You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 13:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Nov 19, 2025 00:43:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27966211376,9sleep30-21swapper/319:08:333
26060210288,9sleep10-21swapper/119:05:171
2804228664,6sleep20-21swapper/219:09:322
2781028463,9sleep00-21swapper/019:06:340
74322610,1sleep125-21ksoftirqd/100:23:341
308342580,2sleep230833-21ssh23:30:172
141082520,0sleep10-21swapper/123:40:351
2817999290,29cyclictest0-21swapper/223:51:432
2817999250,25cyclictest0-21swapper/223:17:152
2817999250,25cyclictest0-21swapper/222:33:322
2818399232,2cyclictest224-21systemd-journal23:44:593
2817999225,5cyclictest12308-21tr23:40:182
2817999210,21cyclictest0-21swapper/223:37:102
2817999210,21cyclictest0-21swapper/222:09:472
28183992014,4cyclictest16884-21grep00:30:313
2817999200,20cyclictest0-21swapper/221:54:562
2817999200,20cyclictest0-21swapper/221:54:562
28174992018,2cyclictest11792-21kworker/u8:100:14:331
2817199207,6cyclictest25563-21munin-run22:20:010
28171992015,5cyclictest0-21swapper/021:19:130
2818399190,2cyclictest0-21swapper/300:23:463
2817999191,6cyclictest0-21swapper/200:35:532
2817999190,19cyclictest0-21swapper/223:02:532
2817499191,0cyclictest141rcu_preempt23:50:501
2817499190,1cyclictest0-21swapper/122:22:321
2818399183,3cyclictest0-21swapper/322:20:213
2817999180,18cyclictest0-21swapper/200:11:172
2817999180,17cyclictest0-21swapper/223:55:242
2817999170,17cyclictest0-21swapper/200:01:022
2817999170,16cyclictest0-21swapper/223:10:042
2817999170,0cyclictest0-21swapper/221:44:202
2817499178,4cyclictest224-21systemd-journal20:35:011
28174991716,1cyclictest15937-21ssh22:35:151
28174991715,1cyclictest0-21swapper/120:40:191
2817499170,1cyclictest141rcu_preempt22:18:551
2817199173,8cyclictest0-21swapper/021:10:150
223252170,0sleep10-21swapper/120:05:011
223252170,0sleep10-21swapper/120:05:001
2818399163,2cyclictest141rcu_preempt23:50:013
2817999160,0cyclictest0-21swapper/221:25:462
28174991614,1cyclictest25-21ksoftirqd/122:05:371
2818399155,9cyclictest7796-21munin-node23:45:123
28183991514,1cyclictest0-21swapper/321:50:173
28183991514,1cyclictest0-21swapper/321:50:163
2817999152,1cyclictest20835-21diskmemload22:26:302
28179991513,1cyclictest32-21ksoftirqd/200:15:202
2817999150,1cyclictest0-21swapper/200:31:452
2817999150,15cyclictest0-21swapper/221:22:362
2817999150,14cyclictest0-21swapper/222:44:352
28174991513,1cyclictest241rcuc/122:10:181
28174991513,1cyclictest0-21swapper/122:25:141
28174991512,2cyclictest25-21ksoftirqd/100:15:271
28174991512,1cyclictest25-21ksoftirqd/121:00:161
2817499150,3cyclictest0-21swapper/100:27:101
2817199159,5cyclictest2874-21kernelversion20:30:200
2817199153,8cyclictest5020-21/usr/sbin/munin22:50:140
2817199152,7cyclictest20844-21munin-run21:09:590
28171991514,1cyclictest0-21swapper/023:23:560
28183991411,2cyclictest21627-21sort20:00:303
28183991411,2cyclictest21627-21sort20:00:293
28179991411,2cyclictest6467-21basename20:40:002
28179991410,3cyclictest1226-21grep20:25:272
28174991412,1cyclictest241rcuc/121:57:171
28174991410,3cyclictest6423-21cron20:40:001
28174991410,3cyclictest25-21ksoftirqd/122:01:531
2817499140,2cyclictest0-21swapper/123:32:051
2817199145,4cyclictest4523-21ssh22:50:000
2817199141,5cyclictest224-21systemd-journal20:41:560
28171991413,1cyclictest0-21swapper/022:00:470
28171991412,2cyclictest0-21swapper/022:09:360
28171991412,1cyclictest5315-21users21:20:280
28171991410,3cyclictest7744-21snmpd23:12:210
28171991410,3cyclictest19841-21cat20:00:010
28171991410,2cyclictest25111-21tr20:10:180
2817199140,12cyclictest9265-21grep22:30:200
2818399139,3cyclictest29363-21sed21:15:193
2818399139,3cyclictest19298-21grep19:55:273
2818399136,3cyclictest493-21systemd-logind00:11:063
2818399135,5cyclictest7535-21apache223:17:533
2818399132,10cyclictest0-21swapper/323:21:383
2818399131,5cyclictest0-21swapper/321:44:473
28183991310,2cyclictest22924-21sh23:25:163
28183991310,2cyclictest19014-21grep22:15:173
28183991310,2cyclictest11866-21tr21:25:243
28183991310,2cyclictest10775-21grep20:45:283
2817999139,3cyclictest8931-21cut19:35:192
2817999139,3cyclictest20372-21grep21:05:282
2817999138,3cyclictest4138-21wc00:20:222
28179991312,1cyclictest61750irq/125-enp2s0-22:37:382
28179991310,2cyclictest12662-21cron19:45:002
2817999130,13cyclictest0-21swapper/221:15:012
2817499139,3cyclictest15865-21grep23:20:171
2817499136,2cyclictest496-21dbus-daemon20:55:241
2817499132,1cyclictest141rcu_preempt21:52:451
2817499132,1cyclictest141rcu_preempt21:52:451
28174991310,2cyclictest13709-21wc20:54:591
2817199139,3cyclictest24513-21cron21:35:000
2817199139,2cyclictest12375-21grep23:40:180
2817199138,3cyclictest15356-21sh21:50:210
2817199138,3cyclictest15356-21sh21:50:200
2817199131,6cyclictest0-21swapper/020:45:160
28171991311,2cyclictest13-21ksoftirqd/023:55:240
28171991311,1cyclictest20835-21diskmemload00:11:290
28171991310,2cyclictest28230-21sed23:50:280
28171991310,2cyclictest25266-21tr21:35:150
28171991310,2cyclictest19542-21grep23:45:150
28171991310,2cyclictest10420-21fgrep00:25:250
92932120,1sleep30-21swapper/322:30:203
2818399129,2cyclictest9256-21grep21:45:293
2818399129,2cyclictest8472-21tr00:00:163
2818399129,2cyclictest5921-21cut23:35:233
2818399129,2cyclictest3675-21tr21:20:173
2818399129,2cyclictest3047-21grep20:30:213
2818399129,2cyclictest29944-21grep22:00:213
2818399129,2cyclictest27958-21tr20:15:223
2818399129,2cyclictest25674-21cron23:05:003
2818399129,2cyclictest24752-21wc20:10:003
2818399129,2cyclictest16041-21tail19:50:203
2818399128,3cyclictest8196-21grep19:35:163
2818399128,3cyclictest28601-21cron22:00:013
2818399128,3cyclictest25082-21grep22:40:303
2818399128,3cyclictest12252-21head19:40:273
2818399124,4cyclictest224-21systemd-journal23:10:013
2818399124,4cyclictest224-21systemd-journal22:49:593
2818399123,5cyclictest30928-21latency_hist19:15:003
2818399123,4cyclictest31807-21munin-plugin-st21:40:003
2817999129,2cyclictest6464-21tail19:30:202
2817999129,2cyclictest31803-21ssh22:45:252
2817999129,2cyclictest25898-21cron23:49:592
2817999129,2cyclictest23678-21tail23:25:202
2817999129,2cyclictest20318-21tr20:00:192
2817999129,2cyclictest20318-21tr20:00:182
2817999129,2cyclictest13164-21grep19:45:172
2817999129,2cyclictest12631-21grep22:10:232
2817999128,3cyclictest27406-21wc22:20:212
2817999128,3cyclictest19200-21tr21:30:252
2817999128,3cyclictest17824-21wc21:00:272
2817999128,2cyclictest22155-21tail21:10:192
2817999124,3cyclictest493-21systemd-logind22:50:232
2817999123,5cyclictest7545-21apache220:10:152
2817999120,11cyclictest11269-21dump-pmu-power20:50:002
2817999120,11cyclictest0-21swapper/221:48:272
2817499129,2cyclictest9116-21grep23:15:201
2817499129,2cyclictest673-21idleruntime-cro19:20:001
2817499129,2cyclictest6035-21cat23:35:241
2817499129,2cyclictest5186-21grep21:20:271
2817499129,2cyclictest22455-21rm23:01:531
2817499129,2cyclictest20326-21grep21:05:281
2817499129,2cyclictest19023-21ssh23:45:001
2817499129,2cyclictest15313-21grep22:35:001
2817499129,2cyclictest1383-21cut23:55:181
2817499129,2cyclictest11642-21grep19:40:211
2817499128,4cyclictest25-21ksoftirqd/121:15:221
2817499128,3cyclictest18464-21diskstats21:30:191
2817499124,4cyclictest224-21systemd-journal00:40:011
28174991210,1cyclictest8913-21sort00:00:171
2817499120,11cyclictest1000-21cat20:25:251
2817199129,2cyclictest6744-21tail23:35:310
2817199129,2cyclictest4062-21ssh00:20:220
2817199129,2cyclictest2599-21cron00:19:590
2817199129,2cyclictest24105-21grep22:40:210
2817199129,2cyclictest16924-21grep19:50:280
2817199129,2cyclictest16415-21tr21:00:150
2817199129,2cyclictest15648-21cut00:05:160
2817199128,3cyclictest6488-21sed19:30:220
2817199126,3cyclictest141rcu_preempt21:56:440
28171991210,1cyclictest8926-21head19:35:180
28171991210,1cyclictest15554-21apt-config22:35:000
87902110,1sleep00-21swapper/000:00:170
2818399119,1cyclictest18672-21grep21:30:203
2818399118,2cyclictest9154-21ssh22:52:363
2818399118,2cyclictest7476-21chrt19:30:363
2818399118,2cyclictest5041-21ssh22:05:203
2818399118,2cyclictest31925-21uname19:15:213
2818399118,2cyclictest31773-21ssh23:30:263
2818399118,2cyclictest29899-21tr20:20:163
2818399118,2cyclictest2259-21grep22:25:213
2818399118,2cyclictest22411-21sort21:10:213
2818399118,2cyclictest21821-21wc00:35:173
2818399118,2cyclictest19487-21cut21:05:203
2818399118,2cyclictest17264-21chrt21:00:193
2818399118,2cyclictest15867-21ssh22:57:153
2818399118,2cyclictest15839-21expr00:05:173
2818399118,2cyclictest14305-21cat20:55:173
2818399118,2cyclictest13647-21awk20:54:593
2818399118,2cyclictest1076-21cat23:10:223
2818399117,2cyclictest11461-21tr22:10:163
2818399115,2cyclictest0-21swapper/320:40:253
2818399111,5cyclictest224-21systemd-journal19:25:003
2818399110,5cyclictest0-21swapper/300:27:063
2818399110,10cyclictest979-21seq23:55:113
2818399110,10cyclictest0-21swapper/300:15:163
2817999119,1cyclictest0-21swapper/219:55:142
2817999118,2cyclictest8878-21chrt00:25:072
2817999118,2cyclictest29328-21idleruntime-cro20:20:012
2817999118,2cyclictest28732-21sh21:37:082
2817999118,2cyclictest25451-21sh22:19:562
2817999118,2cyclictest25249-21sh21:56:522
2817999118,2cyclictest15383-21sh00:05:152
2817999118,2cyclictest13908-21chrt20:55:142
2817999112,2cyclictest224-21systemd-journal23:22:222
2817999110,10cyclictest3734-21chrt20:31:062
2817499119,2cyclictest28423-21sh22:43:311
2817499118,2cyclictest8396-21tr19:35:161
2817499118,2cyclictest5823-21grep22:50:181
2817499118,2cyclictest32105-21chrt22:45:291
2817499118,2cyclictest28317-21ssh23:28:401
2817499118,2cyclictest27959-21grep20:15:221
2817499118,2cyclictest26681-21cat21:35:271
2817499118,2cyclictest25921-21ssh21:13:071
2817499118,2cyclictest1955-21wc19:20:211
2817499118,2cyclictest16133-21grep00:05:191
2817499115,2cyclictest141rcu_preempt23:11:461
2817499113,6cyclictest13383-21ssh22:55:231
2817499111,4cyclictest0-21swapper/119:45:231
28174991110,1cyclictest20835-21diskmemload00:33:161
2817499110,10cyclictest32689-21expr21:40:171
2817499110,10cyclictest22711-21tr20:05:161
2817199118,2cyclictest9978-21ssh21:46:080
2817199118,2cyclictest4032-21taskset22:26:270
2817199118,2cyclictest32755-21head20:25:170
2817199118,2cyclictest28434-21ssh23:05:440
2817199118,2cyclictest26300-21ssh23:26:490
2817199118,2cyclictest23986-21ssh00:36:170
2817199118,2cyclictest21956-21ssh23:01:220
2817199118,2cyclictest20342-21tr20:00:190
2817199118,2cyclictest20342-21tr20:00:180
2817199118,2cyclictest19310-21ssh00:33:040
2817199118,2cyclictest16879-21ssh22:58:170
2817199118,2cyclictest14549-21sed19:45:290
2817199118,2cyclictest13537-21ssh23:18:330
2817199118,2cyclictest12946-21ssh21:25:510
2817199118,2cyclictest1208-21sed21:40:230
2817199118,2cyclictest1205-21cut19:20:160
2817199117,3cyclictest11152-21grep19:40:170
2817199115,2cyclictest224-21systemd-journal19:15:000
2817199112,3cyclictest24587-21perf20:10:010
2818399108,1cyclictest319512sleep320:25:063
2818399108,1cyclictest26728-21taskset20:12:243
2818399107,2cyclictest6186-21taskset20:37:003
2818399107,2cyclictest4076-21grep19:25:193
2818399107,2cyclictest19756-21ssh22:37:143
2818399107,2cyclictest14598-21grep19:45:293
2817999108,1cyclictest85612sleep220:41:002
2817999108,1cyclictest3754-21expr19:25:172
2817999108,1cyclictest29041-21rm22:00:112
2817999107,2cyclictest32616-21grep19:15:292
2817999107,2cyclictest31369-21chrt20:20:352
2817999107,2cyclictest30641-21taskset19:14:082
2817999107,2cyclictest26855-21sort23:05:192
2817999107,2cyclictest23589-21ntpq20:05:222
2817999107,2cyclictest1695-21grep19:20:192
2817999107,2cyclictest15634-21ssh22:57:022
2817999106,3cyclictest15378-21grep19:50:162
2817999102,4cyclictest224-21systemd-journal20:51:452
2817499108,1cyclictest52632sleep119:27:381
2817499108,1cyclictest30258-21tune2fs20:20:181
2817499108,1cyclictest26701-21chrt20:12:011
2817499108,1cyclictest26082-21ssh23:05:101
2817499108,1cyclictest16391-21chrt19:50:221
2817499108,1cyclictest11342-21tune2fs21:25:201
2817499107,2cyclictest7837-21taskset19:34:271
2817499107,2cyclictest7296-21basename21:45:161
2817499107,2cyclictest19830-21lockfile-create20:00:011
2817499104,6cyclictest25-21ksoftirqd/119:10:131
2817499103,3cyclictest224-21systemd-journal20:50:001
2817199108,1cyclictest3716-21ls19:25:170
2817199108,1cyclictest30202-21apt23:30:130
2817199108,1cyclictest26769-21cat22:20:180
2817199108,1cyclictest133872sleep020:51:550
2817199107,2cyclictest4598-21grep20:35:170
2817199107,2cyclictest31824-21cat19:15:210
2817199107,2cyclictest30654-21cat20:20:210
2817199107,2cyclictest11934-21ssh22:10:170
2817199103,6cyclictest29340-21munin-run20:20:010
2817199103,1cyclictest0-21swapper/021:00:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional