You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 02:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Fri Mar 27, 2026 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
387324738,0sleep30-21swapper/318:56:213
411523430,0sleep20-21swapper/218:59:082
391123430,0sleep10-21swapper/118:56:431
399223228,0sleep00-21swapper/018:57:380
42809972,0cyclictest7190-21idleruntime-cro20:15:142
42799975,0cyclictest0-21swapper/122:17:391
42799973,0cyclictest3556-21lldpd19:33:291
42799972,0cyclictest15891-21awk20:45:281
42799970,0cyclictest0-21swapper/123:46:531
42809966,0cyclictest271ktimersoftd/223:09:362
42809964,0cyclictest22650-21iostat_ios22:55:292
42809963,0cyclictest2352-21sh21:00:142
42809962,0cyclictest30252-21munin-node22:20:392
42809962,0cyclictest21916-21munin-node22:35:372
42809960,0cyclictest0-21swapper/200:22:072
42799964,0cyclictest0-21swapper/100:13:041
42799962,0cyclictest6156-21sadc19:50:141
42799962,0cyclictest4775-21awk00:25:271
42799962,0cyclictest3514-21snmpd20:37:321
42799962,0cyclictest3514-21snmpd19:15:011
42799962,0cyclictest3328-21uptime23:00:361
42799962,0cyclictest30757-21awk22:40:271
42799962,0cyclictest28650-21sadc21:40:141
42799962,0cyclictest25302-21awk19:15:271
42799962,0cyclictest2359-21diskmemload23:54:481
42799962,0cyclictest22926-21awk22:35:291
42799961,0cyclictest9197-21sh21:02:001
42799961,0cyclictest5801-21fschecks_time23:05:271
42799961,0cyclictest4134-21latency_hist00:05:131
42799961,0cyclictest32664-21rm21:15:311
42799961,0cyclictest31563-21kthreadcore20:55:301
42799961,0cyclictest2716-21munin-node23:20:361
42799961,0cyclictest26308-21awk19:40:291
42799961,0cyclictest23874-21kthreadcore23:35:301
42799961,0cyclictest22649-21cat23:15:281
42799961,0cyclictest17804-21cut22:50:311
42799961,0cyclictest1587-21cat19:20:311
42799961,0cyclictest1564-21cut22:20:321
42799961,0cyclictest1263-21tune2fs20:10:281
42799961,0cyclictest0-21swapper/123:40:301
42799961,0cyclictest0-21swapper/123:40:301
42799960,0cyclictest8238-21munin-node20:15:281
42799960,0cyclictest7172-21kthreadcore19:25:321
42799960,0cyclictest7107-21kthreadcore22:45:301
42799960,0cyclictest5600-21fschecks_time23:25:251
42799960,0cyclictest32045-21cut00:00:301
42799960,0cyclictest27369-21ssh23:57:211
42799960,0cyclictest25782-21sh21:32:321
42799960,0cyclictest22818-21irqrtprio22:55:291
42799960,0cyclictest22226-21kthreadcore21:10:311
42799960,0cyclictest21254-21sh22:12:591
42799960,0cyclictest20747-21iostat_ios20:00:291
42799960,0cyclictest19770-21iostat_ios19:35:291
42799960,0cyclictest1878-21unixbench_multi00:20:391
42799960,0cyclictest15758-21cut19:05:401
42799960,0cyclictest15587-21cut22:30:301
42799960,0cyclictest13616-21grep21:25:301
42799960,0cyclictest0-21swapper/123:10:301
42789966,0cyclictest3514-21snmpd22:35:340
339960,0migration/3983-21kthreadcore22:20:323
339960,0migration/329493-21sh22:18:493
339960,0migration/329479-21sh22:39:483
339960,0migration/326217-21munin-node00:15:393
339960,0migration/32572-21sh00:22:173
339960,0migration/324737-21munin-node19:35:373
339960,0migration/320416-21sh23:34:263
339960,0migration/318645-21sh00:11:193
339960,0migration/318228-21sh21:28:573
29715060,0irq/26-eth0-rx-4669-21aten_r1power_en23:45:273
29715060,0irq/26-eth0-rx-32153-21ls20:05:403
149960,0migration/031449-21kthreadcore20:55:300
149960,0migration/027941-21sh23:17:080
149960,0migration/027666-21sh23:37:210
149960,0migration/020275-21sh23:13:060
149960,0migration/012872-21sh22:06:530
149960,0migration/012041-21sh23:49:230
42809954,0cyclictest21201-21munin-node23:55:282
42809954,0cyclictest15078-21aten2.4-expect20:45:242
42809953,0cyclictest9221-21sh21:57:292
42809953,0cyclictest8492-21hostname20:40:242
42809953,0cyclictest8323-21if_eth120:15:292
42809953,0cyclictest7736-21kthreadcore00:05:312
42809953,0cyclictest699-21df_inode21:50:252
42809953,0cyclictest6734-21df19:50:252
42809953,0cyclictest5984-21aten2.4-expect22:25:252
42809953,0cyclictest5853-21expr22:45:262
42809953,0cyclictest5827-21diskstats19:25:282
42809953,0cyclictest5546-21sh23:25:252
42809953,0cyclictest4891-21df_inode19:00:262
42809953,0cyclictest4848-21if_eth100:25:282
42809953,0cyclictest4815-21cut21:00:292
42809953,0cyclictest4679-21aten2.4-expect23:45:272
42809953,0cyclictest346-21munin-run20:10:142
42809953,0cyclictest32513-21forks19:45:262
42809953,0cyclictest31442-21df19:20:262
42809953,0cyclictest31027-21rm21:45:582
42809953,0cyclictest29871-21ssh22:40:192
42809953,0cyclictest29712-21sh23:20:272
42809953,0cyclictest29593-21aten2.4-expect23:00:232
42809953,0cyclictest29342-21df_inode23:40:252
42809953,0cyclictest29342-21df_inode23:40:252
42809953,0cyclictest2868-21aten2.4-expect21:20:242
42809953,0cyclictest28504-21grep00:00:142
42809953,0cyclictest27936-21fschecks_time20:30:262
42809953,0cyclictest27292-21aten2.4-expect21:15:242
42809953,0cyclictest26974-21sh21:35:222
42809953,0cyclictest25793-21df_inode19:40:272
42809953,0cyclictest25308-21if_err_eth119:15:272
42809953,0cyclictest23742-21cat23:15:282
42809953,0cyclictest22985-21perl22:15:282
42809953,0cyclictest22056-21df_abs20:50:262
42809953,0cyclictest21359-21sed23:35:252
42809953,0cyclictest21282-21tune2fs20:25:272
42809953,0cyclictest2067-21grep20:35:282
42809953,0cyclictest20602-21df00:15:272
42809953,0cyclictest20348-21perl20:00:262
42809953,0cyclictest19168-21cpuspeed_turbos21:10:262
42809953,0cyclictest19164-21diskstats19:35:262
42809953,0cyclictest18938-21sh21:30:222
42809953,0cyclictest17800-21basename19:10:232
42809953,0cyclictest14529-21sh22:50:292
42809953,0cyclictest14446-21diskstats20:20:262
42809953,0cyclictest14443-21aten2.4-expect22:10:282
42809953,0cyclictest14300-21iostat_ios23:10:292
42809953,0cyclictest14283-21sed22:30:252
42809953,0cyclictest12983-21aten2.4-expect23:30:242
42809953,0cyclictest12895-21perl00:10:282
42809953,0cyclictest12852-21crond19:55:142
42809953,0cyclictest12475-21kthreadcore19:05:392
42809953,0cyclictest12065-21date19:30:142
42809953,0cyclictest11137-21sed21:05:252
42809953,0cyclictest11038-21ssh22:02:092
42809953,0cyclictest10831-21date21:25:142
42809952,0cyclictest12766-21ssh22:06:342
42809951,0cyclictest0-21swapper/223:50:272
42809951,0cyclictest0-21swapper/221:42:092
42799953,0cyclictest0-21swapper/120:50:191
42799952,0cyclictest3514-21snmpd20:40:171
42799950,0cyclictest8446-21cat21:55:141
42799950,0cyclictest6390-21chrt19:50:211
42799950,0cyclictest5780-21latency_hist22:25:141
42799950,0cyclictest44862sleep119:00:221
42799950,0cyclictest32682-21cut21:50:141
42799950,0cyclictest30700-21date21:45:141
42799950,0cyclictest28845-21date21:40:141
42799950,0cyclictest2749-21ssh21:20:141
42799950,0cyclictest27377-21date20:30:141
42799950,0cyclictest26348-21sed20:05:141
42799950,0cyclictest20677-21cat20:25:141
42799950,0cyclictest20203-21sh00:15:141
42799950,0cyclictest13976-21sed20:20:141
42799950,0cyclictest13001-21latency_hist19:55:141
42799950,0cyclictest12887-21cat23:30:141
42799950,0cyclictest12188-21ssh22:05:141
42799950,0cyclictest10697-21ssh21:05:141
42799950,0cyclictest10315-21latency_hist22:00:141
42789955,0cyclictest0-21swapper/023:50:270
339950,0migration/39695-21kthreadcore22:45:313
339950,0migration/38267-21kthreadcore23:25:293
339950,0migration/35727-21kthreadcore21:20:313
339950,0migration/35007-21irqcore00:25:293
339950,0migration/34938-21kthreadcore21:50:303
339950,0migration/34721-21kthreadcore20:35:323
339950,0migration/329460-21fschecks_count23:40:263
339950,0migration/329460-21fschecks_count23:40:263
339950,0migration/328669-21cpuspeed_turbos20:55:253
339950,0migration/323468-21kthreadcore23:15:283
339950,0migration/320773-21aten2.4-expect20:25:253
339950,0migration/320131-21df_abs20:00:263
339950,0migration/316323-21kthreadcore19:30:313
339950,0migration/315478-21irqcore22:10:333
339950,0migration/315275-21cpuspeed_turbos20:45:263
29725050,0irq/27-eth0-tx-8385-21munin-node20:40:250
29725050,0irq/27-eth0-tx-4495-21munin-node19:00:350
29725050,0irq/27-eth0-tx-31117-21munin-node19:20:320
29725050,0irq/27-eth0-tx-27392-21munin-node20:30:320
29725050,0irq/27-eth0-tx-0-21swapper/020:15:250
29715050,0irq/26-eth0-rx-6161-21netstat20:10:343
29715050,0irq/26-eth0-rx-24656-21ps21:10:343
29715050,0irq/26-eth0-rx-19799-21ssh23:11:413
29715050,0irq/26-eth0-rx-18495-21ssh21:09:513
29715050,0irq/26-eth0-rx-16630-21kthreadcore23:50:323
29715050,0irq/26-eth0-rx-15351-21cut19:05:403
29715050,0irq/26-eth0-rx-0-21swapper/323:02:353
29715050,0irq/26-eth0-rx-0-21swapper/322:53:463
29715050,0irq/26-eth0-rx-0-21swapper/322:40:563
29715050,0irq/26-eth0-rx-0-21swapper/322:25:323
29715050,0irq/26-eth0-rx-0-21swapper/322:05:453
29715050,0irq/26-eth0-rx-0-21swapper/321:31:183
29715050,0irq/26-eth0-rx-0-21swapper/300:05:253
29715050,0irq/26-eth0-rx-0-21swapper/300:00:263
29715050,0irq/26-eth0-rx-0-21swapper/300:00:133
149950,0migration/07909-21kthreadcore23:05:310
149950,0migration/07071-21kthreadcore00:25:300
149950,0migration/07019-21kthreadcore22:45:300
149950,0migration/06241-21irqrtprio23:25:280
149950,0migration/04681-21kthreadcore19:45:300
149950,0migration/04355-21kthreadcore21:00:290
149950,0migration/03685-21taskset23:01:290
149950,0migration/03453-21kthreadcore20:35:320
149950,0migration/03421-21fschecks_time21:20:260
149950,0migration/030241-21kthreadcore00:20:310
149950,0migration/030145-21cpuspeed_turbos22:40:250
149950,0migration/029930-21taskset22:20:080
149950,0migration/027970-21idleruntime21:15:260
149950,0migration/026986-21fschecks_time20:05:290
149950,0migration/026915-21taskset20:27:250
149950,0migration/026312-21sh21:34:060
149950,0migration/025463-21munin-node19:15:280
149950,0migration/024342-21kthreadcore20:50:310
149950,0migration/021592-21if_eth123:55:290
149950,0migration/020607-21latency_hist20:25:140
149950,0migration/020464-21gltestperf20:00:270
149950,0migration/019407-21kthreadcore19:10:280
149950,0migration/016064-21kthreadcore22:50:310
149950,0migration/014685-21kthreadcore23:30:300
149950,0migration/012138-21taskset22:25:440
149950,0migration/011657-21taskset19:28:010
149950,0migration/010863-21taskset22:01:380
42789944,0cyclictest4915-21kworker/0:121:56:190
42789943,0cyclictest5924-21kworker/0:022:30:350
42789943,0cyclictest3514-21snmpd19:31:320
42789943,0cyclictest3241-21irqbalance22:57:220
42789943,0cyclictest27973-21sadc00:20:120
42789943,0cyclictest15986-21awk20:45:290
42789943,0cyclictest15314-21awk22:10:320
42789943,0cyclictest11860-21awk21:25:280
339940,0migration/39635-21kthreadcore19:00:313
339940,0migration/35419-21taskset19:25:243
339940,0migration/330884-21kthreadcore23:20:303
339940,0migration/329264-21kthreadcore20:30:303
339940,0migration/328310-21kthreadcore19:40:313
339940,0migration/328076-21kthreadcore19:15:303
339940,0migration/324722-21kthreadcore20:50:313
339940,0migration/316529-21kthreadcore19:55:343
29725040,0irq/27-eth0-tx-0-21swapper/019:05:250
29715040,0irq/26-eth0-rx-0-21swapper/323:37:283
29715040,0irq/26-eth0-rx-0-21swapper/323:05:323
29715040,0irq/26-eth0-rx-0-21swapper/322:55:443
29715040,0irq/26-eth0-rx-0-21swapper/322:32:023
29715040,0irq/26-eth0-rx-0-21swapper/322:00:223
29715040,0irq/26-eth0-rx-0-21swapper/321:56:003
29715040,0irq/26-eth0-rx-0-21swapper/321:45:463
29715040,0irq/26-eth0-rx-0-21swapper/321:41:013
29715040,0irq/26-eth0-rx-0-21swapper/321:36:043
29715040,0irq/26-eth0-rx-0-21swapper/321:15:233
29715040,0irq/26-eth0-rx-0-21swapper/321:01:153
149940,0migration/03292-21runrttasks00:03:210
149940,0migration/025732-21df_abs19:40:270
149940,0migration/020832-21kthreadcore21:10:300
149940,0migration/01660-21kernelversion21:50:290
149940,0migration/013356-21irqcore00:10:310
42789933,0cyclictest5924-21kworker/0:023:40:190
42789933,0cyclictest5924-21kworker/0:023:40:190
42789933,0cyclictest5924-21kworker/0:023:20:170
42789933,0cyclictest5924-21kworker/0:000:05:180
42789933,0cyclictest4915-21kworker/0:122:20:210
42789933,0cyclictest4915-21kworker/0:121:45:150
42789933,0cyclictest4915-21kworker/0:121:40:290
42789933,0cyclictest4915-21kworker/0:121:05:170
42789933,0cyclictest4915-21kworker/0:120:10:210
42789933,0cyclictest4915-21kworker/0:119:50:210
42789932,0cyclictest3514-21snmpd21:35:250
42789930,0cyclictest0-21swapper/019:55:290
42789930,0cyclictest0-21swapper/019:35:270
29715030,0irq/26-eth0-rx-0-21swapper/320:40:253
29715030,0irq/26-eth0-rx-0-21swapper/320:20:253
29715030,0irq/26-eth0-rx-0-21swapper/320:15:253
29715020,0irq/26-eth0-rx-0-21swapper/319:50:213
29715020,0irq/26-eth0-rx-0-21swapper/319:46:363
29715020,0irq/26-eth0-rx-0-21swapper/319:20:283
29715020,0irq/26-eth0-rx-0-21swapper/319:10:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional