You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-05 - 05:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Sep 05, 2026 00:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
146240998924,35cyclictest241164-21sh00:21:461
146240998729,22cyclictest198615-21latency_hist22:50:031
146237998548,32cyclictest16950irq/224-800000000:00:010
146237998527,15cyclictest178854-21ssh22:07:060
146237998438,41cyclictest16850irq/134-800000000:05:020
146237998438,41cyclictest16850irq/134-800000000:05:010
14623799842,37cyclictest203734-21ssh23:00:320
14624099833,41cyclictest151789-21date20:55:031
14624099833,41cyclictest151789-21date20:55:021
146237998128,10cyclictest195875-21ssh22:43:540
146237998127,11cyclictest167793-21ssh21:43:110
146237998125,11cyclictest248793-21ssh00:38:410
14624099801,76cyclictest0-21swapper/100:00:031
146237998040,35cyclictest16850irq/134-800000023:15:010
146237998040,35cyclictest16850irq/134-800000023:15:000
146237998017,17cyclictest16850irq/134-800000000:17:050
14624099791,66cyclictest0-21swapper/119:55:011
146237997929,10cyclictest309-21in:imuxsock20:35:030
146237997929,10cyclictest309-21in:imuxsock20:35:020
14624099772,43cyclictest247772-21ssh00:36:091
14623799773,8cyclictest198622-21ssh22:50:010
14623799772,37cyclictest187656-21ssh22:26:000
146240997634,38cyclictest205125-21sh23:03:541
146237997629,10cyclictest193350-21ssh22:38:240
146237997627,13cyclictest190286-21ssh22:31:410
146237997528,14cyclictest233739-21ssh00:05:350
14623799752,39cyclictest215351-21ssh23:25:430
146240997419,26cyclictest23-21ksoftirqd/122:19:121
146240997419,26cyclictest23-21ksoftirqd/122:19:111
146247997320,47cyclictest275-21dbus-daemon23:30:002
146237997326,9cyclictest169415-21ssh21:46:320
146237997324,17cyclictest16850irq/134-800000022:15:020
146240997141,26cyclictest233130-21sshd00:04:351
146240997141,26cyclictest233130-21sshd00:04:341
146237997135,4cyclictest16850irq/134-800000022:05:010
146237997116,14cyclictest220070-21ssh23:36:150
14624099702,33cyclictest155209-21ssh21:15:291
14624099701,41cyclictest173666-21ssh21:55:361
146237997037,30cyclictest16950irq/224-800000019:37:240
146237997027,12cyclictest186509-21ssh22:23:320
146237996918,17cyclictest16850irq/134-800000023:15:080
146247996831,32cyclictest3467602-21systemd-journal23:47:332
146247996825,39cyclictest158560-21sh21:23:052
14624799682,18cyclictest161690-21ssh21:30:012
146240996821,42cyclictest186177-21rm22:22:511
14624099681,62cyclictest166299-21munin-run21:40:001
146237996824,14cyclictest207683-21ssh23:09:380
14623799682,36cyclictest151249-21latency_hist20:45:020
14624099671,61cyclictest151859-21sshd20:56:251
146237996720,9cyclictest156986-21ssh21:19:490
146247996627,35cyclictest173096-21sh21:54:532
14624099661,61cyclictest214225-21ssh23:23:391
146247996520,39cyclictest187286-21cat22:25:012
146240996521,40cyclictest152427-21diskmemload22:37:381
14624099651,60cyclictest244084-21ssh00:28:271
14624099651,60cyclictest226628-21ssh23:50:151
14624099651,60cyclictest195396-21ssh22:42:431
14624099651,60cyclictest190763-21ssh22:32:521
14624099651,60cyclictest180663-21ssh22:10:501
14624099651,60cyclictest172318-21ssh21:53:001
14624099651,60cyclictest162307-21ssh21:31:081
14624099651,60cyclictest162307-21ssh21:31:081
14624099651,60cyclictest157689-21ssh21:21:001
14624099651,60cyclictest153875-21ssh21:12:581
14624099651,60cyclictest153875-21ssh21:12:571
14624099651,59cyclictest3467602-21systemd-journal23:43:441
14624099651,59cyclictest152135-21sshd21:00:251
14623799653,27cyclictest152436-21ssh21:10:010
14625299643,42cyclictest3467602-21systemd-journal21:10:033
146247996429,31cyclictest205541-21cron23:04:592
146240996422,38cyclictest146664-21cron19:16:591
14624099641,59cyclictest220754-21ssh23:37:561
14624099641,59cyclictest217411-21ssh23:30:131
14624099641,59cyclictest215802-21ssh23:26:521
14624099641,59cyclictest207140-21ssh23:08:141
14624099641,59cyclictest199199-21ssh22:50:491
14624099641,59cyclictest188982-21ssh22:29:111
14624099641,59cyclictest169030-21ssh21:45:321
14624099641,59cyclictest160410-21ssh21:26:531
14624099641,58cyclictest224630-21ssh23:46:061
14624099641,58cyclictest210503-21ssh23:15:071
14624099641,58cyclictest208494-21ssh23:11:031
14624099641,58cyclictest208494-21ssh23:11:021
14624099641,58cyclictest178615-21ssh22:06:261
14624099641,57cyclictest235902-21ssh00:10:171
146237996435,3cyclictest16950irq/224-800000021:10:210
146237996435,3cyclictest16950irq/224-800000021:10:210
146237996424,13cyclictest163698-21ssh21:34:260
146237996424,13cyclictest163698-21ssh21:34:250
146247996320,39cyclictest221635-21latency_hist23:40:032
14624099631,58cyclictest233749-21ssh00:05:371
14624099631,58cyclictest176024-21ssh22:00:561
14624099631,58cyclictest167089-21ssh21:41:281
14624099631,57cyclictest245044-21ssh00:30:071
14624099631,57cyclictest238732-21ssh00:16:401
14623799636,16cyclictest149806-21modprobe20:20:000
146247996224,32cyclictest184765-21sendmail22:20:002
146247996224,32cyclictest184765-21sendmail22:20:002
146247996218,14cyclictest180229-21cat22:10:022
14624099622,56cyclictest3467602-21systemd-journal19:43:191
14624099621,56cyclictest201248-21ssh22:55:091
14624099621,55cyclictest48-21kcompactd019:29:211
14623799622,41cyclictest184766-21munin-plugin-st22:20:000
14623799622,41cyclictest184766-21munin-plugin-st22:20:000
146247996120,37cyclictest30-21ksoftirqd/200:04:412
146247996120,37cyclictest30-21ksoftirqd/200:04:412
146247996118,7cyclictest175379-21rm21:59:512
14624099612,54cyclictest150363-21munin-run20:30:011
14624099611,55cyclictest364-21snmpd19:34:391
14624099611,55cyclictest3467602-21systemd-journal19:21:041
14624099611,55cyclictest148745-21sshd19:56:461
14624099611,55cyclictest148745-21sshd19:56:451
146237996135,20cyclictest159365-21ssh21:25:000
146237996127,12cyclictest217857-21ssh23:31:120
14624799602,16cyclictest181886-21ssh22:13:422
14624799593,15cyclictest200514-21ssh22:53:562
146247995920,34cyclictest210427-21latency_hist23:15:032
146247995920,34cyclictest210427-21latency_hist23:15:022
146247995917,5cyclictest239309-21sh00:18:022
146247995915,10cyclictest148222-21cat19:45:032
146247995912,7cyclictest247193-21sed00:35:022
14624099591,54cyclictest150-21pr/ttyS219:39:011
14624099591,53cyclictest48-21kcompactd020:47:021
14624099591,53cyclictest364-21snmpd20:36:021
14624099591,53cyclictest364-21snmpd20:36:021
14624099591,53cyclictest149774-21sshd20:16:251
146237995918,19cyclictest16850irq/134-800000023:51:070
146237995916,12cyclictest222061-21ssh23:40:250
146247995827,28cyclictest30-21ksoftirqd/222:57:362
146247995823,29cyclictest229234-21sh23:56:062
14624799582,18cyclictest196777-21ssh22:45:312
14624799582,17cyclictest3467602-21systemd-journal00:14:202
14624799582,15cyclictest248866-21ssh00:38:492
146247995819,33cyclictest150841-21cron20:40:002
146247995819,33cyclictest150841-21cron20:40:002
14624099581,52cyclictest150061-21sshd20:20:251
14624099581,52cyclictest149067-21munin-run20:05:001
14623799581,11cyclictest159738-21ssh21:25:190
14624799573,16cyclictest3467602-21systemd-journal20:35:002
146247995723,29cyclictest191819-21latency_hist22:35:022
146247995722,31cyclictest241307-21sh00:22:052
14624799572,17cyclictest166248-21sh21:40:002
14624799572,16cyclictest1-21systemd19:50:002
14624099573,49cyclictest148283-21munin-run19:50:011
14624099572,50cyclictest3467602-21systemd-journal20:41:511
14624099571,52cyclictest364-21snmpd20:32:281
146237995718,18cyclictest16850irq/134-800000000:27:300
14625299569,6cyclictest131rcu_preempt21:32:183
14625299569,6cyclictest131rcu_preempt21:32:183
14624799563,17cyclictest235560-21munin-plugin-st00:09:592
14624799562,16cyclictest168576-21ssh21:45:002
14624799562,13cyclictest152545-21cat21:10:032
146247995618,5cyclictest152427-21diskmemload00:27:072
146247995614,6cyclictest169646-21rm21:47:012
14624099561,51cyclictest149519-21sshd20:10:251
146237995630,22cyclictest3467602-21systemd-journal22:54:590
146237995618,18cyclictest16850irq/134-800000021:35:250
146237995618,17cyclictest16850irq/134-800000023:24:440
146237995617,6cyclictest16850irq/134-800000021:50:230
146237995617,18cyclictest16850irq/134-800000021:55:150
146237995617,18cyclictest16850irq/134-800000000:20:350
146237995617,17cyclictest16850irq/134-800000000:11:580
146252995510,5cyclictest131rcu_preempt23:51:493
146247995546,6cyclictest150893-21kworker/2:121:16:062
14624799553,14cyclictest176297-21ssh22:01:302
146247995523,28cyclictest207878-21latency_hist23:10:032
14624799552,18cyclictest214469-21ssh23:24:092
14624799552,18cyclictest192517-21ssh22:36:292
14624799552,16cyclictest148834-21cat20:00:012
14624799552,16cyclictest148834-21cat20:00:012
146247995521,31cyclictest212547-21sendmail23:20:002
146247995516,4cyclictest217742-21rm23:30:532
146247995515,36cyclictest149824-21latency_hist20:20:002
14624799551,15cyclictest187826-21ssh22:26:232
14625299545,8cyclictest37-21ksoftirqd/322:00:003
14625299545,10cyclictest37-21ksoftirqd/300:00:003
146252995438,12cyclictest180142-21cron22:10:003
146252995436,13cyclictest1-21systemd21:00:003
146252995417,7cyclictest131rcu_preempt00:35:023
146247995424,26cyclictest1-21systemd19:55:012
14624799542,10cyclictest151883-21modprobe20:59:592
14624099541,48cyclictest364-21snmpd19:11:011
14624799532,16cyclictest3467602-21systemd-journal23:50:032
14624799532,15cyclictest150112-21munin-run20:25:012
146247995321,30cyclictest30-21ksoftirqd/223:40:202
146247995319,9cyclictest159230-1kworker/u9:021:33:522
146247995319,9cyclictest159230-1kworker/u9:021:33:522
14623799535,11cyclictest150236-21cat20:25:030
146237995344,5cyclictest16950irq/224-800000023:50:000
14623799533,41cyclictest146698-21modprobe19:20:010
146252995213,6cyclictest131rcu_preempt00:09:023
14624799521,15cyclictest195274-21ssh22:42:282
14624099522,47cyclictest149316-21munin-run20:10:011
14624099521,47cyclictest3467602-21systemd-journal21:05:081
14625299519,6cyclictest131rcu_preempt22:19:373
14625299519,6cyclictest131rcu_preempt22:19:363
146252995114,6cyclictest131rcu_preempt23:38:323
14624799513,15cyclictest147771-21modprobe19:40:012
14624799512,15cyclictest154786-21unin-run21:15:012
14624799512,15cyclictest154786-21unin-run21:15:002
14624799512,15cyclictest150350-21munin-run20:29:592
146237995123,13cyclictest245547-21ssh00:31:210
146252995013,5cyclictest131rcu_preempt23:12:003
146252995013,5cyclictest131rcu_preempt23:12:003
14625299501,13cyclictest37-21ksoftirqd/320:00:013
14625299501,13cyclictest37-21ksoftirqd/320:00:013
14625299498,6cyclictest131rcu_preempt00:37:453
14625299497,8cyclictest37-21ksoftirqd/321:25:013
14625299496,5cyclictest131rcu_preempt21:47:563
146252994940,6cyclictest37-21ksoftirqd/300:02:463
146252994940,6cyclictest37-21ksoftirqd/300:02:453
146252994912,5cyclictest131rcu_preempt22:20:333
14624799492,17cyclictest3467602-21systemd-journal19:31:242
146240994921,23cyclictest146367-21latency_hist19:10:031
146237994934,12cyclictest1-21systemd19:15:010
14625299488,9cyclictest37-21ksoftirqd/323:01:233
14625299488,11cyclictest37-21ksoftirqd/323:45:013
14625299487,10cyclictest37-21ksoftirqd/323:27:593
146252994836,8cyclictest37-21ksoftirqd/300:15:223
146252994832,12cyclictest3467602-21systemd-journal19:40:003
14625299481,8cyclictest242488-21unin-run00:25:013
14624799482,14cyclictest146909-21latency_hist19:20:022
14624799482,13cyclictest151701-21wc20:55:022
14624799482,13cyclictest151701-21wc20:55:022
146237994827,15cyclictest275-21dbus-daemon20:10:010
146237994821,12cyclictest202927-21ssh22:59:110
14625299477,9cyclictest37-21ksoftirqd/323:48:433
14625299477,8cyclictest37-21ksoftirqd/322:10:373
14625299476,9cyclictest37-21ksoftirqd/322:02:303
14625299473,7cyclictest37-21ksoftirqd/321:51:573
146252994737,7cyclictest37-21ksoftirqd/323:25:013
146252994733,11cyclictest37-21ksoftirqd/322:50:253
14625299473,10cyclictest37-21ksoftirqd/322:39:593
146252994729,13cyclictest191692-21cron22:35:003
14625299472,6cyclictest237866-21ssh00:15:013
146247994738,7cyclictest30-21ksoftirqd/219:25:012
14624799473,13cyclictest149167-21latency_hist20:05:022
14624799472,14cyclictest151448-21grep20:50:012
146237994740,4cyclictest16950irq/224-800000020:59:590
14625299469,5cyclictest131rcu_preempt19:35:023
14625299464,9cyclictest37-21ksoftirqd/322:57:133
14625299463,6cyclictest131rcu_preempt21:12:243
14625299463,6cyclictest131rcu_preempt21:12:233
14625299461,8cyclictest0-21swapper/322:41:533
146252994612,6cyclictest131rcu_preempt20:15:033
14624799463,5cyclictest131rcu_preempt20:45:002
14624799461,15cyclictest146246-21grep19:10:012
14625299458,5cyclictest131rcu_preempt23:35:013
146252994533,9cyclictest37-21ksoftirqd/300:26:193
14625299453,10cyclictest37-21ksoftirqd/321:27:423
146252994530,12cyclictest37-21ksoftirqd/322:49:003
14625299451,39cyclictest275-21dbus-daemon19:25:003
146252994512,6cyclictest131rcu_preempt23:16:443
146252994512,6cyclictest131rcu_preempt21:38:283
14623799459,10cyclictest152184-21systemd-run21:05:000
146237994538,4cyclictest16950irq/224-800000020:30:000
14625299447,10cyclictest37-21ksoftirqd/322:29:483
146252994435,7cyclictest37-21ksoftirqd/320:25:033
146252994433,8cyclictest146688-21cron19:20:013
146252994415,9cyclictest37-21ksoftirqd/321:41:403
146252994413,6cyclictest131rcu_preempt23:10:013
14624799442,15cyclictest1-21systemd19:30:012
146237994431,9cyclictest1-21systemd20:00:010
146237994431,9cyclictest1-21systemd20:00:000
14625299439,6cyclictest131rcu_preempt19:30:013
146252994331,8cyclictest3467602-21systemd-journal21:17:253
146252994312,6cyclictest131rcu_preempt20:45:003
146237994319,17cyclictest3467602-21systemd-journal19:50:000
14624799421,14cyclictest152233-21grep21:05:022
146237994230,9cyclictest275-21dbus-daemon20:05:000
146252994127,11cyclictest37-21ksoftirqd/319:46:583
14625299411,5cyclictest0-21swapper/319:43:263
146237994132,6cyclictest19350irq/34-mmc120:32:230
146237994129,9cyclictest12-21ksoftirqd/019:35:010
14623799411,13cyclictest148018-21munin-run19:45:010
14625299409,5cyclictest131rcu_preempt20:10:023
146252994027,9cyclictest309-21in:imuxsock21:02:253
146252994025,12cyclictest37-21ksoftirqd/320:05:003
146237994028,9cyclictest12-21ksoftirqd/019:30:010
146237994028,8cyclictest3467602-21systemd-journal19:53:260
146237994027,8cyclictest309-21in:imuxsock20:15:000
146252993931,5cyclictest150859-21sendmail20:40:013
146252993931,5cyclictest150859-21sendmail20:40:003
146252993927,10cyclictest37-21ksoftirqd/320:45:253
146252993825,10cyclictest37-21ksoftirqd/320:30:003
146247993830,5cyclictest4098005-21kworker/2:220:15:012
146237993827,8cyclictest151373-21cron20:50:000
146252993726,8cyclictest148652-21sed19:55:013
146237993728,7cyclictest12-21ksoftirqd/020:55:020
146237993728,7cyclictest12-21ksoftirqd/020:55:010
14625299367,6cyclictest131rcu_preempt20:35:013
146252993626,7cyclictest37-21ksoftirqd/320:20:003
146247993627,5cyclictest146437-21modprobe19:15:002
146237993626,7cyclictest146278-21cut19:10:010
14625299352,10cyclictest3467718-21lldpd20:51:113
14625299352,10cyclictest3467718-21lldpd20:51:103
14625299349,6cyclictest37-21ksoftirqd/319:11:103
146247993412,17cyclictest149404-21latency_hist20:10:012
14623799343,7cyclictest0-21swapper/019:22:350
146252992822,4cyclictest146261-21latency_hist19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional