You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-16 - 13:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Thu Jul 16, 2026 00:43:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3312430995945,10cyclictest3378916-21munin-run22:00:010
3312431995234,15cyclictest3427419-21sendmail23:20:011
3312431995234,15cyclictest3427419-21sendmail23:20:011
3312431995138,9cyclictest1-21systemd00:05:011
3312431995138,9cyclictest1-21systemd00:05:011
3312431995025,22cyclictest0-21swapper/119:40:001
3312431994937,8cyclictest3451713-21cups-browsed00:06:321
3312430994942,5cyclictest3318442-21systemd-run19:30:010
3312430994942,5cyclictest3318442-21systemd-run19:30:010
3312431994839,6cyclictest3466971-21latency_hist00:25:021
3312431994820,24cyclictest1-21systemd21:35:331
3312432994739,4cyclictest0-21swapper/222:05:012
3312432994737,7cyclictest3403612-21aten2.4-expect22:40:302
3312431994722,22cyclictest3366780-21kworker/u8:14-xprtiod21:50:541
3312432994638,5cyclictest3412215-21cron22:55:012
331243199461,16cyclictest3386209-21ssh22:10:481
331243199461,16cyclictest3386209-21ssh22:10:481
3312430994632,8cyclictest12650irq/32-eth022:20:450
3312432994530,7cyclictest0-21swapper/222:35:412
3312431994537,5cyclictest3463816-21munin-plugin-st00:20:011
3312431994536,6cyclictest3343924-21systemd-run20:55:021
3312431994535,6cyclictest3436457-21cron23:35:011
3312431994521,21cyclictest0-21swapper/123:50:551
3312431994520,22cyclictest3407496-21mailstats22:45:501
3312430994538,4cyclictest3354983-21aten_r2power_en21:20:310
3312430994538,4cyclictest3354983-21aten_r2power_en21:20:310
3312431994438,3cyclictest0-21swapper/120:20:011
3312431994434,6cyclictest3391034-21ssh22:20:001
331243199443,10cyclictest3401291-21ntpq22:35:461
3312431994418,22cyclictest420-21avahi-daemon23:09:561
331243199441,16cyclictest3410243-21sh22:50:391
331243199441,13cyclictest3355637-21ssh21:20:421
331243199441,13cyclictest3355637-21ssh21:20:421
3312430994433,8cyclictest3470254-21/usr/sbin/munin00:30:350
331243099443,34cyclictest3349496-21ssh21:10:410
3312432994334,6cyclictest341rcuc/222:30:412
3312432994334,5cyclictest0-21swapper/222:55:452
3312431994333,7cyclictest421-21dbus-daemon21:50:011
331243199432,18cyclictest3404216-21ssh22:40:401
3312431994320,20cyclictest0-21swapper/121:10:011
3312430994335,4cyclictest1003-21mta-sts-daemon23:10:070
3312430994324,12cyclictest151ktimers/023:00:390
3312430994321,16cyclictest151ktimers/020:45:430
3312432994233,5cyclictest3321442-21kworker/u8:3+xprtiod19:40:012
3312432994231,8cyclictest3371394-21timerwakeupswit21:45:522
331243199423,8cyclictest3331321-21ntpq20:10:411
3312431994233,3cyclictest0-21swapper/120:35:411
3312431994224,11cyclictest3451698-21systemctl00:00:011
3312431994216,24cyclictest0-21swapper/121:20:011
3312431994213,3cyclictest0-21swapper/120:40:521
3312430994233,6cyclictest0-21swapper/021:40:550
3312432994136,2cyclictest0-21swapper/220:20:012
3312432994130,4cyclictest0-21swapper/200:35:442
3312432994127,9cyclictest420-21avahi-daemon00:01:562
3312432994127,9cyclictest420-21avahi-daemon00:01:562
331243199414,11cyclictest3322308-21ntpq19:40:401
3312431994117,21cyclictest0-21swapper/122:55:381
331243199411,12cyclictest3361744-21ntpq21:30:431
331243199411,12cyclictest3361744-21ntpq21:30:431
331243199411,11cyclictest3348408-21diskmemload23:23:151
3312430994135,4cyclictest3322939-21systemd-run19:45:010
3312430994133,5cyclictest3400136-21systemd22:35:020
3312432994032,5cyclictest3460939-21latency_hist00:15:022
331243199405,11cyclictest428-21mta-sts-daemon23:35:211
331243199405,11cyclictest428-21mta-sts-daemon23:35:211
331243199404,11cyclictest3314885-21ntpq19:15:441
3312431994027,4cyclictest171rcu_preempt23:28:161
3312431994023,13cyclictest0-21swapper/123:02:351
331243199401,15cyclictest3316361-21ntpq19:20:441
331243199401,13cyclictest3467510-21df_inode00:25:331
331243199401,11cyclictest3440711-21ssh23:40:471
331243099408,20cyclictest428-21mta-sts-daemon19:30:410
331243099408,20cyclictest428-21mta-sts-daemon19:30:410
3312430994029,7cyclictest3468570-21unixbench_singl00:25:540
3312430994027,10cyclictest3433468-21kworker/u8:3-xprtiod23:45:010
331243099401,34cyclictest3461953-21ssh00:15:440
3312432993931,5cyclictest3342431-21systemd-run20:50:022
3312432993930,5cyclictest3346912-21cron21:05:012
3312432993930,3cyclictest0-21swapper/223:35:422
3312432993930,3cyclictest0-21swapper/223:35:422
3312432993927,9cyclictest3471582-21unixbench_singl00:30:562
3312432993919,4cyclictest171rcu_preempt23:10:432
331243199393,8cyclictest3470962-21ntpq00:30:431
3312431993931,5cyclictest3319937-21systemd-run19:35:011
3312431993931,5cyclictest3319937-21systemd-run19:35:011
3312431993931,5cyclictest1001-21mta-sts-daemon22:24:551
3312431993931,3cyclictest0-21swapper/122:05:401
331243199392,15cyclictest3443605-21ssh23:45:401
331243199392,12cyclictest3377059-21ssh21:55:461
3312431993917,19cyclictest1001-21mta-sts-daemon20:25:411
3312431993915,2cyclictest0-21swapper/123:10:471
3312431993914,3cyclictest0-21swapper/119:45:311
3312430993928,8cyclictest3431894-21xargs23:25:570
3312432993832,3cyclictest0-21swapper/221:30:462
3312432993832,3cyclictest0-21swapper/221:30:462
3312432993831,4cyclictest3461480-21fschecks_time00:15:332
3312432993829,5cyclictest3315433-21sendmail19:20:012
3312432993822,11cyclictest291-21systemd-udevd20:40:442
331243199382,26cyclictest3380079-21ssh22:00:451
3312431993817,17cyclictest0-21swapper/121:25:411
3312431993817,17cyclictest0-21swapper/121:25:411
3312431993815,2cyclictest0-21swapper/121:10:431
331243199381,12cyclictest3394457-21/usr/sbin/munin22:25:331
331243099383,28cyclictest3370864-21ssh21:45:390
3312430993830,5cyclictest3407330-21ntp_offset22:45:460
3312430993816,19cyclictest3442562-21kworker/u8:15+rpciod23:50:450
3312432993728,6cyclictest3419167-21/usr/sbin/munin23:05:372
3312432993720,3cyclictest171rcu_preempt20:05:422
331243199374,10cyclictest421-21dbus-daemon22:34:021
331243199373,7cyclictest3458802-21ssh00:10:411
3312431993715,3cyclictest0-21swapper/119:55:521
331243199371,11cyclictest3328661-21cat20:00:511
3312430993729,5cyclictest3456131-21/usr/sbin/munin00:05:550
3312430993726,8cyclictest1003-21mta-sts-daemon22:55:020
3312430993726,7cyclictest3443778-21ssh23:45:470
331243099371,19cyclictest3360017-21ssh21:28:040
331243099371,19cyclictest3360017-21ssh21:28:040
3312432993629,3cyclictest0-21swapper/221:20:372
3312432993629,3cyclictest0-21swapper/221:20:372
3312432993627,6cyclictest3449308-21gltestperf23:55:322
3312432993627,5cyclictest3385140-21latency_hist22:10:032
3312431993629,3cyclictest0-21swapper/121:00:021
3312431993610,3cyclictest0-21swapper/121:45:021
3312431993610,3cyclictest0-21swapper/120:20:521
3312430993630,3cyclictest0-21swapper/023:30:520
3312430993629,4cyclictest3451642-21munin-run00:00:010
3312430993629,4cyclictest3386650-21grep22:10:540
3312430993629,4cyclictest3386650-21grep22:10:540
3312430993629,4cyclictest3337931-21systemd-run20:35:010
3312430993629,4cyclictest3337931-21systemd-run20:35:010
3312430993624,8cyclictest0-21swapper/022:35:300
3312430993624,6cyclictest3438120-21grep23:35:540
3312430993624,6cyclictest3438120-21grep23:35:540
331243299353,27cyclictest3389248-21ssh22:15:482
3312432993527,5cyclictest3366843-21latency_hist21:40:022
3312431993527,5cyclictest1559552-21cups-browsed20:45:461
331243199351,20cyclictest3474013-21ssh00:35:441
3312430993528,4cyclictest3430531-21ssh23:25:020
3312430993528,4cyclictest0-21swapper/022:55:420
3312430993528,3cyclictest0-21swapper/000:15:000
3312430993526,7cyclictest3375887-21cut21:55:010
3312430993516,13cyclictest12650irq/32-eth021:35:480
331243299343,25cyclictest3407334-21ssh22:45:462
3312432993428,3cyclictest0-21swapper/220:00:352
3312432993425,6cyclictest3348408-21diskmemload23:50:522
3312432993423,7cyclictest3317328-21aten_r2power_vo19:25:312
3312432993423,7cyclictest3317328-21aten_r2power_vo19:25:312
3312431993423,8cyclictest3317798-21ntp_kernel_pll_19:25:401
3312431993423,8cyclictest3317798-21ntp_kernel_pll_19:25:401
3312430993426,5cyclictest1559552-21cups-browsed21:19:200
3312430993425,3cyclictest0-21swapper/019:35:430
3312430993421,10cyclictest3333434-21tr20:20:010
331243299333,25cyclictest3395135-21ssh22:25:442
3312432993325,5cyclictest3445592-21systemd-run23:50:022
3312432993325,5cyclictest3418320-21systemd23:05:022
3312432993324,5cyclictest3312451-21latency_hist19:10:012
3312430993327,4cyclictest3419887-21unixbench-2d23:05:530
3312430993327,3cyclictest0-21swapper/022:10:020
3312430993327,3cyclictest0-21swapper/021:10:010
3312430993326,3cyclictest0-21swapper/022:41:440
3312430993325,4cyclictest1002-21mta-sts-daemon21:30:090
3312430993325,4cyclictest1002-21mta-sts-daemon21:30:090
3312430993320,10cyclictest3325802-21unixbench_multi19:50:530
3312432993227,2cyclictest0-21swapper/222:00:012
3312432993226,3cyclictest0-21swapper/223:20:462
3312432993225,4cyclictest3468481-21ssh00:25:542
3312432993223,4cyclictest351ktimers/223:40:372
3312432993223,3cyclictest0-21swapper/222:20:462
331243299322,24cyclictest665-21snmpd21:25:412
331243299322,24cyclictest665-21snmpd21:25:412
3312432993219,4cyclictest171rcu_preempt20:35:432
3312432993214,3cyclictest171rcu_preempt20:20:432
331243299321,24cyclictest3425568-21rm23:15:472
331243299321,24cyclictest3425568-21rm23:15:472
331243299320,6cyclictest3455918-21ntpq00:05:482
3312431993223,6cyclictest420-21avahi-daemon19:10:311
3312430993226,4cyclictest3324285-21sed19:45:520
3312430993225,4cyclictest1000-21mta-sts-daemon00:39:220
3312430993222,8cyclictest3379783-21cat22:00:370
331243099321,24cyclictest3451713-21cups-browsed00:00:490
331243099321,24cyclictest3451713-21cups-browsed00:00:490
3312432993125,3cyclictest0-21swapper/221:45:012
3312432993124,4cyclictest1001-21mta-sts-daemon21:50:402
3312432993122,5cyclictest1002-21mta-sts-daemon23:26:542
3312432993121,6cyclictest3434802-21mailstats23:30:512
3312431993121,7cyclictest421-21dbus-daemon21:05:021
3312430993127,2cyclictest0-21swapper/022:20:010
3312432993022,4cyclictest0-21swapper/221:10:522
3312432993022,2cyclictest3307122-21kworker/u8:6+xprtiod19:40:502
3312430993024,3cyclictest3346908-21cron21:05:010
3312430993023,3cyclictest0-21swapper/023:20:020
3312430993023,3cyclictest0-21swapper/023:20:020
3312430993021,6cyclictest3465453-21grep00:20:540
3312432992922,3cyclictest0-21swapper/222:10:412
3312432992922,3cyclictest0-21swapper/222:10:412
3312432992921,5cyclictest3352169-21fschecks_time21:15:332
3312432992921,5cyclictest3324344-21sed19:45:532
3312432992914,3cyclictest171rcu_preempt21:05:422
3312432992910,15cyclictest3464911-21diskmemload00:20:422
331243099292,25cyclictest0-21swapper/019:15:370
3312432992820,5cyclictest265-21systemd-journal20:30:012
3312432992820,5cyclictest1-21systemd19:35:012
3312432992820,5cyclictest1-21systemd19:35:012
3312432992818,4cyclictest171rcu_preempt19:10:452
3312432992816,3cyclictest171rcu_preempt19:50:412
3312432992815,6cyclictest171rcu_preempt19:55:392
3312432992813,3cyclictest171rcu_preempt20:50:492
3312432992811,4cyclictest171rcu_preempt20:10:382
3312431992822,3cyclictest0-21swapper/119:50:441
3312431992813,2cyclictest0-21swapper/119:10:021
3312430992823,2cyclictest0-21swapper/019:10:430
3312430992814,11cyclictest3327940-21fschecks_count20:00:340
3312430992810,12cyclictest151ktimers/020:10:400
3312432992719,5cyclictest1559552-21cups-browsed20:32:342
3312432992719,5cyclictest1559552-21cups-browsed20:32:342
3312432992718,3cyclictest171rcu_preempt19:20:412
3312430992719,6cyclictest3394556-21aten_r2power_fr22:25:320
3312430992718,7cyclictest3344431-21fschecks_time20:55:350
3312430992621,2cyclictest0-21swapper/020:45:000
3312430992617,6cyclictest1559552-21cups-browsed19:55:300
331243299257,3cyclictest171rcu_preempt20:55:472
331243199254,14cyclictest428-21mta-sts-daemon20:30:421
331243199254,14cyclictest428-21mta-sts-daemon20:30:421
3312430992514,9cyclictest151ktimers/019:20:460
3312430992514,7cyclictest1000-21mta-sts-daemon20:05:460
3312431992417,4cyclictest3330345-21unixbench_singl20:05:531
3312430992419,2cyclictest0-21swapper/020:40:000
3312430992318,3cyclictest1003-21mta-sts-daemon20:20:280
3312430992317,4cyclictest151ktimers/020:50:510
331243099222,15cyclictest1559552-21cups-browsed20:25:470
331243399183,9cyclictest0-21swapper/322:45:453
331243399182,9cyclictest0-21swapper/321:25:413
331243399182,9cyclictest0-21swapper/321:25:413
331243399182,8cyclictest0-21swapper/300:20:443
331243399181,7cyclictest0-21swapper/323:55:373
331243099183,8cyclictest1-21systemd19:10:010
331243399172,8cyclictest0-21swapper/300:05:533
331243399172,5cyclictest0-21swapper/319:40:383
331243399171,9cyclictest0-21swapper/323:45:403
331243399171,9cyclictest0-21swapper/321:45:433
331243399163,7cyclictest0-21swapper/323:10:483
331243399163,7cyclictest0-21swapper/322:15:483
331243399163,6cyclictest0-21swapper/323:15:443
331243399163,6cyclictest0-21swapper/323:15:443
331243399162,8cyclictest0-21swapper/321:20:443
331243399162,8cyclictest0-21swapper/321:20:443
331243399162,7cyclictest0-21swapper/323:00:383
331243399161,9cyclictest0-21swapper/323:25:473
331243399161,9cyclictest0-21swapper/300:00:483
331243399161,9cyclictest0-21swapper/300:00:483
331243399161,8cyclictest0-21swapper/321:55:433
331243399161,8cyclictest0-21swapper/320:50:483
331243399151,8cyclictest0-21swapper/321:30:423
331243399151,8cyclictest0-21swapper/321:30:423
331243399151,7cyclictest0-21swapper/323:35:443
331243399151,7cyclictest0-21swapper/323:35:443
331243399151,7cyclictest0-21swapper/322:25:443
331243399151,7cyclictest0-21swapper/321:40:443
331243399151,7cyclictest0-21swapper/319:25:393
331243399151,7cyclictest0-21swapper/319:25:393
331243399151,6cyclictest0-21swapper/320:20:443
331243399151,6cyclictest0-21swapper/319:15:383
331243399150,7cyclictest0-21swapper/319:20:423
331243399143,6cyclictest0-21swapper/322:00:443
331243399143,6cyclictest0-21swapper/300:25:473
331243399142,7cyclictest0-21swapper/323:30:413
331243399142,7cyclictest0-21swapper/323:20:433
331243399142,7cyclictest0-21swapper/323:05:363
331243399142,7cyclictest0-21swapper/321:10:013
331243399142,6cyclictest0-21swapper/321:10:423
331243399141,8cyclictest0-21swapper/323:50:443
331243399141,8cyclictest0-21swapper/322:10:483
331243399141,8cyclictest0-21swapper/322:10:483
331243399141,8cyclictest0-21swapper/300:15:393
331243399141,7cyclictest0-21swapper/323:40:403
331243399141,7cyclictest0-21swapper/321:15:423
331243399141,7cyclictest0-21swapper/319:30:433
331243399141,7cyclictest0-21swapper/319:30:433
331243399141,6cyclictest0-21swapper/319:45:413
331243399140,8cyclictest0-21swapper/322:20:473
331243399133,5cyclictest0-21swapper/320:00:013
331243399132,7cyclictest0-21swapper/300:35:443
331243399132,6cyclictest0-21swapper/322:40:323
331243399132,6cyclictest0-21swapper/300:10:443
331243399132,5cyclictest0-21swapper/320:05:013
331243399131,8cyclictest0-21swapper/322:35:403
331243399131,7cyclictest0-21swapper/322:05:423
331243399131,7cyclictest0-21swapper/321:50:433
331243399131,7cyclictest0-21swapper/320:40:423
331243399131,7cyclictest0-21swapper/320:25:473
331243399131,6cyclictest0-21swapper/319:35:473
331243399131,6cyclictest0-21swapper/319:10:453
331243399122,6cyclictest0-21swapper/322:50:403
331243399122,6cyclictest0-21swapper/322:30:443
331243399122,6cyclictest0-21swapper/321:35:423
331243399122,6cyclictest0-21swapper/320:20:013
331243399122,5cyclictest0-21swapper/322:55:303
331243399121,7cyclictest0-21swapper/320:30:443
331243399121,7cyclictest0-21swapper/320:30:443
331243399121,7cyclictest0-21swapper/320:10:013
331243399121,7cyclictest0-21swapper/300:30:483
331243399121,6cyclictest0-21swapper/320:55:463
331243399112,6cyclictest0-21swapper/320:35:413
331243399111,6cyclictest0-21swapper/321:00:393
331243399111,6cyclictest0-21swapper/319:50:413
331243399101,6cyclictest0-21swapper/320:45:373
331243399101,6cyclictest0-21swapper/320:10:283
33124339991,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional