You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-26 - 02:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sun Apr 26, 2026 00:43:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4193372996832,24cyclictest135966-21munin-run23:15:000
4193372995852,3cyclictest0-21swapper/021:35:010
4193373995647,4cyclictest0-21swapper/122:50:511
419337299544,4cyclictest35391-21diskmemload22:20:480
419337299544,4cyclictest35391-21diskmemload22:20:480
4193373995345,5cyclictest15928-21systemd-run20:05:011
4193372995342,7cyclictest59675-21basename21:40:010
4193372995322,27cyclictest161218-21ssh23:45:460
4193372995322,27cyclictest161218-21ssh23:45:460
4193374995143,5cyclictest33921-21systemd-run21:05:012
4193374995143,5cyclictest33921-21systemd-run21:05:012
4193373995042,5cyclictest171867-21cron00:00:001
4193373995042,5cyclictest171867-21cron00:00:001
4193374994838,6cyclictest75708-21systemctl22:00:012
4193372994618,24cyclictest0-21swapper/022:10:480
4193374994537,5cyclictest175893-21cut00:05:002
419337499453,32cyclictest113350-21ssh22:45:542
4193372994538,4cyclictest1054-21mta-sts-daemon22:55:010
419337299443,32cyclictest97157-21ssh22:25:450
4193372994414,2cyclictest0-21swapper/022:05:500
4193372994414,2cyclictest0-21swapper/020:50:440
419337299441,10cyclictest35899-21ssh21:10:320
4193374994334,6cyclictest421-21dbus-daemon23:30:012
4193373994335,5cyclictest59697-21modprobe21:40:011
419337299433,18cyclictest129215-21ssh23:05:450
419337299433,18cyclictest129215-21ssh23:05:450
419337299432,18cyclictest48925-21ssh21:25:480
4193372994318,22cyclictest2389957-21cups-browsed22:57:380
4193374994235,4cyclictest0-21swapper/221:20:442
4193374994234,4cyclictest0-21swapper/221:15:492
4193374994233,5cyclictest187837-21latency_hist00:20:022
4193374994229,9cyclictest191797-21/usr/sbin/munin00:25:012
4193374994229,10cyclictest160607-21cpu23:45:342
4193374994229,10cyclictest160607-21cpu23:45:342
4193373994232,6cyclictest2389953-21cupsd22:08:031
419337299421,14cyclictest113420-21ps22:45:550
4193373994130,7cyclictest0-21swapper/121:25:021
419337299412,10cyclictest141316-21ssh23:20:450
4193372994119,19cyclictest60314-21aten2.4-expect21:40:320
4193372994119,19cyclictest60314-21aten2.4-expect21:40:320
4193372994114,24cyclictest0-21swapper/023:00:340
419337299411,17cyclictest151ktimers/021:15:430
419337299411,16cyclictest109332-21ssh22:40:500
419337299411,15cyclictest171855-21cups-browsed00:15:460
419337299411,14cyclictest72963-21ssh21:55:440
419337499402,31cyclictest137223-21ssh23:15:442
4193373994029,8cyclictest75702-21/usr/sbin/munin22:00:011
4193373994029,8cyclictest196728-21latency00:30:391
4193373994028,9cyclictest135989-21sh23:15:011
419337299409,8cyclictest0-21swapper/000:00:030
419337299409,8cyclictest0-21swapper/000:00:030
419337299406,11cyclictest155928-21ssh23:40:010
419337299401,14cyclictest193572-21grep00:25:530
4193374993933,3cyclictest0-21swapper/200:00:002
4193374993933,3cyclictest0-21swapper/200:00:002
4193374993926,10cyclictest1056-21mta-sts-daemon00:07:172
4193373993933,4cyclictest1054-21mta-sts-daemon21:19:491
4193373993933,3cyclictest0-21swapper/123:25:351
4193373993931,5cyclictest175909-21ssh00:05:011
4193373993928,8cyclictest87812-21ssh22:15:011
419337299399,7cyclictest34904-21/usr/sbin/munin21:05:490
419337299399,7cyclictest34904-21/usr/sbin/munin21:05:490
419337299397,12cyclictest153939-21kworker/u8:0+xprtiod00:10:560
4193372993930,6cyclictest0-21swapper/000:05:010
4193372993919,17cyclictest0-21swapper/000:40:030
4193372993919,17cyclictest0-21swapper/000:40:030
4193372993915,22cyclictest261-21systemd-journal20:06:030
4193372993910,6cyclictest8898-21/usr/sbin/munin19:40:350
4193374993831,4cyclictest107920-21munin-run22:40:012
4193374993828,7cyclictest93695-21unixbench_multi22:20:562
4193374993828,7cyclictest93695-21unixbench_multi22:20:562
4193374993827,8cyclictest420-21avahi-daemon22:30:492
4193374993826,9cyclictest140705-21cpu23:20:332
4193373993830,5cyclictest156510-21aten_r2power_en23:40:321
4193373993830,5cyclictest140119-21latency_hist23:20:021
4193373993828,8cyclictest12203-21needreboot19:50:411
4193373993828,8cyclictest12203-21needreboot19:50:411
4193373993827,8cyclictest153601-21timerandwakeup23:35:531
4193373993823,3cyclictest171rcu_preempt23:45:431
4193373993823,3cyclictest171rcu_preempt23:45:431
4193373993817,15cyclictest421-21dbus-daemon23:20:441
4193373993814,17cyclictest421-21dbus-daemon19:30:431
419337299383,9cyclictest69000-21ntpq21:50:490
419337299383,9cyclictest3304-21ntpq19:20:420
419337299383,8cyclictest12282-21ntpq19:50:430
419337299383,8cyclictest12282-21ntpq19:50:430
419337299383,25cyclictest76943-21diskmemload22:00:440
4193372993821,9cyclictest151ktimers/022:15:470
4193372993821,9cyclictest151ktimers/022:15:470
419337299382,13cyclictest2389957-21cups-browsed20:10:420
4193372993817,4cyclictest103971-21wc22:35:010
4193372993814,3cyclictest0-21swapper/023:25:470
4193372993813,3cyclictest0-21swapper/000:20:430
419337299381,21cyclictest64481-21df21:45:340
419337299381,15cyclictest33891-21cron21:05:000
419337299381,15cyclictest33891-21cron21:05:000
419337299381,15cyclictest149206-21/usr/sbin/munin23:30:440
419337299381,13cyclictest24005-21awk20:30:370
4193374993730,4cyclictest32402-21sendmail21:00:002
4193374993730,3cyclictest0-21swapper/221:15:002
4193374993729,5cyclictest59659-21idleruntime-cro21:40:012
4193374993729,5cyclictest3902-21cron19:25:012
4193374993728,5cyclictest200350-21aten_r2power_po00:35:342
4193374993728,5cyclictest200350-21aten_r2power_po00:35:342
4193374993727,6cyclictest2389957-21cups-browsed22:43:412
419337499372,28cyclictest117206-21ssh22:50:462
4193373993732,3cyclictest171344-21kworker/u8:9-xprtiod00:25:431
4193373993729,5cyclictest111942-21cron22:45:011
4193373993728,6cyclictest55632-21/usr/sbin/munin21:35:021
419337299379,13cyclictest3928-21kworker/u8:9+xprtiod19:30:430
419337299374,10cyclictest196990-21ntpq00:30:450
419337299372,31cyclictest177086-21ssh00:05:420
419337299371,7cyclictest21585-21switchtime20:20:520
4193372993714,3cyclictest0-21swapper/023:40:410
419337299371,16cyclictest7227-21aten_r2power_fr19:35:330
4193372993711,4cyclictest14632-21/usr/sbin/munin20:00:380
4193374993630,3cyclictest0-21swapper/222:25:412
4193374993626,4cyclictest0-21swapper/200:30:452
419337499361,28cyclictest125368-21ssh23:00:542
4193373993628,6cyclictest4193371-21cyclictest23:50:481
4193373993628,5cyclictest23909-21fschecks_count20:30:351
4193373993627,6cyclictest420-21avahi-daemon00:24:241
4193373993627,6cyclictest171855-21cups-browsed00:35:511
4193373993627,6cyclictest171855-21cups-browsed00:35:511
4193373993627,6cyclictest1-21systemd23:35:021
4193372993629,4cyclictest44084-21aten_r2power_en21:20:320
419337299362,17cyclictest105174-21ssh22:35:470
4193372993620,13cyclictest168070-21latency_hist23:55:020
4193374993527,5cyclictest192434-21cpu00:25:322
4193374993527,5cyclictest159934-21idleruntime-cro23:45:012
4193374993526,5cyclictest23413-21systemd-run20:30:012
4193373993527,5cyclictest131983-21systemd-run23:10:011
4193373993527,5cyclictest131983-21systemd-run23:10:011
4193373993526,7cyclictest0-21swapper/121:10:551
4193373993526,6cyclictest5288-21tail19:25:591
4193373993526,6cyclictest171855-21cups-browsed00:07:301
4193374993427,4cyclictest49058-21mailstats21:25:502
4193374993427,3cyclictest0-21swapper/222:55:332
4193374993425,3cyclictest0-21swapper/223:35:422
4193374993424,6cyclictest420-21avahi-daemon22:15:342
4193374993424,6cyclictest420-21avahi-daemon22:15:342
4193373993429,2cyclictest0-21swapper/122:40:011
4193373993426,5cyclictest121805-21unixbench_singl22:55:571
4193373993424,7cyclictest420-21avahi-daemon21:44:131
4193373993424,7cyclictest420-21avahi-daemon21:44:131
419337399342,23cyclictest101201-21ssh22:30:451
4193372993427,5cyclictest1-21systemd23:15:530
4193372993426,5cyclictest26935-21fschecks_time20:40:350
4193372993416,2cyclictest0-21swapper/020:25:530
4193374993325,5cyclictest167858-21cron23:55:002
4193373993325,5cyclictest28418-21fschecks_count20:45:371
4193373993324,6cyclictest19290-21aten_r2power_vo20:15:331
4193373993323,7cyclictest695-21cron21:05:001
4193373993323,7cyclictest695-21cron21:05:001
4193373993320,8cyclictest93385-21/usr/sbin/munin22:20:521
4193373993320,8cyclictest93385-21/usr/sbin/munin22:20:521
4193374993226,3cyclictest0-21swapper/219:35:542
4193374993225,4cyclictest52420-21fschecks_count21:30:372
4193374993225,4cyclictest1054-21mta-sts-daemon21:40:102
4193374993225,4cyclictest1054-21mta-sts-daemon21:40:102
4193374993224,4cyclictest0-21swapper/223:10:462
4193374993223,6cyclictest35391-21diskmemload21:53:452
4193374993223,5cyclictest0-21swapper/223:35:012
4193374993216,8cyclictest35391-21diskmemload00:10:542
4193373993224,5cyclictest99892-21ssh22:30:001
4193373993224,5cyclictest2449-21latency_hist19:20:021
4193373993224,5cyclictest14270-21unixbench_multi19:55:521
4193373993224,5cyclictest127988-21systemd-run23:05:021
4193373993223,7cyclictest21623-21timerandwakeup20:20:521
4193373993223,6cyclictest79734-21cron22:05:011
4193373993216,9cyclictest271ktimers/122:15:461
4193373993216,9cyclictest271ktimers/122:15:461
4193372993219,10cyclictest0-21swapper/019:10:020
4193372993213,5cyclictest261-21systemd-journal20:18:040
4193374993126,2cyclictest0-21swapper/219:30:012
4193374993124,4cyclictest85594-21unixbench_multi22:10:542
4193374993123,5cyclictest20524-21latency_hist20:20:022
4193374993117,9cyclictest67246-21sshd21:49:032
4193373993126,2cyclictest0-21swapper/121:55:021
4193373993124,6cyclictest271ktimers/121:25:491
4193373993124,4cyclictest1055-21mta-sts-daemon20:55:271
4193373993123,5cyclictest64386-21aten_r2power_vo21:45:341
4193373993123,5cyclictest187814-21systemd-run00:20:011
4193372993118,5cyclictest151ktimers/019:55:400
4193374993022,4cyclictest0-21swapper/222:00:532
419337499301,24cyclictest81026-21sh22:05:462
4193373993024,3cyclictest0-21swapper/122:45:351
4193373993022,5cyclictest4193384-21munin-run19:10:011
4193373993022,3cyclictest0-21swapper/100:10:351
4193373993021,6cyclictest2389957-21cups-browsed19:24:181
4193372993023,5cyclictest897-21true19:15:010
4193372993018,9cyclictest421-21dbus-daemon20:40:010
4193374992922,4cyclictest872-21cron19:15:012
4193374992918,8cyclictest25758-21ntp_kernel_pll_20:35:412
4193373992918,3cyclictest10421-21grep19:45:391
4193372992924,3cyclictest10430-21fschecks_time19:45:390
4193372992922,5cyclictest2389957-21cups-browsed20:59:180
4193374992821,4cyclictest15896-21cron20:05:002
4193374992821,4cyclictest1056-21mta-sts-daemon20:50:472
4193373992820,5cyclictest917-21latency_hist19:15:021
4193373992817,8cyclictest23412-21sh20:30:011
4193372992817,9cyclictest5399-21latency_hist19:30:020
4193374992720,3cyclictest0-21swapper/221:05:422
4193374992720,3cyclictest0-21swapper/221:05:422
4193374992719,5cyclictest21898-21cron20:25:012
4193374992719,5cyclictest129084-21needreboot23:05:412
4193374992719,5cyclictest129084-21needreboot23:05:412
4193374992718,6cyclictest6916-21systemd-run19:35:012
419337399275,15cyclictest2389957-21cups-browsed20:05:451
4193374992620,3cyclictest4180860-21kworker/u8:0-xprtiod19:15:462
4193374992619,5cyclictest1-21systemd19:55:012
4193374992619,5cyclictest1-21systemd19:55:012
4193374992619,4cyclictest1055-21mta-sts-daemon19:40:462
4193373992620,3cyclictest0-21swapper/119:40:011
4193373992617,6cyclictest420-21avahi-daemon20:39:071
4193372992620,4cyclictest2389957-21cups-browsed19:16:350
4193374992519,3cyclictest0-21swapper/220:50:012
4193374992517,5cyclictest14324-21unixbench_singl19:55:532
4193374992515,8cyclictest17619-21sshd20:10:032
4193373992518,4cyclictest26947-21gltestperf20:40:351
4193373992518,4cyclictest17716-21aten_r2power_fr20:10:321
4193373992517,5cyclictest34409-21fschecks_count21:05:361
4193373992517,5cyclictest34409-21fschecks_count21:05:361
4193373992513,3cyclictest171rcu_preempt20:55:021
4193372992516,7cyclictest2389957-21cups-browsed20:46:370
4193374992417,4cyclictest36-21ksoftirqd/220:30:462
4193374992416,5cyclictest26929-21fschecks_time20:40:352
4193374992416,5cyclictest17302-21unixbench_multi20:05:552
4193374992415,6cyclictest548-21gdbus19:45:552
4193373992315,5cyclictest0-21swapper/119:45:011
4193374992114,3cyclictest0-21swapper/219:10:022
419337599204,10cyclictest0-21swapper/322:25:493
419337599174,5cyclictest0-21swapper/321:20:433
419337599163,8cyclictest0-21swapper/300:15:463
419337599161,9cyclictest0-21swapper/322:45:543
419337599161,8cyclictest0-21swapper/322:00:443
419337599154,7cyclictest0-21swapper/323:00:553
419337599153,7cyclictest0-21swapper/322:20:453
419337599153,7cyclictest0-21swapper/322:20:453
419337599152,8cyclictest0-21swapper/322:10:483
419337599152,7cyclictest0-21swapper/300:00:443
419337599151,9cyclictest0-21swapper/321:15:483
419337599151,8cyclictest0-21swapper/323:50:443
419337599151,8cyclictest0-21swapper/323:10:493
419337599151,8cyclictest0-21swapper/322:50:453
419337599151,8cyclictest0-21swapper/320:45:453
419337599144,7cyclictest0-21swapper/300:35:483
419337599144,7cyclictest0-21swapper/300:35:483
419337599144,6cyclictest0-21swapper/323:35:463
419337599143,6cyclictest0-21swapper/300:30:433
419337599142,8cyclictest0-21swapper/323:45:423
419337599142,8cyclictest0-21swapper/323:45:423
419337599142,8cyclictest0-21swapper/323:25:533
419337599142,8cyclictest0-21swapper/321:35:443
419337599142,7cyclictest0-21swapper/322:55:363
419337599142,7cyclictest0-21swapper/322:05:433
419337599142,7cyclictest0-21swapper/321:50:513
419337599142,7cyclictest0-21swapper/321:45:443
419337599141,8cyclictest0-21swapper/322:40:453
419337599141,8cyclictest0-21swapper/322:30:423
419337599141,8cyclictest0-21swapper/300:25:403
419337599141,7cyclictest0-21swapper/300:05:423
419337599141,6cyclictest0-21swapper/321:25:323
419337599141,6cyclictest0-21swapper/320:40:463
419337599141,6cyclictest0-21swapper/320:00:463
419337599141,6cyclictest0-21swapper/319:25:453
419337599134,6cyclictest0-21swapper/300:20:433
419337599134,5cyclictest0-21swapper/320:40:013
419337599133,6cyclictest0-21swapper/322:35:483
419337599133,4cyclictest0-21swapper/319:15:453
419337599132,7cyclictest0-21swapper/323:30:343
419337599132,7cyclictest0-21swapper/321:55:303
419337599132,7cyclictest0-21swapper/321:10:403
419337599132,6cyclictest0-21swapper/321:40:353
419337599132,6cyclictest0-21swapper/321:40:353
419337599132,6cyclictest0-21swapper/321:10:003
419337599132,6cyclictest0-21swapper/321:10:003
419337599131,8cyclictest0-21swapper/323:55:363
419337599131,8cyclictest0-21swapper/323:55:363
419337599131,8cyclictest0-21swapper/323:20:403
419337599131,8cyclictest0-21swapper/322:15:313
419337599131,8cyclictest0-21swapper/322:15:313
419337599131,7cyclictest0-21swapper/319:30:493
419337599131,6cyclictest0-21swapper/320:50:423
419337599131,6cyclictest0-21swapper/320:15:453
419337599131,6cyclictest0-21swapper/320:10:413
419337599131,6cyclictest0-21swapper/319:50:413
419337599131,6cyclictest0-21swapper/319:50:413
419337599122,5cyclictest0-21swapper/320:05:453
419337599121,7cyclictest0-21swapper/323:40:373
419337599121,7cyclictest0-21swapper/323:15:333
419337599121,7cyclictest0-21swapper/323:05:383
419337599121,7cyclictest0-21swapper/323:05:383
419337599121,7cyclictest0-21swapper/300:10:423
419337599121,6cyclictest0-21swapper/321:35:023
419337599121,6cyclictest0-21swapper/321:05:003
419337599121,6cyclictest0-21swapper/321:05:003
419337599121,6cyclictest0-21swapper/320:55:413
419337599121,6cyclictest0-21swapper/319:55:463
419337599121,6cyclictest0-21swapper/319:20:473
419337599111,6cyclictest0-21swapper/320:30:413
419337599111,6cyclictest0-21swapper/320:25:443
419337599111,6cyclictest0-21swapper/320:20:383
419337599103,4cyclictest0-21swapper/319:35:433
419337599102,5cyclictest0-21swapper/319:40:443
419337599101,5cyclictest0-21swapper/319:45:453
419337599100,5cyclictest0-21swapper/319:10:323
41933759982,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional