You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-08 - 03:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Wed Apr 08, 2026 00:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1280575995143,5cyclictest1393310-21unixbench_multi23:00:462
1280575995143,5cyclictest1393310-21unixbench_multi23:00:462
1280573995121,7cyclictest0-21swapper/022:55:330
1280575994940,6cyclictest1054-21mta-sts-daemon00:05:322
1280575994940,6cyclictest1054-21mta-sts-daemon00:05:322
1280574994930,16cyclictest547-21gmain23:00:331
1280574994930,16cyclictest547-21gmain23:00:331
1280573994922,7cyclictest1415224-21sort23:35:260
1280575994840,5cyclictest421-21dbus-daemon23:30:012
1280574994837,8cyclictest1401558-21grep23:15:001
1280573994822,4cyclictest439-21systemd-logind21:45:010
1280574994636,7cyclictest421-21dbus-daemon21:45:011
1280574994636,6cyclictest421-21dbus-daemon21:55:011
1280574994636,5cyclictest0-21swapper/121:19:541
128057499462,37cyclictest1402517-21ssh23:15:311
1280573994619,8cyclictest1441814-21unixbench_multi00:15:430
128057599446,33cyclictest1379716-21ssh22:40:362
128057599443,31cyclictest1317821-21ssh21:10:372
1280575994335,5cyclictest1-21systemd23:50:022
128057499435,9cyclictest171rcu_preempt22:40:331
1280574994335,5cyclictest1342227-21ssh21:45:421
1280573994328,11cyclictest12350irq/32-eth021:10:320
1280575994233,6cyclictest1431064-21cups-browsed00:10:222
1280575994233,6cyclictest1431064-21cups-browsed00:10:222
1280575994231,8cyclictest1389083-21fschecks_count22:55:252
1280574994230,9cyclictest1381138-21kworker/u8:4+nfsiod00:00:321
1280574994230,9cyclictest1381138-21kworker/u8:4+nfsiod00:00:321
1280575994134,4cyclictest1056-21mta-sts-daemon00:00:322
1280575994134,4cyclictest1056-21mta-sts-daemon00:00:322
1280575994132,6cyclictest1360887-21unin-run22:15:012
1280575994132,6cyclictest1360887-21unin-run22:15:012
1280575994131,6cyclictest1324173-21munin-plugin-st21:20:012
1280575994129,8cyclictest1403086-21timerwakeupswit23:15:452
1280574994134,4cyclictest0-21swapper/120:20:011
1280573994131,7cyclictest420-21avahi-daemon22:45:320
1280575994032,5cyclictest1334505-21munin-plugin-st21:35:002
1280574994028,9cyclictest421-21dbus-daemon22:07:461
1280574994028,9cyclictest421-21dbus-daemon22:07:461
128057499401,7cyclictest171rcu_preempt21:25:361
1280573994032,6cyclictest600-21gmain21:05:380
1280575993930,4cyclictest0-21swapper/222:00:312
1280575993930,4cyclictest0-21swapper/222:00:312
1280574993932,4cyclictest0-21swapper/122:40:011
1280574993922,13cyclictest1348328-21needreboot21:55:281
1280573993933,4cyclictest1295549-21munin-run20:00:010
1280573993928,5cyclictest261-21systemd-journal23:20:000
1280575993830,5cyclictest1285098-21systemd-run19:25:012
1280575993830,5cyclictest1283558-21sendmail19:20:012
1280575993829,5cyclictest1431077-21ssh00:00:012
1280574993830,5cyclictest1421378-21latency_hist23:45:021
1280574993824,10cyclictest1-21systemd23:50:221
128057499381,30cyclictest1358536-21rm22:10:341
128057499381,30cyclictest1358536-21rm22:10:341
1280573993832,4cyclictest1290994-21head19:40:430
1280573993826,4cyclictest1334624-21latency_hist21:35:020
1280575993728,6cyclictest261-21systemd-journal20:35:002
1280574993729,5cyclictest1332164-21latency21:30:281
1280574993728,6cyclictest1453450-21aten_r2power_fr00:35:251
1280574993727,7cyclictest695-21cron19:25:011
1280575993627,5cyclictest1375119-21latency_hist22:35:022
1280574993630,3cyclictest0-21swapper/123:40:011
1280574993627,4cyclictest0-21swapper/121:10:321
1280574993623,10cyclictest1364904-21aten_r2power_vo22:20:231
128057499362,26cyclictest1444489-21ssh00:20:321
128057499362,26cyclictest1412422-21ssh23:30:351
1280574993621,12cyclictest1437829-21needreboot00:10:291
1280574993621,12cyclictest1437829-21needreboot00:10:291
1280574993614,16cyclictest421-21dbus-daemon19:15:371
1280574993614,12cyclictest1316558-21diskmemload23:55:321
1280573993630,4cyclictest1425461-21ntp_kernel_pll_23:50:340
1280573993627,6cyclictest1375072-21ssh22:35:010
1280575993529,3cyclictest0-21swapper/220:40:012
1280575993527,5cyclictest421-21dbus-daemon23:40:222
128057499353,23cyclictest1335721-21ssh21:35:391
1280574993528,4cyclictest0-21swapper/121:25:021
1280574993526,6cyclictest1-21systemd00:10:011
1280574993526,6cyclictest1-21systemd00:10:011
1280574993526,5cyclictest1424387-21munin-plugin-st23:50:011
1280574993526,5cyclictest1056-21mta-sts-daemon22:55:291
1280574993525,6cyclictest1294060-21idleruntime-cro19:55:011
1280574993525,4cyclictest0-21swapper/100:15:431
128057399353,25cyclictest1321591-21ssh21:15:300
1280573993521,12cyclictest1307435-21unixbench_multi20:35:450
1280575993428,3cyclictest0-21swapper/200:25:362
1280575993428,3cyclictest0-21swapper/200:25:362
1280574993422,9cyclictest1281426-21ntp_kernel_pll_19:10:321
1280573993423,8cyclictest0-21swapper/022:55:010
1280573993423,4cyclictest1335516-21netstat21:35:310
1280573993416,14cyclictest624-21lldpd22:19:000
1280575993327,3cyclictest0-21swapper/223:50:382
1280575993326,4cyclictest1054-21mta-sts-daemon21:00:362
1280575993325,5cyclictest0-21swapper/222:15:212
1280574993325,5cyclictest1303465-21aten_r2power_vo20:25:241
1280574993322,8cyclictest1364384-21sendmail22:20:001
1280573993327,3cyclictest1055-21mta-sts-daemon22:38:400
1280575993226,3cyclictest0-21swapper/220:00:012
1280575993226,3cyclictest0-21swapper/200:40:002
1280575993224,5cyclictest1281077-21fschecks_count19:10:252
1280575993224,4cyclictest1292710-21latency_hist19:50:032
1280575993221,8cyclictest1385202-21cron22:50:032
1280575993220,9cyclictest1382959-21needreboot22:45:302
1280575993218,10cyclictest1451427-21sshd00:30:532
1280574993222,7cyclictest1368661-21fschecks_count22:25:271
1280573993225,4cyclictest1429234-21grep23:55:450
1280573993224,4cyclictest151ktimers/020:45:340
128057599313,21cyclictest1441171-21ssh00:15:302
1280575993125,3cyclictest0-21swapper/221:00:012
1280575993124,4cyclictest1347434-21cat21:55:022
1280575993124,4cyclictest1056-21mta-sts-daemon21:56:032
1280575993124,4cyclictest1055-21mta-sts-daemon21:46:272
128057599312,22cyclictest1338737-21ssh21:40:312
1280575993120,7cyclictest3521708-21cups-browsed22:25:442
1280574993124,4cyclictest0-21swapper/119:45:351
1280574993123,5cyclictest1407914-21systemd-run23:25:011
1280574993123,5cyclictest1386534-21awk22:50:411
1280574993123,4cyclictest0-21swapper/122:00:291
1280574993123,4cyclictest0-21swapper/122:00:291
1280574993119,8cyclictest1290441-21ntp_kernel_pll_19:40:321
1280574993112,15cyclictest271ktimers/120:10:381
1280574993112,14cyclictest1372601-21ld22:30:341
1280573993126,2cyclictest0-21swapper/021:55:010
1280573993123,5cyclictest420-21avahi-daemon22:05:230
1280573993123,5cyclictest420-21avahi-daemon22:05:230
1280573993123,4cyclictest0-21swapper/021:20:380
1280575993024,3cyclictest1286703-21kworker/u8:5-nfsiod19:55:012
1280575993023,4cyclictest1365663-21processes22:20:392
1280575993023,4cyclictest1297073-21munin-plugin-st20:05:012
1280575993023,3cyclictest0-21swapper/223:20:312
1280575993023,3cyclictest0-21swapper/200:20:202
1280575993022,5cyclictest261-21systemd-journal21:10:012
1280575993021,6cyclictest1375806-21/usr/sbin/munin22:35:272
128057499303,19cyclictest1447671-21ssh00:25:341
128057499303,19cyclictest1447671-21ssh00:25:341
1280574993024,3cyclictest0-21swapper/120:55:011
1280574993022,5cyclictest1304587-21kworker/u8:3+xprtiod20:40:011
1280574993020,5cyclictest0-21swapper/122:48:041
128057499301,4cyclictest171rcu_preempt19:55:341
1280573993025,3cyclictest421-21dbus-daemon22:00:020
1280573993025,3cyclictest0-21swapper/023:40:440
1280573993025,3cyclictest0-21swapper/000:05:220
1280573993025,3cyclictest0-21swapper/000:05:220
1280573993024,4cyclictest1446635-21cron00:25:000
1280573993021,6cyclictest1412146-21latency23:30:270
1280573993020,2cyclictest0-21swapper/023:50:020
128057399301,24cyclictest1358590-21ssh22:10:360
128057399301,24cyclictest1358590-21ssh22:10:360
1280575992924,2cyclictest0-21swapper/223:10:002
1280575992924,2cyclictest0-21swapper/223:10:002
1280575992922,4cyclictest1309066-21munin-plugin-st20:45:012
1280575992921,5cyclictest1056-21mta-sts-daemon21:23:192
128057499296,18cyclictest428-21mta-sts-daemon19:35:321
128057499296,18cyclictest428-21mta-sts-daemon19:35:321
1280574992921,5cyclictest1286590-21systemd-run19:30:011
1280574992919,7cyclictest695-21cron20:05:011
1280573992923,4cyclictest1432640-21tail00:00:460
1280573992923,4cyclictest1432640-21tail00:00:460
1280573992923,4cyclictest1306073-21systemd-run20:35:010
1280573992921,6cyclictest1395873-21needreboot23:05:310
1280573992921,6cyclictest1395873-21needreboot23:05:310
1280573992921,5cyclictest261-21systemd-journal23:15:020
1280573992921,5cyclictest0-21swapper/019:25:010
1280573992914,11cyclictest12350irq/32-eth023:00:390
1280573992914,11cyclictest12350irq/32-eth023:00:390
1280575992821,3cyclictest0-21swapper/223:30:322
1280575992821,3cyclictest0-21swapper/221:35:352
1280575992820,5cyclictest1331360-21latency_hist21:30:022
128057599281,20cyclictest1355191-21ssh22:05:362
128057599281,20cyclictest1355191-21ssh22:05:362
1280574992823,2cyclictest0-21swapper/123:10:001
1280574992823,2cyclictest0-21swapper/123:10:001
1280574992822,3cyclictest0-21swapper/121:00:011
1280574992820,5cyclictest1287939-21unixbench_multi19:30:441
1280574992816,9cyclictest1315078-21sh21:05:011
128057399282,21cyclictest1351824-21ssh22:00:330
128057399282,21cyclictest1351824-21ssh22:00:330
1280573992821,4cyclictest1405695-21ntp_kernel_err23:20:310
1280573992821,4cyclictest1288052-21cron19:35:000
1280573992818,7cyclictest1409488-21timerwakeupswit23:25:450
1280573992818,7cyclictest0-21swapper/000:35:000
128057399281,22cyclictest1369058-21ntpq22:25:340
128057599274,17cyclictest3521708-21cups-browsed20:10:332
1280575992719,4cyclictest1310780-21acpi20:50:222
1280575992716,8cyclictest1415613-21ssh23:35:342
128057499274,12cyclictest1316558-21diskmemload23:28:301
1280574992719,5cyclictest1298432-21tail20:05:441
1280574992719,5cyclictest1056-21mta-sts-daemon00:31:061
1280573992722,3cyclictest1054-21mta-sts-daemon19:25:320
128057399272,18cyclictest659-21snmpd19:50:290
1280573992721,3cyclictest0-21swapper/022:45:020
1280573992720,5cyclictest420-21avahi-daemon21:46:190
1280573992719,6cyclictest1292442-21unixbench_multi19:45:430
1280573992717,6cyclictest151ktimers/020:00:370
1280575992620,3cyclictest0-21swapper/219:29:532
1280575992619,4cyclictest421-21dbus-daemon20:20:382
1280574992621,3cyclictest1051-21mta-sts-daemon20:40:231
1280573992620,4cyclictest1329484-21unixbench_singl21:25:450
1280573992615,5cyclictest421-21dbus-daemon20:25:360
1280573992613,11cyclictest0-21swapper/000:40:020
1280575992516,6cyclictest1017-21sendmail-mta19:35:362
1280575992516,6cyclictest1017-21sendmail-mta19:35:362
1280574992514,6cyclictest1316618-21latency_hist21:10:021
128057499251,19cyclictest3521708-21cups-browsed20:20:381
1280573992516,7cyclictest1438474-21unixbench_multi00:10:430
1280573992516,7cyclictest1438474-21unixbench_multi00:10:430
1280573992516,6cyclictest0-21swapper/021:00:420
1280575992420,2cyclictest0-21swapper/220:20:012
1280575992417,4cyclictest1303333-21aten_r2power_cu20:25:232
1280575992417,3cyclictest0-21swapper/223:10:442
1280575992416,5cyclictest1290915-21unixbench-2d19:40:432
128057399247,13cyclictest12450irq/33-eth022:20:350
1280573992419,3cyclictest1054-21mta-sts-daemon20:05:210
1280573992419,2cyclictest0-21swapper/021:00:010
1280573992418,4cyclictest1056-21mta-sts-daemon19:10:220
1280573992417,3cyclictest0-21swapper/020:50:320
128057399241,3cyclictest171rcu_preempt20:40:430
1280575992317,3cyclictest0-21swapper/220:50:012
1280575992311,8cyclictest261-21systemd-journal19:35:012
1280574992315,5cyclictest3521708-21cups-browsed20:47:471
1280574992315,5cyclictest1306048-21cron20:35:001
128057399239,10cyclictest420-21avahi-daemon20:10:330
1280575992216,3cyclictest0-21swapper/220:05:342
128057399221,17cyclictest1447310-21basename00:25:270
128057399221,17cyclictest1447310-21basename00:25:270
128057399221,16cyclictest3521708-21cups-browsed19:15:370
128057399221,15cyclictest1302445-21ntpq20:20:320
1280573992114,3cyclictest0-21swapper/019:40:000
1280573992114,3cyclictest0-21swapper/019:40:000
1280573992112,7cyclictest261-21systemd-journal20:20:010
128057699182,9cyclictest0-21swapper/300:35:343
128057699181,9cyclictest0-21swapper/322:10:223
128057699181,9cyclictest0-21swapper/322:10:223
1280574991812,3cyclictest0-21swapper/119:10:011
128057699172,8cyclictest0-21swapper/323:45:313
128057599178,6cyclictest1280725-21sed19:10:022
128057699164,6cyclictest0-21swapper/321:25:353
128057699163,6cyclictest0-21swapper/322:00:323
128057699163,6cyclictest0-21swapper/322:00:323
128057699162,8cyclictest0-21swapper/300:20:343
128057699162,7cyclictest0-21swapper/300:25:363
128057699162,7cyclictest0-21swapper/300:25:363
128057399169,5cyclictest421-21dbus-daemon19:10:010
128057699154,4cyclictest0-21swapper/319:40:323
128057699153,7cyclictest0-21swapper/321:40:383
128057699151,9cyclictest0-21swapper/321:30:393
128057699151,8cyclictest0-21swapper/322:25:323
128057699151,8cyclictest0-21swapper/322:15:393
128057699151,6cyclictest0-21swapper/322:35:023
128057699151,6cyclictest0-21swapper/319:55:343
128057699143,4cyclictest0-21swapper/320:25:333
128057699142,7cyclictest0-21swapper/323:35:313
128057699142,7cyclictest0-21swapper/323:15:333
128057699142,7cyclictest0-21swapper/321:15:383
128057699141,8cyclictest0-21swapper/323:00:323
128057699141,8cyclictest0-21swapper/323:00:323
128057699141,7cyclictest0-21swapper/323:55:023
128057699141,7cyclictest0-21swapper/322:45:323
128057699141,7cyclictest0-21swapper/321:55:363
128057699141,7cyclictest0-21swapper/321:20:233
128057699141,7cyclictest0-21swapper/320:15:363
128057699141,6cyclictest0-21swapper/320:35:363
128057699141,6cyclictest0-21swapper/319:35:303
128057699141,6cyclictest0-21swapper/319:35:303
128057699135,6cyclictest0-21swapper/323:55:373
128057699133,6cyclictest0-21swapper/323:10:323
128057699133,5cyclictest0-21swapper/322:20:223
128057699133,4cyclictest0-21swapper/321:00:303
128057699132,7cyclictest0-21swapper/323:20:283
128057699132,6cyclictest0-21swapper/323:40:263
128057699132,6cyclictest0-21swapper/322:50:323
128057699132,6cyclictest0-21swapper/321:45:313
128057699132,6cyclictest0-21swapper/321:35:233
128057699132,6cyclictest0-21swapper/320:40:273
128057699132,6cyclictest0-21swapper/300:15:253
128057699132,6cyclictest0-21swapper/300:00:393
128057699132,6cyclictest0-21swapper/300:00:393
128057699131,7cyclictest0-21swapper/323:25:243
128057699131,7cyclictest0-21swapper/322:55:343
128057699131,7cyclictest0-21swapper/322:40:313
128057699131,7cyclictest0-21swapper/319:35:003
128057699131,7cyclictest0-21swapper/300:30:343
128057699131,6cyclictest0-21swapper/319:10:323
128057699124,6cyclictest0-21swapper/320:10:333
128057699123,7cyclictest0-21swapper/321:10:323
128057699123,5cyclictest0-21swapper/300:10:003
128057699123,5cyclictest0-21swapper/300:10:003
128057699122,8cyclictest0-21swapper/321:10:023
128057699122,6cyclictest0-21swapper/323:30:273
128057699122,6cyclictest0-21swapper/322:05:273
128057699122,6cyclictest0-21swapper/322:05:273
128057699122,6cyclictest0-21swapper/300:10:273
128057699122,6cyclictest0-21swapper/300:10:273
128057699122,5cyclictest0-21swapper/323:05:243
128057699122,5cyclictest0-21swapper/323:05:243
128057699121,7cyclictest0-21swapper/321:50:193
128057699121,7cyclictest0-21swapper/320:00:273
128057699121,6cyclictest0-21swapper/319:15:323
128057699113,5cyclictest0-21swapper/322:35:223
128057699111,6cyclictest0-21swapper/320:45:343
128057699111,6cyclictest0-21swapper/320:30:323
128057699111,5cyclictest0-21swapper/319:50:253
128057699105,4cyclictest0-21swapper/319:45:303
128057699103,4cyclictest431ktimers/319:10:013
128057699100,8cyclictest0-21swapper/320:10:033
128057699100,7cyclictest0-21swapper/320:50:283
12805769995,3cyclictest0-21swapper/319:25:363
12805769994,3cyclictest0-21swapper/320:56:263
12805769983,3cyclictest0-21swapper/320:20:303
12805769982,3cyclictest0-21swapper/319:20:333
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional