You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-10 - 04:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Fri Apr 10, 2026 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1286957995939,16cyclictest695-21cron22:15:012
1286957995631,21cyclictest0-21swapper/220:40:012
128695799562,28cyclictest1470694-21ssh00:10:432
1286956995446,5cyclictest1359744-21systemd-run21:55:011
1286956995445,5cyclictest1331091-21sendmail21:20:011
1286957995326,23cyclictest1466077-21fschecks_time00:05:272
1286955995237,12cyclictest1461257-21awk00:00:000
1286957995123,3cyclictest0-21swapper/200:20:012
1286957995040,7cyclictest1322931-21diskmemload21:15:362
1286957995027,20cyclictest1363791-21idleruntime-cro22:00:012
128695599497,31cyclictest1351319-21sh21:44:320
128695599497,31cyclictest1351319-21sh21:44:320
128695799482,22cyclictest1324265-21ssh21:10:412
1286956994830,12cyclictest1377139-21sh22:15:341
1286956994830,12cyclictest1377139-21sh22:15:341
128695599483,39cyclictest659-21snmpd22:45:440
128695799473,20cyclictest1478724-21ssh00:20:372
1286957994722,2cyclictest0-21swapper/200:00:002
1286957994720,5cyclictest1404538-21sh22:50:022
128695799466,13cyclictest1385310-21ssh22:25:372
1286957994624,5cyclictest0-21swapper/200:30:012
1286957994623,3cyclictest0-21swapper/221:20:472
1286956994638,5cyclictest1313932-21run-parts20:40:011
128695599468,21cyclictest1492757-21sh00:38:050
128695599467,28cyclictest1355395-21ssh21:49:360
1286955994626,15cyclictest0-21swapper/023:05:010
128695599462,27cyclictest1056-21mta-sts-daemon23:51:290
128695799455,12cyclictest1450293-21ssh23:45:382
128695799454,18cyclictest1425849-21ssh23:15:372
128695799451,22cyclictest1348669-21ssh21:40:352
128695799451,22cyclictest1348669-21ssh21:40:352
128695799451,20cyclictest1454279-21ssh23:50:352
1286956994536,6cyclictest0-21swapper/119:55:331
1286956994536,5cyclictest1388673-21aten_r2power_en22:30:231
128695599458,35cyclictest1387642-21ssh22:29:000
1286955994538,4cyclictest151ktimers/021:00:350
1286955994525,10cyclictest12350irq/32-eth022:05:340
1286955994521,21cyclictest1409896-21systemctl22:55:410
128695599451,25cyclictest1377923-21ssh22:15:540
128695599451,25cyclictest1377923-21ssh22:15:540
128695599450,43cyclictest1358977-21ssh21:53:360
1286957994436,5cyclictest1491425-21sh00:35:442
1286957994425,16cyclictest0-21swapper/222:50:272
128695699447,27cyclictest1381279-21ssh22:20:361
128695699443,33cyclictest1348611-21ssh21:40:331
128695699443,33cyclictest1348611-21ssh21:40:331
1286956994335,5cyclictest1343325-21ssh21:35:011
1286956994334,6cyclictest1376039-21systemd22:15:021
128695599436,35cyclictest1476716-21ssh00:18:320
1286955994336,4cyclictest1322934-21munin-run21:10:000
128695599432,35cyclictest1450162-21ntpq23:45:330
1286955994318,22cyclictest0-21swapper/000:35:010
128695599431,26cyclictest1463246-21ssh00:01:040
1286957994236,3cyclictest0-21swapper/200:30:402
1286957994231,7cyclictest1444948-21ssh23:40:012
128695599426,34cyclictest1338944-21ssh21:29:270
128695599424,36cyclictest1434673-21ssh23:25:540
1286955994235,4cyclictest151ktimers/019:45:370
128695599421,38cyclictest1342696-21ssh21:33:500
128695599421,36cyclictest1425906-21ntpq23:15:400
128695599420,30cyclictest1325334-21ssh21:11:490
1286956994133,5cyclictest1312504-21latency_hist20:35:021
1286956994132,4cyclictest0-21swapper/100:20:381
1286956994131,6cyclictest420-21avahi-daemon23:33:401
128695699412,30cyclictest1486773-21ssh00:30:331
128695599414,23cyclictest1390411-21diskmemload22:31:310
1286955994135,3cyclictest0-21swapper/021:20:010
128695599413,36cyclictest1375074-21ssh22:13:160
1286955994133,5cyclictest1294456-21kworker/u8:7-xprtiod19:45:010
128695599411,38cyclictest1480826-21ssh00:23:340
128695599411,38cyclictest1407564-21ssh22:53:190
128695599411,37cyclictest1447915-21sh23:42:540
128695599411,37cyclictest1442330-21ssh23:35:420
128695599411,36cyclictest1313319-21ntpq20:35:340
128695599411,35cyclictest1316297-21ntpq20:45:330
1286957994034,3cyclictest0-21swapper/222:15:402
1286957994034,3cyclictest0-21swapper/222:15:402
128695799403,31cyclictest1417753-21ssh23:05:362
128695799403,31cyclictest1417753-21ssh23:05:362
1286957994032,5cyclictest1306453-21systemd-run20:15:012
1286957994015,17cyclictest421-21dbus-daemon21:30:352
1286956994031,6cyclictest1369015-21ntp_kernel_err22:05:331
1286956994031,5cyclictest0-21swapper/119:15:361
1286956994029,8cyclictest1473848-21/usr/sbin/munin00:15:261
1286956994028,8cyclictest1055-21mta-sts-daemon23:20:341
128695599403,33cyclictest1290867-21ntpq19:20:360
128695599402,34cyclictest1305824-21ntpq20:10:330
128695599402,34cyclictest1292350-21ntpq19:25:340
128695599402,34cyclictest1292350-21ntpq19:25:340
128695599402,33cyclictest1307410-21ntpq20:15:390
128695599401,36cyclictest1295325-21ntpq19:35:340
128695599401,34cyclictest1393505-21ntpq22:35:380
128695599401,34cyclictest1310399-21ntpq20:25:380
128695599400,38cyclictest1430768-21ssh23:21:110
128695599400,37cyclictest1334966-21ssh21:24:390
128695599400,30cyclictest1420044-21ssh23:08:570
128695599400,30cyclictest1420044-21ssh23:08:570
1286957993930,5cyclictest1437010-21latency_hist23:30:022
1286957993927,9cyclictest1298253-21needreboot19:45:322
1286956993931,4cyclictest1310922-21kworker/u8:5+rpciod20:30:011
1286956993929,7cyclictest1315963-21fschecks_time20:45:271
1286956993927,9cyclictest1323363-21/usr/sbin/munin21:10:461
128695599392,32cyclictest1365061-21ntpq22:00:380
128695599391,35cyclictest1469382-21ssh00:10:000
128695599391,35cyclictest1361869-21ssh21:56:220
128695599391,35cyclictest1302849-21ntpq20:00:340
128695599391,33cyclictest1301307-21ntpq19:55:340
128695599391,33cyclictest1293786-21ntpq19:30:320
128695599391,32cyclictest1304353-21ntpq20:05:320
128695599391,32cyclictest1304353-21ntpq20:05:320
128695599390,37cyclictest1382389-21ssh22:21:410
128695599390,37cyclictest1345773-21ssh21:36:530
128695599390,37cyclictest1345773-21ssh21:36:530
1286957993830,5cyclictest421-21dbus-daemon21:55:012
128695699384,26cyclictest1466602-21ssh00:05:381
1286956993832,3cyclictest0-21swapper/100:15:021
1286956993830,5cyclictest1417909-21grep23:05:411
1286956993830,5cyclictest1417909-21grep23:05:411
128695599382,31cyclictest1470553-21ntpq00:10:370
128695599382,31cyclictest1314804-21ntpq20:40:320
128695599382,31cyclictest1308812-21ntpq20:20:320
128695599382,29cyclictest1311824-21ntpq20:30:350
128695599381,34cyclictest1299777-21ntpq19:50:310
128695599381,34cyclictest1289408-21ntpq19:15:380
128695599381,33cyclictest1439947-21ssh23:33:140
128695599380,36cyclictest1423140-21ssh23:12:110
128695599380,36cyclictest1398622-21ssh22:41:410
128695599380,30cyclictest1482459-21irqstats00:25:310
1286957993730,4cyclictest0-21swapper/200:00:372
1286957993729,5cyclictest420-21avahi-daemon23:32:232
1286956993730,3cyclictest0-21swapper/100:00:001
1286956993729,5cyclictest1332895-21unixbench_singl21:20:471
1286956993729,4cyclictest0-21swapper/122:00:461
1286956993729,4cyclictest0-21swapper/121:30:021
1286956993728,4cyclictest0-21swapper/122:36:481
128695599379,25cyclictest151ktimers/020:50:350
128695599371,33cyclictest1319155-21memory20:55:300
1286957993629,4cyclictest1396300-21ls22:40:012
1286957993628,5cyclictest1412509-21cron23:00:012
1286957993627,6cyclictest1322931-21diskmemload22:40:282
1286957993623,6cyclictest1051-21mta-sts-daemon23:00:432
1286956993629,4cyclictest1412563-21kthreadd23:00:021
1286956993629,3cyclictest0-21swapper/120:55:011
1286956993628,3cyclictest0-21swapper/100:00:401
1286956993626,7cyclictest3546645-21cups-browsed22:27:271
128695699362,28cyclictest1421823-21ssh23:10:361
128695599362,30cyclictest1287850-21ntpq19:10:340
128695699352,9cyclictest1413629-21ssh23:00:341
1286956993529,3cyclictest0-21swapper/120:20:001
1286956993526,6cyclictest1306453-21systemd-run20:15:011
128695699352,21cyclictest1442031-21sh23:35:331
1286957993427,3cyclictest0-21swapper/222:25:012
1286957993427,3cyclictest0-21swapper/221:25:252
1286957993426,5cyclictest1343747-21acpi21:35:222
1286957993426,5cyclictest1343747-21acpi21:35:222
1286957993425,6cyclictest1311300-21aten_r2power_po20:30:242
1286957993424,7cyclictest421-21dbus-daemon20:50:242
1286957993423,8cyclictest1389800-21timerandwakeup22:30:452
1286956993428,3cyclictest0-21swapper/123:20:011
1286956993426,5cyclictest1405469-21needreboot22:50:311
1286956993426,5cyclictest1296763-21needreboot19:40:301
1286956993426,4cyclictest0-21swapper/100:25:431
1286956993425,4cyclictest0-21swapper/123:40:441
1286956993423,7cyclictest3546642-21cupsd21:35:211
1286956993423,7cyclictest3546642-21cupsd21:35:211
1286956993422,7cyclictest1322809-21unixbench_multi21:05:431
1286956993420,11cyclictest271ktimers/119:45:391
1286956993420,10cyclictest421-21dbus-daemon21:05:011
1286957993324,6cyclictest1291331-21sed19:20:452
1286956993326,4cyclictest1055-21mta-sts-daemon23:25:401
1286957993225,4cyclictest1352775-21ssh21:45:372
1286957993225,3cyclictest0-21swapper/222:00:362
1286957993223,6cyclictest1309442-21latency_hist20:25:012
1286957993222,6cyclictest1289977-21systemd-run19:20:012
1286957993221,8cyclictest1297221-21timerwakeupswit19:40:432
128695699324,17cyclictest1401493-21ssh22:45:351
1286956993225,4cyclictest1450450-21sendmail_mailst23:45:421
1286956993225,3cyclictest0-21swapper/100:35:281
1286956993224,5cyclictest1315379-21unixbench_singl20:40:451
1286956993224,4cyclictest1054-21mta-sts-daemon22:44:541
1286957993124,3cyclictest0-21swapper/223:20:402
1286957993122,6cyclictest1315455-21systemd-run20:45:012
1286957993122,6cyclictest1055-21mta-sts-daemon23:10:362
1286957993121,7cyclictest1368989-21ntp_kernel_err22:05:322
128695699315,20cyclictest1360905-21ssh21:55:361
1286956993124,5cyclictest1318648-21/usr/sbin/munin20:55:361
1286956993122,6cyclictest1293277-21aten_r2power_fr19:30:241
1286956993121,7cyclictest439-21systemd-logind19:15:011
1286957993022,5cyclictest420-21avahi-daemon19:54:272
1286957993022,5cyclictest1310835-21unixbench_multi20:25:462
1286956993023,3cyclictest0-21swapper/119:25:281
1286956993023,3cyclictest0-21swapper/119:25:281
1286956993021,6cyclictest1051-21mta-sts-daemon21:47:301
1286956993014,13cyclictest271ktimers/119:35:341
1286956993010,17cyclictest271ktimers/123:50:381
128695599305,22cyclictest1286911-21/usr/sbin/munin19:10:010
1286957992924,2cyclictest0-21swapper/221:00:012
1286957992922,3cyclictest0-21swapper/223:40:402
128695799291,21cyclictest421-21dbus-daemon20:15:342
1286957992821,4cyclictest518-21rs:main2
1286956992819,6cyclictest439-21systemd-logind19:55:011
1286957992722,3cyclictest0-21swapper/219:35:012
1286957992721,3cyclictest0-21swapper/220:00:012
1286957992719,5cyclictest1295932-21munin-plugin-st19:40:012
1286956992717,6cyclictest439-21systemd-logind20:25:011
1286957992619,3cyclictest0-21swapper/221:10:022
1286957992520,3cyclictest1054-21mta-sts-daemon20:05:402
1286957992520,3cyclictest1054-21mta-sts-daemon20:05:402
1286957992517,5cyclictest548-21gdbus19:27:512
1286957992517,5cyclictest548-21gdbus19:27:512
1286956992518,4cyclictest0-21swapper/119:25:011
128695799242,14cyclictest3546645-21cups-browsed19:10:352
1286956992418,3cyclictest1056-21mta-sts-daemon20:07:371
1286956992418,3cyclictest1056-21mta-sts-daemon20:07:371
1286957992317,4cyclictest1286954-21cyclictest21:03:252
1286956992315,5cyclictest1302462-21fschecks_count20:00:261
128695799212,14cyclictest1316353-21ntpq20:45:362
128695899193,7cyclictest0-21swapper/322:05:343
128695899182,8cyclictest0-21swapper/322:45:353
128695899182,7cyclictest0-21swapper/322:55:343
1286957991813,2cyclictest0-21swapper/219:10:022
128695699182,13cyclictest1286911-21munin-run19:10:011
128695899172,9cyclictest0-21swapper/322:15:413
128695899172,9cyclictest0-21swapper/322:15:413
128695899172,8cyclictest0-21swapper/322:40:373
128695899162,9cyclictest0-21swapper/323:35:403
128695899162,9cyclictest0-21swapper/322:50:323
128695899162,8cyclictest0-21swapper/321:20:373
128695899162,8cyclictest0-21swapper/300:35:333
128695899162,7cyclictest0-21swapper/321:30:353
128695899162,6cyclictest0-21swapper/300:00:333
128695899161,9cyclictest0-21swapper/323:55:333
128695899161,9cyclictest0-21swapper/323:10:363
128695899161,8cyclictest0-21swapper/321:50:343
128695899161,8cyclictest0-21swapper/321:35:363
128695899161,8cyclictest0-21swapper/321:35:363
128695899161,10cyclictest0-21swapper/322:00:333
128695899153,8cyclictest0-21swapper/323:30:363
128695899153,7cyclictest0-21swapper/321:20:013
128695899152,8cyclictest0-21swapper/323:50:423
128695899152,8cyclictest0-21swapper/321:40:413
128695899152,8cyclictest0-21swapper/321:40:413
128695899151,8cyclictest0-21swapper/323:40:373
128695899151,8cyclictest0-21swapper/323:20:323
128695899151,8cyclictest0-21swapper/322:25:413
128695899151,8cyclictest0-21swapper/321:45:423
128695899151,8cyclictest0-21swapper/300:05:323
128695899143,7cyclictest0-21swapper/321:25:253
128695899143,6cyclictest0-21swapper/300:15:343
128695899142,8cyclictest0-21swapper/323:25:363
128695899142,8cyclictest0-21swapper/300:10:493
128695899142,7cyclictest0-21swapper/323:45:343
128695899142,7cyclictest0-21swapper/322:35:333
128695899142,7cyclictest0-21swapper/322:30:403
128695899142,7cyclictest0-21swapper/321:55:423
128695899142,7cyclictest0-21swapper/319:15:363
128695899142,7cyclictest0-21swapper/300:20:383
128695899142,6cyclictest0-21swapper/320:10:323
128695899141,8cyclictest0-21swapper/323:15:353
128695899141,7cyclictest0-21swapper/319:35:323
128695899141,6cyclictest0-21swapper/319:50:323
128695899141,6cyclictest0-21swapper/319:40:323
128695899141,6cyclictest0-21swapper/319:25:313
128695899141,6cyclictest0-21swapper/319:25:313
128695899133,5cyclictest0-21swapper/320:55:343
128695899132,7cyclictest0-21swapper/322:20:333
128695899132,7cyclictest0-21swapper/300:30:243
128695899132,7cyclictest0-21swapper/300:25:243
128695899132,5cyclictest0-21swapper/320:50:323
128695899132,5cyclictest0-21swapper/320:30:353
128695899131,8cyclictest0-21swapper/323:00:303
128695899131,7cyclictest0-21swapper/323:05:363
128695899131,7cyclictest0-21swapper/323:05:363
128695899131,7cyclictest0-21swapper/322:10:363
128695899131,6cyclictest0-21swapper/321:05:343
128695899131,6cyclictest0-21swapper/320:20:353
128695899131,5cyclictest0-21swapper/319:55:323
128695899123,7cyclictest0-21swapper/321:10:303
128695899123,6cyclictest0-21swapper/320:15:243
128695899122,6cyclictest0-21swapper/319:10:323
128695899122,5cyclictest0-21swapper/319:45:363
128695899122,5cyclictest0-21swapper/319:30:343
128695899121,6cyclictest0-21swapper/320:40:333
128695899121,6cyclictest0-21swapper/320:25:333
128695899121,6cyclictest0-21swapper/320:05:353
128695899121,6cyclictest0-21swapper/320:05:353
128695899121,6cyclictest0-21swapper/319:25:003
128695899111,6cyclictest0-21swapper/321:00:353
128695899111,6cyclictest0-21swapper/320:45:343
128695899111,5cyclictest0-21swapper/320:35:323
128695899102,4cyclictest0-21swapper/320:00:313
12869589981,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional