You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-02 - 01:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Tue Jun 02, 2026 00:42:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1381413995343,6cyclictest1535317-21latency_hist00:15:012
1381413995343,6cyclictest1535317-21latency_hist00:15:012
1381412995126,22cyclictest1484291-21cron22:55:011
1381412995034,13cyclictest0-21swapper/122:01:001
1381412994925,20cyclictest0-21swapper/121:26:051
1381412994925,20cyclictest0-21swapper/121:26:051
138141299482,14cyclictest1472844-21sh22:36:031
1381411994820,20cyclictest12350irq/32-eth023:01:010
1381412994739,5cyclictest1477986-21systemd-run22:45:011
1381412994730,14cyclictest1467048-21grep22:26:151
1381413994639,3cyclictest0-21swapper/223:10:022
1381412994636,3cyclictest0-21swapper/100:16:071
138141299463,11cyclictest1431480-21ssh21:31:051
138141299463,11cyclictest1431480-21ssh21:31:051
1381412994619,24cyclictest1-21systemd21:55:011
1381411994638,5cyclictest1-21systemd00:15:010
1381411994638,5cyclictest1-21systemd00:15:010
1381413994536,5cyclictest0-21swapper/223:11:062
1381413994536,5cyclictest0-21swapper/223:11:062
138141299454,8cyclictest1401758-21ntpq20:16:011
1381412994538,3cyclictest0-21swapper/123:35:001
1381412994538,3cyclictest0-21swapper/123:35:001
1381412994522,20cyclictest1505020-21switchtime23:26:141
1381412994522,20cyclictest1505020-21switchtime23:26:141
138141299445,14cyclictest420-21avahi-daemon23:36:041
138141299445,14cyclictest420-21avahi-daemon23:36:041
1381412994425,15cyclictest1471603-21systemd-run22:35:011
1381412994417,24cyclictest548-21gdbus23:22:511
1381412994417,24cyclictest548-21gdbus23:22:511
1381412994417,23cyclictest1525717-21cron00:00:011
1381412994417,23cyclictest1525717-21cron00:00:011
1381412994415,25cyclictest3835474-21cups-browsed19:10:411
1381412994322,18cyclictest0-21swapper/122:46:201
138141299432,16cyclictest1495058-21sh23:11:021
138141299432,16cyclictest1495058-21sh23:11:021
138141299431,16cyclictest1543069-21ssh00:26:091
138141299431,14cyclictest1385357-21sshd19:21:031
1381411994336,4cyclictest1528931-21idleruntime-cro00:05:000
1381411994336,4cyclictest1528931-21idleruntime-cro00:05:000
1381413994233,6cyclictest420-21avahi-daemon20:46:112
1381412994233,6cyclictest1-21systemd22:25:021
1381412994216,23cyclictest3835474-21cups-browsed20:25:131
138141299421,15cyclictest1425073-21rm21:21:041
1381413994130,8cyclictest1505058-21timerandwakeup23:26:152
1381413994130,8cyclictest1505058-21timerandwakeup23:26:152
1381413994128,10cyclictest1452552-21cat22:05:022
1381413994126,12cyclictest1437656-21needreboot21:40:592
1381413994126,12cyclictest1437656-21needreboot21:40:592
1381412994133,5cyclictest1397925-21systemd-run20:05:021
138141299413,13cyclictest1530182-21ssh00:06:031
138141299413,13cyclictest1530182-21ssh00:06:031
138141299413,10cyclictest1549317-21sshd00:36:041
138141299412,7cyclictest1460138-21ssh22:16:051
1381412994123,15cyclictest0-21swapper/119:40:001
138141299412,10cyclictest421-21dbus-daemon19:16:021
138141299411,17cyclictest1456943-21sh22:11:031
1381413994031,5cyclictest1055-21mta-sts-daemon21:53:322
1381413994028,7cyclictest1056-21mta-sts-daemon22:46:052
138141299409,9cyclictest421-21dbus-daemon20:56:041
138141299404,15cyclictest1546050-21ssh00:31:031
138141299403,8cyclictest1406360-21ntpq20:31:061
1381412994032,5cyclictest1516724-21sh23:45:541
138141299401,16cyclictest421-21dbus-daemon23:20:011
138141299401,16cyclictest421-21dbus-daemon23:20:011
1381411994034,4cyclictest1051-21mta-sts-daemon23:21:030
1381411994034,4cyclictest1051-21mta-sts-daemon23:21:030
138141299394,9cyclictest1440979-21ntpq21:46:041
138141299394,9cyclictest1440979-21ntpq21:46:041
138141299393,25cyclictest1453744-21ssh22:06:031
1381412993915,2cyclictest0-21swapper/119:30:011
1381412993914,2cyclictest0-21swapper/121:16:141
1381412993913,2cyclictest0-21swapper/123:41:041
1381412993913,2cyclictest0-21swapper/123:41:041
1381412993913,2cyclictest0-21swapper/123:00:541
1381412993912,5cyclictest0-21swapper/100:20:541
138141299391,12cyclictest1527087-21ntp_states00:01:061
138141299391,12cyclictest1527087-21ntp_states00:01:061
138141299390,14cyclictest1533356-21sshd00:11:041
138141299390,14cyclictest1533356-21sshd00:11:041
1381411993931,5cyclictest1421872-21ntp_kernel_pll_21:16:060
1381411993927,9cyclictest1417389-21munin-plugin-st21:10:000
1381411993927,9cyclictest1417389-21munin-plugin-st21:10:000
138141199392,30cyclictest1539740-21ssh00:21:030
1381413993830,3cyclictest0-21swapper/200:06:042
1381413993830,3cyclictest0-21swapper/200:06:042
1381413993829,6cyclictest439-21systemd-logind23:40:012
1381413993829,6cyclictest439-21systemd-logind23:40:012
1381413993829,6cyclictest0-21swapper/200:20:002
1381413993829,5cyclictest420-21avahi-daemon22:26:342
1381412993830,4cyclictest1438367-21unixbench_singl21:41:141
1381412993830,4cyclictest1438367-21unixbench_singl21:41:141
138141299381,14cyclictest1446708-21aten2.4-expect21:55:521
1381411993829,6cyclictest3835474-21cups-browsed23:26:540
1381411993829,6cyclictest3835474-21cups-browsed23:26:540
1381411993826,5cyclictest1399412-21systemd-run20:10:010
1381413993729,3cyclictest0-21swapper/219:16:012
1381413993727,6cyclictest3835474-21cups-browsed22:30:352
1381413993726,8cyclictest1426986-21idleruntime-cro21:25:002
1381412993715,19cyclictest0-21swapper/119:45:241
138141299371,13cyclictest1434736-21ps21:36:081
138141299371,13cyclictest1434736-21ps21:36:081
138141299371,11cyclictest1418566-21ntpq21:11:001
138141299371,11cyclictest1418566-21ntpq21:11:001
138141199371,29cyclictest1482391-21ssh22:51:050
1381413993629,4cyclictest1481147-21kworker/u8:3-rpciod23:56:132
1381413993629,4cyclictest1481147-21kworker/u8:3-rpciod23:56:132
1381413993629,4cyclictest1420568-21sh21:15:012
1381413993629,4cyclictest1420568-21sh21:15:012
1381413993629,3cyclictest0-21swapper/219:11:052
1381413993628,5cyclictest1476570-21unixbench_multi22:41:142
1381413993627,6cyclictest1528924-21ssh00:05:012
1381413993627,6cyclictest1528924-21ssh00:05:012
1381413993627,6cyclictest1413409-21fschecks_time20:55:582
138141299364,7cyclictest1400265-21ntpq20:11:011
138141299362,15cyclictest1403337-21ntpq20:21:041
1381412993614,2cyclictest0-21swapper/122:55:561
1381413993527,5cyclictest1515463-21sh23:43:042
1381413993527,5cyclictest1515463-21sh23:43:042
1381413993527,5cyclictest0-21swapper/219:40:012
138141299352,9cyclictest1391282-21ntpq19:41:021
138141299352,9cyclictest1391282-21ntpq19:41:021
138141299351,14cyclictest1416911-21ps21:06:091
138141299351,14cyclictest1416911-21ps21:06:091
1381411993529,4cyclictest1513000-21systemd-run23:40:010
1381411993529,4cyclictest1513000-21systemd-run23:40:010
1381411993528,4cyclictest1388969-21latency_hist19:35:020
1381411993527,4cyclictest151ktimers/000:16:020
1381411993524,4cyclictest261-21systemd-journal22:00:000
1381411993512,8cyclictest1382373-21ntpq19:11:060
1381413993427,4cyclictest1545997-21ssh00:31:002
1381413993426,5cyclictest1519331-21grep23:50:002
138141399342,24cyclictest1549318-21ssh00:36:042
138141299342,14cyclictest1491891-21ssh23:06:021
1381411993429,2cyclictest0-21swapper/023:55:520
1381411993429,2cyclictest0-21swapper/023:55:520
1381411993418,10cyclictest1546026-21ntpq00:31:010
138141399336,20cyclictest1460207-21ssh22:16:082
138141399333,22cyclictest1434569-21ssh21:36:022
138141399333,22cyclictest1434569-21ssh21:36:022
1381413993327,3cyclictest0-21swapper/222:00:002
1381413993326,3cyclictest0-21swapper/222:06:042
1381413993325,5cyclictest1415811-21unixbench_singl21:01:172
1381413993325,5cyclictest1415811-21unixbench_singl21:01:172
1381413993324,5cyclictest1412894-21systemd-run20:55:022
1381413993315,14cyclictest351ktimers/200:21:062
138141299331,13cyclictest1410917-21ps20:46:101
138141199338,7cyclictest1478470-21rm22:45:510
1381411993327,4cyclictest1462865-21sort22:20:560
1381411993327,4cyclictest1437998-21mailstats21:41:090
1381411993327,4cyclictest1437998-21mailstats21:41:090
1381411993324,6cyclictest1459680-21fschecks_time22:15:560
1381411993319,11cyclictest1471575-21cron22:35:010
138141199331,27cyclictest659-21snmpd20:46:030
138141399323,24cyclictest1501555-21ssh23:21:032
138141399323,24cyclictest1501555-21ssh23:21:032
138141399323,20cyclictest1440995-21ssh21:46:052
138141399323,20cyclictest1440995-21ssh21:46:052
1381413993223,6cyclictest1381422-21systemd-run19:10:022
1381413993223,4cyclictest1426999-21kworker/u8:4-rpciod21:30:582
1381413993223,4cyclictest1426999-21kworker/u8:4-rpciod21:30:582
1381412993224,5cyclictest0-21swapper/120:55:011
1381412993223,6cyclictest1-21systemd19:10:011
138141299321,13cyclictest1415420-21ps21:01:121
138141299321,13cyclictest1415420-21ps21:01:121
1381411993225,4cyclictest1396933-21fschecks_time20:00:550
1381411993224,6cyclictest1543348-21timerwakeupswit00:26:150
1381411993223,5cyclictest151ktimers/023:06:090
1381411993222,5cyclictest1498307-21sshd23:16:040
1381411993222,5cyclictest1498307-21sshd23:16:040
1381411993220,7cyclictest1406237-21ntpq20:31:000
1381411993216,13cyclictest1394265-21ntpq19:51:020
1381411993216,12cyclictest12350irq/32-eth021:11:060
1381411993216,12cyclictest12350irq/32-eth021:11:060
138141399313,16cyclictest1542844-21ssh00:26:012
1381413993123,3cyclictest0-21swapper/221:26:032
1381413993123,3cyclictest0-21swapper/221:26:032
1381413993121,7cyclictest1507674-21missed_timers23:30:582
1381413993121,7cyclictest1507674-21missed_timers23:30:582
1381413993120,8cyclictest3936059-21munin-node22:36:242
1381411993124,4cyclictest1507452-21ssh23:30:550
1381411993124,4cyclictest1507452-21ssh23:30:550
138141199312,23cyclictest1549344-21rm00:36:060
1381411993121,7cyclictest1401385-21diskstats20:15:540
1381411993120,5cyclictest1403252-21ntpq20:21:000
1381411993119,8cyclictest1383791-21ntpq19:16:010
1381411993119,7cyclictest1472794-21ntpq22:36:020
1381411993118,9cyclictest1389779-21ntpq19:36:020
1381411993117,9cyclictest12350irq/32-eth023:41:010
1381411993117,9cyclictest12350irq/32-eth023:41:010
1381411993116,12cyclictest1428175-21ntpq21:26:010
1381411993116,12cyclictest1428175-21ntpq21:26:010
1381411993114,14cyclictest1415282-21ntpq21:01:050
1381411993114,14cyclictest1415282-21ntpq21:01:050
1381413993023,4cyclictest1408934-21fschecks_time20:40:592
1381413993023,3cyclictest0-21swapper/223:16:042
1381413993023,3cyclictest0-21swapper/223:16:042
1381413993023,3cyclictest0-21swapper/220:36:032
1381413993022,5cyclictest1416957-21sendmail_mailqu21:06:102
1381413993022,5cyclictest1416957-21sendmail_mailqu21:06:102
1381413993022,5cyclictest1399397-21munin-run20:10:012
138141399302,18cyclictest1485584-21ssh22:56:062
1381412993023,4cyclictest1522516-21munin-plugin-st23:55:001
138141199307,7cyclictest1054-21mta-sts-daemon19:21:050
138141199303,20cyclictest1431426-21ssh21:31:030
138141199303,20cyclictest1431426-21ssh21:31:030
1381411993025,2cyclictest0-21swapper/022:41:140
1381411993020,5cyclictest0-21swapper/023:51:010
1381411993020,4cyclictest1532100-21kworker/u8:2-rpciod00:10:000
1381411993020,4cyclictest1532100-21kworker/u8:2-rpciod00:10:000
1381411993019,7cyclictest1392816-21ntpq19:46:050
1381411993018,7cyclictest1453805-21ntpq22:06:050
1381411993017,8cyclictest1404790-21ntpq20:25:590
138141199301,24cyclictest1444078-21ssh21:51:000
1381413992923,3cyclictest0-21swapper/219:35:012
1381413992922,4cyclictest1056-21mta-sts-daemon22:25:022
1381413992922,4cyclictest1054-21mta-sts-daemon23:50:152
1381413992922,4cyclictest1054-21mta-sts-daemon23:01:062
138141399292,23cyclictest1482459-21ssh22:51:082
1381411992924,2cyclictest0-21swapper/021:40:000
1381411992924,2cyclictest0-21swapper/021:40:000
1381411992923,4cyclictest1466839-21awk22:26:130
1381411992923,4cyclictest1407993-21systemctl20:36:100
1381411992918,7cyclictest1395739-21ntpq19:56:000
1381411992915,10cyclictest1485510-21ntpq22:56:020
1381411992914,7cyclictest1391368-21ntpq19:41:060
1381411992914,7cyclictest1391368-21ntpq19:41:060
1381413992820,5cyclictest1-21systemd19:46:112
1381413992818,8cyclictest0-21swapper/219:55:572
1381413992817,8cyclictest1386778-21ntp_kernel_pll_19:26:012
138141399281,22cyclictest1421835-21ntpq21:16:052
1381412992823,2cyclictest0-21swapper/120:45:001
138141199287,13cyclictest12350irq/32-eth023:11:010
138141199287,13cyclictest12350irq/32-eth023:11:010
1381411992823,2cyclictest0-21swapper/021:00:010
138141199282,21cyclictest1450614-21ssh22:01:050
138141199282,21cyclictest1440917-21ssh21:46:020
138141199282,21cyclictest1440917-21ssh21:46:020
1381411992818,6cyclictest1412264-21ntpq20:51:000
1381411992817,7cyclictest1400359-21ntpq20:11:050
1381411992816,8cyclictest1386761-21ntpq19:25:590
138141199281,22cyclictest1456926-21awk22:11:020
138141399272,19cyclictest1397342-21sshd20:01:042
1381413992719,5cyclictest1384931-21fschecks_count19:20:542
1381413992716,8cyclictest1394876-21cron19:55:012
1381411992721,4cyclictest0-21swapper/023:45:580
1381411992716,8cyclictest1409134-21memory20:41:020
1381411992713,10cyclictest1424142-21ssh21:20:230
1381413992620,4cyclictest1054-21mta-sts-daemon20:30:582
1381413992619,4cyclictest1390869-21cpuspeed19:40:542
1381413992619,4cyclictest1390869-21cpuspeed19:40:542
1381413992619,3cyclictest0-21swapper/220:11:132
1381412992618,5cyclictest695-21cron19:55:011
138141399252,16cyclictest1403345-21sshd20:21:042
1381413992518,3cyclictest0-21swapper/220:26:082
1381413992516,6cyclictest1402287-21grep20:16:132
1381412992518,4cyclictest1054-21mta-sts-daemon19:55:521
1381413992417,4cyclictest0-21swapper/222:11:012
138141299244,13cyclictest428-21mta-sts-daemon20:36:031
1381411992415,6cyclictest1381423-21munin-run19:10:020
1381412992313,7cyclictest1398315-21wc20:05:521
138141299221,16cyclictest1388290-21ntpq19:31:011
138141499192,9cyclictest0-21swapper/322:51:103
138141499192,8cyclictest0-21swapper/321:01:083
138141499192,8cyclictest0-21swapper/321:01:083
138141499182,10cyclictest0-21swapper/300:16:033
138141499181,11cyclictest0-21swapper/300:06:043
138141499181,11cyclictest0-21swapper/300:06:043
138141499173,8cyclictest0-21swapper/300:31:033
138141499173,11cyclictest0-21swapper/300:35:583
138141499172,9cyclictest0-21swapper/319:16:023
138141499172,8cyclictest0-21swapper/322:36:073
138141499172,8cyclictest0-21swapper/322:06:023
138141499172,10cyclictest0-21swapper/323:11:043
138141499172,10cyclictest0-21swapper/323:11:043
138141499165,7cyclictest0-21swapper/323:21:003
138141499165,7cyclictest0-21swapper/323:21:003
138141499164,9cyclictest0-21swapper/323:01:013
138141499163,8cyclictest0-21swapper/322:31:063
138141499163,8cyclictest0-21swapper/321:56:043
138141499163,7cyclictest0-21swapper/321:16:033
138141499163,7cyclictest0-21swapper/300:01:013
138141499163,7cyclictest0-21swapper/300:01:013
138141499162,8cyclictest0-21swapper/323:56:073
138141499162,8cyclictest0-21swapper/323:56:073
138141499161,9cyclictest0-21swapper/323:31:013
138141499161,9cyclictest0-21swapper/323:31:013
138141499161,8cyclictest0-21swapper/323:36:033
138141499161,8cyclictest0-21swapper/323:36:033
138141499161,8cyclictest0-21swapper/323:26:023
138141499161,8cyclictest0-21swapper/323:26:023
138141499161,8cyclictest0-21swapper/300:21:003
138141499161,7cyclictest0-21swapper/323:16:023
138141499161,7cyclictest0-21swapper/323:16:023
138141499161,10cyclictest0-21swapper/321:26:023
138141499161,10cyclictest0-21swapper/321:26:023
138141499153,7cyclictest0-21swapper/319:31:063
138141499152,9cyclictest0-21swapper/323:46:043
138141499152,8cyclictest0-21swapper/322:01:003
138141499152,8cyclictest0-21swapper/320:30:023
138141499152,7cyclictest0-21swapper/322:41:003
138141499152,7cyclictest0-21swapper/321:51:023
138141499152,7cyclictest0-21swapper/320:46:033
138141499152,6cyclictest0-21swapper/319:11:033
138141499151,9cyclictest0-21swapper/300:25:523
138141499151,8cyclictest0-21swapper/323:51:053
138141499151,8cyclictest0-21swapper/322:46:083
138141499151,8cyclictest0-21swapper/320:41:093
138141499151,6cyclictest0-21swapper/320:16:023
138141499143,7cyclictest0-21swapper/322:11:023
138141499143,7cyclictest0-21swapper/321:21:003
138141499143,6cyclictest0-21swapper/319:20:593
138141499143,6cyclictest0-21swapper/300:11:043
138141499143,6cyclictest0-21swapper/300:11:043
138141499142,8cyclictest0-21swapper/323:10:013
138141499142,8cyclictest0-21swapper/322:56:063
138141499142,8cyclictest0-21swapper/321:35:553
138141499142,8cyclictest0-21swapper/321:35:553
138141499141,8cyclictest0-21swapper/323:40:523
138141499141,8cyclictest0-21swapper/323:40:523
138141499141,8cyclictest0-21swapper/322:30:013
138141499141,8cyclictest0-21swapper/320:01:083
138141499141,7cyclictest0-21swapper/321:41:013
138141499141,7cyclictest0-21swapper/321:41:013
138141499141,7cyclictest0-21swapper/320:36:043
138141499141,6cyclictest0-21swapper/322:20:193
138141499132,7cyclictest0-21swapper/321:30:033
138141499132,7cyclictest0-21swapper/321:30:033
138141499132,7cyclictest0-21swapper/319:41:033
138141499132,7cyclictest0-21swapper/319:41:033
138141499131,8cyclictest0-21swapper/321:46:043
138141499131,8cyclictest0-21swapper/321:46:043
138141499131,7cyclictest0-21swapper/322:16:013
138141499131,7cyclictest0-21swapper/321:10:573
138141499131,7cyclictest0-21swapper/321:10:573
138141499131,7cyclictest0-21swapper/320:51:063
138141499122,6cyclictest0-21swapper/320:11:043
138141499122,6cyclictest0-21swapper/319:40:003
138141499121,7cyclictest0-21swapper/320:56:053
138141499121,6cyclictest0-21swapper/320:30:523
138141499121,6cyclictest0-21swapper/319:51:013
138141499112,5cyclictest0-21swapper/320:06:023
138141499112,5cyclictest0-21swapper/319:56:013
138141499111,6cyclictest0-21swapper/321:06:023
138141499111,6cyclictest0-21swapper/321:06:023
138141499111,6cyclictest0-21swapper/319:46:043
138141499111,6cyclictest0-21swapper/319:26:003
138141499101,6cyclictest0-21swapper/320:20:553
13814149991,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional