You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-04 - 05:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Tue Aug 04, 2026 00:43:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1076088995444,7cyclictest1158062-21latency_hist22:25:022
1076087995041,5cyclictest0-21swapper/123:40:541
1076086994918,23cyclictest665-21snmpd22:55:500
1076087994840,5cyclictest1176127-21systemd-run22:55:021
1076086994831,5cyclictest1182765-21aten_r2power_vo23:05:410
1076087994636,7cyclictest1086711-21kworker/u8:13-xprtiod21:25:551
1076087994636,7cyclictest1086711-21kworker/u8:13-xprtiod21:25:551
1076088994536,6cyclictest1188233-21grep23:15:012
1076086994527,14cyclictest1218156-40mandb00:04:060
1076087994435,6cyclictest421-21dbus-daemon23:15:011
107608799441,25cyclictest1150014-21diskmemload22:10:521
1076088994334,5cyclictest1103060-21sendmail20:40:012
1076087994329,11cyclictest1138261-21timerwakeupswit21:51:031
107608799431,37cyclictest1213533-21ssh23:55:531
107608699432,35cyclictest151ktimers/022:00:560
107608699432,35cyclictest151ktimers/022:00:560
1076087994235,4cyclictest1179135-21sendmail23:00:011
1076087994234,4cyclictest0-21swapper/123:20:011
1076087994233,5cyclictest1197782-21aten_r2power_en23:30:411
1076088994033,4cyclictest1230591-21munin-run00:25:012
1076088994032,5cyclictest1121282-21ssh21:25:012
1076088994032,5cyclictest1121282-21ssh21:25:012
1076088994030,6cyclictest420-21avahi-daemon20:21:042
1076088994027,9cyclictest1225509-21missed_timers00:15:492
107608799405,30cyclictest1207549-21ssh23:45:591
1076087994031,6cyclictest1142794-21latency_hist22:00:021
1076087994030,6cyclictest1118283-21latency_hist21:20:021
1076087994029,4cyclictest0-21swapper/122:15:571
107608899395,28cyclictest1216678-21rm00:00:552
1076087993931,5cyclictest1109105-21systemd-run21:00:011
1076087993930,6cyclictest1100080-21munin-plugin-st20:30:011
1076087993930,5cyclictest1133559-21munin-plugin-st21:45:011
1076087993921,13cyclictest3575727-21cups-browsed23:00:591
1076087993921,13cyclictest3575727-21cups-browsed23:00:591
1076086993932,5cyclictest1191391-21latency_hist23:20:020
1076086993932,4cyclictest1134629-21needreboot21:45:500
1076088993829,6cyclictest1107564-21munin-plugin-st20:55:012
1076087993831,4cyclictest1197294-21systemd-run23:30:011
1076087993831,3cyclictest0-21swapper/122:21:121
1076088993729,5cyclictest1224514-21latency_hist00:15:012
1076088993728,6cyclictest1133598-21latency_hist21:45:022
1076087993730,3cyclictest0-21swapper/122:01:041
1076087993730,3cyclictest0-21swapper/122:01:041
1076087993729,5cyclictest1127483-21latency_hist21:35:011
1076087993729,5cyclictest1127483-21latency_hist21:35:011
1076086993731,4cyclictest1101573-21cut20:35:010
1076086993730,4cyclictest1159657-21unixbench-2d22:26:090
1076088993629,4cyclictest1160982-21munin-run22:30:012
1076088993628,5cyclictest1212401-21systemd-run23:55:022
1076088993628,4cyclictest0-21swapper/221:15:552
1076087993629,4cyclictest1185206-21idleruntime-cro23:10:001
1076087993628,5cyclictest1233640-21systemd-run00:30:021
1076087993627,5cyclictest1098582-21munin-plugin-st20:25:011
1076086993630,4cyclictest1000-21mta-sts-daemon23:40:530
1076086993628,5cyclictest420-21avahi-daemon22:53:390
1076086993627,7cyclictest1227545-21munin-plugin-st00:20:010
107608899353,24cyclictest1122470-21ssh21:25:542
107608899353,24cyclictest1122470-21ssh21:25:542
1076088993530,2cyclictest0-21swapper/200:40:012
1076088993527,5cyclictest1128216-21gltestperf21:35:452
1076088993526,5cyclictest1079073-21munin-run19:20:012
1076088993525,6cyclictest420-21avahi-daemon23:30:092
107608899352,26cyclictest1153038-21ssh22:15:502
1076087993527,4cyclictest1002-21mta-sts-daemon00:15:551
1076086993528,4cyclictest1003-21mta-sts-daemon22:33:200
1076086993526,6cyclictest1215473-21cups-browsed00:05:100
1076086993524,7cyclictest3575727-21cups-browsed20:40:540
107608699352,22cyclictest3575727-21cups-browsed21:20:580
107608699352,22cyclictest3575727-21cups-browsed21:20:580
1076086993520,10cyclictest12650irq/32-eth021:16:000
1076088993427,4cyclictest1164034-21latency_hist22:35:022
1076088993424,6cyclictest1-21systemd23:30:012
1076088993424,6cyclictest1176107-21idleruntime-cro22:55:012
1076088993422,8cyclictest1180766-21timerwakeupswit23:01:042
1076088993422,8cyclictest1180766-21timerwakeupswit23:01:042
1076087993426,5cyclictest1079075-21munin-plugin-st19:20:011
1076087993426,4cyclictest0-21swapper/122:45:531
1076087993425,7cyclictest1080589-21/usr/sbin/munin19:25:021
1076087993425,6cyclictest1003-21mta-sts-daemon22:05:501
1076087993425,6cyclictest1003-21mta-sts-daemon22:05:501
1076087993424,7cyclictest1165666-21unixbench_multi22:36:021
1076086993428,4cyclictest1235141-21timerandwakeup00:31:030
1076086993428,4cyclictest1128195-21fschecks_count21:35:440
1076086993427,5cyclictest0-21swapper/023:15:020
1076086993423,8cyclictest1223254-21users00:11:020
1076086993421,10cyclictest1103056-21cron20:40:010
107608699341,27cyclictest1201549-21ntp_states23:35:570
1076088993326,4cyclictest1086809-21/usr/sbin/munin19:46:062
1076088993326,4cyclictest1086809-21/usr/sbin/munin19:46:062
1076088993325,5cyclictest420-21avahi-daemon21:10:522
1076088993324,5cyclictest420-21avahi-daemon23:57:302
1076088993322,8cyclictest1204527-21gcc23:40:532
1076088993314,15cyclictest421-21dbus-daemon20:05:512
107608899331,26cyclictest1147115-21ssh22:06:002
107608899331,26cyclictest1147115-21ssh22:06:002
107608899331,26cyclictest1137965-21ssh21:50:582
1076087993325,5cyclictest1220009-21timerandwakeup00:06:021
1076087993325,5cyclictest1105949-21tail20:46:031
1076087993324,5cyclictest1193026-21unixbench_singl23:21:141
107608799331,29cyclictest1168262-21ssh22:40:551
1076086993327,4cyclictest1081098-21fschecks_count19:25:430
1076086993326,4cyclictest0-21swapper/021:40:520
1076086993325,6cyclictest421-21dbus-daemon23:05:010
1076086993325,6cyclictest421-21dbus-daemon23:05:010
1076086993325,4cyclictest1182208-21kworker/u8:15-rpciod23:45:470
1076086993321,3cyclictest0-21swapper/022:20:580
1076088993224,4cyclictest0-21swapper/222:45:562
1076088993224,3cyclictest0-21swapper/222:00:582
1076088993224,3cyclictest0-21swapper/222:00:582
1076088993223,7cyclictest694-21cron00:35:012
1076087993224,5cyclictest1101585-21systemd-run20:35:011
1076087993223,6cyclictest1-21systemd21:25:021
1076087993223,6cyclictest1-21systemd21:25:021
107608799322,23cyclictest1234743-21ssh00:30:521
1076087993221,8cyclictest1093390-21needreboot20:05:471
1076086993226,3cyclictest0-21swapper/022:15:000
1076086993226,3cyclictest0-21swapper/021:30:490
1076086993226,3cyclictest0-21swapper/021:30:490
1076086993225,4cyclictest1239698-21systemd-run00:40:020
1076086993212,15cyclictest1165269-21awk22:35:560
1076088993124,3cyclictest0-21swapper/221:30:542
1076088993124,3cyclictest0-21swapper/221:30:542
1076088993123,5cyclictest1208029-21unixbench-2d23:46:102
1076088993123,5cyclictest1134817-21ntpq21:45:582
1076088993123,5cyclictest1091005-21unixbench_singl19:56:022
1076088993123,5cyclictest1091005-21unixbench_singl19:56:022
1076088993123,2cyclictest0-21swapper/200:05:372
1076087993123,5cyclictest1-21systemd23:40:011
1076087993123,5cyclictest1113834-21grep21:11:031
1076087993123,5cyclictest1001-21mta-sts-daemon20:10:441
1076087993122,6cyclictest1215473-21cups-browsed00:24:381
107608799312,22cyclictest1134914-21ssh21:46:001
1076087993121,7cyclictest1216746-21wc00:00:581
1076086993126,2cyclictest0-21swapper/000:21:030
1076086993125,4cyclictest1086604-21systemd-run19:45:010
1076086993124,4cyclictest1198017-21fschecks_count23:30:440
1076086993124,4cyclictest0-21swapper/022:15:530
107608899304,7cyclictest171rcu_preempt20:00:532
107608899303,20cyclictest1140807-21ssh21:55:512
1076088993023,4cyclictest1002-21mta-sts-daemon23:05:522
1076088993023,3cyclictest0-21swapper/223:35:482
1076088993023,3cyclictest0-21swapper/200:30:012
1076088993022,6cyclictest1164411-21/usr/sbin/munin22:35:432
107608899302,23cyclictest1168200-21ssh22:40:522
107608899302,22cyclictest1189362-21ssh23:15:512
1076087993023,4cyclictest1107466-21tail20:51:031
1076087993017,9cyclictest1003-21mta-sts-daemon19:55:511
1076087993017,9cyclictest1003-21mta-sts-daemon19:55:511
1076087993014,11cyclictest3575727-21cups-browsed19:25:501
1076086993024,3cyclictest0-21swapper/023:50:420
1076086993024,3cyclictest0-21swapper/021:50:440
1076088992924,2cyclictest0-21swapper/220:20:012
1076088992923,2cyclictest0-21swapper/222:15:002
107608799293,19cyclictest1237734-21ssh00:35:491
1076087992923,3cyclictest0-21swapper/121:40:011
1076087992921,5cyclictest694-21cron19:15:011
1076087992920,6cyclictest1159471-21switchtime22:26:081
107608799291,21cyclictest1210661-21sh23:50:581
1076086992924,2cyclictest0-21swapper/000:00:010
1076086992923,4cyclictest1109099-21systemd-run21:00:010
107608699292,22cyclictest1083126-21ps19:30:560
1076086992921,6cyclictest1105419-21ntp_kernel_err20:45:500
1076086992921,5cyclictest1146584-21gltestperf22:05:460
1076086992921,5cyclictest1146584-21gltestperf22:05:460
1076086992920,6cyclictest439-21systemd-logind20:20:020
1076086992920,6cyclictest3575727-21cups-browsed22:42:380
1076088992823,2cyclictest0-21swapper/223:25:012
1076088992822,3cyclictest0-21swapper/220:35:022
1076088992821,5cyclictest1177011-21ssh22:55:452
1076088992821,4cyclictest1000-21mta-sts-daemon19:25:522
1076088992820,5cyclictest421-21dbus-daemon20:26:132
1076087992822,4cyclictest1003-21mta-sts-daemon00:10:381
1076086992824,2cyclictest0-21swapper/021:10:010
1076086992823,2cyclictest0-21swapper/020:00:000
1076086992823,2cyclictest0-21swapper/020:00:000
1076086992822,4cyclictest1091606-21grep20:00:450
1076086992821,5cyclictest623-21sshd00:25:300
1076086992819,6cyclictest1-21systemd19:10:010
1076086992819,6cyclictest1-21systemd19:10:010
1076086992817,8cyclictest0-21swapper/019:40:010
107608699281,20cyclictest1112072-21diskmemload23:25:540
1076088992720,4cyclictest1003-21mta-sts-daemon20:40:552
1076088992719,5cyclictest1089484-21head19:51:022
1076088992719,5cyclictest1089484-21head19:51:022
1076087992722,2cyclictest0-21swapper/119:35:011
1076087992721,3cyclictest1000-21mta-sts-daemon22:32:331
1076087992720,4cyclictest1097074-21munin-run20:20:011
1076087992720,4cyclictest1076107-21latency_hist19:10:011
1076087992720,4cyclictest1076107-21latency_hist19:10:011
1076087992720,4cyclictest1000-21mta-sts-daemon20:00:571
1076087992718,6cyclictest0-21swapper/120:40:011
1076086992721,3cyclictest1079125-21latency_hist19:20:020
1076086992719,4cyclictest1076085-21cyclictest21:55:550
1076086992718,5cyclictest12650irq/32-eth021:10:560
1076086992716,4cyclictest1074746-21kworker/u8:14+xprtiod19:15:020
1076086992714,10cyclictest0-21swapper/020:10:490
1076088992619,4cyclictest0-21swapper/221:10:012
1076088992617,6cyclictest3575727-21cups-browsed19:30:372
1076088992614,7cyclictest627-21lldpd20:45:512
1076087992619,4cyclictest1002-21mta-sts-daemon19:53:461
1076087992619,4cyclictest1002-21mta-sts-daemon19:53:461
1076087992619,4cyclictest0-21swapper/121:10:021
1076087992613,8cyclictest665-21snmpd19:35:501
107608699264,13cyclictest12650irq/32-eth023:20:590
107608699262,5cyclictest171rcu_preempt22:45:580
107608699262,19cyclictest567-21gdbus20:20:540
1076086992618,5cyclictest1093096-21fschecks_count20:05:430
1076088992519,3cyclictest0-21swapper/219:23:292
1076088992517,5cyclictest1110581-21ls21:05:012
107608899250,3cyclictest171rcu_preempt20:55:542
1076087992519,3cyclictest0-21swapper/119:46:131
1076087992519,3cyclictest0-21swapper/119:46:131
1076087992517,5cyclictest487-21rs:main1
107608699252,17cyclictest1122612-21ps21:25:590
107608699252,17cyclictest1122612-21ps21:25:590
1076086992520,3cyclictest1002-21mta-sts-daemon20:50:470
1076086992513,8cyclictest151ktimers/019:20:560
1076086992512,10cyclictest1099114-21fschecks_time20:25:430
1076088992418,3cyclictest0-21swapper/219:15:012
1076086992416,5cyclictest0-21swapper/019:46:040
1076086992416,5cyclictest0-21swapper/019:46:040
1076088992316,5cyclictest3575727-21cups-browsed20:15:022
1076088992316,3cyclictest0-21swapper/219:40:532
1076088992314,6cyclictest1076091-21systemd-run19:10:012
1076088992314,6cyclictest1076091-21systemd-run19:10:012
1076088992313,7cyclictest1085033-21awk19:36:032
1076087992317,3cyclictest0-21swapper/120:40:571
1076086992316,5cyclictest1110293-21timerandwakeup21:01:000
107608799211,9cyclictest265-21systemd-journal19:45:011
107608699212,14cyclictest1088945-21ntpq19:50:500
107608699212,14cyclictest1088945-21ntpq19:50:500
107608999204,9cyclictest0-21swapper/323:45:593
107608999204,9cyclictest0-21swapper/321:15:563
107608999203,9cyclictest0-21swapper/321:30:503
107608999203,9cyclictest0-21swapper/321:30:503
107608999203,8cyclictest0-21swapper/300:05:563
107608999202,10cyclictest0-21swapper/300:20:553
107608999195,7cyclictest0-21swapper/300:04:063
107608999193,9cyclictest0-21swapper/323:40:483
107608999192,11cyclictest0-21swapper/322:45:593
107608999192,11cyclictest0-21swapper/321:11:293
107608999192,10cyclictest0-21swapper/321:55:543
107608999184,9cyclictest0-21swapper/323:15:023
107608999183,8cyclictest0-21swapper/323:30:523
107608999183,7cyclictest0-21swapper/322:20:563
107608999183,10cyclictest0-21swapper/322:35:563
107608999183,10cyclictest0-21swapper/300:10:483
107608999182,9cyclictest0-21swapper/322:50:513
107608999182,9cyclictest0-21swapper/322:10:473
107608999182,10cyclictest0-21swapper/322:26:033
107608999181,9cyclictest0-21swapper/300:35:483
107608999181,10cyclictest0-21swapper/322:00:583
107608999181,10cyclictest0-21swapper/322:00:583
107608999175,9cyclictest0-21swapper/323:25:543
107608999174,8cyclictest0-21swapper/323:00:593
107608999174,8cyclictest0-21swapper/323:00:593
107608999173,9cyclictest0-21swapper/321:35:543
107608999173,8cyclictest0-21swapper/321:46:043
107608999172,9cyclictest0-21swapper/322:15:523
107608999172,9cyclictest0-21swapper/320:20:543
107608999172,8cyclictest0-21swapper/322:30:523
107608999172,8cyclictest0-21swapper/321:50:413
107608999171,9cyclictest0-21swapper/323:35:513
107608999171,7cyclictest0-21swapper/321:00:503
107608999166,8cyclictest0-21swapper/319:40:533
107608999163,9cyclictest0-21swapper/323:56:003
107608999163,8cyclictest0-21swapper/321:40:433
107608999163,8cyclictest0-21swapper/300:15:533
107608999163,7cyclictest0-21swapper/323:15:513
107608999162,9cyclictest0-21swapper/322:40:513
107608999162,9cyclictest0-21swapper/321:20:523
107608999162,9cyclictest0-21swapper/321:20:523
107608999162,8cyclictest0-21swapper/322:05:533
107608999162,8cyclictest0-21swapper/322:05:533
107608999162,8cyclictest0-21swapper/321:25:593
107608999162,8cyclictest0-21swapper/321:25:593
107608999161,9cyclictest0-21swapper/300:30:523
107608999161,7cyclictest0-21swapper/320:35:513
107608999161,7cyclictest0-21swapper/319:30:493
107608999154,7cyclictest0-21swapper/319:25:503
107608999153,8cyclictest0-21swapper/323:20:533
107608999153,6cyclictest0-21swapper/320:40:543
107608999152,8cyclictest0-21swapper/323:50:423
107608999152,8cyclictest0-21swapper/322:55:483
107608999152,8cyclictest0-21swapper/300:25:493
107608999152,7cyclictest0-21swapper/319:20:543
107608999152,7cyclictest0-21swapper/319:15:523
107608999152,6cyclictest0-21swapper/320:55:503
107608999151,8cyclictest0-21swapper/319:55:493
107608999151,8cyclictest0-21swapper/319:55:493
107608999143,7cyclictest0-21swapper/323:05:433
107608999142,7cyclictest0-21swapper/320:50:503
107608999142,7cyclictest0-21swapper/320:20:013
107608999142,7cyclictest0-21swapper/319:50:513
107608999142,7cyclictest0-21swapper/319:50:513
107608999141,6cyclictest0-21swapper/319:46:053
107608999141,6cyclictest0-21swapper/319:46:053
107608999133,5cyclictest0-21swapper/320:45:523
107608999132,6cyclictest0-21swapper/321:05:503
107608999132,6cyclictest0-21swapper/320:30:583
107608999132,6cyclictest0-21swapper/320:25:483
107608999132,6cyclictest0-21swapper/320:10:483
107608999131,7cyclictest0-21swapper/319:35:503
107608999122,6cyclictest0-21swapper/320:00:503
107608999111,6cyclictest0-21swapper/320:05:413
107608999111,6cyclictest0-21swapper/319:10:533
10760899991,6cyclictest0-21swapper/319:10:013
10760899991,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional