You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 11:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Sat Jun 27, 2026 00:43:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1095128997247,21cyclictest0-21swapper/220:45:332
1095126996144,10cyclictest151ktimers/019:40:350
1095127995342,8cyclictest1202158-21/usr/sbin/munin23:05:011
1095127995041,5cyclictest0-21swapper/122:35:011
1095127994838,6cyclictest3545678-21cups-browsed21:55:291
1095126994818,24cyclictest1150835-21sshd21:40:320
1095128994736,6cyclictest0-21swapper/221:40:362
1095126994733,11cyclictest1131269-21latency_hist21:10:030
1095127994636,7cyclictest1174428-21sendmail22:20:011
1095127994436,5cyclictest1159040-21idleruntime-cro21:55:011
1095127994436,4cyclictest0-21swapper/123:45:301
1095126994437,4cyclictest151ktimers/020:50:310
1095126994436,5cyclictest3545678-21cups-browsed22:00:010
1095127994335,5cyclictest1140488-21kworker/u8:2+xprtiod21:45:011
1095127994331,5cyclictest1003-21mta-sts-daemon21:18:001
1095128994232,6cyclictest1107136-21systemd-run19:50:012
1095127994228,11cyclictest1148344-21unixbench_singl21:35:451
1095126994234,5cyclictest1168671-21aten_r2power_cu22:10:220
1095128994133,5cyclictest1116147-21systemd-run20:20:012
1095127994133,5cyclictest1129688-21systemd-run21:05:021
1095126994116,17cyclictest0-21swapper/020:05:360
1095128994032,5cyclictest3545675-21cupsd23:30:262
1095128994032,5cyclictest1098131-21idleruntime-cro19:20:012
1095128994031,5cyclictest1174414-21ssh22:20:012
1095128994030,6cyclictest1-21systemd00:00:022
1095127994034,3cyclictest0-21swapper/122:55:001
1095127994034,3cyclictest0-21swapper/122:55:001
1095127994033,3cyclictest0-21swapper/123:15:321
1095127994032,5cyclictest1260635-21ssh00:40:011
1095126994032,5cyclictest1214861-21aten_r2power_en23:25:230
1095126994032,5cyclictest1154112-21proc_pri21:45:390
1095126994029,4cyclictest1183660-21cron22:35:010
109512699402,28cyclictest1246342-21rm00:15:330
109512899393,30cyclictest1197010-21ssh22:55:302
1095128993932,4cyclictest1169352-21modprobe22:10:332
1095128993930,5cyclictest421-21dbus-daemon22:08:542
1095128993929,7cyclictest1177526-21cut22:25:012
1095127993932,4cyclictest0-21swapper/120:20:011
1095127993931,4cyclictest1122157-21latency_hist20:40:021
1095127993929,6cyclictest421-21dbus-daemon00:29:371
1095126993923,13cyclictest1223678-21munin-run23:40:010
1095126993916,8cyclictest1132251-21ntpq21:10:340
1095126993914,7cyclictest1206992-21sh23:10:580
1095126993914,18cyclictest0-21swapper/019:55:350
1095126993913,11cyclictest1258695-21ntpq00:35:350
1095126993912,21cyclictest1243279-21ssh00:10:340
1095127993830,5cyclictest1220969-21aten_r2power_cu23:35:201
1095127993829,6cyclictest421-21dbus-daemon23:30:371
1095126993825,8cyclictest1123065-21ntpq20:40:400
1095126993825,7cyclictest12750irq/33-eth022:40:360
1095126993825,7cyclictest12750irq/33-eth022:40:360
1095126993824,9cyclictest1156967-21ntpq21:50:290
109512699382,28cyclictest1141550-21ssh21:25:350
109512699382,28cyclictest1141550-21ssh21:25:350
1095126993817,12cyclictest0-21swapper/000:05:010
1095126993814,7cyclictest1191588-21sh22:45:580
1095126993814,7cyclictest1191588-21sh22:45:580
1095128993729,5cyclictest1239135-21kworker/u8:12-xprtiod00:40:012
1095128993729,5cyclictest1128281-21latency_hist21:00:032
1095128993729,3cyclictest0-21swapper/200:25:382
1095128993728,6cyclictest1200623-21/usr/sbin/munin23:00:442
1095127993730,4cyclictest1236158-21latency_hist00:00:031
1095127993729,4cyclictest1123615-21cron20:45:001
1095126993730,4cyclictest1144223-21fschecks_count21:30:250
1095126993728,4cyclictest0-21swapper/023:30:370
1095126993723,8cyclictest1124461-21ntpq20:45:290
1095126993712,7cyclictest1000-21mta-sts-daemon23:50:060
1095128993623,9cyclictest3545678-21cups-browsed21:45:452
1095127993631,2cyclictest0-21swapper/119:50:001
1095127993628,5cyclictest421-21dbus-daemon00:15:301
1095127993617,13cyclictest271ktimers/122:45:301
1095127993617,13cyclictest271ktimers/122:45:301
1095126993628,5cyclictest1099651-21systemd-run19:25:010
1095126993626,7cyclictest1177522-21ssh22:25:010
1095126993625,5cyclictest1129084-21ntpq21:00:400
1095126993624,8cyclictest1096009-21ntpq19:10:310
1095126993623,8cyclictest1172453-21ntpq22:15:320
1095126993620,11cyclictest1166270-21ntpq22:05:320
1095126993615,14cyclictest1200231-21sh23:00:370
1095126993613,6cyclictest1254147-21sh00:28:590
1095126993612,9cyclictest1140442-21sh21:24:570
1095126993612,9cyclictest1140442-21sh21:24:570
1095126993612,8cyclictest1180540-21sh22:29:520
1095128993528,4cyclictest1105693-21latency_hist19:45:022
1095128993520,11cyclictest1255274-21kernelversion00:30:272
1095127993526,6cyclictest421-21dbus-daemon21:27:571
1095127993526,6cyclictest421-21dbus-daemon21:27:571
1095127993526,5cyclictest0-21swapper/122:10:261
1095126993527,5cyclictest0-21swapper/023:00:010
1095126993523,7cyclictest1117082-21ntpq20:20:340
1095126993522,7cyclictest1118602-21ntpq20:25:360
1095126993522,10cyclictest1097336-21load19:15:270
1095126993521,10cyclictest1114023-21ntpq20:10:330
1095126993513,18cyclictest1233240-21/usr/sbin/munin23:55:200
109512699351,23cyclictest0-21swapper/022:35:320
1095128993427,4cyclictest1246927-21grep00:15:442
1095128993426,5cyclictest421-21dbus-daemon22:30:372
1095128993426,5cyclictest1189820-21cron22:45:002
1095128993426,5cyclictest1189820-21cron22:45:002
1095128993426,5cyclictest1162136-21systemctl22:00:012
109512799344,26cyclictest1163340-21ssh22:00:401
1095127993428,3cyclictest0-21swapper/123:25:401
1095127993427,3cyclictest0-21swapper/123:00:001
1095127993426,5cyclictest0-21swapper/121:00:011
1095127993426,4cyclictest0-21swapper/122:05:421
1095127993425,6cyclictest421-21dbus-daemon00:07:041
109512799342,28cyclictest1224805-21ssh23:40:331
109512699349,8cyclictest1148879-21sh21:37:180
1095126993426,5cyclictest1242230-21latency_hist00:10:020
1095126993425,6cyclictest0-21swapper/000:23:320
1095126993423,5cyclictest1100507-21ntpq19:25:330
1095126993422,3cyclictest0-21swapper/020:40:010
1095126993421,8cyclictest1203167-21ntpq23:05:310
1095126993421,8cyclictest1115524-21ntpq20:15:330
1095126993421,7cyclictest1209318-21ntpq23:15:310
1095126993420,8cyclictest1212442-21ntpq23:20:350
1095128993326,4cyclictest1003-21mta-sts-daemon23:41:272
1095128993326,4cyclictest1001-21mta-sts-daemon19:35:012
1095128993326,4cyclictest1001-21mta-sts-daemon19:35:012
1095128993326,3cyclictest0-21swapper/221:20:222
1095128993326,3cyclictest0-21swapper/221:20:222
1095128993325,5cyclictest1223725-21systemd-run23:40:022
1095128993325,5cyclictest1108597-21cron19:55:012
1095128993319,11cyclictest351ktimers/219:10:312
109512899331,26cyclictest1206259-21ssh23:10:312
1095127993328,2cyclictest0-21swapper/123:10:361
1095127993326,4cyclictest1000-21mta-sts-daemon21:12:371
1095127993324,5cyclictest0-21swapper/120:00:011
1095127993322,9cyclictest1146692-21sh21:35:011
1095127993321,9cyclictest1153796-21missed_timers21:45:291
1095126993322,7cyclictest1127504-21ntpq20:55:300
1095126993321,8cyclictest1106463-21ntpq19:45:330
1095126993321,8cyclictest1103585-21ntpq19:35:360
1095126993321,8cyclictest1103585-21ntpq19:35:360
1095126993320,9cyclictest1224451-21irqstats23:40:250
1095126993318,9cyclictest1111092-21ntpq20:00:380
1095128993224,5cyclictest1211373-21cron23:20:012
1095128993224,5cyclictest1113116-21munin-run20:10:012
1095128993224,5cyclictest1110107-21sendmail20:00:012
1095128993223,6cyclictest1131164-21sh21:10:012
1095128993223,5cyclictest1144323-21if_wlan021:30:262
1095128993221,8cyclictest1184725-21ntp_kernel_pll_22:35:322
1095128993220,9cyclictest1141438-21needreboot21:25:312
1095128993220,9cyclictest1141438-21needreboot21:25:312
1095128993215,13cyclictest351ktimers/219:35:302
1095128993215,13cyclictest351ktimers/219:35:302
1095127993226,3cyclictest0-21swapper/100:15:011
1095127993225,4cyclictest1130198-21fschecks_time21:05:241
1095127993225,3cyclictest0-21swapper/122:22:201
1095127993223,6cyclictest485-21in:imuxsock19:55:021
1095127993223,6cyclictest1-21systemd20:05:021
1095126993219,8cyclictest1107968-21ntpq19:50:310
1095126993215,13cyclictest1164022-21ssh22:01:200
1095126993213,15cyclictest1255472-21ntpq00:30:300
1095128993126,2cyclictest0-21swapper/222:25:312
1095128993124,4cyclictest1148294-21unixbench_multi21:35:452
1095128993123,5cyclictest1243845-21unixbench_singl00:10:432
109512899312,25cyclictest1163165-21sshd22:00:322
109512899311,21cyclictest1230883-21ssh23:50:322
1095127993125,3cyclictest0-21swapper/122:25:221
1095127993124,4cyclictest1237397-21switchtime00:00:421
1095127993124,4cyclictest1003-21mta-sts-daemon21:20:261
1095127993124,4cyclictest1003-21mta-sts-daemon21:20:261
1095127993120,8cyclictest1230456-21sh23:50:241
1095126993121,6cyclictest1227892-21ntpq23:45:340
1095126993121,6cyclictest1119973-21ntpq20:30:310
1095126993120,7cyclictest1102003-21ntpq19:30:320
1095126993120,7cyclictest1102003-21ntpq19:30:320
1095126993119,8cyclictest1193889-21ntpq22:50:290
1095126993119,8cyclictest1193889-21ntpq22:50:290
1095126993117,9cyclictest1135276-21ntpq21:15:340
1095128993024,3cyclictest0-21swapper/221:15:012
1095128993024,3cyclictest0-21swapper/220:45:002
1095128993024,3cyclictest0-21swapper/220:30:462
1095128993024,3cyclictest0-21swapper/220:25:012
1095128993023,4cyclictest1129661-21cron21:05:012
1095128993022,5cyclictest1191086-21sendmail_mailqu22:45:392
1095128993022,5cyclictest1191086-21sendmail_mailqu22:45:392
1095128993021,6cyclictest439-21systemd-logind19:25:012
1095128993021,5cyclictest0-21swapper/223:05:322
1095128993020,6cyclictest1236063-21cups-browsed00:02:512
1095128993020,6cyclictest1000-21mta-sts-daemon23:45:312
1095127993025,2cyclictest0-21swapper/120:50:321
1095127993023,3cyclictest0-21swapper/123:05:251
1095127993022,5cyclictest1125135-21systemd-run20:50:011
1095127993022,4cyclictest0-21swapper/122:40:341
1095127993022,4cyclictest0-21swapper/122:40:341
1095127993019,8cyclictest1117623-21munin-plugin-st20:25:021
1095126993020,6cyclictest1095138-21munin-run19:10:010
1095128992922,3cyclictest0-21swapper/220:30:012
1095128992920,6cyclictest420-21avahi-daemon00:07:262
1095127992922,4cyclictest1105614-21cron19:45:001
1095127992921,4cyclictest0-21swapper/122:35:251
1095127992920,6cyclictest0-21swapper/119:10:311
1095127992920,4cyclictest0-21swapper/100:32:551
1095128992822,3cyclictest0-21swapper/221:15:292
1095128992820,5cyclictest1236058-21cupsd00:20:392
1095128992819,7cyclictest1121668-21sendmail_mailqu20:35:382
1095128992817,7cyclictest1214785-21/usr/sbin/munin23:25:242
109512899281,22cyclictest1212530-21ssh23:20:382
109512799282,22cyclictest1212519-21ssh23:20:381
1095127992821,4cyclictest1119340-21/usr/sbin/munin20:30:341
1095127992820,5cyclictest1002-21mta-sts-daemon19:30:351
1095127992820,5cyclictest1002-21mta-sts-daemon19:30:351
1095127992820,2cyclictest0-21swapper/100:25:031
1095127992812,11cyclictest265-21systemd-journal19:15:321
1095128992719,5cyclictest3545678-21cups-browsed21:51:382
1095128992718,6cyclictest539-21wpa_supplicant20:00:242
1095128992718,5cyclictest3545678-21cups-browsed20:51:472
109512899271,21cyclictest1193978-21ssh22:50:332
109512899271,21cyclictest1193978-21ssh22:50:332
1095127992720,5cyclictest1093790-21kworker/u8:3+xprtiod19:10:011
1095127992620,3cyclictest0-21swapper/120:25:211
1095127992615,6cyclictest1101103-21kworker/u8:10+xprtiod20:10:031
1095127992520,2cyclictest0-21swapper/119:25:321
109512799243,13cyclictest1103515-21sshd19:35:321
109512799243,13cyclictest1103515-21sshd19:35:321
109512799242,17cyclictest265-21systemd-journal19:20:321
1095127992418,3cyclictest0-21swapper/120:15:011
1095128992317,3cyclictest0-21swapper/219:10:022
109512899211,15cyclictest1114023-21ntpq20:10:332
1095128992110,8cyclictest1100462-21/usr/sbin/munin19:25:312
109512999201,5cyclictest0-21swapper/322:05:363
109512999183,7cyclictest0-21swapper/300:25:353
109512999182,8cyclictest0-21swapper/321:55:313
109512999172,9cyclictest0-21swapper/323:25:243
109512999171,8cyclictest0-21swapper/321:35:343
109512999171,7cyclictest0-21swapper/320:51:033
109512999163,7cyclictest0-21swapper/321:40:323
109512999162,7cyclictest0-21swapper/323:00:313
109512999161,9cyclictest0-21swapper/300:05:443
109512999161,8cyclictest0-21swapper/321:20:343
109512999161,8cyclictest0-21swapper/321:20:343
109512999161,7cyclictest0-21swapper/319:50:343
109512999154,7cyclictest0-21swapper/323:40:323
109512999154,6cyclictest0-21swapper/322:00:393
109512999153,6cyclictest0-21swapper/323:30:323
109512999152,9cyclictest0-21swapper/322:20:403
109512999152,8cyclictest0-21swapper/323:20:363
109512999152,8cyclictest0-21swapper/300:00:303
109512999152,7cyclictest0-21swapper/323:50:383
109512999152,7cyclictest0-21swapper/300:35:353
109512999152,7cyclictest0-21swapper/300:15:323
109512999152,6cyclictest0-21swapper/321:05:293
109512999151,8cyclictest0-21swapper/322:25:353
109512999151,8cyclictest0-21swapper/321:10:323
109512999151,7cyclictest0-21swapper/322:50:323
109512999151,7cyclictest0-21swapper/322:50:323
109512999151,6cyclictest0-21swapper/320:25:323
109512999151,6cyclictest0-21swapper/319:25:333
109512999151,5cyclictest0-21swapper/320:05:363
109512999143,5cyclictest0-21swapper/322:55:293
109512999142,8cyclictest0-21swapper/323:40:013
109512999142,7cyclictest0-21swapper/322:35:283
109512999142,7cyclictest0-21swapper/321:50:323
109512999142,7cyclictest0-21swapper/321:45:353
109512999142,7cyclictest0-21swapper/319:35:323
109512999142,7cyclictest0-21swapper/319:35:323
109512999142,7cyclictest0-21swapper/300:15:013
109512999142,6cyclictest0-21swapper/319:30:313
109512999142,6cyclictest0-21swapper/319:30:313
109512999141,8cyclictest0-21swapper/323:45:313
109512999141,8cyclictest0-21swapper/323:10:313
109512999141,8cyclictest0-21swapper/321:30:383
109512999141,6cyclictest0-21swapper/321:25:233
109512999141,6cyclictest0-21swapper/321:25:233
109512999141,6cyclictest0-21swapper/320:20:303
109512999141,3cyclictest0-21swapper/319:55:353
109512999136,5cyclictest0-21swapper/323:55:323
109512999132,7cyclictest0-21swapper/322:40:303
109512999132,7cyclictest0-21swapper/322:40:303
109512999132,7cyclictest0-21swapper/322:10:223
109512999132,6cyclictest0-21swapper/320:10:313
109512999132,5cyclictest0-21swapper/320:15:323
109512999132,5cyclictest0-21swapper/319:40:363
109512999131,8cyclictest0-21swapper/322:30:253
109512999131,8cyclictest0-21swapper/300:20:323
109512999131,7cyclictest0-21swapper/322:15:353
109512999131,6cyclictest0-21swapper/320:35:313
109512999131,6cyclictest0-21swapper/319:10:303
109512999123,4cyclictest0-21swapper/320:30:403
109512999122,7cyclictest0-21swapper/323:05:213
109512999122,6cyclictest0-21swapper/322:50:023
109512999122,6cyclictest0-21swapper/322:50:023
109512999122,6cyclictest0-21swapper/319:45:323
109512999122,5cyclictest0-21swapper/319:20:313
109512999121,7cyclictest0-21swapper/320:00:323
109512999121,6cyclictest0-21swapper/321:15:333
109512999113,4cyclictest0-21swapper/320:45:333
109512999112,5cyclictest0-21swapper/300:35:033
109512999111,5cyclictest0-21swapper/319:15:313
109512999102,6cyclictest0-21swapper/323:15:263
109512999101,5cyclictest0-21swapper/321:00:243
10951299991,6cyclictest0-21swapper/320:55:243
10951299991,5cyclictest0-21swapper/320:40:253
10951299981,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional