You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-17 - 08:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Fri Jul 17, 2026 00:43:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1010124995947,7cyclictest1-21systemd22:30:010
1010126995746,7cyclictest1-21systemd00:10:022
1010124995041,6cyclictest3451713-21cups-browsed21:51:060
1010125994940,6cyclictest1003-21mta-sts-daemon23:40:481
1010126994840,5cyclictest1104098-21ssh22:45:002
1010126994837,8cyclictest3451713-21cups-browsed23:50:122
1010125994737,6cyclictest1150562-21kernelversion00:00:391
1010126994636,6cyclictest1084452-21unixbench-2d22:10:542
1010125994630,11cyclictest1123351-21needreboot23:15:411
1010124994637,6cyclictest1160470-21unixbench-2d00:15:580
1010125994434,6cyclictest265-21systemd-journal22:15:491
1010124994435,3cyclictest0-21swapper/021:25:470
1010124994420,17cyclictest151ktimers/021:00:460
1010124994420,17cyclictest151ktimers/021:00:460
1010126994332,8cyclictest1020633-21sh19:45:012
1010125994326,14cyclictest1079813-21kworker/u8:17+events_unbound22:05:471
1010124994334,7cyclictest1046166-21diskmemload00:30:020
1010124994334,7cyclictest1046166-21diskmemload00:30:020
1010126994226,11cyclictest3451713-21cups-browsed22:51:512
1010126994226,11cyclictest3451713-21cups-browsed22:51:512
101012599422,35cyclictest1071846-21sh21:50:461
1010124994234,6cyclictest1064565-21ls21:40:010
1010124994233,6cyclictest1147210-21aten_r2power_vo23:55:320
1010124994229,3cyclictest0-21swapper/023:00:000
1010126994133,5cyclictest1052277-21ssh21:20:012
1010126994129,7cyclictest1116340-21latency_hist23:05:022
1010125994134,4cyclictest1000-21mta-sts-daemon20:00:011
1010125994133,5cyclictest421-21dbus-daemon22:50:021
1010125994133,3cyclictest0-21swapper/122:35:491
1010125994131,5cyclictest1062492-21needreboot21:35:381
1010124994135,4cyclictest1125351-21ssh23:20:010
1010124994129,9cyclictest1079784-21cron22:05:000
1010124994127,11cyclictest1105834-21unixbench_singl22:45:570
101012499411,33cyclictest1047231-21ssh21:10:400
101012499411,33cyclictest1047231-21ssh21:10:400
101012699407,19cyclictest428-21mta-sts-daemon19:45:462
1010126994033,4cyclictest1031108-21munin-run20:20:012
1010126994031,5cyclictest1028129-21systemd-run20:10:012
101012699402,28cyclictest1150924-21ssh00:00:492
1010125994032,5cyclictest1159527-21ssh00:15:391
1010125994032,5cyclictest1158857-21systemd-run00:15:011
1010124994025,11cyclictest1083675-21kernelversion22:10:370
1010126993934,2cyclictest0-21swapper/223:10:492
1010126993931,5cyclictest1142184-21sed23:45:552
1010126993929,7cyclictest1055403-21munin-run21:25:012
1010126993928,8cyclictest1149722-21idleruntime-cro00:00:002
101012599396,26cyclictest1163062-21ssh00:20:471
101012599393,30cyclictest1053436-21ssh21:20:441
1010124993932,4cyclictest1061519-21systemd-run21:35:010
1010124993932,4cyclictest1019141-21systemd-run19:40:010
1010124993931,5cyclictest1120572-21proc_pri23:10:500
1010124993931,4cyclictest1000-21mta-sts-daemon00:07:130
1010126993833,2cyclictest0-21swapper/220:50:022
1010126993831,4cyclictest1160550-21unixbench_singl00:15:592
1010126993831,3cyclictest0-21swapper/222:10:012
1010126993830,5cyclictest1037160-21grep20:40:012
1010126993814,6cyclictest36-21ksoftirqd/222:20:482
101012699381,32cyclictest1065772-21ssh21:40:462
1010125993831,3cyclictest0-21swapper/122:55:011
1010125993831,3cyclictest0-21swapper/122:55:011
1010125993830,5cyclictest1052285-21idleruntime-cro21:20:001
1010125993829,6cyclictest3451713-21cups-browsed23:30:021
1010125993829,5cyclictest3451713-21cups-browsed21:55:521
1010125993829,5cyclictest3451713-21cups-browsed21:55:521
1010125993826,9cyclictest1114185-21missed_timers23:00:411
1010125993826,9cyclictest1068189-21aten_r2power_vo21:45:341
1010125993823,9cyclictest1111283-21sh22:55:461
101012499383,25cyclictest1132486-21ssh23:30:430
1010124993831,4cyclictest1068958-21wc21:45:520
1010124993829,6cyclictest1003-21mta-sts-daemon21:24:550
1010126993729,5cyclictest1088944-21kworker/u8:11+xprtiod22:30:012
1010126993725,9cyclictest1063058-21timerwakeupswit21:35:532
101012599373,27cyclictest1084072-21ssh22:10:491
1010125993731,3cyclictest0-21swapper/121:10:001
1010125993731,3cyclictest0-21swapper/121:10:001
1010125993730,4cyclictest1023543-21head19:50:531
1010125993727,6cyclictest420-21avahi-daemon22:20:151
1010125993727,6cyclictest1001-21mta-sts-daemon23:46:531
101012599371,29cyclictest1102287-21ssh22:40:491
1010124993732,3cyclictest0-21swapper/023:05:010
1010124993730,4cyclictest1038663-21cron20:45:010
1010124993730,4cyclictest1003-21mta-sts-daemon00:10:510
1010124993729,5cyclictest1-21systemd21:00:010
1010124993729,5cyclictest1-21systemd21:00:010
1010124993725,9cyclictest1049642-21aten_r2power_cu21:15:310
1010124993724,10cyclictest1040168-21munin-plugin-st20:50:010
1010126993628,5cyclictest1061489-21cron21:35:012
1010126993628,4cyclictest0-21swapper/200:30:402
1010126993628,4cyclictest0-21swapper/200:30:402
1010126993626,6cyclictest1126365-21needreboot23:20:372
1010126993625,8cyclictest0-21swapper/222:37:492
1010126993613,6cyclictest36-21ksoftirqd/200:20:452
101012699361,28cyclictest3451713-21cups-browsed22:30:442
1010125993630,3cyclictest0-21swapper/120:20:011
1010124993629,4cyclictest1140575-21cron23:45:010
1010124993626,7cyclictest1171897-21latency00:35:380
1010124993618,12cyclictest151ktimers/022:15:430
1010126993525,6cyclictest3451708-21cupsd21:14:462
1010126993525,6cyclictest3451708-21cupsd21:14:462
101012699352,26cyclictest1165958-21ssh00:25:402
101012699352,26cyclictest1165958-21ssh00:25:402
101012699352,24cyclictest665-21snmpd20:10:432
1010125993530,2cyclictest0-21swapper/119:40:011
1010125993528,4cyclictest1014617-21cron19:25:001
1010125993527,5cyclictest1168645-21fschecks_time00:30:341
1010125993527,5cyclictest1168645-21fschecks_time00:30:341
101012599351,25cyclictest1059695-21ssh21:31:011
1010124993528,4cyclictest1000-21mta-sts-daemon21:44:180
1010124993526,6cyclictest485-21in:imuxsock20:04:490
1010124993526,6cyclictest0-21swapper/023:50:410
1010126993426,5cyclictest1117567-21sendmail_mailqu23:05:492
1010126993423,8cyclictest1025102-21/usr/sbin/munin20:00:022
1010126993423,7cyclictest1149763-21cups-browsed00:35:552
101012599343,24cyclictest1077937-21ssh22:00:471
1010125993426,5cyclictest421-21dbus-daemon20:05:011
1010125993426,5cyclictest1056847-21threads21:25:531
1010125993426,5cyclictest0-21swapper/120:40:461
1010125993422,9cyclictest1117107-21kernelversion23:05:371
1010124993428,4cyclictest1171079-21latency_hist00:35:020
1010124993428,4cyclictest1171079-21latency_hist00:35:020
1010124993427,5cyclictest1128864-21sh23:25:330
1010124993426,6cyclictest3451713-21cups-browsed19:50:530
1010124993424,7cyclictest0-21swapper/021:55:440
1010124993424,7cyclictest0-21swapper/021:55:440
1010124993423,9cyclictest1013645-21fschecks_count19:20:330
1010126993327,3cyclictest0-21swapper/222:45:462
1010126993326,4cyclictest1013133-21sendmail19:20:012
1010126993326,4cyclictest1013133-21sendmail19:20:012
1010126993325,5cyclictest1041651-21cron20:55:002
1010126993325,4cyclictest0-21swapper/200:10:432
1010126993324,6cyclictest421-21dbus-daemon23:40:022
1010126993324,6cyclictest3451713-21cups-browsed19:25:352
1010126993311,5cyclictest36-21ksoftirqd/222:15:482
1010125993327,3cyclictest0-21swapper/119:20:001
1010125993327,3cyclictest0-21swapper/119:20:001
1010125993325,5cyclictest1149693-21idleruntime-cro00:00:001
1010124993326,4cyclictest1002-21mta-sts-daemon22:38:550
1010124993326,4cyclictest0-21swapper/023:50:020
1010124993325,5cyclictest420-21avahi-daemon22:34:240
1010124993316,13cyclictest0-21swapper/019:45:500
1010126993225,3cyclictest0-21swapper/221:55:312
1010126993225,3cyclictest0-21swapper/221:55:312
1010126993224,5cyclictest694-21cron21:10:002
1010126993224,5cyclictest694-21cron21:10:002
1010126993212,4cyclictest36-21ksoftirqd/223:15:342
101012599323,22cyclictest1126459-21ssh23:20:411
1010125993225,4cyclictest1098100-21latency_hist22:35:021
1010125993224,5cyclictest1043138-21cron21:00:001
1010125993224,5cyclictest1043138-21cron21:00:001
101012499329,16cyclictest151ktimers/020:05:450
1010124993225,5cyclictest1-21systemd20:15:010
1010124993225,4cyclictest1128580-21latency_hist23:25:020
101012499321,24cyclictest1080865-21ssh22:05:420
101012699316,5cyclictest1022470-21aten2.4-expect19:50:312
1010126993122,6cyclictest1043437-21aten_r2power_cu21:00:302
1010126993122,6cyclictest1043437-21aten_r2power_cu21:00:302
101012699312,22cyclictest1129570-21sshd23:25:502
1010126993110,4cyclictest36-21ksoftirqd/220:25:482
1010126993110,3cyclictest36-21ksoftirqd/220:40:402
101012599314,18cyclictest1132626-21ssh23:30:481
1010125993124,4cyclictest1092499-21aten2.4-expect22:25:351
1010125993123,5cyclictest1032615-21cron20:25:001
1010125993123,5cyclictest1032615-21cron20:25:001
1010125993122,6cyclictest1044600-21unixbench_singl21:00:531
1010125993122,6cyclictest1044600-21unixbench_singl21:00:531
1010125993122,3cyclictest0-21swapper/121:10:521
1010125993122,3cyclictest0-21swapper/121:10:521
1010124993122,6cyclictest420-21avahi-daemon22:41:000
1010124993117,3cyclictest0-21swapper/022:20:560
101012699309,4cyclictest36-21ksoftirqd/222:55:512
101012699303,21cyclictest1068833-21ssh21:45:492
1010126993023,3cyclictest0-21swapper/223:40:432
1010126993022,4cyclictest0-21swapper/219:35:402
1010126993018,7cyclictest1132129-21/usr/sbin/munin23:30:362
101012699301,22cyclictest1077813-21ssh22:00:422
1010126993011,5cyclictest36-21ksoftirqd/220:30:412
1010126993010,5cyclictest36-21ksoftirqd/220:00:362
1010125993023,4cyclictest1001-21mta-sts-daemon23:35:421
1010125993022,5cyclictest1120847-21timerwakeupswit23:10:551
1010125993022,5cyclictest1015079-21cpuspeed19:25:311
1010124993023,4cyclictest1024980-21unixbench_multi19:55:520
1010124993023,4cyclictest1010642-21fschecks_time19:10:360
1010124993023,4cyclictest1010642-21fschecks_time19:10:360
1010124993023,4cyclictest1003-21mta-sts-daemon00:24:370
1010124993021,7cyclictest1110312-21cut22:55:020
1010124993021,7cyclictest1110312-21cut22:55:020
101012699292,19cyclictest1071722-21ssh21:50:412
1010126992912,3cyclictest0-21swapper/221:28:052
1010125992923,3cyclictest0-21swapper/100:05:501
1010125992922,4cyclictest1003-21mta-sts-daemon21:44:421
101012599292,22cyclictest1144740-21ssh23:50:441
1010125992921,5cyclictest3451713-21cups-browsed20:50:461
1010125992921,5cyclictest1022054-21unixbench_singl19:45:551
1010125992919,7cyclictest1165295-21ssh00:25:311
1010125992919,7cyclictest1165295-21ssh00:25:311
1010125992918,8cyclictest1171992-21needreboot00:35:391
1010124992924,3cyclictest1002-21mta-sts-daemon00:00:160
101012499291,21cyclictest1119413-21latency_hist23:10:020
1010126992821,4cyclictest1001-21mta-sts-daemon19:30:392
1010125992822,3cyclictest0-21swapper/120:40:021
1010125992820,5cyclictest3451713-21cups-browsed20:13:511
1010125992818,5cyclictest0-21swapper/119:30:351
101012699279,4cyclictest36-21ksoftirqd/220:55:372
101012699279,4cyclictest36-21ksoftirqd/220:55:372
101012699279,4cyclictest36-21ksoftirqd/219:10:352
101012699279,4cyclictest36-21ksoftirqd/219:10:352
1010126992713,8cyclictest1031995-21ntpq20:20:412
1010126992713,8cyclictest1031995-21ntpq20:20:412
1010126992710,4cyclictest36-21ksoftirqd/219:20:402
1010125992721,4cyclictest1039126-21cpuspeed20:45:331
1010125992719,5cyclictest3451713-21cups-browsed19:41:211
1010124992723,2cyclictest0-21swapper/023:40:010
1010124992719,5cyclictest1-21systemd19:20:010
1010124992719,5cyclictest1-21systemd19:20:010
1010125992619,4cyclictest1035441-21timerwakeupswit20:30:571
1010124992619,4cyclictest421-21dbus-daemon20:50:460
1010124992618,6cyclictest12650irq/32-eth020:15:490
1010124992614,8cyclictest151ktimers/019:40:410
101012599252,19cyclictest1027568-21ntpq20:05:491
1010125992519,3cyclictest0-21swapper/119:10:011
1010125992517,5cyclictest1033644-21proc_pri20:25:481
101012499253,7cyclictest171rcu_preempt21:05:460
101012499253,7cyclictest171rcu_preempt21:05:460
1010124992519,4cyclictest1036877-21timerandwakeup20:35:530
1010124992519,2cyclictest0-21swapper/019:25:410
1010124992517,6cyclictest421-21dbus-daemon20:30:330
1010124992515,8cyclictest265-21systemd-journal20:23:490
1010124992515,8cyclictest265-21systemd-journal20:23:490
1010125992417,4cyclictest1003-21mta-sts-daemon19:10:321
1010125992417,4cyclictest1003-21mta-sts-daemon19:10:321
1010124992418,2cyclictest0-21swapper/020:25:530
1010124992414,6cyclictest1010123-21cyclictest19:10:010
1010126992216,3cyclictest0-21swapper/219:10:012
1010124992114,3cyclictest151ktimers/019:30:400
101012799203,9cyclictest0-21swapper/321:45:433
101012799183,9cyclictest0-21swapper/323:30:493
101012799181,8cyclictest0-21swapper/322:50:443
101012799181,8cyclictest0-21swapper/322:50:443
101012799174,9cyclictest0-21swapper/322:20:453
101012799173,9cyclictest0-21swapper/300:35:443
101012799173,8cyclictest0-21swapper/319:10:443
101012799173,8cyclictest0-21swapper/319:10:443
101012799172,8cyclictest0-21swapper/300:10:583
101012799172,8cyclictest0-21swapper/300:00:433
101012799165,8cyclictest0-21swapper/323:15:433
101012799164,9cyclictest0-21swapper/322:00:313
101012799163,7cyclictest0-21swapper/321:55:403
101012799163,7cyclictest0-21swapper/321:55:403
101012799163,7cyclictest0-21swapper/321:35:453
101012799162,8cyclictest0-21swapper/322:35:433
101012799162,8cyclictest0-21swapper/321:20:433
101012799162,6cyclictest0-21swapper/320:15:413
101012799161,9cyclictest0-21swapper/323:45:413
101012799161,8cyclictest0-21swapper/322:25:483
101012799160,6cyclictest0-21swapper/320:40:403
101012799153,7cyclictest0-21swapper/323:55:413
101012799153,7cyclictest0-21swapper/321:25:493
101012799153,7cyclictest0-21swapper/321:10:393
101012799153,7cyclictest0-21swapper/321:10:393
101012799153,6cyclictest0-21swapper/322:30:443
101012799151,8cyclictest0-21swapper/323:50:453
101012799151,8cyclictest0-21swapper/323:35:363
101012799151,8cyclictest0-21swapper/321:40:423
101012799151,8cyclictest0-21swapper/320:28:363
101012799151,7cyclictest0-21swapper/320:20:433
101012799151,7cyclictest0-21swapper/320:20:433
101012799151,7cyclictest0-21swapper/319:25:403
101012799151,6cyclictest0-21swapper/319:55:403
101012799146,6cyclictest0-21swapper/323:24:503
101012799144,7cyclictest0-21swapper/322:10:413
101012799143,7cyclictest0-21swapper/323:00:413
101012799143,6cyclictest0-21swapper/323:05:413
101012799143,6cyclictest0-21swapper/321:15:423
101012799143,6cyclictest0-21swapper/319:35:403
101012799143,6cyclictest0-21swapper/300:25:363
101012799143,6cyclictest0-21swapper/300:25:363
101012799143,5cyclictest0-21swapper/319:30:403
101012799142,7cyclictest0-21swapper/322:55:353
101012799142,7cyclictest0-21swapper/320:45:483
101012799142,7cyclictest0-21swapper/319:45:023
101012799142,7cyclictest0-21swapper/300:15:403
101012799142,6cyclictest0-21swapper/322:05:373
101012799141,8cyclictest0-21swapper/323:10:373
101012799141,8cyclictest0-21swapper/322:40:363
101012799141,8cyclictest0-21swapper/321:30:593
101012799141,7cyclictest0-21swapper/320:30:483
101012799141,7cyclictest0-21swapper/320:05:503
101012799141,6cyclictest0-21swapper/321:05:403
101012799141,6cyclictest0-21swapper/321:05:403
101012799141,6cyclictest0-21swapper/320:00:463
101012799134,6cyclictest0-21swapper/319:20:403
101012799132,7cyclictest0-21swapper/323:40:443
101012799132,7cyclictest0-21swapper/322:15:433
101012799132,7cyclictest0-21swapper/320:55:433
101012799132,7cyclictest0-21swapper/320:55:433
101012799132,6cyclictest0-21swapper/321:05:003
101012799132,6cyclictest0-21swapper/321:05:003
101012799132,6cyclictest0-21swapper/300:30:343
101012799132,6cyclictest0-21swapper/300:30:343
101012799132,6cyclictest0-21swapper/300:20:333
101012799131,7cyclictest0-21swapper/323:25:463
101012799131,7cyclictest0-21swapper/319:45:463
101012799131,6cyclictest0-21swapper/320:50:353
101012799121,7cyclictest0-21swapper/321:50:423
101012799121,7cyclictest0-21swapper/319:15:503
101012799121,7cyclictest0-21swapper/319:15:503
101012799121,7cyclictest0-21swapper/300:05:423
101012799121,6cyclictest0-21swapper/322:45:033
101012799121,6cyclictest0-21swapper/319:50:403
101012799111,6cyclictest0-21swapper/320:10:413
101012799101,6cyclictest0-21swapper/320:35:353
10101279981,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional