You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-18 - 11:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sat Apr 18, 2026 00:43:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
955913995440,9cyclictest1147567-21ssh00:25:371
955913995440,9cyclictest1147567-21ssh00:25:371
955912995123,22cyclictest1151383-21ssh00:30:430
955913995041,6cyclictest421-21dbus-daemon23:40:311
955913995041,6cyclictest421-21dbus-daemon23:40:311
955912995016,26cyclictest1041649-21ssh22:10:380
955914994941,5cyclictest1097007-21munin-plugin-st23:20:002
95591399481,41cyclictest1020188-21ssh21:45:001
955912994840,5cyclictest955911-21cyclictest22:35:430
95591399474,36cyclictest1001294-21sh21:20:451
955912994737,7cyclictest151ktimers/020:35:380
955912994737,6cyclictest1003984-21latency_hist21:25:010
955914994637,6cyclictest1053147-21aten_r2power_cu22:25:272
955914994636,6cyclictest3345305-21cups-browsed21:54:492
955913994536,6cyclictest1105040-21ssh23:30:011
95591299453,32cyclictest1086408-21grep23:05:450
95591299453,32cyclictest1086408-21grep23:05:450
955912994517,21cyclictest1009157-21ssh21:30:380
955914994436,5cyclictest955917-21systemd-run19:10:012
955914994434,6cyclictest3345305-21cups-browsed22:10:462
955913994435,6cyclictest1054941-21sshd22:26:371
955913994337,3cyclictest0-21swapper/122:50:331
955913994336,3cyclictest0-21swapper/122:10:011
955913994336,3cyclictest0-21swapper/122:10:011
955913994334,5cyclictest1093027-21munin-plugin-st23:15:001
955913994334,5cyclictest1052691-21systemd-run22:25:011
955912994315,23cyclictest1005150-21ssh21:25:390
95591299425,27cyclictest1106210-21ssh23:30:390
955912994233,5cyclictest1056-21mta-sts-daemon21:11:200
955914994132,6cyclictest1061530-21fschecks_count22:35:302
955914994130,8cyclictest1017652-21switchtime21:40:472
955912994131,7cyclictest421-21dbus-daemon22:20:400
955912994127,11cyclictest3345305-21cups-browsed21:55:440
955912994127,11cyclictest1117453-21aten_r2power_po23:45:280
95591299412,32cyclictest1098290-21ssh23:20:400
955912994123,14cyclictest420-21avahi-daemon22:00:520
955912994123,14cyclictest420-21avahi-daemon22:00:520
955914994032,5cyclictest1139441-21systemd-run00:15:012
955914994031,6cyclictest1-21systemd23:05:012
955914994031,6cyclictest1157492-21munin-plugin-st00:40:002
95591499402,32cyclictest1037699-21ssh22:05:432
95591499402,32cyclictest1037699-21ssh22:05:432
955913994032,5cyclictest988879-21cron21:00:011
955913994032,5cyclictest1113072-21latency_hist23:40:031
955913994028,7cyclictest0-21swapper/123:55:371
955912994028,10cyclictest1057608-21sh22:30:330
955914993931,5cyclictest1147215-21uniq00:25:322
955914993931,5cyclictest1147215-21uniq00:25:322
955913993931,5cyclictest1136800-21kernelversion00:10:331
95591399392,33cyclictest1041727-21ssh22:10:401
955912993932,4cyclictest984430-21kworker/u8:14-xprtiod21:05:370
955912993931,5cyclictest1157522-21sendmail-msp00:40:010
955912993929,7cyclictest1128531-21cups-browsed00:25:430
955912993929,7cyclictest1128531-21cups-browsed00:25:430
955912993929,6cyclictest421-21dbus-daemon22:53:300
955912993928,8cyclictest1081943-21latency23:00:330
955912993927,9cyclictest1-21systemd21:15:310
955912993927,9cyclictest1-21systemd21:15:310
955912993927,8cyclictest1054356-21timerwakeupswit22:25:490
955914993832,3cyclictest0-21swapper/223:25:352
955914993830,5cyclictest1089021-21idleruntime-cro23:10:012
955914993830,5cyclictest1089021-21idleruntime-cro23:10:012
955914993825,10cyclictest1113729-21/usr/sbin/munin23:40:322
955914993825,10cyclictest1113729-21/usr/sbin/munin23:40:322
955914993824,10cyclictest3345305-21cups-browsed22:45:372
95591399388,5cyclictest171rcu_preempt22:00:411
95591399388,5cyclictest171rcu_preempt22:00:411
955913993833,2cyclictest0-21swapper/100:20:011
955913993832,3cyclictest0-21swapper/121:40:011
955913993830,5cyclictest976904-21systemctl20:20:011
95591399382,31cyclictest1078157-21ssh22:55:391
95591399382,31cyclictest1078157-21ssh22:55:391
955912993830,5cyclictest974045-21latency_hist20:10:020
95591299382,27cyclictest1013156-21sh21:35:370
955914993729,5cyclictest1029128-21fschecks_time21:55:302
955913993729,6cyclictest1048589-21sendmail22:20:011
955913993729,5cyclictest1153742-21cron00:35:001
955913993728,4cyclictest0-21swapper/122:35:431
95591299373,27cyclictest991896-21diskmemload22:05:510
95591299373,27cyclictest991896-21diskmemload22:05:510
955912993728,4cyclictest0-21swapper/000:20:420
955912993727,6cyclictest620-21sshd00:04:370
95591299372,27cyclictest1133507-21ssh00:05:430
955912993713,14cyclictest0-21swapper/023:10:560
95591499363,28cyclictest1054-21mta-sts-daemon22:00:412
95591499363,28cyclictest1054-21mta-sts-daemon22:00:412
955914993630,3cyclictest981375-21kworker/u8:5-xprtiod21:40:012
955914993628,6cyclictest976897-21/usr/sbin/munin20:20:022
955914993628,5cyclictest1121385-21aten_r2power_vo23:50:272
955914993628,5cyclictest1060812-21latency_hist22:35:022
955914993627,7cyclictest964125-21missed_timers19:35:372
955914993627,6cyclictest1-21systemd00:00:002
955914993627,6cyclictest1093014-21cron23:15:002
955913993628,5cyclictest963416-21systemd-run19:35:011
955913993627,6cyclictest3345305-21cups-browsed22:49:171
955913993626,6cyclictest993078-21/usr/sbin/munin21:10:371
955912993625,6cyclictest12350irq/32-eth023:50:360
95591299362,12cyclictest1070066-21ssh22:45:380
955914993528,4cyclictest970878-21cron20:00:002
955914993528,4cyclictest970878-21cron20:00:002
955914993527,5cyclictest963384-21cron19:35:012
955914993527,5cyclictest1068948-21latency_hist22:45:022
955914993516,7cyclictest990385-21kworker/u8:13-xprtiod21:30:432
955913993528,4cyclictest1122556-21unixbench_multi23:50:481
955913993527,5cyclictest991896-21diskmemload21:10:011
955913993527,5cyclictest975318-21head20:10:501
955913993527,5cyclictest1132355-21ssh00:05:011
955912993530,3cyclictest1080987-21sendmail23:00:010
955912993530,3cyclictest1080987-21sendmail23:00:010
955912993528,5cyclictest1095230-21sh23:16:310
955912993528,4cyclictest151ktimers/020:30:430
955912993527,4cyclictest0-21swapper/023:35:310
955912993523,9cyclictest1116858-21ssh23:45:000
955912993523,9cyclictest1116858-21ssh23:45:000
95591299352,26cyclictest1021443-21ssh21:45:390
955912993514,15cyclictest421-21dbus-daemon19:50:400
955914993424,7cyclictest1005801-21unixbench_singl21:25:502
955914993423,8cyclictest983750-21ntp_kernel_err20:40:362
955913993428,3cyclictest0-21swapper/121:55:011
955913993427,4cyclictest1133825-21switchtime00:05:531
955913993426,5cyclictest1051-21mta-sts-daemon21:50:021
955912993428,4cyclictest1139955-21aten_r2power_po00:15:300
955912993426,5cyclictest1016868-21fschecks_count21:40:300
95591299342,26cyclictest1125789-21ssh23:55:360
95591299342,26cyclictest1045919-21ssh22:15:450
955912993412,5cyclictest14-21ksoftirqd/021:54:560
955914993327,3cyclictest0-21swapper/200:35:012
955914993326,4cyclictest992873-21latency21:10:332
955914993326,4cyclictest1052692-21sh22:25:012
955914993325,5cyclictest3345305-21cups-browsed23:49:412
955914993324,4cyclictest0-21swapper/223:35:432
955913993328,2cyclictest0-21swapper/121:35:011
955913993325,4cyclictest0-21swapper/121:15:431
955913993325,4cyclictest0-21swapper/121:15:431
95591299335,12cyclictest969887-21diskstats19:55:300
95591299335,12cyclictest969887-21diskstats19:55:300
955912993323,5cyclictest0-21swapper/022:40:420
95591299332,26cyclictest979303-21modprobe20:25:370
955912993313,16cyclictest1137022-21ssh00:10:360
955912993312,6cyclictest14-21ksoftirqd/020:20:430
955912993312,5cyclictest14-21ksoftirqd/023:25:360
955914993225,4cyclictest1101039-21cut23:25:012
955914993224,5cyclictest1133157-21/usr/sbin/munin00:05:362
955914993224,5cyclictest1000841-21if_wlan021:20:332
955914993224,4cyclictest969363-21kworker/u8:1-xprtiod21:20:022
955914993224,4cyclictest969363-21kworker/u8:1-xprtiod21:20:022
955914993223,5cyclictest1021410-21ssh21:45:382
955913993226,3cyclictest0-21swapper/123:35:011
955913993224,5cyclictest261-21systemd-journal20:25:011
95591399322,26cyclictest1005335-21grep21:25:451
955912993220,9cyclictest0-21swapper/020:40:440
955912993211,5cyclictest14-21ksoftirqd/019:10:390
955914993123,4cyclictest0-21swapper/222:50:432
955914993121,7cyclictest1081172-21latency_hist23:00:032
955914993121,7cyclictest1081172-21latency_hist23:00:032
95591499311,22cyclictest1045718-21ssh22:15:372
95591399315,3cyclictest171rcu_preempt22:30:361
955913993123,5cyclictest1029318-21latency21:55:321
955912993119,9cyclictest0-21swapper/020:55:010
955914993023,4cyclictest961922-21systemd-run19:30:012
955914993023,4cyclictest1056-21mta-sts-daemon20:35:022
955914993020,8cyclictest974734-21ntp_kernel_err20:10:372
955913993025,2cyclictest0-21swapper/123:20:011
955913993022,5cyclictest1065452-21aten2.4-expect22:40:281
955913993022,4cyclictest0-21swapper/119:55:371
955913993022,4cyclictest0-21swapper/119:55:371
955913993022,3cyclictest0-21swapper/123:10:011
955913993022,3cyclictest0-21swapper/123:10:011
95591299301,22cyclictest967285-21sshd19:45:370
955912993012,12cyclictest965850-21ntpq19:40:410
955912993010,5cyclictest14-21ksoftirqd/020:45:440
955914992923,3cyclictest0-21swapper/200:25:012
95591399295,16cyclictest428-21mta-sts-daemon20:35:431
955913992924,2cyclictest0-21swapper/119:40:011
955913992923,3cyclictest0-21swapper/123:45:281
95591399292,22cyclictest1098278-21ssh23:20:401
955913992912,12cyclictest421-21dbus-daemon20:45:011
95591299299,3cyclictest14-21ksoftirqd/019:35:440
95591299298,4cyclictest14-21ksoftirqd/021:00:420
955912992910,5cyclictest14-21ksoftirqd/020:00:380
955912992910,5cyclictest14-21ksoftirqd/019:15:410
955914992822,3cyclictest0-21swapper/200:00:312
955914992820,5cyclictest990330-21unixbench_singl21:00:492
955914992820,5cyclictest959426-21fschecks_count19:20:302
955914992820,5cyclictest1105772-21ssh23:30:312
955914992820,5cyclictest0-21swapper/219:45:012
95591399285,12cyclictest1055-21mta-sts-daemon19:10:361
955913992821,3cyclictest0-21swapper/100:35:371
955913992818,6cyclictest973816-21unixbench_singl20:05:521
95591299288,11cyclictest14-21ksoftirqd/019:25:380
955912992823,2cyclictest0-21swapper/019:20:440
955912992820,6cyclictest989051-21latency_hist21:00:020
955914992721,3cyclictest0-21swapper/219:20:012
955914992719,5cyclictest421-21dbus-daemon19:55:012
955914992719,4cyclictest0-21swapper/220:25:422
95591399273,14cyclictest1082203-21ssh23:00:391
955913992721,3cyclictest0-21swapper/100:23:091
955913992717,7cyclictest980166-21sh20:30:281
95591299277,4cyclictest14-21ksoftirqd/020:10:430
95591499264,16cyclictest428-21mta-sts-daemon20:05:382
955914992621,2cyclictest0-21swapper/220:45:522
955914992620,3cyclictest0-21swapper/200:15:522
955914992616,7cyclictest982609-21timerandwakeup20:35:482
955913992621,2cyclictest0-21swapper/119:10:011
955913992619,4cyclictest987319-21unixbench_singl20:50:471
95591299269,4cyclictest14-21ksoftirqd/020:15:350
955914992517,5cyclictest3345305-21cups-browsed19:12:152
95591399255,10cyclictest99011550cat21:00:481
955913992519,3cyclictest0-21swapper/120:45:321
955913992517,5cyclictest967826-21unixbench_singl19:45:491
955913992517,5cyclictest959930-21ps19:20:431
95591299257,3cyclictest14-21ksoftirqd/019:30:400
95591399248,13cyclictest961960-21sed19:30:021
955913992418,3cyclictest966433-21kworker/u8:21+events_unbound19:50:401
955913992416,5cyclictest958811-21unixbench_singl19:15:491
95591499235,13cyclictest428-21mta-sts-daemon20:20:382
95591499235,13cyclictest421-21dbus-daemon21:05:402
955914992316,4cyclictest1056-21mta-sts-daemon21:00:012
955914992315,5cyclictest3345305-21cups-browsed19:46:052
95591399232,17cyclictest624-21lldpd19:40:431
955914992216,3cyclictest0-21swapper/220:50:392
95591499221,16cyclictest971884-21ntpq20:00:422
955913992217,2cyclictest0-21swapper/120:25:471
95591399212,6cyclictest1056-21mta-sts-daemon20:05:011
95591299215,4cyclictest14-21ksoftirqd/019:10:010
95591599172,8cyclictest0-21swapper/322:30:383
95591599171,8cyclictest0-21swapper/321:45:003
95591599171,8cyclictest0-21swapper/319:30:373
95591599171,6cyclictest0-21swapper/320:05:423
95591599152,5cyclictest0-21swapper/321:05:383
95591599151,8cyclictest0-21swapper/322:40:383
95591599143,6cyclictest0-21swapper/322:25:373
95591599142,7cyclictest0-21swapper/321:30:373
95591599142,6cyclictest0-21swapper/322:50:373
95591599142,5cyclictest0-21swapper/321:00:443
95591599141,7cyclictest0-21swapper/322:10:383
95591599141,7cyclictest0-21swapper/321:35:393
95591599141,7cyclictest0-21swapper/321:10:433
95591599141,7cyclictest0-21swapper/300:35:453
95591599141,6cyclictest0-21swapper/323:45:363
95591599141,6cyclictest0-21swapper/322:35:433
95591599141,6cyclictest0-21swapper/320:15:363
95591599132,7cyclictest0-21swapper/322:20:403
95591599132,6cyclictest0-21swapper/300:30:403
95591599131,7cyclictest0-21swapper/300:10:363
95591599131,6cyclictest0-21swapper/323:35:383
95591599131,6cyclictest0-21swapper/321:55:323
95591599131,6cyclictest0-21swapper/320:50:363
95591599131,5cyclictest0-21swapper/319:25:373
95591599122,6cyclictest0-21swapper/300:15:323
95591599122,5cyclictest0-21swapper/300:00:423
95591599122,4cyclictest0-21swapper/319:15:373
95591599121,7cyclictest0-21swapper/323:30:313
95591599121,6cyclictest0-21swapper/323:20:403
95591599121,6cyclictest0-21swapper/323:00:363
95591599121,6cyclictest0-21swapper/322:15:393
95591599121,6cyclictest0-21swapper/320:55:363
95591599121,5cyclictest0-21swapper/320:25:383
95591599121,5cyclictest0-21swapper/320:20:393
95591599121,5cyclictest0-21swapper/320:00:353
95591599121,5cyclictest0-21swapper/319:45:383
95591599121,5cyclictest0-21swapper/319:35:423
95591599120,5cyclictest0-21swapper/320:30:413
95591599112,6cyclictest0-21swapper/322:00:373
95591599112,6cyclictest0-21swapper/322:00:373
95591599112,5cyclictest0-21swapper/321:50:403
95591599111,7cyclictest0-21swapper/323:15:273
95591599111,6cyclictest0-21swapper/323:50:273
95591599111,6cyclictest0-21swapper/322:45:313
95591599111,6cyclictest0-21swapper/322:05:363
95591599111,6cyclictest0-21swapper/322:05:363
95591599111,6cyclictest0-21swapper/300:25:013
95591599111,6cyclictest0-21swapper/300:05:373
95591599111,5cyclictest0-21swapper/323:55:313
95591599111,5cyclictest0-21swapper/320:10:323
95591599111,5cyclictest0-21swapper/319:55:373
95591599111,5cyclictest0-21swapper/319:55:373
95591599111,3cyclictest0-21swapper/319:20:423
95591599102,5cyclictest0-21swapper/323:40:313
95591599102,5cyclictest0-21swapper/323:40:313
95591599102,5cyclictest0-21swapper/322:55:333
95591599102,5cyclictest0-21swapper/322:55:333
95591599101,6cyclictest0-21swapper/323:10:403
95591599101,6cyclictest0-21swapper/321:25:023
95591599101,6cyclictest0-21swapper/321:15:273
95591599101,6cyclictest0-21swapper/321:15:273
95591599101,5cyclictest0-21swapper/323:25:313
95591599101,5cyclictest0-21swapper/321:45:273
95591599101,5cyclictest0-21swapper/320:35:393
95591599101,5cyclictest0-21swapper/319:50:373
95591599101,5cyclictest0-21swapper/300:25:293
95591599101,5cyclictest0-21swapper/300:25:293
95591599100,3cyclictest0-21swapper/323:05:303
95591599100,3cyclictest0-21swapper/323:05:303
9559159992,5cyclictest0-21swapper/320:40:313
9559159992,5cyclictest0-21swapper/319:10:363
9559159991,5cyclictest0-21swapper/321:30:023
9559159990,3cyclictest0-21swapper/319:40:363
9559159981,4cyclictest0-21swapper/319:10:023
9559159980,4cyclictest0-21swapper/320:45:273
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional