You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 19:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu Mar 19, 2026 12:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4123463999352,32cyclictest23-21ksoftirqd/110:25:021
4123459998985,2cyclictest0-21swapper/012:10:020
412345999841,79cyclictest0-21swapper/009:35:010
4123459997369,2cyclictest0-21swapper/011:55:020
4123459997332,34cyclictest309-21in:imuxsock11:10:000
412345999729,48cyclictest19350irq/34-mmc109:00:030
412346999685,15cyclictest19848-21cron12:20:003
4123459996820,29cyclictest12-21ksoftirqd/010:15:010
4123463996713,11cyclictest0-21swapper/111:35:011
4123469996624,37cyclictest3467602-21systemd-journal10:59:593
4123469996526,36cyclictest4126744-21sshd08:12:353
4123469996524,36cyclictest120-21kswapd011:10:003
4123463996513,10cyclictest23-21ksoftirqd/110:15:011
412346999641,5cyclictest4150369-21sshd09:56:353
412346599644,43cyclictest4128362-21kworker/u8:309:25:012
4123469996313,12cyclictest275-21dbus-daemon12:00:013
4123459996327,32cyclictest418007650rm11:03:480
4123459996327,32cyclictest418007650rm11:03:470
4123469996226,31cyclictest4170737-21sh10:42:233
412346999613,19cyclictest4156206-21grep10:10:013
412346999613,19cyclictest4156206-21grep10:10:003
4123469996123,34cyclictest3468525-21sendmail-mta10:28:583
4123463996118,8cyclictest23-21ksoftirqd/110:04:011
4123459996126,29cyclictest19350irq/34-mmc107:30:020
4123469996011,11cyclictest309-21in:imuxsock07:35:013
4123469996011,11cyclictest309-21in:imuxsock07:35:013
4123469995923,32cyclictest4139744-21sh09:32:333
4123469995923,32cyclictest3891-21sh11:43:553
412346999592,14cyclictest3467602-21systemd-journal11:30:003
4123469995918,7cyclictest37-21ksoftirqd/311:46:103
412346999591,16cyclictest1-21systemd07:15:003
412346399595,11cyclictest23-21ksoftirqd/109:25:491
4123463995914,10cyclictest23-21ksoftirqd/112:20:541
4123463995914,10cyclictest23-21ksoftirqd/112:20:541
4123463995911,8cyclictest23-21ksoftirqd/112:06:301
412345999595,34cyclictest16950irq/224-800000012:15:020
4123469995819,7cyclictest37-21ksoftirqd/309:42:433
4123469995817,7cyclictest37-21ksoftirqd/309:29:243
4123469995817,5cyclictest4176204-21unin-run10:55:023
4123469995817,5cyclictest4176204-21unin-run10:55:013
4123469995816,10cyclictest37-21ksoftirqd/311:54:103
412346999581,17cyclictest4160652-21systemctl10:20:013
4123465995849,5cyclictest4125510-21kworker/2:209:40:032
4123465995849,5cyclictest4125510-21kworker/2:209:40:022
4123465995843,6cyclictest4125510-21kworker/2:210:18:292
4123459995831,4cyclictest12-21ksoftirqd/009:25:370
4123459995828,13cyclictest18610-21ssh12:16:540
4123459995828,13cyclictest11617-21ssh12:01:010
4123469995723,29cyclictest37-21ksoftirqd/309:11:093
412346999572,19cyclictest4129662-21unin-run09:10:013
412346999572,17cyclictest275-21dbus-daemon12:25:013
412346999572,17cyclictest275-21dbus-daemon12:25:003
4123469995720,4cyclictest4146361-21sh09:47:273
4123469995718,7cyclictest4165516-21sshd10:30:363
4123465995749,5cyclictest19925-21kworker/2:112:31:462
412346399571,17cyclictest654-21ssh11:36:261
412345999572,9cyclictest12-21ksoftirqd/010:36:030
412345999572,39cyclictest5056-21ssh11:46:150
4123459995713,40cyclictest12-21ksoftirqd/010:05:090
4123459995713,40cyclictest12-21ksoftirqd/010:05:090
412346999562,17cyclictest1-21systemd12:40:013
4123469995618,4cyclictest4129638-21diskmemload11:22:343
4123469995618,4cyclictest4129638-21diskmemload11:22:343
4123469995616,6cyclictest4180616-21unin-run11:05:023
4123469995616,6cyclictest4180616-21unin-run11:05:013
4123469995615,6cyclictest4193457-21rm11:34:003
4123469995611,10cyclictest1-21systemd10:50:003
412346399568,12cyclictest23-21ksoftirqd/107:54:591
412346399568,11cyclictest23-21ksoftirqd/110:49:581
412346399567,12cyclictest23-21ksoftirqd/111:24:041
412346399567,12cyclictest23-21ksoftirqd/111:24:041
4123463995611,9cyclictest23-21ksoftirqd/108:25:011
412345999565,5cyclictest131rcu_preempt10:23:590
4123459995621,18cyclictest28696-21ssh12:40:010
4123459995617,34cyclictest12-21ksoftirqd/011:11:440
4123469995536,15cyclictest1-21systemd09:00:023
412346999552,17cyclictest4158427-21systemd-run10:15:003
4123469995518,5cyclictest37-21ksoftirqd/309:23:113
4123469995512,10cyclictest4128181-21cat08:40:023
4123465995547,5cyclictest4128877-21kworker/2:011:18:322
4123465995546,5cyclictest4125510-21kworker/2:211:51:542
4123465995546,5cyclictest4125510-21kworker/2:210:42:462
4123463995533,18cyclictest275-21dbus-daemon12:20:021
4123463995513,13cyclictest23-21ksoftirqd/112:00:011
412345999553,31cyclictest1419-21kworker/u8:111:45:030
4123459995525,15cyclictest4164513-21ssh10:28:350
412345999551,38cyclictest4141851-21ssh09:37:170
412345999551,38cyclictest4141851-21ssh09:37:170
4123469995425,26cyclictest4141121-21sshd09:35:343
4123469995425,26cyclictest4141121-21sshd09:35:343
4123469995420,4cyclictest15337-21sh12:09:463
4123469995417,5cyclictest37-21ksoftirqd/310:01:083
4123469995417,4cyclictest4129638-21diskmemload12:11:473
4123469995416,4cyclictest3467562-21sshd09:53:353
4123465995446,5cyclictest4125510-21kworker/2:211:49:392
4123465995446,5cyclictest19925-21kworker/2:112:27:072
4123465995444,6cyclictest4125510-21kworker/2:210:50:542
4123465995444,6cyclictest4125510-21kworker/2:210:50:532
4123463995420,11cyclictest23-21ksoftirqd/107:20:011
4123463995412,12cyclictest23-21ksoftirqd/111:20:021
4123459995440,11cyclictest12-21ksoftirqd/009:11:570
4123459995426,12cyclictest23161-21ssh12:27:110
412345999542,47cyclictest4167254-21cron10:35:000
4123459995411,4cyclictest12-21ksoftirqd/011:18:010
4123469995319,7cyclictest37-21ksoftirqd/308:35:013
4123469995317,6cyclictest37-21ksoftirqd/309:05:003
4123465995345,5cyclictest4128877-21kworker/2:012:17:502
4123465995345,5cyclictest4125510-21kworker/2:211:01:272
4123465995345,5cyclictest4125510-21kworker/2:211:01:262
4123465995344,6cyclictest4128877-21kworker/2:011:06:012
4123465995344,6cyclictest4128877-21kworker/2:010:33:272
4123465995344,6cyclictest4125510-21kworker/2:211:41:072
4123465995344,6cyclictest4125510-21kworker/2:210:14:302
4123465995341,8cyclictest4128877-21kworker/2:010:30:002
412346399537,11cyclictest23-21ksoftirqd/111:00:021
412346399535,8cyclictest23-21ksoftirqd/111:43:331
412346399534,11cyclictest4145142-21rm09:44:561
4123463995314,8cyclictest23-21ksoftirqd/109:18:201
4123463995314,8cyclictest23-21ksoftirqd/109:18:201
4123463995313,9cyclictest23-21ksoftirqd/112:40:011
4123459995316,29cyclictest12-21ksoftirqd/009:50:240
412345999531,44cyclictest0-21swapper/010:45:440
4123469995223,25cyclictest4124949-21latency_hist07:40:013
4123469995223,24cyclictest37-21ksoftirqd/309:15:533
4123469995223,24cyclictest37-21ksoftirqd/309:15:533
4123469995222,6cyclictest37-21ksoftirqd/312:30:013
4123469995217,6cyclictest37-21ksoftirqd/311:39:073
4123469995216,10cyclictest37-21ksoftirqd/311:17:093
4123465995244,5cyclictest4128877-21kworker/2:011:32:452
4123465995244,5cyclictest4128877-21kworker/2:010:35:072
4123465995244,5cyclictest4125510-21kworker/2:209:50:022
4123465995244,5cyclictest4125510-21kworker/2:209:32:202
4123465995243,6cyclictest4125510-21kworker/2:208:45:022
4123465995243,5cyclictest4125510-21kworker/2:209:30:032
412346399525,6cyclictest23-21ksoftirqd/112:14:481
412346399525,10cyclictest23-21ksoftirqd/109:58:331
4123463995214,10cyclictest23-21ksoftirqd/108:30:011
4123463995214,10cyclictest23-21ksoftirqd/108:30:011
4123459995245,4cyclictest16850irq/134-800000009:23:350
4123459995236,11cyclictest3467602-21systemd-journal08:15:000
4123459995234,4cyclictest16850irq/134-800000011:29:350
4123459995217,9cyclictest12-21ksoftirqd/008:00:010
4123459995217,8cyclictest4189644-21sed11:25:030
4123469995121,4cyclictest4129638-21diskmemload11:11:373
4123469995112,33cyclictest0-21swapper/308:00:003
4123465995143,5cyclictest4128877-21kworker/2:011:28:382
4123465995143,5cyclictest4125510-21kworker/2:210:45:512
4123465995143,5cyclictest4125510-21kworker/2:210:22:032
4123465995143,5cyclictest4125510-21kworker/2:209:57:562
4123465995143,5cyclictest4113512-21kworker/2:307:35:012
4123465995143,5cyclictest4113512-21kworker/2:307:35:012
4123465995143,5cyclictest19925-21kworker/2:112:38:372
4123465995142,5cyclictest4125510-21kworker/2:210:10:012
4123465995142,5cyclictest4125510-21kworker/2:210:10:012
4123465995142,5cyclictest4125510-21kworker/2:209:00:012
4123465995135,8cyclictest4128877-21kworker/2:011:22:082
4123465995135,8cyclictest4128877-21kworker/2:011:22:082
412346399517,8cyclictest23-21ksoftirqd/109:00:011
4123459995134,11cyclictest4151788-21ssh10:00:010
412345999512,45cyclictest4145253-21sort09:45:010
4123459995118,29cyclictest4129681-21latency_hist09:10:020
412345999511,45cyclictest275-21dbus-daemon09:00:010
4123465995043,5cyclictest4125510-21kworker/2:210:02:322
4123465995043,4cyclictest4125510-21kworker/2:209:50:472
4123465995042,5cyclictest4125510-21kworker/2:212:05:132
4123465995042,5cyclictest4125510-21kworker/2:212:03:532
4123465995042,5cyclictest4125510-21kworker/2:211:37:472
4123465995042,5cyclictest19925-21kworker/2:112:22:322
4123465995042,5cyclictest19925-21kworker/2:112:22:312
4123465995041,6cyclictest4125510-21kworker/2:209:15:002
4123465995041,6cyclictest4113512-21kworker/2:307:15:012
4123465995041,5cyclictest4113512-21kworker/2:307:55:012
412346399506,9cyclictest23-21ksoftirqd/111:54:101
412346399505,14cyclictest23-21ksoftirqd/110:08:501
412346399505,14cyclictest23-21ksoftirqd/110:08:501
412346399505,11cyclictest23-21ksoftirqd/109:34:141
412346399504,16cyclictest23-21ksoftirqd/108:45:011
412346399504,10cyclictest23-21ksoftirqd/111:01:131
412346399504,10cyclictest23-21ksoftirqd/111:01:131
412346399503,9cyclictest23-21ksoftirqd/110:40:011
4123463995033,13cyclictest3467602-21systemd-journal10:30:361
4123463995025,17cyclictest275-21dbus-daemon10:45:011
4123463995014,9cyclictest23-21ksoftirqd/111:50:001
4123463995011,10cyclictest23-21ksoftirqd/109:13:091
4123459995043,4cyclictest16850irq/134-800000012:00:020
412345999502,12cyclictest22124-21unin-run12:25:010
412345999502,12cyclictest22124-21unin-run12:25:000
4123459995011,35cyclictest12-21ksoftirqd/007:19:350
4123469994919,4cyclictest4126627-21latency_hist08:10:013
4123469994917,6cyclictest37-21ksoftirqd/312:00:413
412346999491,11cyclictest1-21systemd08:25:013
4123465994942,4cyclictest4125510-21kworker/2:209:40:052
4123465994941,5cyclictest4125510-21kworker/2:212:10:042
4123465994941,5cyclictest4125510-21kworker/2:211:58:522
4123465994941,5cyclictest4125510-21kworker/2:210:59:502
412346399496,9cyclictest23-21ksoftirqd/109:25:001
412346399495,5cyclictest23-21ksoftirqd/112:33:531
412346399494,9cyclictest23-21ksoftirqd/110:54:471
412346399494,9cyclictest23-21ksoftirqd/110:54:471
412345999493,8cyclictest4176180-21modprobe10:55:010
412345999493,8cyclictest4176180-21modprobe10:55:000
4123459994933,12cyclictest275-21dbus-daemon08:50:000
4123459994913,7cyclictest12-21ksoftirqd/009:45:440
4123469994821,5cyclictest25307-21sh12:32:043
4123469994817,4cyclictest4161018-21sh10:20:203
4123465994841,4cyclictest4125510-21kworker/2:209:18:332
4123465994841,4cyclictest4125510-21kworker/2:209:18:332
4123465994839,6cyclictest4128877-21kworker/2:011:12:412
4123465994839,6cyclictest4113512-21kworker/2:308:05:012
412346399486,9cyclictest23-21ksoftirqd/107:45:011
412346399486,7cyclictest23-21ksoftirqd/110:25:181
412346399485,8cyclictest23-21ksoftirqd/111:05:431
4123459994839,5cyclictest16950irq/224-800000011:25:000
4123459994822,12cyclictest4152930-21ssh10:02:190
412345999481,43cyclictest41589111ssh10:15:390
4123465994731,11cyclictest4124965-21tr07:40:002
412346399477,8cyclictest23-21ksoftirqd/107:10:011
412346399476,9cyclictest23-21ksoftirqd/109:46:581
4123463994718,11cyclictest23-21ksoftirqd/112:02:511
4123463994710,8cyclictest23-21ksoftirqd/111:28:431
4123459994739,5cyclictest16950irq/224-800000011:31:260
4123459994724,17cyclictest275-21dbus-daemon07:50:010
4123459994724,17cyclictest275-21dbus-daemon07:50:010
4123459994713,30cyclictest12-21ksoftirqd/007:11:350
4123465994639,4cyclictest4125510-21kworker/2:208:55:022
4123465994639,4cyclictest4113512-21kworker/2:307:28:252
4123465994638,5cyclictest4125510-21kworker/2:208:40:012
4123465994638,5cyclictest4125510-21kworker/2:208:26:012
4123465994638,5cyclictest4125510-21kworker/2:208:26:002
4123465994638,5cyclictest4125510-21kworker/2:208:15:002
4123465994638,5cyclictest4113512-21kworker/2:307:10:032
412346599463,26cyclictest4126775-21kworker/u8:008:25:022
412346399465,10cyclictest23-21ksoftirqd/109:35:051
412346399465,10cyclictest23-21ksoftirqd/109:35:041
412346399463,7cyclictest23-21ksoftirqd/111:14:151
4123463994636,8cyclictest23-21ksoftirqd/109:10:001
4123463994629,12cyclictest4128074-21systemctl08:40:001
4123463994614,8cyclictest0-21swapper/109:54:191
412345999463,3cyclictest131rcu_preempt09:16:330
412345999463,3cyclictest131rcu_preempt09:16:320
4123459994630,10cyclictest275-21dbus-daemon08:55:000
4123459994612,30cyclictest210-21jbd2/mmcblk1p2-12:32:250
4123465994537,5cyclictest4125510-21kworker/2:209:03:142
412346399459,6cyclictest23-21ksoftirqd/112:28:541
412346399457,8cyclictest23-21ksoftirqd/110:16:181
4123463994510,8cyclictest0-21swapper/108:14:251
4123465994437,4cyclictest4125510-21kworker/2:208:20:012
4123465994435,6cyclictest4113512-21kworker/2:307:45:002
4123465994430,11cyclictest30-21ksoftirqd/207:20:012
412346399447,6cyclictest23-21ksoftirqd/108:00:011
4123459994428,13cyclictest1511-21ssh11:38:340
4123459994419,18cyclictest275-21dbus-daemon08:35:000
4123469994328,9cyclictest309-21in:imuxsock07:54:593
4123465994336,4cyclictest4125510-21kworker/2:208:30:142
4123465994334,5cyclictest4125510-21kworker/2:208:47:362
4123465994330,10cyclictest30-21ksoftirqd/209:10:022
412346399436,7cyclictest0-21swapper/107:35:021
412346399436,7cyclictest0-21swapper/107:35:021
412346399431,8cyclictest4126264-21munin-run08:05:001
4123459994328,12cyclictest4178183-21ssh10:59:350
4123469994231,8cyclictest37-21ksoftirqd/307:10:023
4123469994228,10cyclictest4169480-21cron10:39:593
4123469994221,14cyclictest275-21dbus-daemon08:20:003
4123465994236,4cyclictest4113512-21kworker/2:307:58:142
412346599421,3cyclictest131rcu_preempt08:10:012
412346399429,5cyclictest131rcu_preempt08:55:001
4123459994232,5cyclictest309-21in:imuxsock08:17:000
4123459994228,11cyclictest1-21systemd07:35:020
4123459994228,11cyclictest1-21systemd07:35:010
4123459994223,15cyclictest3467602-21systemd-journal07:20:350
412345999421,7cyclictest275-21dbus-daemon08:05:010
412345999421,37cyclictest4170086-21ssh10:40:510
412345999421,37cyclictest0-21swapper/007:40:000
4123469994128,10cyclictest37-21ksoftirqd/308:50:003
4123469994123,14cyclictest275-21dbus-daemon08:05:013
4123469994119,14cyclictest275-21dbus-daemon07:30:013
412346399415,8cyclictest23-21ksoftirqd/108:20:001
4123459994111,20cyclictest0-21swapper/007:10:010
4123469994030,8cyclictest37-21ksoftirqd/307:20:023
4123465994032,5cyclictest4113512-21kworker/2:307:25:012
4123463994032,4cyclictest4129378-21munin-plugin-st09:05:001
412345999405,8cyclictest4125220-21modprobe07:45:000
412345999402,33cyclictest1-21systemd08:27:360
412345999402,33cyclictest1-21systemd08:27:350
412346399399,7cyclictest0-21swapper/108:30:231
4123463993928,8cyclictest4124190-21/usr/sbin/munin07:25:011
4123463993926,9cyclictest23-21ksoftirqd/108:10:011
4123469993829,5cyclictest1-21systemd07:25:003
4123465993830,5cyclictest4113512-21kworker/2:307:48:582
4123465993830,5cyclictest4113512-21kworker/2:307:48:582
412346399385,9cyclictest23-21ksoftirqd/107:40:011
412346999372,12cyclictest150-21pr/ttyS207:45:003
4123469993720,11cyclictest275-21dbus-daemon08:45:013
4123459993721,12cyclictest4126653-21wc08:10:020
4123469993624,9cyclictest37-21ksoftirqd/308:50:353
4123469993624,10cyclictest37-21ksoftirqd/307:50:013
4123469993624,10cyclictest37-21ksoftirqd/307:50:013
412346399367,7cyclictest0-21swapper/107:29:591
4123463993622,9cyclictest23-21ksoftirqd/107:10:491
412346999351,8cyclictest4127574-21unin-run08:30:013
412346999351,8cyclictest4127574-21unin-run08:30:013
4123463993524,9cyclictest23-21ksoftirqd/108:50:001
4123459993525,7cyclictest12-21ksoftirqd/008:45:020
4123459993518,14cyclictest4128051-21cron08:40:000
412345999333,4cyclictest0-21swapper/008:21:580
4123459993226,3cyclictest41256951sshd07:50:230
4123463993017,11cyclictest23-21ksoftirqd/107:45:221
4123463993017,11cyclictest23-21ksoftirqd/107:45:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional