You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-27 - 12:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Apr 27, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1917728975,9sleep00-21swapper/019:05:410
1938828168,9sleep20-21swapper/219:08:242
1921727764,9sleep30-21swapper/319:06:133
1937227659,13sleep10-21swapper/119:08:111
205312480,1sleep30-21swapper/321:14:513
1958499330,2cyclictest25364-21kernelversion00:30:210
1958499314,26cyclictest12390-21diskmemload21:28:290
1958499303,26cyclictest12390-21diskmemload22:14:110
1958499293,25cyclictest12390-21diskmemload22:58:270
1958499292,25cyclictest10071-21apt-key21:05:010
1958499292,19cyclictest0-21swapper/023:16:430
1958499290,3cyclictest0-21swapper/023:43:380
1958499290,2cyclictest0-21swapper/020:25:140
19589992822,3cyclictest25-21ksoftirqd/100:06:491
1958499284,23cyclictest22437-21cron00:10:000
1958499281,26cyclictest2712-21rm22:40:190
1958499281,26cyclictest12390-21diskmemload21:44:340
1958499280,3cyclictest22967-21tail21:35:160
1958499273,23cyclictest31349-21apt-get00:15:140
1958499272,12cyclictest0-21swapper/023:23:390
1958499271,1cyclictest12390-21diskmemload22:19:180
1958499270,3cyclictest5717-21kworker/u8:022:05:080
1958499270,25cyclictest0-21swapper/022:22:360
1958499270,1cyclictest0-21swapper/023:34:080
1958499262,2cyclictest12390-21diskmemload23:13:040
1958499262,23cyclictest12390-21diskmemload22:25:540
1958499262,19cyclictest12935-21ssh21:10:120
1958499261,24cyclictest12390-21diskmemload21:55:040
1958499261,23cyclictest0-21swapper/023:25:250
1958499260,2cyclictest0-21swapper/022:45:130
1958499260,2cyclictest0-21swapper/022:31:550
1958499260,25cyclictest0-21swapper/021:15:320
1958499260,21cyclictest0-21swapper/023:02:390
1959499252,23cyclictest32-21ksoftirqd/200:23:532
1958999250,25cyclictest0-21swapper/123:50:221
1958499251,2cyclictest21157-21perf23:50:010
1958499251,1cyclictest12390-21diskmemload23:06:370
1958499250,2cyclictest12390-21diskmemload23:37:370
1958499250,24cyclictest0-21swapper/000:21:090
1958999240,24cyclictest0-21swapper/100:36:101
1958499242,3cyclictest879-21python22:00:250
1958499242,1cyclictest12390-21diskmemload21:46:330
1958499241,22cyclictest15971-21latency_hist20:10:000
1958499241,1cyclictest0-21swapper/021:32:220
1958499240,0cyclictest0-21swapper/023:54:090
19594992312,10cyclictest55550irq/126-enp2s0-22:30:222
1958499233,19cyclictest11007-21/usr/sbin/munin20:00:000
1958499232,2cyclictest12390-21diskmemload00:11:240
1958499232,1cyclictest0-21swapper/019:40:150
1958499230,22cyclictest0-21swapper/023:56:220
1958499230,1cyclictest0-21swapper/021:54:430
1958499230,16cyclictest0-21swapper/021:22:540
19598992219,1cyclictest28987-21ssh23:35:123
1959499225,5cyclictest3765-21ssh22:21:282
1959499224,6cyclictest20812-21apache_volume20:20:142
1959499220,3cyclictest15531-21ssh22:29:092
1958999220,17cyclictest0-21swapper/122:01:051
1958499223,18cyclictest12390-21diskmemload00:03:190
1958499221,1cyclictest12390-21diskmemload22:37:120
1958499220,21cyclictest0-21swapper/019:35:140
1959499217,12cyclictest20267-21sh00:27:262
19594992118,2cyclictest22441-21cron00:10:012
1959499211,2cyclictest26982-21apache_processe19:25:122
1959499211,2cyclictest12390-21diskmemload00:02:252
19589992115,4cyclictest25-21ksoftirqd/121:39:521
1958999210,1cyclictest0-21swapper/122:30:231
1958499210,20cyclictest0-21swapper/020:40:160
1958499210,16cyclictest0-21swapper/019:30:210
1958499210,0cyclictest0-21swapper/000:25:220
1959499205,13cyclictest12390-21diskmemload22:01:052
19594992012,7cyclictest0-21swapper/222:35:132
1959499200,1cyclictest0-21swapper/221:17:212
1958999202,17cyclictest19099-21latency_hist23:10:011
1958499202,1cyclictest12390-21diskmemload00:37:560
1959899190,19cyclictest0-21swapper/322:14:103
1959499192,12cyclictest32-21ksoftirqd/221:12:172
19594991915,3cyclictest0-21swapper/223:27:332
1959499191,17cyclictest0-21swapper/221:21:532
1959499191,17cyclictest0-21swapper/200:13:292
19594991910,7cyclictest25384-21kernelversion00:30:212
1958999190,2cyclictest0-21swapper/121:40:011
1958999190,0cyclictest0-21swapper/121:53:531
1958499196,8cyclictest18366-21ssh22:50:040
1958499190,1cyclictest7401-21cron20:59:590
1959499188,5cyclictest0-21swapper/223:22:342
1959499188,4cyclictest19121-21sh23:10:002
1959499187,1cyclictest12390-21diskmemload00:38:042
19594991816,1cyclictest12390-21diskmemload23:50:472
19594991811,7cyclictest12390-21diskmemload22:59:102
1959499180,17cyclictest0-21swapper/222:40:042
19589991817,1cyclictest0-21swapper/121:34:221
1958499181,2cyclictest1214-21sed20:45:180
1958499180,3cyclictest25751-21ntp_kernel_pll_19:20:210
1958499180,1cyclictest0-21swapper/019:15:150
1959899170,1cyclictest141rcu_preempt21:24:483
19594991714,1cyclictest12390-21diskmemload23:15:142
19594991713,4cyclictest0-21swapper/223:45:302
19594991711,4cyclictest32-21ksoftirqd/221:28:522
19594991710,7cyclictest12390-21diskmemload23:34:592
1958999179,4cyclictest12390-21diskmemload23:57:411
1958999172,1cyclictest141rcu_preempt23:36:291
19589991714,2cyclictest25-21ksoftirqd/122:53:121
19589991712,2cyclictest29081-21cron19:30:001
1958999171,0cyclictest141rcu_preempt22:41:441
1958999170,1cyclictest0-21swapper/100:14:021
1958499178,8cyclictest29300-21date19:30:010
1958499172,14cyclictest12412-21awk21:10:000
1958499170,1cyclictest0-21swapper/020:10:160
1959899164,7cyclictest17975-21sshd20:13:043
1959499166,6cyclictest0-21swapper/222:53:082
1959499166,5cyclictest8924-21/usr/sbin/munin22:05:232
1959499164,8cyclictest32-21ksoftirqd/222:18:512
1959499164,11cyclictest0-21swapper/223:38:242
1959499163,3cyclictest0-21swapper/221:44:572
1959499161,1cyclictest21445-21sh22:13:142
1959499160,2cyclictest20854-21missed_timers19:10:192
19589991616,0cyclictest0-21swapper/100:32:461
19589991615,1cyclictest241rcuc/123:33:521
1958999161,2cyclictest141rcu_preempt23:43:311
19589991611,2cyclictest25-21ksoftirqd/122:10:521
1958999160,1cyclictest141rcu_preempt23:23:031
1958999160,1cyclictest0-21swapper/121:55:221
1958499168,7cyclictest22895-21chrt20:24:510
1958499161,14cyclictest0-21swapper/020:35:150
1958499160,1cyclictest274492sleep020:32:200
1959499159,5cyclictest927-21users00:15:282
1959499157,7cyclictest0-21swapper/221:34:502
1959499154,11cyclictest0-21swapper/221:55:432
1959499153,6cyclictest0-21swapper/223:04:382
19594991514,1cyclictest32-21ksoftirqd/221:38:512
19594991512,3cyclictest0-21swapper/223:42:442
19594991511,2cyclictest16401-21ssh21:50:252
1958999151,13cyclictest28073-21ssh22:35:431
1958499157,7cyclictest6974-21expr19:50:160
1958499150,2cyclictest0-21swapper/020:50:170
1958499150,0cyclictest0-21swapper/019:45:160
19598991413,1cyclictest0-21swapper/323:13:443
19598991410,3cyclictest21503-21sed19:10:263
19598991410,3cyclictest1562-21wc22:20:193
1959899140,12cyclictest0-21swapper/300:12:163
1959499144,6cyclictest10866-21df_inode22:45:162
19594991412,1cyclictest0-21swapper/220:59:592
1958999149,1cyclictest25-21ksoftirqd/122:55:091
1958999145,8cyclictest0-21swapper/123:25:081
1958999142,1cyclictest141rcu_preempt23:45:161
1958999142,1cyclictest141rcu_preempt00:20:131
19589991413,1cyclictest241rcuc/122:15:211
19589991410,3cyclictest30343-21ssh23:15:451
19589991410,2cyclictest19898-21grep19:10:131
1958499141,9cyclictest225-21systemd-journal20:20:000
19584991411,2cyclictest12916-21grep20:00:260
19584991411,2cyclictest12916-21grep20:00:250
1958499140,1cyclictest0-21swapper/019:10:140
1959899139,2cyclictest4813-21sort19:45:153
1959899138,4cyclictest39-21ksoftirqd/322:00:173
1959899136,3cyclictest0-21swapper/323:05:153
1959899135,7cyclictest16160-31sort00:06:503
1959899135,2cyclictest0-21swapper/323:32:333
19598991310,2cyclictest5247-21cat23:20:223
19598991310,2cyclictest29878-21sed23:15:273
19598991310,2cyclictest28632-21grep19:25:263
19598991310,2cyclictest26227-21cut20:30:173
1959899130,12cyclictest32621-21wc20:44:593
1959499139,3cyclictest16959-21cut20:10:192
1959499139,2cyclictest26707-21cron19:25:002
1959499137,1cyclictest25502-21cut20:30:002
1959499136,1cyclictest0-21swapper/223:10:392
19594991311,1cyclictest31737-21apt-get19:35:142
19594991310,2cyclictest8036-21basename21:45:222
1958999139,3cyclictest23767-21sort00:30:001
1958999138,5cyclictest25-21ksoftirqd/119:15:151
1958999138,4cyclictest10981-21taskset19:59:581
1958999136,5cyclictest141rcu_preempt22:20:251
1958999135,4cyclictest225-21systemd-journal22:10:011
1958999134,4cyclictest29065-21dump-pmu-power21:20:001
1958999132,3cyclictest141rcu_preempt21:05:151
1958999132,0cyclictest141rcu_preempt21:49:481
19589991310,2cyclictest0-21swapper/121:10:551
1959899129,2cyclictest9268-21grep22:05:263
1959899129,2cyclictest3288-21ssh23:58:223
1959899129,2cyclictest16955-21tr00:25:223
1959899129,2cyclictest13388-21idleruntime-cro20:05:013
1959899129,2cyclictest13388-21idleruntime-cro20:05:003
1959899129,2cyclictest10983-21cron20:00:013
1959899128,3cyclictest30932-21cron22:00:003
1959899127,3cyclictest7379-21grep00:20:133
1959499129,2cyclictest20557-21grep20:20:002
1959499129,2cyclictest10510-21sed19:55:262
1959499129,1cyclictest3529-21taskset19:41:292
1959499128,1cyclictest2356-21taskset20:49:462
1959499122,5cyclictest3012-21runrttasks23:58:092
19594991211,1cyclictest32-21ksoftirqd/220:33:042
19594991210,1cyclictest9300-21grep21:00:282
1959499120,11cyclictest14870-21ls20:05:222
1958999129,2cyclictest6234-21cron19:50:011
1958999129,2cyclictest31561-21grep21:20:271
1958999129,2cyclictest25356-21tr19:20:181
1958999129,2cyclictest21524-21grep20:20:191
1958999129,2cyclictest13863-21awk00:05:001
1958999129,2cyclictest12127-21ssh22:45:261
1958999126,1cyclictest141rcu_preempt20:35:121
1958999123,4cyclictest225-21systemd-journal19:35:001
1958999123,1cyclictest141rcu_preempt23:12:491
1958999121,1cyclictest141rcu_preempt20:45:271
1958999120,2cyclictest0-21swapper/121:29:171
1959899119,1cyclictest267002sleep322:55:043
1959899118,2cyclictest8651-21ssh21:26:223
1959899118,2cyclictest7756-21tr21:45:203
1959899118,2cyclictest7582-21cat19:50:203
1959899118,2cyclictest6138-21ssh00:38:203
1959899118,2cyclictest2847-21ssh21:42:373
1959899118,2cyclictest27908-21ssh22:35:353
1959899118,2cyclictest2549-21ssh23:00:003
1959899118,2cyclictest25207-21head22:15:173
1959899118,2cyclictest24099-21ssh00:30:053
1959899118,2cyclictest23388-21grep20:25:153
1959899118,2cyclictest22972-21ntp_states23:50:203
1959899118,2cyclictest22888-21ssh21:15:253
1959899118,2cyclictest22521-21grep20:20:283
1959899118,2cyclictest22385-21sh22:51:533
1959899118,2cyclictest18415-21cut20:15:133
1959899118,2cyclictest15644-21ssh23:45:423
1959899118,2cyclictest1500-21ssh00:15:533
1959899118,2cyclictest14373-21grep21:30:163
1959899118,2cyclictest1414-21ssh22:40:043
1959899118,2cyclictest13532-21ssh00:04:463
1959899118,2cyclictest12620-21tr23:25:153
1959899118,2cyclictest11243-21ssh22:45:183
1959899118,2cyclictest10606-21expr21:05:143
1959899118,2cyclictest10132-21ssh23:42:563
1959899118,2cyclictest10099-21taskset22:25:223
1959899117,3cyclictest2937-21grep20:50:153
1959899117,3cyclictest16800-21sed21:50:293
1959899115,4cyclictest22753-21df_inode21:35:153
1959499119,1cyclictest30615-21wc19:30:222
1959499119,1cyclictest241232sleep219:17:332
1959499119,1cyclictest11545-21grep20:00:142
1959499119,1cyclictest11545-21grep20:00:132
1959499113,4cyclictest225-21systemd-journal21:09:592
1958999119,1cyclictest38150irq/131-snd_hda22:25:141
1958999118,2cyclictest32151-21tail00:15:181
1958999118,2cyclictest26784-21grep20:30:221
1958999118,2cyclictest17959-21taskset20:12:521
1958999118,1cyclictest7422-21dump-pmu-power20:59:591
1958999113,4cyclictest25-21ksoftirqd/123:00:071
1959899109,1cyclictest39-21ksoftirqd/320:06:003
1959899108,1cyclictest3178-21threads19:40:263
1959899108,1cyclictest29853-21taskset20:37:453
1959899108,1cyclictest29545-21egrep19:30:153
1959899108,1cyclictest17375-21cat22:30:133
1959899107,2cyclictest7085-21taskset20:56:003
1959899107,2cyclictest31800-21wget19:35:143
1959899107,2cyclictest25700-21grep19:20:213
1959899107,2cyclictest23306-21cut19:15:203
1959899107,2cyclictest1390-21uname20:45:203
1959499108,1cyclictest6752-21grep19:50:152
1959499103,3cyclictest0-21swapper/219:45:122
1959499101,8cyclictest0-21swapper/220:50:142
1958999108,1cyclictest8035-21cat21:00:161
1958999108,1cyclictest7907-21python19:50:251
1958999108,1cyclictest20235-21chrt20:16:301
1958999108,1cyclictest155082sleep120:06:531
1958999107,2cyclictest3637-21grep20:50:191
1958999107,2cyclictest30579-21tr20:40:141
1958999107,2cyclictest1854-21egrep19:40:151
1958999107,2cyclictest1232-21taskset19:37:471
195989996,2cyclictest7833-21wget21:00:163
195949991,5cyclictest27931-21apache_volume20:35:122
195949990,4cyclictest0-21swapper/220:40:142
195899998,1cyclictest25-21ksoftirqd/120:25:221
195899997,1cyclictest12303-21taskset20:00:191
195899997,1cyclictest12303-21taskset20:00:181
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional