You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-09 - 18:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat May 09, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4935221697,97sleep30-21swapper/307:06:523
49672186165,14sleep00-21swapper/007:07:160
48562140104,25sleep10-21swapper/107:05:491
51132138104,24sleep20-21swapper/207:09:112
38332780,3sleep012-21ksoftirqd/010:54:120
207042610,2sleep20-21swapper/209:54:392
55042560,1sleep20-21swapper/210:06:422
187592520,2sleep10-21swapper/111:53:131
80742510,3sleep38079-21timerwakeupswit09:20:243
80742510,3sleep38079-21timerwakeupswit09:20:243
144852500,3chrt34-21ksoftirqd/310:37:383
58352480,1sleep30-21swapper/312:07:313
112562480,0sleep1211rcuc/109:46:551
303102460,2sleep00-21swapper/009:37:030
51502450,2sleep10-21swapper/110:06:211
307882450,1sleep30-21swapper/309:14:023
5452994412,9cyclictest17805-21grep10:40:140
50312430,1sleep00-21swapper/012:06:440
142292430,1chrt14230-21/usr/sbin/munin11:25:213
545299429,12cyclictest25018-21diskmemload10:25:250
5452994210,22cyclictest25018-21diskmemload09:11:490
295042420,1sleep30-21swapper/308:05:013
5452994112,12cyclictest26499-21sh11:10:200
5452994112,11cyclictest13789-21ssh12:37:460
545299409,12cyclictest16848-21latency_hist10:40:010
5452994012,13cyclictest23679-21ssh10:20:180
545299397,13cyclictest0-21swapper/011:40:120
5452993913,9cyclictest25018-21diskmemload09:21:320
5452993913,9cyclictest25018-21diskmemload09:21:320
5452993912,9cyclictest25018-21diskmemload11:09:530
5452993912,9cyclictest25018-21diskmemload10:16:190
5452993912,9cyclictest25018-21diskmemload09:33:350
5452993911,10cyclictest10144-21ssh11:46:170
545299389,11cyclictest25018-21diskmemload10:32:380
5452993813,14cyclictest25018-21diskmemload10:47:040
5452993813,10cyclictest29300-21unixbench-2d12:00:250
5452993812,8cyclictest14369-21awk09:25:210
5452993811,9cyclictest25018-21diskmemload11:35:210
5452993811,9cyclictest14825-21seq11:01:420
5452993811,11cyclictest24360-21ssh12:21:000
5452993811,10cyclictest7875-21ssh09:45:040
5452993811,10cyclictest4437-21ssh09:18:210
5452993811,10cyclictest26093-21cron11:35:000
5452993811,10cyclictest25018-21diskmemload10:02:110
31862380,1chrt0-21swapper/110:53:341
545299379,11cyclictest3264-21sh11:17:150
545299377,17cyclictest0-21swapper/011:25:140
5452993713,10cyclictest25018-21diskmemload09:50:050
5452993712,8cyclictest25018-21diskmemload12:14:030
5452993710,9cyclictest23750-21ssh09:55:560
5452993710,11cyclictest7843-21ssh12:33:320
5452993710,10cyclictest19948-21ssh11:54:250
545299369,12cyclictest17369-21chrt12:15:420
5452993611,9cyclictest7281-21sed11:20:170
5452993611,9cyclictest10081-21sed08:35:010
5452993610,9cyclictest25018-21diskmemload10:11:450
5452993610,9cyclictest25018-21diskmemload10:11:440
5452993610,8cyclictest9467-21ssh10:58:050
5452993610,14cyclictest636-21runrttasks08:48:030
5452993610,10cyclictest17466-21sed08:50:190
545299359,16cyclictest0-21swapper/008:55:130
545299359,14cyclictest30752-21chrt12:25:420
545299359,14cyclictest30752-21chrt12:25:420
5452993519,12cyclictest20970-21ssh11:55:130
5452993513,8cyclictest25018-21diskmemload09:44:010
545299339,7cyclictest2133-21perl08:15:160
545299334,17cyclictest0-21swapper/010:06:500
5452993310,9cyclictest6823-21sed07:10:240
5452993310,10cyclictest636-21runrttasks09:00:040
5453993210,11cyclictest22-21ksoftirqd/110:39:591
5453993112,6cyclictest22-21ksoftirqd/112:10:121
545299317,8cyclictest0-21swapper/008:40:190
5452993110,8cyclictest636-21runrttasks08:25:160
545399296,11cyclictest22-21ksoftirqd/109:50:141
5452992915,11cyclictest20110-21unixbench_multi07:40:250
5452992915,11cyclictest20110-21unixbench_multi07:40:250
545399289,4cyclictest22-21ksoftirqd/111:09:551
545399288,5cyclictest22-21ksoftirqd/110:03:481
545399288,10cyclictest22-21ksoftirqd/112:02:131
5453992811,4cyclictest22-21ksoftirqd/112:08:231
545299288,7cyclictest0-21swapper/007:45:190
545299287,8cyclictest14835-21memory07:30:190
545299286,8cyclictest0-21swapper/008:05:150
545299286,8cyclictest0-21swapper/008:00:000
5452992817,8cyclictest12360-21fschecks_time07:25:150
545399279,10cyclictest22-21ksoftirqd/111:41:061
545399278,4cyclictest22-21ksoftirqd/112:21:411
545399277,4cyclictest22-21ksoftirqd/111:22:541
545399275,18cyclictest32622-21sort09:15:171
5453992711,4cyclictest22-21ksoftirqd/109:59:081
5453992710,4cyclictest22-21ksoftirqd/110:46:581
545299278,6cyclictest0-21swapper/007:24:540
545299274,9cyclictest0-21swapper/007:15:230
5452992716,7cyclictest1215-21perf08:15:000
210062270,2chrt0-21swapper/310:18:353
139322270,3sleep20-21swapper/210:37:042
58482260,2sleep20-21swapper/211:19:502
5455992610,13cyclictest335-21cron11:40:003
545399264,10cyclictest22-21ksoftirqd/111:36:541
5453992623,2cyclictest0-21swapper/107:11:481
5453992610,4cyclictest22-21ksoftirqd/112:15:391
545299265,8cyclictest0-21swapper/008:35:180
545299265,7cyclictest0-21swapper/007:50:100
545299265,10cyclictest29325-21runrttasks08:04:540
5452992616,7cyclictest23483-21sort09:05:160
251272260,9sleep334-21ksoftirqd/311:57:523
545599251,23cyclictest0-21swapper/307:11:393
545399259,5cyclictest22-21ksoftirqd/111:48:571
545399259,5cyclictest22-21ksoftirqd/111:01:001
545399259,4cyclictest22-21ksoftirqd/111:13:131
545399259,3cyclictest22-21ksoftirqd/110:34:481
545399257,3cyclictest141rcu_preempt10:25:201
5453992513,4cyclictest22-21ksoftirqd/110:41:481
5453992511,2cyclictest141rcu_preempt11:17:051
545399251,11cyclictest26321-21apt-get09:35:011
5452992516,6cyclictest17115-21grep07:35:190
5452992511,11cyclictest5863-21cat08:25:010
307532250,1sleep10-21swapper/109:14:001
5455992415,6cyclictest278-21systemd-journal07:45:013
5455992415,6cyclictest278-21systemd-journal07:45:003
545499244,14cyclictest32218-21ssh09:15:132
545499241,14cyclictest23330-21fschecks_time10:20:152
545399249,4cyclictest22-21ksoftirqd/109:20:021
545399249,4cyclictest22-21ksoftirqd/109:20:011
545399248,5cyclictest22-21ksoftirqd/112:32:441
545399248,4cyclictest22-21ksoftirqd/110:19:331
545399247,2cyclictest141rcu_preempt08:35:001
545399244,11cyclictest22-21ksoftirqd/109:31:281
545399242,10cyclictest22-21ksoftirqd/111:25:081
5453992410,4cyclictest22-21ksoftirqd/109:40:011
5453992410,4cyclictest22-21ksoftirqd/109:29:051
73342230,4sleep3331rcuc/310:32:123
5454992317,4cyclictest28-21ksoftirqd/207:40:002
5454992312,7cyclictest7603-21ntpq09:20:202
5454992312,7cyclictest7603-21ntpq09:20:202
545499230,11cyclictest232862cut07:50:132
5453992317,4cyclictest22-21ksoftirqd/112:30:011
5453992317,4cyclictest22-21ksoftirqd/112:30:001
5453992311,4cyclictest22-21ksoftirqd/110:21:311
545399230,13cyclictest22-21ksoftirqd/110:10:141
545399230,13cyclictest22-21ksoftirqd/110:10:141
545599221,9cyclictest172642sh09:27:413
5455992215,5cyclictest699-21ssh11:40:043
5455992214,6cyclictest17152-21perl08:50:153
545499221,8cyclictest8310-21idleruntime-cro12:09:592
5454992216,4cyclictest28-21ksoftirqd/211:40:002
5454992216,4cyclictest28-21ksoftirqd/208:40:002
5454992213,6cyclictest29739-21tr10:25:152
5454992212,6cyclictest30800-21cpu10:50:112
545399229,4cyclictest22-21ksoftirqd/110:55:591
545399228,4cyclictest22-21ksoftirqd/111:55:461
545399223,5cyclictest0-21swapper/107:35:151
5453992213,6cyclictest23884-21memory09:05:201
5453992213,6cyclictest11672-21cron07:24:591
545399221,10cyclictest27195-21munin-run08:00:001
5453992210,4cyclictest22-21ksoftirqd/112:35:221
5455992113,6cyclictest31016-21ls10:25:263
5455992113,6cyclictest10469-21ntp_states10:10:233
5455992113,6cyclictest10469-21ntp_states10:10:223
545499212,14cyclictest8016-21ssh09:45:112
5454992115,4cyclictest28-21ksoftirqd/211:05:002
5454992115,4cyclictest28-21ksoftirqd/208:55:002
545499211,12cyclictest0-21swapper/207:55:262
545399219,4cyclictest22-21ksoftirqd/111:31:291
545399215,4cyclictest22-21ksoftirqd/108:25:161
545399214,3cyclictest141rcu_preempt07:20:011
254492210,4sleep3331rcuc/307:55:143
545599208,8cyclictest5981-21apt-get08:25:133
545599201,8cyclictest25018-21diskmemload10:40:243
5455992012,6cyclictest939-21cut09:40:003
5455992012,6cyclictest29964-21seq12:00:583
5455992011,7cyclictest4045-21ntpq12:30:203
545499207,3cyclictest141rcu_preempt09:10:122
545499207,11cyclictest1603-21idleruntime-cro12:05:002
545499202,5cyclictest28-21ksoftirqd/208:00:192
545499201,6cyclictest0-21swapper/210:31:232
5454992014,4cyclictest28-21ksoftirqd/212:35:012
5454992013,5cyclictest28-21ksoftirqd/211:00:012
5454992013,5cyclictest15452-21ntp_kernel_pll_08:45:192
5454992013,5cyclictest15255-21seq12:39:142
5454992012,6cyclictest7758-21tr11:20:222
5454992012,6cyclictest3638-21cut08:20:002
5454992012,6cyclictest28996-21expr12:25:142
5454992012,6cyclictest28996-21expr12:25:132
5454992012,6cyclictest20801-21cat09:30:182
5454992011,7cyclictest18465-21mktemp11:05:022
5454992010,7cyclictest560-21dbus-daemon08:10:002
5454992010,7cyclictest20197-21df_inode11:30:142
545399208,8cyclictest31741-21apt-get08:10:121
545399204,4cyclictest13810-21cron07:30:001
545399202,4cyclictest22-21ksoftirqd/108:55:141
5453992012,6cyclictest21873-21ntpq09:00:201
5453992012,5cyclictest14867-21expr08:45:131
5453992011,6cyclictest27698-21expr08:00:131
545599197,10cyclictest10324-21gltestperf12:35:173
545599192,12cyclictest21541-21ssh09:55:143
545599191,5cyclictest15499-21open_inodes08:45:203
5455991914,3cyclictest8646-21hwlatdetect09:45:163
5455991913,4cyclictest25394-21/usr/sbin/munin11:10:113
5455991913,4cyclictest18013-21sh09:51:573
5455991913,3cyclictest23674-21sh09:32:233
5455991912,5cyclictest10525-21expr08:35:153
5455991912,4cyclictest16629-21sh11:51:043
5455991911,6cyclictest31880-21grep08:10:133
5455991911,6cyclictest22983-21seq11:07:563
5455991911,6cyclictest21760-21ls09:00:193
5455991911,6cyclictest13476-21grep11:00:243
5455991910,7cyclictest7832-21ls07:15:133
5455991910,7cyclictest6864-21tune2fs11:20:143
5455991910,5cyclictest52850irq/140-enp3s0-11:30:123
545499199,7cyclictest9841-21pmu-power12:10:212
545499197,10cyclictest28-21ksoftirqd/208:10:222
545499193,9cyclictest26791-21ssh09:35:042
545499191,5cyclictest0-21swapper/210:45:022
545499191,5cyclictest0-21swapper/210:10:262
545499191,5cyclictest0-21swapper/210:10:262
5454991913,4cyclictest23887-21sh09:56:052
5454991911,6cyclictest7209-21sed11:45:002
5454991911,6cyclictest19466-21grep11:30:002
5454991910,7cyclictest19647-21sed09:30:012
5454991910,6cyclictest2440-21grep09:40:222
5453991914,3cyclictest24116-21open_inodes07:50:221
5453991911,6cyclictest20478-21seq07:43:591
5453991911,6cyclictest20478-21seq07:43:591
170152190,2chrt0-21swapper/108:50:131
133192190,2chrt0-21swapper/311:49:293
545599189,6cyclictest26552-21seq10:46:133
545599189,6cyclictest20936-21cat09:00:013
545599181,7cyclictest6222-21ssh11:20:013
5455991813,3cyclictest9688-21sh10:58:203
5455991813,3cyclictest8581-21sh12:10:053
5455991813,3cyclictest24327-21switchtime09:05:243
5455991813,3cyclictest1655-21acpi08:15:113
5455991813,3cyclictest13490-21rt-features08:40:213
5455991812,4cyclictest30601-21sh10:01:053
5455991812,4cyclictest24585-21sh12:21:133
5455991812,3cyclictest6665-21sh10:07:523
5455991811,5cyclictest52850irq/140-enp3s0-09:40:133
5455991811,5cyclictest29079-21expr12:25:133
5455991811,5cyclictest29079-21expr12:25:133
5455991811,5cyclictest21850-21date07:45:193
5455991811,5cyclictest15531-21expr12:15:123
5455991811,5cyclictest15514-21awk07:30:263
5455991810,6cyclictest9785-21sort07:20:103
5455991810,6cyclictest24562-21grep07:50:273
5455991810,6cyclictest13194-21cat07:25:243
5455991810,6cyclictest1-21systemd08:35:003
5455991810,5cyclictest30181-21cat08:05:183
5455991810,5cyclictest23053-21expr10:20:133
545499187,4cyclictest28-21ksoftirqd/211:11:202
545499186,8cyclictest30838-21kworker/u8:010:40:112
545499182,5cyclictest0-21swapper/209:02:562
5454991813,3cyclictest18242-21sh11:52:412
5454991812,4cyclictest25018-21diskmemload10:00:142
5454991811,5cyclictest8301-21expr08:30:132
5454991811,4cyclictest25018-21diskmemload11:45:042
545499181,13cyclictest27276-21apt12:00:002
5454991810,6cyclictest5557-21cat07:10:132
5454991810,6cyclictest4958-21which08:20:222
5454991810,6cyclictest24390-21sed09:05:252
5454991810,6cyclictest15261-21ls07:30:252
5454991810,5cyclictest13212-21missed_timers08:40:182
545399189,6cyclictest14373-21ls08:45:001
545399181,4cyclictest0-21swapper/107:30:161
5453991811,5cyclictest3255-21seq08:16:171
5453991810,6cyclictest30434-21uname08:05:211
545399180,5cyclictest21853-21ls07:45:191
184402180,14sleep20-21swapper/210:16:022
545599171,6cyclictest31236-21perl10:50:143
5455991710,5cyclictest5435-21seq08:21:273
5455991710,4cyclictest25018-21diskmemload09:15:073
545499177,7cyclictest6302-21df08:25:152
545499176,7cyclictest12266-21df_inode07:25:142
545499172,10cyclictest9798-21apt-get07:20:122
545499171,7cyclictest23565-21timerwakeupswit12:20:212
5454991711,4cyclictest28-21ksoftirqd/207:50:002
5454991711,4cyclictest19005-21sh12:17:232
5454991710,5cyclictest19015-21cat08:55:122
545399173,2cyclictest141rcu_preempt08:35:171
5453991710,5cyclictest4915-21sendmail_mailqu08:20:221
5455991610,4cyclictest16413-21cat07:35:143
5454991611,3cyclictest7612-21/usr/sbin/munin07:15:192
5454991610,4cyclictest28-21ksoftirqd/207:45:002
5454991610,4cyclictest28-21ksoftirqd/207:45:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional