You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 19:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri May 15, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
257072240216,17sleep30-21swapper/307:08:533
25608214999,24sleep00-21swapper/007:07:370
25755214696,25sleep10-21swapper/107:09:301
25473214599,39sleep20-21swapper/207:05:512
241472790,2chrt12-21ksoftirqd/011:18:170
241472790,2chrt12-21ksoftirqd/011:18:160
153032610,4sleep015287-21ssh11:35:220
260902590,2sleep20-21swapper/212:08:102
312772580,3sleep20-21swapper/207:20:182
312772580,3sleep20-21swapper/207:20:172
318862560,1sleep00-21swapper/012:35:460
243882560,2sleep124383-21ssh12:30:211
170762540,1sleep217077-21ssh10:49:042
134222530,2sleep30-21swapper/311:58:533
133122520,2sleep00-21swapper/009:33:590
13562510,2sleep30-21swapper/310:36:583
26271994712,25cyclictest22503-21sh10:52:483
5892460,2sleep00-21swapper/008:40:130
2627199440,30cyclictest25949-21ssh11:43:573
145292440,1sleep20-21swapper/212:23:442
26271994210,16cyclictest24665-21sh11:18:483
26271994210,16cyclictest24665-21sh11:18:483
2627199411,23cyclictest4128-21sh09:26:283
2627199411,23cyclictest12307-21ssh11:33:443
2627199411,22cyclictest28970-21ssh11:21:213
2627199401,24cyclictest636-21runrttasks12:12:173
2627199401,23cyclictest22780-21ssh10:29:243
2627199401,23cyclictest2020-21ssh09:49:513
2627199401,23cyclictest2020-21ssh09:49:503
2627199401,23cyclictest18721-21sh12:02:273
2627199401,23cyclictest18696-21sh11:14:313
2627199401,22cyclictest27792-21ssh12:33:233
2627199401,22cyclictest26146-21sh09:43:123
2627199401,22cyclictest10825-21ssh10:44:453
2627199401,21cyclictest19201-21sh12:26:293
2627199401,21cyclictest15999-21cat12:25:013
2627199401,20cyclictest26014-21ssh10:30:553
2627199400,22cyclictest30889-21ssh11:47:133
2627199400,22cyclictest10711-21ssh09:31:213
243182400,1sleep20-21swapper/209:17:392
2627199391,23cyclictest13231-21diskmemload12:37:273
2627199391,23cyclictest13231-21diskmemload12:18:173
2627199391,22cyclictest26915-21sh09:20:013
2627199391,22cyclictest20131-21sh09:38:553
2627199391,21cyclictest961-21sh10:12:373
2627199391,21cyclictest5632-21sh09:51:453
2627199391,21cyclictest27035-21sh10:55:383
2627199391,21cyclictest11536-21sh09:55:543
2627199390,22cyclictest22760-21ssh09:16:053
2627199381,22cyclictest5409-21chrt11:52:313
2627199381,22cyclictest20924-21taskset10:03:353
2627199381,21cyclictest8749-21sh11:06:283
2627199381,21cyclictest19824-21ssh09:14:543
2627199380,21cyclictest16641-21ssh10:48:393
2627199371,20cyclictest26330-21ssh10:07:153
2627199371,19cyclictest400-21ssh11:25:033
2627199370,21cyclictest7221-21ssh10:17:123
2627199370,21cyclictest24501-21ssh12:06:343
2627199370,20cyclictest4594-21taskset07:35:083
2627199361,17cyclictest13231-21diskmemload10:20:533
2627199360,20cyclictest4069-21ssh11:03:293
2627199351,19cyclictest21791-21chrt08:14:403
2627199350,21cyclictest6649-21taskset08:53:283
2627199350,19cyclictest2013-21seq07:27:223
239462350,2chrt0-21swapper/110:54:181
26271993413,18cyclictest11149-21find07:50:103
2627199330,19cyclictest25979-21chrt08:23:113
2627199320,19cyclictest32196-21taskset07:21:073
2627199320,19cyclictest32196-21taskset07:21:063
299612310,8sleep228-21ksoftirqd/210:58:392
2627199312,25cyclictest22929-21needreboot08:15:203
26271993010,17cyclictest421-21ls08:40:133
26271993010,17cyclictest29834-21cat08:30:243
102592300,3sleep10-21swapper/109:30:541
87222290,2chrt0-21swapper/209:54:502
287962290,2sleep20-21swapper/210:09:442
177522290,9sleep122-21ksoftirqd/109:12:481
146082290,2sleep10-21swapper/109:58:591
3662280,4sleep228-21ksoftirqd/210:36:012
2627199289,17cyclictest27859-21sshd07:10:443
26271992811,14cyclictest7017-21cpuspeed_turbos07:40:113
26271992811,14cyclictest278-21systemd-journal08:40:003
26267992816,7cyclictest636-21runrttasks11:20:111
5402270,2sleep00-21swapper/011:25:110
2627199274,20cyclictest8010-21pmu-power08:55:193
2627199274,20cyclictest17962-21cstates08:05:133
2627199271,23cyclictest9201-21cpu09:00:133
26271992710,15cyclictest15449-21basename08:00:103
2627199270,1cyclictest0-21swapper/311:35:103
26267992710,11cyclictest53050irq/142-enp3s0-12:00:111
2627199265,18cyclictest13614-21expr07:55:143
2627199261,22cyclictest28464-21ls08:30:003
2627199260,23cyclictest11469-21grep09:05:133
26267992624,1cyclictest0-21swapper/107:11:421
26266992624,1cyclictest0-21swapper/007:11:210
126452260,2chrt0-21swapper/009:05:240
2627199251,21cyclictest28562-21grep07:15:143
2627199251,21cyclictest2671-21grep08:45:133
2627199251,21cyclictest2474-21find07:30:113
2627199251,21cyclictest11015-21latency_hist07:50:013
26270992522,2cyclictest0-21swapper/207:11:572
2626799252,16cyclictest27147-21ssh09:20:131
149852250,11sleep122-21ksoftirqd/111:35:191
308432240,2sleep00-21swapper/009:22:270
287222240,2sleep20-21swapper/212:34:192
2626799243,14cyclictest3735-21ssh10:15:111
2626799243,13cyclictest21662-21ssh09:40:101
2626799241,18cyclictest9482-21kworker/u8:012:35:091
26266992412,10cyclictest31380-21latency08:35:180
2626699241,17cyclictest15815-21apt-get12:25:050
75622230,11sleep122-21ksoftirqd/112:18:291
26270992313,7cyclictest18687-21ntp_kernel_err08:05:202
2626799239,8cyclictest11581-21ssh10:45:041
2626699234,16cyclictest8877-21grep11:30:250
26266992314,7cyclictest32495-21seq07:24:580
26266992314,7cyclictest32495-21seq07:24:580
26266992314,7cyclictest29224-21memory07:15:200
34012220,2sleep10-21swapper/111:50:281
2627099222,17cyclictest23340-21unixbench_singl09:40:252
26270992212,7cyclictest27392-21apt-get11:45:112
26270992210,9cyclictest26187-21sh11:44:132
2626799229,10cyclictest22-21ksoftirqd/109:40:001
2626799223,13cyclictest2577-21apt-get09:50:121
26267992213,5cyclictest13231-21diskmemload12:10:131
2626699226,2cyclictest141rcu_preempt09:39:520
2626699225,3cyclictest141rcu_preempt10:32:430
2626699223,17cyclictest17536-21fschecks_count10:25:130
2626699222,14cyclictest2589-21ssh09:50:110
26266992215,5cyclictest9132-21sh09:55:020
26266992213,7cyclictest14030-21entropy09:10:140
26266992212,7cyclictest7316-21sensors_temp11:05:230
2626699221,18cyclictest5627-21cat08:50:180
272952210,1chrt27271-21ssh11:20:192
2627099216,12cyclictest28368-21runrttasks09:45:122
2627099216,12cyclictest28368-21runrttasks09:45:112
2627099213,6cyclictest28-21ksoftirqd/209:57:362
2627099212,2cyclictest141rcu_preempt10:10:232
2627099212,13cyclictest28563-21ssh12:10:122
26270992113,6cyclictest4388-21ls12:15:262
26270992113,6cyclictest20568-21ls11:15:202
26270992113,6cyclictest20568-21ls11:15:202
26270992112,6cyclictest10869-21seq07:49:402
2626799213,11cyclictest23818-21ssh10:30:111
26267992113,6cyclictest28930-21fw_packets07:15:161
2626799211,14cyclictest16105-21ssh10:00:131
26267992110,8cyclictest1-21systemd08:19:591
2626699219,8cyclictest1349-21apt-get09:25:110
2626699213,16cyclictest11972-21ssh10:20:240
2626699211,9cyclictest12255-21apt-get11:10:010
26266992112,7cyclictest10076-21perl12:20:160
26266992112,6cyclictest8784-21seq08:58:260
26266992112,6cyclictest22036-21cat09:40:140
26266992111,7cyclictest26986-21cat08:25:170
2626699210,6cyclictest2232-21turbostat07:30:000
123292210,3chrt12-21ksoftirqd/011:57:450
101392210,2sleep1211rcuc/112:20:181
57922200,1sleep10-21swapper/109:28:111
2627099203,4cyclictest28-21ksoftirqd/209:00:202
2627099203,14cyclictest5914-21proc_pri08:50:202
2627099202,13cyclictest640-21snmpd11:05:122
2627099201,5cyclictest601-21apt-get11:25:122
26270992013,4cyclictest4639-21sh11:04:052
26270992012,6cyclictest2399-21sed07:30:002
26270992012,5cyclictest52850irq/140-enp3s0-09:35:112
26270992012,5cyclictest29877-21grep07:15:262
26270992011,6cyclictest22633-21sed08:15:172
2627099200,2cyclictest141rcu_preempt09:11:332
26267992014,4cyclictest22-21ksoftirqd/109:05:001
26267992013,5cyclictest22294-21seq11:16:241
26267992013,5cyclictest22294-21seq11:16:231
26267992012,6cyclictest560-21dbus-daemon09:20:011
26267992012,6cyclictest29721-21sed08:30:231
26267992012,6cyclictest27291-21ntpq08:25:201
26267992012,6cyclictest16484-21seq10:24:461
26267992011,7cyclictest25075-21apt-get10:55:111
26267992011,6cyclictest9803-21seq11:55:251
2626799200,9cyclictest2195-21ls09:50:001
2626799200,9cyclictest2195-21ls09:49:591
2626699208,8cyclictest2488-21apt-get07:30:140
2626699207,4cyclictest52950irq/141-enp3s0-12:15:240
2626699205,3cyclictest141rcu_preempt10:00:250
2626699204,14cyclictest16920-21unixbench_singl08:00:240
2626699202,13cyclictest75-21kswapd010:55:110
2626699201,7cyclictest23074-21awk12:30:080
26266992014,4cyclictest6804-21sh10:40:410
26266992013,4cyclictest21627-21rt-features09:15:220
26266992013,4cyclictest19639-21rt-features10:50:210
26266992012,6cyclictest31872-21seq11:48:140
26266992012,6cyclictest29884-21cat08:30:240
26266992012,6cyclictest29776-21seq11:22:100
26266992012,5cyclictest1199-21seq12:13:510
26266992011,7cyclictest11041-21seq09:04:410
2626699201,16cyclictest17807-21apt-get08:05:120
158152200,3sleep228-21ksoftirqd/208:00:152
2627099199,7cyclictest7596-21cat11:30:142
2627099197,3cyclictest28-21ksoftirqd/211:10:272
2627099193,4cyclictest141rcu_preempt12:35:162
2627099193,3cyclictest13231-21diskmemload11:36:422
2627099193,3cyclictest13231-21diskmemload10:42:312
2627099192,3cyclictest13231-21diskmemload10:18:522
2627099192,12cyclictest366-21apt-get08:40:122
2627099191,4cyclictest16410-21apt-get12:25:102
26270991913,4cyclictest34872sleep209:25:502
26270991913,4cyclictest16234-21sendmail_mailqu12:00:222
26270991912,5cyclictest3296-21awk11:50:252
26270991912,5cyclictest11561-21tune2fs07:50:152
26270991911,6cyclictest7992-21tr07:40:212
26270991911,6cyclictest13580-21expr07:55:132
26270991911,5cyclictest10580-21grep10:20:122
2626799194,5cyclictest636-21runrttasks11:02:081
2626799193,9cyclictest16289-21apt-get12:25:051
2626799191,7cyclictest29485-21/usr/sbin/munin10:10:171
26267991915,2cyclictest172292sleep110:25:121
26267991912,5cyclictest11265-21seq11:32:401
26267991912,4cyclictest13231-21diskmemload11:11:531
26267991911,6cyclictest32736-21seq10:35:551
26267991911,6cyclictest3130-21idleruntime08:45:171
26267991911,5cyclictest13231-21diskmemload10:05:171
2626799191,15cyclictest0-21swapper/107:45:201
26267991910,6cyclictest53050irq/142-enp3s0-11:40:121
26267991910,6cyclictest32499-21cron07:25:001
26267991910,6cyclictest32499-21cron07:24:591
26267991910,6cyclictest278-21systemd-journal08:05:001
2626699198,9cyclictest52950irq/141-enp3s0-10:06:160
2626699197,4cyclictest52950irq/141-enp3s0-10:45:120
2626699193,14cyclictest22288-21grep12:05:160
2626699192,3cyclictest141rcu_preempt11:00:240
2626699192,3cyclictest13231-21diskmemload10:36:260
2626699192,12cyclictest4624-21apt-get07:35:130
26266991914,3cyclictest16017-21ntp_offset12:00:200
26266991912,5cyclictest7813-21open_files07:40:200
26266991912,5cyclictest15090-21seq07:58:560
26266991911,6cyclictest2776-21seq10:14:280
26266991911,6cyclictest23039-21ls08:15:220
26266991911,6cyclictest21643-21seq08:12:440
26266991911,5cyclictest3527-21ls08:45:220
26266991910,6cyclictest9847-21expr07:45:190
26266991910,6cyclictest11485-21df07:50:120
26266991910,5cyclictest13231-21diskmemload11:50:120
114102190,2sleep10-21swapper/109:05:141
2627099188,7cyclictest17118-21apt-get10:25:112
2627099184,2cyclictest141rcu_preempt10:30:112
2627099183,2cyclictest141rcu_preempt09:30:142
2627099182,14cyclictest20637-21latency08:10:162
26270991813,3cyclictest30670-21apt08:35:112
26270991811,5cyclictest5833-21cat07:35:232
26270991811,5cyclictest28136-21ntp_kernel_err09:20:212
26270991811,4cyclictest13231-21diskmemload11:57:362
26270991811,4cyclictest13231-21diskmemload10:00:332
26270991810,6cyclictest4847-21packagekitd08:55:062
26270991810,5cyclictest3814-21cat08:45:252
26270991810,5cyclictest27059-21sed08:25:182
2626799189,7cyclictest14060-21memory07:55:191
2626799189,6cyclictest5778-21seq10:40:191
2626799189,6cyclictest53050irq/142-enp3s0-12:05:121
2626799182,13cyclictest875-21grep07:25:161
26267991813,3cyclictest7085-21open_inodes11:05:211
26267991811,5cyclictest7050-21sed11:30:011
26267991810,6cyclictest20600-21cut08:10:151
26267991810,5cyclictest6841-21date07:40:011
2626699189,6cyclictest20750-21cpu11:40:130
2626699187,3cyclictest52950irq/141-enp3s0-10:15:060
26266991813,3cyclictest28319-21/usr/sbin/munin09:45:200
26266991813,3cyclictest28319-21/usr/sbin/munin09:45:200
26266991810,6cyclictest24601-21grep08:20:160
2627099173,2cyclictest141rcu_preempt10:50:102
2627099172,11cyclictest11336-21apt-get09:05:142
26270991712,3cyclictest28507-21/usr/sbin/munin08:30:092
2627099171,13cyclictest3166-21cut07:30:192
26270991710,5cyclictest24849-21cat08:20:162
2626799179,6cyclictest18953-21mailstats08:05:221
2626799178,4cyclictest0-21swapper/108:55:111
26267991712,3cyclictest339-21/usr/sbin/munin08:40:111
26267991712,3cyclictest27598-21cstates11:45:141
26267991711,3cyclictest7845-21cut07:40:211
2626799171,11cyclictest11163-21apt-get07:50:121
26267991710,5cyclictest31805-21tr08:35:231
2626799169,5cyclictest24256-21grep08:20:131
2626799168,6cyclictest2927-21grep07:30:161
26267991610,4cyclictest5216-21expr08:50:131
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional