You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-06 - 00:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Aug 05, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1870999204199,3cyclictest986-21lspci10:55:093
2324521690,1sleep10-21swapper/112:00:031
18119216696,63sleep20-21swapper/207:05:502
182702144106,26sleep30-21swapper/307:07:513
181282140104,25sleep10-21swapper/107:05:571
182522135100,25sleep00-21swapper/007:07:360
156482770,4sleep3331rcuc/310:17:153
4232710,2sleep10-21swapper/109:40:001
248022640,3sleep024675-21cstates12:25:130
85062620,2sleep30-21swapper/310:11:453
276312600,1sleep30-21swapper/310:01:583
42662580,2sleep30-21swapper/311:45:193
42662580,2sleep30-21swapper/311:45:193
214322580,2sleep30-21swapper/312:22:193
280952570,2sleep334-21ksoftirqd/309:35:173
205912520,2sleep20-21swapper/209:56:362
10212520,2chrt0-21swapper/110:55:091
131362510,1sleep00-21swapper/012:15:380
272272490,1sleep00-21swapper/012:02:340
63572480,4chrt6356-21ssh10:35:141
93212460,2chrt0-21swapper/311:00:213
93212460,2chrt0-21swapper/311:00:213
176282460,2chrt0-21swapper/111:30:321
233682450,2chrt0-21swapper/309:59:253
230032450,2sleep20-21swapper/211:10:232
136962440,1sleep20-21swapper/210:15:262
38062420,1sleep30-21swapper/310:08:443
55882370,3chrt131rcuc/010:57:450
237092370,3sleep228-21ksoftirqd/209:31:032
312722340,2sleep20-21swapper/211:16:452
254632330,2sleep30-21swapper/311:36:443
96472310,2chrt0-21swapper/310:37:053
70192310,2sleep10-21swapper/111:47:221
70192310,2sleep10-21swapper/111:47:221
147982280,8sleep122-21ksoftirqd/109:22:271
18708992712,12cyclictest14960-21apt-get11:05:132
18707992716,6cyclictest0-21swapper/112:10:041
228262260,2sleep00-21swapper/009:30:080
18708992618,6cyclictest30663-21ssh10:53:072
1870799261,24cyclictest0-21swapper/107:11:481
1870699261,24cyclictest0-21swapper/007:11:460
103532260,3chrt0-21swapper/210:13:362
18709992517,6cyclictest20220-21ssh09:27:433
1870899251,1cyclictest0-21swapper/207:11:532
1870799252,16cyclictest19651-21ssh10:45:121
18707992511,9cyclictest53050irq/142-enp3s0-10:15:121
217202240,3sleep10-21swapper/107:35:011
18709992422,1cyclictest0-21swapper/307:12:053
18709992411,4cyclictest12336-21ls12:15:233
18708992416,6cyclictest22244-21ssh10:46:192
18707992413,7cyclictest12595-21grep12:15:251
1870699248,4cyclictest267212sh11:38:010
18706992413,9cyclictest25196-21apt-key08:00:000
18706992413,9cyclictest25196-21apt-key08:00:000
142312240,2sleep00-21swapper/011:28:510
18709992315,6cyclictest31874-21ssh10:54:223
1870699232,14cyclictest2525-21ssh11:20:120
18706992314,7cyclictest6010-21fw_packets10:10:150
104752230,8sleep228-21ksoftirqd/212:14:442
18709992216,4cyclictest27952-21munin-plugin-st08:20:003
1870699220,10cyclictest66050lldpd10:00:560
1870999212,13cyclictest11016-21ssh09:50:083
1870899217,12cyclictest550-21systemd-logind10:55:492
1870899216,6cyclictest28-21ksoftirqd/210:25:002
18708992114,5cyclictest2507-21sh09:10:272
18708992113,6cyclictest31315-21seq08:46:102
1870899211,16cyclictest636-21runrttasks11:35:002
18708992110,2cyclictest29087-21/usr/sbin/munin11:40:232
18707992115,4cyclictest22-21ksoftirqd/112:30:001
18707992113,6cyclictest15469-21sed11:30:021
18707992112,7cyclictest31702-21head12:30:171
18707992112,7cyclictest31702-21head12:30:161
18707992111,8cyclictest3388-21ssh11:20:191
1870699217,10cyclictest25455-21ssh10:25:120
1870699211,9cyclictest3411-21ssh11:45:130
1870699211,9cyclictest3411-21ssh11:45:130
18706992113,6cyclictest6416-21ntp_kernel_pll_12:35:210
18706992111,7cyclictest19098-21python309:55:260
18706992110,7cyclictest52750irq/139-enp3s0-12:30:120
18706992110,7cyclictest52750irq/139-enp3s0-12:30:120
18709992015,3cyclictest34-21ksoftirqd/312:15:003
18709992014,4cyclictest13734-21kernelversion10:40:193
18709992011,6cyclictest1243-21seq09:40:443
18709992010,7cyclictest22635-21ntp_offset11:10:203
18708992014,4cyclictest16921-21sh12:19:322
18708992013,5cyclictest8891-21smartctl11:00:162
18708992013,5cyclictest8891-21smartctl11:00:162
18708992013,5cyclictest4901-21ls11:45:262
18708992013,5cyclictest4901-21ls11:45:252
18708992012,6cyclictest29770-21cut12:05:012
18708992011,7cyclictest560-21dbus-daemon12:00:012
18708992011,6cyclictest23694-21processes11:35:222
1870799209,7cyclictest0-21swapper/109:00:041
18707992012,6cyclictest31478-21ls12:05:241
18707992012,6cyclictest13428-21seq11:52:071
18707992010,7cyclictest25866-21tune2fs10:25:151
1870699201,5cyclictest0-21swapper/009:17:330
18706992015,3cyclictest22969-21cron12:00:000
18706992012,6cyclictest15661-21if_err_enp3s011:05:180
75502190,1chrt0-21swapper/009:46:540
300272190,2sleep00-21swapper/012:05:120
296422190,2chrt0-21swapper/011:15:270
187852190,1chrt0-21swapper/210:44:342
1870999191,8cyclictest2966-21awk11:45:003
1870999191,6cyclictest0-21swapper/311:05:033
18709991913,4cyclictest34-21ksoftirqd/309:00:003
18709991913,4cyclictest22258-21turbostat07:40:003
18709991912,4cyclictest1855-21diskmemload09:31:003
18709991911,6cyclictest5774-21df12:35:153
18709991911,6cyclictest28857-21grep11:15:183
18709991911,6cyclictest26129-21date08:05:013
18709991911,5cyclictest24810-21cat12:25:133
18709991910,7cyclictest560-21dbus-daemon12:10:003
18709991910,7cyclictest27525-21wc08:15:013
1870999190,5cyclictest0-21swapper/307:57:493
1870999190,5cyclictest0-21swapper/307:57:493
1870899192,12cyclictest6327-21apt-get10:35:152
18708991914,3cyclictest832-21sh09:40:182
18708991913,4cyclictest3922-21sh10:33:002
18708991913,4cyclictest28456-21sh10:26:442
18708991912,5cyclictest6763-21expr12:35:252
18708991912,4cyclictest10555-21sh09:18:232
18708991911,6cyclictest10516-21swap11:25:252
18708991911,5cyclictest19285-21timerandwakeup12:20:232
18708991910,7cyclictest17324-21grep09:55:012
1870799192,12cyclictest12960-21apt-get10:40:131
1870799192,11cyclictest22472-21apt-get11:35:131
1870799191,7cyclictest17720-21ntp_states11:55:191
1870799191,4cyclictest0-21swapper/111:15:321
18707991914,3cyclictest30707-21unixbench-2d11:40:241
18707991914,3cyclictest10637-21sh11:01:061
18707991914,3cyclictest10637-21sh11:01:051
18707991913,4cyclictest1855-21diskmemload09:47:071
18707991913,4cyclictest1855-21diskmemload09:16:001
18707991913,3cyclictest1855-21diskmemload09:56:051
18707991912,5cyclictest1855-21diskmemload09:51:241
18707991912,4cyclictest1855-21diskmemload11:12:201
18707991912,4cyclictest1855-21diskmemload09:43:171
18707991911,6cyclictest6700-21run-parts12:35:241
18707991911,6cyclictest6639-21which10:10:211
18707991911,6cyclictest32006-21seq10:30:081
18707991911,6cyclictest29741-21seq10:52:111
18707991911,6cyclictest24849-21seq09:32:131
18707991911,5cyclictest278-21systemd-journal09:30:001
1870699198,3cyclictest198572rm09:27:210
18706991913,4cyclictest22121-21gltestperf11:10:150
18706991913,4cyclictest17750-21sh12:20:060
18706991913,4cyclictest14885-21sh09:22:320
18706991913,4cyclictest14871-21sh10:40:370
18706991913,3cyclictest24700-21sh10:48:500
18706991912,5cyclictest28554-21sh10:50:590
18706991911,6cyclictest8805-21expr11:00:150
18706991911,6cyclictest8805-21expr11:00:150
18706991911,6cyclictest560-21dbus-daemon08:55:000
18706991911,6cyclictest23033-21seq07:44:310
18706991911,6cyclictest13002-21seq10:15:190
18706991911,6cyclictest10996-21expr11:50:180
1870999182,2cyclictest141rcu_preempt10:28:333
1870999181,4cyclictest0-21swapper/312:00:033
18709991813,3cyclictest21561-21sh11:58:343
18709991811,5cyclictest4688-21seq12:34:443
18709991811,5cyclictest4688-21seq12:34:443
18709991811,5cyclictest34-21ksoftirqd/308:10:003
18709991811,5cyclictest21050-21sed10:45:243
18709991811,5cyclictest11332-21which11:50:213
18709991810,6cyclictest560-21dbus-daemon08:35:003
1870899189,7cyclictest21533-21cron07:35:002
1870899186,9cyclictest29250-21sort08:30:002
1870899181,8cyclictest32622-21python310:05:252
18708991813,3cyclictest15901-21sh09:23:352
18708991812,4cyclictest9913-21sh11:50:042
18708991812,4cyclictest9523-21sh09:48:542
18708991812,4cyclictest6623-21sh11:22:522
18708991812,4cyclictest31271-21sh09:38:312
18708991812,4cyclictest2331-21sh12:32:222
18708991812,4cyclictest2331-21sh12:32:212
18708991811,5cyclictest560-21dbus-daemon10:05:002
18708991811,5cyclictest25710-21gzip12:25:232
18708991811,5cyclictest17848-21ssh09:25:202
18708991810,6cyclictest30237-21expr12:05:142
18708991810,6cyclictest23257-21cut07:45:002
1870799187,8cyclictest278-21systemd-journal07:45:051
1870799182,11cyclictest17837-21apt-get12:20:131
18707991813,3cyclictest29796-21latency_hist08:35:001
18707991812,4cyclictest835-21sh10:05:421
18707991811,5cyclictest3216-21seq09:11:101
18707991810,6cyclictest25792-21wc10:00:241
18707991810,6cyclictest20486-21cut07:25:011
1870699189,7cyclictest6828-21tune2fs10:35:170
1870699189,6cyclictest560-21dbus-daemon08:50:000
1870699189,6cyclictest30692-21sed11:40:240
1870699189,6cyclictest15925-21grep11:30:130
1870699182,11cyclictest31286-21apt-get10:05:140
18706991813,3cyclictest8193-21sh12:12:260
18706991813,3cyclictest4706-21sh09:44:140
18706991813,3cyclictest3572-21sh09:11:320
18706991813,3cyclictest28984-21sh09:36:120
18706991813,3cyclictest2807-21sh10:31:520
18706991812,4cyclictest12841-21sh09:50:290
18706991811,5cyclictest27129-21seq08:12:200
18706991811,5cyclictest21394-21seq07:33:110
18706991811,5cyclictest19381-21expr10:20:160
18706991810,6cyclictest1460-21seq09:05:040
1870999179,6cyclictest32299-21date08:55:003
1870999179,6cyclictest31733-21date08:50:003
1870999179,6cyclictest31157-21grep08:45:003
1870999179,6cyclictest31157-21grep08:45:003
1870999179,6cyclictest20941-21pluginstate07:30:013
1870999179,5cyclictest20098-21mailstats10:20:233
1870999176,9cyclictest0-21swapper/310:30:453
1870999174,3cyclictest1855-21diskmemload11:30:563
1870999171,7cyclictest2339-21ssh11:20:033
18709991713,2cyclictest9136-21/usr/sbin/munin11:25:213
18709991712,3cyclictest9742-21sh09:49:073
18709991712,3cyclictest7399-21sh09:15:123
18709991712,3cyclictest2176-21sh09:10:073
18709991712,3cyclictest12924-21sh09:20:323
18709991712,3cyclictest1235-21turbostat.cron09:05:003
18709991710,5cyclictest22968-21seq07:43:393
18708991712,3cyclictest1238-21idleruntime-cro09:05:002
1870799179,6cyclictest20775-21seq07:28:021
1870799179,6cyclictest16039-21awk11:05:231
1870799172,11cyclictest26537-21apt-get08:10:041
18707991712,3cyclictest636-21runrttasks08:18:541
18707991712,3cyclictest31620-21latency_hist08:50:001
18707991711,4cyclictest22-21ksoftirqd/108:10:001
18707991711,3cyclictest1855-21diskmemload10:20:071
18707991710,5cyclictest30502-21date08:40:001
18707991710,5cyclictest22217-21seq07:39:451
1870699179,6cyclictest560-21dbus-daemon07:50:000
1870699178,6cyclictest354-21cron09:00:000
1870699171,7cyclictest19717-21munin-run07:20:000
18706991710,5cyclictest29894-21ls08:35:000
18706991710,5cyclictest28888-21seq08:26:200
18706991710,5cyclictest21916-21seq07:37:090
18706991710,5cyclictest1152-21seq09:04:040
1870999169,5cyclictest278-21systemd-journal08:40:003
1870999168,6cyclictest21611-21grep07:35:013
1870999161,7cyclictest640-21snmpd07:16:223
18709991610,4cyclictest23695-21munin-plugin-st07:50:003
1870899169,5cyclictest31160-21ls08:45:002
1870899169,5cyclictest31160-21ls08:45:002
1870899169,5cyclictest27516-21cat08:15:012
1870899168,6cyclictest28308-21seq08:21:392
1870899168,5cyclictest278-21systemd-journal07:30:002
18708991611,3cyclictest24289-21latency_hist07:55:002
18708991610,4cyclictest1841-21sh09:10:012
1870799169,5cyclictest31043-21grep08:45:001
1870799169,5cyclictest31043-21grep08:45:001
1870799169,5cyclictest23242-21sed07:45:001
1870799168,6cyclictest28268-21seq08:21:061
18707991612,2cyclictest1354-21cron09:10:001
1870799161,13cyclictest380-21perf09:00:001
1870799161,10cyclictest24715-21apt-get07:55:041
1870799161,10cyclictest24715-21apt-get07:55:041
18707991610,4cyclictest32212-21cron08:55:001
1870699169,5cyclictest26330-21seq08:07:300
1870699168,6cyclictest30345-21seq08:39:360
18706991611,3cyclictest636-21runrttasks08:21:040
18706991611,3cyclictest636-21runrttasks08:15:070
18706991611,3cyclictest31081-21grep08:45:000
18706991611,3cyclictest31081-21grep08:45:000
18706991610,4cyclictest25432-21gpgv08:00:010
1870999159,4cyclictest0-21swapper/307:55:013
1870999157,5cyclictest278-21systemd-journal08:20:013
1870999150,2cyclictest201482chrt07:22:433
1870899159,4cyclictest636-21runrttasks08:57:392
1870899159,4cyclictest0-21swapper/207:20:002
1870899158,5cyclictest27694-21seq08:16:492
1870899158,5cyclictest25178-21sed08:00:002
1870899158,5cyclictest25178-21sed08:00:002
1870899158,5cyclictest20410-21cut07:25:012
1870899156,7cyclictest26497-21seq08:09:432
18708991511,3cyclictest29788-21dump-pmu-power08:39:552
18708991510,3cyclictest29621-21sshd08:33:022
1870799158,5cyclictest25777-21seq08:03:091
1870799156,6cyclictest278-21systemd-journal08:30:001
18707991510,3cyclictest19724-21grep07:20:001
1870699157,6cyclictest20216-21seq07:23:360
18706991510,3cyclictest636-21runrttasks07:25:150
1870999149,3cyclictest1845-21latency_hist09:10:013
1870999145,7cyclictest29182-21awk08:30:013
1870899149,3cyclictest636-21runrttasks08:50:322
1870899149,3cyclictest23779-21latency_hist07:50:002
18708991412,1cyclictest0-21swapper/207:36:342
1870899141,1cyclictest0-21swapper/208:03:122
1870799149,3cyclictest636-21runrttasks07:51:381
1870699148,4cyclictest24444-21cut07:55:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional