You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-10 - 14:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun May 10, 2026 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14048221698,97sleep20-21swapper/207:07:032
14076217298,25sleep00-21swapper/007:07:240
14191217197,25sleep10-21swapper/107:08:541
123682157100,31sleep30-21swapper/307:05:033
130772730,1chrt13078-21/usr/sbin/munin08:20:172
98562700,1sleep00-21swapper/009:15:240
109952700,2sleep30-21swapper/312:04:183
30572660,10sleep012-21ksoftirqd/009:58:580
297962600,2sleep30-21swapper/309:00:143
302582590,1sleep30-21swapper/311:54:513
217822560,2sleep10-21swapper/110:12:041
207592540,1sleep20-21swapper/209:24:132
35952530,1sleep30-21swapper/311:58:523
95772520,1sleep20-21swapper/212:26:362
323042500,1sleep30-21swapper/309:32:163
227962490,9sleep334-21ksoftirqd/311:49:053
49132480,2sleep20-21swapper/209:11:532
37782480,1sleep30-21swapper/309:59:413
262022470,2sleep30-21swapper/311:03:423
252512470,2sleep00-21swapper/011:26:030
241662470,2sleep30-21swapper/308:45:203
294972460,2sleep00-21swapper/011:30:040
170252460,2sleep317024-21ssh12:32:193
201382440,2sleep20-21swapper/209:47:162
193352440,2sleep10-21swapper/112:34:381
31462430,2chrt0-21swapper/212:21:552
31462430,2chrt0-21swapper/212:21:552
307452430,2chrt0-21swapper/110:42:451
259352430,1chrt25936-21/usr/sbin/munin08:50:153
250472430,2chrt0-21swapper/312:14:533
132492430,1sleep10-21swapper/112:05:201
176932410,1sleep20-21swapper/212:32:582
276152390,1sleep20-21swapper/210:16:102
320252380,1sleep10-21swapper/111:31:051
244562330,2sleep334-21ksoftirqd/312:38:023
86452320,3chrt0-21swapper/211:38:252
34342310,2chrt0-21swapper/210:46:082
85992300,2sleep30-21swapper/312:25:373
1455499300,4cyclictest34-21ksoftirqd/312:25:003
1455499300,4cyclictest34-21ksoftirqd/312:25:003
1455499299,9cyclictest34-21ksoftirqd/308:15:113
85532280,9sleep012-21ksoftirqd/011:38:190
33202280,2sleep30-21swapper/309:35:133
190142280,2sleep00-21swapper/011:45:250
1455499277,9cyclictest34-21ksoftirqd/312:09:333
1455499277,9cyclictest34-21ksoftirqd/311:11:023
1455499277,9cyclictest34-21ksoftirqd/310:50:283
1455499273,17cyclictest26724-21ssh10:40:123
1455499271,6cyclictest34-21ksoftirqd/311:20:203
1455199271,25cyclictest0-21swapper/007:13:020
304982260,1chrt0-21swapper/311:05:593
1455499266,9cyclictest34-21ksoftirqd/311:25:203
1455499260,7cyclictest14722-21ls10:30:263
1455299263,9cyclictest5373-21ssh12:00:101
1455299261,24cyclictest0-21swapper/107:13:291
257562250,7sleep2271rcuc/208:50:142
250102250,2sleep00-21swapper/011:50:190
1455499257,8cyclictest34-21ksoftirqd/309:25:303
1455499257,10cyclictest34-21ksoftirqd/307:40:083
1455499257,10cyclictest34-21ksoftirqd/307:40:083
1455499256,9cyclictest34-21ksoftirqd/311:34:363
1455499256,9cyclictest34-21ksoftirqd/309:40:223
1455499256,10cyclictest34-21ksoftirqd/310:45:433
1455499255,11cyclictest34-21ksoftirqd/309:24:053
1455499255,11cyclictest34-21ksoftirqd/308:28:353
1455499254,6cyclictest34-21ksoftirqd/311:15:233
14554992516,6cyclictest25238-21ssh10:38:593
1455499251,4cyclictest34-21ksoftirqd/307:55:013
1455499251,1cyclictest0-21swapper/307:13:183
1455499250,6cyclictest4687-21ls08:00:173
14553992514,6cyclictest24142-21sh11:50:112
1455399251,23cyclictest0-21swapper/207:12:562
301092240,8sleep012-21ksoftirqd/010:18:410
214692240,2sleep20-21swapper/210:11:442
1455499248,9cyclictest34-21ksoftirqd/309:54:173
1455499248,8cyclictest34-21ksoftirqd/312:15:423
1455499245,8cyclictest34-21ksoftirqd/310:56:303
1455499244,11cyclictest34-21ksoftirqd/310:28:383
1455499243,4cyclictest34-21ksoftirqd/309:45:183
14554992415,7cyclictest34-21ksoftirqd/310:09:593
1455499241,11cyclictest34-21ksoftirqd/310:19:513
14554992410,4cyclictest34-21ksoftirqd/309:16:483
14552992414,7cyclictest19226-21df_inode07:20:141
14552992414,7cyclictest13173-21sort12:30:011
1455199244,11cyclictest6210-21ssh10:25:120
1455199243,14cyclictest16181-21ssh11:20:110
267992230,2sleep2271rcuc/207:35:262
1455499236,8cyclictest34-21ksoftirqd/310:00:363
14554992310,4cyclictest34-21ksoftirqd/310:10:013
1455499230,7cyclictest34-21ksoftirqd/308:45:003
1455499230,7cyclictest11743-21/usr/sbin/munin11:40:213
1455499230,4cyclictest34-21ksoftirqd/309:10:233
1455499230,13cyclictest34-21ksoftirqd/308:10:233
1455299231,20cyclictest19451-21gltestperf12:10:141
1455199239,5cyclictest12-21ksoftirqd/007:40:140
1455199239,5cyclictest12-21ksoftirqd/007:40:130
14551992316,4cyclictest20826-21turbostat.cron08:40:000
14551992313,7cyclictest13514-21switchtime12:05:230
1455499224,4cyclictest34-21ksoftirqd/311:37:183
1455499221,7cyclictest34-21ksoftirqd/308:20:193
1455499221,7cyclictest34-21ksoftirqd/307:20:153
14554992216,4cyclictest34-21ksoftirqd/307:40:003
1455499221,4cyclictest34-21ksoftirqd/310:20:183
1455399227,7cyclictest0-21swapper/211:23:272
14553992213,6cyclictest4485-21grep09:35:242
14553992213,6cyclictest27982-21fw_packets08:55:172
14553992210,8cyclictest26348-21kernelversion12:15:192
1455299227,4cyclictest66050lldpd08:02:301
14552992214,6cyclictest21719-21gltestperf11:00:141
14552992213,7cyclictest26753-21ls08:50:231
14552992213,6cyclictest26467-21sensors_temp07:35:241
1455199222,14cyclictest4590-21apt-get10:00:030
1455199221,5cyclictest0-21swapper/009:38:090
288672210,4chrt28872-21cpuspeed_turbos09:30:130
1455499216,8cyclictest34-21ksoftirqd/308:55:433
1455499215,8cyclictest34-21ksoftirqd/307:30:123
1455499214,9cyclictest34-21ksoftirqd/308:34:113
1455499214,9cyclictest34-21ksoftirqd/308:34:103
1455499213,5cyclictest34-21ksoftirqd/309:05:203
1455499212,5cyclictest34-21ksoftirqd/307:45:203
14554992115,4cyclictest34-21ksoftirqd/308:10:013
14554992115,4cyclictest34-21ksoftirqd/307:55:003
14554992111,2cyclictest141rcu_preempt07:19:523
1455499210,6cyclictest18946-21apt-get08:35:123
1455399214,3cyclictest141rcu_preempt11:00:362
14553992114,5cyclictest28-21ksoftirqd/210:10:002
14553992113,6cyclictest5775-21ntp_kernel_pll_10:00:202
14553992112,6cyclictest4293-21seq11:10:302
1455299219,4cyclictest91612sh10:03:071
14552992112,5cyclictest1285-21diskmemload11:15:041
14552992111,7cyclictest303-21ntpq07:50:241
14552992110,8cyclictest23754-21grep08:45:161
1455299210,9cyclictest107142rm10:28:141
1455199214,7cyclictest636-21runrttasks12:17:260
1455199213,13cyclictest28202-21apt-get11:05:120
14551992113,6cyclictest32655-21sed07:50:210
14551992111,7cyclictest19300-21fschecks_time12:10:130
115022210,2chrt0-21swapper/010:05:120
196342200,4sleep219632-21cat10:35:002
153752200,5sleep015381-21cpuspeed_turbos09:20:140
14554992012,5cyclictest21740-21sed07:25:183
1455399208,4cyclictest141rcu_preempt11:30:022
1455399208,4cyclictest141rcu_preempt07:45:002
1455399208,4cyclictest141rcu_preempt07:45:002
1455399205,4cyclictest28-21ksoftirqd/209:59:522
1455399202,2cyclictest141rcu_preempt09:25:572
14553992015,3cyclictest53050irq/142-enp3s0-09:20:002
1455399201,3cyclictest141rcu_preempt09:35:002
14553992013,5cyclictest18178-21ls08:30:242
14553992013,5cyclictest18178-21ls08:30:242
14553992012,6cyclictest21067-21ls12:35:182
14553992012,6cyclictest20990-21ls10:35:202
14553992011,6cyclictest887-21threads10:20:232
1455399200,6cyclictest19327-21tune2fs07:20:152
1455299209,7cyclictest12909-21gdbus10:30:011
1455299202,13cyclictest22909-21apt-get11:25:131
14552992015,4cyclictest22-21ksoftirqd/111:55:001
14552992012,6cyclictest28037-21fw_packets07:40:151
14552992012,6cyclictest28037-21fw_packets07:40:151
14552992012,6cyclictest15357-21ls10:55:201
14552992011,7cyclictest560-21dbus-daemon08:20:001
14552992011,7cyclictest3484-21seq07:57:521
14552992011,7cyclictest26469-21memory12:15:181
14552992011,6cyclictest23221-21find09:50:011
1455299200,17cyclictest52850irq/140-enp3s0-09:40:141
1455199201,6cyclictest21180-21apt-get11:00:110
14551992015,3cyclictest9243-21timerwakeupswit10:50:230
14551992015,3cyclictest17020-21sh11:45:000
14551992013,5cyclictest4746-21sh12:00:000
14551992013,4cyclictest636-21runrttasks10:48:330
14551992012,6cyclictest18217-21awk07:15:230
14551992011,6cyclictest9610-21grep08:10:250
14551992011,6cyclictest560-21dbus-daemon12:05:000
1455199201,15cyclictest13371-21/usr/sbin/munin12:30:110
144522200,2sleep00-21swapper/008:25:000
107122200,1sleep20-21swapper/210:28:142
1455399199,7cyclictest18198-21cut11:45:192
1455399198,2cyclictest28-21ksoftirqd/208:05:012
1455399192,12cyclictest8808-21apt-get11:15:042
1455399191,4cyclictest27445-21irqstats10:40:182
14553991914,3cyclictest13926-21sh11:41:562
14553991913,4cyclictest14581-21sh12:06:072
14553991912,4cyclictest23509-21sh07:30:142
14553991911,6cyclictest7968-21ls10:50:122
14553991911,6cyclictest5837-21tune2fs12:00:132
14553991911,6cyclictest32138-21run-parts11:55:222
14553991911,6cyclictest24798-21sensors_temp09:50:212
14553991910,6cyclictest11140-21sed08:15:202
1455299199,7cyclictest16538-21df_inode11:20:131
1455299198,10cyclictest5542-21ssh09:36:111
1455299191,9cyclictest26128-21rm10:39:531
14552991913,4cyclictest26859-21sh10:15:251
14552991913,4cyclictest2648-21sh12:21:251
14552991913,4cyclictest2648-21sh12:21:251
14552991913,4cyclictest24175-21proc_pri07:30:211
14552991913,4cyclictest21110-21sh09:24:341
14552991912,5cyclictest31945-21pluginstate11:55:211
14552991912,5cyclictest17567-21tr11:45:131
14552991912,4cyclictest25375-21sh09:27:061
14552991911,6cyclictest8363-21expr08:10:141
14552991911,6cyclictest560-21dbus-daemon10:54:591
14552991911,6cyclictest3348-21memory11:10:181
14552991911,6cyclictest28594-21gzip08:55:241
14552991911,6cyclictest16867-21cat08:30:131
14552991911,6cyclictest16867-21cat08:30:131
14552991911,6cyclictest12024-21sensors_temp11:40:231
1455199199,7cyclictest25379-21apt-get08:50:110
1455199199,3cyclictest52750irq/139-enp3s0-09:50:170
1455199198,8cyclictest10687-21df_inode08:15:150
1455199197,9cyclictest0-21swapper/009:45:010
1455199196,4cyclictest141rcu_preempt09:27:000
1455199194,11cyclictest23244-21apt-get08:45:120
1455199192,10cyclictest636-21runrttasks10:35:130
1455199191,8cyclictest16940-21/usr/sbin/munin09:45:140
1455199191,6cyclictest0-21swapper/012:25:000
1455199191,6cyclictest0-21swapper/012:25:000
1455199191,6cyclictest0-21swapper/011:14:390
1455199191,4cyclictest0-21swapper/010:20:550
14551991913,3cyclictest20754-21sendmail11:00:000
14551991912,5cyclictest2611-21grep07:55:190
14551991911,6cyclictest28120-21cat10:40:240
14551991911,6cyclictest21727-21grep07:25:170
14551991911,6cyclictest18823-21diskstats10:10:150
1455199191,14cyclictest6646-21apt-get12:25:120
14551991910,7cyclictest13775-21hwlatdetect10:30:170
1455399189,6cyclictest15507-21grep08:25:192
1455399182,4cyclictest28-21ksoftirqd/211:25:142
14553991813,3cyclictest9769-21sh09:40:112
14553991813,3cyclictest30952-21sh11:06:272
14553991813,3cyclictest29653-21/usr/sbin/munin09:00:182
14553991813,3cyclictest20938-21sh12:10:452
14553991811,5cyclictest6310-21expr08:05:132
14553991811,5cyclictest32741-21ntp_kernel_pll_07:50:232
14553991811,5cyclictest29881-21cat07:45:132
14553991811,5cyclictest21240-21expr07:25:132
14553991811,5cyclictest20911-21date08:40:002
14553991811,5cyclictest1265-21cron09:10:002
14553991810,6cyclictest8940-21ls08:10:182
14553991810,6cyclictest14995-21tune2fs10:55:172
1455299184,10cyclictest21441-21mailstats12:35:221
1455299182,11cyclictest75-21kswapd010:45:111
1455299182,11cyclictest29679-21apt-get09:00:141
1455299182,11cyclictest21036-21apt-get07:25:131
1455299182,11cyclictest1694-21apt-get09:10:111
14552991813,3cyclictest636-21runrttasks09:32:031
14552991813,3cyclictest31310-21sh11:06:481
14552991812,4cyclictest5669-21sh10:24:551
14552991812,4cyclictest30203-21sh09:55:111
14552991812,4cyclictest14018-21sh09:19:151
14552991811,5cyclictest21331-21cat08:40:131
14552991811,5cyclictest12681-21cat08:20:141
14552991811,5cyclictest11418-21seq10:05:081
14552991811,4cyclictest9931-21sh11:39:421
14552991810,6cyclictest20623-21seq08:37:321
14552991810,6cyclictest17966-21ls09:45:231
14552991810,6cyclictest14916-21cat08:25:141
1455199188,7cyclictest0-21swapper/007:30:210
1455199183,9cyclictest31816-21apt-get09:05:130
14551991811,5cyclictest27222-21seq07:39:090
14551991811,5cyclictest0-21swapper/011:15:160
14551991811,4cyclictest1285-21diskmemload12:38:160
1455399178,6cyclictest21611-21gltestperf08:40:162
14553991710,5cyclictest3727-21date08:00:012
14552991710,5cyclictest7483-21unixbench_multi08:05:241
14552991710,5cyclictest30872-21swap07:45:231
1455199179,5cyclictest17905-21readlink08:30:220
1455199179,5cyclictest17905-21readlink08:30:220
1455199178,6cyclictest30037-21fschecks_count09:00:160
1455199171,4cyclictest0-21swapper/008:00:120
14551991711,4cyclictest29409-21latency_hist09:00:010
1455199171,13cyclictest0-21swapper/007:45:120
14551991710,5cyclictest1684-21awk09:10:090
14551991710,5cyclictest16130-21awk08:25:260
14553991611,3cyclictest16711-21/usr/sbin/munin07:15:112
14553991610,4cyclictest23509-21expr08:45:132
1455299169,5cyclictest32080-21expr09:05:141
1455299169,5cyclictest16899-21cat07:15:131
1455199163,11cyclictest6593-21sed08:05:150
14551991612,2cyclictest21658-21hddtemp_smartct08:40:160
14551991611,3cyclictest19403-21gltestperf07:20:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional