You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-26 - 15:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Jun 26, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
218312206172,23sleep10-21swapper/107:09:431
217432157101,30sleep00-21swapper/007:08:350
215132142106,25sleep30-21swapper/307:05:363
215122140104,25sleep20-21swapper/207:05:352
2423321130,9sleep228-21ksoftirqd/210:56:202
71272910,2sleep20-21swapper/211:08:122
78782650,2chrt0-21swapper/209:32:172
76302640,2sleep00-21swapper/011:08:420
75602640,3sleep30-21swapper/311:55:493
41282630,2sleep30-21swapper/310:17:483
179492610,2sleep30-21swapper/310:04:283
179492610,2sleep30-21swapper/310:04:283
172312610,2sleep30-21swapper/312:27:223
293922590,1sleep10-21swapper/111:00:201
15282590,2sleep30-21swapper/307:35:203
265472580,2sleep10-21swapper/109:22:001
287632570,3sleep2271rcuc/212:35:282
319402560,2sleep10-21swapper/110:38:451
115812560,2sleep20-21swapper/209:35:132
43642520,1sleep00-21swapper/011:54:210
162962520,2sleep00-21swapper/010:02:490
162962520,2sleep00-21swapper/010:02:480
298062510,3sleep029793-21ssh11:25:110
292022510,1sleep30-21swapper/311:00:183
286452510,1sleep10-21swapper/111:24:201
172322510,3sleep317234-21grep11:15:203
76002500,2chrt0-21swapper/109:31:591
228772500,2sleep20-21swapper/208:25:152
170962500,2sleep30-21swapper/312:03:273
281822490,1sleep30-21swapper/310:11:063
271522490,3sleep30-21swapper/311:22:493
301512480,2sleep10-21swapper/112:36:531
105082480,1sleep00-21swapper/011:58:470
91022470,2sleep30-21swapper/309:33:303
40252470,3sleep10-21swapper/107:40:241
99302460,2sleep20-21swapper/212:21:492
140692450,1sleep00-21swapper/012:25:150
26462440,1sleep10-21swapper/110:40:231
76542430,1sleep20-21swapper/209:05:122
35262420,2chrt0-21swapper/210:17:112
278102400,1sleep00-21swapper/010:35:190
111092400,1sleep228-21ksoftirqd/210:23:052
323572390,13sleep012-21ksoftirqd/008:45:230
317562370,6chrt31757-21sh11:25:433
92102330,1sleep20-21swapper/209:57:292
2212599311,10cyclictest20066-21ssh11:17:300
22128993010,4cyclictest34-21ksoftirqd/310:05:203
22128993010,4cyclictest34-21ksoftirqd/310:05:203
215782300,2sleep30-21swapper/312:06:133
2212899289,5cyclictest34-21ksoftirqd/310:31:123
22128992821,5cyclictest34-21ksoftirqd/311:00:013
100062280,8sleep334-21ksoftirqd/312:21:533
2212899269,4cyclictest34-21ksoftirqd/310:27:203
2212899261,16cyclictest34-21ksoftirqd/312:14:583
22128992610,3cyclictest141rcu_preempt10:23:153
22125992624,1cyclictest0-21swapper/007:10:020
22125992617,6cyclictest29991-21dpkg09:25:110
2212599261,20cyclictest17573-21sh11:40:030
130732260,10sleep228-21ksoftirqd/210:48:382
284062250,2sleep00-21swapper/011:47:400
2212899259,4cyclictest34-21ksoftirqd/311:50:033
2212899258,10cyclictest34-21ksoftirqd/311:07:433
2212899257,4cyclictest34-21ksoftirqd/311:48:403
2212899256,11cyclictest34-21ksoftirqd/310:51:313
2212899256,11cyclictest34-21ksoftirqd/310:51:303
2212899253,13cyclictest34-21ksoftirqd/309:52:183
22128992518,5cyclictest34-21ksoftirqd/309:20:003
22128992510,4cyclictest34-21ksoftirqd/312:38:473
2212799259,9cyclictest52950irq/141-enp3s0-12:30:112
22127992518,5cyclictest17181-21sh10:51:012
22127992518,5cyclictest17181-21sh10:51:002
94402240,2sleep10-21swapper/110:21:241
27762240,2sleep30-21swapper/312:16:243
27762240,2sleep30-21swapper/312:16:233
2212899249,4cyclictest34-21ksoftirqd/311:40:113
2212899249,3cyclictest141rcu_preempt11:33:463
2212899248,4cyclictest34-21ksoftirqd/310:48:173
2212899248,2cyclictest141rcu_preempt10:40:493
2212899242,11cyclictest34-21ksoftirqd/309:48:513
22128992418,4cyclictest34-21ksoftirqd/311:40:013
22128992418,4cyclictest34-21ksoftirqd/308:20:003
22128992411,9cyclictest5447-21apt-get09:55:123
2212799240,9cyclictest264722sh09:21:562
2212699241,6cyclictest10892-21sh11:59:091
2212899239,7cyclictest141rcu_preempt09:27:173
2212899239,2cyclictest141rcu_preempt11:10:293
2212899238,2cyclictest141rcu_preempt12:30:223
2212899236,9cyclictest34-21ksoftirqd/309:11:463
2212899233,9cyclictest34-21ksoftirqd/309:21:383
22128992314,7cyclictest679-21cron08:50:003
22127992313,6cyclictest52950irq/141-enp3s0-10:05:112
22127992313,6cyclictest52950irq/141-enp3s0-10:05:112
2212699235,15cyclictest278-21systemd-journal10:05:011
2212699235,15cyclictest278-21systemd-journal10:05:001
22125992317,4cyclictest12339-21timerwakeupswit11:35:250
22125992314,7cyclictest5098-21ntpq11:30:210
247082220,2sleep20-21swapper/212:09:202
2212899229,2cyclictest141rcu_preempt07:15:213
2212899226,4cyclictest34-21ksoftirqd/309:42:313
22128992217,3cyclictest34-21ksoftirqd/307:25:003
22128992215,5cyclictest34-21ksoftirqd/308:40:003
2212799222,18cyclictest10911-21grep11:35:132
2212799221,9cyclictest17080-21apt-get11:40:042
22127992212,6cyclictest10245-21tail09:10:162
2212799221,15cyclictest9580-21ssh11:10:132
22125992212,7cyclictest20369-21cat08:20:120
2212599221,10cyclictest143662uname08:05:220
22125992210,3cyclictest1035-21hwlatdetect07:35:150
2212899217,2cyclictest141rcu_preempt07:45:253
2212899217,2cyclictest141rcu_preempt07:45:243
2212899215,2cyclictest141rcu_preempt10:36:113
22128992115,4cyclictest34-21ksoftirqd/309:40:013
22128992115,4cyclictest34-21ksoftirqd/308:10:013
22128992115,4cyclictest34-21ksoftirqd/307:35:003
22128992112,6cyclictest560-21dbus-daemon08:25:003
2212799219,3cyclictest28-21ksoftirqd/210:39:032
2212799211,4cyclictest26406-21sed12:10:152
22127992113,6cyclictest25218-21ntp_kernel_pll_07:15:242
22127992113,6cyclictest19049-21ntp_states08:15:202
2212699212,3cyclictest116952ssh12:23:341
22126992115,4cyclictest22-21ksoftirqd/111:30:001
22126992113,6cyclictest8270-21ntpq09:05:201
22126992113,6cyclictest28169-21ls08:35:241
22126992112,7cyclictest3487-21tune2fs08:55:161
22126992112,7cyclictest28902-21seq09:48:081
22126992111,7cyclictest12972-21cat10:00:191
22126992111,7cyclictest12972-21cat10:00:181
22126992110,8cyclictest22168-21apt-get07:10:131
2212599217,11cyclictest23938-21grep11:20:190
22125992113,6cyclictest8274-21sed07:50:230
22125992112,7cyclictest20738-21perl12:30:150
22125992112,6cyclictest26661-21grep12:10:180
22125992112,6cyclictest22895-21cat09:20:000
2212599211,18cyclictest5600-21fschecks_count09:00:140
22125992110,9cyclictest28539-21cstates11:00:130
61842200,8sleep06183-21/usr/sbin/munin09:55:190
2212899209,3cyclictest141rcu_preempt08:55:183
2212899207,2cyclictest141rcu_preempt09:00:173
2212899206,5cyclictest34-21ksoftirqd/308:00:193
22128992014,5cyclictest34-21ksoftirqd/307:15:003
22128992012,6cyclictest7717-21diskstats09:05:133
22128992011,6cyclictest28598-21cat07:25:093
22127992014,4cyclictest636-21runrttasks10:10:102
22127992012,6cyclictest9332-21expr07:55:142
22127992012,6cyclictest5246-21apt-get09:00:112
22127992012,6cyclictest17754-21apt-get09:40:122
22127992011,7cyclictest28980-21smartctl11:00:172
22127992011,6cyclictest24459-21awk11:45:102
22126992014,4cyclictest9974-21sh11:34:381
22126992012,6cyclictest6483-21ls09:00:231
22126992012,6cyclictest31398-21cat09:25:241
22126992011,7cyclictest29492-21ntpq07:25:181
22126992011,7cyclictest13347-21seq09:12:201
22126992011,3cyclictest9281-21diskmemload12:09:501
22126992010,7cyclictest22789-21cat08:25:151
2212599205,13cyclictest16072-21processes10:50:220
2212599205,13cyclictest16072-21processes10:50:220
22125992014,4cyclictest9939-21missed_timers07:55:200
22125992014,4cyclictest16538-21sh10:26:480
22125992014,4cyclictest12-21ksoftirqd/008:40:000
22125992012,6cyclictest922-21ls12:15:180
22125992012,6cyclictest922-21ls12:15:170
22125992012,6cyclictest8543-21python309:05:220
22125992011,7cyclictest29003-21cut07:25:140
22125992011,7cyclictest18662-21ls12:04:590
2212899192,11cyclictest7009-21apt-get07:50:103
22128991914,4cyclictest34-21ksoftirqd/308:30:003
22128991914,4cyclictest34-21ksoftirqd/308:15:013
22128991913,4cyclictest34-21ksoftirqd/308:35:003
22128991913,4cyclictest34-21ksoftirqd/308:35:003
2212799197,4cyclictest141rcu_preempt12:00:002
2212799192,12cyclictest29734-21ssh11:25:112
22127991914,3cyclictest3871-21cron09:30:002
22127991914,3cyclictest30979-21dump-pmu-power08:45:002
22127991914,3cyclictest21325-21processes08:20:212
22127991913,4cyclictest5464-21idleruntime07:45:192
22127991913,4cyclictest5464-21idleruntime07:45:182
22127991913,4cyclictest3629-21sh09:53:372
22127991912,5cyclictest2052-21run-parts08:50:232
22127991912,4cyclictest9281-21diskmemload10:40:492
22127991912,4cyclictest9281-21diskmemload09:19:192
22127991911,6cyclictest7912-21ntpq07:50:192
22127991911,6cyclictest15124-21seq12:01:272
2212799191,15cyclictest20870-21irqstats10:30:152
22127991910,6cyclictest871-21cut07:35:152
2212699199,7cyclictest8565-21forks10:45:161
2212699199,7cyclictest18981-21processes09:40:221
2212699192,12cyclictest32488-21apt-get12:15:121
2212699192,12cyclictest32488-21apt-get12:15:111
2212699192,12cyclictest13447-21apt-get08:05:131
2212699192,10cyclictest13418-21ssh10:25:101
22126991913,4cyclictest5921-21sh11:07:001
22126991913,4cyclictest53050irq/142-enp3s0-10:50:141
22126991913,4cyclictest53050irq/142-enp3s0-10:50:131
22126991913,4cyclictest29308-21sh11:48:331
22126991913,4cyclictest21616-21sh11:42:351
22126991912,5cyclictest30806-21seq08:42:591
22126991912,5cyclictest2835-21sh10:16:291
22126991912,5cyclictest16808-21seq09:39:291
22126991912,5cyclictest12040-21seq11:11:111
2212699191,16cyclictest32720-21cat09:50:241
22126991911,6cyclictest28407-21ssh12:11:181
22126991911,6cyclictest26619-21cat10:10:191
22126991911,6cyclictest17430-21seq08:12:011
22126991911,6cyclictest12773-21entropy12:00:141
2212699191,16cyclictest11722-21needreboot11:35:191
22126991911,5cyclictest5659-21sed07:45:201
22126991911,5cyclictest5659-21sed07:45:201
22126991910,7cyclictest22253-21date08:25:001
22126991910,6cyclictest16664-21entropy11:15:161
2212699190,16cyclictest20057-21cut12:30:011
2212599199,6cyclictest52750irq/139-enp3s0-12:20:130
2212599198,8cyclictest560-21dbus-daemon08:45:010
2212599198,8cyclictest12-21ksoftirqd/009:41:390
2212599192,12cyclictest636-21runrttasks08:55:120
2212599191,9cyclictest10624-21sh09:35:000
2212599191,6cyclictest17356-21munin-plugin-st09:40:000
2212599191,4cyclictest0-21swapper/010:24:120
22125991914,3cyclictest745-21sh09:50:430
22125991914,3cyclictest25613-21processes08:30:220
22125991914,3cyclictest25613-21processes08:30:220
22125991913,4cyclictest30919-21sh10:13:520
22125991912,5cyclictest24675-21seq10:56:460
22125991912,5cyclictest24375-21grep09:20:200
22125991912,4cyclictest9281-21diskmemload10:40:350
22125991911,6cyclictest1743-21seq10:15:250
22125991910,6cyclictest19887-21sed12:05:180
206672190,4sleep10-21swapper/112:30:131
2212899189,6cyclictest1017-21cat08:50:123
2212899187,3cyclictest141rcu_preempt07:57:393
2212899181,14cyclictest30420-21cat08:40:253
2212799187,9cyclictest26484-21awk08:35:002
2212799187,9cyclictest26484-21awk08:35:002
2212799182,12cyclictest75-21kswapd012:25:102
22127991813,3cyclictest31624-21fschecks_time11:50:132
22127991811,5cyclictest5631-21ls11:30:262
22127991811,5cyclictest27120-21cat08:35:132
22127991811,5cyclictest24366-21sensors11:20:242
22127991811,5cyclictest12446-21cat10:00:132
22127991811,5cyclictest12446-21cat10:00:132
22127991811,5cyclictest11498-21expr08:00:122
22127991811,4cyclictest9281-21diskmemload10:25:082
2212799181,14cyclictest9281-21diskmemload09:45:152
22127991810,6cyclictest905-21date08:50:012
22127991810,6cyclictest340-21cat07:35:002
2212699189,7cyclictest700-21vmstat11:50:381
22126991813,3cyclictest1035-21hwlatdetect07:35:151
22126991812,4cyclictest25361-21sh10:33:551
22126991811,5cyclictest31551-21expr08:45:151
22126991811,5cyclictest27958-21awk07:20:271
22126991811,4cyclictest9281-21diskmemload09:15:271
22126991810,6cyclictest19295-21smartctl08:15:231
2212599189,6cyclictest5201-21df07:45:150
2212599189,6cyclictest5201-21df07:45:150
2212599182,14cyclictest11346-21tail09:10:250
22125991813,3cyclictest8939-21apt-get11:10:030
22125991811,5cyclictest28091-21pmu-power12:35:210
22125991811,5cyclictest21167-21ntp_kernel_pll_10:30:180
22125991811,5cyclictest15978-21cat08:10:130
22125991810,5cyclictest8836-21idleruntime10:45:170
22125991810,5cyclictest3171-21grep07:40:150
2212899179,6cyclictest4151-21grep07:40:253
2212799179,6cyclictest17136-21grep08:10:242
2212799177,8cyclictest0-21swapper/211:15:142
2212799171,12cyclictest3652-21ntp_states07:40:202
22127991710,5cyclictest4828-21seq08:58:192
22127991710,5cyclictest32589-21wc12:15:122
22127991710,5cyclictest32589-21wc12:15:122
22127991710,5cyclictest27108-21cat07:20:172
2212699179,6cyclictest12082-21cat08:00:181
2212699179,5cyclictest7224-21cat07:50:121
2212699171,7cyclictest6500-21ntp_states09:55:211
22126991713,2cyclictest21533-21/usr/sbin/munin10:55:131
22126991712,3cyclictest30725-21/usr/sbin/munin07:30:261
22126991712,3cyclictest24559-21/usr/sbin/munin08:30:141
22126991712,3cyclictest24559-21/usr/sbin/munin08:30:141
22126991711,4cyclictest25659-21timerwakeupswit07:15:281
22126991710,5cyclictest10680-21seq07:55:361
2212599179,6cyclictest26195-21cron07:20:000
2212599179,6cyclictest21433-21seq10:06:020
2212599179,6cyclictest21433-21seq10:06:010
2212599179,6cyclictest19579-21awk08:15:260
2212599178,6cyclictest11631-21df08:00:130
2212599171,14cyclictest32641-21munin-run07:35:000
22125991710,5cyclictest24544-21cut08:30:010
22125991710,5cyclictest1233-21expr08:50:130
22127991611,3cyclictest22143-21/usr/sbin/munin07:10:192
22127991611,3cyclictest13500-21cpuspeed_turbos08:05:132
22127991610,4cyclictest28803-21expr07:25:112
2212699167,7cyclictest2776-21seq08:54:221
2212599163,8cyclictest24486-21apt-get09:45:120
22125991611,3cyclictest27262-21missed_timers07:20:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional