You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-17 - 21:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun May 17, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3966217797,25sleep20-21swapper/207:09:122
3806217398,26sleep30-21swapper/307:07:083
37052151115,25sleep10-21swapper/107:05:501
37212143101,34sleep00-21swapper/007:06:010
201962630,2sleep20-21swapper/211:08:152
134422630,4sleep312238-21kworker/3:212:15:233
197002580,2sleep30-21swapper/311:31:363
228332570,4chrt0-21swapper/210:21:042
285372540,1sleep20-21swapper/210:01:052
13222540,2sleep10-21swapper/110:05:181
65462530,2sleep30-21swapper/310:09:533
160102500,2sleep3331rcuc/310:40:233
55162490,2sleep30-21swapper/307:10:243
68432470,2sleep228-21ksoftirqd/209:20:282
4305994713,23cyclictest4951-21sh10:32:173
430599447,27cyclictest29811-21sh09:38:263
82382410,1sleep20-21swapper/211:23:322
430599418,17cyclictest6783-21mailstats12:10:233
120502410,1chrt0-21swapper/109:49:451
430599409,16cyclictest23456-21diskmemload10:49:103
430599401,26cyclictest23456-21diskmemload09:28:053
121152390,1sleep10-21swapper/112:39:241
163392380,3chrt4300-21cyclictest11:29:581
430599364,21cyclictest0-21swapper/311:20:443
4305993610,13cyclictest23456-21diskmemload10:51:393
430599340,6cyclictest4474-21ssh12:09:263
430599331,27cyclictest11051-21needreboot11:25:193
430599331,27cyclictest0-21swapper/309:50:123
430599330,5cyclictest17205-21ssh11:54:453
300352330,1sleep10-21swapper/111:15:221
430599320,6cyclictest20460-21unixbench_singl12:20:253
430599319,20cyclictest20826-21sed07:45:243
430599310,6cyclictest23456-21diskmemload12:04:453
430599310,28cyclictest28221-21cat08:05:133
39012310,2chrt0-21swapper/311:44:443
430599309,18cyclictest4754-21expr11:45:123
430599307,20cyclictest13082-21cstates07:30:123
430599301,26cyclictest22483-21/usr/sbin/munin09:05:213
430599300,8cyclictest8823-21diskstats08:35:133
430599300,4cyclictest26499-21ssh12:25:223
430599300,3cyclictest26545-21ssh10:24:493
430599300,3cyclictest1847-21ssh09:41:103
299162300,0sleep0131rcuc/009:38:320
252982300,2sleep20-21swapper/209:59:382
430599298,18cyclictest18497-21swap07:40:253
430599296,20cyclictest15804-21cut11:05:133
430599292,22cyclictest0-21swapper/310:15:113
430599291,4cyclictest26616-21ssh10:00:183
4305992913,14cyclictest14934-21sh11:04:423
4305992911,16cyclictest24115-21tune2fs07:55:143
4305992911,15cyclictest27649-21seq08:03:263
4305992911,15cyclictest23051-21sh09:33:213
430599290,4cyclictest12391-21taskset07:26:333
430599290,3cyclictest9930-21ssh12:37:103
430599290,3cyclictest7685-21/usr/sbin/munin10:10:183
430599290,3cyclictest22361-21memory07:50:173
430499292,18cyclictest2321-21ssh10:55:122
430599289,16cyclictest27800-21hostname10:25:173
430599289,16cyclictest27800-21hostname10:25:173
430599289,16cyclictest15777-21latency07:35:173
430599287,18cyclictest6147-21grep12:35:003
430599287,18cyclictest219262sleep309:56:143
430599283,22cyclictest29710-21latency_hist11:39:593
430599282,2cyclictest0-21swapper/311:10:143
430599282,2cyclictest0-21swapper/311:10:143
430599282,23cyclictest511-21uname08:15:163
430599282,19cyclictest2317-21apt-get10:55:133
430599280,4cyclictest8418-21diskstats10:35:133
430599280,4cyclictest12978-21expr08:45:133
430599280,3cyclictest17439-21cstates08:55:143
430599280,3cyclictest17439-21cstates08:55:133
430599280,2cyclictest72612sleep309:20:533
140472280,2chrt0-21swapper/111:51:321
430599277,17cyclictest9462-21cat07:20:183
430599276,18cyclictest6741-21fschecks_count08:30:143
430599276,18cyclictest20072-21interrupts09:00:183
430599271,8cyclictest704-21ssh11:17:523
430599271,23cyclictest24011-21tr09:10:123
430599270,3cyclictest23456-21diskmemload11:55:063
78162260,1sleep00-21swapper/012:35:230
430599264,19cyclictest2340-21expr08:20:133
430599263,6cyclictest31200-21entropy09:15:153
430599261,9cyclictest11107-21/usr/sbin/munin08:40:153
430599261,5cyclictest6055-21sh09:45:113
430599260,3cyclictest15466-21diskstats08:50:143
4303992618,6cyclictest12512-21sh12:15:141
4303992615,8cyclictest18341-21perl11:55:151
430399261,2cyclictest0-21swapper/107:11:411
4302992615,7cyclictest12786-21apt-get08:45:120
4302992614,7cyclictest53050irq/142-enp3s0-11:40:130
161012260,3sleep012-21ksoftirqd/010:40:240
122902260,3chrt0-21swapper/211:02:012
109472260,4chrt10941-21ssh11:50:011
430599253,19cyclictest30927-21load08:10:193
430599251,4cyclictest6989-21grep07:15:173
430599251,4cyclictest6989-21grep07:15:173
4304992517,6cyclictest29910-21cat11:40:002
430299250,24cyclictest0-21swapper/007:12:000
288352250,3sleep228824-21ssh10:25:252
288352250,3sleep228824-21ssh10:25:252
228312250,2chrt0-21swapper/111:34:471
430599241,9cyclictest4209-21/usr/sbin/munin08:25:133
4304992422,1cyclictest0-21swapper/207:12:082
430499242,16cyclictest25132-21ssh09:35:132
430299241,7cyclictest30296-21ssh11:15:230
4304992312,7cyclictest30276-21cut09:15:002
430399232,13cyclictest30656-21sh09:15:041
430399232,12cyclictest26159-21chrt10:00:131
4302992310,11cyclictest24826-21latency_hist09:35:000
325282230,5sleep2141rcu_preempt09:15:272
222372230,1sleep00-21swapper/009:05:180
147902230,1sleep20-21swapper/211:28:222
4304992216,4cyclictest8153-21ssh11:47:182
4304992214,6cyclictest19137-21df12:20:142
4304992211,7cyclictest1698-21expr10:30:132
430399226,4cyclictest0-21swapper/111:14:131
430399226,4cyclictest0-21swapper/111:14:121
430399222,14cyclictest75-21kswapd010:10:131
430399222,14cyclictest10648-21taskset08:40:101
4303992212,7cyclictest52950irq/141-enp3s0-11:20:251
4302992215,4cyclictest8954-21/usr/sbin/munin11:00:180
4302992214,6cyclictest9201-21hddtemp_smartct07:20:150
4302992212,7cyclictest4954-21df_inode11:45:140
4304992112,7cyclictest24080-21entropy07:55:142
4304992111,8cyclictest12472-21ls09:50:012
4304992111,7cyclictest15452-21df08:50:132
430399212,15cyclictest5042-21apt-get09:20:111
430399212,13cyclictest28088-21ssh10:50:121
4303992116,3cyclictest29693-21cron11:40:001
4303992113,5cyclictest31832-21sh09:40:141
4303992110,8cyclictest6668-21df08:30:141
430299212,13cyclictest5060-21ssh09:20:100
4302992115,4cyclictest21685-21sh09:55:580
4302992113,5cyclictest6075-21latency_hist12:35:000
4302992112,7cyclictest32145-21python309:15:250
286542211,3sleep128834-21sed10:25:261
286542211,3sleep128834-21sed10:25:251
430499208,10cyclictest0-21swapper/211:19:272
4304992015,3cyclictest1538-21sh08:17:002
4304992014,4cyclictest29238-21switchtime08:05:252
4304992012,5cyclictest7754-21sed10:10:182
4304992011,7cyclictest636-21runrttasks11:34:142
4304992011,7cyclictest15607-21missed_timers10:40:192
4304992011,7cyclictest11716-21grep12:15:012
430499200,18cyclictest27243-21sh11:13:402
430499200,18cyclictest27243-21sh11:13:402
430399203,7cyclictest22326-21cat10:45:191
430399201,4cyclictest0-21swapper/109:11:551
4303992012,6cyclictest3973-21seq11:44:491
4303992012,6cyclictest31286-21cat12:05:121
4303992012,6cyclictest19180-21seq07:44:441
4303992012,6cyclictest18373-21seq11:06:251
430299207,8cyclictest12-21ksoftirqd/012:15:110
4302992015,3cyclictest31039-21sh10:51:410
4302992012,6cyclictest7469-21seq10:58:550
4302992012,6cyclictest5772-21seq10:09:060
4302992012,6cyclictest4373-21cat08:25:130
4302992012,6cyclictest15113-21python310:15:230
4302992011,7cyclictest25437-21ls09:10:230
117032200,1sleep20-21swapper/209:25:112
430499198,2cyclictest28-21ksoftirqd/210:48:532
430499191,9cyclictest9745-21users10:35:242
4304991913,4cyclictest636-21runrttasks11:52:412
4304991913,3cyclictest657-21sshd12:02:582
4304991912,5cyclictest19364-21cat11:55:232
4304991912,4cyclictest23456-21diskmemload09:54:092
4304991911,6cyclictest6977-21tune2fs12:35:142
4304991911,6cyclictest32036-21awk11:40:262
4304991911,6cyclictest3154-21which08:20:222
4304991911,5cyclictest12911-21acpi07:30:102
4304991910,7cyclictest1228-21cat10:05:182
4304991910,6cyclictest22890-21cat09:05:242
430499190,17cyclictest26731-21python312:25:232
430399198,4cyclictest141rcu_preempt12:25:131
430399192,12cyclictest13800-21ssh10:15:121
430399191,8cyclictest15104-21ssh09:51:041
4303991915,3cyclictest22806-21ssh09:33:071
4303991914,3cyclictest22982-21sh12:22:571
4303991913,4cyclictest5496-21cpu12:10:131
4303991913,4cyclictest22-21ksoftirqd/111:05:011
4303991913,3cyclictest26727-21sh10:25:001
4303991912,5cyclictest560-21dbus-daemon08:30:001
4303991912,5cyclictest24141-21seq09:58:271
4303991911,6cyclictest32656-21forks08:15:131
4303991911,6cyclictest25763-21cut08:00:001
4303991911,6cyclictest21582-21find07:50:111
4303991910,7cyclictest22348-21cat09:05:181
4303991910,6cyclictest8487-21cut10:35:141
430299192,12cyclictest10645-21apt-get08:40:100
4302991914,3cyclictest7443-21sh12:10:400
4302991914,3cyclictest13283-21sh11:50:460
4302991913,4cyclictest9985-21sh10:35:390
4302991913,4cyclictest6266-21sh10:33:380
4302991913,4cyclictest4592-21sh09:43:560
4302991913,4cyclictest28929-21sh12:03:070
4302991913,4cyclictest20505-21sh11:08:350
4302991912,5cyclictest30809-21cat08:10:180
4302991912,5cyclictest19046-21seq07:43:000
4302991912,5cyclictest18938-21pluginstate11:55:200
4302991911,6cyclictest26892-21cat10:25:010
4302991911,6cyclictest23570-21cat11:35:120
4302991911,6cyclictest17064-21cron07:40:010
4302991911,6cyclictest12903-21find07:30:090
4302991911,5cyclictest53050irq/142-enp3s0-10:45:130
4302991910,7cyclictest4214-21awk11:20:190
282242190,1sleep00-21swapper/010:00:470
430499189,6cyclictest22521-21ntpq07:50:192
430499182,12cyclictest392-21hddtemp_smartct12:30:152
4304991813,3cyclictest7140-21kernelversion07:15:182
4304991813,3cyclictest7140-21kernelversion07:15:182
4304991813,3cyclictest15418-21fschecks_count07:35:142
4304991812,4cyclictest10382-21dump-pmu-power08:40:002
4304991811,5cyclictest18709-21unixbench-2d07:40:262
4304991811,5cyclictest13889-21tr08:45:242
4304991810,6cyclictest9203-21smartctl07:20:152
4304991810,6cyclictest560-21dbus-daemon12:05:222
4304991810,6cyclictest32673-21expr09:40:192
4304991810,6cyclictest30876-21cat08:10:192
4304991810,6cyclictest27489-21seq08:01:212
430399187,9cyclictest4720-21ssh10:32:041
430399181,4cyclictest0-21swapper/112:01:581
4303991813,3cyclictest12241-21unixbench_singl07:25:251
4303991812,4cyclictest5398-21ssh10:56:481
4303991812,4cyclictest31092-21ntp_states08:10:231
4303991812,4cyclictest26692-21unixbench-2d09:35:251
4303991812,4cyclictest17372-21sh10:41:221
4303991811,5cyclictest9296-21awk07:20:171
4303991811,5cyclictest7263-21sed07:15:191
4303991811,5cyclictest7263-21sed07:15:191
4303991811,5cyclictest27347-21awk08:00:291
4303991811,5cyclictest20708-21wc09:00:261
430299182,12cyclictest21056-21kworker/u8:012:25:130
430299182,11cyclictest28039-21apt-get08:05:130
4302991813,3cyclictest10190-21sh10:11:500
4302991812,4cyclictest2698-21sh12:07:380
4302991812,4cyclictest25550-21sh11:11:570
4302991812,4cyclictest25550-21sh11:11:560
4302991812,4cyclictest16386-21sh09:52:230
4302991812,4cyclictest11812-21cpuspeed_turbos09:25:130
4302991812,4cyclictest11410-21mailstats11:25:230
4302991811,5cyclictest8187-21seq07:17:490
4302991811,5cyclictest8187-21seq07:17:480
4302991811,5cyclictest6915-21ls09:45:180
4302991811,5cyclictest29624-21seq10:26:130
4302991811,5cyclictest29624-21seq10:26:120
4302991811,5cyclictest22554-21pluginstate07:50:200
4302991811,5cyclictest19700-21sed12:20:190
4302991811,5cyclictest17279-21fschecks_count11:30:160
4302991811,5cyclictest16115-21grep08:50:200
4302991810,6cyclictest26173-21ls08:00:160
4302991810,6cyclictest25693-21cut08:00:010
4302991810,6cyclictest20620-21sensors_temp07:45:240
9832170,2sleep123456-21diskmemload12:30:221
430499179,6cyclictest7592-21wc08:30:232
430499179,6cyclictest560-21dbus-daemon09:05:002
430499179,6cyclictest14390-21hddtemp_smartct10:15:152
430499176,9cyclictest0-21swapper/210:51:432
4304991712,3cyclictest14336-21sh12:15:562
4304991710,5cyclictest10955-21expr08:40:122
430399176,7cyclictest22-21ksoftirqd/108:05:121
430399172,10cyclictest17275-21apt-get08:55:131
430399172,10cyclictest17275-21apt-get08:55:131
4303991712,3cyclictest20152-21latency07:45:181
4303991712,3cyclictest18205-21latency_hist09:30:001
4303991711,3cyclictest9014-21cut08:35:151
4303991710,5cyclictest13112-21expr07:30:121
430299179,6cyclictest1562-21seq08:17:180
4302991711,4cyclictest18034-21missed_timers08:55:200
4302991711,4cyclictest18034-21missed_timers08:55:190
4302991710,5cyclictest8276-21date08:35:000
4302991710,5cyclictest3812-21seq08:23:280
430499169,5cyclictest6099-21dump-pmu-power08:30:012
430499169,5cyclictest19175-21irqstats09:30:162
4304991611,3cyclictest17249-21/usr/sbin/munin08:55:152
4304991611,3cyclictest17249-21/usr/sbin/munin08:55:152
430499161,11cyclictest10787-21apt-get07:25:122
430399169,5cyclictest4152-21sed08:25:021
4303991611,3cyclictest16329-21switchtime08:50:221
430399161,11cyclictest15069-21runrttasks07:35:111
4302991611,3cyclictest8461-21/usr/sbin/munin08:35:220
4302991611,3cyclictest20332-21ntp_kernel_pll_09:00:220
430299161,11cyclictest10787-21apt-get07:25:140
430499158,5cyclictest19721-21expr07:45:132
4303991510,3cyclictest13389-21if_enp3s008:45:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional