You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 02:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8s.osadl.org (updated Thu Mar 19, 2026 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30152221598,96sleep20-21swapper/207:07:122
292492156101,29sleep10-21swapper/107:05:181
302402139102,26sleep30-21swapper/307:08:213
284672137102,24sleep00-21swapper/007:05:050
1755721120,1sleep00-21swapper/011:58:260
45482660,2sleep20-21swapper/209:48:142
31122660,2sleep10-21swapper/110:35:161
81882640,1sleep20-21swapper/209:26:342
135112640,2sleep00-21swapper/010:19:170
47152630,9sleep334-21ksoftirqd/312:12:493
33282620,2chrt0-21swapper/310:35:183
16352580,3sleep0131rcuc/011:22:230
199702550,1sleep20-21swapper/210:24:022
75112520,3sleep07498-21ssh12:15:120
308582520,4sleep00-21swapper/007:10:130
307152500,1sleep20-21swapper/212:08:072
101022500,4sleep110103-21sh09:28:311
170792480,2chrt0-21swapper/109:57:191
130032430,1sleep10-21swapper/109:54:591
117262430,1sleep10-21swapper/112:18:061
172542410,2chrt0-21swapper/110:45:241
161562410,1sleep00-21swapper/011:09:390
191352320,2sleep30-21swapper/311:10:543
106802320,7sleep3331rcuc/310:16:273
210632310,1sleep10-21swapper/112:25:121
3064499283,17cyclictest25682-21ssh09:40:111
253052280,3sleep1211rcuc/111:15:251
30644992718,6cyclictest1413-21sh11:22:101
240882270,2chrt0-21swapper/112:03:121
31282260,1sleep10-21swapper/109:46:481
3064499261,24cyclictest0-21swapper/107:10:041
87252250,3sleep10-21swapper/111:51:171
3064699251,10cyclictest28927-21memory10:30:183
3064599251,1cyclictest0-21swapper/207:10:002
3064499253,15cyclictest10238-21sh11:05:101
3064499251,16cyclictest4719-21sh11:25:131
3064499251,16cyclictest4719-21sh11:25:131
3064699249,6cyclictest13044-21ls11:55:123
3064699242,9cyclictest29180-21ssh10:55:113
3064499243,12cyclictest14056-21ssh12:20:101
30644992414,6cyclictest17435-21diskmemload11:55:121
30643992416,6cyclictest30165-21ssh12:07:330
198602240,2sleep10-21swapper/109:35:211
30646992312,7cyclictest20240-21cat12:00:173
3064399237,4cyclictest66050lldpd11:47:350
3064399237,4cyclictest66050lldpd09:57:030
253372230,2sleep30-21swapper/309:40:003
57082220,5sleep0121ksoftirqd/011:25:200
57082220,5sleep0121ksoftirqd/011:25:200
3064699223,13cyclictest640-21snmpd10:05:123
30646992212,7cyclictest21804-21ntp_states12:25:193
30646992211,9cyclictest15380-21sh10:44:593
3064699221,18cyclictest560-21dbus-daemon10:00:003
3064599221,11cyclictest28082-21runrttasks10:30:122
3064499222,8cyclictest636-21runrttasks10:50:131
3064499222,15cyclictest0-21swapper/109:05:131
30644992212,7cyclictest24429-21cron11:39:591
30643992212,7cyclictest21523-21python310:00:230
23022220,5sleep30-21swapper/307:19:593
109972220,6sleep3560-21dbus-daemon11:30:003
109972220,6sleep3560-21dbus-daemon11:29:593
3064699211,7cyclictest10962sh09:21:133
30646992115,4cyclictest30778-21sh12:08:113
30646992115,3cyclictest32122-21cat09:44:593
30646992113,6cyclictest3320-21ls12:35:233
3064699211,18cyclictest16910-21ntpq10:45:213
30646992111,5cyclictest6575-21tr09:25:223
3064599219,8cyclictest21968-21apt-get08:05:112
3064599217,5cyclictest2487-21grep10:35:112
3064599212,13cyclictest11418-21ssh11:30:112
30645992113,6cyclictest12560-21ls08:55:262
30645992112,7cyclictest13110-21awk09:30:212
3064599210,18cyclictest13690-21cat09:55:122
3064499213,13cyclictest75-21kswapd011:45:111
30644992114,4cyclictest1423-21sh12:10:171
30644992112,7cyclictest21355-21apt-get10:25:111
30644992112,6cyclictest53050irq/142-enp3s0-10:00:131
30644992112,6cyclictest29829-21hostname10:55:161
30644992112,5cyclictest53050irq/142-enp3s0-11:03:071
3064399212,13cyclictest6726-21apt-get09:50:110
3064399211,8cyclictest3023-21sh12:11:080
30643992114,5cyclictest12-21ksoftirqd/009:40:000
30643992113,6cyclictest6626-21seq11:01:470
30643992113,6cyclictest27249-21ls08:15:170
30643992113,6cyclictest10417-21ls08:50:240
30643992112,6cyclictest9965-21memory10:40:170
30643992112,6cyclictest5668-21pluginstate07:25:210
30643992112,6cyclictest17519-21sed09:10:000
30643992111,7cyclictest2502-21date08:35:000
30643992111,6cyclictest52750irq/139-enp3s0-11:50:040
30643992111,5cyclictest18188-21cat11:10:210
97082200,1sleep09712-21latency07:35:170
3064699202,12cyclictest3512-21apt-get11:00:113
30646992014,4cyclictest34-21ksoftirqd/310:05:013
30646992012,6cyclictest22356-21cut08:05:153
30646992012,5cyclictest30329-21/usr/sbin/munin11:20:123
30646992011,7cyclictest20081-21expr08:00:113
3064699201,16cyclictest3830-21timerandwakeup07:20:233
3064699201,11cyclictest75-21kswapd011:35:103
30646992010,7cyclictest11551-21cat07:40:133
3064599202,12cyclictest29162-21apt-get10:55:102
3064599202,11cyclictest9102-21apt-get10:40:102
3064599201,9cyclictest22687-21python310:25:222
3064599201,8cyclictest4465-21idleruntime-cro07:24:592
3064599201,7cyclictest88442sh11:27:522
3064599201,7cyclictest88442sh11:27:522
30645992012,6cyclictest7154-21ntpq11:50:192
30645992012,6cyclictest5131-21cat08:40:132
30645992012,6cyclictest3771-21which08:35:222
30645992011,6cyclictest14728-21cat07:45:242
3064599200,4cyclictest0-21swapper/209:51:522
3064499202,13cyclictest4837-21apt-get08:40:131
3064499202,13cyclictest14672-21sh10:20:111
30644992012,6cyclictest32220-21seq10:09:021
30644992012,6cyclictest10061-21cut07:35:221
30644992012,5cyclictest21209-21grep08:00:231
30644992011,7cyclictest1284-21load08:30:181
30644992011,6cyclictest29074-21ntp_kernel_pll_10:30:191
30644992010,7cyclictest560-21dbus-daemon08:50:001
3064499200,9cyclictest66050lldpd07:47:011
3064499200,18cyclictest26124-21wc08:15:001
3064399209,7cyclictest1509-21sh10:10:120
30643992012,6cyclictest23790-21seq08:09:250
30643992011,7cyclictest19143-21ntp_kernel_pll_11:35:190
30643992011,6cyclictest12856-21date09:30:170
290422200,8sleep228-21ksoftirqd/208:20:142
183332200,3sleep1211rcuc/109:35:001
170382200,2chrt0-21swapper/307:50:253
48832190,1sleep10-21swapper/112:36:431
3064699198,7cyclictest52950irq/141-enp3s0-09:15:123
3064699192,12cyclictest75-21kswapd012:30:113
3064699191,5cyclictest0-21swapper/311:31:523
30646991913,3cyclictest18212-21sed07:55:153
30646991912,5cyclictest14734-21sort10:20:133
30646991912,4cyclictest17435-21diskmemload10:50:193
30646991911,6cyclictest416-21expr09:45:153
30646991911,6cyclictest3878-21wc08:35:243
30646991911,6cyclictest31406-21cat08:25:163
30646991911,6cyclictest25088-21awk08:10:213
30646991910,7cyclictest8106-21cat12:15:173
30646991910,7cyclictest26389-21ls11:40:233
30646991910,6cyclictest18119-21ls09:10:123
3064599199,7cyclictest24434-21df_inode08:10:142
3064599199,7cyclictest15613-21cat09:05:142
3064599192,11cyclictest26503-21apt-get08:15:102
30645991914,3cyclictest31766-21kernelversion09:20:192
30645991914,3cyclictest28264-21sh09:17:262
30645991914,3cyclictest23412-21sh10:01:552
30645991914,3cyclictest16190-21sh11:57:032
30645991913,4cyclictest7173-21sh12:15:032
30645991913,4cyclictest2988-21sh12:11:052
30645991913,4cyclictest28673-21sh10:05:292
30645991913,4cyclictest20642-21sh10:48:312
30645991912,5cyclictest15629-21grep12:20:232
30645991912,4cyclictest16186-21kernelversion07:50:182
30645991911,6cyclictest9914-21cat11:05:012
30645991911,6cyclictest31688-21ls11:45:132
30645991911,6cyclictest30809-21apt08:25:112
30645991911,6cyclictest28333-21cat12:30:172
30645991910,7cyclictest7721-21expr08:45:182
30645991910,7cyclictest7315-21perl07:30:142
30645991910,7cyclictest2176-21df12:35:122
30645991910,6cyclictest9790-21missed_timers07:35:192
30645991910,6cyclictest2623-21date08:35:002
30644991913,4cyclictest19794-21sh11:11:341
30644991912,4cyclictest636-21runrttasks12:34:271
3064399193,3cyclictest0-21swapper/008:40:120
3064399192,12cyclictest23658-21chrt11:15:120
3064399192,11cyclictest19830-21apt-get08:00:120
3064399191,8cyclictest0-21swapper/012:20:120
3064399191,7cyclictest1842-21users09:45:270
30643991914,3cyclictest5912-21grep09:25:150
30643991914,3cyclictest24847-21apt11:40:090
30643991913,4cyclictest4472-21sh10:35:460
30643991913,4cyclictest31536-21sh09:44:350
30643991913,4cyclictest2545-21sh10:59:290
30643991913,4cyclictest19411-21sh10:47:170
30643991913,4cyclictest14677-21sh11:31:560
30643991912,5cyclictest7904-21awk08:45:210
30643991912,5cyclictest22081-21cat10:25:150
30643991911,6cyclictest31568-21memory08:25:190
30643991911,6cyclictest27160-21grep12:30:010
30643991911,6cyclictest24504-21apt09:15:100
30643991911,6cyclictest21714-21seq12:00:510
30643991911,6cyclictest1992-21grep12:35:120
30643991911,6cyclictest14704-21ls07:45:240
30643991911,6cyclictest13336-21date07:45:010
30643991910,6cyclictest7603-21seq07:30:160
30643991910,6cyclictest3824-21mailstats08:35:230
30643991910,6cyclictest32025-21ntp_kernel_err09:20:210
18582190,2chrt0-21swapper/311:46:213
12242190,2sleep10-21swapper/109:21:191
3064699188,7cyclictest17427-21cron09:09:593
3064699187,7cyclictest34-21ksoftirqd/311:15:233
3064699181,8cyclictest14353-21rm09:31:013
30646991813,3cyclictest19627-21sh12:24:163
30646991812,4cyclictest8410-21sh11:50:593
3064699181,12cyclictest8995-21apt-get07:35:123
30646991810,6cyclictest5973-21cat07:25:243
30646991810,6cyclictest10476-21expr11:05:123
30646991810,5cyclictest7712-21netstat09:50:193
3064599185,11cyclictest26533-21unixbench_singl11:40:222
3064599182,11cyclictest368-21apt-get07:15:132
3064599182,11cyclictest21141-21cat12:25:142
3064599181,8cyclictest25174-21ssh12:04:182
3064599181,6cyclictest25275-21chrt10:51:312
30645991813,3cyclictest21333-21sh11:36:542
30645991813,3cyclictest10148-21sh10:15:562
30645991811,5cyclictest18978-21cat09:35:122
30645991811,5cyclictest18568-21ntp_kernel_pll_07:55:182
30645991811,5cyclictest18166-21cat09:10:122
30645991811,5cyclictest17346-21cut11:10:132
30645991811,5cyclictest10619-21entropy11:05:142
30645991811,5cyclictest10432-21cat08:50:232
30645991811,4cyclictest3827-21sh10:11:192
30645991810,6cyclictest4496-21cut11:25:002
30645991810,6cyclictest24320-21cat11:15:172
30645991810,6cyclictest20043-21ls08:00:122
30645991810,5cyclictest0-21swapper/209:40:032
3064499189,7cyclictest6610-21cron07:30:001
3064499189,6cyclictest1838-21grep07:15:251
3064499188,8cyclictest22-21ksoftirqd/107:20:111
3064499182,12cyclictest25909-21ntp_states11:40:181
3064499181,7cyclictest23865-21dump-pmu-power08:10:001
3064499181,5cyclictest0-21swapper/111:32:561
30644991813,3cyclictest29064-21sh09:18:161
30644991811,5cyclictest1893-21irqstats10:10:161
30644991811,4cyclictest9072-21/usr/sbin/munin10:40:161
30644991811,4cyclictest10236-21mailstats08:50:221
30644991810,6cyclictest26294-21grep12:05:111
30644991810,6cyclictest16448-21ntp_states07:50:201
3064399189,7cyclictest27688-21sed10:05:170
3064399188,6cyclictest52750irq/139-enp3s0-10:20:120
3064399182,12cyclictest22476-21apt-get10:50:130
3064399182,11cyclictest2578-21apt-get07:20:120
3064399182,11cyclictest17672-21apt-get07:55:110
3064399182,10cyclictest15534-21apt-get07:50:120
30643991814,2cyclictest11251-21apt08:55:120
30643991812,4cyclictest30406-21sh12:31:350
30643991811,5cyclictest18094-21cat09:10:120
3064399181,12cyclictest24110-21taskset08:10:120
30643991810,6cyclictest30637-21date08:25:000
30643991810,6cyclictest28453-21entropy10:30:140
3064699179,6cyclictest6917-21cat07:30:113
3064699179,5cyclictest2032-21missed_timers10:10:173
3064699177,7cyclictest11253-21apt-get08:55:133
30646991713,2cyclictest13775-21fschecks_count07:45:163
30646991712,3cyclictest9085-21/usr/sbin/munin08:50:243
3064699171,11cyclictest8131-21chrt08:45:223
30646991710,5cyclictest31884-21sed07:10:243
30646991710,5cyclictest27061-21smartctl08:15:163
3064599179,6cyclictest4873-21expr07:25:142
3064599179,6cyclictest15483-21sed09:05:002
3064599178,7cyclictest12465-21cat07:40:232
3064499179,6cyclictest4610-21grep08:40:001
3064499179,6cyclictest27835-21sed08:15:241
3064499173,9cyclictest7920-21ssh10:15:121
3064499172,10cyclictest13411-21apt-get09:00:121
3064499171,6cyclictest13129-21cron09:00:001
30644991713,2cyclictest22-21ksoftirqd/107:30:201
30644991712,3cyclictest11230-21/usr/sbin/munin07:40:241
30644991711,4cyclictest19293-21sed09:10:221
30644991710,5cyclictest31586-21sed08:25:181
30644991710,5cyclictest18668-21head07:55:201
3064399179,6cyclictest14331-21ntpq09:00:220
223622170,1sleep30-21swapper/310:25:193
3064699168,6cyclictest1692-21python308:30:223
3064699162,10cyclictest28665-21apt-get08:20:113
30646991611,3cyclictest6726-21dump-pmu-power08:45:013
30644991610,4cyclictest29973-21timerwakeupswit08:20:231
3064399168,6cyclictest1143-21cat07:15:190
30646991510,3cyclictest14318-21ntp_offset09:00:223
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional