You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-29 - 02:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon Sep 28, 2026 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
306122180122,34sleep00-21swapper/007:09:260
303522163111,30sleep30-21swapper/307:06:033
304072162110,28sleep10-21swapper/107:06:451
303112162109,28sleep20-21swapper/207:05:302
321562770,3chrt32140-21ssh12:05:241
222332640,4sleep10-21swapper/111:35:051
82212610,1sleep20-21swapper/210:36:272
32332610,2sleep30-21swapper/308:35:143
203682600,2sleep00-21swapper/009:58:260
203682600,2sleep00-21swapper/009:58:250
115882570,2sleep10-21swapper/112:38:511
56792540,2sleep10-21swapper/109:47:351
226242500,1sleep10-21swapper/109:35:471
314282490,1sleep20-21swapper/210:54:132
121612490,1sleep30-21swapper/311:03:333
312762480,1sleep00-21swapper/011:17:200
158722480,2chrt0-21swapper/011:30:150
90212470,2sleep30-21swapper/310:14:013
66662470,1sleep00-21swapper/011:23:210
321152470,2sleep10-21swapper/111:41:411
30934994511,17cyclictest4066-21sh10:57:193
177282440,3chrt17724-21munin-run07:55:002
285152430,1sleep00-21swapper/009:16:490
3093499425,21cyclictest825-21sh09:44:333
3093499421,28cyclictest0-21swapper/311:51:523
135382420,2sleep2271rcuc/210:16:332
3093499414,21cyclictest31233-21sh11:17:163
3093499414,21cyclictest27969-21sh12:26:323
30934994123,16cyclictest6260-21ssh09:48:083
3093499411,21cyclictest17472-21/usr/sbin/munin10:20:113
3093499406,19cyclictest6649-21kworker/u8:112:11:143
3093499406,18cyclictest18091-21diskmemload12:20:583
3093499404,21cyclictest9810-21sh12:37:053
3093499401,36cyclictest0-21swapper/309:31:273
3093499401,21cyclictest19073-21ssh10:45:133
109422400,1sleep30-21swapper/310:15:153
3093499391,25cyclictest20037-21sh11:56:423
3093499381,22cyclictest26049-21kworker/u8:111:49:483
3093499381,20cyclictest18091-21diskmemload11:24:403
3093499380,1cyclictest0-21swapper/310:04:413
245312380,3sleep00-21swapper/012:00:180
3093499379,14cyclictest29444-21sh09:17:433
3093499371,4cyclictest25883-21sh09:39:013
3093499371,22cyclictest0-21swapper/309:53:273
3093499371,21cyclictest0-21swapper/311:08:383
3093499363,30cyclictest15108-21cron11:29:593
3093499362,19cyclictest18091-21diskmemload12:06:563
30934993617,16cyclictest0-21swapper/311:12:513
3093499353,23cyclictest0-21swapper/311:35:113
3093499352,19cyclictest25443-21kworker/u8:009:20:503
3093499351,20cyclictest6814-21taskset07:29:173
3093499345,15cyclictest0-21swapper/310:39:213
3093299341,30cyclictest18618-21/usr/sbin/munin10:20:211
3093199341,30cyclictest24447-21sh10:49:040
30934993317,15cyclictest3013-21ssh10:33:013
3093499331,29cyclictest28138-21unixbench-2d08:15:253
3093499330,4cyclictest648-21ntpd09:57:393
3093499330,4cyclictest648-21ntpd09:57:383
3093299331,29cyclictest24422-21ssh11:12:171
3093499321,18cyclictest0-21swapper/309:12:423
3093499321,18cyclictest0-21swapper/309:12:423
30932993223,7cyclictest31116-21ssh10:05:331
3093499319,20cyclictest30624-21cron10:30:003
3093499313,23cyclictest0-21swapper/312:15:113
30934993112,17cyclictest24936-21awk12:00:223
3093499310,4cyclictest1827-21awk07:15:213
3093499310,2cyclictest29108-21apt-get11:40:103
3093499310,1cyclictest0-21swapper/310:52:013
3093299313,4cyclictest24760-21grep10:25:151
3093299311,4cyclictest27737-21ssh10:50:351
3093299311,27cyclictest17554-21wc09:05:261
3093199313,26cyclictest27992-21kworker/u8:110:01:380
30931993127,2cyclictest0-21swapper/010:34:060
3093499307,20cyclictest9976-21cat07:35:183
3093499306,21cyclictest23002-21expr08:05:173
3093499303,24cyclictest21671-21timerwakeupswit08:00:253
3093499301,5cyclictest31574-21ssh10:06:013
3093499300,4cyclictest15365-21ssh11:30:023
30931993023,4cyclictest515-21irqbalance11:14:090
124322300,9sleep012-21ksoftirqd/010:15:280
3093499299,18cyclictest822-21sed12:30:223
3093499299,17cyclictest20071-21cron08:00:003
3093499293,22cyclictest34-21ksoftirqd/307:50:133
3093499290,3cyclictest7803-21pmu-power09:25:213
30933992921,5cyclictest18091-21diskmemload09:39:562
3093399291,26cyclictest0-21swapper/208:20:162
3093399291,25cyclictest14063-21df_inode09:30:132
3093299291,25cyclictest9099-21apt-get08:50:011
3093299291,24cyclictest0-21swapper/109:19:081
3093299290,26cyclictest13248-21cmp09:30:001
3093199291,3cyclictest17873-21sh10:44:160
3093199291,3cyclictest13077-21apt-get09:30:020
3093499286,20cyclictest7722-21sed08:45:143
3093499284,21cyclictest25442-21needreboot08:10:203
3093499281,3cyclictest12547-21cstates10:40:133
3093499281,24cyclictest29246-21diskstats08:20:163
3093499281,24cyclictest15578-21munin-run07:50:003
3093499281,24cyclictest13170-21grep08:55:243
3093499280,3cyclictest7117-21/usr/sbin/munin07:30:253
30933992823,3cyclictest30651-21munin-run10:29:592
3093299281,25cyclictest13749-21ssh10:16:461
3093299281,25cyclictest0-21swapper/110:44:051
3093299281,25cyclictest0-21swapper/110:31:131
3093299281,25cyclictest0-21swapper/110:01:381
3093299281,25cyclictest0-21swapper/109:40:491
3093299281,25cyclictest0-21swapper/109:22:081
30931992823,3cyclictest30068-21python308:20:240
3093199281,3cyclictest5947-21ssh10:10:590
3093199281,3cyclictest3488-21ssh09:22:190
3093199281,25cyclictest0-21swapper/012:09:190
3093199280,3cyclictest15124-21ssh09:54:580
227792280,3sleep222773-21ssh11:35:152
3093499272,21cyclictest0-21swapper/307:10:123
3093499271,23cyclictest5048-21/usr/sbin/munin08:40:153
3093499271,23cyclictest17046-21grep09:05:203
3093499270,3cyclictest32209-21python308:25:213
3093499270,3cyclictest1092-21df08:30:143
30933992722,3cyclictest29732-21perl10:05:142
30933992719,6cyclictest30103-21ssh09:18:232
3093399271,24cyclictest5677-21ssh09:47:352
3093399271,24cyclictest0-21swapper/209:55:462
3093399271,24cyclictest0-21swapper/209:55:452
3093299271,2cyclictest22471-21apt-get08:05:131
3093299271,25cyclictest0-21swapper/111:32:311
3093299271,24cyclictest0-21swapper/111:08:281
3093299271,24cyclictest0-21swapper/110:10:131
3093299271,24cyclictest0-21swapper/109:56:251
3093299271,24cyclictest0-21swapper/109:56:241
3093299271,24cyclictest0-21swapper/108:01:481
3093299270,26cyclictest53050irq/142-enp3s0-11:03:301
30931992723,2cyclictest18091-21diskmemload09:40:250
30931992723,2cyclictest0-21swapper/009:39:270
3093199271,2cyclictest0-21swapper/010:37:030
3093199271,24cyclictest0-21swapper/010:26:230
3093199271,24cyclictest0-21swapper/009:03:560
3093199271,24cyclictest0-21swapper/008:35:230
3093199271,24cyclictest0-21swapper/008:01:480
3093499268,16cyclictest15739-21cron09:04:593
3093499261,22cyclictest9508-21systemd08:50:023
3093499261,22cyclictest3356-21tail07:20:143
3093499260,23cyclictest13020-21sed07:40:233
30933992623,2cyclictest30929-21cyclictest10:48:052
30933992620,4cyclictest5778-21ssh10:10:492
30933992615,7cyclictest22420-21apt-get10:00:122
3093399261,2cyclictest0-21swapper/211:19:022
3093399261,2cyclictest0-21swapper/210:34:422
3093399261,2cyclictest0-21swapper/209:40:372
3093399261,23cyclictest0-21swapper/211:21:062
3093399261,23cyclictest0-21swapper/211:12:422
3093399261,23cyclictest0-21swapper/210:20:152
3093399261,23cyclictest0-21swapper/208:35:142
3093399261,23cyclictest0-21swapper/208:05:212
3093399261,1cyclictest0-21swapper/210:42:392
30932992624,1cyclictest0-21swapper/110:48:591
30932992624,1cyclictest0-21swapper/110:35:011
30932992624,1cyclictest0-21swapper/109:50:501
30932992624,1cyclictest0-21swapper/108:49:061
30932992623,2cyclictest0-21swapper/111:23:451
3093299261,24cyclictest0-21swapper/111:16:511
3093299261,1cyclictest0-21swapper/109:30:571
3093299261,1cyclictest0-21swapper/109:10:151
3093299261,1cyclictest0-21swapper/109:10:141
3093299260,24cyclictest0-21swapper/112:00:011
3093299260,24cyclictest0-21swapper/111:55:041
3093299260,24cyclictest0-21swapper/108:36:061
30931992624,1cyclictest0-21swapper/010:50:430
3093199261,24cyclictest0-21swapper/011:54:430
3093199261,24cyclictest0-21swapper/011:28:080
3093199261,23cyclictest0-21swapper/008:05:150
3093199261,1cyclictest0-21swapper/011:08:180
3093199261,1cyclictest0-21swapper/009:45:160
3093199261,1cyclictest0-21swapper/008:40:170
30933992520,3cyclictest2969-21ssh09:21:482
3093399251,23cyclictest0-21swapper/212:08:262
3093399251,23cyclictest0-21swapper/211:49:192
3093399251,23cyclictest0-21swapper/211:02:242
3093399251,23cyclictest0-21swapper/209:51:492
3093399251,23cyclictest0-21swapper/208:01:382
3093399251,1cyclictest0-21swapper/211:32:552
3093399251,1cyclictest0-21swapper/210:59:462
3093399251,1cyclictest0-21swapper/208:49:522
3093399250,23cyclictest0-21swapper/212:00:022
3093299251,1cyclictest0-21swapper/111:28:171
30931992523,1cyclictest0-21swapper/011:02:550
30931992523,1cyclictest0-21swapper/010:20:210
30931992523,1cyclictest0-21swapper/008:50:040
30931992523,1cyclictest0-21swapper/008:49:050
3093199251,1cyclictest0-21swapper/011:55:040
3093199250,24cyclictest0-21swapper/010:05:330
30933992422,1cyclictest0-21swapper/211:28:242
3093399240,23cyclictest0-21swapper/211:06:472
3093399240,1cyclictest0-21swapper/209:05:262
30932992422,1cyclictest0-21swapper/108:17:391
30931992422,1cyclictest0-21swapper/009:10:200
30931992422,1cyclictest0-21swapper/009:10:190
19642240,2sleep10-21swapper/107:15:231
30933992321,1cyclictest9882-21ssh09:26:542
30933992316,5cyclictest28-21ksoftirqd/211:45:002
30931992321,1cyclictest0-21swapper/011:43:540
30933992220,1cyclictest0-21swapper/211:58:442
30933992213,6cyclictest31296-21df07:10:142
3093299221,14cyclictest18091-21diskmemload11:45:111
3093199222,14cyclictest4885-21ssh12:10:120
262732220,2sleep00-21swapper/012:25:210
3093199212,13cyclictest11538-21apt-get07:40:090
30931992119,1cyclictest0-21swapper/009:26:340
30931992116,3cyclictest15750-21apt-get09:05:010
30931992113,5cyclictest27209-21fw_packets08:15:160
195492211,2sleep20-21swapper/207:55:262
30933992015,3cyclictest911-21timerandwakeup12:30:242
30933992012,6cyclictest14028-21entropy07:45:152
30933992012,6cyclictest11569-21smartctl11:50:212
30933992010,7cyclictest9557-21apt-get08:50:102
3093299209,2cyclictest22-21ksoftirqd/112:15:161
3093299201,9cyclictest32725-21/usr/sbin/munin12:30:181
30932992013,4cyclictest18091-21diskmemload12:13:271
30932992012,6cyclictest8000-21netstat07:30:231
30932992011,7cyclictest12916-21ls07:40:231
30931992012,6cyclictest2635-21wc07:20:000
3093199201,17cyclictest1367-21ssh12:30:370
30931992010,8cyclictest629-21cstates10:55:140
174472200,2sleep00-21swapper/012:19:310
30933991911,6cyclictest26548-21wc12:25:232
30933991911,6cyclictest12687-21grep07:40:202
30933991910,7cyclictest560-21dbus-daemon12:14:592
30933991910,6cyclictest6590-21tr12:35:132
3093399190,6cyclictest20441-21ssh09:10:362
3093399190,6cyclictest20441-21ssh09:10:352
3093299196,3cyclictest22-21ksoftirqd/112:20:141
30932991914,3cyclictest18733-21hwlatdetect07:55:161
30932991911,6cyclictest26584-21ls12:25:241
30932991911,6cyclictest11124-21seq07:39:411
30932991911,6cyclictest10413-21expr11:50:121
30932991911,5cyclictest22-21ksoftirqd/108:10:121
30932991910,7cyclictest5037-21seq08:40:041
30931991913,4cyclictest23513-21sh12:23:510
30931991911,6cyclictest25559-21pluginstate08:10:230
30931991911,6cyclictest2148-21date08:30:250
30931991911,6cyclictest17670-21seq07:54:420
147912190,2sleep20-21swapper/209:00:212
3093399182,12cyclictest31051-21apt-get08:25:112
3093399181,4cyclictest0-21swapper/212:17:142
30933991812,4cyclictest10040-21missed_timers07:35:202
30933991811,5cyclictest821-21awk08:30:122
30933991811,5cyclictest5192-21expr07:25:132
30933991811,5cyclictest25548-21open_inodes08:10:212
30933991810,6cyclictest2002-21sed07:15:242
3093299186,4cyclictest22-21ksoftirqd/108:30:201
30932991815,2cyclictest0-21swapper/108:20:181
30932991813,3cyclictest12619-21ntp_states08:55:201
30932991811,5cyclictest6019-21which07:25:221
30932991811,5cyclictest15115-21ls09:00:231
30932991811,5cyclictest1371-21sed10:55:181
30931991811,5cyclictest5597-21cat07:25:180
30931991811,5cyclictest19833-21seq07:57:190
30931991810,6cyclictest4602-21seq07:23:200
30931991810,5cyclictest13750-21cat07:45:120
3093399179,6cyclictest7355-21expr07:30:152
3093399179,6cyclictest27873-21which08:15:232
3093399178,6cyclictest278-21systemd-journal08:59:592
30933991712,3cyclictest22787-21sh12:23:072
3093399171,12cyclictest6234-21python308:40:232
3093299179,6cyclictest31029-21find08:25:101
3093299178,7cyclictest31482-21grep07:10:171
3093199179,6cyclictest32556-21users08:25:240
30931991712,3cyclictest7832-21sh12:35:240
30931991711,5cyclictest32050-21gzip07:10:230
30931991710,5cyclictest8094-21awk07:30:240
30931991710,5cyclictest22590-21expr11:35:130
30931991710,5cyclictest11851-21expr08:55:120
30931991710,5cyclictest10353-21rt-features07:35:230
30931991710,4cyclictest3452-21apt-get11:45:120
3093399169,5cyclictest3076-21expr07:20:122
3093299169,5cyclictest3676-21sed07:20:181
3093299169,3cyclictest22-21ksoftirqd/107:45:111
30932991610,4cyclictest17088-21sed07:50:241
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional