You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-28 - 20:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue Jul 28, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
251352235181,29sleep20-21swapper/207:09:122
250802181160,15sleep10-21swapper/107:08:311
251282171101,63sleep00-21swapper/007:09:060
249902159104,27sleep30-21swapper/307:07:213
1224321320,2sleep00-21swapper/011:59:180
1224321320,2sleep00-21swapper/011:59:180
2519927558,10sleep20-21swapper/207:10:002
65622720,1sleep20-21swapper/209:53:402
307132680,3sleep00-21swapper/011:00:150
229632660,2sleep20-21swapper/209:17:082
36812600,2sleep20-21swapper/211:52:222
299562600,1sleep10-21swapper/111:47:561
223382590,2sleep10-21swapper/109:16:301
167852580,1sleep316786-21sh10:50:033
133442500,1sleep30-21swapper/312:00:103
292252490,2chrt0-21swapper/209:45:322
204912490,1sleep00-21swapper/011:16:200
191462490,2sleep018933-21latency_hist09:15:010
120332480,2sleep112032-21ssh11:10:191
275522460,1sleep00-21swapper/012:35:100
275522460,1sleep00-21swapper/012:35:100
4872450,1sleep0131rcuc/011:50:150
135682450,1sleep10-21swapper/112:24:441
32432430,2sleep10-21swapper/109:50:251
11332420,1sleep00-21swapper/011:25:430
185262410,5sleep012-21ksoftirqd/010:02:120
287672400,1sleep10-21swapper/107:15:201
287672400,1sleep10-21swapper/107:15:201
187152390,1sleep318717-21rm09:38:323
153892380,2chrt0-21swapper/310:24:473
327192370,1sleep10-21swapper/109:25:011
277822340,3sleep10-21swapper/110:57:551
220952330,5sleep322102-21sh12:30:243
314892320,3chrt0-21swapper/211:49:292
2547599317,8cyclictest28-21ksoftirqd/210:21:252
2547599308,8cyclictest28-21ksoftirqd/211:15:212
25475993027,2cyclictest271rcuc/210:31:172
137162300,8sleep228-21ksoftirqd/212:24:532
25475992912,8cyclictest28-21ksoftirqd/210:19:132
2547599284,9cyclictest28-21ksoftirqd/210:52:452
2547599282,7cyclictest28-21ksoftirqd/212:04:412
25475992820,6cyclictest24951-21rm12:32:452
2547499281,2cyclictest0-21swapper/107:21:421
2547399281,25cyclictest0-21swapper/007:22:320
14312280,3sleep20-21swapper/211:26:022
117862280,3chrt0-21swapper/309:57:103
299062270,2chrt0-21swapper/012:11:590
25476992717,8cyclictest3915-21ssh09:27:063
2547699271,24cyclictest0-21swapper/307:22:193
2547599274,15cyclictest521-21ssh09:25:132
2547599273,7cyclictest28-21ksoftirqd/211:21:022
25475992723,3cyclictest271rcuc/212:38:552
25475992723,3cyclictest271rcuc/212:38:552
25475992723,3cyclictest271rcuc/209:43:232
25474992723,2cyclictest0-21swapper/107:26:271
2547499271,12cyclictest326142sh09:48:581
150202270,3sleep315017-21ssh12:25:183
2547599264,7cyclictest28-21ksoftirqd/210:45:192
2547599264,6cyclictest28-21ksoftirqd/210:27:192
2547599264,6cyclictest28-21ksoftirqd/210:27:192
2547599263,8cyclictest28-21ksoftirqd/209:13:452
2547599263,7cyclictest28-21ksoftirqd/209:55:582
2547599263,7cyclictest28-21ksoftirqd/209:21:222
25475992622,3cyclictest0-21swapper/207:23:172
25475992610,6cyclictest28-21ksoftirqd/210:09:452
2547499269,4cyclictest167862sh10:50:021
25474992616,5cyclictest636-21runrttasks10:35:131
2547499261,14cyclictest13533-21ntp_states11:35:221
283462250,2sleep00-21swapper/009:20:470
2547699256,8cyclictest141rcu_preempt11:05:133
25476992514,6cyclictest0-21swapper/310:35:133
2547599255,7cyclictest28-21ksoftirqd/209:39:382
2547599254,7cyclictest28-21ksoftirqd/209:33:352
2547599253,7cyclictest28-21ksoftirqd/211:56:462
2547599253,7cyclictest28-21ksoftirqd/211:56:452
2547599253,7cyclictest28-21ksoftirqd/211:42:002
2547599252,6cyclictest28-21ksoftirqd/210:37:362
25475992521,3cyclictest271rcuc/212:09:062
2547599251,1cyclictest0-21swapper/207:18:012
2547599251,1cyclictest0-21swapper/207:18:002
2547599250,7cyclictest28-21ksoftirqd/210:42:092
25473992523,1cyclictest0-21swapper/007:26:080
25473992523,1cyclictest0-21swapper/007:17:350
25473992523,1cyclictest0-21swapper/007:17:350
25473992511,9cyclictest15913-21kworker/u8:009:50:130
180352250,3sleep228-21ksoftirqd/211:39:222
25476992422,1cyclictest0-21swapper/307:26:183
25476992422,1cyclictest0-21swapper/307:17:323
25476992422,1cyclictest0-21swapper/307:17:313
2547699241,5cyclictest0-21swapper/311:43:073
2547599248,6cyclictest28-21ksoftirqd/207:43:152
2547599247,7cyclictest28-21ksoftirqd/212:26:242
2547599245,7cyclictest28-21ksoftirqd/212:14:412
25475992422,1cyclictest0-21swapper/207:26:082
25475992420,3cyclictest271rcuc/212:16:302
25475992420,3cyclictest271rcuc/211:10:022
25475992420,3cyclictest271rcuc/210:00:162
25475992419,3cyclictest271rcuc/211:33:592
25475992419,3cyclictest271rcuc/208:45:322
2547599240,7cyclictest1838-21ssh11:02:432
2547499242,7cyclictest0-21swapper/110:25:001
232292240,2sleep10-21swapper/110:05:261
2547699231,8cyclictest4140-21perl10:40:163
2547699231,7cyclictest9313-21users12:20:263
2547699231,5cyclictest2031-21df09:50:133
25476992315,6cyclictest13036-21diskstats07:55:163
25476992313,7cyclictest26660-21memory11:45:193
25476992313,7cyclictest24939-21users08:20:243
2547599234,7cyclictest28-21ksoftirqd/209:00:142
2547599233,5cyclictest28-21ksoftirqd/211:05:472
2547599232,7cyclictest28-21ksoftirqd/210:13:062
25475992318,4cyclictest271rcuc/210:57:262
25475992318,3cyclictest271rcuc/208:51:532
2547499233,14cyclictest31394-21ssh11:25:041
2547499233,14cyclictest14858-21apt-get08:00:121
25474992313,6cyclictest52850irq/140-enp3s0-10:40:161
2547699229,3cyclictest66050lldpd08:39:033
2547699227,2cyclictest0-21swapper/310:05:173
2547699225,3cyclictest12200-21diskmemload11:27:063
25476992216,4cyclictest34-21ksoftirqd/307:40:013
25476992213,6cyclictest4342-21fw_packets08:50:163
25476992213,6cyclictest278-21systemd-journal08:30:013
2547599223,8cyclictest28-21ksoftirqd/207:30:192
25475992218,3cyclictest271rcuc/208:10:122
2547599221,7cyclictest28-21ksoftirqd/208:34:502
25475992217,4cyclictest271rcuc/207:59:462
25475992217,3cyclictest271rcuc/208:06:122
25475992217,3cyclictest271rcuc/208:06:112
25475992212,7cyclictest4784-21memory07:35:202
2547599220,7cyclictest28-21ksoftirqd/208:17:272
2547499228,5cyclictest22-21ksoftirqd/112:25:261
2547499225,3cyclictest12200-21diskmemload09:33:101
2547499223,13cyclictest22715-21ssh10:30:131
2547499222,14cyclictest32446-21ssh11:50:121
2547499221,6cyclictest20949-21apt-get12:30:141
25474992214,4cyclictest1929-21sh12:15:211
2547399227,6cyclictest3247-21gltestperf10:15:170
2547399223,3cyclictest20629-21apt-get09:40:120
2547399221,9cyclictest242672sh12:32:030
25473992213,7cyclictest1633-21awk09:25:210
25473992213,6cyclictest3718-21awk10:40:140
2547399221,18cyclictest7148-21apt12:20:000
74272210,1sleep30-21swapper/308:55:253
2547699218,6cyclictest1382-21wc09:50:013
2547699215,3cyclictest141rcu_preempt12:15:183
2547699214,3cyclictest12200-21diskmemload12:11:173
2547699214,3cyclictest12200-21diskmemload09:34:493
2547699211,7cyclictest864-21seq12:39:283
2547699211,7cyclictest864-21seq12:39:283
25476992115,4cyclictest34-21ksoftirqd/311:20:003
25476992112,7cyclictest13497-21ntpq11:35:223
25476992112,6cyclictest3686-21cut08:50:003
25476992112,6cyclictest21490-21ls10:05:003
25475992118,2cyclictest271rcuc/207:48:022
2547599211,7cyclictest28-21ksoftirqd/208:37:032
25475992117,3cyclictest271rcuc/209:09:482
25475992117,3cyclictest271rcuc/208:25:542
2547499214,3cyclictest12200-21diskmemload11:43:261
2547499211,5cyclictest26675-21apt-get12:10:131
2547499211,3cyclictest141rcu_preempt12:05:121
25474992113,6cyclictest31359-21open_inodes11:00:211
2547399218,9cyclictest15913-21kworker/u8:009:55:100
2547399216,4cyclictest141rcu_preempt10:10:180
25473992113,6cyclictest465-21ntpq08:40:200
25473992113,6cyclictest25290-21smartctl11:20:150
25473992113,6cyclictest19362-21seq10:51:080
25473992112,6cyclictest19238-21tr11:40:130
2547399211,17cyclictest24545-21grep10:55:190
92392200,3sleep09238-21ssh12:20:250
64892200,2chrt0-21swapper/011:05:410
266882201,16sleep026704-21sh10:08:430
2547699208,8cyclictest2661-21apt-get10:15:133
2547699204,3cyclictest12200-21diskmemload12:08:433
2547699204,3cyclictest12200-21diskmemload11:56:273
2547699204,3cyclictest12200-21diskmemload11:56:273
2547699203,4cyclictest12200-21diskmemload10:28:173
2547699203,4cyclictest12200-21diskmemload10:28:173
2547699203,3cyclictest12200-21diskmemload09:42:003
2547699202,12cyclictest21279-21apt-get08:15:153
2547699201,5cyclictest24251-21chrt10:55:173
2547699201,5cyclictest10265-21apt-get10:45:113
25476992012,6cyclictest17821-21ls08:05:193
25476992012,6cyclictest17821-21ls08:05:193
25476992012,4cyclictest12200-21diskmemload11:14:103
25476992011,5cyclictest52950irq/141-enp3s0-11:00:123
25475992016,3cyclictest271rcuc/208:21:052
25475992016,3cyclictest271rcuc/208:00:242
2547599200,7cyclictest1362-21taskset08:43:372
2547599200,7cyclictest12149-21taskset07:51:212
2547499204,3cyclictest12200-21diskmemload12:00:351
2547499204,3cyclictest12200-21diskmemload11:56:151
2547499204,3cyclictest12200-21diskmemload11:56:151
2547499204,3cyclictest12200-21diskmemload11:31:371
2547499204,3cyclictest12200-21diskmemload09:35:011
2547499204,13cyclictest22260-21cron10:30:001
2547499204,13cyclictest22260-21cron10:30:001
2547499203,3cyclictest12200-21diskmemload11:06:061
2547499203,3cyclictest12200-21diskmemload10:14:321
2547499202,4cyclictest12200-21diskmemload10:48:541
2547499202,13cyclictest14001-21ssh09:10:251
2547499201,5cyclictest9560-21grep09:55:211
2547499201,4cyclictest141rcu_preempt09:40:111
2547499201,3cyclictest141rcu_preempt08:40:121
25474992012,6cyclictest9018-21grep10:20:001
25474992012,5cyclictest12200-21diskmemload10:02:471
25474992011,7cyclictest278-21systemd-journal08:20:001
2547499200,5cyclictest0-21swapper/111:19:211
2547499200,4cyclictest0-21swapper/108:32:311
2547399208,4cyclictest141rcu_preempt07:30:210
2547399208,10cyclictest12-21ksoftirqd/011:15:000
25473992012,6cyclictest6580-21expr07:40:120
25473992012,6cyclictest27163-21awk07:10:250
25473992012,6cyclictest12867-21cat07:55:140
25473992011,7cyclictest8641-21date09:30:230
25473992011,7cyclictest560-21dbus-daemon09:45:230
25473992011,7cyclictest11358-21open_inodes10:45:210
2547699193,3cyclictest12200-21diskmemload10:11:513
2547699193,3cyclictest12200-21diskmemload09:22:333
2547699191,5cyclictest8180-21taskset11:31:063
2547699191,5cyclictest0-21swapper/310:32:113
25476991913,3cyclictest20228-21grep08:10:223
25476991912,5cyclictest34-21ksoftirqd/311:50:223
25476991912,5cyclictest24886-21ls11:20:113
25476991912,4cyclictest12200-21diskmemload09:12:243
25476991911,6cyclictest11300-21awk09:05:233
2547699191,15cyclictest8903-21grep07:45:153
25476991910,6cyclictest27382-21seq07:12:183
2547499198,7cyclictest23436-21apt-get08:20:101
2547499193,3cyclictest141rcu_preempt09:25:211
2547499193,11cyclictest24031-21apt-get11:20:051
2547499192,11cyclictest1886-21apt-get07:30:131
25474991912,5cyclictest8494-21expr09:00:151
25474991911,6cyclictest10432-21grep09:05:151
25474991910,5cyclictest12200-21diskmemload12:35:131
25474991910,5cyclictest12200-21diskmemload12:35:131
2547399191,6cyclictest12832-21ssh11:35:160
25473991914,3cyclictest8225-21sh11:31:100
25473991914,3cyclictest28229-21gltestperf08:30:160
25473991914,3cyclictest12773-21sh10:22:060
25473991913,4cyclictest27119-21sh11:45:240
25473991913,4cyclictest23528-21sh09:17:420
25473991913,4cyclictest23514-21sh12:07:140
25473991913,4cyclictest16640-21sh12:02:000
25473991912,5cyclictest9678-21ls07:45:230
25473991912,5cyclictest30232-21sed10:35:180
25473991912,5cyclictest16435-21grep10:25:140
25473991912,5cyclictest16435-21grep10:25:130
25473991911,6cyclictest6572-21smartctl08:55:160
25473991911,6cyclictest27120-21seq08:25:410
25473991911,6cyclictest20945-21seq08:14:100
25473991911,6cyclictest15608-21sensors_temp12:25:240
25473991911,6cyclictest10960-21tune2fs07:50:150
25473991910,7cyclictest17456-21tune2fs08:05:150
25473991910,7cyclictest17456-21tune2fs08:05:150
2547699189,7cyclictest2877-21grep07:30:213
2547699187,9cyclictest52950irq/141-enp3s0-09:17:073
2547699186,2cyclictest34-21ksoftirqd/308:30:223
25476991812,4cyclictest8232-21cpuspeed_turbos09:00:153
25476991811,5cyclictest6483-21ls07:40:133
25476991810,6cyclictest1544-21date08:45:003
2547599180,6cyclictest28-21ksoftirqd/208:56:122
2547499189,7cyclictest9560-21tr07:45:221
2547499189,7cyclictest26787-21cut07:10:211
2547499181,7cyclictest25559-21/usr/sbin/munin08:25:221
25474991811,5cyclictest20298-21swap08:10:241
25474991811,5cyclictest14716-21cat07:59:591
25474991810,6cyclictest6634-21cat08:55:171
25473991813,3cyclictest25691-21sh10:31:420
25473991813,3cyclictest21254-21/usr/sbin/munin08:15:160
25473991812,4cyclictest15884-21sh09:35:380
25473991811,5cyclictest15179-21seq08:00:130
25473991810,6cyclictest4443-21hostname08:50:160
25473991810,6cyclictest2965-21cat08:45:250
2547399180,4cyclictest12-21ksoftirqd/009:05:000
2547699175,4cyclictest52950irq/141-enp3s0-08:00:173
25476991712,3cyclictest11069-21gltestperf07:50:173
2547499178,7cyclictest560-21dbus-daemon07:55:001
2547499178,7cyclictest29836-21/usr/sbin/munin08:35:251
25474991711,4cyclictest5755-21cron08:55:001
25474991710,5cyclictest18443-21awk08:05:251
25474991710,5cyclictest18443-21awk08:05:251
2547399173,12cyclictest5440-21tail07:35:260
25473991710,5cyclictest30169-21expr08:35:130
25473991710,5cyclictest11995-21seq09:07:340
25474991611,3cyclictest4001-21/usr/sbin/munin07:35:211
2547499161,10cyclictest6292-21chrt07:40:121
2547499161,10cyclictest1744-21apt-get08:45:121
25473991611,3cyclictest24520-21sendmail_mailqu08:20:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional