You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-26 - 16:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Aug 26, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
151152237165,24sleep20-21swapper/207:06:172
152692235163,59sleep00-21swapper/007:08:150
151012202166,24sleep30-21swapper/307:06:053
150642202165,24sleep10-21swapper/107:05:351
1586821740,4sleep21549999cyclictest08:20:202
2773621700,2sleep01548799cyclictest11:15:170
236612920,1sleep30-21swapper/307:25:153
15499995956,2cyclictest0-21swapper/207:25:492
172262560,2sleep20-21swapper/207:10:252
318212550,1sleep20-21swapper/211:25:132
54542540,5sleep05452-21netstat09:10:190
111632530,5sleep21549999cyclictest08:10:172
102452530,1sleep00-21swapper/009:20:260
42752500,2sleep30-21swapper/311:35:183
212072450,1sleep30-21swapper/311:00:183
255162410,2sleep10-21swapper/111:10:151
15487993711,12cyclictest27158-21munin-run07:35:010
20402340,5sleep30-21swapper/310:15:243
1549999291,6cyclictest0-21swapper/210:52:182
1548799293,12cyclictest4204-21diskmemload09:20:010
230932281,3sleep123184-21/usr/sbin/munin09:50:221
1550599288,2cyclictest87350irq/126-enp1s0-11:30:243
15487992818,3cyclictest0-21swapper/011:26:290
15487992717,3cyclictest0-21swapper/009:57:510
1549999261,5cyclictest0-21swapper/210:25:382
1549999260,6cyclictest31084-21kworker/u8:111:32:062
1549599261,21cyclictest18646-21perf12:10:011
1549599260,20cyclictest0-21swapper/109:57:061
1548799262,12cyclictest3246-21munin-run11:35:010
15487992618,3cyclictest0-21swapper/010:52:180
15487992615,3cyclictest0-21swapper/009:43:390
110502260,5sleep121-21rcuc/111:50:211
224382250,1sleep00-21swapper/011:05:020
1549999255,3cyclictest0-21swapper/211:00:142
1549999251,6cyclictest0-21swapper/210:32:212
1549999251,5cyclictest0-21swapper/212:32:142
1549999251,5cyclictest0-21swapper/210:07:592
1548799257,5cyclictest0-21swapper/012:08:280
1550599241,5cyclictest0-21swapper/309:50:473
1550599241,11cyclictest15692-21/usr/sbin/munin07:10:253
1549999242,5cyclictest0-21swapper/209:43:462
15499992411,3cyclictest87650irq/129-enp1s0-12:35:132
1549999240,3cyclictest4652-21kworker/u8:010:46:412
1549599245,11cyclictest755-21rngd12:20:201
1549599245,11cyclictest755-21rngd12:20:201
1549599241,5cyclictest16390-21diskstats10:50:141
1548799247,5cyclictest0-21swapper/010:01:130
1548799247,5cyclictest0-21swapper/010:01:130
1548799242,6cyclictest0-21swapper/010:20:330
1548799241,11cyclictest13690-21fschecks_time09:30:140
1548799240,11cyclictest12383-21munin-run08:15:010
1549999232,5cyclictest8564-1kworker/u9:110:18:492
1549599231,9cyclictest0-21swapper/109:24:321
15495992317,3cyclictest27475-21ntp_states10:00:201
15495992317,3cyclictest27475-21ntp_states10:00:201
15495992313,3cyclictest0-21swapper/111:32:071
1549599231,19cyclictest3822-21cpu07:55:111
1549599231,19cyclictest28968-21diskstats10:05:141
1549599231,19cyclictest28211-21perl12:30:141
1548799237,6cyclictest0-21swapper/011:38:020
15487992313,3cyclictest0-21swapper/009:25:400
1548799231,11cyclictest7250-21/usr/sbin/munin10:30:250
1548799231,10cyclictest24841-21/usr/sbin/munin11:10:200
67432221,3sleep06816-21irqstats08:00:170
19852221,3sleep22245-21sed07:50:172
1550599221,8cyclictest0-21swapper/310:08:293
1549999221,7cyclictest8564-1kworker/u9:112:18:002
1549999221,6cyclictest0-21swapper/211:37:102
1549999221,5cyclictest0-21swapper/209:51:122
1549999221,3cyclictest0-21swapper/209:57:512
1549599221,9cyclictest0-21swapper/108:40:491
15495992215,2cyclictest0-21swapper/111:05:171
1549599221,19cyclictest0-21swapper/108:04:551
1549599221,19cyclictest0-21swapper/107:42:111
1549599221,18cyclictest16421-21taskset12:04:541
15495992210,3cyclictest755-21rngd10:37:241
1548799227,3cyclictest0-21swapper/008:18:550
1548799221,4cyclictest19230-21head10:55:190
1548799221,10cyclictest554-21/usr/sbin/munin10:15:130
262522210,7sleep121-21rcuc/112:25:161
1550599215,9cyclictest0-21swapper/309:14:163
1550599213,2cyclictest41-21ksoftirqd/309:49:553
1550599212,2cyclictest4652-21kworker/u8:010:49:323
1550599211,7cyclictest0-21swapper/310:20:263
1550599211,7cyclictest0-21swapper/309:56:093
1550599211,6cyclictest0-21swapper/310:57:483
1550599211,6cyclictest0-21swapper/307:40:153
1550599211,11cyclictest808-21snmpd07:46:523
1549999211,5cyclictest8564-1kworker/u9:109:31:492
1549999210,6cyclictest8564-1kworker/u9:112:25:452
1549999210,10cyclictest8564-1kworker/u9:112:05:122
1549599218,4cyclictest14806-21users09:30:251
1549599217,3cyclictest29590-21vmstat08:50:271
1549599216,13cyclictest18657-21kworker/u8:108:58:351
1549599214,8cyclictest31074-21sshd11:23:301
1549599213,6cyclictest0-21swapper/110:10:401
1549599211,6cyclictest13249-21strings11:55:191
15495992113,4cyclictest0-21swapper/110:46:411
1549599211,18cyclictest0-21swapper/111:36:361
1549599211,18cyclictest0-21swapper/111:27:571
1549599211,18cyclictest0-21swapper/109:44:061
1549599211,18cyclictest0-21swapper/107:45:211
15495992111,3cyclictest0-21swapper/110:32:211
15495992111,3cyclictest0-21swapper/109:35:501
1548799218,5cyclictest0-21swapper/011:43:040
1548799217,6cyclictest0-21swapper/010:45:310
1548799211,6cyclictest20785-21/usr/sbin/munin07:20:130
15487992116,3cyclictest27253-21grep08:45:240
1548799211,11cyclictest0-21swapper/011:59:470
1548799211,10cyclictest0-21swapper/008:56:500
1548799210,19cyclictest0-21swapper/012:14:300
1548799210,15cyclictest0-21swapper/011:46:520
194122200,7sleep2311ktimersoftd/208:30:122
1550599205,9cyclictest20503-40dnf07:19:323
1550599205,5cyclictest24944-21cpuspeed_turbos11:10:123
1550599205,4cyclictest755-21rngd12:05:213
1550599203,8cyclictest0-21swapper/312:29:143
1550599203,6cyclictest32322-21sed10:10:253
1550599202,11cyclictest839-21lldpd10:04:143
1550599202,11cyclictest839-21lldpd10:04:143
1550599202,10cyclictest27039-21/usr/sbin/munin11:15:183
1550599201,9cyclictest0-21swapper/311:59:353
1550599201,7cyclictest0-21swapper/309:32:573
1550599201,3cyclictest0-21swapper/312:01:293
1550599201,11cyclictest20785-21/usr/sbin/munin07:20:173
1550599201,11cyclictest17618-21/usr/sbin/munin09:40:173
1550599201,11cyclictest17001-21/usr/sbin/munin08:25:193
1550599201,11cyclictest1602-21/usr/sbin/munin07:50:173
1550599200,11cyclictest401ktimersoftd/307:35:013
1549999207,3cyclictest0-21swapper/208:43:402
1549999201,8cyclictest0-21swapper/211:50:202
1549999201,7cyclictest0-21swapper/212:13:202
1549999201,6cyclictest8564-1kworker/u9:111:42:372
1549999201,6cyclictest0-21swapper/212:22:482
1549999201,6cyclictest0-21swapper/212:22:472
1549999201,6cyclictest0-21swapper/210:23:452
1549999201,6cyclictest0-21swapper/210:01:562
1549999201,6cyclictest0-21swapper/210:01:562
1549999201,6cyclictest0-21swapper/209:17:322
1549999201,5cyclictest8564-1kworker/u9:111:58:012
1549999201,5cyclictest0-21swapper/209:25:402
1549999201,16cyclictest32548-21tr09:00:122
15495992016,2cyclictest20841-21perf12:15:011
1549599201,4cyclictest0-21swapper/109:25:251
1549599201,2cyclictest4262-21wc09:10:011
15495992011,3cyclictest0-21swapper/110:15:481
1549599201,11cyclictest23030-21latency_hist12:20:011
1548799202,9cyclictest0-21swapper/012:31:260
1548799201,8cyclictest0-21swapper/010:13:300
1548799201,17cyclictest0-21swapper/010:07:590
1548799201,11cyclictest25824-21fschecks_count12:25:130
1548799201,10cyclictest0-21swapper/011:54:330
1548799201,10cyclictest0-21swapper/009:45:190
1548799200,9cyclictest0-21swapper/011:20:230
1548799200,8cyclictest0-21swapper/012:00:030
1548799200,8cyclictest0-21swapper/009:52:330
1548799200,19cyclictest0-21swapper/010:25:260
1548799200,18cyclictest0-21swapper/012:15:140
176522191,3sleep017915-21timerwakeupswit07:15:020
1550599195,2cyclictest87650irq/129-enp1s0-12:21:373
1550599195,2cyclictest87650irq/129-enp1s0-12:21:373
1550599195,2cyclictest39-21rcuc/309:10:013
1550599194,3cyclictest39-21rcuc/307:35:153
1550599192,4cyclictest39-21rcuc/310:50:123
1550599191,8cyclictest6322-21swap10:25:263
1550599191,7cyclictest0-21swapper/311:07:053
1550599191,6cyclictest0-21swapper/312:33:313
1550599191,6cyclictest0-21swapper/311:51:033
1550599191,5cyclictest0-21swapper/309:39:313
1550599191,3cyclictest0-21swapper/311:22:413
1550599191,10cyclictest6627-21/usr/sbin/munin09:15:123
1550599191,10cyclictest31431-21/usr/sbin/munin11:25:233
1550599191,10cyclictest21401-21/usr/sbin/munin08:35:283
1550599191,10cyclictest18866-21/usr/sbin/munin12:10:133
1550599191,10cyclictest11639-21/usr/sbin/munin10:40:213
1550599191,10cyclictest0-21swapper/308:50:233
1550599190,10cyclictest0-21swapper/311:45:543
1549999198,3cyclictest0-21swapper/211:45:262
1549999195,4cyclictest4420-21gltestperf07:55:152
1549999192,7cyclictest30-21rcuc/208:45:232
1549999191,8cyclictest0-21swapper/207:19:192
1549999191,6cyclictest31958-21pmu-power10:10:222
1549999191,6cyclictest0-21swapper/211:16:512
1549999191,5cyclictest0-21swapper/211:08:522
1549599197,3cyclictest0-21swapper/111:41:391
1549599196,2cyclictest0-21swapper/109:10:211
1549599196,12cyclictest5601-21kworker/u8:008:07:351
1549599194,12cyclictest19167-21missed_timers10:55:181
1549599193,12cyclictest4462-40dnf10:20:331
15495991916,2cyclictest0-21swapper/108:46:401
15495991916,2cyclictest0-21swapper/108:37:291
15495991915,3cyclictest0-21swapper/110:44:151
15495991914,3cyclictest23116-21grep07:25:121
15495991913,3cyclictest7455-21uname09:15:191
1549599191,17cyclictest0-21swapper/107:36:181
1549599191,16cyclictest0-21swapper/111:45:231
1549599191,16cyclictest0-21swapper/110:25:051
1549599191,15cyclictest27155-21cpuspeed_turbos11:15:121
1549599190,18cyclictest0-21swapper/107:50:221
1549599190,18cyclictest0-21swapper/107:34:001
1548799198,9cyclictest24626-21awk11:10:020
1548799195,5cyclictest0-21swapper/012:39:050
1548799192,9cyclictest0-21swapper/010:39:430
1548799191,5cyclictest12318-21smartctl10:40:140
1548799191,15cyclictest25279-21sshd08:41:300
1548799190,3cyclictest0-21swapper/009:37:140
52402181,2sleep20-21swapper/209:10:182
160892180,3sleep20-21swapper/212:00:472
1550599186,4cyclictest401ktimersoftd/312:15:223
1550599182,3cyclictest8318-21kworker/u8:009:20:253
1550599182,15cyclictest0-21swapper/308:21:543
1550599181,8cyclictest10310-21cat08:10:023
1550599181,4cyclictest0-21swapper/309:26:063
15505991812,3cyclictest9573-21cpuspeed_turbos10:35:143
15505991812,3cyclictest808-21snmpd08:16:593
1550599181,11cyclictest0-21swapper/307:55:133
1550599181,10cyclictest839-21lldpd08:48:423
1549999186,4cyclictest13161-21grep08:15:152
1549999185,5cyclictest7379-21timerandwakeup08:00:242
1549999184,6cyclictest32-21ksoftirqd/207:40:242
1549999182,6cyclictest21956-21perl08:35:162
1549999181,8cyclictest30-21rcuc/207:35:142
1549999181,7cyclictest29383-21cpuspeed_turbos11:20:122
1549999181,6cyclictest0-21swapper/210:56:192
1549999181,6cyclictest0-21swapper/210:42:222
1549999181,5cyclictest0-21swapper/208:55:232
15499991812,3cyclictest10059-21fgrep09:20:232
15499991811,4cyclictest15795-21fschecks_count09:35:152
1549999181,10cyclictest0-21swapper/208:27:332
15495991814,3cyclictest0-21swapper/109:45:211
1549599181,3cyclictest0-21swapper/111:01:491
1549599181,16cyclictest0-21swapper/109:00:421
1549599181,16cyclictest0-21swapper/108:33:341
1549599181,16cyclictest0-21swapper/108:23:161
1549599181,16cyclictest0-21swapper/108:15:431
1549599181,16cyclictest0-21swapper/108:11:241
1549599181,16cyclictest0-21swapper/107:12:561
1549599181,15cyclictest0-21swapper/108:26:201
1549599181,14cyclictest21699-21memory07:20:181
1549599180,17cyclictest0-21swapper/112:35:111
1548799189,5cyclictest17660-21latency_hist07:15:020
1548799186,3cyclictest0-21swapper/007:50:570
1548799184,8cyclictest0-21swapper/007:45:480
1548799183,3cyclictest87550irq/128-enp1s0-08:35:150
1548799182,2cyclictest28367-21needreboot07:35:190
1548799182,13cyclictest32382-21/usr/sbin/munin09:00:220
1548799181,3cyclictest0-21swapper/008:25:130
1548799181,15cyclictest0-21swapper/008:50:510
1548799180,10cyclictest23562-21expr12:20:140
1548799180,10cyclictest23562-21expr12:20:140
288012171,13sleep228902-21memory08:50:192
269192171,2sleep20-21swapper/207:32:142
214502171,2sleep20-21swapper/209:45:512
1550599171,8cyclictest87350irq/126-enp1s0-08:00:113
1550599171,7cyclictest32382-21/usr/sbin/munin09:00:223
1550599171,7cyclictest0-21swapper/308:56:303
1550599171,6cyclictest8062-21irqstats10:30:193
1550599171,6cyclictest11475-21ntp_states08:10:233
1550599171,6cyclictest0-21swapper/311:40:203
1550599171,6cyclictest0-21swapper/308:30:113
1549999173,5cyclictest9796-21users08:05:262
1549999173,5cyclictest30-21rcuc/207:45:242
1549999171,5cyclictest0-21swapper/210:35:252
1549999171,3cyclictest0-21swapper/211:11:242
1549599170,15cyclictest0-21swapper/107:15:021
1548799171,3cyclictest0-21swapper/009:09:480
1548799171,14cyclictest5793-21perf08:00:010
1548799171,13cyclictest8209-21/usr/sbin/munin08:05:120
15487991711,3cyclictest30438-21cat07:40:190
15487991710,4cyclictest20047-21/usr/sbin/munin08:30:170
1550599169,4cyclictest24835-21/usr/sbin/munin08:40:223
1550599161,14cyclictest0-21swapper/312:38:393
1549999161,6cyclictest21119-21df07:20:122
15499991612,3cyclictest0-21swapper/209:06:082
1548799168,6cyclictest23119-21cpuspeed_turbos07:25:110
1548799159,3cyclictest15120-21expr08:20:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional