You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 00:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Mon Nov 17, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4062842212555,65sleep30-21swapper/307:06:123
406279828757,9sleep20-21swapper/207:05:392
406301128354,9sleep10-21swapper/107:08:211
406309428061,13sleep00-21swapper/007:09:240
406326599280,20cyclictest279784-21chrt12:09:353
406326599280,18cyclictest0-21swapper/310:01:193
406326599271,8cyclictest0-21swapper/310:30:213
406326599271,7cyclictest0-21swapper/312:20:353
406326599271,7cyclictest0-21swapper/311:23:473
406326599261,7cyclictest0-21swapper/312:27:383
406326599252,18cyclictest85210-21chrt10:05:153
406326599251,7cyclictest0-21swapper/311:34:393
406326599251,20cyclictest44108-21taskset09:36:333
406326599251,19cyclictest153099-21chrt10:49:173
406326599251,16cyclictest12727-21chrt09:16:083
2137482251,11sleep04192999-21diskmemload11:26:540
4063265992421,2cyclictest0-21swapper/311:50:013
661842231,15sleep066243-21ntpq09:50:220
661842231,15sleep066243-21ntpq09:50:210
406326599236,11cyclictest321069-21taskset12:32:123
406326599232,14cyclictest21730-21taskset09:23:133
406326599231,18cyclictest77185-21turbostat10:00:003
406326599231,18cyclictest77185-21turbostat09:59:593
406326599231,18cyclictest46547-21chrt09:40:133
406326599231,18cyclictest187655-21chrt11:10:173
406326599230,8cyclictest0-21swapper/310:22:523
3222692231,3sleep20-21swapper/212:33:482
1141122231,3sleep00-21swapper/010:22:410
406326599221,8cyclictest4167981-21taskset08:45:183
4063265992218,2cyclictest0-21swapper/310:26:593
406326599221,18cyclictest145925-21turbostat10:44:593
406326599221,17cyclictest4145114-21turbostat08:25:003
406326599221,16cyclictest159268-21chrt10:51:243
406326599221,14cyclictest199184-21chrt11:18:353
406326599220,2cyclictest0-21swapper/309:25:233
406326599220,1cyclictest0-21swapper/312:10:453
2868522221,3sleep00-21swapper/012:12:330
41134282211,4sleep24113634-21diskstats07:55:162
41134282211,4sleep24113634-21diskstats07:55:152
40964952211,5sleep24096659-21/usr/sbin/munin07:35:272
406326599212,14cyclictest92829-21taskset10:10:153
406326599212,13cyclictest5062-21taskset09:11:183
4063265992112,6cyclictest0-21swapper/312:03:393
406326599211,16cyclictest0-21swapper/311:59:153
406326599211,15cyclictest293381-21chrt12:15:243
406326599211,12cyclictest0-21swapper/311:06:173
406326599211,10cyclictest4102420-21taskset07:43:493
406326599211,10cyclictest0-21swapper/309:45:243
406326599210,1cyclictest0-21swapper/308:55:213
406326599210,17cyclictest228-1kworker/3:1H+kblockd08:06:383
406326599208,3cyclictest0-21swapper/308:00:133
406326599201,9cyclictest0-21swapper/311:25:233
406326599201,9cyclictest0-21swapper/311:05:003
406326599201,8cyclictest0-21swapper/310:36:423
4063265992017,1cyclictest0-21swapper/308:40:163
4063265992015,3cyclictest66550irq/131-enp1s0-TxRx-309:54:223
4063265992015,3cyclictest66550irq/131-enp1s0-TxRx-309:54:213
406326599201,16cyclictest0-21swapper/311:37:403
406326599201,16cyclictest0-21swapper/309:32:153
406326599201,15cyclictest243988-21chrt11:45:333
406326599201,15cyclictest238682-21chrt11:44:563
406326599201,15cyclictest0-21swapper/310:59:153
406326599201,14cyclictest328477-21chrt12:35:443
406326599201,11cyclictest4177190-21sleep08:54:593
1821552201,4sleep24192999-21diskmemload11:06:042
532632191,4sleep24192999-21diskmemload09:43:412
41942602191,3sleep10-21swapper/109:10:171
406326599191,15cyclictest4102529-21turbostat07:49:553
406326599191,13cyclictest106284-21taskset10:17:383
406326599191,12cyclictest66550irq/131-enp1s0-TxRx-307:38:013
2533812191,3sleep00-21swapper/011:52:010
2069862191,12sleep24192999-21diskmemload11:23:442
406326599181,9cyclictest15950irq/132-ahci[0000:00:17.0]07:55:153
406326599181,9cyclictest15950irq/132-ahci[0000:00:17.0]07:55:143
406326599181,8cyclictest0-21swapper/307:15:123
406326599181,13cyclictest4193006-21turbostat09:10:013
406326599181,13cyclictest4155978-21turbostat08:35:003
406326599181,11cyclictest4139580-21taskset08:16:453
3301572181,4sleep14192999-21diskmemload12:38:151
3297332181,12sleep20-21swapper/212:37:402
2777762181,3sleep20-21swapper/212:06:412
2546292181,4sleep24192999-21diskmemload11:53:512
2453192181,4sleep0245332-21ssh11:47:310
2074362181,3sleep10-21swapper/111:24:321
1514122181,5sleep14192999-21diskmemload10:46:061
1051312181,2sleep00-21swapper/010:15:340
802022171,8sleep281491-21memory10:00:212
54942171,12sleep25506-21ssh09:12:072
406326599171,8cyclictest0-21swapper/307:50:133
4063265991711,3cyclictest40861332sleep307:26:353
406326599171,13cyclictest228-1kworker/3:1H+kblockd08:27:133
406326599171,11cyclictest4080404-21taskset07:20:253
406326599170,14cyclictest0-21swapper/307:10:213
354182171,11sleep235689-21unixbench_singl09:30:262
2775562170,3sleep00-21swapper/012:06:230
2221202171,4sleep24192999-21diskmemload11:33:232
406326599161,9cyclictest4182717-21taskset09:00:003
406326599161,10cyclictest4161155-21chrt08:38:453
406326599160,13cyclictest228-1kworker/3:1H+kblockd08:10:083
406326599160,13cyclictest0-21swapper/307:32:293
3018312161,10sleep0301876-21ntpq12:20:230
2370232160,1sleep1237037-21ssh11:41:551
1606212161,9sleep04192999-21diskmemload10:53:440
1136792161,2sleep10-21swapper/110:21:581
989302151,8sleep14192999-21diskmemload10:13:001
851802151,6sleep24192999-21diskmemload10:05:152
845272151,3sleep04192999-21diskmemload10:04:330
40960462151,4sleep04096093-21ntpq07:35:210
357592151,9sleep04192999-21diskmemload09:30:300
3218592151,9sleep14192999-21diskmemload12:33:151
272742151,9sleep227328-21ntpq09:25:222
2296952150,1sleep10-21swapper/111:38:021
2136192151,8sleep14192999-21diskmemload11:26:391
1918952150,2sleep00-21swapper/011:14:320
1223342151,6sleep20-21swapper/210:28:322
537432141,3sleep0151ktimers/009:44:330
41362842141,8sleep04138069-21cut08:15:210
40676092140,1sleep20-21swapper/207:10:232
388742147,3sleep138666-21/usr/sbin/munin09:35:161
2651222141,8sleep0266731-21grep12:00:190
1467882141,8sleep0147230-21cut10:45:170
76682131,6sleep17802-21diskstats09:15:151
41785172131,7sleep04180237-21kthreadcore08:55:170
41620872131,7sleep14162141-21kthreadcore08:40:151
41515222131,3sleep14150951-21/usr/sbin/munin08:30:171
41416102131,8sleep14143388-21cut08:20:191
41139182131,7sleep14115179-21kthreadcore07:55:191
41139182131,7sleep14115179-21kthreadcore07:55:181
351152131,7sleep135173-21ntpq09:30:211
3118022131,6sleep24192999-21diskmemload12:26:172
222002130,1sleep20-21swapper/209:24:132
2053062131,7sleep00-21swapper/011:20:460
1665732131,7sleep14192999-21diskmemload10:56:041
1372532131,8sleep24192999-21diskmemload10:36:542
1307452131,7sleep0130753-21ssh10:34:400
1216122131,5sleep00-21swapper/010:27:050
1053332131,6sleep10-21swapper/110:15:551
712352123,4sleep272872-21cut09:55:212
712352123,4sleep272872-21cut09:55:202
524552121,3sleep10-21swapper/109:42:201
41919742121,5sleep24192012-21ntpq09:05:222
41276652121,8sleep00-21swapper/008:05:200
41086712121,7sleep14110328-21kthreadcore07:50:181
40969282121,6sleep14096924-40gst-plugin-scan07:37:551
393092121,5sleep040776-21ps09:35:180
3030542121,7sleep1303065-21ssh12:21:081
2967812127,2sleep00-21swapper/012:19:220
2961082120,3sleep10-21swapper/112:18:231
2885152121,4sleep24192999-21diskmemload12:14:482
2863202121,5sleep14192999-21diskmemload12:11:501
2610702121,3sleep10-21swapper/111:56:101
2534822121,4sleep14192999-21diskmemload11:52:101
2465502121,3sleep10-21swapper/111:49:241
2395522121,4sleep20-21swapper/211:45:162
2386692120,5sleep20-21swapper/211:44:572
1988012120,1sleep00-21swapper/011:17:560
1979382121,7sleep1197943-21ssh11:16:251
1907482121,5sleep20-21swapper/211:12:262
1904362120,2sleep10-21swapper/111:11:531
1753502120,6sleep20-21swapper/211:02:442
1745472121,5sleep00-21swapper/011:01:210
1744322121,7sleep1174445-21rm11:01:101
1609542121,5sleep24192999-21diskmemload10:54:262
133702121,3sleep00-21swapper/009:17:140
1304112121,5sleep24192999-21diskmemload10:34:052
1062762121,4sleep24192999-21diskmemload10:17:382
985922111,3sleep00-21swapper/010:12:230
912352114,2sleep04063259-21cyclictest10:07:570
836972113,3sleep1331ktimers/110:02:461
704402111,6sleep00-21swapper/009:55:190
704402111,6sleep00-21swapper/009:55:180
684332111,4sleep24192999-21diskmemload09:53:162
684332111,4sleep24192999-21diskmemload09:53:152
614222110,1sleep00-21swapper/009:49:330
603762111,3sleep20-21swapper/209:47:432
599952111,4sleep14192999-21diskmemload09:46:581
54822111,4sleep04192999-21diskmemload09:12:050
450622111,3sleep20-21swapper/209:38:052
41504122111,5sleep04150488-21munin-run08:30:000
41391512110,7sleep10-21swapper/108:15:311
41343902110,1sleep20-21swapper/208:12:472
3130542111,3sleep10-21swapper/112:28:011
3129412111,5sleep04192999-21diskmemload12:27:520
299802111,3sleep10-21swapper/109:28:581
2936072111,4sleep24192999-21diskmemload12:15:272
2762272111,5sleep1276283-21ntpq12:05:241
2696822111,3sleep10-21swapper/112:01:481
2696502111,4sleep24192999-21diskmemload12:01:452
2631622110,7sleep0263173-21ssh11:59:170
2630442111,3sleep20-21swapper/211:59:072
2302592111,3sleep20-21swapper/211:38:592
2292572111,3sleep04192999-21diskmemload11:37:160
2211922111,3sleep00-21swapper/011:31:200
207392111,3sleep10-21swapper/109:21:321
1978642111,3sleep20-21swapper/211:16:182
1828802111,3sleep14192999-21diskmemload11:07:191
1780092111,6sleep0179614-21kthreadcore11:05:170
1664742111,3sleep24192999-21diskmemload10:55:532
1538512111,4sleep14192999-21diskmemload10:50:121
1530882111,4sleep24192999-21diskmemload10:49:172
152732111,3sleep00-21swapper/009:20:110
1450812110,4sleep14192999-21diskmemload10:42:301
1443332111,4sleep24192999-21diskmemload10:41:152
1369162111,3sleep10-21swapper/110:36:161
1291812111,3sleep10-21swapper/110:31:561
1137702111,4sleep24192999-21diskmemload10:22:062
982152101,2sleep20-21swapper/210:11:392
919832101,2sleep10-21swapper/110:09:181
744432101,3sleep10-21swapper/109:55:281
744432101,3sleep10-21swapper/109:55:271
621532101,4sleep162367-21diskstats09:50:151
621532101,4sleep162367-21diskstats09:50:151
41827572101,5sleep14182992-21cat09:00:151
41510732101,4sleep24151238-21diskstats08:30:142
41300672101,5sleep04130651-21tr08:10:160
41225432101,4sleep14122649-21ntpq08:00:191
41067652101,4sleep24106816-21ntpq07:45:212
40979852100,5sleep04098826-21cut07:40:190
40974472101,4sleep14097705-21df_inode07:40:161
40811232101,5sleep14081329-21date07:25:011
40707722101,4sleep24070791-21apt-get07:15:122
3299462101,2sleep00-21swapper/012:37:580
3191972101,5sleep0319254-21/usr/sbin/munin12:30:250
3026072101,3sleep20-21swapper/212:20:332
2370612100,3sleep00-21swapper/011:41:580
2196002101,5sleep1219896-21tail11:30:211
2134452101,4sleep24192999-21diskmemload11:26:162
1655802101,4sleep0165636-21ntpq10:55:230
1431382101,3sleep00-21swapper/010:40:210
1381592101,3sleep00-21swapper/010:39:120
1207562101,3sleep10-21swapper/110:25:381
4173224291,4sleep14174951-21cut08:50:171
4170881291,4sleep24170929-21ntpq08:45:222
4123435290,1sleep20-21swapper/208:02:132
4112150291,4sleep24112203-21ntp_states07:50:232
4080770290,1sleep20-21swapper/207:20:382
4067504291,4sleep04067551-21ntpq07:10:220
4182107280,1sleep20-21swapper/208:55:442
4166587281,3sleep04166611-21munin-plugin-st08:45:000
4162647281,3sleep24164372-21cut08:40:162
4140690281,3sleep04140728-21kthreadcore08:20:190
4124763281,3sleep24125857-21cat08:05:182
4124170281,3sleep14123668-21apt-get08:05:091
4108568281,4sleep04108666-21tail07:50:170
4100595281,3sleep20-21swapper/207:40:212
4081043280,6sleep00-21swapper/007:24:040
4069452281,3sleep14070758-21memory07:15:021
4065640281,3sleep10-21swapper/107:10:191
28085280,4sleep028099-21ssh09:25:430
12600280,5sleep20-21swapper/209:15:562
4187643270,1sleep20-21swapper/209:03:452
4187508270,1sleep00-21swapper/009:02:030
4182227270,1sleep10-21swapper/108:57:161
4172157270,2sleep00-21swapper/008:50:130
4157866271,2sleep10-21swapper/108:35:191
4149433271,2sleep24149482-21ntpq08:25:222
4145365270,2sleep10-21swapper/108:25:131
4135234271,3sleep20-21swapper/208:15:202
4130653271,2sleep10-21swapper/108:10:171
4122974271,3sleep04123204-21unixbench_multi08:00:260
4091349270,6sleep10-21swapper/107:30:471
4088744271,2sleep20-21swapper/207:30:192
4086122270,1sleep20-21swapper/207:26:262
4081836271,2sleep14081952-21irqstats07:25:161
4071053271,3sleep00-21swapper/007:15:130
4192990260,2sleep04192999-21diskmemload09:10:000
4161220260,1sleep20-21swapper/208:39:372
4156778261,2sleep00-21swapper/008:35:170
4154899260,2sleep00-21swapper/008:30:210
4144420261,2sleep24140124-21/usr/sbin/munin08:20:282
4113846260,1sleep00-21swapper/007:55:180
4113846260,1sleep00-21swapper/007:55:170
4106645261,2sleep10-21swapper/107:45:201
4091383260,1sleep00-21swapper/007:31:140
4084873260,2sleep04085358-21strings07:25:190
4105287250,1sleep00-21swapper/007:45:190
4187743240,1sleep10-21swapper/109:05:011
4172435240,1sleep20-21swapper/208:50:142
4167129240,1sleep00-21swapper/008:45:150
4166612240,1sleep10-21swapper/108:45:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional