You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 15:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Mar 11, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1112648212352,67sleep30-21swapper/307:07:183
135406521210,6sleep00-21swapper/010:11:240
111278727956,6sleep00-21swapper/007:09:000
111265927860,12sleep10-21swapper/107:07:251
111264727860,12sleep20-21swapper/207:07:182
1112988992411,10cyclictest1541358-21turbostat12:04:553
111298899221,3cyclictest0-21swapper/311:20:463
111298899211,15cyclictest0-21swapper/309:40:133
111298899208,9cyclictest1141552-21turbostat07:39:553
111298899201,16cyclictest0-21swapper/311:55:193
111298899201,16cyclictest0-21swapper/311:55:183
111298899201,15cyclictest1416358-21taskset10:46:463
111298899201,12cyclictest0-21swapper/309:45:573
111298899200,2cyclictest0-21swapper/311:51:543
111298899200,17cyclictest0-21swapper/309:32:083
14697712191,4sleep00-21swapper/011:17:590
111298899199,6cyclictest1173397-21taskset08:01:453
111298899198,7cyclictest1168317-21turbostat07:59:593
111298899191,7cyclictest1608874-21chrt12:36:153
1112988991916,1cyclictest0-21swapper/311:15:163
111298899191,15cyclictest0-21swapper/309:16:503
111298899191,15cyclictest0-21swapper/309:16:503
111298899191,14cyclictest66950irq/132-enp1s0-TxRx-309:10:203
111298899191,14cyclictest0-21swapper/311:01:023
111298899191,14cyclictest0-21swapper/310:50:193
111298899191,14cyclictest0-21swapper/310:02:423
1112988991910,7cyclictest0-21swapper/309:20:093
111298899181,5cyclictest0-21swapper/307:26:423
1112988991814,2cyclictest0-21swapper/311:34:203
1112988991814,2cyclictest0-21swapper/309:37:463
111298899181,14cyclictest1353942-21chrt10:11:153
111298899181,14cyclictest0-21swapper/312:22:113
111298899181,14cyclictest0-21swapper/312:18:023
111298899181,14cyclictest0-21swapper/312:10:133
111298899181,14cyclictest0-21swapper/311:45:443
111298899181,14cyclictest0-21swapper/311:35:523
111298899181,14cyclictest0-21swapper/311:27:273
111298899181,14cyclictest0-21swapper/311:11:183
111298899181,14cyclictest0-21swapper/310:57:213
111298899181,14cyclictest0-21swapper/310:44:143
111298899181,14cyclictest0-21swapper/310:31:293
111298899181,14cyclictest0-21swapper/310:16:023
111298899180,16cyclictest0-21swapper/309:56:133
111298899180,15cyclictest0-21swapper/312:27:023
16100612171,3sleep20-21swapper/212:37:512
15229242170,4sleep11243187-21diskmemload11:48:481
15052522171,3sleep20-21swapper/211:38:392
13546852171,2sleep20-21swapper/210:12:112
12836482170,8sleep11283659-21ssh09:30:421
111298899172,11cyclictest1243208-21turbostat09:10:003
111298899171,14cyclictest0-21swapper/310:35:063
111298899171,14cyclictest0-21swapper/309:50:033
111298899171,13cyclictest0-21swapper/312:30:113
111298899171,13cyclictest0-21swapper/311:42:053
111298899171,13cyclictest0-21swapper/308:25:183
111298899171,12cyclictest1152361-21turbostat07:45:013
111298899170,15cyclictest0-21swapper/310:24:293
111298899170,14cyclictest0-21swapper/310:25:023
111298899170,14cyclictest0-21swapper/309:25:583
15645562161,4sleep11243187-21diskmemload12:10:311
15134742161,2sleep00-21swapper/011:42:520
14958612160,1sleep00-21swapper/011:32:390
14343052165,5sleep1331ktimers/110:57:251
14337022161,2sleep20-21swapper/210:56:372
13719362161,3sleep00-21swapper/010:21:520
13648952160,2sleep00-21swapper/010:19:070
13613972162,7sleep11361456-21ntpq10:15:191
13439742161,10sleep11344039-21ntpq10:05:211
111298899167,5cyclictest1210924-21turbostat08:40:003
111298899161,3cyclictest0-21swapper/312:05:193
111298899161,13cyclictest0-21swapper/311:05:213
1112988991610,4cyclictest0-21swapper/307:30:273
111298899160,13cyclictest0-21swapper/310:05:083
111298899160,13cyclictest0-21swapper/308:50:163
111298899160,13cyclictest0-21swapper/308:47:403
111298899160,13cyclictest0-21swapper/308:05:253
111298899160,13cyclictest0-21swapper/307:15:193
14686752151,4sleep21243187-21diskmemload11:16:342
14267112151,2sleep20-21swapper/210:53:572
14099822151,2sleep20-21swapper/210:44:592
13753042151,9sleep11375526-21hddtemp_smartct10:25:131
12781782151,4sleep01278211-21munin-run09:30:000
12486142151,4sleep21248628-21ssh09:10:382
12112042151,6sleep21211444-21cstates08:40:122
111298899150,3cyclictest0-21swapper/307:50:013
111298899150,13cyclictest0-21swapper/308:15:133
111298899150,13cyclictest0-21swapper/307:12:083
111298899150,12cyclictest0-21swapper/308:20:163
111298899150,12cyclictest0-21swapper/308:10:183
15564802146,3sleep01243187-21diskmemload12:06:510
15477382141,3sleep10-21swapper/112:01:511
15371322141,6sleep21537190-21ntpq11:55:182
15371322141,6sleep21537190-21ntpq11:55:182
14533782140,4sleep20-21swapper/211:09:282
14170552140,2sleep20-21swapper/210:47:432
12752772141,8sleep11243187-21diskmemload09:26:131
12327892141,5sleep11232790-21/usr/sbin/munin09:00:111
12197872141,3sleep20-21swapper/208:45:172
111298899143,7cyclictest0-21swapper/308:55:403
1112988991411,2cyclictest0-21swapper/307:20:163
1112988991411,1cyclictest0-21swapper/309:01:183
111298899140,13cyclictest0-21swapper/307:45:103
111298899140,12cyclictest0-21swapper/308:30:133
111298899140,10cyclictest0-21swapper/308:40:203
15927512131,2sleep10-21swapper/112:28:041
15733992130,2sleep00-21swapper/012:15:410
15316532131,2sleep10-21swapper/111:53:441
13907522131,8sleep21390766-21sh10:33:082
13808912131,5sleep20-21swapper/210:26:432
13290572130,1sleep20-21swapper/209:58:012
12926812131,8sleep21292692-21ssh09:36:042
12866322130,2sleep00-21swapper/009:34:370
12381312131,8sleep01238386-21df09:05:150
16090442121,5sleep00-21swapper/012:36:290
15863782121,7sleep01586459-21ssh12:25:150
14949182121,5sleep11494929-21ssh11:31:241
14614582121,3sleep01461472-21ssh11:13:350
14520102120,8sleep11243187-21diskmemload11:07:391
14396292120,2sleep00-21swapper/011:00:160
14256382121,7sleep11425649-21ssh10:52:311
14232992121,6sleep01423343-21ntpq10:50:170
14175362120,7sleep11243187-21diskmemload10:48:231
14062022120,2sleep10-21swapper/110:40:251
13900162121,7sleep01390027-21ssh10:32:100
13896722121,7sleep11243187-21diskmemload10:31:431
13478962121,5sleep20-21swapper/210:09:472
13403372122,6sleep01341445-21kthreadcore10:05:160
13294042121,6sleep11243187-21diskmemload09:58:281
13186772121,7sleep21318692-21ssh09:50:472
13105822121,8sleep10-21swapper/109:46:361
12652382121,5sleep11265296-21ssh09:20:191
12368432121,5sleep21236895-21ntpq09:00:192
12208892121,6sleep11220940-21ntpq08:45:201
11896152122,5sleep21189639-21munin-run08:20:002
11833042121,6sleep01183356-21ntpq08:10:200
11467442121,7sleep11146684-31find07:36:591
11290982121,7sleep01129821-21cat07:20:190
15812902111,5sleep21581347-21ntpq12:20:202
15770442110,2sleep10-21swapper/112:20:111
15665652111,6sleep01566572-21ssh12:13:120
15554982111,5sleep20-21swapper/212:05:322
15472452111,4sleep20-21swapper/212:01:112
15411042111,5sleep11243187-21diskmemload11:59:421
15411042111,5sleep11243187-21diskmemload11:59:421
15304472111,3sleep20-21swapper/211:52:082
15217432111,7sleep01521757-21ssh11:47:140
14982612110,1sleep10-21swapper/111:35:131
14965312111,5sleep21243187-21diskmemload11:33:372
14886692111,3sleep00-21swapper/011:29:470
14608662111,4sleep11243187-21diskmemload11:12:481
14335302111,3sleep00-21swapper/010:56:230
14161132110,7sleep01416128-21ssh10:46:290
14016212111,7sleep01401888-21sh10:40:120
13993982112,6sleep01399401-21sh10:37:390
13574942111,6sleep21357206-21/usr/sbin/munin10:15:122
13385492111,3sleep20-21swapper/210:03:582
13315352111,6sleep01331874-21grep10:00:140
13211842111,6sleep01321198-21ssh09:54:060
13047892111,6sleep01304820-21apt-get09:45:120
13023662111,6sleep01302380-21ssh09:42:150
13021292111,4sleep11243187-21diskmemload09:41:561
12954452111,3sleep00-21swapper/009:39:420
12740292112,4sleep21274083-21ntpq09:25:212
12650872110,7sleep01265237-21nfs4_client09:20:180
11566412111,5sleep21156688-21ntpq07:45:162
11362362111,5sleep01136263-21munin-run07:30:000
11352182111,5sleep21135256-21ntpq07:25:222
11248862111,6sleep01125085-21grep07:15:260
16090042101,3sleep11243187-21diskmemload12:36:251
16027572101,3sleep10-21swapper/112:34:461
15999722101,3sleep00-21swapper/012:31:040
15987772101,5sleep21598839-21ntpq12:30:182
15912332101,4sleep21243187-21diskmemload12:26:052
15823312101,2sleep00-21swapper/012:20:540
15744852101,3sleep20-21swapper/212:17:072
15743262101,2sleep10-21swapper/112:16:541
15676212101,3sleep21243187-21diskmemload12:14:332
15566012100,1sleep10-21swapper/112:07:001
15469742101,3sleep01243187-21diskmemload12:00:480
15312382100,2sleep00-21swapper/011:53:120
15215152101,4sleep21243187-21diskmemload11:46:562
15130522101,2sleep20-21swapper/211:42:192
15109932101,4sleep11511059-21ntp_states11:40:221
15029832101,2sleep00-21swapper/011:35:360
14882442101,2sleep20-21swapper/211:29:112
14791322101,2sleep10-21swapper/111:23:461
14778182101,3sleep20-21swapper/211:22:032
14766912101,3sleep00-21swapper/011:20:330
14709992101,4sleep11243187-21diskmemload11:19:371
14548472101,5sleep21456428-21kthreadcore11:10:162
14366592101,4sleep21436682-21apt-get11:00:092
13934602101,5sleep21394475-21kthreadcore10:35:162
13927802101,3sleep11243187-21diskmemload10:35:071
13815792101,5sleep01381593-21ssh10:27:360
13735062101,2sleep21373398-21fwupd10:23:542
13537652101,3sleep10-21swapper/110:11:011
13364592100,7sleep11243187-21diskmemload10:01:121
13293872101,5sleep01329403-21ssh09:58:270
13185212101,3sleep10-21swapper/109:50:341
13098342101,6sleep21243187-21diskmemload09:45:372
12924262101,2sleep10-21swapper/109:35:421
12586742101,4sleep21243187-21diskmemload09:17:242
12586742101,4sleep21243187-21diskmemload09:17:242
12586152101,4sleep11243187-21diskmemload09:17:191
12586152101,4sleep11243187-21diskmemload09:17:181
12576812101,5sleep01257695-21ssh09:16:050
12576812101,5sleep01257695-21ssh09:16:040
12502532101,4sleep00-21swapper/009:12:490
12261502101,4sleep21226188-21ntpq08:50:182
12167822101,4sleep01216801-21apt-get08:45:110
12099332101,4sleep01209987-21ntpq08:35:210
11899192101,4sleep21190174-21df08:20:122
11726352100,5sleep10-21swapper/108:00:181
11609942101,5sleep21161821-21memory07:50:182
11579162101,4sleep01157935-21apt-get07:50:120
11569962101,6sleep01157270-21unixbench_singl07:45:230
1540539291,2sleep00-21swapper/011:58:570
1540539291,2sleep00-21swapper/011:58:570
1488272290,2sleep10-21swapper/111:29:141
1451436290,4sleep00-21swapper/011:06:540
1443193290,2sleep10-21swapper/111:02:331
1372410290,1sleep10-21swapper/110:22:291
1296098291,3sleep21296127-21apt-get09:40:132
1283509291,2sleep20-21swapper/209:30:302
1266673291,2sleep20-21swapper/209:21:202
1249361291,3sleep11243187-21diskmemload09:11:371
1227440291,3sleep21227459-21apt-get08:55:112
1226488291,4sleep01226647-21cat08:50:230
1210214291,4sleep11210439-21unixbench_multi08:35:261
1206330291,4sleep21206401-21irqstats08:35:142
1201922291,5sleep00-21swapper/008:30:160
1200808291,5sleep11200915-21fschecks_time08:30:131
1169684291,5sleep21171402-21kthreadcore08:00:162
1157912290,2sleep10-21swapper/107:50:091
1147585291,4sleep11147798-21hddtemp_smartct07:40:151
1145918291,4sleep01145972-21ntpq07:35:200
1121064291,4sleep21121775-21cut07:15:162
1242042281,3sleep21242159-21ntpq09:05:212
1215382281,3sleep01215524-21mailstats08:40:220
1189375280,1sleep10-21swapper/108:17:031
1173331280,3sleep072-21kcompactd008:00:540
1164629281,3sleep01166384-21kthreadcore07:55:170
1141521280,1sleep10-21swapper/107:34:461
1140848281,3sleep01136496-21/usr/sbin/munin07:30:250
1129822281,4sleep10-21swapper/107:20:191
1118245280,1sleep20-21swapper/207:14:552
1113183281,4sleep01113341-21cstates07:10:120
1232352270,1sleep10-21swapper/108:58:071
1211182270,2sleep10-21swapper/108:40:111
1205403270,1sleep20-21swapper/208:32:552
1199994270,1sleep10-21swapper/108:26:361
1195242271,3sleep00-21swapper/008:25:100
1194612270,1sleep00-21swapper/008:20:340
1194603270,1sleep10-21swapper/108:20:271
1181538271,3sleep11179242-21/usr/sbin/munin08:10:161
1178748270,1sleep10-21swapper/108:07:051
1178644270,3sleep20-21swapper/208:05:452
1173916270,2sleep00-21swapper/008:05:110
1156519271,2sleep11156640-21ntpq07:45:151
1152291270,1sleep20-21swapper/207:44:172
1136495270,2sleep20-21swapper/207:30:102
1125866270,2sleep20-21swapper/207:20:122
1125468270,1sleep10-21swapper/107:18:351
1238389261,2sleep10-21swapper/109:05:161
1229533261,2sleep01231281-21latency08:55:170
1189463260,1sleep00-21swapper/008:18:060
1168033260,1sleep10-21swapper/107:56:351
1147293260,4sleep00-21swapper/007:40:100
1146697261,2sleep20-21swapper/207:36:482
1113153260,1sleep10-21swapper/107:10:051
1233078250,2sleep00-21swapper/009:00:140
1226882250,3sleep10-21swapper/108:51:161
1179534250,1sleep20-21swapper/208:10:132
1167176250,1sleep20-21swapper/207:55:192
1135367250,1sleep10-21swapper/107:25:251
1194963240,1sleep20-21swapper/208:25:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional