You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-23 - 16:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sun Aug 23, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
119559221090,4sleep13056-21PK-Backend07:15:021
139099830,64rtkit-daemon0-21swapper/307:05:243
119311428239,14sleep10-21swapper/107:06:481
119305628039,14sleep00-21swapper/007:06:040
119188826643,14sleep20-21swapper/207:05:082
12776162570,4sleep13056-21PK-Backend11:05:201
12776162570,4sleep13056-21PK-Backend11:05:191
11938862570,4sleep0161rcuc/007:10:130
12486902500,2sleep20-21swapper/209:45:002
12896562370,3sleep30-21swapper/311:40:193
12854162350,4sleep00-21swapper/011:30:010
12570262340,6sleep234-21ksoftirqd/210:05:222
119352499340,18cyclictest1251853-21taskset09:52:233
119352499330,21cyclictest1308109-21munin-run12:35:003
119352499321,18cyclictest1285410-21latency_hist11:30:013
119352499310,18cyclictest0-21swapper/308:58:303
119352499310,16cyclictest1246599-21taskset09:36:573
119352499310,15cyclictest1259394-21date10:15:003
119352499300,28cyclictest436-20systemd-journal10:40:003
119352499300,18cyclictest0-21swapper/311:34:283
119352499300,18cyclictest0-21swapper/311:07:493
119352499300,18cyclictest0-21swapper/311:07:493
119352499300,16cyclictest2156-21gnome-shell09:33:093
119352499300,16cyclictest0-21swapper/309:41:083
119352499300,16cyclictest0-21swapper/307:43:353
119352499291,15cyclictest1225786-21apt08:40:113
119352499290,17cyclictest0-21swapper/311:15:393
119352499290,17cyclictest0-21swapper/308:38:003
119352499290,16cyclictest1195088-21sshd07:14:123
119352499280,17cyclictest0-21swapper/308:45:503
119352499280,16cyclictest0-21swapper/311:58:503
119352499280,16cyclictest0-21swapper/310:40:263
119352499280,16cyclictest0-21swapper/307:53:373
119352499280,15cyclictest1293937-21latency_hist11:55:013
119352499271,24cyclictest1210845-21ntp_kernel_pll_07:55:193
119352499271,15cyclictest0-21swapper/312:18:353
1193524992710,15cyclictest1201198-21wc07:30:003
119352499270,16cyclictest0-21swapper/309:48:373
139099260,18rtkit-daemon0-21swapper/208:20:212
139099260,18rtkit-daemon0-21swapper/208:20:212
119352499264,11cyclictest0-21swapper/309:06:113
1193524992612,12cyclictest1217796-21ntp_states08:15:203
1193515992618,6cyclictest1271412-21perf10:50:001
119352499256,16cyclictest11959332sleep307:15:143
119352499253,20cyclictest1218998-21cstates08:20:133
119352499253,20cyclictest1218998-21cstates08:20:133
1193524992520,4cyclictest0-21swapper/309:56:513
119352499251,21cyclictest1221383-21python308:25:203
119352499250,23cyclictest0-21swapper/312:20:113
13015132240,4sleep13056-21PK-Backend12:15:131
12750942240,2sleep30-21swapper/311:00:003
119352499247,15cyclictest1207395-21cat07:45:203
119352499241,21cyclictest2156-21gnome-shell07:33:543
1193524992412,10cyclictest1237178-21ntp_states09:10:203
119352499240,22cyclictest1895067-21snapd10:03:433
119352499240,22cyclictest1288022-21ntp_states11:35:193
119352499240,22cyclictest1272171-21kernelversion10:50:183
139099230,15rtkit-daemon0-21swapper/207:15:202
119352499238,13cyclictest1261348-21cat10:20:133
119352499232,19cyclictest1282564-21uname11:20:163
1193524992315,6cyclictest1212720-21python308:00:213
1193524992312,10cyclictest1232947-21/usr/sbin/munin09:00:163
119352499231,20cyclictest1211-21kerneloops08:08:103
1193524992311,11cyclictest1-21systemd08:55:003
1193524992311,10cyclictest1199405-21idleruntime-cro07:24:593
119352499230,22cyclictest1305491-21sed12:25:173
119352499230,1cyclictest0-21swapper/310:45:593
119352499230,1cyclictest0-21swapper/309:25:223
1193515992311,10cyclictest1260415-21python310:15:211
119350799230,21cyclictest1295679-21munin-run12:00:000
139099220,14rtkit-daemon0-21swapper/209:00:212
12123342220,9chrt13-21ksoftirqd/008:00:170
119352499229,11cyclictest1240389-21cstates09:20:133
119352499228,12cyclictest436-20systemd-journal12:10:013
119352499228,12cyclictest1203738-21cat07:35:173
119352499227,13cyclictest1308369-21cpu12:35:123
119352499226,14cyclictest1299973-21expr12:10:123
119352499221,19cyclictest1278973-21df11:10:143
119352499220,21cyclictest1256513-21basename10:05:163
119352499220,21cyclictest1-21systemd11:50:003
119352499220,20cyclictest1238565-21grep09:15:153
119352499220,20cyclictest1216768-21cron08:15:003
1193515992220,1cyclictest3056-21PK-Backend07:57:521
1193515992220,1cyclictest0-21swapper/110:36:501
1193515992219,2cyclictest3056-21PK-Backend10:24:461
1193515992219,2cyclictest3056-21PK-Backend09:53:431
1193515992219,2cyclictest3056-21PK-Backend09:29:091
1193515992219,2cyclictest3056-21PK-Backend08:39:071
1193515992219,2cyclictest3056-21PK-Backend07:33:181
1193515992219,1cyclictest3056-21PK-Backend11:43:481
119351599220,20cyclictest3056-21PK-Backend12:07:411
119351599220,20cyclictest3056-21PK-Backend08:56:311
119351599220,20cyclictest3056-21PK-Backend08:16:231
119351599220,20cyclictest3056-21PK-Backend07:20:191
1193507992220,1cyclictest0-21swapper/007:55:100
119350799220,21cyclictest0-21swapper/010:04:000
12497092210,8sleep2331rcuc/209:45:212
119352499212,15cyclictest1275150-21apt11:00:123
1193524992119,1cyclictest0-21swapper/308:30:063
1193524992116,4cyclictest0-21swapper/310:15:163
1193524992112,6cyclictest1296166-21apt-get12:00:033
119352499210,20cyclictest0-21swapper/310:32:263
1193519992118,2cyclictest0-21swapper/211:04:002
119351999210,20cyclictest0-21swapper/207:52:242
1193515992119,1cyclictest3056-21PK-Backend12:23:291
1193515992119,1cyclictest3056-21PK-Backend11:59:441
1193515992119,1cyclictest3056-21PK-Backend11:30:431
1193515992119,1cyclictest3056-21PK-Backend11:17:251
1193515992119,1cyclictest3056-21PK-Backend09:00:571
1193515992119,1cyclictest0-21swapper/110:34:461
1193515992118,2cyclictest3056-21PK-Backend10:05:341
1193515992118,2cyclictest3056-21PK-Backend09:05:231
1193515992118,2cyclictest3056-21PK-Backend08:03:251
1193515992118,2cyclictest3056-21PK-Backend07:54:411
1193515992118,2cyclictest3056-21PK-Backend07:46:081
119351599210,20cyclictest3056-21PK-Backend10:03:301
119351599210,19cyclictest3056-21PK-Backend12:30:591
119351599210,19cyclictest3056-21PK-Backend12:00:441
119351599210,19cyclictest3056-21PK-Backend11:25:371
119351599210,19cyclictest3056-21PK-Backend10:26:491
119351599210,19cyclictest3056-21PK-Backend09:35:391
1193507992119,1cyclictest0-21swapper/012:23:590
1193507992119,1cyclictest0-21swapper/012:11:490
1193507992119,1cyclictest0-21swapper/012:08:590
1193507992119,1cyclictest0-21swapper/011:10:390
1193507992119,1cyclictest0-21swapper/009:06:480
119350799210,2cyclictest0-21swapper/008:13:420
119350799210,20cyclictest0-21swapper/010:42:410
119350799210,20cyclictest0-21swapper/010:28:340
119350799210,20cyclictest0-21swapper/009:44:130
119350799210,20cyclictest0-21swapper/009:24:340
119350799210,20cyclictest0-21swapper/009:00:330
119350799210,20cyclictest0-21swapper/008:56:560
119350799210,20cyclictest0-21swapper/007:47:120
119350799210,20cyclictest0-21swapper/007:25:430
119352499206,12cyclictest1264034-21timerwakeupswit10:25:223
1193519992018,1cyclictest0-21swapper/208:26:442
1193519992017,2cyclictest0-21swapper/209:05:042
1193519992012,6cyclictest1277868-21sed11:05:232
1193519992012,6cyclictest1277868-21sed11:05:232
119351999200,1cyclictest0-21swapper/210:25:152
119351999200,1cyclictest0-21swapper/208:15:092
119351999200,19cyclictest0-21swapper/212:21:202
119351999200,19cyclictest0-21swapper/211:57:072
119351999200,19cyclictest0-21swapper/210:50:012
119351999200,19cyclictest0-21swapper/210:46:432
119351999200,19cyclictest0-21swapper/210:44:272
119351999200,19cyclictest0-21swapper/208:39:572
1193515992018,1cyclictest3056-21PK-Backend12:26:201
1193515992018,1cyclictest3056-21PK-Backend11:46:521
1193515992018,1cyclictest3056-21PK-Backend11:21:311
1193515992018,1cyclictest3056-21PK-Backend11:13:191
1193515992018,1cyclictest3056-21PK-Backend09:58:371
1193515992018,1cyclictest3056-21PK-Backend09:16:531
1193515992018,1cyclictest3056-21PK-Backend08:53:071
1193515992018,1cyclictest3056-21PK-Backend08:28:331
1193515992018,1cyclictest3056-21PK-Backend07:37:021
1193515992018,1cyclictest3056-21PK-Backend07:10:051
1193515992018,1cyclictest0-21swapper/109:21:191
119351599200,1cyclictest3056-21PK-Backend10:55:421
119351599200,19cyclictest3056-21PK-Backend12:10:311
119351599200,19cyclictest0-21swapper/108:07:241
119351599200,18cyclictest3056-21PK-Backend09:32:481
1193507992018,1cyclictest0-21swapper/012:30:420
119350799200,1cyclictest3056-21PK-Backend08:26:540
119350799200,19cyclictest3056-21PK-Backend10:37:330
119350799200,19cyclictest0-21swapper/012:38:120
119350799200,19cyclictest0-21swapper/010:31:380
119350799200,19cyclictest0-21swapper/010:07:040
119350799200,19cyclictest0-21swapper/009:57:040
119350799200,19cyclictest0-21swapper/009:38:320
119350799200,19cyclictest0-21swapper/007:40:160
119350799200,19cyclictest0-21swapper/007:35:440
1193519991917,1cyclictest0-21swapper/212:30:532
1193519991917,1cyclictest0-21swapper/212:16:412
1193519991917,1cyclictest0-21swapper/212:00:112
1193519991917,1cyclictest0-21swapper/210:20:072
1193519991917,1cyclictest0-21swapper/210:02:302
1193519991917,1cyclictest0-21swapper/209:51:272
1193519991914,3cyclictest1286056-21latency11:30:162
1193519991912,5cyclictest1205076-21grep07:40:122
1193519991912,5cyclictest1199375-21cron07:25:002
1193519991911,6cyclictest883-21dbus-daemon07:33:552
1193519991911,6cyclictest1241251-21awk09:20:212
1193519991911,6cyclictest1208168-21sed07:50:012
119351999190,1cyclictest3056-21PK-Backend10:35:142
119351999190,1cyclictest0-21swapper/211:44:502
119351999190,1cyclictest0-21swapper/209:59:262
119351999190,1cyclictest0-21swapper/209:17:342
119351999190,1cyclictest0-21swapper/209:11:192
119351999190,1cyclictest0-21swapper/208:34:562
119351999190,1cyclictest0-21swapper/207:35:202
119351999190,1cyclictest0-21swapper/207:13:302
119351999190,18cyclictest0-21swapper/212:36:342
119351999190,18cyclictest0-21swapper/212:27:012
119351999190,18cyclictest0-21swapper/212:08:092
119351999190,18cyclictest0-21swapper/211:54:022
119351999190,18cyclictest0-21swapper/211:38:412
119351999190,18cyclictest0-21swapper/211:16:452
119351999190,18cyclictest0-21swapper/209:37:492
1193515991916,2cyclictest3056-21PK-Backend10:43:591
119351599190,1cyclictest3056-21PK-Backend12:35:391
119351599190,1cyclictest3056-21PK-Backend11:52:471
119351599190,1cyclictest3056-21PK-Backend11:39:421
119351599190,1cyclictest3056-21PK-Backend09:40:191
119350799193,14cyclictest1261022-21perf10:20:010
1193507991917,1cyclictest0-21swapper/011:20:050
1193507991917,1cyclictest0-21swapper/010:54:360
1193507991917,1cyclictest0-21swapper/009:32:510
119350799190,1cyclictest0-21swapper/012:16:290
119350799190,1cyclictest0-21swapper/011:53:510
119350799190,1cyclictest0-21swapper/010:24:280
119351999187,9cyclictest1273775-21df10:55:142
1193519991816,1cyclictest3056-21PK-Backend10:34:142
1193519991816,1cyclictest0-21swapper/208:55:102
1193519991816,1cyclictest0-21swapper/208:51:452
1193519991816,1cyclictest0-21swapper/208:44:082
1193519991815,2cyclictest0-21swapper/209:26:402
1193519991811,6cyclictest1300651-21sed12:10:212
1193515991816,1cyclictest3056-21PK-Backend09:12:271
1193515991816,1cyclictest3056-21PK-Backend07:25:071
1193515991812,4cyclictest1223777-21idleruntime-cro08:35:001
1193507991816,1cyclictest0-21swapper/012:29:400
1193507991816,1cyclictest0-21swapper/012:02:030
1193507991816,1cyclictest0-21swapper/010:58:560
1193507991816,1cyclictest0-21swapper/009:50:090
1193507991816,1cyclictest0-21swapper/009:25:340
1193507991816,1cyclictest0-21swapper/008:32:410
1193507991816,1cyclictest0-21swapper/008:17:540
1193507991816,1cyclictest0-21swapper/007:51:110
1193507991816,1cyclictest0-21swapper/007:20:560
1193507991815,2cyclictest0-21swapper/009:19:200
1193507991813,4cyclictest13-21ksoftirqd/009:15:000
119350799180,1cyclictest0-21swapper/010:45:590
12760272170,10sleep013-21ksoftirqd/011:00:210
119351999179,6cyclictest1283960-21tr11:25:122
1193519991715,1cyclictest0-21swapper/211:24:092
1193519991715,1cyclictest0-21swapper/208:48:212
1193519991715,1cyclictest0-21swapper/207:27:502
1193519991712,3cyclictest1257932-21cpuspeed_turbos10:10:122
1193519991712,3cyclictest1210613-21kernelversion07:55:172
1193519991711,5cyclictest1259764-21ls10:15:132
1193519991710,5cyclictest1244705-21sed09:30:222
1193515991715,1cyclictest3056-21PK-Backend07:41:351
1193515991714,2cyclictest3056-21PK-Backend10:10:411
119350799179,6cyclictest1225402-21nm-dispatcher08:38:550
119350799172,13cyclictest1288734-21cron11:40:010
1193507991715,1cyclictest0-21swapper/007:31:590
1193507991712,4cyclictest1228220-21awk08:45:190
1193507991712,3cyclictest1277010-21tr11:05:130
1193507991712,3cyclictest1277010-21tr11:05:130
1193507991711,5cyclictest1214477-21ls08:05:230
139099160,7rtkit-daemon1279236-21sed11:10:182
1193519991610,5cyclictest34-21ksoftirqd/211:45:162
1193515991614,1cyclictest3056-21PK-Backend09:47:481
1193515991614,1cyclictest3056-21PK-Backend08:44:561
119351599161,11cyclictest1215321-21apt08:10:111
119350799169,6cyclictest1196643-21grep07:15:190
1193507991612,3cyclictest1292203-21dump-pmu-power11:50:000
119351999157,6cyclictest1213875-21expr08:05:142
1193519991512,2cyclictest1212599-21ntp_states08:00:192
1193519991510,4cyclictest3056-21PK-Backend08:12:392
119350799159,5cyclictest1286496-21grep11:30:210
119350799159,5cyclictest1219786-21sed08:20:230
119350799159,5cyclictest1219786-21sed08:20:220
119350799159,4cyclictest1290663-21sed11:45:010
119350799158,5cyclictest1229852-21expr08:50:130
1193507991513,1cyclictest0-21swapper/009:48:060
1193507991510,4cyclictest1257950-21tr10:10:120
1193515991412,1cyclictest3056-21PK-Backend11:00:011
1193515991410,3cyclictest3056-21PK-Backend08:49:151
119350799148,5cyclictest1280961-21grep11:15:200
119350799144,9cyclictest1225923-21cut08:40:120
1193515991311,1cyclictest3056-21PK-Backend10:54:401
119351599111,8cyclictest1220247-21turbostat.cron08:25:001
119351599111,8cyclictest1220247-21turbostat.cron08:24:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional