You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-04 - 17:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Mon May 04, 2026 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
137262421430,4sleep00-21swapper/007:50:210
14214522910,2sleep30-21swapper/310:10:183
139099800,70rtkit-daemon0-21swapper/007:05:460
135644327939,13sleep20-21swapper/207:06:162
135658926740,21sleep30-21swapper/307:08:103
135670126343,13sleep10-21swapper/107:09:351
14409542580,4sleep10-21swapper/111:05:211
14168902500,4sleep03056-21PK-Backend10:00:010
14387712470,1sleep20-21swapper/211:00:122
13831852350,4chrt3056-21PK-Backend08:20:233
1356856992910,6cyclictest1456232-21sed11:50:000
1356856992910,6cyclictest1441235-21sed11:05:240
1356865992819,6cyclictest1400983-21munin-plugin-st09:14:593
135685699289,5cyclictest1418390-21grep10:00:210
135685699289,5cyclictest1418390-21grep10:00:210
135685699278,5cyclictest1361285-21expr07:20:130
1356856992711,14cyclictest1366825-21missed_timers07:35:160
135685699267,5cyclictest1442497-21ls11:10:120
135685699267,5cyclictest1396531-21kernelversion09:00:160
135685699266,5cyclictest1406552-21cat09:30:110
1356856992611,3cyclictest1413021-21sshd09:45:390
135685699261,10cyclictest1373490-21apt07:55:110
135685699259,5cyclictest1402906-21date09:20:010
135685699256,5cyclictest1414105-21cat09:50:170
1356856992520,4cyclictest0-21swapper/011:27:320
1356856992510,2cyclictest1369772-21perf07:45:000
135685699250,1cyclictest0-21swapper/009:24:020
1356865992414,8cyclictest1367764-21cron07:40:013
135686099242,20cyclictest1358579-21munin-run07:15:011
135685699249,3cyclictest1436837-21cpuspeed_turbos10:55:120
135685699249,3cyclictest1405657-21xargs09:25:210
135685699249,3cyclictest1095-21runrttasks07:18:320
135685699247,4cyclictest1461935-21idleruntime12:05:150
135685699247,3cyclictest0-21swapper/007:30:220
1356856992419,4cyclictest0-21swapper/010:08:350
135685699240,9cyclictest1375421-21cpuspeed_turbos08:00:130
1356865992313,6cyclictest1431249-21perf10:40:003
135686599230,21cyclictest1431902-21hddtemp_smartct10:40:163
1356862992315,6cyclictest1386504-21python308:30:202
135685699239,2cyclictest1095-21runrttasks08:19:280
135685699238,3cyclictest1463790-21latency12:10:160
135685699238,3cyclictest0-21swapper/011:55:220
135685699237,4cyclictest13-21ksoftirqd/008:05:190
135685699237,3cyclictest1358019-21timerandwakeup07:10:210
1356856992317,5cyclictest0-21swapper/010:49:050
1356856992317,4cyclictest1426042-21ls10:25:000
1356856992315,6cyclictest1469318-21python312:25:210
1356856992310,3cyclictest1465119-21tr12:15:120
1356856992310,3cyclictest1095-21runrttasks07:29:050
135685699230,22cyclictest1447384-21sh11:25:000
135686599222,7cyclictest42-21ksoftirqd/311:30:123
1356865992220,1cyclictest3056-21PK-Backend08:19:283
1356865992220,1cyclictest0-21swapper/311:17:403
1356865992219,2cyclictest3056-21PK-Backend09:48:583
1356865992219,2cyclictest0-21swapper/307:50:093
135686599221,18cyclictest1423698-21python310:15:213
1356862992220,1cyclictest0-21swapper/212:06:322
135686299220,21cyclictest0-21swapper/209:26:262
135686299220,20cyclictest1401724-21irqstats09:15:162
1356860992220,1cyclictest3056-21PK-Backend10:09:201
1356860992220,1cyclictest0-21swapper/108:35:301
1356860992219,2cyclictest3056-21PK-Backend12:09:111
135686099220,21cyclictest0-21swapper/108:14:151
135686099220,20cyclictest3056-21PK-Backend10:29:201
135686099220,20cyclictest3056-21PK-Backend09:15:061
135686099220,20cyclictest0-21swapper/112:16:351
135685699227,3cyclictest0-21swapper/011:00:220
135685699221,8cyclictest1379562-21python308:10:200
1356856992217,4cyclictest0-21swapper/010:29:510
1356856992210,3cyclictest1095-21runrttasks08:31:370
1356856992210,2cyclictest1460732-21xargs12:00:210
139099210,12rtkit-daemon26-21ksoftirqd/107:42:221
135686599218,9cyclictest1399236-21cron09:10:003
1356865992119,1cyclictest3056-21PK-Backend09:34:393
1356865992119,1cyclictest3056-21PK-Backend07:47:453
1356865992119,1cyclictest0-21swapper/312:10:393
1356865992119,1cyclictest0-21swapper/312:08:033
1356865992119,1cyclictest0-21swapper/310:26:293
1356865992119,1cyclictest0-21swapper/308:46:463
1356865992118,2cyclictest3056-21PK-Backend11:04:573
1356865992116,3cyclictest42-21ksoftirqd/311:45:023
1356865992113,6cyclictest1446623-21ntpq11:20:173
1356865992113,5cyclictest1395487-21sshd08:56:223
1356865992110,8cyclictest436-20systemd-journal10:50:003
135686599210,20cyclictest3056-21PK-Backend08:33:353
135686599210,20cyclictest0-21swapper/311:27:333
135686599210,1cyclictest0-21swapper/310:00:213
135686599210,1cyclictest0-21swapper/310:00:203
135686599210,19cyclictest3056-21PK-Backend07:22:443
135686599210,19cyclictest3056-21PK-Backend07:12:033
1356862992119,1cyclictest0-21swapper/212:32:342
1356862992119,1cyclictest0-21swapper/211:56:042
1356862992119,1cyclictest0-21swapper/211:44:222
1356862992119,1cyclictest0-21swapper/211:06:092
1356862992119,1cyclictest0-21swapper/210:15:192
1356862992119,1cyclictest0-21swapper/209:47:412
1356862992119,1cyclictest0-21swapper/208:55:502
1356862992119,1cyclictest0-21swapper/208:15:082
1356862992119,1cyclictest0-21swapper/208:11:572
1356862992119,1cyclictest0-21swapper/207:45:072
1356862992119,1cyclictest0-21swapper/207:32:502
1356862992119,1cyclictest0-21swapper/207:22:022
135686299210,20cyclictest0-21swapper/212:37:352
135686299210,20cyclictest0-21swapper/212:01:182
135686299210,20cyclictest0-21swapper/210:56:502
135686299210,20cyclictest0-21swapper/210:02:082
135686299210,20cyclictest0-21swapper/210:02:082
135686299210,20cyclictest0-21swapper/209:32:552
135686299210,20cyclictest0-21swapper/208:35:502
135686299210,19cyclictest436-20systemd-journal11:40:002
1356860992119,1cyclictest3056-21PK-Backend09:07:271
1356860992119,1cyclictest3056-21PK-Backend09:00:521
1356860992119,1cyclictest0-21swapper/111:51:071
1356860992119,1cyclictest0-21swapper/111:11:111
1356860992119,1cyclictest0-21swapper/109:58:111
1356860992119,1cyclictest0-21swapper/108:07:191
1356860992119,1cyclictest0-21swapper/107:33:111
1356860992119,1cyclictest0-21swapper/107:16:221
1356860992113,5cyclictest1429502-21cron10:35:011
135686099210,20cyclictest3056-21PK-Backend09:42:231
135686099210,20cyclictest3056-21PK-Backend08:29:081
135686099210,20cyclictest0-21swapper/107:48:271
135686099210,19cyclictest3056-21PK-Backend12:39:321
135686099210,19cyclictest3056-21PK-Backend10:01:021
135686099210,19cyclictest3056-21PK-Backend10:01:011
135686099210,19cyclictest0-21swapper/108:16:391
135685699218,2cyclictest1095-21runrttasks09:37:240
135685699217,4cyclictest1457807-21systemctl11:54:590
1356856992119,1cyclictest3056-21PK-Backend10:41:490
1356856992119,1cyclictest0-21swapper/008:50:290
1356856992119,1cyclictest0-21swapper/007:49:110
14483942200,4sleep13056-21PK-Backend11:25:191
135686599202,12cyclictest978-21snmpd12:00:113
1356865992018,1cyclictest3056-21PK-Backend11:55:593
1356865992018,1cyclictest3056-21PK-Backend11:06:463
1356865992018,1cyclictest3056-21PK-Backend09:51:493
1356865992018,1cyclictest0-21swapper/312:24:523
1356865992018,1cyclictest0-21swapper/310:59:283
1356865992018,1cyclictest0-21swapper/310:07:033
1356865992018,1cyclictest0-21swapper/308:41:463
1356865992017,2cyclictest0-21swapper/307:15:183
135686599201,5cyclictest1401377-21cpuspeed_turbos09:15:123
1356865992014,4cyclictest1429516-21turbostat10:35:003
1356865992012,5cyclictest1095-21runrttasks08:50:183
135686599200,2cyclictest1387378-21apt08:35:113
135686599200,1cyclictest3056-21PK-Backend08:01:523
135686599200,1cyclictest3056-21PK-Backend07:58:413
135686599200,1cyclictest0-21swapper/309:21:553
135686599200,19cyclictest3056-21PK-Backend11:53:223
135686599200,19cyclictest3056-21PK-Backend09:36:273
135686599200,19cyclictest0-21swapper/312:35:413
135686599200,19cyclictest0-21swapper/311:38:163
135686599200,19cyclictest0-21swapper/310:50:223
135686599200,19cyclictest0-21swapper/310:23:403
1356862992018,1cyclictest0-21swapper/211:49:502
1356862992018,1cyclictest0-21swapper/211:10:502
1356862992018,1cyclictest0-21swapper/209:35:462
1356862992018,1cyclictest0-21swapper/209:13:282
1356862992018,1cyclictest0-21swapper/208:51:372
1356862992018,1cyclictest0-21swapper/208:44:012
1356862992018,1cyclictest0-21swapper/207:25:322
1356862992012,6cyclictest1371758-21apt07:50:102
135686299200,19cyclictest0-21swapper/212:19:582
135686299200,19cyclictest0-21swapper/211:33:262
135686299200,19cyclictest0-21swapper/211:25:222
135686299200,19cyclictest0-21swapper/211:21:452
135686299200,19cyclictest0-21swapper/210:43:512
135686299200,19cyclictest0-21swapper/210:30:072
135686299200,19cyclictest0-21swapper/210:10:392
135686299200,19cyclictest0-21swapper/209:58:152
135686299200,19cyclictest0-21swapper/209:50:312
135686299200,19cyclictest0-21swapper/209:44:512
135686299200,19cyclictest0-21swapper/209:21:452
135686299200,19cyclictest0-21swapper/209:00:032
135686299200,19cyclictest0-21swapper/207:42:432
135686299200,19cyclictest0-21swapper/207:36:352
1356860992018,1cyclictest3056-21PK-Backend12:02:211
1356860992018,1cyclictest3056-21PK-Backend11:40:241
1356860992018,1cyclictest3056-21PK-Backend10:36:381
1356860992018,1cyclictest3056-21PK-Backend09:20:331
1356860992018,1cyclictest0-21swapper/109:54:201
1356860992018,1cyclictest0-21swapper/109:48:381
1356860992018,1cyclictest0-21swapper/109:33:161
1356860992018,1cyclictest0-21swapper/108:31:311
1356860992018,1cyclictest0-21swapper/107:51:381
1356860992012,6cyclictest1361407-21df07:20:131
1356860992011,7cyclictest1459160-2101-ifupdown11:55:341
135686099200,1cyclictest0-21swapper/107:29:271
135686099200,19cyclictest0-21swapper/109:36:561
135685699207,3cyclictest0-21swapper/008:58:040
1356856992018,1cyclictest0-21swapper/012:24:150
1356856992018,1cyclictest0-21swapper/011:37:250
1356856992018,1cyclictest0-21swapper/010:36:210
1356856992018,1cyclictest0-21swapper/008:20:350
135685699201,5cyclictest1409918-21apt09:40:110
135685699200,19cyclictest0-21swapper/012:35:030
135685699200,19cyclictest0-21swapper/011:15:480
1356865991917,1cyclictest3056-21PK-Backend11:45:313
1356865991917,1cyclictest3056-21PK-Backend09:57:313
1356865991917,1cyclictest0-21swapper/311:14:503
1356865991913,4cyclictest1465726-21ntp_states12:15:193
135686599191,11cyclictest42-21ksoftirqd/308:25:193
135686599190,3cyclictest3056-21PK-Backend08:05:213
135686599190,1cyclictest3056-21PK-Backend07:31:573
135686299198,2cyclictest141rcu_preempt07:15:232
1356862991917,1cyclictest0-21swapper/212:25:582
1356862991917,1cyclictest0-21swapper/212:14:582
1356862991917,1cyclictest0-21swapper/210:51:232
1356862991917,1cyclictest0-21swapper/210:23:382
1356862991917,1cyclictest0-21swapper/208:47:262
1356862991915,3cyclictest1377631-21grep08:05:182
135686299190,1cyclictest0-21swapper/211:16:172
135686299190,1cyclictest0-21swapper/209:05:032
1356860991917,1cyclictest3056-21PK-Backend12:28:581
1356860991917,1cyclictest3056-21PK-Backend12:21:351
1356860991917,1cyclictest0-21swapper/107:37:441
1356860991912,5cyclictest1433627-21awk10:45:151
135686099190,1cyclictest3056-21PK-Backend11:35:581
135686099190,1cyclictest3056-21PK-Backend11:34:101
135686099190,1cyclictest3056-21PK-Backend10:23:531
135686099190,1cyclictest3056-21PK-Backend10:12:581
135686099190,1cyclictest3056-21PK-Backend08:41:061
135686099190,18cyclictest0-21swapper/111:47:271
135686099190,18cyclictest0-21swapper/108:24:231
135685699194,3cyclictest0-21swapper/010:50:110
1356856991917,1cyclictest0-21swapper/011:31:100
1356856991913,4cyclictest1453507-21needreboot11:40:160
135686599186,7cyclictest141rcu_preempt09:41:443
135686599183,13cyclictest1468912-21if_err_enp2s012:25:173
1356865991811,5cyclictest1470417-21ls12:30:143
135686599180,5cyclictest3056-21PK-Backend09:25:213
1356862991816,1cyclictest0-21swapper/212:22:342
1356862991812,5cyclictest1457284-21sed11:50:232
135686099189,7cyclictest1375817-21memory08:00:171
1356860991816,1cyclictest3056-21PK-Backend08:49:171
1356860991816,1cyclictest0-21swapper/109:26:001
1356860991810,6cyclictest1374388-21sed07:55:211
135686099180,1cyclictest3056-21PK-Backend11:22:411
1356856991816,1cyclictest3056-21PK-Backend10:16:070
1356856991816,1cyclictest0-21swapper/009:13:540
1356856991816,1cyclictest0-21swapper/008:25:210
1356856991812,1cyclictest0-21swapper/009:08:530
135685699180,1cyclictest0-21swapper/008:47:040
1356865991715,1cyclictest3056-21PK-Backend08:14:083
135686599170,3cyclictest141rcu_preempt07:42:323
135686299172,11cyclictest1419110-21apt10:05:112
1356862991715,1cyclictest0-21swapper/210:36:352
1356862991715,1cyclictest0-21swapper/210:26:292
1356862991712,3cyclictest1434691-21sh10:50:002
1356862991712,3cyclictest1375884-21needreboot08:00:182
1356862991711,4cyclictest1385457-21wc08:30:002
1356860991715,1cyclictest3056-21PK-Backend10:53:471
1356860991715,1cyclictest3056-21PK-Backend09:11:411
1356860991714,2cyclictest3056-21PK-Backend10:42:041
135686599163,8cyclictest1396053-21apt09:00:113
1356862991610,5cyclictest1357592-21grep07:10:172
1356860991614,1cyclictest3056-21PK-Backend10:57:261
1356860991613,2cyclictest3056-21PK-Backend10:17:371
1356856991614,1cyclictest0-21swapper/010:30:530
1356856991611,4cyclictest1421709-21sshd10:10:220
1356865991511,2cyclictest42-21ksoftirqd/307:26:303
1356862991511,3cyclictest1382143-21/usr/sbin/munin08:20:182
1356860991513,1cyclictest3056-21PK-Backend12:14:591
1356860991513,1cyclictest0-21swapper/111:01:051
135685699157,6cyclictest883-21dbus-daemon08:45:000
135686099140,5cyclictest3056-21PK-Backend08:50:101
135685699148,5cyclictest1387702-21cat08:35:110
135685699147,5cyclictest1471601-21anacron12:34:220
135686299134,8cyclictest2174-21JS10:27:482
135686099137,5cyclictest1471158-21timerandwakeup12:30:221
135686099128,3cyclictest3056-21PK-Backend11:19:281
135686099128,3cyclictest3056-21PK-Backend08:58:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional