You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-10 - 23:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Wed Jun 10, 2026 12:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
138736021470,4sleep33056-21PK-Backend09:55:193
139099890,71rtkit-daemon0-21swapper/307:05:063
132778627939,14sleep00-21swapper/007:07:440
132766327241,25sleep10-21swapper/107:06:071
132764826444,14sleep20-21swapper/207:05:562
13943272600,4sleep13056-21PK-Backend10:15:151
13943272600,4sleep13056-21PK-Backend10:15:151
14164432530,4sleep33056-21PK-Backend11:20:123
13676422510,7sleep0131ksoftirqd/009:00:150
13822342440,4sleep342-21ksoftirqd/309:40:193
14269542430,2sleep00-21swapper/011:50:120
139099390,3rtkit-daemon3056-21PK-Backend10:46:393
14331812380,2sleep03056-21PK-Backend12:05:240
139099380,3rtkit-daemon3056-21PK-Backend09:45:053
139099370,3rtkit-daemon3056-21PK-Backend09:03:563
139099360,3rtkit-daemon3056-21PK-Backend09:14:403
139099360,3rtkit-daemon0-21swapper/310:28:253
13758012300,9sleep013-21ksoftirqd/009:25:000
13919352260,3chrt0-21swapper/210:10:022
139099250,4rtkit-daemon1389-21rtkit-daemon12:25:023
13517492240,2sleep20-21swapper/208:15:132
132812599242,20cyclictest0-21swapper/207:45:182
139099230,15rtkit-daemon3056-21PK-Backend10:38:023
1328121992314,7cyclictest1414550-21cat11:15:101
1328119992321,1cyclictest0-21swapper/008:20:300
132811999231,20cyclictest1371080-21df_inode09:10:120
1328128992220,1cyclictest3056-21PK-Backend11:31:163
1328128992220,1cyclictest3056-21PK-Backend09:35:373
1328128992220,1cyclictest0-21swapper/308:55:013
1328128992220,1cyclictest0-21swapper/308:51:233
1328128992215,5cyclictest883-21dbus-daemon07:45:003
1328128992214,6cyclictest436-20systemd-journal07:40:013
1328128992213,7cyclictest1350657-21ntp_states08:10:183
1328125992220,1cyclictest0-21swapper/211:28:322
1328125992220,1cyclictest0-21swapper/211:10:062
1328125992220,1cyclictest0-21swapper/211:02:432
1328125992220,1cyclictest0-21swapper/209:36:322
1328125992220,1cyclictest0-21swapper/209:17:322
1328125992220,1cyclictest0-21swapper/208:20:062
1328125992215,5cyclictest1422726-21sshd11:36:342
132812599220,20cyclictest0-21swapper/212:03:062
1328121992220,1cyclictest0-21swapper/112:10:521
1328121992220,1cyclictest0-21swapper/111:11:581
1328121992220,1cyclictest0-21swapper/109:55:001
1328121992220,1cyclictest0-21swapper/107:54:211
1328121992218,3cyclictest1402152-21cron10:40:011
132812199220,21cyclictest3056-21PK-Backend12:07:141
132812199220,20cyclictest3056-21PK-Backend07:11:411
1328119992220,1cyclictest0-21swapper/011:48:020
1328119992220,1cyclictest0-21swapper/009:37:500
1328119992215,6cyclictest1440863-21cron12:30:000
139099210,15rtkit-daemon3056-21PK-Backend10:56:253
139099210,15rtkit-daemon3056-21PK-Backend07:50:003
13313852210,3sleep234-21ksoftirqd/207:15:192
1328128992119,1cyclictest3056-21PK-Backend12:34:573
1328128992119,1cyclictest3056-21PK-Backend12:15:503
1328128992119,1cyclictest3056-21PK-Backend12:15:503
1328128992119,1cyclictest3056-21PK-Backend12:03:343
1328128992119,1cyclictest3056-21PK-Backend11:12:373
1328128992119,1cyclictest3056-21PK-Backend09:07:193
1328128992119,1cyclictest3056-21PK-Backend07:54:183
1328128992119,1cyclictest0-21swapper/308:04:053
1328128992119,1cyclictest0-21swapper/307:27:553
1328128992119,1cyclictest0-21swapper/307:10:103
1328128992113,6cyclictest42-21ksoftirqd/308:30:113
132812899211,18cyclictest1330324-21diskstats07:15:033
132812899210,20cyclictest3056-21PK-Backend09:16:443
132812899210,20cyclictest0-21swapper/308:45:213
132812899210,1cyclictest3056-21PK-Backend10:34:383
132812899210,19cyclictest3056-21PK-Backend11:50:293
132812899210,19cyclictest3056-21PK-Backend11:15:283
1328125992119,1cyclictest0-21swapper/212:28:142
1328125992119,1cyclictest0-21swapper/212:16:052
1328125992119,1cyclictest0-21swapper/212:16:042
1328125992119,1cyclictest0-21swapper/211:57:252
1328125992119,1cyclictest0-21swapper/211:41:092
1328125992119,1cyclictest0-21swapper/211:23:252
1328125992117,2cyclictest4176944-21snapd09:31:192
132812599210,20cyclictest3056-21PK-Backend11:17:162
132812599210,20cyclictest0-21swapper/210:45:532
1328121992119,1cyclictest3056-21PK-Backend09:18:361
1328121992119,1cyclictest3056-21PK-Backend08:52:411
1328121992119,1cyclictest3056-21PK-Backend08:46:261
1328121992119,1cyclictest3056-21PK-Backend07:26:421
1328121992119,1cyclictest0-21swapper/111:35:371
1328121992119,1cyclictest0-21swapper/111:20:571
1328121992119,1cyclictest0-21swapper/111:08:551
1328121992119,1cyclictest0-21swapper/110:04:121
1328121992119,1cyclictest0-21swapper/109:40:391
1328121992119,1cyclictest0-21swapper/109:10:181
1328121992118,2cyclictest3056-21PK-Backend07:15:411
1328121992113,6cyclictest1377377-21cron09:30:001
132812199210,2cyclictest0-21swapper/111:55:181
132812199210,20cyclictest3056-21PK-Backend11:34:371
132812199210,20cyclictest0-21swapper/109:05:381
132812199210,20cyclictest0-21swapper/109:04:361
132812199210,19cyclictest3056-21PK-Backend11:43:081
132812199210,19cyclictest3056-21PK-Backend08:44:361
132812199210,19cyclictest3056-21PK-Backend08:36:581
132812199210,19cyclictest3056-21PK-Backend08:33:341
1328119992119,1cyclictest0-21swapper/012:34:540
1328119992119,1cyclictest0-21swapper/011:18:010
1328119992119,1cyclictest0-21swapper/009:33:440
1328119992119,1cyclictest0-21swapper/008:38:220
1328119992119,1cyclictest0-21swapper/008:10:160
1328119992119,1cyclictest0-21swapper/007:31:370
1328119992118,2cyclictest0-21swapper/007:55:170
1328119992113,6cyclictest436-20systemd-journal09:45:000
132811999210,7cyclictest1400441-21perf10:35:000
132811999210,3cyclictest1389241-21cat10:00:220
132811999210,20cyclictest0-21swapper/012:24:120
132811999210,1cyclictest0-21swapper/008:04:150
139099200,13rtkit-daemon0-21swapper/210:26:062
1328128992018,1cyclictest0-21swapper/311:45:163
1328128992018,1cyclictest0-21swapper/309:32:343
1328128992018,1cyclictest0-21swapper/309:23:473
1328128992018,1cyclictest0-21swapper/308:26:093
1328128992012,6cyclictest913-21gdbus08:20:003
132812899200,19cyclictest3056-21PK-Backend10:40:333
132812899200,19cyclictest0-21swapper/310:15:253
132812899200,19cyclictest0-21swapper/310:15:253
132812899200,19cyclictest0-21swapper/308:41:093
132812899200,19cyclictest0-21swapper/307:30:123
132812899200,18cyclictest3056-21PK-Backend11:05:073
1328125992018,1cyclictest0-21swapper/212:05:562
1328125992018,1cyclictest0-21swapper/208:55:482
1328125992018,1cyclictest0-21swapper/208:52:582
1328125992018,1cyclictest0-21swapper/208:04:582
1328125992018,1cyclictest0-21swapper/207:52:552
1328125992015,4cyclictest978-21snmpd12:37:532
1328125992015,4cyclictest978-21snmpd10:54:592
1328125992015,3cyclictest978-21snmpd08:45:262
1328125992013,6cyclictest1361314-21sed08:40:212
1328125992013,5cyclictest1428226-21perf11:55:012
132812599200,1cyclictest0-21swapper/212:22:322
132812599200,1cyclictest0-21swapper/211:45:012
132812599200,1cyclictest0-21swapper/210:30:222
132812599200,19cyclictest0-21swapper/212:13:132
132812599200,19cyclictest0-21swapper/210:09:432
132812599200,19cyclictest0-21swapper/209:57:262
132812599200,19cyclictest0-21swapper/209:45:572
132812599200,19cyclictest0-21swapper/208:31:492
1328121992018,1cyclictest0-21swapper/112:39:111
1328121992018,1cyclictest0-21swapper/112:21:471
1328121992018,1cyclictest0-21swapper/109:35:321
1328121992017,2cyclictest3056-21PK-Backend08:11:371
1328121992017,2cyclictest3056-21PK-Backend08:06:231
1328121992017,2cyclictest3056-21PK-Backend07:57:441
1328121992017,2cyclictest0-21swapper/111:28:551
1328121992012,6cyclictest1440876-21cron12:30:011
1328121992011,7cyclictest1436406-21awk12:15:141
1328121992011,7cyclictest1436406-21awk12:15:141
132812199200,1cyclictest0-21swapper/111:03:471
132812199200,19cyclictest3056-21PK-Backend08:04:341
132812199200,19cyclictest0-21swapper/109:46:491
132812199200,19cyclictest0-21swapper/107:35:411
132812199200,18cyclictest3056-21PK-Backend10:14:121
1328119992018,1cyclictest0-21swapper/012:02:010
1328119992018,1cyclictest0-21swapper/011:28:010
1328119992018,1cyclictest0-21swapper/008:32:350
1328119992018,1cyclictest0-21swapper/007:51:040
1328119992017,2cyclictest0-21swapper/009:50:200
1328119992013,5cyclictest13-21ksoftirqd/010:55:100
132811999200,1cyclictest0-21swapper/012:15:060
132811999200,1cyclictest0-21swapper/012:15:060
132811999200,19cyclictest3056-21PK-Backend09:17:420
132811999200,19cyclictest1410837-21cut11:05:000
132811999200,19cyclictest0-21swapper/011:14:580
132811999200,19cyclictest0-21swapper/008:17:060
132811999200,19cyclictest0-21swapper/007:23:590
132811999200,19cyclictest0-21swapper/007:13:450
132811999200,18cyclictest3056-21PK-Backend07:40:290
14064672190,3chrt3056-21PK-Backend10:50:173
1328128991917,1cyclictest3056-21PK-Backend11:00:273
1328128991917,1cyclictest3056-21PK-Backend10:03:023
1328128991917,1cyclictest0-21swapper/310:12:343
1328128991917,1cyclictest0-21swapper/309:26:393
1328128991914,3cyclictest1390897-21python310:05:213
1328128991914,3cyclictest1345585-21python307:55:213
1328128991912,5cyclictest1423925-21grep11:40:193
1328128991911,6cyclictest883-21dbus-daemon08:09:593
132812899190,17cyclictest1359662-21users08:35:243
1328125991917,1cyclictest0-21swapper/209:10:022
1328125991917,1cyclictest0-21swapper/209:03:192
1328125991917,1cyclictest0-21swapper/208:28:102
1328125991917,1cyclictest0-21swapper/208:14:392
1328125991917,1cyclictest0-21swapper/207:59:452
1328125991917,1cyclictest0-21swapper/207:33:422
1328125991915,3cyclictest1095-21runrttasks07:13:132
1328125991913,4cyclictest1385699-21python309:50:192
132812599190,1cyclictest0-21swapper/209:25:022
132812599190,1cyclictest0-21swapper/209:22:112
132812599190,1cyclictest0-21swapper/207:25:512
132812599190,18cyclictest0-21swapper/212:31:042
132812599190,18cyclictest0-21swapper/208:07:362
1328121991917,1cyclictest3056-21PK-Backend10:32:031
1328121991917,1cyclictest3056-21PK-Backend07:46:571
1328121991916,2cyclictest3056-21PK-Backend09:59:051
1328121991912,5cyclictest1404218-21awk10:45:121
1328121991911,6cyclictest1-21systemd07:35:001
132812199190,1cyclictest3056-21PK-Backend08:17:391
132812199190,18cyclictest3056-21PK-Backend08:26:311
132811999193,14cyclictest1342714-21cron07:50:000
1328119991917,1cyclictest0-21swapper/012:35:550
1328119991917,1cyclictest0-21swapper/011:57:220
1328119991917,1cyclictest0-21swapper/010:21:100
1328119991917,1cyclictest0-21swapper/010:21:100
1328119991917,1cyclictest0-21swapper/009:27:510
1328119991917,1cyclictest0-21swapper/008:44:100
1328119991917,1cyclictest0-21swapper/008:27:210
1328119991917,1cyclictest0-21swapper/008:05:160
1328119991914,3cyclictest1338523-21wc07:35:190
132811999191,3cyclictest1384578-21awk09:50:000
132811999190,1cyclictest0-21swapper/011:24:570
132811999190,1cyclictest0-21swapper/011:09:500
132811999190,1cyclictest0-21swapper/010:09:200
132811999190,18cyclictest0-21swapper/011:30:520
132811999190,18cyclictest0-21swapper/010:10:210
1328128991816,1cyclictest3056-21PK-Backend10:20:333
1328128991816,1cyclictest3056-21PK-Backend10:20:323
1328128991816,1cyclictest0-21swapper/311:35:563
1328128991812,4cyclictest1354381-21cat08:20:213
132812899180,17cyclictest3056-21PK-Backend11:55:433
1328125991816,1cyclictest0-21swapper/209:09:002
1328125991816,1cyclictest0-21swapper/208:37:032
1328125991816,1cyclictest0-21swapper/207:39:302
1328125991814,3cyclictest1095-21runrttasks10:18:412
1328125991814,3cyclictest1095-21runrttasks10:18:412
1328125991813,3cyclictest978-21snmpd10:22:312
1328125991813,3cyclictest978-21snmpd10:22:312
1328121991816,1cyclictest3056-21PK-Backend12:30:051
1328121991816,1cyclictest3056-21PK-Backend10:08:171
1328121991816,1cyclictest0-21swapper/112:01:001
1328121991816,1cyclictest0-21swapper/110:55:491
1328121991816,1cyclictest0-21swapper/108:55:311
1328119991816,1cyclictest0-21swapper/008:59:040
1328119991813,3cyclictest1405628-21latency_hist10:50:010
1328119991812,5cyclictest1403444-21expr10:40:210
1328119991812,4cyclictest1330088-21forks07:15:020
1328128991715,1cyclictest3056-21PK-Backend12:36:473
1328128991715,1cyclictest3056-21PK-Backend09:53:283
1328128991715,1cyclictest0-21swapper/307:24:303
132812899170,15cyclictest3056-21PK-Backend12:20:033
132812899170,15cyclictest1434311-21df_inode12:10:143
132812899170,15cyclictest1432607-21hddtemp_smartct12:05:173
1328125991715,1cyclictest0-21swapper/210:39:312
1328125991711,4cyclictest1419995-21cpuspeed_turbos11:30:122
132812199176,2cyclictest141rcu_preempt07:21:011
1328121991715,1cyclictest3056-21PK-Backend10:23:381
1328121991715,1cyclictest3056-21PK-Backend10:23:381
1328121991710,5cyclictest436-20systemd-journal11:50:001
1328121991710,5cyclictest1405870-21/usr/sbin/munin10:50:221
1328119991715,1cyclictest0-21swapper/011:44:250
1328119991715,1cyclictest0-21swapper/010:25:150
1328119991715,1cyclictest0-21swapper/009:08:230
1328119991715,1cyclictest0-21swapper/008:49:100
1328119991712,3cyclictest1386662-21cpuspeed_turbos09:55:120
1328119991711,4cyclictest1400831-21cut10:35:120
132811999170,1cyclictest0-21swapper/008:50:590
1328121991614,1cyclictest0-21swapper/109:30:251
1328121991612,3cyclictest1403688-21sshd10:42:351
1328119991611,3cyclictest1406504-21missed_timers10:50:170
1328119991610,5cyclictest1434997-21sed12:10:230
1328128991513,1cyclictest3056-21PK-Backend11:26:503
132812599159,5cyclictest34-21ksoftirqd/210:44:382
1328125991513,1cyclictest0-21swapper/209:44:572
1328125991511,3cyclictest34-21ksoftirqd/207:40:202
1328125991511,3cyclictest1407602-21/usr/sbin/munin10:55:182
132812599151,12cyclictest1332393-21tr07:20:122
1328125991510,3cyclictest1388793-21kernelversion10:00:162
1328121991513,1cyclictest3056-21PK-Backend08:21:181
132812199150,13cyclictest1426932-21grep11:50:121
1328119991513,1cyclictest0-21swapper/011:38:100
132812599149,3cyclictest1411148-21cpuspeed11:05:112
132812199149,2cyclictest26-21ksoftirqd/109:20:211
1328121991410,2cyclictest26-21ksoftirqd/107:41:071
132811999149,4cyclictest1334461-21cat07:25:160
1328119991412,1cyclictest0-21swapper/010:15:150
1328119991412,1cyclictest0-21swapper/010:15:150
132812199139,2cyclictest26-21ksoftirqd/110:25:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional