You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 19:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Apr 28, 2026 12:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1390991140,84rtkit-daemon0-21swapper/307:05:163
221471321110,4sleep20-21swapper/208:50:182
218084421070,3sleep33056-21PK-Backend07:15:163
217728027838,13sleep10-21swapper/107:06:041
217723926841,19sleep20-21swapper/207:05:332
217734626344,13sleep00-21swapper/007:06:550
22816462520,6sleep10-21swapper/112:00:201
22079652510,3sleep23056-21PK-Backend08:30:182
22079652510,3sleep23056-21PK-Backend08:30:182
22025022490,5sleep20-21swapper/208:15:182
22445782480,3sleep30-21swapper/310:15:173
22317332470,3sleep03056-21PK-Backend09:40:010
21965402450,4sleep10-21swapper/108:00:131
22415912440,3sleep12241592-21vmstat10:05:241
139099420,5rtkit-daemon2188835-21runrttasks07:39:483
22644252410,3sleep13056-21PK-Backend11:10:191
22286662410,2sleep00-21swapper/009:30:130
22286662410,2sleep00-21swapper/009:30:130
139099400,3rtkit-daemon2204199-21strings08:20:173
139099390,3rtkit-daemon3056-21PK-Backend11:16:503
139099390,3rtkit-daemon3056-21PK-Backend10:06:383
217773299387,18cyclictest2208737-21turbostat08:39:543
2177732993411,20cyclictest436-20systemd-journal11:35:003
2177732993411,20cyclictest436-20systemd-journal11:35:003
217773299319,20cyclictest2182011-21expr07:20:133
2177732993012,16cyclictest2255085-21ntp_kernel_err10:45:193
2177732993010,18cyclictest2178923-21cat07:10:253
217773299300,28cyclictest2235688-21fw_packets09:50:153
217773299291,11cyclictest2226609-21apt09:25:113
217773299290,27cyclictest2251284-21df_inode10:35:143
217773299288,18cyclictest2293620-21cat12:35:143
217773299288,18cyclictest2229081-21ntpq09:30:203
217773299288,18cyclictest2229081-21ntpq09:30:193
217773299286,20cyclictest2291865-21cat12:30:183
217773299283,23cyclictest42-21ksoftirqd/309:49:463
2177732992812,14cyclictest2200560-21open_files08:10:193
2177732992811,15cyclictest978-21snmpd12:07:473
217773299280,26cyclictest2247653-21cat10:25:113
22931832270,3chrt3056-21PK-Backend12:35:001
217773299271,4cyclictest2195170-21memory07:55:163
217773299270,4cyclictest2260946-21rt-features11:00:193
217773299270,3cyclictest2264424-21python311:10:203
217773299270,3cyclictest2253722-21users10:40:223
217773299270,25cyclictest2201768-21/usr/sbin/munin08:15:203
217773299268,16cyclictest1095-21runrttasks10:13:003
217773299267,17cyclictest2285909-21wc12:15:013
217773299267,17cyclictest2262416-21missed_timers11:05:193
217773299266,18cyclictest2275938-21ls11:45:113
217773299261,4cyclictest2215935-21apt08:55:133
217773299260,3cyclictest22468192chrt10:20:343
217773299260,24cyclictest2249399-21cstates10:30:123
217773299258,15cyclictest2193071-21cstates07:50:143
217773299257,16cyclictest2199106-21timerwakeupswit08:05:233
217773299256,17cyclictest0-21swapper/309:40:193
217773299255,18cyclictest2205656-21hddtemp_smartct08:25:153
217773299255,18cyclictest2205656-21hddtemp_smartct08:25:153
217773299253,6cyclictest2230444-21cat09:35:123
217773299253,6cyclictest2184084-21cat07:25:133
217773299251,20cyclictest2273974-21apt11:40:123
217772999259,14cyclictest886-21NetworkManager09:09:442
2177723992521,3cyclictest2019-21gmain12:40:010
139099250,18rtkit-daemon0-21swapper/307:30:383
139099250,18rtkit-daemon0-21swapper/307:30:373
21895102240,5sleep342-21ksoftirqd/307:40:143
217773299247,15cyclictest2287811-21/usr/sbin/munin12:20:143
217773299241,9cyclictest2290432-21python312:25:213
217773299240,9cyclictest2191417-21cat07:45:173
217773299240,8cyclictest2272634-21/usr/sbin/munin11:35:183
217773299240,3cyclictest2277912-21irqstats11:50:153
217773299240,19cyclictest3056-21PK-Backend09:07:523
217772799241,3cyclictest0-21swapper/108:40:181
217773299231,7cyclictest2226554-21date09:25:013
217773299230,7cyclictest2286260-21expr12:15:093
2177729992321,1cyclictest0-21swapper/209:53:522
217772999231,3cyclictest2217189-21python308:57:352
217772399230,21cyclictest2257838-21apt-get10:55:020
139099230,15rtkit-daemon3056-21PK-Backend08:32:103
139099230,15rtkit-daemon3056-21PK-Backend08:32:103
217773299227,13cyclictest22702062sleep311:29:553
217773299222,19cyclictest3056-21PK-Backend10:57:343
2177732992220,1cyclictest3056-21PK-Backend09:55:043
217773299220,9cyclictest2222617-21turbostat.cron09:15:003
2177729992220,1cyclictest0-21swapper/212:21:342
2177729992220,1cyclictest0-21swapper/210:53:002
2177729992220,1cyclictest0-21swapper/210:10:412
2177729992219,2cyclictest0-21swapper/212:27:092
2177729992219,2cyclictest0-21swapper/210:03:112
2177729992219,2cyclictest0-21swapper/207:17:092
2177729992211,8cyclictest883-21dbus-daemon10:30:002
2177729992211,8cyclictest883-21dbus-daemon10:25:002
217772999220,20cyclictest2192334-21apt-get07:50:022
217772999220,20cyclictest0-21swapper/210:45:162
2177727992215,5cyclictest2217402-21cron08:59:591
2177727992214,6cyclictest2179378-21cron07:15:011
2177723992215,5cyclictest2263286-21cron11:10:010
217772399220,1cyclictest0-21swapper/009:17:430
139099220,15rtkit-daemon3056-21PK-Backend08:52:073
217773299214,15cyclictest978-21snmpd11:20:373
2177732992118,2cyclictest3056-21PK-Backend10:52:163
217773299211,4cyclictest0-21swapper/309:00:183
217773299210,7cyclictest2197171-21taskset08:00:213
217773299210,6cyclictest2282032-21chrt12:00:553
217773299210,20cyclictest0-21swapper/308:45:083
217772999212,13cyclictest2187346-21apt07:35:122
2177729992119,1cyclictest0-21swapper/212:31:082
2177729992119,1cyclictest0-21swapper/212:01:002
2177729992119,1cyclictest0-21swapper/211:02:332
2177729992119,1cyclictest0-21swapper/210:36:302
2177729992119,1cyclictest0-21swapper/209:02:492
2177729992119,1cyclictest0-21swapper/208:42:002
2177729992119,1cyclictest0-21swapper/208:23:352
2177729992119,1cyclictest0-21swapper/208:14:502
2177729992119,1cyclictest0-21swapper/207:40:022
2177729992119,1cyclictest0-21swapper/207:30:492
2177729992119,1cyclictest0-21swapper/207:30:482
2177729992114,4cyclictest2199524-21sh08:10:002
217772999210,20cyclictest0-21swapper/211:27:202
217772999210,20cyclictest0-21swapper/211:13:552
217772999210,20cyclictest0-21swapper/207:27:452
217772799210,19cyclictest3056-21PK-Backend07:55:161
2177723992115,4cyclictest2220890-21munin-run09:10:000
217772399210,20cyclictest0-21swapper/011:43:490
139099210,3rtkit-daemon1389-21rtkit-daemon12:37:282
139099210,14rtkit-daemon0-21swapper/210:16:582
139099210,14rtkit-daemon0-21swapper/207:46:072
139099210,13rtkit-daemon0-21swapper/207:58:532
2177732992018,1cyclictest3056-21PK-Backend11:55:433
217773299200,5cyclictest2239623-21taskset10:00:203
217773299200,4cyclictest2223634-21sed09:15:173
2177729992018,1cyclictest0-21swapper/211:08:142
2177729992018,1cyclictest0-21swapper/209:43:442
2177729992018,1cyclictest0-21swapper/209:37:292
2177729992012,6cyclictest2285270-21cat12:10:202
217772999200,1cyclictest3056-21PK-Backend11:50:192
217772999200,19cyclictest0-21swapper/211:45:052
217772999200,19cyclictest0-21swapper/211:17:482
217772999200,19cyclictest0-21swapper/210:34:282
217772999200,19cyclictest0-21swapper/209:49:132
2177727992018,1cyclictest0-21swapper/108:28:081
2177727992018,1cyclictest0-21swapper/108:28:081
2177727992017,2cyclictest3056-21PK-Backend09:34:331
2177727992017,2cyclictest3056-21PK-Backend09:34:331
2177727992012,6cyclictest2263298-21cron11:10:011
2177727992011,7cyclictest2222215-21awk09:10:211
217772799200,19cyclictest3056-21PK-Backend10:37:191
217772799200,18cyclictest3056-21PK-Backend08:18:011
2177723992018,1cyclictest0-21swapper/011:54:170
2177723992018,1cyclictest0-21swapper/010:07:110
2177723992013,5cyclictest2277181-21ls11:49:590
217772399200,1cyclictest3056-21PK-Backend07:25:070
217772399200,1cyclictest0-21swapper/008:50:110
217772399200,19cyclictest3056-21PK-Backend12:20:530
217772399200,19cyclictest0-21swapper/009:52:360
217772399200,19cyclictest0-21swapper/008:09:020
217772399200,19cyclictest0-21swapper/008:04:560
217772399200,19cyclictest0-21swapper/007:36:570
217772399200,19cyclictest0-21swapper/007:33:530
217772399200,19cyclictest0-21swapper/007:33:520
139099200,14rtkit-daemon3056-21PK-Backend08:42:033
139099200,14rtkit-daemon0-21swapper/210:41:002
139099200,13rtkit-daemon3056-21PK-Backend07:10:012
2177729991917,1cyclictest0-21swapper/211:31:122
2177729991917,1cyclictest0-21swapper/211:31:122
2177729991917,1cyclictest0-21swapper/211:24:302
2177729991917,1cyclictest0-21swapper/209:58:322
2177729991917,1cyclictest0-21swapper/208:37:212
2177729991917,1cyclictest0-21swapper/208:27:122
2177729991917,1cyclictest0-21swapper/208:27:122
2177729991916,2cyclictest28650irq/123-ahci[0009:20:322
2177729991912,5cyclictest2228649-21cat09:30:142
2177729991912,5cyclictest2228649-21cat09:30:142
2177729991911,6cyclictest2227680-21users09:25:222
217772999190,1cyclictest0-21swapper/211:43:162
217772999190,1cyclictest0-21swapper/210:58:422
217772999190,1cyclictest0-21swapper/209:19:312
2177727991917,1cyclictest3056-21PK-Backend11:47:081
2177727991917,1cyclictest0-21swapper/111:26:061
2177727991917,1cyclictest0-21swapper/111:24:171
2177727991912,5cyclictest2213181-21ntpq08:45:191
2177727991912,5cyclictest2199066-21sed08:05:221
2177727991911,6cyclictest2279564-21fw_packets11:55:161
2177727991911,6cyclictest2240234-21cron10:05:011
2177727991911,6cyclictest2237428-21which09:55:141
2177727991911,6cyclictest2207625-21df_inode08:30:121
2177727991911,6cyclictest2207625-21df_inode08:30:121
217772799190,1cyclictest3056-21PK-Backend12:33:111
217772799190,1cyclictest3056-21PK-Backend09:52:241
217772799190,1cyclictest3056-21PK-Backend09:36:081
217772799190,1cyclictest0-21swapper/111:32:421
217772799190,1cyclictest0-21swapper/111:32:411
2177723991917,1cyclictest0-21swapper/012:27:160
2177723991917,1cyclictest0-21swapper/012:15:190
2177723991917,1cyclictest0-21swapper/011:37:480
2177723991917,1cyclictest0-21swapper/011:25:170
2177723991917,1cyclictest0-21swapper/011:13:560
2177723991917,1cyclictest0-21swapper/010:43:410
2177723991917,1cyclictest0-21swapper/010:38:000
2177723991917,1cyclictest0-21swapper/010:14:270
2177723991917,1cyclictest0-21swapper/009:48:590
2177723991917,1cyclictest0-21swapper/009:40:540
2177723991917,1cyclictest0-21swapper/009:28:240
2177723991917,1cyclictest0-21swapper/008:40:030
2177723991917,1cyclictest0-21swapper/008:33:350
2177723991917,1cyclictest0-21swapper/008:33:350
2177723991917,1cyclictest0-21swapper/008:22:530
2177723991917,1cyclictest0-21swapper/007:41:030
2177723991917,1cyclictest0-21swapper/007:22:170
2177723991912,5cyclictest436-20systemd-journal07:15:020
2177723991912,5cyclictest2266043-21ntp_states11:15:200
2177723991911,6cyclictest2281619-21ntpq12:00:200
217772399190,1cyclictest0-21swapper/011:56:550
217772399190,1cyclictest0-21swapper/009:56:150
217772399190,1cyclictest0-21swapper/007:59:490
217772399190,18cyclictest0-21swapper/011:02:320
217772399190,18cyclictest0-21swapper/010:46:310
217772399190,18cyclictest0-21swapper/010:28:250
217772399190,18cyclictest0-21swapper/010:19:550
217772399190,18cyclictest0-21swapper/008:27:200
217772399190,18cyclictest0-21swapper/008:27:200
217772399190,18cyclictest0-21swapper/007:51:500
217772399190,18cyclictest0-21swapper/007:49:470
139099190,13rtkit-daemon0-21swapper/207:23:412
22749662180,3sleep1251rcuc/111:40:231
2177729991816,1cyclictest0-21swapper/211:56:332
2177729991811,5cyclictest2196305-21cat08:00:102
2177729991810,6cyclictest2273323-21seq11:35:252
2177727991816,1cyclictest3056-21PK-Backend12:17:101
2177727991816,1cyclictest3056-21PK-Backend10:45:291
2177727991816,1cyclictest0-21swapper/109:15:061
2177727991816,1cyclictest0-21swapper/109:05:471
2177727991816,1cyclictest0-21swapper/107:17:241
2177727991814,2cyclictest2194122-21chrt07:52:391
2177727991812,4cyclictest2208951-21date08:35:011
2177727991811,5cyclictest2214389-21expr08:50:141
2177727991810,6cyclictest2256277-21sort10:50:011
2177727991810,6cyclictest2192334-21apt-get07:50:001
217772799180,1cyclictest3056-21PK-Backend12:29:401
217772399189,6cyclictest2222852-21latency_hist09:15:010
2177723991816,1cyclictest3056-21PK-Backend08:58:020
2177723991816,1cyclictest0-21swapper/012:30:280
2177723991816,1cyclictest0-21swapper/011:30:590
2177723991816,1cyclictest0-21swapper/011:30:590
2177723991816,1cyclictest0-21swapper/011:24:160
2177723991810,6cyclictest436-20systemd-journal09:25:000
2177723991810,6cyclictest2257364-21sed10:50:210
217772399180,1cyclictest0-21swapper/008:37:130
2177729991715,1cyclictest0-21swapper/212:15:132
217772999171,12cyclictest2284111-21sleep12:10:012
2177729991710,5cyclictest2222612-21apt-get09:15:002
2177727991715,1cyclictest0-21swapper/110:10:351
2177727991711,5cyclictest2278933-21systemd-run11:55:011
2177727991711,5cyclictest2258580-21ls10:55:131
2177727991711,5cyclictest2218717-21sed09:00:221
217772799171,14cyclictest2266756-21cron11:20:001
2177723991715,1cyclictest0-21swapper/010:23:460
2177723991714,2cyclictest3056-21PK-Backend08:15:230
2177723991710,5cyclictest2239208-21cat10:00:160
2177729991614,1cyclictest0-21swapper/208:49:302
217772799169,5cyclictest2260235-21grep11:00:121
217772799169,5cyclictest2254112-21cron10:44:591
217772799164,11cyclictest2285074-21ntp_kernel_err12:10:161
217772799164,10cyclictest2231808-21sort09:40:131
2177727991614,1cyclictest3056-21PK-Backend11:37:141
2177727991612,3cyclictest2248388-21threads10:25:211
2177727991610,5cyclictest2225320-21cat09:20:151
217772399169,5cyclictest2283370-21ntpq12:05:190
217772399168,6cyclictest2217903-21expr09:00:120
2177723991610,4cyclictest2212597-21cpuspeed_turbos08:45:130
2177727991513,1cyclictest0-21swapper/109:45:271
2177727991513,1cyclictest0-21swapper/108:10:091
2177727991512,2cyclictest3056-21PK-Backend07:24:471
2177727991510,3cyclictest2283581-21timerandwakeup12:05:211
217772799150,14cyclictest3056-21PK-Backend07:40:491
217772799150,14cyclictest3056-21PK-Backend07:38:121
217772399159,4cyclictest2156-21gnome-shell07:10:090
2177723991513,1cyclictest3056-21PK-Backend10:30:280
217772799149,4cyclictest2250043-21sensors10:30:211
2177727991411,2cyclictest2287811-21/usr/sbin/munin12:20:211
2177727991410,3cyclictest3056-21PK-Backend10:19:151
217772799140,1cyclictest3056-21PK-Backend08:24:571
217772799140,1cyclictest3056-21PK-Backend07:30:361
217772799140,1cyclictest3056-21PK-Backend07:30:361
217772399148,4cyclictest2201409-21apt-key08:15:010
2177723991412,1cyclictest3056-21PK-Backend12:10:320
217772999136,5cyclictest2240226-21cron10:09:552
217772799139,3cyclictest3056-21PK-Backend10:23:531
217772799139,3cyclictest3056-21PK-Backend09:27:221
217772799139,3cyclictest3056-21PK-Backend07:26:531
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional