You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 21:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed May 13, 2026 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
236006021330,3sleep23056-21PK-Backend11:45:192
226527821250,2sleep10-21swapper/107:20:121
227227721140,4sleep00-21swapper/007:40:000
236482721010,4sleep33056-21PK-Backend12:00:123
225941828152,17sleep10-21swapper/107:05:131
139099750,65rtkit-daemon0-21swapper/007:08:370
225941026444,14sleep30-21swapper/307:05:063
226062026243,13sleep20-21swapper/207:06:352
22741222610,3sleep33056-21PK-Backend07:45:013
23681812530,4sleep23056-21PK-Backend12:10:132
22974102510,3sleep30-21swapper/308:50:133
22706102510,4sleep20-21swapper/207:35:012
23770852430,4sleep22261020-21cyclictest12:35:162
2261030994312,10cyclictest2307419-21apt-key09:20:001
226103099326,11cyclictest2275595-21seq07:49:341
226103099305,13cyclictest2343050-21cat11:00:011
226103099299,6cyclictest2313079-21runrttasks09:34:531
23144142280,4sleep20-21swapper/209:35:222
226103099289,11cyclictest2305584-21runrttasks09:14:281
226103099284,15cyclictest0-21swapper/111:05:121
139099280,4rtkit-daemon1389-21rtkit-daemon07:15:361
2261033992713,12cyclictest2291283-21unin-run08:35:002
2261033992712,14cyclictest1095-21runrttasks07:44:552
226103099274,9cyclictest0-21swapper/111:25:121
226103099265,8cyclictest0-21swapper/108:45:111
2261030992614,10cyclictest2303934-21basename09:10:011
139099260,3rtkit-daemon1389-21rtkit-daemon08:00:421
226103399259,3cyclictest978-21snmpd08:37:242
226103399259,3cyclictest2286047-21sh08:20:002
226103399257,3cyclictest0-21swapper/208:09:582
226103399256,5cyclictest2283328-21cut08:10:162
226103399254,5cyclictest2305193-21grep09:10:212
2261033992520,4cyclictest0-21swapper/212:28:352
226103399251,8cyclictest2334218-21munin-run10:34:592
2261033992510,4cyclictest2298692-21cron08:55:002
226103399250,23cyclictest0-21swapper/211:59:082
226103099259,14cyclictest2290511-21grep08:30:181
2261030992514,9cyclictest2331950-21find10:25:211
2261030992514,8cyclictest436-20systemd-journal07:32:351
2261030992512,11cyclictest2337683-21awk10:44:591
139099250,3rtkit-daemon1389-21rtkit-daemon09:32:270
2261035992417,5cyclictest42-21ksoftirqd/308:13:083
226103399246,5cyclictest2267722-21cut07:25:212
226103399244,7cyclictest1-21systemd08:44:592
226103399241,7cyclictest2370138-21irqstats12:15:152
226103399241,7cyclictest2329012-21perf10:20:012
2261030992420,3cyclictest0-21swapper/110:03:131
226103399239,5cyclictest2316169-21cut09:40:212
226103399239,2cyclictest2263426-21/usr/sbin/munin07:15:192
226103399238,3cyclictest2353647-21latency_hist11:30:002
226103399237,3cyclictest2361194-21expr11:50:112
226103399233,9cyclictest2343859-21ntpq11:00:172
226103399231,8cyclictest2262091-21python307:10:212
2261033992318,4cyclictest0-21swapper/210:22:372
2261033992318,4cyclictest0-21swapper/209:25:452
2261033992310,11cyclictest2288029-21acpi08:25:112
226103399230,8cyclictest1095-21runrttasks08:20:222
226103399230,21cyclictest2347736-21/usr/sbin/munin11:10:212
226103099235,6cyclictest2328281-21ps10:15:181
2261030992314,6cyclictest436-20systemd-journal08:20:011
2261030992313,8cyclictest2314877-21apt-config09:40:011
226103099230,9cyclictest2156-21gnome-shell12:06:081
2261035992220,1cyclictest3056-21PK-Backend07:18:383
2261035992219,2cyclictest3056-21PK-Backend08:46:533
2261035992219,2cyclictest3056-21PK-Backend07:45:013
2261035992218,2cyclictest3056-21PK-Backend11:08:343
226103599220,20cyclictest3056-21PK-Backend12:17:013
226103599220,20cyclictest3056-21PK-Backend10:04:463
226103399229,2cyclictest1-21systemd10:10:002
226103399225,4cyclictest2322320-21cpu10:00:112
226103399222,4cyclictest2351211-21python311:20:222
2261033992220,1cyclictest0-21swapper/212:22:542
226103399221,8cyclictest2306289-21diskstats09:15:152
226103399221,7cyclictest2376352-21munin-run12:35:002
226103399221,7cyclictest2355085-21wc11:30:212
226103399221,7cyclictest2355085-21wc11:30:212
226103399220,8cyclictest2269392-21ntp_states07:30:192
226103399220,20cyclictest3056-21PK-Backend10:46:232
226103399220,20cyclictest0-21swapper/209:33:142
226103099229,11cyclictest2347885-21timerwakeupswit11:10:231
226103099228,12cyclictest2376663-21sort12:35:111
226103099224,7cyclictest0-21swapper/107:50:421
226103099223,7cyclictest0-21swapper/111:30:021
226103099223,7cyclictest0-21swapper/111:30:011
226103099222,2cyclictest0-21swapper/111:15:111
2261030992214,6cyclictest2289508-21cron08:29:591
2261030992213,7cyclictest2332885-21grep10:30:131
2261030992212,9cyclictest1-21systemd12:05:001
2261030992212,8cyclictest2363180-21if_enp2s011:55:141
2261030992212,8cyclictest2322015-21cron10:00:011
2261030992212,8cyclictest2313739-21kworker/u8:010:45:201
2261030992212,7cyclictest2293022-21gpgv08:40:011
2261030992211,9cyclictest2307846-21gdbus09:20:011
2261030992211,9cyclictest2278446-21processes07:55:191
2261026992220,1cyclictest0-21swapper/012:16:110
2261026992220,1cyclictest0-21swapper/012:06:520
2261026992220,1cyclictest0-21swapper/011:29:200
2261026992220,1cyclictest0-21swapper/008:41:300
2261026992220,1cyclictest0-21swapper/008:05:330
2261026992212,8cyclictest883-21dbus-daemon09:03:110
226102699220,21cyclictest0-21swapper/010:36:350
139099220,15rtkit-daemon3056-21PK-Backend12:32:413
139099220,15rtkit-daemon3056-21PK-Backend11:51:093
139099220,15rtkit-daemon3056-21PK-Backend11:49:223
22873202210,3sleep00-21swapper/008:20:230
2261035992119,1cyclictest3056-21PK-Backend12:05:053
2261035992119,1cyclictest3056-21PK-Backend10:40:503
2261035992119,1cyclictest3056-21PK-Backend09:15:253
2261035992119,1cyclictest3056-21PK-Backend09:10:593
2261035992119,1cyclictest3056-21PK-Backend08:21:113
2261035992119,1cyclictest3056-21PK-Backend07:28:463
2261035992118,2cyclictest3056-21PK-Backend11:41:593
226103599210,20cyclictest3056-21PK-Backend11:00:153
226103599210,20cyclictest3056-21PK-Backend10:37:593
226103599210,1cyclictest3056-21PK-Backend12:28:583
226103599210,19cyclictest3056-21PK-Backend10:15:553
226103599210,19cyclictest3056-21PK-Backend09:53:513
226103399218,4cyclictest978-21snmpd09:21:142
226103399216,3cyclictest3056-21PK-Backend12:05:162
226103399216,3cyclictest0-21swapper/207:48:272
226103399214,4cyclictest0-21swapper/210:40:152
226103399212,4cyclictest2342270-21python310:55:202
2261033992119,1cyclictest0-21swapper/212:03:002
226103399211,5cyclictest2266652-21idleruntime-cro07:25:012
226103399210,8cyclictest2317676-21memory09:45:172
226103399210,8cyclictest2302820-21diskstats09:05:102
226103399210,20cyclictest0-21swapper/211:41:042
226103399210,19cyclictest0-21swapper/207:50:212
2261030992119,1cyclictest0-21swapper/111:04:511
2261030992114,5cyclictest2318976-21cpuspeed_turbos09:50:111
2261030992114,5cyclictest2311392-21awk09:30:001
2261030992113,6cyclictest2362059-21sed11:50:211
2261030992113,6cyclictest2272594-21cut07:40:131
2261030992113,6cyclictest2261938-21awk07:10:191
2261030992112,7cyclictest2375818-21awk12:30:201
2261030992112,7cyclictest2369408-21turbostat.cron12:15:011
2261030992112,7cyclictest2359760-21uname11:45:161
2261030992112,7cyclictest2356393-21cat11:35:171
2261030992112,7cyclictest2335056-21taskset10:35:151
2261030992112,7cyclictest2281786-21/usr/sbin/munin08:05:171
2261030992112,6cyclictest2293334-21date08:40:011
2261030992111,8cyclictest2357682-21expr11:40:111
2261030992111,8cyclictest2297205-21taskset08:50:021
2261030992111,8cyclictest2284978-21/usr/sbin/munin08:15:151
2261030992111,8cyclictest1095-21runrttasks10:10:121
226103099210,20cyclictest0-21swapper/110:20:031
2261026992119,1cyclictest0-21swapper/012:33:020
2261026992119,1cyclictest0-21swapper/012:11:310
2261026992119,1cyclictest0-21swapper/011:36:510
2261026992119,1cyclictest0-21swapper/010:48:440
2261026992119,1cyclictest0-21swapper/010:16:540
2261026992119,1cyclictest0-21swapper/010:01:540
2261026992119,1cyclictest0-21swapper/008:04:320
2261026992119,1cyclictest0-21swapper/007:25:250
2261026992119,1cyclictest0-21swapper/007:12:080
2261026992118,2cyclictest3056-21PK-Backend11:45:220
2261026992118,2cyclictest3056-21PK-Backend08:12:170
2261026992118,1cyclictest3056-21PK-Backend11:30:230
2261026992118,1cyclictest3056-21PK-Backend11:30:230
226102699210,2cyclictest2332305-21pool-udisksd10:28:030
226102699210,20cyclictest0-21swapper/011:51:510
226102699210,20cyclictest0-21swapper/011:09:270
226102699210,20cyclictest0-21swapper/009:59:030
226102699210,20cyclictest0-21swapper/008:55:080
226102699210,20cyclictest0-21swapper/008:35:410
226102699210,20cyclictest0-21swapper/008:26:430
226102699210,20cyclictest0-21swapper/007:51:350
226102699210,1cyclictest0-21swapper/009:38:360
2261035992018,1cyclictest3056-21PK-Backend10:30:413
2261035992018,1cyclictest3056-21PK-Backend10:22:233
2261035992018,1cyclictest3056-21PK-Backend09:44:463
2261035992018,1cyclictest3056-21PK-Backend08:57:553
2261035992018,1cyclictest3056-21PK-Backend08:41:533
2261035992018,1cyclictest0-21swapper/307:35:343
2261035992017,2cyclictest3056-21PK-Backend12:36:283
226103599200,1cyclictest3056-21PK-Backend11:55:463
226103599200,1cyclictest3056-21PK-Backend10:59:143
226103599200,1cyclictest3056-21PK-Backend10:27:513
226103599200,1cyclictest3056-21PK-Backend08:28:553
226103599200,19cyclictest3056-21PK-Backend11:33:423
226103599200,19cyclictest3056-21PK-Backend11:33:413
226103599200,19cyclictest3056-21PK-Backend08:33:403
226103599200,19cyclictest3056-21PK-Backend08:19:353
226103599200,18cyclictest3056-21PK-Backend07:13:313
226103399206,3cyclictest978-21snmpd08:00:322
226103399203,4cyclictest436-20systemd-journal08:00:002
2261033992018,1cyclictest0-21swapper/210:13:022
2261033992018,1cyclictest0-21swapper/209:58:512
2261033992018,1cyclictest0-21swapper/208:47:452
226103399201,5cyclictest2356177-21diskstats11:35:152
226103399200,2cyclictest0-21swapper/211:06:292
226103399200,1cyclictest0-21swapper/208:55:372
226103399200,19cyclictest0-21swapper/210:53:052
226103099209,9cyclictest2373215-21cpuspeed12:25:111
2261030992013,6cyclictest2372264-21python312:20:211
2261030992013,5cyclictest2351816-21munin-run11:25:001
2261030992012,7cyclictest2371121-21cron12:20:011
2261030992012,6cyclictest2324182-21cut10:05:121
2261030992012,6cyclictest2299110-21cpuspeed_turbos08:55:111
2261030992012,6cyclictest2283029-21cut08:10:131
2261030992011,7cyclictest2315844-21/usr/sbin/munin09:40:161
2261030992011,7cyclictest2267284-21df07:25:121
2261030992010,8cyclictest2318065-21cat09:45:201
2261030992010,8cyclictest2270940-21expr07:35:141
226103099200,19cyclictest0-21swapper/109:04:191
2261026992018,1cyclictest0-21swapper/012:20:510
2261026992018,1cyclictest0-21swapper/012:04:020
2261026992018,1cyclictest0-21swapper/011:16:090
2261026992018,1cyclictest0-21swapper/009:42:280
2261026992018,1cyclictest0-21swapper/009:22:460
2261026992018,1cyclictest0-21swapper/009:19:090
2261026992018,1cyclictest0-21swapper/008:52:320
2261026992018,1cyclictest0-21swapper/008:45:430
2261026992018,1cyclictest0-21swapper/007:21:340
2261026992015,3cyclictest2351687-21chrt11:23:390
2261026992014,5cyclictest883-21dbus-daemon09:10:000
226102699200,19cyclictest0-21swapper/012:27:210
226102699200,19cyclictest0-21swapper/011:10:280
226102699200,19cyclictest0-21swapper/010:23:230
226102699200,19cyclictest0-21swapper/008:19:340
226102699200,19cyclictest0-21swapper/007:58:520
226102699200,19cyclictest0-21swapper/007:42:430
226102699200,19cyclictest0-21swapper/007:33:500
226102699200,19cyclictest0-21swapper/007:18:160
139099200,14rtkit-daemon3056-21PK-Backend11:29:313
139099200,14rtkit-daemon3056-21PK-Backend09:37:103
226103599199,2cyclictest42-21ksoftirqd/307:25:013
2261035991917,1cyclictest3056-21PK-Backend12:23:303
2261035991917,1cyclictest3056-21PK-Backend11:22:333
2261035991917,1cyclictest3056-21PK-Backend09:21:413
2261035991917,1cyclictest3056-21PK-Backend08:06:373
2261035991917,1cyclictest0-21swapper/309:45:473
2261035991916,2cyclictest3056-21PK-Backend10:47:183
2261035991916,2cyclictest3056-21PK-Backend07:32:093
226103599190,1cyclictest3056-21PK-Backend11:15:033
226103599190,1cyclictest3056-21PK-Backend10:11:163
226103599190,1cyclictest3056-21PK-Backend08:37:393
226103599190,1cyclictest3056-21PK-Backend07:57:183
226103599190,1cyclictest3056-21PK-Backend07:51:033
2261033991917,1cyclictest0-21swapper/211:18:062
2261033991917,1cyclictest0-21swapper/210:39:392
2261030991911,6cyclictest2339808-21grep10:50:111
2261026991917,1cyclictest0-21swapper/010:57:160
2261026991917,1cyclictest0-21swapper/010:33:450
2261026991917,1cyclictest0-21swapper/010:08:230
2261026991917,1cyclictest0-21swapper/009:13:410
2261026991917,1cyclictest0-21swapper/007:47:090
2261026991912,5cyclictest2377661-21cat12:35:200
2261026991912,5cyclictest2363582-21grep11:55:200
226102699190,1cyclictest0-21swapper/011:41:300
226102699190,1cyclictest0-21swapper/010:51:350
226102699190,1cyclictest0-21swapper/010:13:030
226102699190,1cyclictest0-21swapper/009:47:060
226102699190,18cyclictest0-21swapper/011:00:080
226102699190,18cyclictest0-21swapper/009:54:230
226102699190,17cyclictest2310767-21python309:25:200
139099190,13rtkit-daemon3056-21PK-Backend09:57:163
2261035991816,1cyclictest3056-21PK-Backend11:11:253
2261035991816,1cyclictest3056-21PK-Backend09:09:093
2261035991816,1cyclictest3056-21PK-Backend08:03:473
2261035991816,1cyclictest0-21swapper/310:05:483
2261035991815,2cyclictest3056-21PK-Backend12:13:223
2261035991815,2cyclictest3056-21PK-Backend11:38:223
226103599180,1cyclictest3056-21PK-Backend10:50:563
226103599180,17cyclictest3056-21PK-Backend09:33:363
2261033991810,6cyclictest2164-21gdbus09:03:122
226103399180,4cyclictest1-21systemd10:30:002
2261035991715,1cyclictest0-21swapper/309:26:193
2261033991715,1cyclictest0-21swapper/209:54:112
226102699179,6cyclictest2289789-21apt08:30:110
226102699170,16cyclictest2336358-21tr10:40:120
2261035991614,1cyclictest0-21swapper/309:00:313
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional