You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-10 - 20:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Aug 10, 2026 12:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
224639424110,10sleep10-21swapper/107:05:151
230463121050,3sleep10-21swapper/109:50:011
226466821030,3sleep33056-21PK-Backend07:55:143
224670928343,33sleep00-21swapper/007:05:230
139099830,70rtkit-daemon0-21swapper/207:05:282
224738727740,14sleep30-21swapper/307:08:563
22858252550,5sleep10-21swapper/108:55:191
22645712480,3sleep00-21swapper/007:55:100
23318582410,2sleep00-21swapper/011:05:190
23385982380,1sleep00-21swapper/011:25:160
23385982380,1sleep00-21swapper/011:25:160
23139372370,7sleep2341ksoftirqd/210:15:112
22718832350,2sleep33056-21PK-Backend08:15:153
2247624993513,20cyclictest2279877-21grep08:40:003
224762499327,6cyclictest883-21dbus-daemon11:15:013
2247624993212,18cyclictest2258916-21dpkg07:40:013
224761699304,14cyclictest0-21swapper/112:01:121
224762499299,18cyclictest2363088-21grep12:35:003
2247624992913,14cyclictest2255976-21ntp_states07:30:183
224762499290,27cyclictest2312277-21fw_packets10:10:163
139099290,22rtkit-daemon3056-21PK-Backend11:50:223
224762499287,5cyclictest2277206-21cat08:30:163
2247624992811,15cyclictest2285320-21/usr/sbin/munin08:55:233
224762499280,26cyclictest2274083-21swap08:20:213
224762499279,17cyclictest2262459-21/usr/sbin/munin07:50:223
224762499278,17cyclictest2342596-21sed11:35:223
224762499276,5cyclictest2295335-21seq09:23:293
224762499276,5cyclictest2261826-21who07:45:203
224762499272,4cyclictest0-21swapper/309:25:113
2247624992710,15cyclictest2249950-21timerandwakeup07:15:043
224762499270,24cyclictest3056-21PK-Backend10:15:113
2247619992716,9cyclictest2258526-21unin-run07:39:592
224761699279,17cyclictest883-21dbus-daemon09:10:011
224761699279,16cyclictest2360456-21ntpq12:25:201
224761699277,18cyclictest883-21dbus-daemon09:44:111
224761699271,13cyclictest2270921-21unattended-upgr08:11:381
2247616992711,14cyclictest2261512-21awk07:45:171
139099270,19rtkit-daemon3056-21PK-Backend08:42:253
224762499268,16cyclictest2287082-21cpu09:00:123
224762499267,17cyclictest2292414-21cpuspeed_turbos09:15:143
224762499267,17cyclictest2289786-21timerandwakeup09:05:203
224762499261,5cyclictest2316858-21sort10:25:003
224762499261,5cyclictest2316858-21sort10:25:003
224762499261,10cyclictest1-21systemd10:34:343
224762499260,25cyclictest0-21swapper/311:41:393
224762499260,19cyclictest0-21swapper/312:29:533
224761699268,16cyclictest1095-21runrttasks09:50:551
224761699261,4cyclictest2175-21JS10:27:481
224762499258,4cyclictest42-21ksoftirqd/311:45:163
224762499258,15cyclictest2317293-21cstates10:25:123
224762499257,16cyclictest0-21swapper/307:25:143
224762499255,5cyclictest2252170-21sh07:20:153
224762499255,4cyclictest978-21snmpd10:40:183
224762499252,8cyclictest2276277-21sshd08:28:313
224762499251,8cyclictest2297559-21/usr/sbin/munin09:30:123
224762499251,7cyclictest2321288-21ntp_states10:35:193
224762499251,6cyclictest2247788-21/usr/sbin/munin07:10:223
224762499250,6cyclictest2336574-21grep11:20:143
224762499250,6cyclictest2336574-21grep11:20:133
224762499250,24cyclictest978-21snmpd11:18:053
224762499250,24cyclictest2258333-21sshd07:38:123
224762499250,19cyclictest3056-21PK-Backend08:09:553
224761699258,15cyclictest2284045-21if_enp2s008:50:161
2247616992516,7cyclictest883-21dbus-daemon09:39:071
224762499247,3cyclictest3056-21PK-Backend11:05:023
224762499241,7cyclictest2299207-21apt09:35:093
224762499240,22cyclictest3056-21PK-Backend12:00:143
224761699248,14cyclictest2342130-21ls11:35:181
224761699248,14cyclictest2308368-21cstates10:00:141
224761699248,14cyclictest2283348-21sh08:50:001
224761699247,15cyclictest2352767-21expr12:05:131
224761699245,5cyclictest2267005-21sed08:00:231
2247616992419,4cyclictest0-21swapper/109:01:171
224761699241,11cyclictest2331581-21diskstats11:05:131
224761699240,8cyclictest0-21swapper/109:55:131
224761699240,22cyclictest0-21swapper/110:55:061
23282892230,8sleep013-21ksoftirqd/010:55:200
224762499237,3cyclictest3056-21PK-Backend11:59:433
224762499230,8cyclictest2291938-21seq09:14:233
224762499230,7cyclictest2338443-21expr11:25:123
224762499230,7cyclictest2338443-21expr11:25:123
224761699238,13cyclictest26-21ksoftirqd/110:15:121
224761699238,13cyclictest2291070-21latency09:10:171
224761699237,15cyclictest2298507-21timerwakeupswit09:30:231
224761699235,16cyclictest2348616-21latency_hist11:55:011
224761699233,3cyclictest2311020-21swap10:05:221
224761699232,17cyclictest2355990-21apt12:15:101
224761699232,17cyclictest2355990-21apt12:15:101
224761699232,17cyclictest2329475-21apt11:00:121
224761699231,9cyclictest2271852-21diskstats08:15:131
224761699230,1cyclictest3056-21PK-Backend11:12:161
2247624992220,1cyclictest3056-21PK-Backend08:01:433
224762499221,6cyclictest2306120-21chrt09:54:443
224762499221,6cyclictest1-21systemd10:10:003
224762499221,5cyclictest2341339-21dump-pmu-power11:35:003
2247624992211,9cyclictest3056-21PK-Backend10:58:083
224762499220,7cyclictest2364682-21awk12:39:593
224762499220,6cyclictest2357225-21chrt12:17:123
224762499220,6cyclictest2357225-21chrt12:17:123
2247619992220,1cyclictest0-21swapper/212:19:422
2247619992220,1cyclictest0-21swapper/212:19:422
2247619992220,1cyclictest0-21swapper/210:54:062
2247619992214,6cyclictest2293741-21latency_hist09:20:002
224761699226,5cyclictest2278173-21systemctl08:35:001
224761699224,5cyclictest2265525-21seq07:58:121
224761699224,4cyclictest2280137-21awk08:40:121
2247616992219,2cyclictest3056-21PK-Backend10:27:081
2247616992218,3cyclictest3056-21PK-Backend07:35:461
224761699221,4cyclictest2278596-21cpuspeed_turbos08:35:131
224761699221,3cyclictest0-21swapper/110:45:111
224761699220,20cyclictest3056-21PK-Backend10:11:331
2247612992215,5cyclictest2309747-21grep10:05:000
139099220,15rtkit-daemon3056-21PK-Backend08:13:563
224762499210,19cyclictest3056-21PK-Backend11:00:433
224762499210,19cyclictest3056-21PK-Backend10:03:313
2247619992119,1cyclictest0-21swapper/212:38:562
2247619992119,1cyclictest0-21swapper/212:33:562
2247619992119,1cyclictest0-21swapper/212:22:202
2247619992119,1cyclictest0-21swapper/212:06:252
2247619992119,1cyclictest0-21swapper/211:51:172
2247619992119,1cyclictest0-21swapper/211:22:252
2247619992119,1cyclictest0-21swapper/211:22:242
2247619992119,1cyclictest0-21swapper/210:55:072
2247619992119,1cyclictest0-21swapper/210:35:052
2247619992119,1cyclictest0-21swapper/210:31:422
2247619992119,1cyclictest0-21swapper/210:23:172
2247619992119,1cyclictest0-21swapper/210:23:172
2247619992119,1cyclictest0-21swapper/209:23:012
2247619992119,1cyclictest0-21swapper/209:03:472
2247619992119,1cyclictest0-21swapper/208:58:132
2247619992119,1cyclictest0-21swapper/208:24:342
2247619992119,1cyclictest0-21swapper/208:06:292
2247619992118,2cyclictest0-21swapper/212:27:212
2247619992118,2cyclictest0-21swapper/211:46:512
2247619992118,2cyclictest0-21swapper/209:47:262
2247619992117,3cyclictest2302540-21latency_hist09:45:002
2247619992113,6cyclictest2343902-21cat11:40:182
224761999210,20cyclictest0-21swapper/210:02:552
224761699217,12cyclictest2250150-21cpuspeed_turbos07:15:121
224761699215,3cyclictest2363924-21ntpq12:35:181
224761699215,2cyclictest0-21swapper/107:20:281
224761699215,14cyclictest0-21swapper/112:30:141
224761699213,6cyclictest2268644-21sed08:05:231
224761699211,9cyclictest2338140-21apt11:25:111
224761699211,9cyclictest2338140-21apt11:25:101
2247616992119,1cyclictest3056-21PK-Backend10:21:191
2247616992119,1cyclictest3056-21PK-Backend10:21:191
2247616992119,1cyclictest3056-21PK-Backend07:10:581
2247616992119,1cyclictest0-21swapper/112:11:321
2247616992119,1cyclictest0-21swapper/108:26:361
2247616992118,2cyclictest3056-21PK-Backend09:17:191
224761699211,7cyclictest2318793-21/usr/sbin/munin10:30:251
2247612992118,2cyclictest0-21swapper/008:11:370
224761299210,19cyclictest3056-21PK-Backend08:20:220
139099210,13rtkit-daemon0-21swapper/210:07:382
2247624992018,1cyclictest3056-21PK-Backend12:06:123
224762499200,7cyclictest2283065-21chrt08:46:393
224762499200,19cyclictest3056-21PK-Backend09:57:233
224762499200,19cyclictest3056-21PK-Backend09:41:003
2247619992018,1cyclictest0-21swapper/212:01:592
2247619992018,1cyclictest0-21swapper/211:35:092
2247619992018,1cyclictest0-21swapper/211:06:162
2247619992018,1cyclictest0-21swapper/209:34:362
2247619992018,1cyclictest0-21swapper/208:38:052
2247619992018,1cyclictest0-21swapper/207:47:022
2247619992018,1cyclictest0-21swapper/207:32:492
2247619992018,1cyclictest0-21swapper/207:26:272
2247619992017,2cyclictest0-21swapper/211:11:432
2247619992013,5cyclictest2263225-21ntp_states07:50:192
224761999200,1cyclictest0-21swapper/211:32:182
224761999200,1cyclictest0-21swapper/208:50:222
224761999200,19cyclictest0-21swapper/211:55:442
224761999200,19cyclictest0-21swapper/211:03:242
224761999200,19cyclictest0-21swapper/209:28:422
224761999200,19cyclictest0-21swapper/209:09:352
224761999200,19cyclictest0-21swapper/208:25:212
224761999200,19cyclictest0-21swapper/208:19:272
224761999200,19cyclictest0-21swapper/207:41:552
224761999200,19cyclictest0-21swapper/207:16:062
224761699201,9cyclictest2253879-21df_inode07:25:151
2247616992018,1cyclictest3056-21PK-Backend11:21:091
2247616992018,1cyclictest3056-21PK-Backend11:21:091
2247616992018,1cyclictest3056-21PK-Backend08:21:291
2247616992018,1cyclictest0-21swapper/111:56:371
2247616992018,1cyclictest0-21swapper/107:44:041
2247616992017,2cyclictest3056-21PK-Backend11:44:211
224761699200,7cyclictest2359184-21apt-get12:24:591
224761699200,7cyclictest2322770-21cut10:40:171
224761699200,5cyclictest2326934-21sshd10:51:361
224761699200,4cyclictest436-20systemd-journal09:24:591
2247612992018,1cyclictest0-21swapper/011:50:040
2247612992018,1cyclictest0-21swapper/010:12:430
2247612992018,1cyclictest0-21swapper/009:00:100
2247612992017,2cyclictest3056-21PK-Backend11:22:390
2247612992017,2cyclictest3056-21PK-Backend11:22:390
2247612992013,5cyclictest2315865-21wc10:20:160
2247612992013,5cyclictest2315865-21wc10:20:160
2247612992012,6cyclictest436-20systemd-journal07:45:000
224761299200,1cyclictest0-21swapper/012:20:390
224761299200,19cyclictest3056-21PK-Backend09:05:100
224761299200,19cyclictest0-21swapper/012:00:300
224761299200,19cyclictest0-21swapper/011:48:150
224761299200,19cyclictest0-21swapper/010:51:080
224761299200,19cyclictest0-21swapper/010:36:160
224761299200,19cyclictest0-21swapper/010:19:050
224761299200,19cyclictest0-21swapper/009:50:190
224761299200,19cyclictest0-21swapper/009:39:040
224761299200,19cyclictest0-21swapper/009:19:510
224761299200,19cyclictest0-21swapper/007:13:300
224761299200,18cyclictest3056-21PK-Backend12:30:250
139099200,14rtkit-daemon0-21swapper/209:56:262
2247624991917,1cyclictest3056-21PK-Backend10:45:563
224762499191,4cyclictest2359184-21apt12:25:003
224762499190,6cyclictest2326952-21chrt10:51:493
224762499190,6cyclictest2303328-21chrt09:45:143
224762499190,1cyclictest3056-21PK-Backend12:12:143
2247619991917,1cyclictest0-21swapper/210:48:252
2247619991917,1cyclictest0-21swapper/210:26:412
2247619991917,1cyclictest0-21swapper/209:10:562
2247619991917,1cyclictest0-21swapper/207:23:302
2247619991917,1cyclictest0-21swapper/207:13:012
224761999190,1cyclictest0-21swapper/211:16:092
224761999190,1cyclictest0-21swapper/208:13:592
224761999190,18cyclictest0-21swapper/211:26:512
224761999190,18cyclictest0-21swapper/211:26:502
224761999190,18cyclictest0-21swapper/209:38:222
224761699198,10cyclictest2322125-21latency_hist10:40:011
224761699193,3cyclictest0-21swapper/109:25:451
2247616991917,1cyclictest3056-21PK-Backend11:15:551
2247616991917,1cyclictest3056-21PK-Backend07:31:331
224761699191,4cyclictest1-21systemd07:55:011
224761699190,18cyclictest3056-21PK-Backend11:46:581
2247612991917,1cyclictest3056-21PK-Backend09:14:370
2247612991917,1cyclictest0-21swapper/011:10:230
2247612991917,1cyclictest0-21swapper/009:49:310
2247612991917,1cyclictest0-21swapper/009:33:560
2247612991917,1cyclictest0-21swapper/008:42:400
2247612991917,1cyclictest0-21swapper/007:54:400
2247612991917,1cyclictest0-21swapper/007:46:490
2247612991916,2cyclictest3056-21PK-Backend11:42:130
2247612991914,4cyclictest2330945-21perf11:05:010
2247612991914,3cyclictest2279905-21systemctl08:40:000
2247612991913,4cyclictest2271451-21sshd08:15:100
2247612991912,5cyclictest2294179-21tr09:20:130
2247612991911,6cyclictest2282413-21sed08:45:160
2247612991910,7cyclictest2324384-21df_inode10:45:140
224761299190,1cyclictest3056-21PK-Backend11:35:570
224761299190,1cyclictest0-21swapper/011:58:420
224761299190,1cyclictest0-21swapper/009:27:280
224761299190,1cyclictest0-21swapper/007:37:160
224761299190,18cyclictest3056-21PK-Backend08:26:500
224761299190,18cyclictest0-21swapper/012:13:360
224761299190,18cyclictest0-21swapper/010:27:160
224761299190,18cyclictest0-21swapper/009:59:250
224761299190,18cyclictest0-21swapper/008:33:000
224762499182,3cyclictest3056-21PK-Backend08:53:173
224761999189,7cyclictest2322865-21memory10:40:182
2247619991816,1cyclictest0-21swapper/212:10:022
2247619991816,1cyclictest0-21swapper/209:53:362
2247619991812,5cyclictest2277992-21seq08:33:172
2247619991812,4cyclictest34-21ksoftirqd/210:10:212
2247612991816,1cyclictest0-21swapper/012:18:020
2247612991816,1cyclictest0-21swapper/012:18:020
2247612991816,1cyclictest0-21swapper/010:09:320
2247612991816,1cyclictest0-21swapper/007:34:390
2247612991812,5cyclictest2267876-21expr08:05:140
2247612991811,5cyclictest883-21dbus-daemon09:00:000
224761299180,1cyclictest0-21swapper/007:19:390
2247619991715,1cyclictest0-21swapper/208:49:482
224761299173,13cyclictest2301433-21irqstats09:40:160
2247612991715,1cyclictest0-21swapper/010:43:390
2247612991712,3cyclictest2363175-21acpi12:35:100
224761999165,10cyclictest2281606-21sh08:44:592
224761999163,12cyclictest2266625-21needreboot08:00:192
224761299169,5cyclictest13-21ksoftirqd/008:50:130
2247612991614,1cyclictest0-21swapper/008:00:210
224761299160,14cyclictest1-21systemd11:19:590
224761999150,13cyclictest2265711-20unin-run08:00:012
224761299158,5cyclictest2352516-21basename12:05:100
224761299157,6cyclictest436-20systemd-journal10:35:000
2247612991513,1cyclictest0-21swapper/011:31:320
2247612991512,2cyclictest2359697-21/usr/sbin/munin12:25:120
224761299149,3cyclictest22538852sleep007:25:160
2247612991411,2cyclictest2251928-21cpuspeed_turbos07:20:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional