You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 03:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu May 28, 2026 00:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
216608021230,2sleep30-21swapper/322:00:113
213989021050,3sleep20-21swapper/220:45:112
22001572980,4sleep33056-21PK-Backend23:35:233
210560227439,13sleep30-21swapper/319:09:343
139099740,62rtkit-daemon0-21swapper/219:08:112
210505527053,9sleep10-21swapper/119:05:261
210529126343,14sleep00-21swapper/019:05:330
21308222530,6sleep1882-21avahi-daemon20:20:001
21157522430,4sleep10-21swapper/119:35:131
21278412390,4sleep013-21ksoftirqd/020:10:130
210577899394,22cyclictest0-21swapper/222:10:412
210577899384,22cyclictest0-21swapper/221:17:112
210577899372,23cyclictest0-21swapper/223:54:142
210577899371,23cyclictest0-21swapper/219:22:072
210577899370,20cyclictest2174118-21chrt22:21:512
210577899370,20cyclictest2174118-21chrt22:21:512
210577899365,17cyclictest0-21swapper/220:03:292
210577899362,21cyclictest0-21swapper/219:51:062
210577899360,21cyclictest2147942-21taskset21:08:132
210577899360,20cyclictest2118461-21chrt19:43:202
21162362350,3sleep03056-21PK-Backend19:35:200
210577899353,18cyclictest0-21swapper/200:04:052
210577899352,18cyclictest0-21swapper/223:43:442
210577899352,18cyclictest0-21swapper/222:34:512
210577899352,18cyclictest0-21swapper/222:27:412
210577899352,18cyclictest0-21swapper/221:03:282
210577899351,19cyclictest0-21swapper/221:47:552
210577899351,19cyclictest0-21swapper/200:19:152
210577899350,21cyclictest2182799-21seq22:46:562
210577899343,18cyclictest0-21swapper/220:20:492
210577899342,18cyclictest0-21swapper/200:32:042
210577899340,19cyclictest2120267-21chrt19:49:182
210577899340,19cyclictest2112773-21chrt19:25:362
210577899340,19cyclictest2112773-21chrt19:25:362
210577899332,18cyclictest0-21swapper/221:33:232
210577899331,22cyclictest2127273-21turbostat.cron20:14:552
210577899331,18cyclictest0-21swapper/221:58:052
210577899330,19cyclictest2212660-21chrt00:12:592
210577899330,19cyclictest2186917-21cat23:00:112
210577899322,17cyclictest0-21swapper/223:38:022
210577899320,20cyclictest2204000-21chrt23:48:132
210577899320,18cyclictest2180964-21chrt22:40:362
210577899320,18cyclictest2137593-21taskset20:38:582
210577899311,18cyclictest2221305-21chrt00:37:372
210577899311,18cyclictest0-21swapper/222:52:032
210577899310,1cyclictest0-21swapper/219:30:192
210577899310,19cyclictest2172497-21chrt22:18:182
210577899310,19cyclictest2167411-21taskset22:04:472
210577899310,18cyclictest2216238-21chrt00:24:222
210577899310,18cyclictest2108466-21cstates19:15:132
210577899307,12cyclictest1095-21runrttasks00:29:032
210577899301,16cyclictest0-21swapper/220:53:332
210577899300,18cyclictest21971552sleep223:29:322
2105778992913,14cyclictest2206538-21cat23:55:152
210577899280,15cyclictest2189146-21sed23:05:162
210577899270,25cyclictest2148599-21/usr/sbin/munin21:10:142
2105773992718,7cyclictest2145269-21entropy21:00:151
210577899252,18cyclictest2197480-21apt23:30:112
210577899251,22cyclictest2163358-21awk21:50:192
2105778992512,11cyclictest978-21snmpd19:57:582
210577899250,13cyclictest0-21swapper/220:07:572
2105780992419,3cyclictest436-20systemd-journal00:34:553
210577899249,13cyclictest2116300-21cat19:35:212
210577899241,20cyclictest2157459-21cpuspeed_turbos21:35:132
2105773992414,7cyclictest436-20systemd-journal23:05:011
2105770992416,6cyclictest2118594-21cron19:45:000
2105780992314,7cyclictest883-21dbus-daemon00:28:163
210577899238,14cyclictest2193823-21grep23:20:002
210577899238,13cyclictest2195052-21cat23:20:202
210577899238,13cyclictest2195052-21cat23:20:202
210577899232,17cyclictest2185055-21apt22:55:112
210577899231,19cyclictest2168573-21python322:05:212
210577899230,20cyclictest1095-21runrttasks22:35:112
2105773992317,5cyclictest26-21ksoftirqd/120:32:581
2105773992316,5cyclictest2151707-21sshd21:18:331
2105770992321,1cyclictest0-21swapper/020:15:220
2105770992320,2cyclictest0-21swapper/023:35:230
21453172220,4sleep33056-21PK-Backend21:00:163
2105780992215,5cyclictest1-21systemd19:34:323
210578099221,20cyclictest42-21ksoftirqd/321:08:003
210577899225,15cyclictest978-21snmpd20:27:142
210577899222,17cyclictest2138817-21python320:40:202
2105778992217,4cyclictest0-21swapper/220:32:382
210577899220,20cyclictest2209612-21munin-node00:05:112
210577899220,20cyclictest2129967-21awk20:15:172
2105773992219,2cyclictest3056-21PK-Backend00:14:351
2105773992211,8cyclictest26-21ksoftirqd/121:20:201
2105770992220,1cyclictest0-21swapper/021:21:330
2105770992220,1cyclictest0-21swapper/020:51:370
2105770992220,1cyclictest0-21swapper/020:32:320
2105770992220,1cyclictest0-21swapper/019:51:560
2105770992220,1cyclictest0-21swapper/019:34:050
139099220,3rtkit-daemon3056-21PK-Backend21:42:283
139099220,15rtkit-daemon3056-21PK-Backend20:29:093
139099220,14rtkit-daemon0-21swapper/320:38:323
21492032210,4chrt3056-21PK-Backend21:10:161
2105780992113,6cyclictest2168414-21ntp_kernel_err22:05:183
210578099211,18cyclictest2213103-21apt00:15:113
2105778992119,1cyclictest0-21swapper/221:23:282
210577899211,18cyclictest2156-21gnome-shell19:12:092
210577899211,17cyclictest436-20systemd-journal23:15:002
210577899210,19cyclictest2154635-21proc_pri21:25:182
210577899210,19cyclictest2154635-21proc_pri21:25:182
210577899210,18cyclictest2159106-21ls21:40:112
2105773992119,1cyclictest3056-21PK-Backend22:54:461
2105773992119,1cyclictest3056-21PK-Backend22:20:391
2105773992119,1cyclictest3056-21PK-Backend22:20:391
2105773992119,1cyclictest3056-21PK-Backend00:36:121
2105773992119,1cyclictest0-21swapper/123:05:071
2105773992119,1cyclictest0-21swapper/119:13:011
2105773992119,1cyclictest0-21swapper/100:32:131
2105773992118,2cyclictest3056-21PK-Backend00:25:501
2105773992113,6cyclictest26-21ksoftirqd/120:05:011
2105773992113,6cyclictest2201683-21ntp_states23:40:191
2105773992112,7cyclictest26-21ksoftirqd/121:50:091
2105773992112,3cyclictest26-21ksoftirqd/121:00:011
2105773992111,7cyclictest26-21ksoftirqd/120:50:211
210577399210,20cyclictest3056-21PK-Backend21:09:551
210577399210,20cyclictest0-21swapper/123:35:271
210577399210,19cyclictest3056-21PK-Backend21:46:521
210577399210,19cyclictest3056-21PK-Backend19:24:161
2105770992119,1cyclictest0-21swapper/023:34:360
2105770992119,1cyclictest0-21swapper/023:11:030
2105770992119,1cyclictest0-21swapper/021:59:520
2105770992119,1cyclictest0-21swapper/021:38:290
2105770992119,1cyclictest0-21swapper/021:30:590
2105770992119,1cyclictest0-21swapper/020:22:180
2105770992119,1cyclictest0-21swapper/000:39:520
2105770992113,6cyclictest2145020-21expr21:00:130
210577099210,2cyclictest2186496-21check-new-relea22:59:550
210577099210,20cyclictest0-21swapper/022:31:210
210577099210,20cyclictest0-21swapper/022:01:480
210577099210,20cyclictest0-21swapper/000:02:200
21353462200,4sleep33056-21PK-Backend20:30:213
2105780992018,1cyclictest3056-21PK-Backend20:07:303
2105780992018,1cyclictest0-21swapper/323:45:493
2105780992018,1cyclictest0-21swapper/319:49:123
2105780992015,4cyclictest978-21snmpd22:27:453
2105780992015,4cyclictest42-21ksoftirqd/322:23:023
2105780992015,4cyclictest42-21ksoftirqd/322:23:023
2105780992015,3cyclictest978-21snmpd23:31:563
2105780992015,3cyclictest2212233-21python300:10:193
2105780992012,6cyclictest2183681-21df22:50:123
210578099201,17cyclictest2205284-21python323:50:203
210578099200,19cyclictest0-21swapper/322:45:143
210578099200,19cyclictest0-21swapper/322:16:073
210578099200,19cyclictest0-21swapper/319:39:113
210578099200,19cyclictest0-21swapper/300:05:173
210577899201,17cyclictest2143129-21/usr/sbin/munin20:55:232
2105773992018,1cyclictest3056-21PK-Backend22:35:331
2105773992018,1cyclictest3056-21PK-Backend19:45:251
2105773992018,1cyclictest3056-21PK-Backend19:30:521
2105773992018,1cyclictest0-21swapper/121:41:521
2105773992018,1cyclictest0-21swapper/121:39:561
2105773992018,1cyclictest0-21swapper/120:25:071
2105773992013,5cyclictest2186704-21sed23:00:011
2105773992012,6cyclictest26-21ksoftirqd/121:25:111
2105773992012,6cyclictest26-21ksoftirqd/121:25:111
2105773992012,6cyclictest26-21ksoftirqd/120:10:011
2105773992012,6cyclictest2191730-21apt-get23:14:591
2105773992012,3cyclictest26-21ksoftirqd/120:47:091
2105773992011,3cyclictest26-21ksoftirqd/123:15:171
210577399200,1cyclictest3056-21PK-Backend00:20:171
210577399200,19cyclictest3056-21PK-Backend22:46:471
210577399200,19cyclictest3056-21PK-Backend22:26:271
210577399200,19cyclictest3056-21PK-Backend22:07:411
210577399200,19cyclictest3056-21PK-Backend20:40:201
210577399200,19cyclictest3056-21PK-Backend00:06:451
210577399200,19cyclictest0-21swapper/123:26:021
2105770992018,1cyclictest0-21swapper/023:22:260
2105770992018,1cyclictest0-21swapper/023:22:250
2105770992018,1cyclictest0-21swapper/023:16:440
2105770992018,1cyclictest0-21swapper/022:43:120
2105770992018,1cyclictest0-21swapper/020:45:500
2105770992013,5cyclictest2107961-21cat19:15:030
2105770992012,6cyclictest2183062-21perf22:50:000
210577099200,1cyclictest0-21swapper/022:05:400
210577099200,19cyclictest3056-21PK-Backend22:25:340
210577099200,19cyclictest0-21swapper/023:02:040
210577099200,19cyclictest0-21swapper/022:13:580
210577099200,19cyclictest0-21swapper/021:47:220
210577099200,19cyclictest0-21swapper/021:40:250
210577099200,19cyclictest0-21swapper/021:13:280
210577099200,19cyclictest0-21swapper/019:49:130
210577099200,19cyclictest0-21swapper/019:24:590
139099200,14rtkit-daemon3056-21PK-Backend22:12:223
210578099198,8cyclictest42-21ksoftirqd/319:15:183
2105780991917,1cyclictest0-21swapper/322:37:433
2105780991917,1cyclictest0-21swapper/321:47:403
2105780991915,3cyclictest2216305-21turbostat00:25:003
2105780991911,7cyclictest2110408-21awk19:20:153
2105780991911,6cyclictest2165742-21pluginstate22:00:013
2105780991911,6cyclictest2107436-21cron19:15:003
210578099190,1cyclictest3056-21PK-Backend23:20:293
210578099190,1cyclictest3056-21PK-Backend23:20:283
210578099190,1cyclictest0-21swapper/319:28:373
210578099190,1cyclictest0-21swapper/319:28:373
210578099190,1cyclictest0-21swapper/300:35:453
210578099190,18cyclictest0-21swapper/323:55:583
210578099190,17cyclictest2123198-21python319:55:193
2105773991917,1cyclictest3056-21PK-Backend23:45:421
2105773991917,1cyclictest3056-21PK-Backend22:42:081
2105773991917,1cyclictest3056-21PK-Backend19:44:041
2105773991917,1cyclictest3056-21PK-Backend19:25:381
2105773991917,1cyclictest3056-21PK-Backend19:25:381
2105773991917,1cyclictest3056-21PK-Backend00:16:451
2105773991917,1cyclictest3056-21PK-Backend00:04:341
2105773991917,1cyclictest0-21swapper/121:30:291
2105773991916,2cyclictest3056-21PK-Backend23:30:211
2105773991916,2cyclictest3056-21PK-Backend21:56:191
2105773991912,6cyclictest2132048-21sed20:20:211
2105773991911,6cyclictest26-21ksoftirqd/123:20:121
2105773991911,6cyclictest26-21ksoftirqd/123:20:121
2105773991911,6cyclictest2176863-21awk22:30:141
2105773991910,2cyclictest26-21ksoftirqd/120:10:211
210577399190,1cyclictest0-21swapper/123:50:491
210577399190,18cyclictest0-21swapper/122:13:291
2105770991917,1cyclictest0-21swapper/023:55:520
2105770991917,1cyclictest0-21swapper/022:52:310
2105770991917,1cyclictest0-21swapper/021:05:240
2105770991917,1cyclictest0-21swapper/020:36:580
2105770991917,1cyclictest0-21swapper/020:00:080
2105770991917,1cyclictest0-21swapper/000:18:020
2105770991916,2cyclictest3056-21PK-Backend23:50:450
2105770991911,6cyclictest2196302-21cat23:25:160
2105770991910,7cyclictest2163092-21memory21:50:160
210577099190,1cyclictest0-21swapper/022:35:140
210577099190,1cyclictest0-21swapper/021:26:330
210577099190,1cyclictest0-21swapper/021:26:330
210577099190,1cyclictest0-21swapper/020:41:580
210577099190,1cyclictest0-21swapper/020:06:570
210577099190,1cyclictest0-21swapper/019:59:200
210577099190,18cyclictest3056-21PK-Backend23:40:300
210578099189,7cyclictest2185633-21memory22:55:163
2105780991816,1cyclictest0-21swapper/321:36:393
2105780991815,2cyclictest3056-21PK-Backend20:02:303
2105780991814,3cyclictest978-21snmpd20:59:333
2105780991811,2cyclictest42-21ksoftirqd/319:50:103
2105780991810,6cyclictest436-20systemd-journal21:52:323
210578099180,8cyclictest0-21swapper/321:18:013
210578099180,17cyclictest3056-21PK-Backend23:13:183
210578099180,17cyclictest0-21swapper/323:28:463
210578099180,16cyclictest2177992-21cut22:35:003
210577399189,3cyclictest26-21ksoftirqd/119:15:231
2105773991816,1cyclictest3056-21PK-Backend23:56:441
2105773991816,1cyclictest3056-21PK-Backend19:52:491
2105773991816,1cyclictest0-21swapper/120:35:211
2105773991816,1cyclictest0-21swapper/119:55:331
2105770991816,1cyclictest0-21swapper/023:49:570
2105770991816,1cyclictest0-21swapper/023:05:360
2105770991816,1cyclictest0-21swapper/022:23:380
2105770991816,1cyclictest0-21swapper/022:23:380
2105770991816,1cyclictest0-21swapper/022:17:500
2105770991816,1cyclictest0-21swapper/020:56:520
2105770991816,1cyclictest0-21swapper/019:26:220
2105770991816,1cyclictest0-21swapper/019:26:210
2105770991811,5cyclictest2150933-21load21:15:170
2105770991811,5cyclictest2106495-21expr19:10:160
2105780991715,1cyclictest0-21swapper/323:43:073
2105780991715,1cyclictest0-21swapper/322:42:443
210578099171,1cyclictest141rcu_preempt23:05:013
210578099171,14cyclictest436-20systemd-journal00:05:013
2105780991711,4cyclictest2117941-21/usr/sbin/munin19:40:193
210578099170,3cyclictest2138434-21irqstats20:40:163
210578099170,16cyclictest3056-21PK-Backend20:12:303
210577099172,11cyclictest2132745-21apt20:25:100
2105770991715,1cyclictest0-21swapper/000:10:590
210578099160,3cyclictest2149697-21users21:10:223
210578099160,14cyclictest21323692sleep320:23:473
2105773991614,1cyclictest3056-21PK-Backend22:16:471
2105773991614,1cyclictest3056-21PK-Backend22:03:141
2105770991614,1cyclictest0-21swapper/000:05:180
2105770991612,3cyclictest2214814-21/usr/sbin/munin00:20:160
210578099158,6cyclictest2157136-21sed21:35:003
210578099155,2cyclictest141rcu_preempt23:10:013
210578099151,5cyclictest141rcu_preempt21:20:103
2105780991513,1cyclictest3056-21PK-Backend20:47:453
21539312140,3sleep33056-21PK-Backend21:25:103
21539312140,3sleep33056-21PK-Backend21:25:103
2105780991410,3cyclictest3056-21PK-Backend20:53:343
210578099131,1cyclictest141rcu_preempt20:15:203
2105780991311,1cyclictest3056-21PK-Backend23:15:293
210577099139,2cyclictest1095-21runrttasks00:30:180
210577099137,4cyclictest2216854-21expr00:25:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional