You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 12:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat Jun 27, 2026 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
228916022260,8sleep20-21swapper/219:05:202
230129221400,4sleep20-21swapper/219:40:012
229896421300,3sleep13056-21PK-Backend19:30:211
228971527640,9sleep00-21swapper/019:05:580
23970962640,11sleep234-21ksoftirqd/200:10:112
228973326443,15sleep30-21swapper/319:06:133
228989626241,14sleep10-21swapper/119:08:201
23670722500,4sleep342-21ksoftirqd/322:45:123
23290482470,3sleep00-21swapper/020:55:170
23147052380,4sleep126-21ksoftirqd/120:15:171
2290255993119,7cyclictest1095-21runrttasks20:24:201
2290269993014,13cyclictest2354400-21perf22:10:003
2290255992818,7cyclictest436-20systemd-journal23:20:001
22969702270,8sleep013-21ksoftirqd/019:25:200
229026999274,21cyclictest1095-21runrttasks19:30:353
23516482260,9sleep126-21ksoftirqd/122:00:191
23516482260,9sleep126-21ksoftirqd/122:00:181
229026999250,5cyclictest2294303-21cron19:19:593
2290260992516,7cyclictest2291103-21/usr/sbin/munin19:10:162
2290260992413,9cyclictest2384688-21unin-run23:35:002
139099240,16rtkit-daemon3056-21PK-Backend23:49:143
229026999238,3cyclictest3056-21PK-Backend23:09:173
229026999232,2cyclictest42-21ksoftirqd/320:17:393
2290260992315,6cyclictest1788610-21gdbus20:55:272
2290255992316,5cyclictest2328147-21munin-run20:55:001
229025499230,6cyclictest2317295-21ld-2.23.so20:24:210
229025499230,4cyclictest2322988-21cut20:40:000
139099230,16rtkit-daemon3056-21PK-Backend20:40:163
139099230,15rtkit-daemon3056-21PK-Backend22:51:323
2290269992220,1cyclictest3056-21PK-Backend21:18:503
2290269992219,2cyclictest3056-21PK-Backend21:26:473
2290269992219,2cyclictest3056-21PK-Backend21:07:343
2290269992219,2cyclictest3056-21PK-Backend21:07:343
229026999220,20cyclictest3056-21PK-Backend20:49:163
229026999220,20cyclictest3056-21PK-Backend19:54:403
2290255992214,6cyclictest436-20systemd-journal23:49:591
229025599220,20cyclictest3056-21PK-Backend21:38:121
229025499228,13cyclictest13-21ksoftirqd/019:20:000
229025499225,6cyclictest13-21ksoftirqd/020:00:110
2290254992220,1cyclictest0-21swapper/022:04:030
2290254992220,1cyclictest0-21swapper/022:04:030
2290254992220,1cyclictest0-21swapper/021:24:420
2290254992220,1cyclictest0-21swapper/021:04:000
2290254992220,1cyclictest0-21swapper/020:53:400
2290254992220,1cyclictest0-21swapper/020:08:100
2290254992220,1cyclictest0-21swapper/019:51:010
2290254992220,1cyclictest0-21swapper/000:38:490
2290254992218,3cyclictest0-21swapper/019:33:090
229025499220,21cyclictest0-21swapper/021:54:020
2290269992119,1cyclictest3056-21PK-Backend23:21:043
2290269992119,1cyclictest3056-21PK-Backend22:38:263
2290269992119,1cyclictest3056-21PK-Backend20:55:593
2290269992119,1cyclictest3056-21PK-Backend19:57:393
2290269992119,1cyclictest3056-21PK-Backend00:12:353
2290269992119,1cyclictest0-21swapper/322:25:483
2290269992119,1cyclictest0-21swapper/321:41:273
2290269992119,1cyclictest0-21swapper/319:43:593
2290269992113,6cyclictest883-21dbus-daemon19:30:013
229026999210,20cyclictest3056-21PK-Backend23:42:423
229026999210,20cyclictest0-21swapper/321:35:393
229026999210,1cyclictest3056-21PK-Backend21:00:513
229026999210,19cyclictest3056-21PK-Backend21:53:043
229026999210,19cyclictest3056-21PK-Backend20:08:533
229026999210,19cyclictest3056-21PK-Backend20:04:283
229026999210,19cyclictest3056-21PK-Backend19:45:283
229026999210,19cyclictest3056-21PK-Backend19:13:243
2290260992113,6cyclictest883-21dbus-daemon19:35:002
229026099210,7cyclictest2317449-21netplan20:24:322
229026099210,20cyclictest0-21swapper/220:53:562
2290255992119,1cyclictest0-21swapper/123:41:341
2290255992112,7cyclictest883-21dbus-daemon22:50:011
229025599210,20cyclictest3056-21PK-Backend22:22:531
229025599210,20cyclictest0-21swapper/121:15:341
229025499217,4cyclictest13-21ksoftirqd/022:20:270
2290254992119,1cyclictest0-21swapper/023:16:020
2290254992119,1cyclictest0-21swapper/023:02:430
2290254992119,1cyclictest0-21swapper/023:02:430
2290254992119,1cyclictest0-21swapper/022:36:410
2290254992119,1cyclictest0-21swapper/022:31:540
2290254992119,1cyclictest0-21swapper/021:57:410
2290254992119,1cyclictest0-21swapper/019:37:550
2290254992115,5cyclictest13-21ksoftirqd/000:00:000
2290254992113,6cyclictest2353112-21expr22:05:120
2290254992113,6cyclictest2324933-21awk20:45:110
2290254992113,6cyclictest1-21systemd23:40:010
229025499210,20cyclictest0-21swapper/023:42:380
229025499210,20cyclictest0-21swapper/023:20:280
229025499210,20cyclictest0-21swapper/019:56:210
229025499210,1cyclictest0-21swapper/022:57:020
139099210,15rtkit-daemon3056-21PK-Backend00:38:313
23467232200,3sleep10-21swapper/121:45:221
2290269992018,1cyclictest3056-21PK-Backend21:55:473
2290269992018,1cyclictest3056-21PK-Backend00:18:093
2290269992018,1cyclictest0-21swapper/300:30:063
2290269992018,1cyclictest0-21swapper/300:20:463
2290269992013,4cyclictest2359312-21switchtime22:20:213
229026999200,7cyclictest2403161-21df_inode00:25:143
229026999200,1cyclictest3056-21PK-Backend23:12:123
229026999200,19cyclictest0-21swapper/323:39:093
229026999200,19cyclictest0-21swapper/323:30:513
229026999200,19cyclictest0-21swapper/321:33:503
229026999200,19cyclictest0-21swapper/321:22:493
229026999200,19cyclictest0-21swapper/320:20:033
229026999200,19cyclictest0-21swapper/319:36:153
2290260992018,1cyclictest0-21swapper/222:51:172
2290260992018,1cyclictest0-21swapper/219:52:252
2290260992018,1cyclictest0-21swapper/200:25:062
2290260992012,6cyclictest2387236-21memory23:40:142
2290260992011,7cyclictest2361875-21apt22:30:112
229026099200,1cyclictest0-21swapper/222:55:512
229026099200,1cyclictest0-21swapper/220:35:302
229026099200,19cyclictest0-21swapper/223:13:002
229026099200,19cyclictest0-21swapper/222:21:372
229026099200,19cyclictest0-21swapper/220:06:452
229026099200,18cyclictest2298003-21sed19:30:002
2290255992018,1cyclictest0-21swapper/123:25:541
2290255992018,1cyclictest0-21swapper/121:08:181
2290255992018,1cyclictest0-21swapper/121:08:171
2290255992018,1cyclictest0-21swapper/120:41:481
2290255992018,1cyclictest0-21swapper/119:58:411
2290255992018,1cyclictest0-21swapper/100:25:281
2290255992016,2cyclictest2374003-21apt23:05:111
2290255992012,6cyclictest2370299-21pluginstate22:55:001
229025599200,1cyclictest3056-21PK-Backend22:18:201
229025599200,1cyclictest0-21swapper/119:25:151
229025599200,19cyclictest0-21swapper/123:55:541
229025599200,19cyclictest0-21swapper/120:32:481
229025599200,19cyclictest0-21swapper/120:07:331
229025599200,19cyclictest0-21swapper/119:50:231
229025599200,18cyclictest3056-21PK-Backend19:36:241
229025599200,18cyclictest3056-21PK-Backend00:24:551
229025599200,18cyclictest0-21swapper/100:39:131
229025499204,9cyclictest13-21ksoftirqd/022:45:020
229025499203,9cyclictest13-21ksoftirqd/000:31:390
2290254992018,1cyclictest0-21swapper/023:45:150
2290254992018,1cyclictest0-21swapper/023:34:410
2290254992018,1cyclictest0-21swapper/023:26:220
2290254992018,1cyclictest0-21swapper/022:53:440
2290254992018,1cyclictest0-21swapper/021:28:410
2290254992018,1cyclictest0-21swapper/021:10:220
2290254992018,1cyclictest0-21swapper/020:19:190
2290254992018,1cyclictest0-21swapper/000:11:380
2290254992018,1cyclictest0-21swapper/000:09:350
2290254992018,1cyclictest0-21swapper/000:01:040
2290254992015,4cyclictest13-21ksoftirqd/022:45:020
2290254992014,5cyclictest13-21ksoftirqd/020:15:000
229025499200,1cyclictest0-21swapper/023:11:010
229025499200,1cyclictest0-21swapper/019:11:390
229025499200,19cyclictest0-21swapper/022:29:310
229025499200,19cyclictest0-21swapper/021:32:400
229025499200,19cyclictest0-21swapper/021:05:490
229025499200,19cyclictest0-21swapper/021:05:490
229025499200,19cyclictest0-21swapper/020:34:190
229025499200,19cyclictest0-21swapper/000:15:100
229025499200,10cyclictest13-21ksoftirqd/021:45:190
139099200,14rtkit-daemon3056-21PK-Backend21:49:033
23901552190,4sleep33056-21PK-Backend23:50:113
23901552190,4sleep33056-21PK-Backend23:50:113
2290269991917,1cyclictest3056-21PK-Backend23:58:293
2290269991917,1cyclictest3056-21PK-Backend23:18:073
2290269991917,1cyclictest3056-21PK-Backend22:00:203
2290269991917,1cyclictest3056-21PK-Backend22:00:203
2290269991917,1cyclictest0-21swapper/322:32:443
229026999191,6cyclictest2355739-21/usr/sbin/munin22:10:233
229026999191,5cyclictest2295375-21ps19:20:193
229026999191,15cyclictest2357526-21python322:15:203
229026999190,1cyclictest3056-21PK-Backend20:53:143
229026999190,1cyclictest3056-21PK-Backend20:30:163
229026999190,1cyclictest3056-21PK-Backend20:26:053
2290260991917,1cyclictest3056-21PK-Backend23:00:232
2290260991917,1cyclictest3056-21PK-Backend23:00:232
2290260991917,1cyclictest0-21swapper/223:35:042
2290260991917,1cyclictest0-21swapper/222:37:042
2290260991917,1cyclictest0-21swapper/221:50:202
2290260991917,1cyclictest0-21swapper/219:46:432
2290260991917,1cyclictest0-21swapper/219:19:332
2290260991912,6cyclictest2360445-21expr22:25:142
229026099190,1cyclictest3056-21PK-Backend00:03:502
229026099190,1cyclictest0-21swapper/222:15:482
229026099190,1cyclictest0-21swapper/221:22:562
229026099190,1cyclictest0-21swapper/220:45:312
229026099190,1cyclictest0-21swapper/200:32:092
229026099190,18cyclictest3056-21PK-Backend21:47:162
229026099190,18cyclictest0-21swapper/220:19:282
2290255991917,1cyclictest3056-21PK-Backend23:37:081
2290255991917,1cyclictest0-21swapper/121:30:141
2290255991917,1cyclictest0-21swapper/121:00:401
2290255991917,1cyclictest0-21swapper/119:48:551
2290255991917,1cyclictest0-21swapper/100:16:431
2290255991917,1cyclictest0-21swapper/100:05:341
2290255991916,2cyclictest2390985-21smartctl23:50:201
2290255991916,2cyclictest2390985-21smartctl23:50:201
229025599190,1cyclictest3056-21PK-Backend22:32:341
229025599190,1cyclictest3056-21PK-Backend22:29:151
229025599190,1cyclictest0-21swapper/123:21:271
229025599190,1cyclictest0-21swapper/122:14:411
229025599190,1cyclictest0-21swapper/120:45:051
229025599190,18cyclictest3056-21PK-Backend21:22:161
229025599190,18cyclictest3056-21PK-Backend19:18:001
229025599190,18cyclictest0-21swapper/122:44:501
229025599190,18cyclictest0-21swapper/121:51:361
229025599190,18cyclictest0-21swapper/121:11:561
229025599190,18cyclictest0-21swapper/120:29:301
229025599190,18cyclictest0-21swapper/119:10:151
229025599190,18cyclictest0-21swapper/100:12:371
229025499196,5cyclictest13-21ksoftirqd/022:11:300
229025499196,4cyclictest13-21ksoftirqd/021:40:540
229025499195,3cyclictest13-21ksoftirqd/022:17:140
2290254991917,1cyclictest3056-21PK-Backend19:40:180
2290254991917,1cyclictest0-21swapper/023:51:240
2290254991917,1cyclictest0-21swapper/023:51:240
2290254991917,1cyclictest0-21swapper/023:05:080
2290254991917,1cyclictest0-21swapper/020:42:240
2290254991917,1cyclictest0-21swapper/020:26:490
2290254991913,5cyclictest13-21ksoftirqd/019:50:000
2290269991816,1cyclictest3056-21PK-Backend21:10:183
2290269991816,1cyclictest3056-21PK-Backend20:14:553
2290269991816,1cyclictest3056-21PK-Backend00:07:353
2290269991815,2cyclictest3056-21PK-Backend00:00:323
2290260991816,1cyclictest0-21swapper/221:39:522
2290260991816,1cyclictest0-21swapper/221:12:072
2290260991816,1cyclictest0-21swapper/219:44:202
2290260991815,2cyclictest2393326-21latency_hist00:00:002
2290260991811,5cyclictest2352408-21apt-get22:05:002
2290260991811,5cyclictest2352408-21apt-get22:05:002
2290260991810,6cyclictest2401921-21python300:20:192
229026099180,1cyclictest0-21swapper/221:03:492
2290255991814,2cyclictest2321475-21apt20:35:111
2290255991813,3cyclictest2312354-21grep20:10:131
2290255991812,5cyclictest2349916-21ntp_kernel_err21:55:191
2290255991811,5cyclictest2372918-21grep23:00:181
2290255991811,5cyclictest2372918-21grep23:00:181
2290255991811,5cyclictest2363586-21cat22:35:091
2290255991811,5cyclictest2309742-21sed20:00:211
229025499186,3cyclictest13-21ksoftirqd/021:36:280
229025499186,3cyclictest13-21ksoftirqd/019:20:280
229025499181,7cyclictest13-21ksoftirqd/000:20:030
2290254991816,1cyclictest0-21swapper/000:26:320
2290269991715,1cyclictest3056-21PK-Backend22:58:133
2290260991715,1cyclictest0-21swapper/221:59:542
2290260991715,1cyclictest0-21swapper/220:14:552
2290260991712,4cyclictest2307413-21if_enp2s019:55:172
2290260991711,5cyclictest2407848-21sed00:40:012
2290260991710,5cyclictest2396351-21grep00:05:232
229026099170,16cyclictest0-21swapper/221:40:472
2290255991715,1cyclictest3056-21PK-Backend19:21:371
2290255991715,1cyclictest3056-21PK-Backend00:31:361
2290255991715,1cyclictest0-21swapper/100:03:311
2290255991713,3cyclictest2353795-21sensors_temp22:05:211
2290255991712,4cyclictest2384163-21timerandwakeup23:30:201
2290255991710,5cyclictest2371176-21cat22:55:151
2290255991710,5cyclictest2339549-21ntpq21:25:201
2290254991715,1cyclictest0-21swapper/021:15:510
2290269991614,1cyclictest3056-21PK-Backend23:29:223
2290269991614,1cyclictest0-21swapper/323:00:363
2290269991614,1cyclictest0-21swapper/323:00:363
229026099169,5cyclictest2378431-21awk23:15:162
229026099169,5cyclictest2320226-21cut20:30:182
229026099169,5cyclictest2294856-21cat19:20:122
229026099168,6cyclictest2354890-21cat22:10:132
229026099166,9cyclictest2379878-21tr23:20:112
229026099164,2cyclictest141rcu_preempt00:18:122
2290260991614,1cyclictest0-21swapper/221:29:262
2290260991611,3cyclictest2375470-21latency_hist23:10:002
2290260991610,4cyclictest2390617-21sed23:50:172
2290260991610,4cyclictest2390617-21sed23:50:162
229026099160,1cyclictest0-21swapper/220:02:522
2290255991612,3cyclictest2375970-21/usr/sbin/munin23:10:161
229025599161,12cyclictest2328379-21/usr/sbin/munin20:55:231
2290269991513,1cyclictest3056-21PK-Backend20:35:573
229026099159,5cyclictest2365978-21cat22:40:162
229026099158,6cyclictest2353397-21cat22:05:152
229026099153,11cyclictest2319479-21munin-run20:30:002
2290260991513,1cyclictest3056-21PK-Backend21:30:192
2290260991513,1cyclictest0-21swapper/223:25:312
2290260991513,1cyclictest0-21swapper/222:47:392
2290260991513,1cyclictest0-21swapper/221:09:042
2290260991513,1cyclictest0-21swapper/221:09:032
2290260991511,3cyclictest34-21ksoftirqd/220:40:012
2290260991510,3cyclictest2389036-21tail23:45:162
2290255991511,2cyclictest2302990-21latency_hist19:45:021
2290269991412,1cyclictest3056-21PK-Backend22:44:073
2290260991411,2cyclictest2335320-21/usr/sbin/munin21:15:162
2290255991412,1cyclictest0-21swapper/121:43:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional