You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 05:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Apr 09, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
367269522420,20sleep013-21ksoftirqd/019:05:170
377763221330,3sleep23776896-21/usr/sbin/munin00:00:172
367934221130,4sleep13056-21PK-Backend19:20:201
367934221130,4sleep13056-21PK-Backend19:20:191
372001221020,3sleep20-21swapper/221:15:212
367226727747,24sleep20-21swapper/219:05:122
367309626640,19sleep30-21swapper/319:05:263
367335426243,14sleep10-21swapper/119:06:041
37122822530,2sleep30-21swapper/320:55:173
37499872450,3sleep13056-21PK-Backend22:40:211
37097312450,8sleep013-21ksoftirqd/020:50:000
367392199413,24cyclictest0-21swapper/222:33:002
367392199413,22cyclictest0-21swapper/221:51:312
367392199388,21cyclictest978-21snmpd19:31:292
367392199374,18cyclictest3722278-21taskset21:24:122
367392199373,20cyclictest0-21swapper/219:28:382
367392199370,19cyclictest3724051-21chrt21:29:402
367392199363,20cyclictest0-21swapper/220:06:042
367392199363,20cyclictest0-21swapper/220:04:002
367392199362,23cyclictest0-21swapper/219:18:562
367392199362,23cyclictest0-21swapper/219:10:312
367392199362,23cyclictest0-21swapper/200:11:172
367392199362,18cyclictest0-21swapper/222:40:382
367392199361,20cyclictest3755497-21taskset22:59:102
367392199361,20cyclictest3734246-21chrt21:57:032
367392199360,29cyclictest3785326-21chrt00:24:592
367392199360,23cyclictest37571582sleep223:03:152
367392199360,21cyclictest3748386-21chrt22:36:532
367392199360,21cyclictest3690454-21chrt19:53:162
367392199360,20cyclictest3725698-21chrt21:33:332
367392199360,19cyclictest3704462-21chrt20:34:552
367392199360,19cyclictest3686835-21chrt19:41:172
367392199352,26cyclictest3729260-21taskset21:44:482
367392199352,18cyclictest0-21swapper/223:30:342
367392199352,18cyclictest0-21swapper/220:40:082
367392199351,24cyclictest0-21swapper/221:46:042
367392199350,26cyclictest3700825-21chrt20:22:452
367392199350,22cyclictest3705853-21taskset20:35:282
367392199350,22cyclictest3690950-21cpuspeed19:55:112
367392199350,20cyclictest3786848-21chrt00:27:162
367392199350,20cyclictest3774609-21chrt23:51:272
367392199350,19cyclictest3727512-21chrt21:39:352
367392199342,18cyclictest0-21swapper/220:10:052
367392199341,21cyclictest3744950-21taskset22:27:182
367392199341,21cyclictest3688626-21taskset19:47:032
367392199341,19cyclictest0-21swapper/220:57:542
367392199340,26cyclictest3764140-21chrt23:23:522
367392199340,25cyclictest3762185-21chrt23:16:032
367392199340,22cyclictest3753798-21chrt22:54:372
367392199340,22cyclictest3709934-21chrt20:50:022
367392199340,21cyclictest3765850-21taskset23:28:332
367392199340,21cyclictest3713668-21chrt21:00:092
37385852330,3sleep0161rcuc/022:10:120
367392199330,24cyclictest0-21swapper/222:21:422
367392199330,21cyclictest3760519-21runrttasks23:11:532
367392199330,21cyclictest3751860-21chrt22:47:002
367392199330,18cyclictest978-21snmpd00:07:582
367392199330,18cyclictest978-21snmpd00:07:582
367392199330,18cyclictest3771677-21taskset23:45:072
367392199326,24cyclictest3789079-21munin-plugin-st00:35:002
367392199320,20cyclictest3056-21PK-Backend22:04:412
367392199320,19cyclictest3769087-21taskset23:35:352
367392199320,19cyclictest3699059-21chrt20:17:182
367392199320,19cyclictest3699059-21chrt20:17:182
367392199320,18cyclictest1095-21runrttasks23:41:532
367392199301,17cyclictest0-21swapper/219:23:212
367392199301,17cyclictest0-21swapper/219:23:212
367392199300,19cyclictest0-21swapper/200:36:302
367392199300,17cyclictest1095-21runrttasks20:49:422
367392199297,20cyclictest3717841-21basename21:10:142
367392199291,26cyclictest3740525-21expr22:15:122
3673921992912,15cyclictest3738645-21cut22:10:122
367392199272,21cyclictest3701290-21apt20:25:132
367392199270,16cyclictest1095-21runrttasks19:36:212
139099270,4rtkit-daemon1389-21rtkit-daemon00:28:553
367392199266,18cyclictest3757986-21basename23:05:152
367392199260,15cyclictest1095-21runrttasks00:17:332
139099260,4rtkit-daemon1389-21rtkit-daemon22:44:293
37114912250,4sleep30-21swapper/320:55:013
367392199247,15cyclictest0-21swapper/222:09:102
367392199246,17cyclictest0-21swapper/223:57:492
367392199241,21cyclictest3716574-21ls21:05:192
367392999230,21cyclictest3745894-21/usr/sbin/munin22:30:163
3673915992320,2cyclictest3056-21PK-Backend19:50:561
3673915992315,6cyclictest3724090-21munin-run21:30:001
367391599230,3cyclictest3738875-21diskstats22:10:141
367392999224,16cyclictest3745209-21unin-run22:30:003
3673929992219,2cyclictest3748385-21taskset22:36:523
3673929992216,5cyclictest0-21swapper/319:38:543
3673929992213,7cyclictest3771172-21apt-get23:45:003
367392999220,21cyclictest0-21swapper/300:21:023
3673915992220,1cyclictest0-21swapper/123:04:281
367391599220,20cyclictest3056-21PK-Backend19:16:231
367391599220,20cyclictest3056-21PK-Backend00:25:201
3673906992220,1cyclictest0-21swapper/023:39:330
3673906992220,1cyclictest0-21swapper/022:20:240
3673906992220,1cyclictest0-21swapper/019:59:170
3673906992215,5cyclictest3765987-21cron23:30:000
367390699220,21cyclictest3056-21PK-Backend23:49:330
139099220,3rtkit-daemon1389-21rtkit-daemon20:23:323
36993532210,3sleep30-21swapper/320:20:013
36993532210,3sleep30-21swapper/320:20:003
3673929992114,5cyclictest3725435-21users21:30:203
367392999211,17cyclictest3765373-21python323:25:203
367392999210,19cyclictest3056-21PK-Backend23:02:273
3673915992119,1cyclictest3056-21PK-Backend23:44:161
3673915992119,1cyclictest3056-21PK-Backend22:50:291
3673915992119,1cyclictest3056-21PK-Backend21:55:071
3673915992119,1cyclictest3056-21PK-Backend21:01:341
3673915992119,1cyclictest3056-21PK-Backend19:42:591
3673915992119,1cyclictest3056-21PK-Backend19:36:021
3673915992119,1cyclictest3056-21PK-Backend00:03:301
3673915992119,1cyclictest0-21swapper/123:38:291
3673915992119,1cyclictest0-21swapper/122:32:451
3673915992119,1cyclictest0-21swapper/122:07:511
3673915992119,1cyclictest0-21swapper/119:57:381
3673915992119,1cyclictest0-21swapper/119:29:531
3673915992118,2cyclictest3056-21PK-Backend20:33:561
367391599210,20cyclictest0-21swapper/100:20:201
367391599210,19cyclictest3056-21PK-Backend20:56:121
367391599210,19cyclictest3056-21PK-Backend20:19:421
367391599210,19cyclictest3056-21PK-Backend20:19:421
3673906992119,1cyclictest3056-21PK-Backend19:40:030
3673906992119,1cyclictest3056-21PK-Backend00:20:020
3673906992119,1cyclictest0-21swapper/023:58:460
3673906992119,1cyclictest0-21swapper/023:16:070
3673906992119,1cyclictest0-21swapper/022:36:340
3673906992119,1cyclictest0-21swapper/021:17:310
3673906992119,1cyclictest0-21swapper/021:02:510
3673906992119,1cyclictest0-21swapper/020:57:300
3673906992119,1cyclictest0-21swapper/020:52:090
3673906992119,1cyclictest0-21swapper/020:22:220
3673906992119,1cyclictest0-21swapper/020:17:500
3673906992119,1cyclictest0-21swapper/020:17:500
3673906992119,1cyclictest0-21swapper/020:11:060
3673906992119,1cyclictest0-21swapper/019:21:050
3673906992119,1cyclictest0-21swapper/019:21:040
3673906992119,1cyclictest0-21swapper/000:32:400
3673906992111,9cyclictest3694104-21unin-run20:05:000
367390699210,20cyclictest0-21swapper/021:51:170
367390699210,20cyclictest0-21swapper/021:38:190
367390699210,20cyclictest0-21swapper/019:27:000
367390699210,20cyclictest0-21swapper/000:02:100
367390699210,19cyclictest3056-21PK-Backend23:30:200
37688582200,3sleep30-21swapper/323:35:213
367392999203,15cyclictest918-21rs:main3
3673929992018,1cyclictest3056-21PK-Backend23:06:393
3673929992018,1cyclictest0-21swapper/323:16:403
3673929992018,1cyclictest0-21swapper/321:16:493
3673929992018,1cyclictest0-21swapper/320:39:583
3673929992018,1cyclictest0-21swapper/320:25:113
3673929992018,1cyclictest0-21swapper/300:18:253
3673929992018,1cyclictest0-21swapper/300:10:473
3673929992017,2cyclictest3056-21PK-Backend20:05:373
367392999201,4cyclictest3686535-21python319:40:193
3673929992012,6cyclictest3718315-21sed21:10:203
3673929992011,7cyclictest3750764-21/usr/sbin/munin22:45:173
367392999200,19cyclictest0-21swapper/322:53:143
367392999200,19cyclictest0-21swapper/321:46:363
367392999200,19cyclictest0-21swapper/319:45:163
367392999200,19cyclictest0-21swapper/319:21:303
367392999200,19cyclictest0-21swapper/319:21:303
367392999200,19cyclictest0-21swapper/300:31:163
367391599209,9cyclictest3776674-40install00:00:021
3673915992018,1cyclictest3056-21PK-Backend22:58:411
3673915992018,1cyclictest3056-21PK-Backend20:51:391
3673915992018,1cyclictest3056-21PK-Backend19:45:211
3673915992018,1cyclictest3056-21PK-Backend19:13:511
3673915992018,1cyclictest3056-21PK-Backend00:11:071
3673915992018,1cyclictest3056-21PK-Backend00:06:541
3673915992018,1cyclictest3056-21PK-Backend00:06:541
3673915992018,1cyclictest0-21swapper/120:05:431
3673915992017,2cyclictest3056-21PK-Backend21:45:331
3673915992012,6cyclictest3707962-21munin-run20:45:001
367391599200,19cyclictest3056-21PK-Backend23:15:171
367391599200,19cyclictest3056-21PK-Backend21:12:141
367391599200,19cyclictest3056-21PK-Backend20:25:031
367391599200,19cyclictest3056-21PK-Backend20:10:511
367391599200,19cyclictest3056-21PK-Backend00:30:201
3673906992018,1cyclictest0-21swapper/023:53:460
3673906992018,1cyclictest0-21swapper/021:10:360
3673906992018,1cyclictest0-21swapper/021:08:590
3673906992018,1cyclictest0-21swapper/020:41:430
3673906992018,1cyclictest0-21swapper/019:16:580
3673906992018,1cyclictest0-21swapper/000:13:120
3673906992017,2cyclictest0-21swapper/022:05:040
3673906992017,2cyclictest0-21swapper/020:05:110
3673906992013,5cyclictest3721313-21basename21:20:160
367390699200,19cyclictest0-21swapper/023:44:330
367390699200,19cyclictest0-21swapper/019:31:320
367390699200,19cyclictest0-21swapper/000:05:350
367390699200,19cyclictest0-21swapper/000:05:350
367392999192,15cyclictest3726238-21cpuspeed_turbos21:35:103
3673929991917,1cyclictest3056-21PK-Backend23:57:093
3673929991917,1cyclictest3056-21PK-Backend20:02:403
3673929991917,1cyclictest0-21swapper/323:52:563
3673929991917,1cyclictest0-21swapper/323:12:273
3673929991917,1cyclictest0-21swapper/322:18:543
3673929991917,1cyclictest0-21swapper/321:25:203
3673929991917,1cyclictest0-21swapper/321:00:333
3673929991917,1cyclictest0-21swapper/320:40:453
3673929991917,1cyclictest0-21swapper/320:10:443
3673929991917,1cyclictest0-21swapper/300:38:053
3673929991917,1cyclictest0-21swapper/300:08:113
3673929991917,1cyclictest0-21swapper/300:08:113
3673929991917,1cyclictest0-21swapper/300:00:333
3673929991914,4cyclictest978-21snmpd20:49:473
3673929991914,3cyclictest3766622-21diskstats23:30:133
3673929991914,3cyclictest3762781-21apt23:20:113
3673929991914,3cyclictest3741979-21apt22:20:103
3673929991912,5cyclictest3733748-21/usr/sbin/munin21:55:183
367392999190,1cyclictest3056-21PK-Backend19:56:123
367392999190,1cyclictest3056-21PK-Backend19:27:263
367392999190,1cyclictest0-21swapper/322:03:333
367392999190,1cyclictest0-21swapper/319:50:033
367392999190,1cyclictest0-21swapper/319:16:023
367392999190,18cyclictest0-21swapper/321:23:453
367392999190,18cyclictest0-21swapper/319:10:283
367392999190,17cyclictest3716547-21python321:05:183
3673915991917,1cyclictest3056-21PK-Backend23:34:161
3673915991917,1cyclictest3056-21PK-Backend22:00:421
3673915991917,1cyclictest3056-21PK-Backend21:36:481
3673915991917,1cyclictest0-21swapper/119:31:291
3673915991916,2cyclictest3056-21PK-Backend23:29:161
367391599190,1cyclictest3056-21PK-Backend20:47:061
367391599190,1cyclictest3056-21PK-Backend00:16:551
3673906991917,1cyclictest0-21swapper/023:21:550
3673906991917,1cyclictest0-21swapper/022:54:310
3673906991917,1cyclictest0-21swapper/019:10:500
3673906991917,1cyclictest0-21swapper/000:25:020
3673906991912,5cyclictest3730627-21uptime21:45:220
367390699190,1cyclictest0-21swapper/020:35:470
367390699190,1cyclictest0-21swapper/020:32:490
367390699190,1cyclictest0-21swapper/020:26:550
367390699190,1cyclictest0-21swapper/019:36:520
367390699190,1cyclictest0-21swapper/000:36:040
367390699190,18cyclictest0-21swapper/023:10:190
367390699190,18cyclictest0-21swapper/000:17:250
3673929991816,1cyclictest3056-21PK-Backend19:32:453
3673929991811,6cyclictest3703925-21sed20:30:193
3673915991816,1cyclictest3056-21PK-Backend21:07:411
3673915991816,1cyclictest3056-21PK-Backend20:21:171
3673915991816,1cyclictest3056-21PK-Backend00:38:451
3673915991816,1cyclictest0-21swapper/123:49:171
3673915991816,1cyclictest0-21swapper/120:00:361
3673915991815,2cyclictest3056-21PK-Backend21:50:201
3673915991815,2cyclictest3056-21PK-Backend21:31:281
3673915991811,5cyclictest3747283-21cat22:35:121
3673915991811,5cyclictest3721824-21cat21:20:211
367390699184,6cyclictest1095-21runrttasks23:01:050
3673906991817,1cyclictest0-21swapper/022:30:110
3673906991816,1cyclictest0-21swapper/022:59:310
3673906991816,1cyclictest0-21swapper/022:49:310
3673906991816,1cyclictest0-21swapper/021:32:110
3673906991816,1cyclictest0-21swapper/021:26:020
3673906991811,5cyclictest3729488-21grep21:45:020
367392999179,6cyclictest3731540-21expr21:50:123
3673929991713,3cyclictest42-21ksoftirqd/322:14:393
3673929991712,4cyclictest42-21ksoftirqd/321:45:023
367392999170,3cyclictest3754486-21df22:55:213
3673915991715,1cyclictest3056-21PK-Backend23:09:291
3673915991715,1cyclictest3056-21PK-Backend22:25:361
3673915991715,1cyclictest0-21swapper/121:19:101
3673915991714,2cyclictest3056-21PK-Backend20:39:021
3673906991712,3cyclictest3736249-21idleruntime-cro22:04:590
3673906991711,5cyclictest3758539-21sed23:05:210
3673906991710,5cyclictest3749794-21grep22:40:190
3673915991614,1cyclictest3056-21PK-Backend23:54:171
3673906991610,5cyclictest3690676-21sed19:55:020
367390699160,14cyclictest436-20systemd-journal22:30:000
3673929991511,2cyclictest42-21ksoftirqd/322:05:143
3673915991513,1cyclictest3056-21PK-Backend23:23:281
3673915991512,2cyclictest3056-21PK-Backend21:43:101
3673915991510,3cyclictest3760772-21sh23:15:001
367390699159,4cyclictest2156-21gnome-shell19:47:100
3673906991510,3cyclictest3733466-21if_enp2s021:55:150
3673915991410,3cyclictest3056-21PK-Backend22:21:361
367390699142,11cyclictest3740374-21tr22:15:110
367391599119,1cyclictest3056-21PK-Backend22:49:411
367391599118,2cyclictest37414072chrt22:16:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional