You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-31 - 12:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Jul 31, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
61344828464,14sleep00-21swapper/019:07:430
61264627246,17sleep10-21swapper/119:05:191
139099690,59rtkit-daemon0-21swapper/319:05:453
61218126444,13sleep20-21swapper/219:05:122
6361342550,5sleep3636136-21latency_hist20:10:013
61379499439,22cyclictest650156-21seq20:49:402
61379499439,22cyclictest650156-21seq20:49:402
6434752390,1sleep10-21swapper/120:30:121
613794993515,18cyclictest630328-21perf19:55:002
6767412340,4sleep33056-21PK-Backend22:05:023
61379499329,21cyclictest640422-21ls20:20:182
6158482300,6sleep20-21swapper/219:15:022
139099300,4rtkit-daemon1389-21rtkit-daemon21:22:193
61379499299,5cyclictest647503-21open_inodes20:40:192
61379499299,5cyclictest642592-21sed20:25:212
613794992911,17cyclictest650442-21/usr/sbin/munin20:50:202
613794992911,17cyclictest34-21ksoftirqd/220:05:222
613794992910,18cyclictest1095-21runrttasks22:09:272
139099290,4rtkit-daemon1389-21rtkit-daemon23:17:043
61379499289,17cyclictest692860-21cstates22:50:122
61379499282,4cyclictest0-21swapper/220:15:112
61379499281,4cyclictest0-21swapper/200:10:152
61379499280,6cyclictest690400-21latency_hist22:45:002
61379499279,16cyclictest1095-21runrttasks22:10:432
61379499278,17cyclictest718041-21wc00:00:222
61379499278,17cyclictest695065-21tail22:55:192
61379499277,18cyclictest0-21swapper/222:28:582
61379499272,5cyclictest701357-21apt23:15:132
61379499271,9cyclictest619047-21python319:20:212
61379499271,4cyclictest0-21swapper/219:55:112
61379499270,20cyclictest0-21swapper/220:56:292
139099270,4rtkit-daemon1389-21rtkit-daemon22:50:053
139099270,4rtkit-daemon1389-21rtkit-daemon20:04:282
139099270,4rtkit-daemon1389-21rtkit-daemon19:53:563
61379499269,15cyclictest712130-21sshd23:45:122
61379499268,15cyclictest715342-21latency_hist23:55:012
61379499267,4cyclictest719834-21sensors00:05:222
61379499267,4cyclictest659021-21latency_hist21:15:012
61379499266,6cyclictest691401-21cat22:45:172
61379499266,3cyclictest663256-21/usr/sbin/munin21:25:192
61379499265,5cyclictest436-20systemd-journal19:35:002
61379499261,8cyclictest82332-21update-manager21:41:012
61379499261,5cyclictest0-21swapper/221:20:122
613792992618,6cyclictest697593-21cron23:05:001
139099260,4rtkit-daemon1389-21rtkit-daemon22:37:053
61379499257,3cyclictest0-21swapper/221:00:222
61379499256,4cyclictest660121-21sed21:15:232
61379499255,7cyclictest82332-21update-manager23:06:052
61379499255,5cyclictest730465-21processes00:35:222
61379499255,4cyclictest722998-21expr00:15:132
61379499254,4cyclictest628143-21timerwakeupswit19:45:222
61379499252,7cyclictest0-21swapper/223:35:112
61379499252,5cyclictest703089-21apt23:20:122
61379499251,9cyclictest715890-21hddtemp_smartct23:55:162
613794992519,5cyclictest0-21swapper/223:44:122
613794992519,5cyclictest0-21swapper/223:26:422
61379499251,8cyclictest645653-21/usr/sbin/munin20:35:172
61379499251,7cyclictest725929-21awk00:25:012
61379499251,7cyclictest683487-21munin-run22:25:012
61379499251,7cyclictest625351-21/usr/sbin/munin19:40:102
61379499251,6cyclictest625227-21latency_hist19:40:012
61379499251,5cyclictest0-21swapper/221:05:102
139099250,4rtkit-daemon1389-21rtkit-daemon22:44:433
61379499248,2cyclictest688661-21cron22:40:002
61379499247,4cyclictest28650irq/123-ahci[0023:35:002
61379499247,3cyclictest34-21ksoftirqd/222:18:032
61379499244,4cyclictest0-21swapper/222:35:002
61379499241,8cyclictest728302-21/usr/sbin/munin00:30:162
61379499241,8cyclictest728302-21/usr/sbin/munin00:30:152
61379499241,8cyclictest701115-21munin-run23:15:002
61379499241,8cyclictest696513-21diskstats23:00:182
61379499241,6cyclictest637057-21ntp_offset20:10:182
613794992416,6cyclictest642993-21cron20:34:562
139099240,3rtkit-daemon1389-21rtkit-daemon20:23:053
6422772230,4sleep13056-21PK-Backend20:25:191
61379899230,21cyclictest3056-21PK-Backend23:51:423
61379499237,3cyclictest673047-21/usr/sbin/munin21:55:222
61379499237,3cyclictest664919-21load21:30:182
61379499235,6cyclictest672044-21open_files21:50:202
61379499235,4cyclictest666165-21cpuspeed21:35:132
61379499231,7cyclictest619649-21apt-get19:25:032
613794992317,5cyclictest0-21swapper/221:45:102
61379499230,8cyclictest726541-21/usr/sbin/munin00:25:152
61379499230,8cyclictest615460-21munin-run19:15:012
613788992320,2cyclictest0-21swapper/023:07:310
613788992320,2cyclictest0-21swapper/021:37:050
61378899230,21cyclictest643667-21hddtemp_smartct20:30:150
6851862220,4sleep33056-21PK-Backend22:29:563
61379899220,10cyclictest721188-21/usr/sbin/munin00:10:203
61379499221,6cyclictest675660-21grep22:00:222
61379299228,12cyclictest626886-21unin-run19:45:001
613792992219,2cyclictest0-21swapper/121:54:011
61378899225,3cyclictest630386-21latency_hist19:55:000
613788992220,1cyclictest0-21swapper/022:18:370
613788992220,1cyclictest0-21swapper/019:13:010
613788992220,1cyclictest0-21swapper/000:26:590
613788992220,1cyclictest0-21swapper/000:14:360
613788992219,2cyclictest686148-21strings22:30:190
613788992219,2cyclictest626238-21python319:40:190
613788992219,2cyclictest3056-21PK-Backend23:35:560
61378899220,21cyclictest0-21swapper/022:11:410
61378899220,21cyclictest0-21swapper/020:48:470
61378899220,21cyclictest0-21swapper/020:48:470
613798992119,1cyclictest3056-21PK-Backend21:40:483
613798992118,2cyclictest3056-21PK-Backend22:17:533
613798992118,2cyclictest3056-21PK-Backend20:59:183
613798992113,6cyclictest657527-21grep21:10:123
61379899210,20cyclictest636737-21ls20:10:153
61379899210,20cyclictest0-21swapper/323:08:023
61379899210,19cyclictest3056-21PK-Backend23:11:063
613792992115,5cyclictest667529-21cron21:40:001
61379299210,20cyclictest3056-21PK-Backend00:31:521
61379299210,20cyclictest3056-21PK-Backend00:31:521
61379299210,20cyclictest0-21swapper/122:36:331
613788992119,1cyclictest0-21swapper/023:00:480
613788992119,1cyclictest0-21swapper/022:56:350
613788992119,1cyclictest0-21swapper/022:29:450
613788992119,1cyclictest0-21swapper/021:55:440
613788992119,1cyclictest0-21swapper/021:28:140
61378899211,7cyclictest617893-21wc19:20:010
613788992116,3cyclictest13-21ksoftirqd/022:00:120
613788992113,6cyclictest652945-21ntp_states20:55:210
613788992112,7cyclictest13-21ksoftirqd/021:40:120
61378899210,20cyclictest0-21swapper/023:42:320
61378899210,20cyclictest0-21swapper/021:08:120
61378899210,20cyclictest0-21swapper/000:08:000
7263022200,3sleep13056-21PK-Backend00:25:141
6227812200,4sleep30-21swapper/319:30:233
61379899206,12cyclictest42-21ksoftirqd/322:30:213
613798992018,1cyclictest3056-21PK-Backend20:27:413
613798992018,1cyclictest0-21swapper/322:55:113
613798992018,1cyclictest0-21swapper/320:47:153
613798992018,1cyclictest0-21swapper/320:47:143
613798992018,1cyclictest0-21swapper/320:41:413
613798992018,1cyclictest0-21swapper/319:55:103
613798992018,1cyclictest0-21swapper/319:24:553
613798992018,1cyclictest0-21swapper/300:36:503
613798992017,2cyclictest3056-21PK-Backend21:52:113
613798992017,2cyclictest3056-21PK-Backend21:46:573
613798992017,2cyclictest3056-21PK-Backend20:31:543
613798992017,2cyclictest3056-21PK-Backend19:25:433
613798992017,2cyclictest0-21swapper/320:54:383
613798992017,1cyclictest0-21swapper/323:35:333
613798992014,5cyclictest42-21ksoftirqd/322:15:013
61379899200,1cyclictest3056-21PK-Backend23:46:143
61379899200,1cyclictest3056-21PK-Backend21:38:453
61379899200,1cyclictest0-21swapper/321:19:463
61379899200,1cyclictest0-21swapper/319:15:213
61379899200,19cyclictest3056-21PK-Backend23:28:423
61379899200,19cyclictest3056-21PK-Backend00:03:103
61379899200,19cyclictest0-21swapper/323:55:463
61379899200,19cyclictest0-21swapper/323:20:533
61379899200,19cyclictest0-21swapper/320:39:103
61379899200,19cyclictest0-21swapper/320:16:253
61379899200,19cyclictest0-21swapper/319:13:403
61379899200,19cyclictest0-21swapper/300:08:383
61379899200,18cyclictest633026-21/usr/sbin/munin20:00:163
61379899200,18cyclictest3056-21PK-Backend19:35:433
613792992018,1cyclictest3056-21PK-Backend23:19:041
613792992018,1cyclictest0-21swapper/123:26:221
613792992018,1cyclictest0-21swapper/121:40:021
613792992017,2cyclictest3056-21PK-Backend23:38:111
613792992017,2cyclictest3056-21PK-Backend22:53:231
613792992012,6cyclictest617184-21cat19:15:231
61379299200,19cyclictest3056-21PK-Backend22:49:231
61379299200,19cyclictest3056-21PK-Backend21:15:151
61379299200,19cyclictest3056-21PK-Backend21:03:251
61379299200,19cyclictest3056-21PK-Backend19:59:101
61379299200,19cyclictest3056-21PK-Backend19:48:221
61379299200,19cyclictest3056-21PK-Backend00:35:101
61379299200,19cyclictest3056-21PK-Backend00:07:111
61379299200,19cyclictest0-21swapper/123:54:191
61379299200,19cyclictest0-21swapper/123:42:511
61379299200,19cyclictest0-21swapper/119:36:111
61379299200,19cyclictest0-21swapper/119:25:241
61379299200,18cyclictest3056-21PK-Backend21:10:561
61378899203,15cyclictest683480-21grep22:25:010
61378899202,7cyclictest713722-21apt-get23:50:040
61378899202,13cyclictest706569-21apt23:30:120
613788992018,1cyclictest3056-21PK-Backend19:22:280
613788992018,1cyclictest0-21swapper/021:49:230
613788992018,1cyclictest0-21swapper/021:24:430
613788992018,1cyclictest0-21swapper/020:52:110
613788992018,1cyclictest0-21swapper/019:55:130
613788992018,1cyclictest0-21swapper/019:38:100
613788992018,1cyclictest0-21swapper/000:30:180
613788992018,1cyclictest0-21swapper/000:30:180
613788992018,1cyclictest0-21swapper/000:00:370
613788992014,4cyclictest731127-21cron00:40:000
613788992014,4cyclictest704564-21latency_hist23:25:000
613788992013,5cyclictest665073-21ntpq21:30:180
613788992011,7cyclictest13-21ksoftirqd/019:30:130
613788992011,6cyclictest13-21ksoftirqd/000:15:210
61378899200,19cyclictest0-21swapper/022:05:530
61378899200,19cyclictest0-21swapper/020:23:460
61378899200,19cyclictest0-21swapper/019:47:220
61378899200,19cyclictest0-21swapper/000:21:450
613798991917,1cyclictest3056-21PK-Backend22:01:433
613798991917,1cyclictest3056-21PK-Backend21:57:593
613798991917,1cyclictest0-21swapper/321:02:293
613798991917,1cyclictest0-21swapper/300:31:373
613798991917,1cyclictest0-21swapper/300:31:363
613798991916,2cyclictest3056-21PK-Backend23:00:443
613798991916,2cyclictest3056-21PK-Backend21:31:223
613798991911,6cyclictest436-20systemd-journal00:25:013
61379899190,7cyclictest710902-21smartctl23:40:233
61379899190,1cyclictest3056-21PK-Backend23:31:263
61379899190,1cyclictest3056-21PK-Backend19:42:123
61379899190,1cyclictest0-21swapper/322:49:503
61379899190,1cyclictest0-21swapper/300:28:523
61379899190,18cyclictest0-21swapper/321:09:253
613792991917,1cyclictest3056-21PK-Backend22:07:541
613792991917,1cyclictest3056-21PK-Backend19:13:001
613792991917,1cyclictest0-21swapper/123:05:061
613792991917,1cyclictest0-21swapper/122:55:531
613792991917,1cyclictest0-21swapper/122:15:571
613792991913,5cyclictest684492-21ntpq22:25:201
613792991912,5cyclictest682112-21sshd22:20:111
613792991910,7cyclictest685840-21df22:30:131
61379299190,1cyclictest3056-21PK-Backend20:39:121
61379299190,1cyclictest3056-21PK-Backend19:52:411
61379299190,1cyclictest0-21swapper/123:48:521
61379299190,1cyclictest0-21swapper/120:55:341
61379299190,18cyclictest3056-21PK-Backend22:14:501
61379299190,18cyclictest0-21swapper/123:30:481
61379299190,18cyclictest0-21swapper/123:23:181
61379299190,18cyclictest0-21swapper/123:11:481
61379299190,18cyclictest0-21swapper/122:03:351
613788991917,1cyclictest0-21swapper/023:57:180
613788991917,1cyclictest0-21swapper/020:13:520
613788991917,1cyclictest0-21swapper/019:29:310
613788991913,5cyclictest883-21dbus-daemon21:10:590
613788991912,5cyclictest436-20systemd-journal20:00:110
613788991911,6cyclictest13-21ksoftirqd/021:50:120
613788991911,2cyclictest13-21ksoftirqd/020:10:010
613788991910,7cyclictest13-21ksoftirqd/023:45:190
61378899190,1cyclictest3056-21PK-Backend23:15:210
61378899190,1cyclictest0-21swapper/022:41:280
61378899190,1cyclictest0-21swapper/021:03:000
61378899190,1cyclictest0-21swapper/020:19:470
613798991816,1cyclictest0-21swapper/319:47:523
613798991812,4cyclictest724009-21pool-udisksd00:18:033
613792991816,1cyclictest3056-21PK-Backend21:07:101
613792991816,1cyclictest3056-21PK-Backend20:53:321
613792991816,1cyclictest3056-21PK-Backend20:44:461
613792991816,1cyclictest3056-21PK-Backend20:11:001
613792991816,1cyclictest3056-21PK-Backend00:11:171
613792991816,1cyclictest0-21swapper/121:48:141
613792991816,1cyclictest0-21swapper/119:31:531
61379299180,1cyclictest3056-21PK-Backend20:04:171
61378899189,7cyclictest687675-21cat22:35:150
613788991816,1cyclictest0-21swapper/021:15:440
613788991811,6cyclictest13-21ksoftirqd/023:10:070
613788991811,6cyclictest13-21ksoftirqd/020:40:000
613788991810,2cyclictest13-21ksoftirqd/020:44:590
613798991715,1cyclictest0-21swapper/321:26:563
61379299179,6cyclictest436-20systemd-journal22:00:001
613792991715,1cyclictest3056-21PK-Backend21:20:421
613792991715,1cyclictest3056-21PK-Backend20:18:581
613792991715,1cyclictest0-21swapper/100:03:061
613792991711,5cyclictest664308-21seq21:30:021
61378899179,6cyclictest705263-21cat23:25:160
613788991711,5cyclictest691551-21wc22:45:190
613798991610,5cyclictest683622-21sed22:25:013
613792991613,2cyclictest3056-21PK-Backend20:08:021
613792991613,2cyclictest3056-21PK-Backend00:20:021
61379299160,2cyclictest3056-21PK-Backend19:25:001
613788991610,5cyclictest642852-21seq20:28:140
61378899160,1cyclictest0-21swapper/022:52:570
61379299157,6cyclictest918-21rs:main1
61379299151,2cyclictest141rcu_preempt00:16:071
613792991512,2cyclictest3056-21PK-Backend23:58:261
613792991510,4cyclictest649699-21sed20:45:211
613792991510,4cyclictest649699-21sed20:45:211
61379299139,3cyclictest3056-21PK-Backend20:22:541
61379299119,1cyclictest3056-21PK-Backend22:40:321
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional