You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-22 - 13:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Jul 22, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
390986622140,5sleep13056-21PK-Backend19:05:121
394210521330,2sleep10-21swapper/120:35:111
391634421090,4sleep33056-21PK-Backend19:20:183
139099860,48rtkit-daemon0-21swapper/319:08:023
391009527245,19sleep20-21swapper/219:05:132
391129426444,13sleep00-21swapper/019:09:500
39717502560,1chrt3970658-21/usr/sbin/munin21:55:231
39627442510,4sleep33056-21PK-Backend21:30:223
39688352490,1sleep00-21swapper/021:50:000
40153842480,1sleep10-21swapper/100:00:141
39343502440,1chrt3934351-21sshd20:11:171
39193752440,4sleep33056-21PK-Backend19:30:133
139099390,3rtkit-daemon0-21swapper/221:36:542
139099380,4rtkit-daemon0-21swapper/219:25:362
139099380,3rtkit-daemon0-21swapper/220:19:292
139099370,3rtkit-daemon0-21swapper/223:35:522
139099370,3rtkit-daemon0-21swapper/221:56:432
139099370,3rtkit-daemon0-21swapper/221:27:322
139099370,3rtkit-daemon0-21swapper/221:24:522
139099370,3rtkit-daemon0-21swapper/220:32:152
139099360,3rtkit-daemon0-21swapper/221:34:312
139099350,3rtkit-daemon0-21swapper/221:45:082
139099350,3rtkit-daemon0-21swapper/219:38:032
139099350,3rtkit-daemon0-21swapper/219:30:322
391144599248,14cyclictest3963223-21sh21:35:001
3911445992415,7cyclictest4027511-21unin-run00:35:011
391144199241,2cyclictest0-21swapper/023:37:070
40262572230,5sleep00-21swapper/000:30:120
3911449992317,4cyclictest3952319-21basename21:05:002
3911445992315,6cyclictest3913040-21sshd19:14:171
391144199233,18cyclictest1895009-21snapd23:11:130
3911441992317,5cyclictest4022366-21gpgv00:20:010
3911441992314,7cyclictest3949689-21memory20:55:180
3911441992314,7cyclictest3917705-21df_inode19:25:140
39156772220,3sleep00-21swapper/019:20:110
3911453992220,1cyclictest3056-21PK-Backend21:16:353
3911453992218,2cyclictest3056-21PK-Backend22:37:393
391145399220,20cyclictest3056-21PK-Backend22:47:193
3911449992220,1cyclictest0-21swapper/223:47:272
3911449992220,1cyclictest0-21swapper/223:47:272
3911449992220,1cyclictest0-21swapper/220:11:172
3911445992220,1cyclictest0-21swapper/119:33:001
3911445992219,2cyclictest3056-21PK-Backend19:48:211
3911445992219,2cyclictest3056-21PK-Backend19:48:211
391144599220,21cyclictest0-21swapper/119:56:331
391144599220,20cyclictest436-20systemd-journal19:40:171
391144199223,17cyclictest3977609-21munin-run22:15:000
3911441992214,6cyclictest436-20systemd-journal20:30:010
3911441992214,6cyclictest3947376-21apt20:50:120
139099220,14rtkit-daemon0-21swapper/319:45:373
139099220,14rtkit-daemon0-21swapper/319:45:373
39905242210,2sleep10-21swapper/122:50:121
391145399212,1cyclictest3056-21PK-Backend00:35:483
3911453992119,1cyclictest3056-21PK-Backend23:32:553
3911453992119,1cyclictest3056-21PK-Backend21:43:043
3911453992119,1cyclictest0-21swapper/323:41:073
3911453992119,1cyclictest0-21swapper/323:26:203
3911453992118,2cyclictest3056-21PK-Backend23:16:333
3911453992118,2cyclictest3056-21PK-Backend22:26:243
3911453992118,2cyclictest3056-21PK-Backend20:05:173
3911453992118,2cyclictest3056-21PK-Backend00:17:443
391145399210,20cyclictest3056-21PK-Backend22:43:343
391145399210,20cyclictest3056-21PK-Backend21:25:213
391145399210,20cyclictest3056-21PK-Backend20:18:553
391145399210,20cyclictest3056-21PK-Backend00:23:453
391145399210,19cyclictest3056-21PK-Backend20:42:283
391145399210,19cyclictest3056-21PK-Backend20:42:283
391145399210,19cyclictest3056-21PK-Backend19:27:243
3911449992119,1cyclictest0-21swapper/223:52:272
3911449992119,1cyclictest0-21swapper/223:30:512
3911449992119,1cyclictest0-21swapper/223:20:032
3911449992119,1cyclictest0-21swapper/222:55:162
3911449992119,1cyclictest0-21swapper/222:37:112
3911449992119,1cyclictest0-21swapper/222:24:062
3911449992119,1cyclictest0-21swapper/220:41:112
3911449992119,1cyclictest0-21swapper/220:41:112
3911449992119,1cyclictest0-21swapper/220:28:002
3911449992119,1cyclictest0-21swapper/219:52:242
3911449992119,1cyclictest0-21swapper/219:45:022
3911449992119,1cyclictest0-21swapper/219:45:012
3911449992119,1cyclictest0-21swapper/219:10:472
3911449992119,1cyclictest0-21swapper/200:13:572
3911449992119,1cyclictest0-21swapper/200:01:402
3911449992113,6cyclictest3995445-21cron23:05:002
391144999210,20cyclictest0-21swapper/223:59:512
391144999210,20cyclictest0-21swapper/221:54:392
391144999210,20cyclictest0-21swapper/221:40:402
391144999210,20cyclictest0-21swapper/220:09:282
391144999210,20cyclictest0-21swapper/220:00:012
391144999210,20cyclictest0-21swapper/219:24:062
391144999210,20cyclictest0-21swapper/200:39:462
3911445992119,1cyclictest3056-21PK-Backend22:32:411
3911445992119,1cyclictest0-21swapper/123:54:331
3911445992119,1cyclictest0-21swapper/123:31:481
3911445992119,1cyclictest0-21swapper/123:16:141
3911445992119,1cyclictest0-21swapper/122:19:021
3911445992119,1cyclictest0-21swapper/121:50:161
3911445992119,1cyclictest0-21swapper/121:45:091
3911445992119,1cyclictest0-21swapper/121:40:491
3911445992119,1cyclictest0-21swapper/121:27:451
3911445992119,1cyclictest0-21swapper/121:03:111
3911445992119,1cyclictest0-21swapper/120:58:381
3911445992119,1cyclictest0-21swapper/120:16:541
3911445992119,1cyclictest0-21swapper/120:06:471
3911445992119,1cyclictest0-21swapper/100:24:411
3911445992113,6cyclictest4019408-21python300:10:211
391144599210,20cyclictest3056-21PK-Backend23:43:591
391144599210,20cyclictest3056-21PK-Backend23:25:131
391144599210,20cyclictest0-21swapper/122:14:301
391144599210,20cyclictest0-21swapper/120:47:221
391144199212,17cyclictest3913117-21turbostat19:15:000
3911441992119,1cyclictest0-21swapper/019:46:440
3911441992119,1cyclictest0-21swapper/019:46:440
3911441992114,5cyclictest883-21dbus-daemon22:38:030
3911441992114,5cyclictest436-20systemd-journal00:19:280
3911441992113,7cyclictest3915337-21cron19:20:000
3911441992113,7cyclictest2019-21gmain22:05:000
3911441992113,6cyclictest4012813-21sshd23:52:110
391144199210,20cyclictest0-21swapper/022:31:580
391144199210,20cyclictest0-21swapper/019:34:340
139099210,14rtkit-daemon0-21swapper/300:08:433
139099210,14rtkit-daemon0-21swapper/200:19:422
39532492200,2sleep30-21swapper/321:05:203
3911453992018,1cyclictest3056-21PK-Backend22:51:043
3911453992018,1cyclictest3056-21PK-Backend22:15:563
3911453992018,1cyclictest3056-21PK-Backend21:56:023
3911453992018,1cyclictest3056-21PK-Backend21:38:113
3911453992018,1cyclictest3056-21PK-Backend20:55:393
3911453992018,1cyclictest0-21swapper/323:49:183
3911453992018,1cyclictest0-21swapper/323:49:183
3911453992018,1cyclictest0-21swapper/322:30:093
3911453992018,1cyclictest0-21swapper/321:22:023
3911453992014,4cyclictest3926129-21cpuspeed_turbos19:50:103
391145399200,1cyclictest3056-21PK-Backend21:04:313
391145399200,1cyclictest3056-21PK-Backend20:38:083
391145399200,19cyclictest3056-21PK-Backend22:09:143
391145399200,19cyclictest3056-21PK-Backend20:48:573
391145399200,19cyclictest3056-21PK-Backend00:32:243
391145399200,19cyclictest3056-21PK-Backend00:29:473
391145399200,19cyclictest3056-21PK-Backend00:11:423
391145399200,19cyclictest0-21swapper/319:10:353
391145399200,18cyclictest3056-21PK-Backend20:31:063
3911449992018,1cyclictest0-21swapper/222:18:452
3911449992018,1cyclictest0-21swapper/219:57:242
3911449992014,4cyclictest3937000-21expr20:20:132
3911449992013,5cyclictest436-20systemd-journal22:05:002
391144999200,19cyclictest0-21swapper/223:41:392
391144999200,19cyclictest0-21swapper/222:29:272
391144999200,19cyclictest0-21swapper/221:14:172
391144999200,19cyclictest0-21swapper/220:51:392
3911445992018,1cyclictest3056-21PK-Backend00:08:121
3911445992018,1cyclictest0-21swapper/122:37:131
3911445992018,1cyclictest0-21swapper/122:23:351
3911445992018,1cyclictest0-21swapper/121:13:331
3911445992018,1cyclictest0-21swapper/121:09:261
3911445992018,1cyclictest0-21swapper/119:29:011
3911445992013,6cyclictest3958754-21awk21:20:211
3911445992013,5cyclictest4014181-21python323:55:211
3911445992012,6cyclictest4022012-21cron00:20:001
391144599200,1cyclictest0-21swapper/100:29:551
391144599200,19cyclictest3056-21PK-Backend23:40:001
391144599200,19cyclictest0-21swapper/122:40:591
391144599200,19cyclictest0-21swapper/121:38:401
391144599200,19cyclictest0-21swapper/120:32:011
391144599200,19cyclictest0-21swapper/120:25:321
391144599200,19cyclictest0-21swapper/120:24:451
391144599200,19cyclictest0-21swapper/119:19:421
391144599200,19cyclictest0-21swapper/100:38:471
3911441992018,1cyclictest0-21swapper/023:41:200
3911441992018,1cyclictest0-21swapper/023:25:310
3911441992018,1cyclictest0-21swapper/022:27:250
3911441992018,1cyclictest0-21swapper/021:39:050
3911441992018,1cyclictest0-21swapper/021:06:200
3911441992018,1cyclictest0-21swapper/020:21:590
3911441992018,1cyclictest0-21swapper/020:17:400
3911441992018,1cyclictest0-21swapper/020:11:590
3911441992018,1cyclictest0-21swapper/020:01:450
3911441992018,1cyclictest0-21swapper/019:57:320
3911441992018,1cyclictest0-21swapper/019:52:320
3911441992018,1cyclictest0-21swapper/000:38:450
3911441992018,1cyclictest0-21swapper/000:29:530
3911441992018,1cyclictest0-21swapper/000:01:070
3911441992013,6cyclictest3989265-21ntpq22:45:190
3911441992013,6cyclictest3943649-21apt20:40:140
3911441992013,6cyclictest3943649-21apt20:40:140
3911441992013,5cyclictest3980360-21open_files22:20:190
3911441992012,6cyclictest3993611-21rm23:00:000
391144199200,1cyclictest0-21swapper/021:28:440
391144199200,19cyclictest0-21swapper/021:43:240
391144199200,19cyclictest0-21swapper/020:37:540
391144199200,19cyclictest0-21swapper/020:31:250
391144199200,19cyclictest0-21swapper/000:07:090
139099200,14rtkit-daemon0-21swapper/319:40:353
139099200,13rtkit-daemon0-21swapper/322:00:083
391145399194,13cyclictest3976827-21netstat22:10:183
3911453991917,1cyclictest3056-21PK-Backend23:50:063
3911453991917,1cyclictest3056-21PK-Backend22:22:393
3911453991916,2cyclictest3056-21PK-Backend21:47:243
391145399190,1cyclictest3056-21PK-Backend21:53:053
391145399190,1cyclictest3056-21PK-Backend21:10:333
391145399190,1cyclictest3056-21PK-Backend20:51:543
391145399190,1cyclictest3056-21PK-Backend20:14:023
391145399190,1cyclictest3056-21PK-Backend19:17:323
391145399190,18cyclictest3056-21PK-Backend19:37:323
3911449991917,1cyclictest0-21swapper/222:15:002
3911449991917,1cyclictest0-21swapper/200:27:152
3911449991914,3cyclictest3997182-21apt23:10:002
391144999190,1cyclictest0-21swapper/223:28:142
391144999190,1cyclictest0-21swapper/222:51:032
391144999190,1cyclictest0-21swapper/222:34:472
391144999190,1cyclictest0-21swapper/221:07:272
391144999190,1cyclictest0-21swapper/220:47:062
391144999190,18cyclictest0-21swapper/223:15:032
3911445991917,1cyclictest0-21swapper/121:16:151
3911445991917,1cyclictest0-21swapper/119:22:051
3911445991912,6cyclictest3989998-21cron22:50:001
3911445991912,6cyclictest3947819-21cat20:50:161
391144599190,1cyclictest0-21swapper/123:00:391
391144599190,1cyclictest0-21swapper/122:27:201
391144599190,1cyclictest0-21swapper/120:41:291
391144599190,1cyclictest0-21swapper/120:41:281
391144599190,1cyclictest0-21swapper/120:00:451
391144599190,1cyclictest0-21swapper/119:38:351
3911441991917,1cyclictest0-21swapper/022:09:130
3911441991917,1cyclictest0-21swapper/021:10:260
3911441991917,1cyclictest0-21swapper/021:00:250
3911441991917,1cyclictest0-21swapper/020:49:440
3911441991913,4cyclictest4009984-21cpuspeed_turbos23:45:130
3911441991913,4cyclictest4009984-21cpuspeed_turbos23:45:130
3911441991912,6cyclictest3971387-21ntp_kernel_err21:55:180
3911441991912,5cyclictest4014068-21awk23:55:190
3911441991912,5cyclictest3999933-21sh23:15:210
3911441991912,5cyclictest3995978-21cat23:05:130
3911441991912,5cyclictest3993919-21rm23:00:010
391144199190,18cyclictest0-21swapper/023:31:190
391144199190,18cyclictest0-21swapper/000:13:090
3911453991816,1cyclictest3056-21PK-Backend23:14:573
3911453991816,1cyclictest3056-21PK-Backend00:01:283
3911453991813,3cyclictest3929206-21munin-plugin-st20:00:003
391144999189,6cyclictest436-20systemd-journal19:20:002
3911449991816,1cyclictest0-21swapper/200:20:122
3911449991816,1cyclictest0-21swapper/200:08:432
3911449991812,5cyclictest3923416-21awk19:40:212
3911449991812,4cyclictest3988688-21tr22:45:132
3911449991811,5cyclictest436-20systemd-journal20:40:012
3911449991811,5cyclictest3957851-21cut21:20:012
3911449991810,6cyclictest3974782-21df_inode22:05:172
3911445991816,1cyclictest0-21swapper/122:09:221
3911445991811,5cyclictest3926269-21expr19:50:121
3911445991810,6cyclictest3993474-21apt-get23:00:001
3911441991816,1cyclictest3056-21PK-Backend21:21:210
3911441991812,5cyclictest3922763-21cat19:40:140
3911441991811,5cyclictest436-20systemd-journal20:07:170
391144199181,15cyclictest3921479-21cut19:35:180
3911453991715,1cyclictest3056-21PK-Backend20:21:533
3911449991715,1cyclictest0-21swapper/220:57:472
3911449991710,6cyclictest3988297-21systemd22:45:002
3911445991715,1cyclictest3056-21PK-Backend23:23:371
3911445991715,1cyclictest3056-21PK-Backend22:02:051
391144199177,9cyclictest13-21ksoftirqd/023:24:090
3911441991712,3cyclictest3987385-21missed_timers22:40:170
3911441991711,5cyclictest3969343-21fw_packets21:50:150
391144199171,13cyclictest3977874-21apt22:15:120
3911453991614,1cyclictest3056-21PK-Backend22:55:513
3911453991614,1cyclictest3056-21PK-Backend20:27:343
3911453991611,4cyclictest4001310-21cat23:20:153
3911453991610,4cyclictest4007297-21wc23:35:213
391144999162,12cyclictest4026186-21cpuspeed_turbos00:30:122
391144599168,6cyclictest3996448-21ntpq23:05:181
3911445991613,2cyclictest4011319-21latency_hist23:50:001
3911445991613,2cyclictest4011319-21latency_hist23:50:001
391144199169,5cyclictest3961246-21apt-get21:30:010
3911441991611,3cyclictest3990359-21cpuspeed_turbos22:50:120
3911441991610,4cyclictest3956350-21cat21:15:120
391145399159,4cyclictest3997381-21date23:10:013
3911445991512,2cyclictest3997462-21apt23:10:121
3911453991412,1cyclictest3056-21PK-Backend23:00:383
3911453991410,3cyclictest42-21ksoftirqd/320:02:493
3911453991410,3cyclictest3056-21PK-Backend23:57:513
391144999137,5cyclictest3998257-21grep23:10:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional