You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-11-17 - 14:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Tue Nov 17, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
311343221400,2sleep20-21swapper/223:45:212
301380721110,4sleep30-21swapper/319:05:243
311981921030,4sleep30-21swapper/300:05:163
301712721000,2sleep13056-21PK-Backend19:15:131
301409727639,9sleep00-21swapper/019:06:060
139099740,62rtkit-daemon0-21swapper/119:07:441
301417126444,14sleep20-21swapper/219:07:042
30201402630,4sleep13056-21PK-Backend19:24:561
30209042550,8sleep234-21ksoftirqd/219:25:172
30955562290,13chrt42-21ksoftirqd/322:55:223
3014568992720,5cyclictest3129796-21cron00:34:591
301457599260,11cyclictest30463352sensors_temp20:35:213
30463362240,3sleep10-21swapper/120:35:211
301457599248,14cyclictest3125457-21grep00:20:183
301457599240,5cyclictest3102528-21latency23:15:173
301457599240,3cyclictest3056603-21ntp_offset21:05:183
301457099242,20cyclictest3090745-21systemd-udevd22:44:322
30734532230,4chrt3056-21PK-Backend21:55:011
301457599238,13cyclictest3032770-21cstates20:00:123
3014575992318,3cyclictest3024290-21hddtemp_smartct19:35:143
3014575992316,4cyclictest3044345-21python320:30:213
3014570992317,5cyclictest3126343-21/usr/sbin/munin00:24:592
3014570992315,6cyclictest3101707-21cron23:15:002
139099230,16rtkit-daemon3056-21PK-Backend19:57:391
139099230,15rtkit-daemon3056-21PK-Backend20:16:291
30510702220,9chrt26-21ksoftirqd/120:50:131
3014575992215,5cyclictest3115637-21/usr/sbin/munin23:55:013
3014575992210,10cyclictest3069515-21systemd21:45:013
301457599220,9cyclictest3070739-21ntp_states21:45:173
301457599220,20cyclictest3074095-21df_inode21:55:133
3014570992220,1cyclictest0-21swapper/221:44:042
3014570992220,1cyclictest0-21swapper/220:44:022
3014570992219,2cyclictest0-21swapper/221:30:192
3014570992215,5cyclictest3073405-21mktemp21:55:002
301456499226,14cyclictest1-21systemd00:00:010
3014564992220,1cyclictest0-21swapper/023:44:200
3014564992220,1cyclictest0-21swapper/023:35:280
3014564992220,1cyclictest0-21swapper/022:31:130
3014564992214,6cyclictest3019436-21ntp_states19:20:190
301456499220,21cyclictest0-21swapper/000:17:470
139099220,16rtkit-daemon3056-21PK-Backend23:37:061
139099220,15rtkit-daemon3056-21PK-Backend19:29:301
139099220,14rtkit-daemon0-21swapper/022:40:120
30530702210,4sleep13056-21PK-Backend20:55:181
3014575992118,2cyclictest0-21swapper/323:45:183
3014575992116,3cyclictest3018443-21idleruntime-cro19:20:013
3014575992115,4cyclictest2156-21gnome-shell19:55:093
3014575992114,5cyclictest42-21ksoftirqd/321:00:113
3014575992113,6cyclictest42-21ksoftirqd/321:20:083
301457599211,18cyclictest3116644-21ntp_states23:55:203
301457599211,18cyclictest3092805-21apt22:50:113
301457599210,19cyclictest0-21swapper/320:51:323
3014570992119,1cyclictest0-21swapper/223:59:302
3014570992119,1cyclictest0-21swapper/223:53:152
3014570992119,1cyclictest0-21swapper/222:22:302
3014570992119,1cyclictest0-21swapper/220:19:482
3014570992119,1cyclictest0-21swapper/220:14:482
3014570992119,1cyclictest0-21swapper/219:23:112
301457099210,20cyclictest34-21ksoftirqd/223:44:092
301457099210,20cyclictest3086808-21find22:30:212
301457099210,20cyclictest0-21swapper/222:58:332
301457099210,20cyclictest0-21swapper/220:32:122
301457099210,20cyclictest0-21swapper/219:51:162
3014568992119,1cyclictest0-21swapper/120:40:211
3014564992119,1cyclictest0-21swapper/023:29:140
3014564992119,1cyclictest0-21swapper/023:24:480
3014564992119,1cyclictest0-21swapper/022:54:460
3014564992119,1cyclictest0-21swapper/022:39:040
3014564992119,1cyclictest0-21swapper/022:05:580
3014564992119,1cyclictest0-21swapper/021:28:280
3014564992119,1cyclictest0-21swapper/021:28:280
3014564992119,1cyclictest0-21swapper/021:21:170
3014564992119,1cyclictest0-21swapper/021:03:130
3014564992119,1cyclictest0-21swapper/020:39:130
3014564992119,1cyclictest0-21swapper/020:22:030
3014564992119,1cyclictest0-21swapper/000:38:140
3014564992119,1cyclictest0-21swapper/000:01:180
3014564992115,4cyclictest3115636-21apt23:55:010
301456499210,20cyclictest3056-21PK-Backend20:11:290
301456499210,20cyclictest3056-21PK-Backend00:05:430
301456499210,20cyclictest0-21swapper/023:01:480
301456499210,20cyclictest0-21swapper/023:00:000
301456499210,20cyclictest0-21swapper/022:46:560
301456499210,20cyclictest0-21swapper/020:56:170
301456499210,20cyclictest0-21swapper/020:07:300
301456499210,19cyclictest0-21swapper/023:10:420
301456499210,19cyclictest0-21swapper/022:26:000
139099210,16rtkit-daemon3056-21PK-Backend23:29:351
139099210,15rtkit-daemon0-21swapper/019:25:560
139099210,14rtkit-daemon3056-21PK-Backend20:05:031
139099210,13rtkit-daemon0-21swapper/019:49:590
31274622200,4sleep03056-21PK-Backend00:25:220
30851512200,2sleep13056-21PK-Backend22:25:201
3014575992017,2cyclictest0-21swapper/300:19:383
3014575992013,6cyclictest3099521-21sed23:05:213
3014575992013,5cyclictest42-21ksoftirqd/323:25:203
3014575992013,5cyclictest3122593-21basename00:14:593
3014575992013,5cyclictest1-21systemd22:14:593
3014575992012,6cyclictest436-20systemd-journal21:17:003
3014575992012,6cyclictest3082401-21apt22:20:133
301457599201,17cyclictest3100992-21ntp_states23:10:163
301457599200,2cyclictest3047797-21grep20:40:173
301457599200,19cyclictest0-21swapper/320:14:283
301457599200,19cyclictest0-21swapper/300:30:203
301457599200,18cyclictest0-21swapper/320:05:023
3014570992018,1cyclictest0-21swapper/222:25:072
3014570992018,1cyclictest0-21swapper/220:00:152
301457099201,18cyclictest1095-21runrttasks23:20:122
301457099200,1cyclictest0-21swapper/221:55:402
301457099200,19cyclictest0-21swapper/223:36:052
301457099200,19cyclictest0-21swapper/223:19:072
301457099200,19cyclictest0-21swapper/223:00:222
301457099200,19cyclictest0-21swapper/222:51:292
301457099200,19cyclictest0-21swapper/221:13:222
301457099200,19cyclictest0-21swapper/221:02:272
301457099200,19cyclictest0-21swapper/220:36:252
301457099200,19cyclictest0-21swapper/220:25:362
301457099200,19cyclictest0-21swapper/219:42:512
301457099200,19cyclictest0-21swapper/219:38:252
301457099200,19cyclictest0-21swapper/219:32:232
301457099200,19cyclictest0-21swapper/219:32:232
301456899200,1cyclictest3056-21PK-Backend20:02:571
3014564992018,1cyclictest0-21swapper/022:16:480
3014564992018,1cyclictest0-21swapper/022:00:110
3014564992018,1cyclictest0-21swapper/020:46:030
3014564992018,1cyclictest0-21swapper/020:42:380
3014564992018,1cyclictest0-21swapper/020:32:380
301456499200,1cyclictest0-21swapper/023:15:080
301456499200,1cyclictest0-21swapper/021:15:440
301456499200,19cyclictest0-21swapper/000:20:240
139099200,7rtkit-daemon3056-21PK-Backend23:12:311
139099200,14rtkit-daemon3056-21PK-Backend00:06:511
139099200,14rtkit-daemon0-21swapper/021:11:100
139099200,14rtkit-daemon0-21swapper/019:41:250
139099200,13rtkit-daemon0-21swapper/022:20:080
139099200,13rtkit-daemon0-21swapper/020:27:490
31000012190,3sleep1251rcuc/123:10:001
30311192190,4sleep00-21swapper/019:55:140
3014575991914,3cyclictest3104587-21python323:20:213
3014575991913,5cyclictest42-21ksoftirqd/321:35:203
3014575991913,4cyclictest42-21ksoftirqd/322:45:203
3014575991913,4cyclictest42-21ksoftirqd/322:25:103
3014575991912,5cyclictest3065449-21switchtime21:30:213
3014575991911,6cyclictest3118391-21ntpq00:00:193
3014575991911,6cyclictest3063670-21python321:25:213
3014575991911,6cyclictest3063670-21python321:25:213
3014575991910,7cyclictest3040830-21users20:20:233
301457599190,4cyclictest3053074-21strings20:55:173
301457599190,1cyclictest0-21swapper/319:32:513
301457599190,1cyclictest0-21swapper/319:32:503
301457599190,18cyclictest0-21swapper/321:50:063
301457599190,18cyclictest0-21swapper/320:16:043
301457599190,18cyclictest0-21swapper/319:20:133
301457599190,18cyclictest0-21swapper/300:37:503
301457599190,17cyclictest3025636-21apt19:40:143
301457099192,5cyclictest0-21swapper/221:45:032
3014570991917,1cyclictest0-21swapper/220:57:062
3014570991917,1cyclictest0-21swapper/219:46:292
3014570991917,1cyclictest0-21swapper/200:32:432
3014570991917,1cyclictest0-21swapper/200:12:012
3014570991915,3cyclictest978-21snmpd20:23:272
3014570991912,5cyclictest3106158-21ntp_kernel_err23:25:192
3014570991912,5cyclictest3066716-21cat21:35:172
3014570991911,6cyclictest3123861-21ntpq00:15:172
3014570991911,6cyclictest3107990-21ntpq23:30:192
301457099190,1cyclictest0-21swapper/222:05:412
301457099190,1cyclictest0-21swapper/200:26:412
301457099190,18cyclictest0-21swapper/221:18:562
301457099190,18cyclictest0-21swapper/200:01:202
3014568991917,1cyclictest3056-21PK-Backend22:58:511
301456899190,1cyclictest3056-21PK-Backend23:44:561
301456899190,1cyclictest3056-21PK-Backend22:20:121
301456899190,1cyclictest3056-21PK-Backend21:47:271
301456899190,18cyclictest3056-21PK-Backend20:21:091
3014564991917,1cyclictest0-21swapper/023:33:400
3014564991917,1cyclictest0-21swapper/022:11:470
3014564991917,1cyclictest0-21swapper/021:34:020
3014564991917,1cyclictest0-21swapper/021:05:360
3014564991917,1cyclictest0-21swapper/020:04:190
3014564991917,1cyclictest0-21swapper/000:10:580
139099190,13rtkit-daemon0-21swapper/019:51:520
30669532180,4sleep00-21swapper/021:35:190
301457599188,8cyclictest1-21systemd22:34:143
3014575991816,1cyclictest0-21swapper/323:00:233
3014575991814,3cyclictest978-21snmpd23:34:083
3014575991811,6cyclictest3078062-21python322:05:213
3014575991811,5cyclictest3111267-21cat23:40:143
3014575991811,5cyclictest3029479-21cat19:50:143
301457599180,6cyclictest3015815-21users19:10:233
301457599180,16cyclictest3089332-21apt22:40:103
301457599180,16cyclictest3076045-21memory22:00:203
301457099189,5cyclictest3130069-21apt00:35:132
3014570991816,1cyclictest0-21swapper/220:49:162
3014570991816,1cyclictest0-21swapper/219:10:132
3014570991812,5cyclictest3061916-21gzip21:20:182
3014570991812,4cyclictest28650irq/123-ahci[0021:05:202
3014570991811,5cyclictest436-20systemd-journal22:49:592
3014570991811,5cyclictest3079110-21grep22:10:112
3014570991811,5cyclictest3075746-21expr22:00:152
3014570991811,5cyclictest3064298-21systemctl21:30:002
3014570991811,5cyclictest3064298-21systemctl21:29:592
3014570991811,5cyclictest3016678-21cat19:15:022
301457099180,1cyclictest0-21swapper/222:19:522
3014568991816,1cyclictest3056-21PK-Backend19:54:321
3014568991816,1cyclictest0-21swapper/122:10:121
3014568991816,1cyclictest0-21swapper/100:22:271
3014568991811,6cyclictest3055002-21sshd21:00:201
3014568991811,5cyclictest3045129-21wc20:35:011
3014564991816,1cyclictest3056-21PK-Backend21:45:240
3014564991816,1cyclictest0-21swapper/023:45:230
3014564991816,1cyclictest0-21swapper/019:37:080
3014564991816,1cyclictest0-21swapper/019:34:310
3014564991816,1cyclictest0-21swapper/019:34:310
3014564991816,1cyclictest0-21swapper/000:34:030
3014564991813,3cyclictest3099428-21switchtime23:05:190
3014575991712,3cyclictest3126976-21diskstats00:25:153
3014575991712,3cyclictest3109488-21latency23:35:173
3014575991712,3cyclictest3057605-21apt21:10:133
3014575991712,3cyclictest3027338-21/usr/sbin/munin19:45:113
301457599171,14cyclictest3087572-21/usr/sbin/munin22:35:153
301457599170,5cyclictest3020739-21expr19:25:133
301457599170,15cyclictest3081043-21cat22:15:143
301457599170,15cyclictest3041749-21apt20:25:123
301457099174,11cyclictest3087886-21cstates22:35:122
3014570991715,1cyclictest0-21swapper/223:08:272
3014570991710,5cyclictest3031505-21cut19:55:202
301456899178,8cyclictest3056-21PK-Backend19:40:001
3014568991715,1cyclictest3056-21PK-Backend23:55:361
3014568991715,1cyclictest3056-21PK-Backend23:23:471
3014568991715,1cyclictest3056-21PK-Backend21:10:561
3014568991715,1cyclictest3056-21PK-Backend21:08:141
3014568991715,1cyclictest0-21swapper/120:47:521
3014568991714,2cyclictest3056-21PK-Backend23:17:041
3014568991714,2cyclictest3056-21PK-Backend21:25:581
3014568991714,2cyclictest3056-21PK-Backend21:25:581
3014568991711,5cyclictest3127218-21grep00:25:181
301456899170,6cyclictest3056-21PK-Backend19:48:171
301456899170,15cyclictest3056-21PK-Backend22:00:121
3014564991714,2cyclictest3056-21PK-Backend19:11:470
3014564991711,5cyclictest3050664-21awk20:50:080
3014564991710,5cyclictest3073378-21cat21:55:010
301456499170,7cyclictest3069465-21turbostat21:44:590
301457099166,8cyclictest3119818-21if_enp2s000:05:162
3014570991614,1cyclictest0-21swapper/220:51:052
3014570991610,5cyclictest3035036-21ntpq20:05:192
301456899169,5cyclictest3023085-21cat19:30:221
301456899169,5cyclictest3023085-21cat19:30:221
3014568991614,1cyclictest3056-21PK-Backend22:46:151
3014568991614,1cyclictest3056-21PK-Backend21:55:181
3014568991613,2cyclictest3056-21PK-Backend21:30:511
301456899160,15cyclictest3056-21PK-Backend00:14:571
301456899160,15cyclictest0-21swapper/119:11:261
301456899160,15cyclictest0-21swapper/100:35:391
3014564991611,4cyclictest13-21ksoftirqd/022:00:010
30432082150,7sleep1251rcuc/120:30:001
301457599150,14cyclictest3056-21PK-Backend20:46:593
301456899150,6cyclictest3056-21PK-Backend22:50:191
301456899150,6cyclictest3056-21PK-Backend00:05:001
301456899150,6cyclictest26-21ksoftirqd/100:15:151
301456899150,14cyclictest3056-21PK-Backend23:54:011
301456899150,14cyclictest3056-21PK-Backend23:31:171
301456899150,14cyclictest3056-21PK-Backend21:19:431
301456899150,13cyclictest3056-21PK-Backend22:35:471
301456499158,5cyclictest3037479-21cron20:19:560
3014564991511,3cyclictest3016446-21munin-node19:15:020
3014568991412,1cyclictest3056-21PK-Backend23:46:311
3014568991412,1cyclictest3056-21PK-Backend22:40:201
301456899141,11cyclictest3036155-21tr20:10:121
3014568991410,3cyclictest3069760-21apt-key21:45:011
3014568991410,3cyclictest3056-21PK-Backend21:36:531
301456899140,1cyclictest3056-21PK-Backend22:34:121
301456899140,1cyclictest3056-21PK-Backend22:09:241
301456899140,13cyclictest3056-21PK-Backend23:02:371
3014568991310,2cyclictest3025609-21/usr/sbin/munin19:40:191
301456899130,5cyclictest3056-21PK-Backend21:20:111
139099110,4rtkit-daemon3056-21PK-Backend22:15:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional