You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-13 - 21:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Thu Aug 13, 2026 12:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
396508621300,5sleep33056-21PK-Backend08:05:193
139099850,72rtkit-daemon0-21swapper/107:07:021
394397027455,13sleep20-21swapper/207:09:502
394248827343,21sleep30-21swapper/307:05:043
394374127049,14sleep00-21swapper/007:06:500
40150282470,11sleep013-21ksoftirqd/010:30:030
40343432460,3sleep30-21swapper/311:25:123
39912832430,3sleep13056-21PK-Backend09:20:201
40566742420,4sleep00-21swapper/012:30:100
394411599390,27cyclictest3948105-21taskset07:18:000
40084512380,3sleep30-21swapper/310:10:193
394411599380,26cyclictest3967241-21chrt08:12:580
394411599380,17cyclictest3963647-21chrt08:01:280
394411599370,19cyclictest1095-21runrttasks07:38:260
394411599360,19cyclictest1095-21runrttasks07:12:030
394411599360,18cyclictest3960747-21grep07:55:110
394411599360,18cyclictest3960124-21sshd07:50:430
394411599360,17cyclictest3953111-21seq07:30:320
39768422340,2sleep30-21swapper/308:40:163
394411599340,18cyclictest3949157-21missed_timers07:20:180
394411599330,25cyclictest3968995-21runrttasks08:18:120
394411599330,18cyclictest3951593-21taskset07:28:190
394411599330,18cyclictest1095-21runrttasks07:43:140
394411599330,17cyclictest3957239-21/usr/sbin/munin07:45:180
394411599321,15cyclictest2156-21gnome-shell10:07:080
394411599310,17cyclictest3974352-21perf08:39:560
394412299306,8cyclictest4009112-21taskset10:12:332
394412299306,8cyclictest4005445-21taskset10:00:392
394412299306,11cyclictest3946475-21hddtemp_smartct07:15:022
3944122993010,18cyclictest4052959-21unin-run12:20:012
394411599303,13cyclictest0-21swapper/008:54:590
394411599301,27cyclictest4018506-21/usr/sbin/munin10:40:140
394411599301,16cyclictest3965500-21chrt08:08:010
394412299297,9cyclictest3956397-21python307:40:222
394412299296,9cyclictest3964682-21irqstats08:05:152
394412299294,10cyclictest3988222-21taskset09:11:592
139099280,4rtkit-daemon1389-21rtkit-daemon10:56:512
394412299276,8cyclictest3959407-21hddtemp_smartct07:50:162
394412299276,8cyclictest3951022-21ntp_states07:25:182
394412299276,8cyclictest3949127-21memory07:20:172
3944122992711,14cyclictest3998881-21/usr/sbin/munin09:45:002
394412299266,8cyclictest3952879-21python307:30:222
3944122992618,6cyclictest4054646-21cron12:25:012
3944115992610,14cyclictest3982259-21awk08:55:180
394412299259,14cyclictest436-20systemd-journal08:25:292
394412299255,6cyclictest3958741-21munin-run07:50:012
3944122992513,8cyclictest4041062-21apt11:45:102
3944122992513,8cyclictest4041062-21apt11:45:102
3944122992513,7cyclictest4014534-21apt-get10:30:052
394411599259,14cyclictest3997348-21date09:40:010
394411599259,14cyclictest3989328-21grep09:15:170
3944115992512,11cyclictest4059530-21python312:35:220
3944115992511,12cyclictest4041036-21/usr/sbin/munin11:45:180
3944115992511,12cyclictest4041036-21/usr/sbin/munin11:45:170
139099250,4rtkit-daemon1389-21rtkit-daemon12:25:552
3944122992412,9cyclictest436-20systemd-journal09:25:002
3944115992418,5cyclictest3056-21PK-Backend10:21:080
394411599240,1cyclictest0-21swapper/010:54:260
394412499235,16cyclictest3991931-21cron09:25:013
394412299238,13cyclictest4047912-21cat12:05:002
3944122992315,6cyclictest4030382-21cron11:15:002
3944122992312,9cyclictest4028649-21cron11:09:592
394411999230,21cyclictest3056-21PK-Backend12:23:121
394411599239,12cyclictest4024710-21timerandwakeup10:55:210
394411599239,12cyclictest4010402-21cut10:15:210
3944115992318,4cyclictest0-21swapper/010:45:550
3944115992310,11cyclictest3991138-21grep09:20:180
394411599230,1cyclictest0-21swapper/011:18:470
394411599230,1cyclictest0-21swapper/009:41:120
139099230,4rtkit-daemon1389-21rtkit-daemon10:54:332
39954162220,3sleep23995408-21turbostat09:35:012
3944124992220,1cyclictest0-21swapper/312:06:113
3944124992220,1cyclictest0-21swapper/309:34:573
394412499220,20cyclictest0-21swapper/310:30:123
394412299226,6cyclictest3973249-21df08:30:122
3944122992217,3cyclictest0-21swapper/210:20:002
3944122992213,7cyclictest4058187-21dpkg12:35:002
3944122992213,7cyclictest4018681-21cut10:40:122
3944122992213,7cyclictest4004054-21cron10:00:002
3944122992213,5cyclictest4042799-21apt11:50:132
3944122992211,8cyclictest0-21swapper/212:10:132
3944119992220,1cyclictest3056-21PK-Backend10:57:281
3944119992220,1cyclictest3056-21PK-Backend08:44:321
3944119992220,1cyclictest0-21swapper/109:29:131
3944119992220,1cyclictest0-21swapper/107:26:311
3944119992219,2cyclictest3056-21PK-Backend10:17:251
3944119992219,2cyclictest3056-21PK-Backend10:02:391
3944119992219,2cyclictest3056-21PK-Backend09:57:241
3944119992219,2cyclictest3056-21PK-Backend08:31:411
3944119992214,6cyclictest26-21ksoftirqd/107:50:061
394411999220,20cyclictest3056-21PK-Backend12:04:471
394411999220,20cyclictest3056-21PK-Backend11:29:181
394411599228,12cyclictest4036811-21timerandwakeup11:30:200
394411599228,12cyclictest0-21swapper/008:27:220
394411599225,15cyclictest4046742-21kernelversion12:00:180
3944115992220,1cyclictest0-21swapper/011:27:310
3944115992220,1cyclictest0-21swapper/010:28:100
394411599220,20cyclictest3979587-21grep08:49:590
394411599220,1cyclictest0-21swapper/012:17:330
139099220,14rtkit-daemon0-21swapper/010:13:160
139099220,14rtkit-daemon0-21swapper/010:00:230
3944124992119,1cyclictest0-21swapper/312:38:093
3944124992119,1cyclictest0-21swapper/311:55:023
3944124992119,1cyclictest0-21swapper/310:01:463
3944124992119,1cyclictest0-21swapper/309:11:523
3944124992119,1cyclictest0-21swapper/308:56:313
3944124992119,1cyclictest0-21swapper/308:51:293
3944124992119,1cyclictest0-21swapper/308:31:503
3944124992119,1cyclictest0-21swapper/308:21:143
394412499211,8cyclictest4043274-21irqstats11:50:163
3944124992118,2cyclictest3056-21PK-Backend07:59:123
3944124992113,6cyclictest4054055-21python312:20:203
3944124992113,6cyclictest4017515-21ntp_kernel_pll_10:35:173
3944124992113,6cyclictest3989454-21ntpq09:15:183
394412499210,9cyclictest3985785-21pluginstate09:05:193
394412499210,20cyclictest0-21swapper/309:01:173
394412499210,19cyclictest4028663-21ls11:10:003
394412299218,11cyclictest4002323-21cron09:54:592
394412299217,5cyclictest3983842-21cat09:00:162
394412299215,6cyclictest3944907-21memory07:10:162
3944122992115,4cyclictest4045395-21taskset11:55:222
3944122992115,3cyclictest3995677-21apt09:35:122
3944122992114,5cyclictest3986786-21latency_hist09:10:012
3944122992112,7cyclictest4012823-21sleep10:25:012
3944122992111,8cyclictest4040051-21ntpq11:40:182
3944122992111,5cyclictest4037601-21apt11:35:102
394412299210,20cyclictest0-21swapper/211:20:452
3944119992119,1cyclictest3056-21PK-Backend12:06:491
3944119992119,1cyclictest3056-21PK-Backend11:11:061
3944119992119,1cyclictest3056-21PK-Backend10:31:461
3944119992119,1cyclictest3056-21PK-Backend09:53:331
3944119992119,1cyclictest3056-21PK-Backend09:40:081
3944119992119,1cyclictest3056-21PK-Backend09:12:371
3944119992119,1cyclictest0-21swapper/112:31:371
3944119992119,1cyclictest0-21swapper/111:49:181
3944119992119,1cyclictest0-21swapper/111:49:181
3944119992119,1cyclictest0-21swapper/108:11:131
3944119992119,1cyclictest0-21swapper/108:04:361
3944119992119,1cyclictest0-21swapper/107:42:201
3944119992119,1cyclictest0-21swapper/107:37:121
3944119992119,1cyclictest0-21swapper/107:20:021
3944119992119,1cyclictest0-21swapper/107:17:461
3944119992118,2cyclictest3056-21PK-Backend11:53:391
394411999210,20cyclictest3056-21PK-Backend10:23:411
394411999210,20cyclictest0-21swapper/109:48:061
394411999210,20cyclictest0-21swapper/107:55:181
394411999210,20cyclictest0-21swapper/107:13:271
394411999210,19cyclictest3056-21PK-Backend10:41:331
394411999210,19cyclictest3056-21PK-Backend08:45:201
394411599216,13cyclictest0-21swapper/011:40:210
394411599215,14cyclictest4029055-21tr11:10:120
394411599214,15cyclictest4033199-21processes11:20:190
394411599213,16cyclictest0-21swapper/009:49:220
3944115992119,1cyclictest0-21swapper/012:13:280
3944115992119,1cyclictest0-21swapper/012:05:520
3944115992116,4cyclictest0-21swapper/009:58:440
3944115992115,5cyclictest0-21swapper/011:05:490
394411599211,19cyclictest4002353-21idleruntime-cro09:55:000
394411599211,18cyclictest3969703-21cstates08:20:140
394411599210,20cyclictest0-21swapper/009:01:320
139099210,15rtkit-daemon3056-21PK-Backend10:49:201
139099210,15rtkit-daemon3056-21PK-Backend08:50:361
139099210,14rtkit-daemon0-21swapper/109:32:131
394412499207,11cyclictest42-21ksoftirqd/307:20:133
3944124992018,1cyclictest3056-21PK-Backend07:33:163
3944124992018,1cyclictest3056-21PK-Backend07:17:003
3944124992018,1cyclictest0-21swapper/312:28:213
3944124992018,1cyclictest0-21swapper/311:16:093
3944124992018,1cyclictest0-21swapper/310:58:243
3944124992018,1cyclictest0-21swapper/309:47:483
3944124992018,1cyclictest0-21swapper/309:37:403
3944124992018,1cyclictest0-21swapper/309:29:163
3944124992014,4cyclictest42-21ksoftirqd/307:45:013
3944124992013,5cyclictest4026041-21/usr/sbin/munin11:00:173
3944124992012,6cyclictest436-20systemd-journal12:05:003
394412499200,19cyclictest3056-21PK-Backend07:28:023
394412499200,19cyclictest0-21swapper/312:16:243
394412499200,19cyclictest0-21swapper/310:44:323
394412499200,19cyclictest0-21swapper/310:27:563
394412499200,19cyclictest0-21swapper/308:28:533
394412499200,18cyclictest3056-21PK-Backend07:36:003
394412499200,10cyclictest40407982cron11:45:003
3944122992014,4cyclictest978-21snmpd10:09:132
3944122992013,6cyclictest3989533-21wc09:15:212
3944122992012,6cyclictest4036049-21cpuspeed_turbos11:30:122
3944122992012,6cyclictest4016957-21tr10:35:122
3944122992012,6cyclictest3978792-21awk08:45:172
3944122992011,6cyclictest4049450-21dump-pmu-power12:09:592
3944122992010,8cyclictest4014619-21ls10:30:012
3944122992010,7cyclictest3976354-21apt08:40:112
394412299200,19cyclictest0-21swapper/208:23:482
394412299200,19cyclictest0-21swapper/208:15:032
3944119992018,1cyclictest3056-21PK-Backend10:36:051
3944119992018,1cyclictest3056-21PK-Backend10:25:511
3944119992018,1cyclictest3056-21PK-Backend10:13:211
3944119992018,1cyclictest3056-21PK-Backend09:36:501
3944119992018,1cyclictest3056-21PK-Backend08:37:351
3944119992018,1cyclictest3056-21PK-Backend08:26:001
3944119992018,1cyclictest0-21swapper/107:33:411
3944119992017,2cyclictest3056-21PK-Backend11:59:471
3944119992017,2cyclictest3056-21PK-Backend11:39:461
394411599203,15cyclictest0-21swapper/012:25:170
3944115992018,1cyclictest0-21swapper/012:22:470
394411599200,1cyclictest0-21swapper/009:30:180
394411599200,19cyclictest0-21swapper/011:56:450
394411599200,19cyclictest0-21swapper/011:35:230
394411599200,18cyclictest4043448-21needreboot11:50:180
394411599200,18cyclictest3977054-21awk08:40:170
139099200,17rtkit-daemon3946201-21munin-node11:16:252
139099200,14rtkit-daemon3056-21PK-Backend09:19:141
139099200,14rtkit-daemon3056-21PK-Backend09:08:470
139099200,13rtkit-daemon0-21swapper/108:18:211
39998902190,1sleep20-21swapper/209:45:182
394412499191,7cyclictest4035867-21apt11:30:103
3944124991917,1cyclictest0-21swapper/311:46:243
3944124991917,1cyclictest0-21swapper/311:46:243
3944124991917,1cyclictest0-21swapper/311:36:163
3944124991917,1cyclictest0-21swapper/311:14:063
3944124991917,1cyclictest0-21swapper/310:24:253
3944124991917,1cyclictest0-21swapper/309:56:333
3944124991917,1cyclictest0-21swapper/309:40:233
3944124991916,2cyclictest3056-21PK-Backend07:46:273
3944124991914,3cyclictest3974621-21apt08:35:113
394412499190,1cyclictest0-21swapper/312:13:483
394412499190,1cyclictest0-21swapper/309:50:313
394412499190,18cyclictest3056-21PK-Backend08:10:343
394412499190,18cyclictest0-21swapper/311:21:433
394412299199,8cyclictest0-21swapper/208:50:092
3944122991913,3cyclictest4034134-21apt11:25:122
3944122991912,5cyclictest978-21snmpd08:35:172
3944122991912,5cyclictest3981732-21grep08:55:112
3944122991911,5cyclictest4058608-21cnf-update-db12:35:012
3944122991911,5cyclictest4026197-21ntp_states11:00:192
3944122991910,7cyclictest4020440-21grep10:45:132
3944122991910,7cyclictest3993147-21/usr/sbin/munin09:25:212
394412299190,1cyclictest0-21swapper/207:37:522
3944119991917,1cyclictest3056-21PK-Backend11:40:191
3944119991917,1cyclictest3056-21PK-Backend11:08:091
3944119991916,2cyclictest3056-21PK-Backend09:00:061
394411999190,1cyclictest3056-21PK-Backend12:27:521
394411999190,1cyclictest3056-21PK-Backend12:16:301
394411999190,1cyclictest3056-21PK-Backend09:05:331
394411999190,1cyclictest3056-21PK-Backend08:55:061
394411599193,14cyclictest3987546-21latency09:10:150
394411599193,14cyclictest0-21swapper/011:00:400
394411599193,14cyclictest0-21swapper/010:35:100
3944115991917,1cyclictest0-21swapper/008:30:430
394411599190,17cyclictest3992999-21grep09:25:200
3944124991816,1cyclictest3056-21PK-Backend08:19:533
394412499181,13cyclictest1095-21runrttasks10:50:113
394412499180,5cyclictest3978263-21cpuspeed_turbos08:45:093
394412499180,16cyclictest4056671-21apt12:30:113
394412299188,8cyclictest3965907-21spatcher08:10:072
3944122991810,6cyclictest3962454-21sort08:00:122
3944119991816,1cyclictest3056-21PK-Backend11:34:321
3944119991816,1cyclictest3056-21PK-Backend11:21:401
3944119991816,1cyclictest3056-21PK-Backend11:19:371
3944119991816,1cyclictest3056-21PK-Backend10:07:191
3944119991815,2cyclictest3056-21PK-Backend10:50:031
3944119991811,5cyclictest3957978-21ntpq07:45:181
394412499179,6cyclictest436-20systemd-journal10:50:003
394412499171,3cyclictest4007377-21lsb_release10:07:353
3944119991715,1cyclictest3056-21PK-Backend12:37:041
3944119991714,2cyclictest3056-21PK-Backend08:21:541
3944119991711,5cyclictest3964261-21ls08:05:091
394412499169,6cyclictest4009969-21df10:15:143
3944124991614,1cyclictest3056-21PK-Backend07:13:153
394412299168,6cyclictest3961321-21chrt07:55:152
3944119991613,2cyclictest3056-21PK-Backend12:14:261
3944124991513,1cyclictest3056-21PK-Backend08:02:163
3944124991513,1cyclictest3056-21PK-Backend07:54:383
394411999128,3cyclictest26-21ksoftirqd/111:03:261
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional