You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 18:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Fri Nov 28, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
39774789912030,1198cyclictest278725-21sleep12:40:003
39774789911370,1133cyclictest1492482sleep311:19:563
39774789911320,1128cyclictest2139682sleep311:59:553
3977478999581,952cyclictest4148310-21turbostat09:20:003
190898290,10rtkit-daemon180901-21ssh11:38:070
6069022418,3sleep060821-21irqcore10:25:170
397747899199,7cyclictest1803232sleep311:36:453
397747899181,14cyclictest27645-21turbostat10:05:003
397747899176,8cyclictest132892-21turbostat11:10:003
397747899176,7cyclictest4043797-21turbostat08:00:003
397747899167,6cyclictest214020-21turbostat12:04:543
397747899161,12cyclictest4119713-21turbostat09:00:003
397747899161,11cyclictest4173367-21chrt09:35:133
775222150,4sleep078598-21tr10:35:150
3977478991513,1cyclictest0-21swapper/308:29:263
3977478991513,1cyclictest0-21swapper/308:25:003
3977478991512,1cyclictest0-21swapper/312:05:163
3977478991512,1cyclictest0-21swapper/310:29:273
3977478991512,1cyclictest0-21swapper/309:25:403
3977478991512,1cyclictest0-21swapper/307:25:133
3977478991512,1cyclictest0-21swapper/307:11:573
397747899151,11cyclictest76030-21taskset10:34:413
397747899151,11cyclictest4100636-21turbostat08:45:013
397747899151,11cyclictest172745-21taskset11:33:033
397747899151,10cyclictest254092-21taskset12:24:083
397747899150,6cyclictest0-21swapper/309:20:123
397747899150,12cyclictest157659-21taskset11:25:023
190898150,5rtkit-daemon4146245-21ssh09:15:252
190898150,2rtkit-daemon0-21swapper/008:30:160
684592141,4sleep168618-21diskstats10:30:141
397747899146,6cyclictest843322sleep310:39:583
397747899145,6cyclictest2610732sleep312:26:373
397747899143,8cyclictest4094369-21turbostat08:40:013
397747899141,9cyclictest360552sleep310:10:123
3977478991412,1cyclictest0-21swapper/307:36:323
3977478991412,0cyclictest0-21swapper/310:24:413
3977478991412,0cyclictest0-21swapper/308:46:353
3977478991411,1cyclictest0-21swapper/312:30:233
3977478991411,1cyclictest0-21swapper/310:50:223
397747899141,10cyclictest9548-21taskset09:50:323
397747899141,10cyclictest90447-21taskset10:40:253
397747899141,10cyclictest51285-21taskset10:18:333
397747899141,10cyclictest4189960-21taskset09:45:163
397747899141,10cyclictest4187291-21chrt09:40:263
397747899141,10cyclictest4139212-21taskset09:12:293
397747899141,10cyclictest4082467-21taskset08:30:143
397747899141,10cyclictest4068944-21chrt08:20:003
397747899141,10cyclictest4047674-21taskset08:00:183
397747899141,10cyclictest3998891-21turbostat07:25:013
397747899141,10cyclictest33024-21chrt10:05:203
397747899141,10cyclictest33024-21chrt10:05:193
397747899141,10cyclictest230486-21taskset12:10:013
397747899141,10cyclictest18614-21chrt09:57:583
397747899140,11cyclictest3991678-21taskset07:15:213
397747899140,11cyclictest188712-21chrt11:42:253
397747899135,5cyclictest41651772sleep309:30:063
397747899131,9cyclictest41262842sleep309:05:143
397747899131,9cyclictest4119951-21taskset09:00:033
397747899131,9cyclictest4050331-21taskset08:05:103
397747899131,9cyclictest4011388-21taskset07:31:093
397747899131,9cyclictest139013-21taskset11:10:233
3977478991311,1cyclictest0-21swapper/310:55:223
397747899131,10cyclictest92706-21taskset10:45:033
397747899131,10cyclictest116973-21taskset11:00:023
397747899130,10cyclictest4059335-21chrt08:10:153
397747899130,10cyclictest4030952-21taskset07:49:563
397747899130,10cyclictest4018651-21taskset07:40:013
397747899130,10cyclictest204451-21taskset11:51:313
397747899130,10cyclictest196692-21chrt11:47:133
397747899130,10cyclictest1562002sleep311:22:103
397747899121,9cyclictest4037166-21taskset07:50:473
397747899121,8cyclictest238567-21chrt12:15:073
397747899120,9cyclictest4112766-21taskset08:50:263
2548012120,1sleep0254856-21cpuspeed_turbos12:25:100
190898120,7rtkit-daemon4005725-21apt-get07:30:111
41893762110,1sleep00-21swapper/009:45:000
41221112110,8sleep14124177-21cut09:00:181
40314872110,3sleep00-21swapper/007:50:140
39993902111,5sleep23999423-21apt-get07:25:112
2062262111,2sleep2206261-21apt-get11:55:112
190898110,7rtkit-daemon81364-21packagekitd10:20:560
190898110,7rtkit-daemon4157668-21ssh09:25:142
190898110,7rtkit-daemon204481-21ssh11:51:330
190898110,6rtkit-daemon278521-21sh12:39:312
190898110,6rtkit-daemon100550-21munin-run10:50:000
190898110,5rtkit-daemon197107-21ssh11:48:050
190898110,3rtkit-daemon4056115-21kworker/u16:0+events_power_efficient08:51:292
95742100,3sleep29582-21ssh09:50:362
55172100,7sleep07484-21cut09:50:190
41460112101,7sleep14146178-21ssh09:15:251
40750622105,2sleep0951254-21kworker/0:0+kacpid08:23:310
2385582101,7sleep2238566-21ssh12:15:062
2143502102,4sleep2214387-21apt-get12:00:112
190898100,7rtkit-daemon81364-21packagekitd08:39:152
190898100,6rtkit-daemon244547-21ssh12:15:440
190898100,5rtkit-daemon236751-21ssh12:11:220
190898100,5rtkit-daemon213328-21ssh11:58:260
190898100,3rtkit-daemon674-21lldpd09:13:322
190898100,3rtkit-daemon188197-21ssh11:41:190
190898100,1rtkit-daemon0-21swapper/012:20:240
1511422101,6sleep2152947-21kthreadcore11:20:162
1222802101,6sleep0122327-21ntp_states11:00:240
82426291,5sleep182434-21ssh10:35:361
45601291,2sleep10-21swapper/110:15:151
4105945291,4sleep14106022-21ntp_states08:45:191
4044443292,4sleep04044732-21cut08:00:160
4023753291,6sleep24023826-21/usr/sbin/munin07:40:212
3994354290,1sleep23996364-21kthreadcore07:20:192
253673290,6sleep1253681-21ssh12:23:161
24977291,5sleep025112-21/usr/sbin/munin10:00:210
19089890,5rtkit-daemon671-21ntpd08:19:240
19089890,5rtkit-daemon4163642-21kworker/u16:2+events_power_efficient10:06:332
19089890,5rtkit-daemon4163642-21kworker/u16:2+events_power_efficient10:06:322
19089890,5rtkit-daemon319-21systemd-udevd09:05:382
19089890,4rtkit-daemon85066-21hddtemp_smartct10:40:130
19089890,4rtkit-daemon73861-21python310:30:230
19089890,4rtkit-daemon51065-21ssh10:18:040
19089890,4rtkit-daemon4012399-21diskstats07:35:141
19089890,3rtkit-daemon667-21snmpd07:10:361
19089890,3rtkit-daemon3983979-21munin-node09:02:562
157823291,5sleep0157937-21/usr/sbin/munin11:25:150
149653291,4sleep1149656-21apt-get11:20:111
12168290,6sleep212897-21cut09:55:172
107161290,6sleep0107169-21ssh10:51:420
68619281,5sleep20-21swapper/210:30:162
60308281,3sleep160332-21apt-get10:25:131
4178650281,4sleep04178783-21python309:35:240
4166171280,5sleep24167079-21tr09:30:172
4157154281,3sleep14157193-21apt-get09:25:101
4107522280,5sleep04107890-21irqcore08:50:150
4056607281,3sleep14056640-21apt-get08:10:091
4044059280,4sleep14044081-21apt-get08:00:131
3992857281,3sleep13992879-21apt-get07:20:141
36260281,4sleep136056-21/usr/sbin/munin10:10:151
278627280,1sleep00-21swapper/012:39:460
264364280,5sleep1265816-21kthreadcore12:30:211
262880281,3sleep0262915-21apt-get12:30:150
229840280,5sleep0229849-21ssh12:09:120
19089880,5rtkit-daemon81364-21packagekitd09:42:252
19089880,5rtkit-daemon671-21ntpd09:45:212
19089880,5rtkit-daemon667-21snmpd08:46:272
19089880,5rtkit-daemon1928-21sendmail-mta08:06:580
19089880,5rtkit-daemon1928-21sendmail-mta07:58:380
19089880,5rtkit-daemon1928-21sendmail-mta07:51:291
19089880,4rtkit-daemon671-21ntpd08:10:090
19089880,4rtkit-daemon671-21ntpd07:25:501
19089880,4rtkit-daemon667-21snmpd11:14:210
19089880,4rtkit-daemon3983979-21munin-node12:33:572
19089880,4rtkit-daemon25736-21ssh10:00:342
19089880,4rtkit-daemon220610-21ssh12:01:250
19089880,4rtkit-daemon1-21systemd07:15:021
19089880,3rtkit-daemon173208-21ssh11:34:120
189806281,1sleep10-21swapper/111:45:011
150008280,5sleep0150150-21irqcore11:20:140
147911280,2sleep1147915-21ssh11:16:481
110437280,5sleep20-21swapper/210:55:162
93284270,4sleep194277-21irqstats10:45:181
61824271,3sleep263689-21cut10:25:182
52229271,3sleep252263-21apt-get10:20:102
4191570270,5sleep14193588-21kthreadcore09:45:191
4148958271,3sleep04149113-21df_inode09:20:130
4134077270,4sleep04135644-21cat09:10:160
4120959271,3sleep04122110-21tr09:00:170
4075734270,4sleep14075889-21diskstats08:25:131
3972382270,4sleep13973412-21grep07:05:191
3971600270,4sleep30-21swapper/307:05:173
260055270,4sleep1260170-21awk12:25:191
254291270,5sleep2254299-21ssh12:24:412
19513271,4sleep019541-21apt10:00:000
19089870,4rtkit-daemon81364-21packagekitd12:26:552
19089870,4rtkit-daemon671-21ntpd11:09:072
19089870,4rtkit-daemon667-21snmpd10:55:020
19089870,4rtkit-daemon473-21gmain08:35:080
19089870,4rtkit-daemon473-21gmain07:55:081
19089870,4rtkit-daemon4132287-21diskmemload11:10:262
19089870,4rtkit-daemon3983979-21munin-node11:19:250
19089870,4rtkit-daemon3983979-21munin-node11:06:090
19089870,4rtkit-daemon1928-21sendmail-mta07:40:031
19089870,3rtkit-daemon671-21ntpd09:35:072
19089870,3rtkit-daemon667-21snmpd10:11:002
19089870,3rtkit-daemon4148896-21ssh09:20:092
19089870,3rtkit-daemon3983979-21munin-node08:40:202
19089870,3rtkit-daemon1928-21sendmail-mta10:02:191
172610271,3sleep2172618-21ssh11:32:462
142302270,5sleep2143175-21kthreadcore11:15:162
138759270,4sleep1138865-21ssh11:10:211
11024271,3sleep111032-21ssh09:54:041
108945270,3sleep1108982-21apt-get10:55:111
99515260,1sleep20-21swapper/210:47:392
89926261,2sleep10-21swapper/110:40:201
84057260,1sleep20-21swapper/210:39:222
42345260,1sleep00-21swapper/010:11:350
4164631260,3sleep04164633-21ssh09:29:260
4135645260,3sleep14137584-21/usr/sbin/munin09:10:161
4128292260,3sleep14130228-21cut09:05:201
4127266261,2sleep00-21swapper/009:05:190
4115094261,2sleep00-21swapper/008:55:170
4113122260,3sleep14113136-21apt-get08:55:021
4089255260,3sleep14090140-21tr08:35:141
4075699261,2sleep00-21swapper/008:25:110
4075182260,3sleep24075202-21apt-get08:25:022
4074569260,3sleep24074813-21users08:20:242
4068758260,1sleep10-21swapper/108:18:281
4062883261,2sleep20-21swapper/208:15:122
4049957260,1sleep20-21swapper/208:03:302
4011690260,3sleep24011705-21apt-get07:35:012
3993240260,3sleep03993422-21cut07:20:170
3991395260,4sleep03991459-21ntp_states07:15:190
3984104260,3sleep23983732-21apt-get07:15:012
30012260,3sleep132092-21cut10:05:191
30012260,3sleep132092-21cut10:05:181
270940260,1sleep10-21swapper/112:35:141
238815260,3sleep1239020-21hddtemp_smartct12:15:131
237896260,1sleep20-21swapper/212:14:062
237172260,1sleep10-21swapper/112:12:251
228495260,1sleep20-21swapper/212:05:572
221206260,1sleep10-21swapper/112:02:511
213545261,2sleep10-21swapper/111:58:551
205422260,3sleep10-21swapper/111:53:491
196778260,1sleep20-21swapper/211:47:252
196147260,1sleep10-21swapper/111:45:561
19089860,3rtkit-daemon81364-21packagekitd08:55:042
19089860,2rtkit-daemon671-21ntpd07:45:301
188862260,1sleep20-21swapper/211:42:532
180414260,1sleep20-21swapper/211:36:592
179264261,2sleep1179371-21mailstats11:35:221
167441261,2sleep10-21swapper/111:30:171
165349260,3sleep2165357-21ssh11:29:472
124351260,1sleep10-21swapper/111:04:051
100911260,4sleep1100870-21/usr/sbin/munin10:50:151
91948250,3sleep20-21swapper/210:43:502
59928250,2sleep159936-21ssh10:24:581
49413250,3sleep249518-21grep10:15:202
4184018251,2sleep10-21swapper/109:40:171
4171755250,3sleep10-21swapper/109:32:041
4165443250,3sleep04165650-21hddtemp_smartct09:30:140
4154668250,3sleep10-21swapper/109:20:261
4140566250,1sleep00-21swapper/009:15:070
4106787250,1sleep00-21swapper/008:48:590
4100381250,1sleep10-21swapper/108:42:001
4087899250,3sleep10-21swapper/108:33:301
4082323250,2sleep20-21swapper/208:30:142
4051908251,2sleep10-21swapper/108:05:171
4030816250,1sleep20-21swapper/207:48:162
4030697250,1sleep00-21swapper/007:46:400
4024682250,3sleep04024860-21latency_hist07:45:000
4011382250,1sleep20-21swapper/207:31:052
4011365250,1sleep00-21swapper/007:30:500
3980739250,3sleep03982623-21cat07:10:190
3977646250,2sleep20-21swapper/207:10:122
3973413250,2sleep20-21swapper/207:05:202
27963250,2sleep028100-21cpuspeed_turbos10:05:130
27963250,2sleep028100-21cpuspeed_turbos10:05:130
205681250,3sleep2205689-21ssh11:54:232
19487250,3sleep119495-21ssh09:59:571
157676250,3sleep10-21swapper/111:25:051
131187250,3sleep1131195-21ssh11:06:091
116701250,3sleep2116972-21date11:00:022
107754250,3sleep20-21swapper/210:53:032
4190544240,2sleep04191567-21kthreadcore09:45:180
4179242240,3sleep10-21swapper/109:35:411
4100597241,1sleep00-21swapper/008:44:470
4075090240,3sleep10-21swapper/108:23:521
4062500240,3sleep20-21swapper/208:13:242
4040835240,2sleep20-21swapper/207:55:162
4001730240,2sleep00-21swapper/007:25:180
3971401240,1sleep00-21swapper/007:05:130
222412240,2sleep1222418-21ssh12:05:081
4107893230,1sleep10-21swapper/108:50:161
4050525230,1sleep20-21swapper/208:05:142
4031668230,1sleep20-21swapper/207:50:152
4012410230,1sleep00-21swapper/007:35:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional