You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 21:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Wed Nov 19, 2025 12:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
40204139912650,1261cyclictest1181552sleep310:35:003
40204139912110,1207cyclictest2964712sleep312:24:563
40204139912050,1202cyclictest40205542sleep307:10:013
40204139911920,1188cyclictest46232sleep309:24:563
40204139911350,1131cyclictest1178872sleep310:34:573
40204139911091,1103cyclictest4035102-21sleep07:20:003
4020413992990,296cyclictest231774-21turbostat11:49:553
4020413992118,1cyclictest0-21swapper/311:25:003
4020413992017,1cyclictest0-21swapper/309:40:013
402041399197,8cyclictest304559-21turbostat12:30:003
402041399188,7cyclictest2406292sleep311:50:193
4020413991815,1cyclictest0-21swapper/311:10:273
402041399164,9cyclictest4174485-21taskset09:10:133
4020413991613,1cyclictest0-21swapper/312:10:153
4020413991613,1cyclictest0-21swapper/309:40:223
4020413991613,1cyclictest0-21swapper/308:35:173
633402151,11sleep265343-21grep10:00:182
402041399158,4cyclictest85601-21turbostat10:15:003
402041399158,4cyclictest189419-21taskset11:15:443
402041399156,6cyclictest2543972sleep311:56:123
4020413991513,1cyclictest0-21swapper/309:31:143
4020413991513,1cyclictest0-21swapper/309:31:143
4020413991512,1cyclictest0-21swapper/308:25:233
4020413991512,1cyclictest0-21swapper/308:08:163
4020413991512,1cyclictest0-21swapper/307:34:023
402041399151,11cyclictest74297-21taskset10:05:193
402041399151,11cyclictest53555-21taskset09:55:133
402041399151,11cyclictest264420-21chrt12:05:023
402041399151,11cyclictest174788-21turbostat11:10:003
402041399149,3cyclictest222434-21chrt11:37:213
402041399149,3cyclictest222434-21chrt11:37:203
402041399145,7cyclictest158095-21taskset10:58:543
4020413991412,1cyclictest0-21swapper/310:47:223
4020413991412,1cyclictest0-21swapper/310:25:393
4020413991412,1cyclictest0-21swapper/309:25:563
4020413991412,1cyclictest0-21swapper/308:12:183
4020413991412,1cyclictest0-21swapper/307:20:473
4020413991412,1cyclictest0-21swapper/307:20:463
4020413991412,0cyclictest0-21swapper/311:27:483
4020413991411,1cyclictest0-21swapper/312:39:173
4020413991411,1cyclictest0-21swapper/308:44:263
402041399141,10cyclictest99818-21chrt10:20:263
402041399141,10cyclictest4173727-21taskset09:07:573
402041399141,10cyclictest4086116-21chrt08:00:183
402041399141,10cyclictest4047155-21taskset07:25:283
402041399141,10cyclictest310334-21taskset12:30:263
402041399141,10cyclictest231330-21chrt11:44:013
402041399141,10cyclictest231330-21chrt11:44:013
402041399141,10cyclictest164412-21taskset11:00:253
402041399140,7cyclictest0-21swapper/309:50:223
402041399140,11cyclictest4189685-21chrt09:18:443
402041399140,11cyclictest149115-21chrt10:51:393
2953662140,5sleep0295368-21ssh12:22:260
190898140,2rtkit-daemon0-21swapper/208:05:062
402041399131,9cyclictest66921-21taskset10:00:223
402041399131,9cyclictest4161631-21taskset09:00:053
402041399131,9cyclictest4160538-21taskset08:55:233
402041399131,9cyclictest4148799-21taskset08:50:063
402041399131,9cyclictest4129070-21taskset08:30:513
402041399131,9cyclictest4110856-21taskset08:20:053
402041399131,9cyclictest4066364-21taskset07:41:353
402041399131,9cyclictest4059873-21taskset07:35:423
402041399131,9cyclictest37305-21taskset09:45:123
402041399131,9cyclictest261579-21taskset12:00:203
402041399131,9cyclictest213557-21taskset11:30:303
4020413991312,0cyclictest0-21swapper/312:18:483
4020413991312,0cyclictest0-21swapper/310:42:313
4020413991312,0cyclictest0-21swapper/308:18:223
4020413991310,1cyclictest0-21swapper/310:15:233
402041399130,10cyclictest4148346-21chrt08:47:253
402041399130,10cyclictest4074595-21taskset07:50:163
402041399130,10cyclictest4072880-21taskset07:49:563
40196112131,8sleep04019877-21grep07:05:270
41488062120,2sleep00-21swapper/008:50:120
40416292121,8sleep04041674-21grep07:25:140
402041399121,9cyclictest4085126-21taskset07:56:023
190898120,7rtkit-daemon4104895-21grep08:15:162
190898120,3rtkit-daemon4026811-21munin-node12:07:042
993392113,6sleep0423-21dbus-daemon10:20:250
2184012116,3sleep10-21swapper/111:35:181
2184012116,3sleep10-21swapper/111:35:181
1920262110,8sleep0192888-21sed11:20:190
1080902110,4sleep1108098-21ssh10:25:581
911282104,3sleep20-21swapper/210:15:212
711662101,6sleep172219-21kthreadcore10:05:191
612212101,6sleep061270-21awk10:00:000
41050232101,6sleep04106018-21irqstats08:15:180
2556492100,7sleep0255658-21ssh11:59:070
2476972100,7sleep1247705-21ssh11:54:261
190898100,6rtkit-daemon667-21snmpd08:00:392
190898100,6rtkit-daemon436-21systemd-logind08:51:382
190898100,6rtkit-daemon186636-21kthreadcore11:15:202
190898100,5rtkit-daemon667-21snmpd10:57:391
190898100,5rtkit-daemon4156839-21ps08:55:182
190898100,5rtkit-daemon4142685-21df08:45:132
190898100,5rtkit-daemon4072038-21ntp_states07:45:212
190898100,3rtkit-daemon421-21avahi-daemon12:39:470
190898100,3rtkit-daemon4173904-21diskmemload10:12:140
190898100,3rtkit-daemon12504-21ssh09:29:270
1835512101,6sleep10-21swapper/111:15:151
1809492101,5sleep2181145-21users11:10:282
1661212100,2sleep20-21swapper/211:03:422
1518832101,6sleep0152912-21cut10:55:180
1450482100,7sleep2147100-21tr10:50:192
60083290,6sleep260089-21ssh09:57:222
4141467291,4sleep14141633-21awk08:40:271
4091978290,4sleep04092000-21apt-get08:05:120
4040527291,4sleep24040652-21python307:20:232
4040527291,4sleep24040652-21python307:20:232
4035755291,5sleep04036468-21cut07:20:180
4035755291,5sleep04036468-21cut07:20:170
28249290,6sleep128257-21ssh09:38:291
280665290,1sleep2280768-21cpuspeed_turbos12:15:142
247498290,2sleep2247506-21ssh11:53:552
229798290,6sleep2229804-21ssh11:40:332
229798290,6sleep2229804-21ssh11:40:322
207742291,4sleep0207776-21apt-get11:30:120
19089890,5rtkit-daemon436-21systemd-logind07:51:372
19089890,5rtkit-daemon149221-21runrttasks10:51:521
19089890,4rtkit-daemon4178031-21cut09:10:192
19089890,4rtkit-daemon307999-21/usr/sbin/munin12:30:190
19089890,4rtkit-daemon19304-21ssh09:31:300
19089890,4rtkit-daemon19304-21ssh09:31:300
19089890,3rtkit-daemon4026811-21munin-node10:03:080
19089890,1rtkit-daemon0-21swapper/210:42:352
141585290,6sleep0141593-21ssh10:48:060
4647281,5sleep24670-21munin-run09:25:002
4189769281,4sleep24189773-21ssh09:19:042
4159823281,4sleep04160331-21memory08:55:210
4115944280,4sleep24116015-21ntp_states08:20:212
4054341281,4sleep14054437-21cut07:35:171
4047915281,4sleep14048058-21df07:30:171
4047879281,3sleep04047912-21apt-get07:30:150
4026640281,4sleep04026684-21munin-run07:15:010
241634280,5sleep0242789-21cut11:50:200
2355280,5sleep02534-21grep09:20:250
200789282,4sleep2201165-21kthreadcore11:25:172
19089880,5rtkit-daemon319-21systemd-udevd09:47:371
19089880,4rtkit-daemon671-21ntpd11:30:131
19089880,4rtkit-daemon667-21snmpd11:57:012
19089880,4rtkit-daemon667-21snmpd11:25:080
19089880,4rtkit-daemon667-21snmpd11:00:151
19089880,4rtkit-daemon667-21snmpd09:43:461
19089880,4rtkit-daemon438-21wpa_supplicant07:55:562
19089880,4rtkit-daemon421-21avahi-daemon10:05:030
19089880,4rtkit-daemon421-21avahi-daemon07:13:362
19089880,4rtkit-daemon4041786-21df_inode07:25:152
19089880,4rtkit-daemon4026811-21munin-node10:49:101
19089880,4rtkit-daemon4026811-21munin-node10:35:022
19089880,4rtkit-daemon4026811-21munin-node10:17:231
19089880,4rtkit-daemon4026811-21munin-node08:32:402
19089880,4rtkit-daemon302839-21ssh12:25:580
19089880,4rtkit-daemon280450-21latency_hist12:15:000
19089880,4rtkit-daemon169410-21kthreadcore11:05:211
19089880,3rtkit-daemon671-21ntpd10:20:131
19089880,3rtkit-daemon237886-21users11:45:262
19089880,1rtkit-daemon0-21swapper/212:04:482
19089880,1rtkit-daemon0-21swapper/111:58:221
19089880,1rtkit-daemon0-21swapper/109:57:191
141601280,6sleep2141609-21ssh10:48:082
12026280,6sleep20-21swapper/209:28:202
95044271,3sleep295949-21sh10:20:182
8604270,1sleep10-21swapper/109:25:191
72222270,5sleep274296-21grep10:05:202
4189291270,4sleep14189299-21ssh09:17:541
4182501271,3sleep04182540-21apt-get09:15:120
4181928270,4sleep04181932-21ssh09:14:280
4126025271,4sleep04126459-21kthreadcore08:30:210
4117152270,4sleep14117295-21df08:25:141
4015272271,3sleep14016147-21kthreadcore07:05:201
318293270,4sleep2318371-21grep12:35:262
27600270,1sleep00-21swapper/009:37:000
26356270,4sleep226427-21ntp_states09:35:222
231489270,1sleep10-21swapper/111:44:291
231489270,1sleep10-21swapper/111:44:281
224118270,4sleep0224171-21fschecks_count11:40:140
224118270,4sleep0224171-21fschecks_count11:40:140
214820270,4sleep2214828-21ssh11:33:272
207103270,5sleep1207113-21ssh11:29:181
198181270,1sleep20-21swapper/211:22:252
19089870,4rtkit-daemon81364-21packagekitd09:50:091
19089870,4rtkit-daemon674-21lldpd09:05:412
19089870,4rtkit-daemon473-21gmain08:25:092
19089870,4rtkit-daemon473-21gmain07:05:012
19089870,4rtkit-daemon421-21avahi-daemon09:00:262
19089870,4rtkit-daemon4026811-21munin-node11:39:472
19089870,4rtkit-daemon4026811-21munin-node11:39:462
19089870,4rtkit-daemon4026811-21munin-node11:10:411
19089870,4rtkit-daemon4026811-21munin-node09:14:091
19089870,4rtkit-daemon4026811-21munin-node07:30:112
19089870,3rtkit-daemon81364-21packagekitd10:30:292
19089870,3rtkit-daemon674-21lldpd08:41:412
19089870,3rtkit-daemon671-21ntpd07:15:012
19089870,3rtkit-daemon667-21snmpd12:15:140
19089870,3rtkit-daemon421-21avahi-daemon12:10:191
19089870,3rtkit-daemon4026811-21munin-node11:20:101
19089870,3rtkit-daemon4026811-21munin-node08:12:332
19089870,3rtkit-daemon4026811-21munin-node07:41:312
19089870,3rtkit-daemon4026811-21munin-node07:39:402
19089870,1rtkit-daemon0-21swapper/208:37:472
110179271,3sleep0110197-21apt-get10:30:140
103600270,5sleep0105177-21cut10:25:210
92787260,3sleep092795-21ssh10:17:550
83828260,1sleep10-21swapper/110:10:551
67840260,1sleep10-21swapper/110:01:291
4135967260,1sleep00-21swapper/008:40:100
4135509260,1sleep10-21swapper/108:37:431
4128207262,2sleep1321rcuc/108:30:221
4106019260,3sleep14107463-21cut08:15:191
4099720260,3sleep04101143-21cut08:10:170
4085081260,1sleep10-21swapper/107:55:281
4069023261,2sleep10-21swapper/107:45:191
4060589260,3sleep04060622-21apt-get07:40:120
4036470261,3sleep14037703-21kthreadcore07:20:181
4036470261,3sleep14037703-21kthreadcore07:20:181
4020568260,3sleep10-21swapper/107:10:131
4014490260,3sleep30-21swapper/307:05:183
35162260,1sleep00-21swapper/009:40:500
317422260,1sleep10-21swapper/112:35:221
298300260,4sleep1298337-21kthreadcore12:25:191
296799260,1sleep20-21swapper/212:25:082
296495260,3sleep2296511-21grep12:25:012
295225260,1sleep10-21swapper/112:22:041
280629261,2sleep1280662-21apt-get12:15:131
280140260,1sleep20-21swapper/212:14:342
271823260,1sleep00-21swapper/012:09:030
264436260,1sleep10-21swapper/112:05:081
262729260,3sleep0262737-21ssh12:01:420
238340260,1sleep00-21swapper/011:46:310
217288260,3sleep0218400-21cut11:35:180
217288260,3sleep0218400-21cut11:35:170
20023260,1sleep10-21swapper/109:33:171
20023260,1sleep10-21swapper/109:33:161
19089860,3rtkit-daemon674-21lldpd09:40:402
183929260,3sleep0184261-21irqcore11:15:180
181217260,3sleep0181223-21ssh11:10:330
167062261,2sleep00-21swapper/011:05:160
142775261,2sleep0142812-21apt-get10:50:130
132716260,3sleep1132722-21ssh10:41:531
126263260,1sleep00-21swapper/010:40:030
125965262,2sleep0151ktimers/010:39:590
116608260,1sleep10-21swapper/110:31:591
52218250,1sleep00-21swapper/009:52:510
51485250,3sleep20-21swapper/209:51:062
4168083250,1sleep10-21swapper/109:05:161
4161832251,2sleep00-21swapper/009:00:180
4155768250,3sleep14156519-21irqrtprio08:55:181
4150263251,2sleep10-21swapper/108:50:181
4148492250,1sleep00-21swapper/008:49:160
4135376250,1sleep00-21swapper/008:35:580
4122163250,3sleep00-21swapper/008:25:210
4116861250,2sleep10-21swapper/108:25:011
4104106250,1sleep10-21swapper/108:12:281
4091403250,1sleep00-21swapper/008:01:050
4081522250,3sleep04083642-21grep07:55:190
4067877251,2sleep00-21swapper/007:45:180
4060580250,1sleep10-21swapper/107:40:051
4043305250,3sleep14045371-21grep07:25:191
4029394251,2sleep10-21swapper/107:15:171
38647251,2sleep20-21swapper/209:45:202
312171250,3sleep20-21swapper/212:33:502
305095250,3sleep14173904-21diskmemload12:30:161
233960251,2sleep10-21swapper/111:45:201
19089850,1rtkit-daemon0-21swapper/210:29:402
174129250,2sleep2174136-21sh11:08:342
166694251,2sleep00-21swapper/011:05:010
125428251,2sleep1125436-21ssh10:38:441
85334240,2sleep20-21swapper/210:14:292
43544240,3sleep00-21swapper/009:46:300
4169351240,2sleep00-21swapper/009:05:190
4148314240,3sleep10-21swapper/108:47:001
4116802240,3sleep00-21swapper/008:24:160
4055609240,2sleep00-21swapper/007:35:200
4034729240,3sleep00-21swapper/007:15:440
3697240,2sleep13705-21ssh09:22:421
262279240,2sleep1262287-21ssh12:00:391
13480240,2sleep20-21swapper/209:30:182
13480240,2sleep20-21swapper/209:30:172
4166542230,1sleep10-21swapper/109:00:231
4092003230,1sleep10-21swapper/108:05:131
4087620230,1sleep10-21swapper/108:00:211
4078060230,1sleep10-21swapper/107:50:191
4073327230,1sleep00-21swapper/007:50:140
150529230,1sleep20-21swapper/210:55:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional