You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 06:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Congatec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Mon Nov 24, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
40495159912161,1211cyclictest329592sleep321:25:013
40495159912160,1212cyclictest3396692sleep300:34:563
40495159912160,1212cyclictest2752192sleep323:55:013
190898240,8rtkit-daemon235610-21sh23:30:152
4049515992219,1cyclictest0-21swapper/321:55:183
190898200,1rtkit-daemon0-21swapper/119:05:341
190898190,5rtkit-daemon321103-21ntp_states00:20:200
404951599187,7cyclictest267013-21turbostat23:49:593
404951599186,9cyclictest283058-21turbostat00:04:553
404951599186,8cyclictest4139478-21turbostat20:20:003
4049515991714,1cyclictest0-21swapper/323:20:013
4049515991714,1cyclictest0-21swapper/320:55:013
404951599171,13cyclictest24761-21taskset21:20:013
404951599171,11cyclictest251465-21taskset23:40:143
404951599166,7cyclictest341197-21taskset00:35:163
4049515991613,1cyclictest0-21swapper/319:45:003
2024502161,6sleep2202470-21/usr/sbin/munin23:10:012
404951599155,6cyclictest40835922sleep319:35:163
404951599151,9cyclictest2672832sleep323:50:133
4049515991513,1cyclictest0-21swapper/319:15:033
4049515991513,1cyclictest0-21swapper/300:27:033
4049515991513,0cyclictest0-21swapper/321:07:003
4049515991513,0cyclictest0-21swapper/300:10:203
4049515991512,1cyclictest0-21swapper/322:50:183
4049515991512,1cyclictest0-21swapper/321:00:063
4049515991512,1cyclictest0-21swapper/320:55:223
4049515991512,1cyclictest0-21swapper/320:37:313
4049515991512,1cyclictest0-21swapper/319:12:183
404951599151,11cyclictest57147-21chrt21:40:003
404951599151,11cyclictest129654-21turbostat22:25:003
404951599151,11cyclictest105363-21turbostat22:09:593
404951599151,10cyclictest159879-21taskset22:40:223
404951599150,7cyclictest0-21swapper/323:10:143
190898150,10rtkit-daemon4139770-21apt-get20:20:141
190898150,10rtkit-daemon4139770-21apt-get20:20:141
715732140,7sleep171581-21ssh21:46:411
404951599146,5cyclictest1445442sleep322:32:143
404951599145,6cyclictest415962sleep321:30:163
404951599145,6cyclictest3140682sleep300:17:143
404951599141,9cyclictest40957222sleep319:45:143
4049515991412,1cyclictest0-21swapper/321:21:033
4049515991412,1cyclictest0-21swapper/319:23:023
4049515991412,0cyclictest0-21swapper/323:36:303
4049515991412,0cyclictest0-21swapper/323:07:153
4049515991412,0cyclictest0-21swapper/320:21:013
4049515991412,0cyclictest0-21swapper/320:21:013
4049515991411,1cyclictest0-21swapper/323:01:143
4049515991411,1cyclictest0-21swapper/322:19:163
4049515991411,1cyclictest0-21swapper/321:47:553
4049515991411,1cyclictest0-21swapper/319:25:163
404951599141,10cyclictest81064-21turbostat21:54:593
404951599141,10cyclictest4176945-21taskset20:47:143
404951599141,10cyclictest4154644-21taskset20:30:163
404951599141,10cyclictest4114098-21taskset19:57:313
404951599141,10cyclictest4079378-21chrt19:30:173
404951599141,10cyclictest297360-21taskset00:05:273
404951599141,10cyclictest184133-21taskset22:55:303
404951599141,10cyclictest16056-21taskset21:13:323
41082952133,5sleep14108328-21apt-get19:55:141
404951599134,6cyclictest1357362sleep322:25:383
404951599133,7cyclictest148382-21taskset22:35:183
404951599133,7cyclictest148382-21taskset22:35:183
404951599131,9cyclictest49562-21taskset21:35:163
404951599131,9cyclictest4167606-21taskset20:40:183
404951599131,9cyclictest4132715-21taskset20:10:243
404951599131,9cyclictest4120904-21taskset20:05:123
404951599131,9cyclictest4107684-21chrt19:50:263
404951599131,9cyclictest315754-21chrt00:20:013
404951599131,9cyclictest235131-21taskset23:30:043
4049515991312,0cyclictest0-21swapper/322:02:143
4049515991311,1cyclictest0-21swapper/322:46:433
4049515991311,0cyclictest0-21swapper/323:20:123
404951599130,10cyclictest4151808-21chrt20:27:443
818722121,8sleep082871-21/usr/sbin/munin21:55:160
41526852121,2sleep10-21swapper/120:30:151
404951599121,9cyclictest4120307-21taskset20:01:033
4049515991210,1cyclictest0-21swapper/323:26:183
404951599120,9cyclictest105634-21taskset22:10:103
40493822121,7sleep04049394-21munin-run19:10:010
2437102120,4sleep0244709-21irqstats23:35:150
2185102121,8sleep0218535-21munin-run23:20:010
190898120,7rtkit-daemon346494-21ssh00:37:040
190898120,5rtkit-daemon1928-21sendmail-mta23:21:311
1622292121,8sleep2162392-21diskstats22:45:122
190898110,4rtkit-daemon338355-21ssh00:31:540
190898110,3rtkit-daemon4055945-21munin-node23:16:571
142132111,7sleep014270-21/usr/sbin/munin21:10:210
1374522111,7sleep0137460-21ssh22:29:310
493552100,4sleep249559-21hddtemp_smartct21:35:162
3473942100,7sleep2347402-21ssh00:39:092
327022101,5sleep132729-21munin-run21:25:011
3079392100,7sleep1308119-21sed00:15:191
26332101,4sleep12666-21apt-get21:05:131
2042392101,7sleep0206071-21grep23:10:180
190898100,6rtkit-daemon667-21snmpd19:12:091
190898100,5rtkit-daemon324283-21hddtemp_smartct00:25:160
190898100,4rtkit-daemon421-21avahi-daemon23:02:081
190898100,4rtkit-daemon15101-21ssh21:11:151
73358291,4sleep073391-21apt-get21:50:120
4100985290,6sleep14101106-21cat19:45:241
4058424290,6sleep14058674-21sed19:15:151
4058223290,6sleep04058423-21hddtemp_smartct19:15:140
40495159997,1cyclictest0-21swapper/319:10:003
39314290,6sleep039322-21ssh21:26:360
339687291,5sleep2339699-21sh00:35:002
290756290,1sleep00-21swapper/000:04:170
233751291,6sleep2233757-21ssh23:28:042
23341296,1sleep00-21swapper/021:16:490
220047290,6sleep0221093-21grep23:20:210
19089890,6rtkit-daemon667-21snmpd21:03:561
19089890,6rtkit-daemon4055945-21munin-node00:09:330
19089890,5rtkit-daemon33537-21cat21:25:171
19089890,5rtkit-daemon319-21systemd-udevd20:09:381
19089890,5rtkit-daemon1928-21sendmail-mta21:39:410
19089890,5rtkit-daemon112413-21ssh22:12:371
19089890,4rtkit-daemon50734-21cat21:35:181
19089890,4rtkit-daemon4055945-21munin-node19:35:171
19089890,4rtkit-daemon273942-21ssh23:52:250
19089890,4rtkit-daemon113668-21ssh22:15:021
19089890,3rtkit-daemon4055945-21munin-node00:23:211
19089890,3rtkit-daemon313343-21ssh00:15:310
19089890,1rtkit-daemon0-21swapper/222:09:412
154276291,6sleep0154299-21/usr/sbin/munin22:40:130
136228291,5sleep1136236-21ssh22:26:401
132282290,7sleep2134326-21grep22:25:212
13068291,5sleep214060-21memory21:10:192
129628290,6sleep2129854-21latency_hist22:25:012
95415282,1sleep2171rcu_preempt22:00:482
81424281,4sleep181579-21df_inode21:55:141
62649280,1sleep062804-21timerwakeupswit21:40:230
57224281,4sleep157267-21ssh21:40:111
48516280,1sleep20-21swapper/221:34:152
4191668281,4sleep04192682-21grep21:00:190
4172059281,4sleep04172927-21tr20:45:170
4160542280,6sleep04162605-21tr20:35:180
4113611281,4sleep04113848-21unixbench_multi19:55:250
4108331281,4sleep24108482-21df19:55:152
4102033281,5sleep24102125-21tr19:50:142
4101997281,3sleep14102030-21apt-get19:50:131
4089425281,3sleep04089458-21apt-get19:40:140
261441280,5sleep0263457-21cut23:45:170
193006281,5sleep2193012-21ssh23:01:472
19089880,5rtkit-daemon4055945-21munin-node20:29:001
19089880,5rtkit-daemon0-21swapper/120:15:191
19089880,4rtkit-daemon96624-21ssh22:03:370
19089880,4rtkit-daemon671-21ntpd19:22:251
19089880,4rtkit-daemon667-21snmpd00:12:560
19089880,4rtkit-daemon46739-21ssh21:30:231
19089880,4rtkit-daemon438-21wpa_supplicant20:51:181
19089880,4rtkit-daemon436-21systemd-logind23:56:370
19089880,4rtkit-daemon424-21NetworkManager20:49:061
19089880,4rtkit-daemon4070780-21df19:25:151
19089880,4rtkit-daemon4055945-21munin-node22:30:032
19089880,4rtkit-daemon4055945-21munin-node20:36:031
19089880,4rtkit-daemon1928-21sendmail-mta21:46:262
19089880,3rtkit-daemon202163-21ssh23:09:230
19089880,1rtkit-daemon0-21swapper/121:16:301
19089880,1rtkit-daemon0-21swapper/023:40:370
184902281,5sleep0667-21snmpd22:57:200
178358281,3sleep1178395-21apt-get22:55:121
170910280,6sleep2171819-21tr22:50:172
162196281,4sleep1162222-21apt-get22:45:111
146045281,4sleep1146080-21apt-get22:35:131
146045281,4sleep1146080-21apt-get22:35:131
104465280,1sleep00-21swapper/022:07:560
97237270,4sleep197246-21cron22:05:001
4189295270,4sleep04189556-21unixbench_singl20:55:230
4169763270,1sleep00-21swapper/020:40:200
4153542270,5sleep24154643-21cat20:30:162
4139458270,5sleep04139578-21latency_hist20:20:000
4119711271,3sleep04119775-21ntp_states20:00:200
4082183270,1sleep00-21swapper/019:30:240
4076872270,3sleep14076905-21apt-get19:30:131
4072137270,5sleep04074160-21tr19:25:190
334384271,3sleep1336465-21kthreadcore00:30:181
30341270,4sleep030472-21python321:20:230
297514270,1sleep20-21swapper/200:05:392
284367270,4sleep1285510-21grep00:00:171
280345270,1sleep10-21swapper/123:55:201
25048270,4sleep225085-21apt-get21:20:132
242901270,4sleep2243149-21ssh23:35:012
234178270,4sleep1234187-21ssh23:29:071
194280271,4sleep0194288-21ssh23:04:460
19089870,3rtkit-daemon671-21ntpd21:50:042
19089870,3rtkit-daemon667-21snmpd20:10:201
19089870,3rtkit-daemon667-21snmpd19:40:501
19089870,3rtkit-daemon4165063-21entropy20:40:151
19089870,3rtkit-daemon4055945-21munin-node22:40:071
19089870,3rtkit-daemon4055945-21munin-node22:23:391
19089870,3rtkit-daemon4055945-21munin-node20:01:191
19089870,3rtkit-daemon201074-21ssh23:06:451
19089870,3rtkit-daemon175980-21tail22:50:241
19089870,2rtkit-daemon4187223-21cut20:55:171
176574270,4sleep0176582-21ssh22:51:390
159626270,5sleep2159878-21/usr/sbin/munin22:40:222
146000270,1sleep20-21swapper/222:35:062
146000270,1sleep20-21swapper/222:35:062
128689270,4sleep0128697-21ssh22:22:540
112022270,5sleep08666-21diskmemload22:11:480
81580260,4sleep281705-21perl21:55:152
7968261,2sleep00-21swapper/021:05:240
42598262,2sleep00-21swapper/021:30:170
4108038261,2sleep00-21swapper/019:55:000
4095758260,4sleep04095915-21diskstats19:45:150
4088820260,1sleep20-21swapper/219:35:512
4083159260,4sleep04083311-21df_inode19:35:120
4078278261,3sleep24079377-21cat19:30:182
4069039260,4sleep24064332-21/usr/sbin/munin19:20:212
4058675261,2sleep20-21swapper/219:15:162
323874261,2sleep20-21swapper/200:25:142
323863260,4sleep1323873-21acpi00:25:131
314196260,1sleep20-21swapper/200:17:412
289213260,1sleep20-21swapper/200:00:372
275316260,2sleep2275340-21apt-get23:55:132
273059260,1sleep10-21swapper/123:50:271
257727260,1sleep10-21swapper/123:42:061
244710261,2sleep1245775-21ssh23:35:151
241250260,1sleep10-21swapper/123:31:081
232797260,3sleep0232805-21ssh23:25:460
224787260,4sleep2224793-21ssh23:20:502
218395260,1sleep20-21swapper/223:19:462
19089860,3rtkit-daemon667-21snmpd22:10:402
19089860,3rtkit-daemon473-21gmain23:11:381
19089860,3rtkit-daemon1928-21sendmail-mta21:44:012
16972261,2sleep20-21swapper/221:15:122
162178260,3sleep0162195-21basename22:45:100
152000260,1sleep00-21swapper/022:35:530
152000260,1sleep00-21swapper/022:35:520
143531260,4sleep0143724-21users22:30:240
139146260,4sleep1140262-21cut22:30:171
120886260,1sleep20-21swapper/222:18:352
113756261,2sleep00-21swapper/022:15:130
74845250,3sleep176074-21kthreadcore21:50:171
73016251,2sleep00-21swapper/021:50:000
4183945250,1sleep20-21swapper/220:55:092
4178872250,3sleep04180271-21cut20:50:180
4176989250,3sleep20-21swapper/220:47:502
4158112250,1sleep00-21swapper/020:32:470
4147292251,2sleep00-21swapper/020:25:180
4145479250,1sleep00-21swapper/020:21:450
4145479250,1sleep00-21swapper/020:21:450
4139773251,2sleep24139858-21cpuspeed_turbos20:20:152
4139773251,2sleep24139858-21cpuspeed_turbos20:20:142
4139257250,1sleep20-21swapper/220:17:262
4132337250,2sleep04132384-21ps20:10:230
4126886250,1sleep20-21swapper/220:09:412
4126717250,1sleep00-21swapper/020:07:310
4091619250,2sleep20-21swapper/219:40:202
4055498250,1sleep20-21swapper/219:12:312
4043458250,2sleep24043422-21/usr/sbin/munin19:05:132
39232250,1sleep20-21swapper/221:26:212
345952250,1sleep10-21swapper/100:35:471
315446250,2sleep20-21swapper/200:20:012
306268250,3sleep10-21swapper/100:12:511
299249250,0sleep10-21swapper/100:09:511
266838250,3sleep2266845-21sh23:49:362
185393250,1sleep20-21swapper/222:58:312
104505250,3sleep1104513-21ssh22:08:011
4166319240,1sleep20-21swapper/220:40:182
4101685240,3sleep20-21swapper/219:49:172
4055541240,3sleep00-21swapper/019:13:050
4043241,1sleep20-21swapper/221:05:182
306693240,1sleep20-21swapper/200:13:522
267580240,2sleep2267762-21ssh23:50:162
253445240,2sleep20-21swapper/223:40:182
2238240,3sleep20-21swapper/221:03:132
208190240,2sleep20-21swapper/223:10:252
4182335230,1sleep20-21swapper/220:50:202
4158543230,1sleep20-21swapper/220:35:132
4151750230,1sleep20-21swapper/220:27:002
4127167230,1sleep20-21swapper/220:10:132
4114766230,1sleep20-21swapper/220:00:152
4070991230,1sleep20-21swapper/219:25:172
4065792230,1sleep00-21swapper/019:20:190
265152230,1sleep10-21swapper/123:45:341
235105230,1sleep00-21swapper/023:30:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional