You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 12:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Nov 17, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
260302240164,25sleep10-21swapper/119:08:331
258122237161,24sleep30-21swapper/319:05:453
258432234164,57sleep20-21swapper/219:06:082
258142203164,25sleep00-21swapper/019:05:460
3147021520,4sleep22624999cyclictest20:30:272
2211721260,2sleep32625499cyclictest21:25:153
26249999389,3cyclictest0-21swapper/222:13:122
26249998480,3cyclictest0-21swapper/222:32:082
26249998077,2cyclictest0-21swapper/220:50:582
154502480,2sleep10-21swapper/121:10:161
26238993910,14cyclictest9935-21munin-run19:45:010
26249993633,2cyclictest0-21swapper/222:48:342
2624599361,15cyclictest20561-21diskstats22:35:151
98752340,2sleep30-21swapper/320:55:253
98752340,2sleep30-21swapper/320:55:253
2624599342,20cyclictest12637-21gunzip22:15:241
2624599331,16cyclictest0-21swapper/100:25:341
2624599331,15cyclictest0-21swapper/100:03:021
2624599321,9cyclictest0-21swapper/120:20:141
2624599321,14cyclictest0-21swapper/119:21:481
2624599320,14cyclictest0-21swapper/121:41:301
2623899323,13cyclictest19359-21munin-run21:20:010
2624599311,12cyclictest0-21swapper/123:29:101
2624599311,12cyclictest0-21swapper/122:00:281
2624599311,12cyclictest0-21swapper/122:00:281
2624599311,12cyclictest0-21swapper/121:02:561
2624599302,11cyclictest4179-21memory23:10:221
26245993017,3cyclictest29696-40dnf20:29:501
2624599301,15cyclictest0-21swapper/120:52:061
2624599301,12cyclictest0-21swapper/123:48:561
2624599301,12cyclictest0-21swapper/121:17:241
2624599300,14cyclictest0-21swapper/123:06:021
66512290,1sleep20-21swapper/223:15:262
270392290,4sleep227042-21latency21:35:222
2624999290,17cyclictest86550irq/126-enp1s0-21:34:022
2624599292,17cyclictest27775-21users21:35:301
26245992917,8cyclictest0-21swapper/121:25:261
26245992917,3cyclictest7772-40dnf00:31:181
2624599291,5cyclictest0-21swapper/123:23:581
2624599291,12cyclictest0-21swapper/122:22:041
2624599291,12cyclictest0-21swapper/120:42:321
2624599291,12cyclictest0-21swapper/100:10:101
2624599291,11cyclictest0-21swapper/120:49:381
2624599291,10cyclictest0-21swapper/122:49:061
2624599290,12cyclictest0-21swapper/120:15:141
26238992919,2cyclictest28631-21kworker/0:223:01:160
26238992912,15cyclictest31309-21awk19:20:010
26245992812,3cyclictest0-21swapper/123:32:141
2624599281,14cyclictest0-21swapper/100:15:261
2624599281,13cyclictest0-21swapper/121:59:261
2624599281,12cyclictest0-21swapper/121:30:561
2624599281,11cyclictest22787-21sshd23:54:501
2624599281,11cyclictest0-21swapper/123:44:301
2624599281,11cyclictest0-21swapper/100:37:021
2624599281,10cyclictest0-21swapper/119:55:201
2624599280,11cyclictest0-21swapper/121:23:021
2624599280,10cyclictest0-21swapper/122:43:461
2623899283,8cyclictest28631-21kworker/0:223:05:200
2623899282,15cyclictest27930-21sed22:50:250
2623899281,12cyclictest16239-21awk23:40:020
2624599271,12cyclictest0-21swapper/122:32:231
2624599271,11cyclictest0-21swapper/123:38:421
2624599270,12cyclictest0-21swapper/122:09:241
2624599270,11cyclictest0-21swapper/121:53:021
2623899274,13cyclictest25946-21sh21:35:010
2623899271,9cyclictest1230-21kworker/0:119:50:270
2623899271,9cyclictest1230-21kworker/0:119:50:270
2624599261,8cyclictest0-21swapper/122:53:261
2624599261,13cyclictest0-21swapper/120:00:241
26245992610,3cyclictest0-21swapper/119:53:201
26245992610,3cyclictest0-21swapper/119:53:201
2624599260,12cyclictest0-21swapper/123:01:061
2624599260,11cyclictest0-21swapper/122:10:381
2623899267,6cyclictest0-21swapper/023:27:580
2623899265,8cyclictest7656-21/usr/sbin/munin23:20:180
2623899265,14cyclictest24589-21unixbench_singl23:55:300
2623899261,13cyclictest22830-21ntpq21:25:230
2623899261,12cyclictest31647-21munin-run00:15:010
2624599251,6cyclictest0-21swapper/122:58:381
26245992515,3cyclictest0-21swapper/100:20:241
2624599251,13cyclictest0-21swapper/122:29:301
2624599251,10cyclictest0-21swapper/119:19:481
26245992510,3cyclictest21963-21irqstats20:10:201
26245992510,3cyclictest0-21swapper/119:39:121
2623899259,2cyclictest28386-21/usr/sbin/munin21:40:220
2623899257,15cyclictest6809-21awk22:05:010
2623899257,15cyclictest6809-21awk22:05:010
2623899254,11cyclictest26698-21sort00:00:270
277522240,2sleep20-21swapper/220:25:142
2624599247,8cyclictest0-21swapper/119:48:421
2624599241,7cyclictest0-21swapper/121:07:101
2624599241,15cyclictest0-21swapper/123:18:161
2624599240,17cyclictest0-21swapper/120:37:491
2624599240,14cyclictest0-21swapper/123:59:381
2623899249,14cyclictest0-21swapper/022:00:010
2623899247,15cyclictest31938-21idleruntime-cro20:35:000
2623899243,10cyclictest0-21swapper/000:15:180
2624599238,3cyclictest0-21swapper/119:25:401
2624599231,19cyclictest18987-21cpu20:05:161
2624599231,14cyclictest0-21swapper/121:47:501
2624599231,14cyclictest0-21swapper/120:34:221
2624599231,11cyclictest0-21swapper/119:31:361
2624599230,9cyclictest0-21swapper/100:07:221
2624599230,9cyclictest0-21swapper/100:07:221
2623899234,9cyclictest0-21swapper/022:05:170
2623899234,6cyclictest20518-21missed_timers21:20:230
2623899234,16cyclictest25603-21cut22:45:200
2623899233,9cyclictest0-21swapper/022:25:160
2623899233,17cyclictest10778-21/usr/sbin/munin21:00:250
2623899232,9cyclictest9906-21vmstat00:35:290
2623899232,8cyclictest22495-21kworker/0:023:52:340
2623899232,11cyclictest5600-21kworker/0:100:28:200
2623899231,12cyclictest30259-21kworker/u8:022:25:010
2623899231,11cyclictest0-21swapper/022:10:180
2623899231,10cyclictest25243-21kworker/u8:200:20:220
173052231,2sleep317408-21vmstat22:25:303
2625499226,3cyclictest0-21swapper/320:49:103
26254992216,3cyclictest28283-21grep20:25:173
2625499221,5cyclictest15567-21kworker/u8:122:37:463
2625499221,5cyclictest0-21swapper/321:43:243
2624999221,2cyclictest0-21swapper/200:35:202
2624599227,3cyclictest0-21swapper/119:10:581
2623899229,4cyclictest18367-21diskstats22:30:180
2623899229,2cyclictest31652-21kworker/0:120:36:550
2623899226,3cyclictest1527-21kworker/0:020:40:170
2623899224,12cyclictest3820-21perl23:10:180
2623899223,8cyclictest88050irq/133-enp2s0-21:07:280
2623899223,12cyclictest0-21swapper/021:36:450
2623899222,7cyclictest16009-21kworker/0:123:43:240
2623899222,17cyclictest6784-21netstat19:35:220
2623899221,9cyclictest28302-21kworker/0:319:26:100
2623899221,9cyclictest18409-21kworker/0:020:06:500
26238992211,8cyclictest13677-40dnf23:30:500
2623899221,12cyclictest1527-21kworker/0:020:58:490
2623899221,12cyclictest1527-21kworker/0:020:58:490
2623899221,10cyclictest29420-21kworker/0:220:29:550
26238992210,3cyclictest4186-21/usr/sbin/munin20:45:250
26238992210,2cyclictest21939-21kworker/0:222:40:080
29692211,3sleep33031-21fschecks_count21:55:193
2625499212,8cyclictest8081-21munin-run00:35:013
2625499212,11cyclictest25252-21/usr/sbin/munin00:00:193
2625499211,5cyclictest0-21swapper/300:10:023
2625499211,5cyclictest0-21swapper/300:10:023
2624999211,8cyclictest0-21swapper/219:47:202
2624999211,6cyclictest0-21swapper/222:23:072
26249992110,3cyclictest5002-21cstates22:00:162
26249992110,3cyclictest5002-21cstates22:00:162
2624599211,15cyclictest0-21swapper/120:56:121
2624599211,15cyclictest0-21swapper/120:56:121
2624599211,11cyclictest0-21swapper/119:42:041
2623899217,6cyclictest6385-21/usr/sbin/munin20:50:210
2623899217,11cyclictest673-21df21:50:140
2623899215,13cyclictest10153-21/usr/sbin/munin19:45:140
2623899212,6cyclictest30587-21/usr/sbin/munin21:45:280
2623899211,8cyclictest13490-21sort22:20:020
26238992114,4cyclictest21323-21grep22:35:250
26238992112,6cyclictest28167-21idleruntime00:05:190
26238992112,6cyclictest28167-21idleruntime00:05:190
2623899211,11cyclictest31524-21/usr/sbin/munin19:20:110
2623899211,11cyclictest21151-21/usr/sbin/munin20:10:220
317822200,3sleep2311ktimersoftd/221:45:252
2625499209,3cyclictest31216-21/usr/sbin/munin23:00:213
2625499209,3cyclictest15192-21/usr/sbin/munin21:10:263
2625499209,3cyclictest12050-21/usr/sbin/munin23:30:223
2625499201,6cyclictest0-21swapper/300:26:303
2625499201,5cyclictest0-21swapper/322:00:103
2625499201,5cyclictest0-21swapper/322:00:103
2625499201,4cyclictest0-21swapper/323:51:213
2625499201,11cyclictest8651-21sshd22:05:493
2625499201,11cyclictest8307-21/usr/sbin/munin00:35:153
2625499201,11cyclictest14634-21snmpd20:40:493
2624999206,3cyclictest5030-21sshd23:12:352
2624999201,3cyclictest0-21swapper/222:53:052
2624999201,12cyclictest0-21swapper/221:40:572
2623899205,12cyclictest19754-21runrttasks23:45:270
2623899201,6cyclictest0-21swapper/023:18:420
26238992011,5cyclictest7177-21ntp_states00:30:260
2625499195,4cyclictest30992-21perf23:00:013
2625499191,10cyclictest23970-40dnf21:30:133
2625499191,10cyclictest14634-21snmpd23:56:223
2624999199,2cyclictest86750irq/128-enp1s0-23:25:222
2624999197,3cyclictest19821-21ps23:45:282
2624999195,8cyclictest7772-40dnf00:31:192
2624999191,8cyclictest0-21swapper/221:20:212
2624999191,7cyclictest689-21/usr/sbin/munin00:15:262
2624999191,3cyclictest0-21swapper/223:53:132
2624999191,15cyclictest24385-21basename22:45:012
2623899194,3cyclictest1230-21kworker/0:119:32:340
2623899191,7cyclictest241902latency20:15:180
26238991911,5cyclictest27699-21smart_sda19:10:280
2623899191,15cyclictest14548-21/usr/sbin/munin19:55:180
2623899191,10cyclictest25548-21/usr/sbin/munin20:20:200
2623899191,10cyclictest0-21swapper/021:10:030
26238991910,6cyclictest29385-21/usr/sbin/munin22:55:170
242442191,2sleep30-21swapper/322:43:243
163382191,2sleep20-21swapper/219:57:522
14292191,3sleep30-21swapper/320:36:153
95922181,2sleep30-21swapper/319:40:553
79942181,3sleep30-21swapper/320:50:363
35552181,2sleep20-21swapper/219:30:102
33422181,3sleep30-21swapper/300:20:513
282482181,2sleep20-21swapper/219:13:082
2625499187,4cyclictest86550irq/126-enp1s0-21:10:013
2625499183,8cyclictest86550irq/126-enp1s0-21:35:193
2625499181,7cyclictest8004-21diskstats23:20:153
2625499181,7cyclictest32544-21cut00:15:183
2625499181,6cyclictest26935-21grep22:50:143
26254991813,3cyclictest30345-21tail00:10:203
26254991812,3cyclictest28426-21munin-setwarn19:15:003
26254991812,3cyclictest25917-21grep20:20:163
2625499181,16cyclictest0-21swapper/320:09:473
2625499181,12cyclictest86550irq/126-enp1s0-19:48:473
2625499181,11cyclictest14634-21snmpd22:10:493
2625499181,10cyclictest14634-21snmpd19:20:343
2624999184,6cyclictest88150irq/134-enp2s0-20:05:302
2624999182,7cyclictest30-21rcuc/221:25:182
2624999181,8cyclictest32125-21grep19:20:152
2624999181,6cyclictest3467-21sed20:40:282
2624999181,6cyclictest14828-21kworker/2:120:35:262
2624999181,6cyclictest0-21swapper/219:25:162
2624999181,5cyclictest0-21swapper/222:37:532
26249991813,3cyclictest6929-21wc19:35:252
26249991812,3cyclictest17168-21cut20:00:182
2624999181,15cyclictest0-21swapper/221:55:042
2624999181,10cyclictest0-21swapper/221:51:312
198122180,2sleep30-21swapper/322:32:563
153052180,2sleep30-21swapper/322:21:323
74332171,2sleep30-21swapper/319:36:263
311302171,3sleep20-21swapper/219:18:022
2625499175,2cyclictest0-21swapper/322:15:163
2625499174,4cyclictest30846-21expr21:45:153
2625499173,5cyclictest0-21swapper/323:48:223
2625499171,6cyclictest14666-21grep23:35:163
2625499171,6cyclictest0-21swapper/323:11:023
2625499171,6cyclictest0-21swapper/323:05:173
2625499171,6cyclictest0-21swapper/319:30:243
2625499171,5cyclictest2395-21munin-run21:55:013
26254991712,3cyclictest836-21runrttasks19:55:203
26254991712,3cyclictest29954-21/usr/sbin/munin20:30:213
26254991712,3cyclictest22614-21grep20:10:273
26254991712,2cyclictest9851-21/usr/sbin/munin23:25:163
26254991711,3cyclictest24051-21idleruntime20:15:183
26254991711,3cyclictest16933-21fschecks_time23:40:163
2625499171,13cyclictest1340-21/usr/sbin/munin19:25:183
2625499171,10cyclictest0-21swapper/321:00:533
2624999175,3cyclictest12050irq/123-ahci[0023:00:012
2624999172,12cyclictest246082sleep220:15:262
2624999171,9cyclictest0-21swapper/222:25:042
2624999171,7cyclictest7676-21cpu23:20:132
2624999171,6cyclictest0-21swapper/223:42:552
2624999171,6cyclictest0-21swapper/223:07:292
26249991713,3cyclictest0-21swapper/223:32:582
2624999171,2cyclictest0-21swapper/220:45:072
26249991712,3cyclictest12593-21processes22:15:232
2624999171,15cyclictest0-21swapper/223:39:202
2624999171,15cyclictest0-21swapper/221:01:502
26249991711,3cyclictest30348-21gltestperf00:10:202
2624999171,13cyclictest28090-21grep00:05:182
2624999171,13cyclictest28090-21grep00:05:182
262392171,2sleep30-21swapper/322:45:453
26238991710,4cyclictest17660-21memory20:00:210
246682171,2sleep20-21swapper/223:55:492
214682171,3sleep30-21swapper/321:24:083
2625499161,9cyclictest0-21swapper/321:15:173
2625499161,6cyclictest5460-21/usr/sbin/munin23:15:223
26254991611,3cyclictest836-21runrttasks20:03:583
26254991611,3cyclictest14067-21systemd-journal19:51:553
26254991611,3cyclictest14067-21systemd-journal19:51:553
26254991610,3cyclictest30032-21basename19:15:173
2624999165,3cyclictest86850irq/129-enp1s0-20:20:242
2624999165,3cyclictest14958-21diskmemload22:06:112
2624999162,6cyclictest15192-21/usr/sbin/munin21:10:222
2624999161,8cyclictest0-21swapper/200:20:062
2624999161,5cyclictest0-21swapper/221:15:202
2624999161,5cyclictest0-21swapper/200:02:482
26249991612,3cyclictest0-21swapper/219:43:272
26249991612,3cyclictest0-21swapper/200:29:372
2624999161,14cyclictest0-21swapper/223:02:572
2624999161,14cyclictest0-21swapper/220:14:122
26249991611,3cyclictest9635-21ntp_offset20:55:212
26249991611,3cyclictest9635-21ntp_offset20:55:212
26249991611,3cyclictest14026-21ntpq21:05:232
2624999161,10cyclictest0-21swapper/219:53:502
2624999161,10cyclictest0-21swapper/219:53:502
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional