You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 09:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Dec 02, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305492235163,59sleep20-21swapper/219:08:142
304622230156,60sleep00-21swapper/019:07:060
305652204166,25sleep10-21swapper/119:08:251
305622204166,25sleep30-21swapper/319:08:233
307869910198,2cyclictest0-21swapper/223:23:022
30786999995,3cyclictest0-21swapper/220:32:142
30786998380,2cyclictest0-21swapper/200:11:052
30786997766,6cyclictest0-21swapper/221:30:332
115332770,3sleep30-21swapper/300:30:173
178682590,1sleep10-21swapper/123:30:251
11252580,4sleep121-21rcuc/119:15:021
155202510,5sleep115522-21pmu-power23:25:231
292192500,1sleep30-21swapper/321:30:163
30786994138,2cyclictest0-21swapper/222:38:592
3077499371,24cyclictest25210-21ntp_states21:20:220
108422360,18sleep010844-21kernelversion23:15:200
3078299339,6cyclictest17885-40dnf21:05:191
3077499321,19cyclictest0-21swapper/022:51:260
30792993112,5cyclictest1317-40dnf00:06:233
3078299315,17cyclictest0-21swapper/100:25:211
3077499312,12cyclictest11-21rcuc/020:22:500
30774993112,16cyclictest5674-21munin-run19:25:010
3079299309,5cyclictest0-21swapper/322:05:133
3078299301,21cyclictest26473-21ntp_states23:50:201
3077499301,23cyclictest0-21swapper/000:25:100
3079299291,6cyclictest19510-21sa121:10:013
3078299291,22cyclictest0-21swapper/120:15:181
3078299291,20cyclictest2541-21latency_hist21:45:021
3077499290,12cyclictest21185-21kworker/0:222:39:360
3077499290,10cyclictest0-21swapper/020:10:060
3079299283,16cyclictest0-21swapper/323:45:223
3079299283,16cyclictest0-21swapper/323:45:223
3079299281,18cyclictest0-21swapper/321:53:103
3078299288,10cyclictest30781-21cpu21:35:131
3078299288,10cyclictest30781-21cpu21:35:131
3078299284,12cyclictest0-21swapper/100:30:201
3078299282,13cyclictest0-21swapper/121:15:131
3078299280,22cyclictest0-21swapper/120:12:291
3077499281,12cyclictest0-21swapper/019:45:500
3077499281,11cyclictest0-21swapper/023:50:540
3079299276,13cyclictest86550irq/126-enp1s0-21:35:553
3079299276,13cyclictest86550irq/126-enp1s0-21:35:553
3079299270,17cyclictest0-21swapper/300:11:333
3078299272,15cyclictest0-21swapper/122:43:231
3078299272,15cyclictest0-21swapper/122:43:231
30782992713,12cyclictest28891-21kworker/u8:222:53:511
3078299271,24cyclictest25194-21awk23:50:001
3078299271,24cyclictest25194-21awk23:50:001
3077499271,9cyclictest0-21swapper/023:24:220
3077499271,14cyclictest0-21swapper/019:51:260
3077499271,12cyclictest30146-21df00:00:140
3077499271,10cyclictest0-21swapper/022:07:420
3077499271,10cyclictest0-21swapper/020:04:140
3077499270,14cyclictest0-21swapper/021:54:140
3077499270,13cyclictest0-21swapper/023:56:380
3077499270,12cyclictest0-21swapper/023:35:420
3079299263,16cyclictest0-21swapper/322:12:053
3079299261,19cyclictest0-21swapper/323:04:143
3079299261,18cyclictest0-21swapper/300:17:193
3078299262,15cyclictest0-21swapper/120:07:411
3078299261,22cyclictest4286-21latency23:00:201
3078299261,16cyclictest0-21swapper/119:59:361
3078299261,10cyclictest1951-21cpuspeed_turbos00:10:151
3077499267,3cyclictest0-21swapper/022:42:060
3077499267,3cyclictest0-21swapper/022:42:060
3077499264,12cyclictest17885-40dnf21:05:190
30774992620,3cyclictest9091-21/usr/sbin/munin23:10:250
3077499261,13cyclictest0-21swapper/023:00:140
3077499261,11cyclictest0-21swapper/023:45:340
3077499261,11cyclictest0-21swapper/023:45:340
3077499260,13cyclictest0-21swapper/021:31:260
3077499260,11cyclictest0-21swapper/021:11:420
3079299252,16cyclictest0-21swapper/322:40:433
3079299252,16cyclictest0-21swapper/322:40:433
3079299251,18cyclictest0-21swapper/323:54:363
3079299251,18cyclictest0-21swapper/323:41:343
3079299251,18cyclictest0-21swapper/322:53:093
3079299251,18cyclictest0-21swapper/322:27:433
3079299251,18cyclictest0-21swapper/300:03:563
3079299251,17cyclictest86550irq/126-enp1s0-21:46:273
3079299251,17cyclictest0-21swapper/323:38:553
3079299251,17cyclictest0-21swapper/322:22:063
3079299251,16cyclictest13280-21ntp_states23:20:223
30792992510,9cyclictest0-21swapper/321:44:083
3079299250,7cyclictest0-21swapper/323:59:503
3078299252,11cyclictest6942-21munin-run21:55:021
3078299252,11cyclictest17958-21ntp_states19:50:221
30782992513,3cyclictest0-21swapper/120:49:231
3078299251,21cyclictest768-21in:imjournal22:08:171
3078299251,21cyclictest14116-21sh22:10:131
3078299251,16cyclictest19509-21diskmemload22:02:291
3078299251,15cyclictest0-21swapper/120:55:481
3078299251,11cyclictest7843-21hddtemp_smartct21:55:191
3077499257,15cyclictest489-21munin-run19:15:000
3077499251,14cyclictest0-21swapper/020:33:580
3077499251,12cyclictest0-21swapper/022:48:460
3077499251,10cyclictest0-21swapper/021:16:300
3077499250,11cyclictest0-21swapper/022:57:260
3077499250,11cyclictest0-21swapper/020:08:060
3079299243,14cyclictest86550irq/126-enp1s0-22:15:443
30792992417,4cyclictest0-21swapper/323:06:413
3079299241,5cyclictest0-21swapper/322:35:063
3079299241,17cyclictest0-21swapper/321:16:363
3079299241,16cyclictest0-21swapper/321:21:023
3079299241,15cyclictest0-21swapper/321:13:033
3079299240,13cyclictest0-21swapper/322:01:113
3078299248,4cyclictest0-21swapper/122:49:011
3078299247,3cyclictest623-21cat19:15:001
3078299242,18cyclictest29574-21munin-run00:00:011
30782992420,3cyclictest0-21swapper/120:20:441
3078299241,6cyclictest17209-21sendmail_mailqu22:15:231
3078299241,21cyclictest965-21perf22:59:561
3078299241,16cyclictest0-21swapper/100:17:401
3078299241,15cyclictest0-21swapper/122:36:201
3078299241,15cyclictest0-21swapper/122:34:441
3078299241,15cyclictest0-21swapper/121:26:431
3078299241,15cyclictest0-21swapper/121:24:311
3078299241,15cyclictest0-21swapper/120:33:341
3078299241,11cyclictest21444-21ntp_states22:25:231
3078299241,11cyclictest0-21swapper/121:11:031
3078299241,10cyclictest581-21systemd-journal00:23:511
3078299241,10cyclictest5669-21sh19:25:011
3077499246,3cyclictest0-21swapper/021:04:300
3077499241,15cyclictest0-21swapper/020:59:100
3077499241,12cyclictest0-21swapper/023:30:300
3077499241,12cyclictest0-21swapper/023:05:260
3077499241,11cyclictest0-21swapper/022:04:300
3077499240,10cyclictest0-21swapper/023:41:500
3079299237,3cyclictest0-21swapper/322:30:203
30792992317,3cyclictest19509-21diskmemload23:10:193
3079299231,12cyclictest0-21swapper/323:15:083
3079299231,11cyclictest0-21swapper/321:25:433
3078299239,3cyclictest21355-21diskstats23:40:151
3078299238,3cyclictest18511-21df22:20:141
3078299232,19cyclictest0-21swapper/119:48:071
3078299231,9cyclictest4684-21diskstats20:35:151
30782992313,3cyclictest0-21swapper/120:04:341
30782992313,2cyclictest0-21swapper/121:02:331
3078299231,21cyclictest0-21swapper/123:11:161
3078299231,20cyclictest0-21swapper/100:08:231
3078299231,19cyclictest6998-21cut20:40:161
3078299231,16cyclictest0-21swapper/120:50:161
3078299231,11cyclictest0-21swapper/100:02:071
3078299231,10cyclictest3082-21expr21:45:151
3078299230,15cyclictest0-21swapper/121:31:041
3077499231,9cyclictest0-21swapper/023:25:100
3077499231,8cyclictest0-21swapper/020:52:220
3077499231,13cyclictest17844-21kworker/0:122:20:260
3077499231,13cyclictest0-21swapper/021:40:380
3077499231,13cyclictest0-21swapper/019:17:180
3077499231,13cyclictest0-21swapper/000:08:140
3077499231,12cyclictest0-21swapper/021:55:340
3077499231,12cyclictest0-21swapper/019:44:220
3077499231,11cyclictest0-21swapper/021:38:380
3077499231,11cyclictest0-21swapper/021:38:380
3077499230,16cyclictest0-21swapper/022:11:580
3077499230,12cyclictest0-21swapper/000:33:180
3079299227,9cyclictest0-21swapper/322:50:003
3079299222,5cyclictest16667-21munin-run19:50:013
3079299222,11cyclictest32304-21/usr/sbin/munin20:25:253
30792992218,3cyclictest0-21swapper/320:16:483
30786992216,3cyclictest5311-21find00:15:242
3078299221,19cyclictest0-21swapper/119:29:311
3078299221,19cyclictest0-21swapper/119:29:311
3078299221,14cyclictest0-21swapper/120:26:071
3078299221,10cyclictest768-21in:imjournal19:33:131
3077499226,9cyclictest784-21NetworkManager20:38:020
3077499220,13cyclictest0-21swapper/019:25:500
3077499220,13cyclictest0-21swapper/019:25:500
3077499220,12cyclictest0-21swapper/019:33:020
3079299219,6cyclictest0-21swapper/323:25:063
3079299218,3cyclictest0-21swapper/320:35:203
3079299212,5cyclictest9601-21sed00:25:233
3079299211,6cyclictest23265-21latency_hist20:05:013
3079299211,6cyclictest0-21swapper/323:34:143
3079299211,14cyclictest0-21swapper/322:57:043
3079299211,14cyclictest0-21swapper/300:39:383
3079299211,11cyclictest13130-21/usr/sbin/munin20:55:153
3078299219,2cyclictest0-21swapper/123:36:131
3078299215,14cyclictest12524-21cpu19:40:141
3078299211,2cyclictest5394-21perf23:09:561
3078299211,19cyclictest0-21swapper/119:36:011
3078299211,10cyclictest0-21swapper/123:18:231
3077499217,3cyclictest0-21swapper/000:12:540
3077499215,7cyclictest1-21systemd00:40:010
3077499211,13cyclictest0-21swapper/021:47:580
3077499211,11cyclictest0-21swapper/019:57:340
3077499210,13cyclictest0-21swapper/020:46:460
3077499210,10cyclictest0-21swapper/019:35:100
4792200,4sleep23078699cyclictest00:05:182
3079299205,3cyclictest86850irq/129-enp1s0-21:00:173
3079299205,3cyclictest12050irq/123-ahci[0019:45:013
3079299202,5cyclictest8445-21df_inode19:30:153
3079299201,6cyclictest3423-21switchtime20:30:263
30792992014,3cyclictest12113-21sshd19:38:123
3079299201,10cyclictest19090-21sshd19:55:123
3078699208,3cyclictest23918-21awk21:20:002
3078699201,6cyclictest30301-21grep22:45:232
3078699201,15cyclictest16589-21grep22:15:162
3078299201,3cyclictest0-21swapper/123:21:251
3078299201,17cyclictest0-21swapper/100:35:251
3077499201,11cyclictest6535-21/usr/sbin/munin20:40:220
3077499201,11cyclictest0-21swapper/022:30:300
3077499201,11cyclictest0-21swapper/022:15:420
30774992010,8cyclictest86550irq/126-enp1s0-20:26:510
3077499200,10cyclictest27902-21kworker/0:020:18:160
264842200,7sleep230-21rcuc/223:50:212
45592191,2sleep20-21swapper/221:47:502
3079299194,3cyclictest86550irq/126-enp1s0-20:10:133
3079299191,9cyclictest16894-21/usr/sbin/munin19:50:173
3079299191,9cyclictest0-21swapper/319:22:133
30792991917,1cyclictest0-21swapper/320:50:143
30792991914,3cyclictest7485-21processes00:20:213
3079299191,10cyclictest7798-21sshd19:29:123
3079299191,10cyclictest7798-21sshd19:29:123
3079299191,10cyclictest0-21swapper/321:58:063
3078699192,3cyclictest12050irq/123-ahci[0022:55:012
3078699191,6cyclictest23638-21ntp_states22:30:222
3078699191,6cyclictest0-21swapper/223:38:202
3078699191,5cyclictest21863-21wc23:40:192
3078699191,10cyclictest0-21swapper/223:04:562
3078699191,10cyclictest0-21swapper/222:09:152
286852191,3sleep20-21swapper/222:41:542
286852191,3sleep20-21swapper/222:41:542
109162191,3sleep20-21swapper/220:50:062
3079299181,5cyclictest7093-21perl20:40:163
3079299181,15cyclictest0-21swapper/320:22:293
3078699182,8cyclictest86550irq/126-enp1s0-23:10:142
3078699181,8cyclictest10035-21egrep23:15:122
3078699181,7cyclictest23759-21perl23:45:152
3078699181,7cyclictest23759-21perl23:45:152
30786991812,3cyclictest26428-21cat21:25:152
3078699181,15cyclictest0-21swapper/221:20:122
30786991811,4cyclictest30-21rcuc/200:34:252
3077499180,11cyclictest27018-21kworker/0:300:15:160
18982181,2sleep20-21swapper/220:29:582
34632171,2sleep20-21swapper/219:19:572
3079299176,3cyclictest8733-21/usr/sbin/munin20:45:183
3079299171,6cyclictest0-21swapper/320:05:293
30792991711,3cyclictest1826-21expr19:15:143
3078699177,3cyclictest86550irq/126-enp1s0-23:55:172
3078699174,11cyclictest25473-21sa120:10:012
3078699172,4cyclictest0-21swapper/221:58:012
3078699171,9cyclictest0-21swapper/219:12:132
3078699171,7cyclictest0-21swapper/222:13:482
3078699171,7cyclictest0-21swapper/200:20:162
3078699171,4cyclictest0-21swapper/222:55:272
3078699171,3cyclictest12050irq/123-ahci[0020:20:002
30786991712,3cyclictest7082-21unixbench-2d23:05:282
30786991712,3cyclictest6561-21users21:50:272
30786991712,3cyclictest31368-21grep21:35:152
30786991712,3cyclictest31368-21grep21:35:152
30786991712,3cyclictest17757-21expr21:05:132
30786991712,3cyclictest17520-21gltestperf19:50:162
3078699171,14cyclictest0-21swapper/219:40:042
30786991711,3cyclictest836-21runrttasks22:20:112
30786991711,3cyclictest6399-21grep19:25:172
30786991711,3cyclictest6399-21grep19:25:172
30786991711,3cyclictest30187-21cpuspeed_turbos20:20:142
30774991711,3cyclictest7802-21grep00:20:250
220992170,2sleep20-21swapper/222:27:042
170962171,2sleep20-21swapper/221:02:302
3079299161,4cyclictest30998-21egrep19:10:133
3078699166,3cyclictest86550irq/126-enp1s0-23:25:132
3078699165,3cyclictest86750irq/128-enp1s0-20:10:182
3078699162,12cyclictest10741-21grep22:00:252
3078699161,7cyclictest0-21swapper/223:34:342
3078699161,6cyclictest8733-21/usr/sbin/munin20:45:212
3078699161,6cyclictest21293-21/usr/sbin/munin20:00:272
3078699161,6cyclictest12260-21latency_hist19:40:012
3078699161,6cyclictest0-21swapper/219:45:252
3078699161,5cyclictest1-21systemd19:30:092
3078699161,5cyclictest0-21swapper/200:00:132
30786991612,3cyclictest0-21swapper/221:40:452
3078699161,14cyclictest0-21swapper/219:21:532
30786991611,3cyclictest9207-21if_enp1s000:25:162
30786991611,3cyclictest19823-21cat21:10:122
30786991610,4cyclictest0-21swapper/200:35:242
3078699160,7cyclictest15091-21crond21:00:012
3077499168,5cyclictest27287-21/usr/sbin/munin21:25:230
3077499161,6cyclictest20493-21tr22:25:140
3078699151,6cyclictest6535-21/usr/sbin/munin20:40:132
30786991510,3cyclictest5088-21if_enp2s020:35:172
30786991510,3cyclictest19789-21idleruntime19:55:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional