You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 08:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Nov 26, 2025 00:44:34)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
274682239162,25sleep00-21swapper/019:09:390
272712238163,25sleep10-21swapper/119:07:061
273912236164,58sleep20-21swapper/219:08:402
273802232163,55sleep30-21swapper/319:08:333
27600991246,111cyclictest0-21swapper/221:08:062
96212710,4sleep12759699cyclictest22:05:271
200472600,5sleep22760099cyclictest22:30:162
198182590,4sleep32760799cyclictest21:15:223
60282580,10sleep232-21ksoftirqd/222:00:012
13312520,2sleep30-21swapper/323:00:233
160552440,1sleep20-21swapper/223:35:152
2759699343,22cyclictest4159-21kworker/u8:223:10:011
2759099346,12cyclictest26689-21awk20:20:010
2759699313,20cyclictest7628-21irqstats23:15:191
2759699311,24cyclictest0-21swapper/122:48:581
2759699311,10cyclictest16312-21diskmemload23:26:591
2760099306,4cyclictest551-21sh00:15:012
2759699301,22cyclictest0-21swapper/122:16:011
2759699301,22cyclictest0-21swapper/122:16:011
2759699290,26cyclictest0-21swapper/121:51:501
2759099295,11cyclictest31707-21idleruntime-cro21:45:010
2759099292,12cyclictest7252-21idleruntime-cro00:30:010
2759699281,20cyclictest0-21swapper/121:30:241
2759699280,22cyclictest0-21swapper/123:35:291
27600992723,3cyclictest0-21swapper/200:09:562
2760099271,6cyclictest0-21swapper/222:37:392
2759699271,6cyclictest6443-21proc_pri00:25:231
163582270,6sleep011-21rcuc/021:10:010
27596992611,13cyclictest17284-21kworker/u8:223:41:031
2759099261,12cyclictest2849-21munin-run00:20:010
27600992522,2cyclictest0-21swapper/200:04:572
2759699250,16cyclictest0-21swapper/123:32:391
2759099256,16cyclictest5046-21munin-run00:25:010
2759099256,16cyclictest5046-21munin-run00:25:010
2759099251,12cyclictest25124-21awk21:30:010
2760099245,6cyclictest7647-21unixbench-2d22:00:262
2759699247,11cyclictest0-21swapper/100:05:271
2759699246,9cyclictest0-21swapper/122:14:571
27596992419,3cyclictest0-21swapper/119:50:311
27596992419,2cyclictest182402sleep121:11:391
27596992419,2cyclictest182402sleep121:11:391
27596992415,2cyclictest9679-21seq20:54:521
27596992414,3cyclictest0-21swapper/120:35:191
2759699241,17cyclictest4159-21kworker/u8:223:14:061
2759699241,17cyclictest0-21swapper/122:55:021
2759699241,17cyclictest0-21swapper/122:30:181
2759699241,17cyclictest0-21swapper/120:45:271
2759699241,17cyclictest0-21swapper/120:41:211
2759699241,16cyclictest0-21swapper/122:40:231
2759699241,16cyclictest0-21swapper/121:55:191
2759699241,16cyclictest0-21swapper/120:55:251
2759699241,12cyclictest14116-21awk21:05:021
27596992411,11cyclictest17284-21kworker/u8:200:10:041
27596992410,11cyclictest8340-21latency00:30:181
2759699240,18cyclictest0-21swapper/100:38:561
2759699240,16cyclictest0-21swapper/121:47:081
2759699240,16cyclictest0-21swapper/121:40:311
310852231,18sleep131318-21sed19:15:141
2760799232,11cyclictest21917-21diskstats22:35:143
2760099234,6cyclictest0-21swapper/221:40:132
2759699233,11cyclictest18513-21dump-pmu-power21:19:571
27596992319,3cyclictest0-21swapper/120:22:241
27596992315,2cyclictest0-21swapper/120:30:351
27596992314,3cyclictest0-21swapper/121:35:351
2759699231,2cyclictest0-21swapper/123:03:551
2759699231,16cyclictest0-21swapper/122:22:511
2759699231,16cyclictest0-21swapper/100:18:011
2759699231,14cyclictest0-21swapper/123:50:401
2759699231,10cyclictest16308-21sa121:10:021
2759699230,5cyclictest0-21swapper/121:26:201
2759699230,16cyclictest0-21swapper/120:00:131
2759699230,16cyclictest0-21swapper/119:30:191
2759699230,15cyclictest0-21swapper/123:49:161
2759699230,15cyclictest0-21swapper/122:51:431
2759699230,15cyclictest0-21swapper/100:20:111
2759699230,15cyclictest0-21swapper/100:20:111
2759099236,8cyclictest0-21swapper/022:00:010
2759099236,4cyclictest0-21swapper/022:40:010
2759099233,7cyclictest17816-40dnf19:59:330
27590992317,3cyclictest32338-21latency_hist23:00:010
2760799223,16cyclictest29069-21grep21:35:303
2760099228,5cyclictest10028-21awk00:35:162
2760099223,6cyclictest24069-21unixbench_singl20:10:302
2760099222,16cyclictest19207-21tune2fs21:15:152
2760099221,7cyclictest28775-21tail22:50:132
2759699226,3cyclictest9859-21perl19:40:151
2759699225,10cyclictest18097-21missed_timers22:25:191
2759699222,10cyclictest27459-21latency00:00:211
2759699221,8cyclictest23683-21ps20:10:271
27596992218,3cyclictest0-21swapper/122:03:211
27596992217,3cyclictest0-21swapper/120:05:141
27596992217,3cyclictest0-21swapper/120:05:141
2759699221,3cyclictest86550irq/126-enp1s0-19:59:331
2759699221,13cyclictest0-21swapper/121:24:391
2759699220,16cyclictest0-21swapper/122:35:241
2759699220,16cyclictest0-21swapper/119:35:191
2759699220,14cyclictest0-21swapper/123:21:411
2759699220,14cyclictest0-21swapper/123:21:411
2759099227,5cyclictest836-21runrttasks00:14:360
2759099225,15cyclictest27944-21munin-run22:50:010
2760799217,3cyclictest0-21swapper/300:26:573
2760799216,4cyclictest0-21swapper/322:02:243
2760799213,7cyclictest0-21swapper/300:18:523
2760799212,6cyclictest7538-21tr00:30:123
2760799211,3cyclictest2758-21df23:05:153
2760799211,10cyclictest25183-21latency_hist21:30:013
2760099215,6cyclictest12102-21grep19:45:162
2760099212,7cyclictest16544-21/usr/sbin/munin21:10:232
2760099212,7cyclictest16544-21/usr/sbin/munin21:10:232
2760099212,6cyclictest9013-21sed22:05:182
2759699217,8cyclictest0-21swapper/119:20:171
27596992114,2cyclictest0-21swapper/119:45:151
2759699211,15cyclictest0-21swapper/120:25:371
2759699210,16cyclictest0-21swapper/119:29:241
2759699210,15cyclictest0-21swapper/119:10:071
2759099215,7cyclictest780-21rngd21:35:290
2759099213,16cyclictest29760-21munin-run19:15:010
2759099212,11cyclictest24401-21/usr/sbin/munin23:55:240
2759099211,9cyclictest784-21NetworkManager20:02:040
27590992114,4cyclictest5924-21netstat19:30:200
27590992114,4cyclictest27689-21ntp_states00:00:240
42932201,16sleep34336-21smartctl00:20:253
42932201,16sleep34336-21smartctl00:20:253
2760799209,2cyclictest9554-21ps22:05:253
2760799208,7cyclictest86550irq/126-enp1s0-21:51:213
2760799207,3cyclictest24748-21diskstats23:55:143
2760799206,5cyclictest0-21swapper/323:47:273
2760799206,11cyclictest784-21NetworkManager20:37:203
2760799203,9cyclictest0-21swapper/321:34:413
27607992013,4cyclictest25964-21sensors_temp20:15:253
2760799201,12cyclictest0-21swapper/321:48:573
2760799201,11cyclictest7106-21/usr/sbin/munin19:35:143
2760799201,11cyclictest4911-21/usr/sbin/munin19:30:263
2760799201,11cyclictest0-21swapper/321:21:373
2760799201,10cyclictest0-21swapper/323:39:303
2760799200,7cyclictest0-21swapper/323:15:363
2760799200,11cyclictest0-21swapper/323:51:213
2760099205,3cyclictest12050irq/123-ahci[0021:40:022
2760099203,2cyclictest0-21swapper/222:15:172
2760099203,2cyclictest0-21swapper/222:15:172
2760099202,15cyclictest28493-21ls19:10:182
2760099201,6cyclictest16498-21grep19:55:162
27600992014,3cyclictest31859-21unixbench_multi22:55:262
27600992014,3cyclictest21141-21head23:45:232
2760099201,17cyclictest0-21swapper/223:30:542
2760099201,12cyclictest0-21swapper/221:25:492
2760099201,11cyclictest0-21swapper/223:05:292
2759699201,17cyclictest0-21swapper/120:15:201
2759699201,14cyclictest0-21swapper/123:56:331
2759099209,3cyclictest11209-21/usr/sbin/munin23:25:280
2759099202,10cyclictest1780-21/usr/sbin/munin21:50:160
2759099201,8cyclictest7870-21chrt23:15:230
2759099201,6cyclictest0-21swapper/021:21:550
27590992011,7cyclictest1991-21sshd23:02:340
2759099201,11cyclictest17810-21/usr/sbin/munin23:40:170
27590992010,3cyclictest16318-21timerandwakeup22:20:260
126522201,15sleep012747-21/usr/sbin/munin19:45:240
87912191,3sleep20-21swapper/219:36:362
2760799196,4cyclictest0-21swapper/323:11:003
2760799195,4cyclictest0-21swapper/322:11:103
2760799193,6cyclictest0-21swapper/323:42:453
2760799191,9cyclictest31041-21/usr/sbin/munin00:10:163
2760799191,7cyclictest581-21systemd-journal23:30:013
2760799191,3cyclictest26601-40updatedb00:00:103
2760799191,12cyclictest0-21swapper/319:10:173
2760799191,11cyclictest0-21swapper/321:55:023
2760799191,10cyclictest413-21/usr/sbin/munin19:20:203
2760799191,10cyclictest3342-21/usr/sbin/munin20:40:173
2760099196,3cyclictest86550irq/126-enp1s0-23:25:132
2760099195,3cyclictest86550irq/126-enp1s0-23:40:142
2760099193,5cyclictest780-21rngd23:52:592
2760099192,9cyclictest0-21swapper/222:49:022
2760099192,6cyclictest19692-21kworker/2:020:40:162
2760099192,5cyclictest28174-21smart_sda20:20:242
2760099192,15cyclictest0-21swapper/220:50:062
2760099191,7cyclictest784-21NetworkManager20:57:152
2760099191,6cyclictest20308-21/usr/sbin/munin20:05:202
2760099191,6cyclictest20308-21/usr/sbin/munin20:05:202
2760099191,5cyclictest7396-21perl23:15:172
2760099191,17cyclictest0-21swapper/220:16:562
27600991911,5cyclictest12123-40dnf21:00:092
2760099191,11cyclictest0-21swapper/200:17:542
2760099191,10cyclictest0-21swapper/200:27:522
2759099196,5cyclictest3884-21unixbench-2d23:05:280
2759099196,4cyclictest10061-21fschecks_count00:35:160
2759099191,7cyclictest0-21swapper/019:21:340
2759099191,3cyclictest8793-21sshd19:36:340
27590991912,4cyclictest15701-21cpuspeed_turbos23:35:120
2759099191,12cyclictest0-21swapper/021:31:330
111632190,4sleep3401ktimersoftd/300:35:333
103382191,3sleep00-21swapper/022:09:590
74362181,2sleep00-21swapper/020:49:180
2760799187,2cyclictest18095-21missed_timers22:25:183
2760799185,4cyclictest0-21swapper/321:44:363
2760799185,3cyclictest0-21swapper/322:22:443
2760799183,4cyclictest0-21swapper/322:43:113
2760799181,9cyclictest5535-21/usr/sbin/munin20:45:193
2760799181,9cyclictest0-21swapper/322:59:003
2760799181,7cyclictest13115-21df_inode22:15:133
2760799181,7cyclictest13115-21df_inode22:15:133
2760799181,5cyclictest0-21swapper/322:48:433
2760799181,3cyclictest28751-21perl22:50:133
2760799181,3cyclictest10182-21ps23:20:253
2760799181,3cyclictest10182-21ps23:20:253
27607991813,3cyclictest30673-21/usr/sbin/munin19:15:243
27607991813,3cyclictest21435-21pmu-power20:05:223
27607991813,3cyclictest21435-21pmu-power20:05:223
2760799181,11cyclictest0-21swapper/323:31:473
2760799181,10cyclictest0-21swapper/322:31:373
2760799181,10cyclictest0-21swapper/321:14:183
2760799181,10cyclictest0-21swapper/321:14:183
2760799181,10cyclictest0-21swapper/300:06:553
2760099185,3cyclictest16312-21diskmemload21:49:002
2760099184,3cyclictest86750irq/128-enp1s0-20:45:182
2760099184,3cyclictest86550irq/126-enp1s0-21:20:282
2760099184,3cyclictest7536-21cpuspeed_turbos00:30:122
2760099181,7cyclictest0-21swapper/222:23:412
2760099181,5cyclictest4622-21/usr/sbin/munin23:10:182
2760099181,5cyclictest4144-21ntp_states00:20:222
2760099181,5cyclictest4144-21ntp_states00:20:222
2760099181,5cyclictest30673-21/usr/sbin/munin19:15:102
2760099181,5cyclictest30179-21ntp_states20:25:222
2760099181,5cyclictest1780-21/usr/sbin/munin21:50:192
2760099181,5cyclictest13699-21/usr/sbin/munin19:50:152
2760099181,5cyclictest10928-21diskstats22:10:162
2760099181,4cyclictest24593-21kernelversion22:40:202
2760099181,4cyclictest10276-21smart_sda23:20:272
2760099181,4cyclictest10276-21smart_sda23:20:272
27600991812,3cyclictest9320-21cpu19:40:132
27600991811,4cyclictest961-21grep19:20:152
2759099187,3cyclictest14634-21snmpd23:51:480
2759099185,4cyclictest0-21swapper/022:30:010
27590991812,3cyclictest9856-21cat20:55:010
2759099181,11cyclictest0-21swapper/020:10:260
2759099181,10cyclictest0-21swapper/023:13:090
2759099181,10cyclictest0-21swapper/019:50:250
210592181,2sleep00-21swapper/023:45:210
2760799172,12cyclictest24067-21grep20:10:303
2760799171,9cyclictest0-21swapper/319:25:063
2760799171,7cyclictest15430-21open_inodes21:05:213
2760799171,7cyclictest13202-21ntp_states21:00:233
2760799171,3cyclictest9304-21/usr/sbin/munin19:40:183
27607991712,3cyclictest18103-21/usr/sbin/munin20:00:133
27607991712,3cyclictest13917-21expr19:50:133
27607991712,3cyclictest11269-21latency_hist19:45:023
27607991712,2cyclictest14634-21snmpd20:21:093
2760799171,14cyclictest0-21swapper/320:25:083
2760799171,13cyclictest31309-21/usr/sbin/munin20:30:163
2760099171,6cyclictest0-21swapper/223:00:032
2760099171,5cyclictest18103-21/usr/sbin/munin20:00:212
2760099171,4cyclictest17174-21/usr/sbin/munin22:25:132
2760099171,4cyclictest0-21swapper/223:59:222
27600991712,3cyclictest2713-21/usr/sbin/munin19:25:132
2760099171,15cyclictest0-21swapper/220:31:302
27600991710,4cyclictest2175-21ntp_kernel_err20:35:222
2759099173,7cyclictest86550irq/126-enp1s0-23:34:440
2759099172,12cyclictest28171-21/usr/sbin/munin22:50:110
2759099171,9cyclictest0-21swapper/022:30:110
2759099171,7cyclictest29110-21/usr/sbin/munin20:25:220
2759099171,7cyclictest1123-21/usr/sbin/munin20:35:140
2759099171,6cyclictest0-21swapper/022:41:070
2759099171,6cyclictest0-21swapper/020:55:120
2759099171,2cyclictest0-21swapper/019:15:140
27590991712,3cyclictest29450-21tail00:05:170
27590991712,3cyclictest20858-21cat21:20:010
27590991712,3cyclictest17636-21open_files21:10:230
27590991712,3cyclictest17636-21open_files21:10:230
2759099171,15cyclictest7-21ksoftirqd/019:45:010
2759099171,13cyclictest20950-21gltestperf20:05:150
2759099171,13cyclictest20950-21gltestperf20:05:150
2759099171,10cyclictest0-21swapper/020:40:340
2759099171,10cyclictest0-21swapper/019:25:240
2760799161,6cyclictest0-21swapper/320:55:263
27607991612,3cyclictest0-21swapper/319:59:503
2760799160,10cyclictest0-21swapper/320:50:143
27600991611,3cyclictest4970-21cpu19:30:142
27600991611,3cyclictest26437-21open_inodes21:30:262
2759099166,3cyclictest86750irq/128-enp1s0-22:10:230
2759099163,4cyclictest31985-21cpu21:45:130
2759099161,8cyclictest0-21swapper/022:15:140
2759099161,8cyclictest0-21swapper/022:15:140
2759099161,6cyclictest31309-21/usr/sbin/munin20:30:200
2759099161,6cyclictest0-21swapper/022:00:180
2759099161,6cyclictest0-21swapper/021:01:080
27590991611,3cyclictest27951-21ntp_kernel_pll_20:20:200
2759099160,7cyclictest0-21swapper/000:30:170
2759099151,7cyclictest0-21swapper/023:20:170
2759099151,7cyclictest0-21swapper/023:20:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional