You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-16 - 23:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Tue Dec 16, 2025 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
154702232163,56sleep20-21swapper/207:06:082
155752201164,24sleep30-21swapper/307:07:303
155782200165,23sleep10-21swapper/107:07:331
154362200165,23sleep00-21swapper/007:05:430
959821590,4sleep21586299cyclictest10:35:012
15858996764,2cyclictest0-21swapper/109:21:151
42012610,4sleep123-21ksoftirqd/111:35:161
40822600,1sleep30-21swapper/311:35:143
63982570,1sleep00-21swapper/008:00:120
251162540,2sleep20-21swapper/212:20:242
57412520,2sleep20-21swapper/210:25:142
1586999358,16cyclictest28887-21grep10:05:013
15869993512,15cyclictest26499-21NetworkManager09:59:393
1585299345,13cyclictest16183-21awk10:50:000
15869993211,12cyclictest32743-21/usr/sbin/munin09:00:263
15869993210,10cyclictest16587-21/usr/sbin/munin09:35:163
15869993210,10cyclictest16587-21/usr/sbin/munin09:35:163
1585899323,13cyclictest19835-40dnf09:43:501
1585299323,14cyclictest18992-21munin-run12:10:010
15869993111,12cyclictest800-21snmpd11:42:133
1585299319,10cyclictest0-21swapper/011:05:010
1585299311,14cyclictest24970-21munin-run11:10:010
166922300,1sleep10-21swapper/110:50:121
15869993010,11cyclictest11106-21df_inode08:10:133
1585899301,17cyclictest831-21runrttasks10:25:171
1586999295,20cyclictest26604-21ntp_states07:30:213
1586999295,10cyclictest2885-21memory07:50:173
15852992910,16cyclictest29358-21munin-run11:20:020
285062280,4sleep123-21ksoftirqd/107:35:181
1585899287,12cyclictest23343-21/usr/sbin/munin07:25:171
1585899285,11cyclictest3209-21/usr/sbin/munin10:20:161
1585899281,23cyclictest30944-21munin-run10:10:021
1586999278,7cyclictest13865-40dnf10:43:573
1586999277,6cyclictest26298-21/usr/sbin/munin11:10:223
1586999273,13cyclictest87450irq/129-enp1s0-07:40:193
1586999273,11cyclictest20791-21/usr/sbin/munin11:00:153
1585899273,2cyclictest88450irq/133-enp2s0-11:54:541
15858992716,3cyclictest25485-21memory09:55:201
1585899270,14cyclictest22468-21ntp_kernel_pll_12:15:211
15852992710,14cyclictest23400-21idleruntime-cro12:20:010
43382260,4sleep121-21rcuc/107:55:121
1586999264,9cyclictest11390-21vmstat10:35:263
1586999264,8cyclictest28348-21/usr/sbin/munin08:50:183
1586999263,14cyclictest87150irq/126-enp1s0-07:15:193
1586999263,14cyclictest87150irq/126-enp1s0-07:15:193
1586999263,12cyclictest11378-21/usr/sbin/munin09:25:173
1586999262,14cyclictest831-21runrttasks09:50:033
1586999261,14cyclictest0-21swapper/311:15:153
1586299261,12cyclictest0-21swapper/209:00:472
1585899265,9cyclictest19607-21fschecks_count12:10:141
15858992616,3cyclictest14878-21sed10:45:171
1585899261,12cyclictest26622-21irqstats12:25:171
1585899260,16cyclictest27607-21kernelversion10:00:181
1585299261,12cyclictest3605-21awk11:35:010
1586999259,13cyclictest22952-21sshd07:23:073
1586999257,6cyclictest2437-21irqstats11:30:183
1586999257,5cyclictest18019-21crond07:15:013
1586999254,14cyclictest28975-21/usr/sbin/munin10:05:153
1586999253,13cyclictest0-21swapper/311:30:023
1586999252,8cyclictest27337-1kworker/3:1H09:21:593
15869992517,5cyclictest23691-21diskstats07:25:133
15869992511,4cyclictest30198-21/usr/sbin/munin12:35:143
1585899255,9cyclictest3589-21latency_hist11:35:011
1585899253,19cyclictest3205-21grep09:05:161
1585899253,18cyclictest13776-21irqstats08:15:181
1585899253,10cyclictest24495-21unixbench_multi11:05:251
1585899252,8cyclictest11761-21latency_hist10:40:011
1585899252,12cyclictest30116-21cut11:20:151
15858992520,2cyclictest1-21systemd11:44:081
1585899251,5cyclictest25762-40dnf08:43:081
1585899251,2cyclictest88350irq/132-enp2s0-09:02:131
1585899251,11cyclictest6505-21tr08:00:121
1585899251,10cyclictest28488-21tr08:50:111
1585299256,16cyclictest6756-21munin-run09:15:010
1586999247,14cyclictest18909-21sshd12:09:073
1586999246,14cyclictest18071-21grep09:40:123
1586999245,7cyclictest19564-21df_inode12:10:123
1586999245,6cyclictest6177-21cat08:00:013
1586999245,6cyclictest22997-21/usr/sbin/munin11:05:193
1586999245,16cyclictest831-21runrttasks08:02:293
1586999243,6cyclictest0-21swapper/310:23:073
1586999242,12cyclictest0-21swapper/312:15:173
15869992414,6cyclictest5555-21/usr/sbin/munin09:10:173
1586999241,14cyclictest0-21swapper/308:33:413
1586999241,12cyclictest0-21swapper/309:19:383
1585899249,3cyclictest6217-21sed09:10:251
1585899247,9cyclictest18579-21awk08:25:231
1585899247,13cyclictest8562-21diskstats11:45:141
1585899244,12cyclictest13344-21latency_hist09:30:011
1585899243,9cyclictest0-21swapper/107:35:021
1585899243,13cyclictest30546-21/usr/sbin/munin08:55:191
1585899243,11cyclictest788-21NetworkManager07:44:411
1585899242,2cyclictest88450irq/133-enp2s0-09:32:301
1585899242,11cyclictest18085-21cat07:15:011
1585899241,17cyclictest0-21swapper/112:08:241
1585899241,17cyclictest0-21swapper/110:30:191
1585899241,17cyclictest0-21swapper/109:37:361
1585899241,17cyclictest0-21swapper/109:37:361
1585899241,14cyclictest0-21swapper/109:51:491
1585899241,12cyclictest21055-21latency_hist07:20:011
1585899241,12cyclictest21055-21latency_hist07:20:011
1585299245,16cyclictest25923-21munin-run08:45:020
1585299243,17cyclictest10187-21munin-run11:50:010
1585299242,9cyclictest19686-21kworker/0:009:49:100
1585299242,10cyclictest1408-21perf11:30:020
1586999238,11cyclictest29986-21basename12:35:023
1586999237,9cyclictest29590-21/usr/sbin/munin11:20:243
1586999236,5cyclictest5750irq/9-acpi08:05:223
1586999234,6cyclictest12964-21/usr/sbin/munin08:15:183
1586999234,16cyclictest831-21runrttasks12:26:433
1586999233,7cyclictest5407-21/usr/sbin/munin10:25:183
1586999233,17cyclictest16398-21/usr/sbin/munin10:50:173
1586999231,7cyclictest0-21swapper/307:45:183
15869992315,5cyclictest32431-21smart_sda10:10:213
15869992314,5cyclictest8671-21ntp_states10:30:203
15869992313,7cyclictest14198-21/usr/sbin/munin11:55:263
15869992313,7cyclictest14198-21/usr/sbin/munin11:55:263
1586999231,10cyclictest16089-21sshd10:49:103
1585899237,9cyclictest0-21swapper/112:40:011
1585899237,13cyclictest19696-21open_inodes10:55:221
1585899237,13cyclictest12676-21/usr/sbin/munin11:55:131
1585899237,13cyclictest12676-21/usr/sbin/munin11:55:131
1585899236,3cyclictest21991-21taskset09:48:141
1585899231,18cyclictest2378-21/usr/sbin/munin10:15:251
1585899231,16cyclictest0-21swapper/111:02:391
1585899231,12cyclictest18172-21munin-node08:10:251
1585899231,10cyclictest21522-21crond08:35:001
1585899230,12cyclictest28115-21latency_hist08:50:011
1585299238,3cyclictest21139-21kworker/0:012:32:090
1585299237,4cyclictest25969-21kworker/0:107:19:300
1585299237,4cyclictest25969-21kworker/0:107:19:290
1585299230,11cyclictest12396-21perf11:55:010
1586999229,10cyclictest1565-21chrt10:15:153
1586999228,3cyclictest87150irq/126-enp1s0-11:45:133
1586999228,10cyclictest18933-21/usr/sbin/munin10:55:143
15869992214,5cyclictest10763-21diskstats11:50:153
15869992213,6cyclictest15160-21/usr/sbin/munin08:20:173
15869992212,7cyclictest2592-21/usr/sbin/munin09:05:263
15869992212,7cyclictest25022-21ntp_states08:40:213
1585899224,10cyclictest32338-21expr07:45:131
1585899221,3cyclictest29113-21taskset11:17:011
1585899221,3cyclictest12117-21tr10:40:121
1585899221,11cyclictest9302-21iwlist08:05:151
1585899221,10cyclictest25586-21munin-run12:25:021
1585899220,16cyclictest0-21swapper/109:16:491
1585299229,2cyclictest21139-21kworker/0:012:27:370
1585299229,2cyclictest1548-21kworker/0:207:58:210
1585299228,2cyclictest17444-21kworker/0:109:36:460
1585299228,2cyclictest17444-21kworker/0:109:36:460
1585299226,4cyclictest25969-21kworker/0:107:20:060
1585299226,4cyclictest24610-21kworker/0:012:02:220
1585299226,4cyclictest24610-21kworker/0:011:55:460
1585299226,4cyclictest24610-21kworker/0:011:55:460
1585299226,4cyclictest24321-21kworker/0:108:52:580
1585299226,4cyclictest24321-21kworker/0:108:46:380
15852992212,2cyclictest19343-21awk08:30:000
1585299221,10cyclictest26009-21kworker/0:007:31:100
310362210,4sleep01585299cyclictest11:20:260
1586999215,4cyclictest27361-21chrt08:45:233
15869992113,6cyclictest30546-21/usr/sbin/munin08:55:133
15869992113,4cyclictest14261-21/usr/sbin/munin09:30:163
15869992112,6cyclictest869-21sssd_nss12:00:133
15869992112,6cyclictest27903-21/usr/sbin/munin07:35:123
15869992112,6cyclictest24139-21/usr/sbin/munin12:20:123
15869992112,6cyclictest21755-21/usr/sbin/munin08:35:143
15869992112,6cyclictest20175-21/usr/sbin/munin09:45:193
15869992112,6cyclictest17361-21/usr/sbin/munin08:25:163
15858992113,3cyclictest0-21swapper/111:28:151
1585899211,18cyclictest0-21swapper/108:35:121
1585899211,17cyclictest15753-21missed_timers12:00:191
1585899211,16cyclictest25330-21tr11:10:111
1585899211,10cyclictest0-21swapper/107:20:191
1585299218,3cyclictest18154-21kworker/0:008:30:180
1585299218,2cyclictest1986-21kworker/0:209:08:500
1585299217,3cyclictest24610-21kworker/0:011:41:140
1585299217,2cyclictest12625-21kworker/0:208:14:450
1585299211,9cyclictest7599-21kworker/0:010:33:330
1585299211,9cyclictest18154-21kworker/0:008:37:020
1585299211,10cyclictest23085-21kworker/0:307:27:300
176982201,3sleep00-21swapper/007:10:570
15858992013,4cyclictest2318-21diskstats07:50:141
15858992011,2cyclictest32232-21ntpq10:10:191
1585299209,3cyclictest25196-21/usr/sbin/munin11:10:110
1585299206,3cyclictest6625-21kworker/0:009:15:100
1585299206,2cyclictest26429-21kworker/0:210:08:010
1585299206,2cyclictest21139-21kworker/0:012:22:010
1585299205,4cyclictest30135-21kworker/0:008:58:180
1585299205,4cyclictest21139-21kworker/0:012:38:250
1585299205,4cyclictest16633-21kworker/0:212:14:050
1585299204,4cyclictest24610-21kworker/0:011:38:540
15852992014,3cyclictest27361-21grep10:00:150
1585299201,11cyclictest13575-21/usr/sbin/munin09:30:240
1585299201,11cyclictest11026-21kworker/0:210:37:050
14252201,2sleep00-21swapper/007:46:080
1586299191,8cyclictest20650-21awk08:30:212
1586299191,6cyclictest20175-21/usr/sbin/munin09:45:192
1586299191,3cyclictest0-21swapper/211:50:212
1586299191,3cyclictest0-21swapper/209:05:192
15862991912,4cyclictest25521-21head08:40:252
15858991915,3cyclictest0-21swapper/108:22:271
1585899191,16cyclictest0-21swapper/112:30:111
1585299194,3cyclictest30135-21kworker/0:009:02:460
293342181,3sleep20-21swapper/211:19:522
28902181,2sleep20-21swapper/210:19:052
1586299181,9cyclictest4558-21diskmemload10:10:172
1586299181,9cyclictest2303-21df07:50:132
1586299181,7cyclictest30269-21df07:40:122
15862991813,3cyclictest28818-21grep08:50:142
15862991812,4cyclictest869-21sssd_nss11:10:222
15862991812,3cyclictest6158-21cpuspeed_turbos11:40:122
15862991812,3cyclictest15668-21grep10:45:262
15862991812,3cyclictest15306-21cpuspeed_turbos08:20:122
1585299185,4cyclictest788-21NetworkManager10:29:400
1585299185,4cyclictest29355-21kworker/0:107:38:420
1585299184,8cyclictest19835-40dnf09:43:490
1585299183,4cyclictest13684-21kworker/0:010:50:130
15852991812,4cyclictest0-21swapper/008:20:210
1586299177,7cyclictest14216-21gltestperf09:30:152
1586299175,3cyclictest32114-21systemd-journal12:39:282
1586299171,6cyclictest17710-21diskstats08:25:142
1586299171,6cyclictest0-21swapper/211:30:182
15862991712,2cyclictest22227-21fschecks_time08:35:132
1586299171,14cyclictest0-21swapper/208:00:072
15862991711,3cyclictest9816-21sendmail_mailtr08:05:232
15862991711,3cyclictest23339-21df_inode11:05:142
15862991711,3cyclictest17040-21cpu12:05:122
15862991711,3cyclictest11977-21date10:40:022
1586299171,13cyclictest11134-21cut08:10:142
1586299171,11cyclictest0-21swapper/212:15:202
1585299175,2cyclictest88650irq/135-enp2s0-10:55:410
1585299174,3cyclictest11026-21kworker/0:210:42:010
1585299172,12cyclictest1969-21/usr/sbin/munin07:50:150
1585299171,4cyclictest12964-21/usr/sbin/munin08:15:150
1585299171,3cyclictest19686-21kworker/0:009:55:050
15852991710,4cyclictest10747-21unixbench_singl09:20:270
1585299170,14cyclictest11909-21sort09:25:160
1586299169,4cyclictest15175-21/usr/sbin/munin12:00:152
1586299165,3cyclictest87150irq/126-enp1s0-07:55:232
1586299163,6cyclictest30-21rcuc/211:47:592
1586299162,11cyclictest31897-21tr11:25:132
1586299161,9cyclictest0-21swapper/210:00:272
1586299161,9cyclictest0-21swapper/207:30:162
1586299161,6cyclictest20791-21/usr/sbin/munin11:00:232
1586299161,2cyclictest0-21swapper/212:28:152
15862991612,3cyclictest0-21swapper/210:54:562
15862991612,3cyclictest0-21swapper/209:57:512
15862991612,3cyclictest0-21swapper/209:21:352
15862991612,3cyclictest0-21swapper/208:56:542
15862991612,3cyclictest0-21swapper/207:28:002
1586299161,14cyclictest0-21swapper/211:36:552
1586299161,14cyclictest0-21swapper/210:56:512
1586299161,14cyclictest0-21swapper/210:39:512
1586299161,14cyclictest0-21swapper/208:45:392
15862991611,3cyclictest31130-21unixbench_singl11:20:262
15862991611,3cyclictest30046-21ntp_states10:05:232
15862991611,3cyclictest22458-21cat09:50:122
15862991611,3cyclictest20477-21sensors_temp12:10:232
15862991611,3cyclictest19281-21expr07:15:142
15862991611,3cyclictest19281-21expr07:15:132
15862991611,3cyclictest13847-21sort08:15:202
15862991611,3cyclictest11963-21gltestperf09:25:162
1586299161,12cyclictest7081-21tr09:15:122
15862991610,4cyclictest0-21swapper/207:35:132
15862991610,3cyclictest522-21latency07:45:182
15862991610,3cyclictest16062-21/usr/sbin/munin07:10:262
1585299165,2cyclictest88650irq/135-enp2s0-10:22:370
1585299163,2cyclictest88550irq/134-enp2s0-10:17:370
1585299161,3cyclictest29932-21/usr/sbin/munin07:40:180
15852991611,3cyclictest9631-21ntp_states08:05:190
15852991611,3cyclictest31172-21/usr/sbin/munin10:10:210
15852991611,3cyclictest24346-21munin-run09:55:020
1586299159,3cyclictest18582-21gltestperf09:40:152
1586299159,3cyclictest0-21swapper/210:20:142
1586299154,3cyclictest5020-21cstates09:10:122
1586299153,3cyclictest21145-21/usr/sbin/munin07:20:112
1586299151,5cyclictest28007-21/usr/sbin/munin12:30:182
1586299151,4cyclictest13676-21ntp_states11:55:212
1586299151,4cyclictest13676-21ntp_states11:55:212
1586299141,6cyclictest0-21swapper/209:35:262
1586299141,6cyclictest0-21swapper/209:35:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional