You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-19 - 09:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Apr 19, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
223382239164,24sleep20-21swapper/219:09:112
223462218188,21sleep00-21swapper/019:09:170
223472215165,37sleep10-21swapper/119:09:181
222212204166,25sleep30-21swapper/319:07:433
22494999591,3cyclictest0-21swapper/222:13:092
16242600,2sleep011-21rcuc/023:15:010
188032570,1sleep30-21swapper/321:25:163
85972540,1sleep30-21swapper/321:00:243
294272510,1sleep10-21swapper/120:35:221
157062480,1sleep20-21swapper/221:20:012
39862470,1sleep30-21swapper/322:05:173
173042450,4sleep121-21rcuc/120:10:011
22499994313,18cyclictest415-40dnf23:10:233
2249999335,10cyclictest31358-21kworker/u8:000:10:033
2248099335,13cyclictest32560-21munin-run20:45:000
2249999326,13cyclictest808-21snmpd20:15:213
22499993220,7cyclictest17070-40dnf20:08:573
2248099324,13cyclictest25645-21idleruntime-cro00:10:010
2249999318,13cyclictest12582-21/usr/sbin/munin21:10:223
2249999315,12cyclictest22191-21sleep21:35:013
2249999307,10cyclictest2012-21/usr/sbin/munin23:15:123
2249999306,17cyclictest14857-21head22:30:143
2249999306,15cyclictest30610-21df00:20:133
2249999305,15cyclictest11205-21diskmemload23:25:153
2249999299,11cyclictest25642-21taskset20:26:003
2249999298,11cyclictest87650irq/129-enp1s0-23:50:223
2249999297,9cyclictest10565-21sh19:55:023
2249999296,15cyclictest29761-21timerandwakeup20:35:273
2249999296,13cyclictest31652-21cat23:10:013
2249999294,16cyclictest16434-21/usr/sbin/munin22:35:133
2249499291,6cyclictest0-21swapper/221:10:522
2248899291,16cyclictest0-21swapper/123:54:001
2248099291,13cyclictest25970-21munin-run20:30:010
22480992911,15cyclictest30990-21awk21:55:000
2249999287,12cyclictest12079-21sort19:55:243
2249999287,12cyclictest12079-21sort19:55:243
2249999286,6cyclictest11205-21diskmemload22:20:043
2249999285,14cyclictest2394-21latency_hist20:50:013
2249999285,11cyclictest11204-21munin-run21:10:013
2249999285,11cyclictest11204-21munin-run21:10:013
2249999284,11cyclictest30041-21turbostat.cron00:20:003
2249999283,10cyclictest19852-21gunzip22:40:213
22499992811,6cyclictest7646-21/usr/sbin/munin22:15:263
22499992811,6cyclictest7646-21/usr/sbin/munin22:15:263
2249499281,6cyclictest0-21swapper/222:09:582
2249499281,4cyclictest0-21swapper/220:35:262
2249999278,11cyclictest29741-21latency_hist19:25:023
2249999277,11cyclictest28777-21latency_hist21:50:023
2249999276,9cyclictest4825-21/usr/sbin/munin20:55:263
2249999276,6cyclictest11205-21diskmemload23:45:403
2249999275,9cyclictest12390-21diskstats22:25:143
2249999274,14cyclictest22999-21processes20:20:213
2249999273,15cyclictest2207-21proc_pri22:00:243
2249999273,14cyclictest8265-21/usr/sbin/munin23:30:233
2249999272,16cyclictest12973-21cstates23:40:133
22499992712,12cyclictest458-21sshd21:56:563
2249499271,7cyclictest0-21swapper/222:03:162
2249499271,5cyclictest0-21swapper/200:36:032
2248899270,24cyclictest0-21swapper/122:09:581
22480992716,3cyclictest0-21swapper/000:25:200
2249999268,6cyclictest23343-21cstates22:50:123
2249999268,11cyclictest29129-21runrttasks23:01:353
2249999263,8cyclictest26670-21cat21:45:023
2249999263,14cyclictest11205-21diskmemload21:19:333
2249499263,7cyclictest415-40dnf23:10:232
2249499261,5cyclictest0-21swapper/221:50:262
2249499261,5cyclictest0-21swapper/221:42:092
2248099262,11cyclictest22187-21munin-run21:35:010
22480992619,3cyclictest0-21swapper/021:10:520
2248099261,7cyclictest13673-21kworker/0:022:29:100
2249999256,7cyclictest827-21runrttasks22:58:123
2249999256,16cyclictest4604-21perf20:55:013
2249999255,10cyclictest664-21sed00:25:173
2249999251,15cyclictest827-21runrttasks00:35:243
2249999251,13cyclictest0-21swapper/319:30:153
2249499251,4cyclictest0-21swapper/200:14:062
2248899259,3cyclictest19371-21grep23:55:141
2248899251,4cyclictest0-21swapper/100:31:221
2248899250,3cyclictest0-21swapper/100:14:021
2248099256,10cyclictest0-21swapper/022:09:560
22480992518,3cyclictest0-21swapper/000:36:030
22480992516,3cyclictest0-21swapper/022:01:010
22480992515,3cyclictest0-21swapper/000:34:560
2249999243,10cyclictest87650irq/129-enp1s0-19:15:233
2249999242,10cyclictest0-21swapper/300:01:403
2249999241,11cyclictest0-21swapper/320:10:203
2249499246,10cyclictest0-21swapper/222:40:412
2249499243,5cyclictest26435-21kworker/u8:221:56:132
2249499241,7cyclictest28865-21grep23:00:272
2249499241,4cyclictest0-21swapper/223:42:582
2249499241,4cyclictest0-21swapper/200:25:202
2248899244,3cyclictest31808-21head21:55:151
22488992420,3cyclictest0-21swapper/120:40:241
22488992419,3cyclictest11841-21wc22:25:011
2248899241,4cyclictest6092-21users00:35:251
2248899241,11cyclictest22537-21tr21:35:121
2248899241,11cyclictest19880-21sendmail_mailtr22:40:231
2248899240,6cyclictest0-21swapper/122:51:481
2248899240,13cyclictest0-21swapper/100:07:361
22480992417,3cyclictest0-21swapper/022:19:000
22480992417,3cyclictest0-21swapper/022:19:000
22480992412,3cyclictest827-21runrttasks23:43:550
315362230,5sleep12248899cyclictest00:20:241
2249999239,3cyclictest26220-21cpu20:30:103
2249999235,5cyclictest2299-21cstates19:35:133
2249999235,4cyclictest87650irq/129-enp1s0-20:00:203
2249999234,7cyclictest29006-21/usr/sbin/munin21:50:153
2249999233,9cyclictest827-21runrttasks22:13:323
2249999233,6cyclictest87650irq/129-enp1s0-21:35:233
2249999233,12cyclictest24453-21sshd19:12:323
2249999233,11cyclictest6401-21/usr/sbin/munin19:45:143
2249999231,12cyclictest808-21snmpd20:40:243
22499992310,5cyclictest20276-21nfs4_client23:55:213
2249499233,7cyclictest32-21ksoftirqd/200:00:172
2249499233,6cyclictest30-21rcuc/222:30:242
2249499231,3cyclictest0-21swapper/222:35:082
2249499231,2cyclictest0-21swapper/222:19:002
2249499231,2cyclictest0-21swapper/222:19:002
2249499231,11cyclictest20440-21cstates21:30:132
2249499231,11cyclictest0-21swapper/221:00:112
2248899239,9cyclictest0-21swapper/122:49:451
2248899237,2cyclictest12680-21grep23:40:121
22488992318,2cyclictest2092-21ntpq22:00:211
2248899231,6cyclictest9156-21tail22:15:261
2248899231,6cyclictest9156-21tail22:15:261
2248899231,6cyclictest14469-21runrttasks21:15:181
2248899231,4cyclictest18355-21df_inode21:25:121
2248899231,2cyclictest0-21swapper/122:30:531
2248899231,19cyclictest32703-21expr00:25:141
2248899231,19cyclictest11512-21grep21:10:121
2248899231,16cyclictest0-21swapper/121:40:381
2248899231,10cyclictest684-21diskstats20:45:151
2248899230,6cyclictest0-21swapper/123:03:521
2248899230,5cyclictest0-21swapper/122:56:181
2248899230,5cyclictest0-21swapper/100:03:481
2248899230,3cyclictest0-21swapper/123:10:241
2248899230,2cyclictest87650irq/129-enp1s0-00:17:551
2248899230,17cyclictest0-21swapper/123:17:111
2248899230,15cyclictest0-21swapper/122:37:541
22480992316,3cyclictest0-21swapper/022:35:080
22480992313,5cyclictest0-21swapper/023:19:260
2248099231,11cyclictest3980-21munin-run19:40:000
2248099231,11cyclictest0-21swapper/019:27:330
2249999225,9cyclictest87650irq/129-enp1s0-23:35:533
2249999224,9cyclictest810-21sshd21:20:543
22499992213,5cyclictest4217-21diskstats23:20:143
22499992211,7cyclictest26219-21df_inode00:10:143
2249999221,11cyclictest0-21swapper/322:45:233
2249999220,10cyclictest0-21swapper/319:30:013
2249499225,4cyclictest17399-21/usr/sbin/munin21:20:272
2249499223,4cyclictest7782-21kworker/u8:123:34:142
2249499221,7cyclictest6116-21cat20:55:252
2249499221,7cyclictest28160-21fschecks_count19:20:152
2249499221,7cyclictest22637-21kworker/u8:223:20:242
2249499221,7cyclictest14221-21gzip20:00:242
2249499221,6cyclictest18314-21smart_sda23:50:242
2249499221,3cyclictest0-21swapper/223:18:542
2249499221,3cyclictest0-21swapper/200:06:092
2249499220,5cyclictest1615-1kworker/u9:100:34:342
2248899228,1cyclictest0-21swapper/119:15:131
2248899224,3cyclictest29850-21kernelversion21:50:171
22488992218,2cyclictest2139-21cpuspeed_turbos19:35:121
22488992218,2cyclictest0-21swapper/119:25:131
2248899221,3cyclictest17734-21df20:10:121
2248899221,3cyclictest16962-21pmu-power21:20:221
2248899221,18cyclictest12338-21cstates22:25:121
2248899221,13cyclictest11503-21ntpq23:35:201
2248899221,13cyclictest0-21swapper/123:22:421
2248899221,10cyclictest23758-21ntp_states19:10:211
2248899220,6cyclictest0-21swapper/123:27:211
2248899220,6cyclictest0-21swapper/120:50:121
2248899220,5cyclictest0-21swapper/123:47:361
2248099228,8cyclictest0-21swapper/022:23:340
22480992215,3cyclictest0-21swapper/000:14:060
22480992213,3cyclictest0-21swapper/021:21:280
2248099221,13cyclictest8265-21/usr/sbin/munin23:30:140
65562210,2sleep00-21swapper/019:45:130
2249999214,6cyclictest808-21snmpd19:40:143
22499992112,6cyclictest2309-21/usr/sbin/munin00:30:193
2249499218,2cyclictest30-21rcuc/219:25:242
2249499217,2cyclictest30-21rcuc/220:50:262
2249499215,4cyclictest32-21ksoftirqd/220:10:212
2249499214,5cyclictest25145-21ntpq19:15:032
2249499213,6cyclictest0-21swapper/223:35:412
2249499213,3cyclictest32-21ksoftirqd/223:49:132
2249499212,6cyclictest7908-21grep19:45:252
2249499212,6cyclictest23484-21ntp_states21:35:212
2249499212,5cyclictest22808-21munin-run22:50:012
2249499211,7cyclictest27506-21sort20:30:222
2249499211,6cyclictest808-21snmpd20:45:242
2249499211,6cyclictest31109-21grep23:05:252
2249499211,4cyclictest1615-1kworker/u9:123:59:392
2249499211,4cyclictest10913-21ntp_states22:20:222
2248899217,11cyclictest27983-21ps21:45:231
2248899213,9cyclictest32166-21/usr/sbin/munin19:30:121
22488992115,3cyclictest28840-21ntp_states19:20:221
2248899211,3cyclictest0-21swapper/122:12:401
2248899211,2cyclictest87650irq/129-enp1s0-23:32:021
2248899211,2cyclictest87650irq/129-enp1s0-19:40:151
2248899211,17cyclictest7079-21awk21:00:101
2248899211,17cyclictest25136-21head20:25:221
2248899211,17cyclictest11192-21crond21:10:011
2248899211,17cyclictest11192-21crond21:10:011
22488992110,3cyclictest14956-21cat20:04:561
2248899210,9cyclictest0-21swapper/120:15:111
2248899210,4cyclictest0-21swapper/123:05:391
2248099217,3cyclictest23091-21hddtemp_smartct21:35:160
2248099216,8cyclictest0-21swapper/022:42:010
2248099214,2cyclictest88950irq/132-enp2s0-20:24:490
22480992113,3cyclictest0-21swapper/023:59:370
22480992113,2cyclictest27950-21tr21:45:220
2248099211,12cyclictest29635-21/usr/sbin/munin23:05:120
22480992111,2cyclictest21971-21head22:45:220
22480992110,7cyclictest0-21swapper/021:43:590
2249499208,3cyclictest0-21swapper/221:46:312
2249499205,4cyclictest11205-21diskmemload22:55:512
2249499203,4cyclictest5761-21sed19:40:262
2249499202,6cyclictest6067-21/usr/sbin/munin23:25:122
2249499201,7cyclictest773-21NetworkManager20:21:292
2249499201,7cyclictest28410-21df00:15:122
2249499201,6cyclictest0-21swapper/222:50:462
2249499201,6cyclictest0-21swapper/221:28:552
2249499201,5cyclictest0-21swapper/222:27:242
2249499201,12cyclictest0-21swapper/200:20:142
2249499201,10cyclictest0-21swapper/219:56:592
2249499201,10cyclictest0-21swapper/219:56:592
2249499201,10cyclictest0-21swapper/219:34:122
22488992016,3cyclictest0-21swapper/120:30:571
22488992016,3cyclictest0-21swapper/120:23:081
22488992016,3cyclictest0-21swapper/119:59:311
22488992016,3cyclictest0-21swapper/119:59:311
22488992016,3cyclictest0-21swapper/119:54:381
22488992015,2cyclictest4996-21wc20:55:141
2248899200,5cyclictest0-21swapper/121:33:561
2248099209,3cyclictest0-21swapper/023:23:010
2248099206,5cyclictest0-21swapper/022:50:270
2248099202,15cyclictest26303-21grep20:30:100
2248099201,6cyclictest0-21swapper/022:59:220
2248099201,5cyclictest0-21swapper/021:56:070
22480992015,3cyclictest14138-21pmu-power20:00:240
2248099201,11cyclictest4203-21/usr/sbin/munin19:40:180
2248099200,5cyclictest0-21swapper/023:00:260
2248099200,18cyclictest0-21swapper/022:34:290
2249499196,4cyclictest9742-21pmu-power19:50:222
2249499194,3cyclictest87550irq/128-enp1s0-19:35:142
2249499193,3cyclictest16061-21sed20:05:192
2249499191,5cyclictest23996-21/usr/sbin/munin20:25:122
22494991913,4cyclictest21145-21grep20:15:252
22494991913,3cyclictest10552-21switchtime21:05:252
22494991913,3cyclictest10552-21switchtime21:05:252
2249499190,10cyclictest0-21swapper/219:12:242
22488991915,3cyclictest0-21swapper/119:45:171
2248099196,6cyclictest0-21swapper/023:50:180
2248099194,3cyclictest0-21swapper/021:25:080
2248099191,9cyclictest0-21swapper/023:26:510
2248099191,8cyclictest87550irq/128-enp1s0-19:55:180
2248099191,8cyclictest87550irq/128-enp1s0-19:55:180
2248099191,5cyclictest0-21swapper/021:15:160
22480991912,2cyclictest0-21swapper/023:35:220
2248099191,16cyclictest0-21swapper/023:45:100
2248099191,15cyclictest6508-21ntp_states22:10:200
2248099191,10cyclictest21447-21/usr/sbin/munin00:00:200
2248099190,4cyclictest0-21swapper/019:50:240
2248099190,15cyclictest0-21swapper/000:24:310
2249499181,10cyclictest0-21swapper/220:40:302
2248099187,1cyclictest0-21swapper/019:20:240
22480991813,3cyclictest28811-21if_enp1s000:15:160
2248099181,14cyclictest9837-21tail21:05:160
2248099181,14cyclictest9837-21tail21:05:160
22480991811,2cyclictest0-21swapper/020:35:260
2248099171,15cyclictest0-21swapper/020:05:500
2248099171,13cyclictest18388-21needreboot20:10:180
22480991710,4cyclictest1460-21ntpq20:45:200
2248099170,4cyclictest0-21swapper/019:30:180
2248099166,3cyclictest25580-21/usr/sbin/munin19:15:250
2248099162,12cyclictest22688-21/usr/sbin/munin19:10:240
22480991611,3cyclictest810-21sshd20:17:550
2248099161,12cyclictest7016-21/usr/sbin/munin21:00:110
2248099161,12cyclictest4825-21/usr/sbin/munin20:55:150
2248099153,3cyclictest2624-21/usr/sbin/munin20:50:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional