You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-19 - 23:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Sep 19, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
145482238163,24sleep20-21swapper/207:07:542
146462236162,24sleep10-21swapper/107:09:101
146502232164,56sleep00-21swapper/007:09:130
144252202165,24sleep30-21swapper/307:06:183
14808998986,2cyclictest0-21swapper/207:32:382
14808996663,2cyclictest0-21swapper/212:06:002
153662590,2sleep20-21swapper/208:20:252
14808995753,3cyclictest0-21swapper/210:08:152
154672500,4sleep21480899cyclictest12:01:342
1479899386,23cyclictest5629-21/usr/sbin/munin10:25:250
1479899386,23cyclictest5629-21/usr/sbin/munin10:25:250
1479899381,24cyclictest16992-21cp07:15:020
1479899371,28cyclictest27820-21cat10:05:010
1479899371,21cyclictest3510-21diskmemload12:30:150
1479899361,26cyclictest24142-21/usr/sbin/munin11:10:130
1479899361,24cyclictest10419-21cpuspeed_turbos09:25:120
1479899361,22cyclictest18168-21/usr/sbin/munin12:10:260
1479899355,21cyclictest87550irq/128-enp1s0-07:50:240
1479899351,26cyclictest0-21swapper/011:15:190
1479899351,23cyclictest32500-21cstates10:15:150
1479899351,23cyclictest30551-21sed11:25:010
1479899341,25cyclictest2116-21unixbench_singl11:30:290
1479899341,24cyclictest8964-21cstates10:35:150
1479899341,23cyclictest8130-21/usr/sbin/munin09:20:280
1479899341,23cyclictest17934-21latency_hist12:10:020
1479899341,22cyclictest7739-21ls08:05:140
1479899341,22cyclictest7739-21ls08:05:130
1479899341,22cyclictest15346-21/usr/sbin/munin10:50:150
1479899341,20cyclictest27295-21/usr/sbin/munin08:50:130
1479899341,20cyclictest25097-21/usr/sbin/munin08:45:150
1479899341,17cyclictest0-21swapper/011:06:190
1479899337,18cyclictest0-21swapper/012:35:200
1479899331,24cyclictest26023-21/usr/sbin/munin12:25:260
1479899323,20cyclictest24477-21/usr/sbin/munin07:30:180
1479899321,28cyclictest7679-21ls10:30:230
1479899321,28cyclictest11614-21sshd08:14:200
1479899321,24cyclictest3010-21latency10:20:190
1479899321,20cyclictest22475-21proc_pri09:50:210
1479899321,20cyclictest10238-21latency11:50:180
1479899321,19cyclictest3501-21idleruntime-cro09:10:010
1479899321,19cyclictest20553-21/usr/sbin/munin11:00:180
1479899321,18cyclictest14397-21mailstats10:45:240
1479899320,18cyclictest0-21swapper/008:15:180
1479899312,26cyclictest18502-21/usr/sbin/munin08:30:240
1479899312,24cyclictest87350irq/126-enp1s0-09:55:200
14798993113,15cyclictest13761-21/usr/sbin/munin12:00:250
1479899311,27cyclictest16921-21/usr/sbin/munin09:40:240
1479899311,26cyclictest909-21sssd_nss11:38:190
1479899311,26cyclictest909-21sssd_nss11:38:190
1479899311,22cyclictest30120-21/usr/sbin/munin10:10:140
1479899311,22cyclictest0-21swapper/011:40:130
1479899310,27cyclictest11902-21df11:55:150
1479899310,22cyclictest16305-21/usr/sbin/munin08:25:170
1479899310,20cyclictest0-21swapper/009:35:210
1479899303,20cyclictest87350irq/126-enp1s0-09:10:120
1479899302,24cyclictest5937-21/usr/sbin/munin09:15:160
1479899302,12cyclictest5750irq/9-acpi09:00:240
1479899301,25cyclictest31097-21forks11:25:160
1479899301,21cyclictest87550irq/128-enp1s0-07:25:180
1479899301,19cyclictest87350irq/126-enp1s0-07:55:160
1479899301,18cyclictest87350irq/126-enp1s0-07:15:130
1479899301,17cyclictest22418-21cat12:20:020
14798993010,17cyclictest20503-21fschecks_count07:20:150
14815992910,10cyclictest0-21swapper/309:50:123
1479899292,24cyclictest18359-21irqstats10:55:200
1479899291,25cyclictest5306-21/usr/sbin/munin08:00:130
1479899291,25cyclictest31080-21/usr/sbin/munin07:45:260
1479899291,25cyclictest27922-21/usr/sbin/munin10:05:240
1479899291,25cyclictest26679-21/usr/sbin/munin07:35:160
1479899291,25cyclictest19123-21/usr/sbin/munin09:45:160
1479899291,25cyclictest17113-21munin-node07:40:120
1479899291,25cyclictest14107-21/usr/sbin/munin08:20:260
1479899291,25cyclictest12223-21/usr/sbin/munin10:40:270
1479899291,21cyclictest87350irq/126-enp1s0-12:24:490
1479899290,26cyclictest29494-21/usr/sbin/munin08:55:190
1479899290,15cyclictest87650irq/129-enp1s0-09:30:140
1479899281,24cyclictest7519-21entropy11:45:140
1479899281,24cyclictest23248-21diskstats08:40:160
1479899281,14cyclictest87550irq/128-enp1s0-08:35:150
1480899273,3cyclictest12050irq/123-ahci[0012:00:012
14804992721,3cyclictest29259-21latency_hist08:55:021
1480899266,4cyclictest22365-21fschecks_count11:05:152
1480499261,19cyclictest0-21swapper/111:22:371
1480499261,19cyclictest0-21swapper/110:11:281
81442250,4sleep11480499cyclictest11:45:191
180462250,1sleep30-21swapper/312:10:023
1481599257,4cyclictest0-21swapper/312:15:243
1480499251,18cyclictest0-21swapper/112:08:321
1481599241,5cyclictest4742-21dump-pmu-power11:40:023
1481599241,5cyclictest4742-21dump-pmu-power11:40:023
1480499241,6cyclictest839-21lldpd11:05:181
245782230,4sleep11480499cyclictest07:30:121
1481599239,3cyclictest23315-21ntp_kernel_err07:25:203
1481599238,3cyclictest5410-21sshd09:11:203
1481599231,11cyclictest475-21/usr/sbin/munin11:30:283
1481599231,11cyclictest30735-21/usr/sbin/munin11:25:153
1481599231,11cyclictest28871-21df11:20:143
1481599231,11cyclictest20176-21sshd08:31:203
1480499231,9cyclictest24668-21pmu-power09:55:241
1480499231,17cyclictest0-21swapper/112:17:281
1480499231,17cyclictest0-21swapper/110:56:011
1480499231,17cyclictest0-21swapper/110:39:391
1480499231,17cyclictest0-21swapper/110:15:341
41622220,5sleep230-21rcuc/207:55:212
244022220,4sleep230-21rcuc/208:40:282
1481599228,3cyclictest19123-21/usr/sbin/munin09:45:203
1481599228,3cyclictest14722-21/usr/sbin/munin09:35:253
1481599226,5cyclictest755-21rngd08:36:263
1481599222,9cyclictest12301-21munin-run09:30:013
1481599221,2cyclictest457-21gzip09:00:223
1481599221,2cyclictest25857-21chrt08:45:183
1481599221,14cyclictest0-21swapper/311:00:243
1481599221,10cyclictest808-21snmpd10:50:313
1481599221,10cyclictest16921-21/usr/sbin/munin09:40:213
1481599220,10cyclictest2176-21cpu10:20:123
1480899221,11cyclictest26341-21/usr/sbin/munin11:15:222
1480499226,4cyclictest23-21ksoftirqd/111:55:221
14804992216,3cyclictest28934-21netstat10:05:191
1480499221,3cyclictest26707-21cpu07:35:111
1480499221,16cyclictest0-21swapper/112:38:111
1480499221,11cyclictest23009-21cpuspeed_turbos08:40:161
1480499221,11cyclictest16974-21perf07:15:011
14804992210,3cyclictest20711-21users09:45:251
1480499220,16cyclictest0-21swapper/108:20:241
1481599216,3cyclictest87350irq/126-enp1s0-11:45:173
1481599212,5cyclictest30068-21memory12:35:203
1481599211,6cyclictest0-21swapper/309:18:393
1480899216,4cyclictest0-21swapper/212:21:292
1480899211,6cyclictest0-21swapper/209:22:382
1480499212,13cyclictest87650irq/129-enp1s0-10:45:161
1480499211,8cyclictest22284-21/usr/sbin/munin07:25:161
1480499211,18cyclictest0-21swapper/109:04:161
1480499211,15cyclictest0-21swapper/109:28:191
1480499211,14cyclictest0-21swapper/109:54:361
1480499211,14cyclictest0-21swapper/108:45:571
1480499211,14cyclictest0-21swapper/107:50:041
1480499211,14cyclictest0-21swapper/107:48:131
1480499211,12cyclictest12852-40dnf10:44:221
1480499210,16cyclictest0-21swapper/111:32:281
1480499210,15cyclictest0-21swapper/109:18:201
192182201,15sleep219311-21irqstats08:30:172
1481599209,3cyclictest0-21swapper/311:55:233
1481599208,3cyclictest26531-21irqstats10:00:213
1481599205,6cyclictest9663-21memory10:35:203
1481599205,4cyclictest3510-21diskmemload11:14:063
1481599205,4cyclictest3510-21diskmemload10:40:573
1481599204,4cyclictest3510-21diskmemload09:57:373
1481599203,13cyclictest39-21rcuc/308:50:193
1481599202,15cyclictest1920-21idleruntime-cro10:20:013
1481599201,10cyclictest16561-21sed07:10:263
1480899207,2cyclictest311ktimersoftd/210:00:002
1480899202,9cyclictest30-21rcuc/210:30:012
1480899202,9cyclictest30-21rcuc/210:30:012
14808992015,3cyclictest32-21ksoftirqd/209:55:012
14808992015,3cyclictest17515-21diskstats07:15:032
1480899201,12cyclictest0-21swapper/210:59:172
1480499209,3cyclictest23630-21ntp_states12:20:221
1480499208,4cyclictest3875-21/usr/sbin/munin11:35:241
1480499208,4cyclictest3875-21/usr/sbin/munin11:35:241
1480499201,6cyclictest2538-21fschecks_count10:20:151
14804992016,2cyclictest21747-21/usr/sbin/munin08:35:221
1480499201,5cyclictest25627-21unixbench-2d11:10:271
14804992015,2cyclictest19241-21ntp_states12:10:221
1480499201,2cyclictest0-21swapper/108:33:421
1480499201,14cyclictest0-21swapper/112:29:581
1480499201,14cyclictest0-21swapper/111:27:081
1480499201,14cyclictest0-21swapper/109:41:141
1480499201,14cyclictest0-21swapper/109:09:411
1480499201,14cyclictest0-21swapper/108:26:381
1480499201,14cyclictest0-21swapper/108:16:011
1480499201,14cyclictest0-21swapper/108:11:211
1480499201,11cyclictest8130-21/usr/sbin/munin09:20:261
1480499201,11cyclictest29844-21diskstats08:55:161
1480499201,11cyclictest29764-21latency07:40:181
1480499201,10cyclictest15714-21cut10:50:171
1481599197,3cyclictest808-21snmpd07:42:383
1481599194,4cyclictest13536-21fschecks_count10:45:153
1481599192,4cyclictest30566-21ntp_states08:55:233
1481599191,6cyclictest0-21swapper/312:10:153
1481599191,5cyclictest3503-21latency_hist09:10:013
1481599191,4cyclictest0-21swapper/310:25:223
1481599191,4cyclictest0-21swapper/310:25:223
1481599191,2cyclictest13386-21sed08:15:253
1481599191,11cyclictest0-21swapper/312:20:173
1481599191,10cyclictest12528-21/usr/sbin/munin09:30:263
1481599191,10cyclictest0-21swapper/308:40:263
1481599190,12cyclictest0-21swapper/310:05:173
1480899198,3cyclictest87350irq/126-enp1s0-12:25:152
1480899195,6cyclictest30-21rcuc/210:48:502
1480899194,8cyclictest6932-40dnf11:45:032
1480899192,6cyclictest28928-21munin-run12:35:012
1480899192,12cyclictest87350irq/126-enp1s0-11:25:152
1480899191,8cyclictest9640-21cat08:10:012
1480899191,8cyclictest9640-21cat08:10:012
1480899191,8cyclictest790-21in:imjournal09:01:012
1480899191,3cyclictest18268-21timerandwakeup09:40:242
14808991911,5cyclictest19162-21strings12:10:212
1480899191,10cyclictest0-21swapper/211:30:282
1480499196,11cyclictest4732-21crond11:40:011
1480499191,5cyclictest14440-21hddtemp_smartct12:00:191
1480499191,5cyclictest0-21swapper/110:01:111
14804991915,3cyclictest0-21swapper/108:00:261
14804991915,3cyclictest0-21swapper/107:19:141
1480499191,16cyclictest3510-21diskmemload10:28:331
1480499191,16cyclictest3510-21diskmemload10:28:331
1480499191,16cyclictest0-21swapper/111:53:031
1480499191,11cyclictest0-21swapper/112:30:241
1480499191,10cyclictest7861-21diskstats08:05:141
1480499191,10cyclictest7861-21diskstats08:05:141
1480499191,10cyclictest26656-21expr11:15:131
1480499191,10cyclictest12872-21df_inode09:30:151
1480499190,18cyclictest0-21swapper/111:00:161
1480499190,18cyclictest0-21swapper/109:35:181
1480499190,17cyclictest0-21swapper/110:30:131
1480499190,14cyclictest0-21swapper/107:59:081
1481599186,3cyclictest9893-21/usr/sbin/munin08:10:133
1481599185,4cyclictest3510-21diskmemload09:21:183
1481599181,9cyclictest0-21swapper/307:55:283
1481599181,7cyclictest26721-21idleruntime-cro12:30:003
1481599181,7cyclictest14095-21df12:00:153
1481599181,7cyclictest0-21swapper/310:11:323
1481599181,5cyclictest23010-21ntp_states11:05:213
1481599181,5cyclictest19855-21latency_hist07:20:013
1481599181,4cyclictest839-21lldpd07:47:173
1481599180,6cyclictest0-21swapper/307:39:423
1480899184,6cyclictest755-21rngd11:10:082
1480899184,2cyclictest311ktimersoftd/210:35:012
1480899183,4cyclictest0-21swapper/209:13:082
1480899181,8cyclictest0-21swapper/207:10:122
1480899181,7cyclictest10417-21ntpq11:50:202
1480899181,6cyclictest0-21swapper/211:43:562
1480899181,6cyclictest0-21swapper/209:29:332
1480899181,4cyclictest0-21swapper/210:20:552
14808991813,3cyclictest25097-21/usr/sbin/munin08:45:132
1480899181,2cyclictest0-21swapper/209:35:462
14808991812,3cyclictest12082-21grep10:40:252
1480899181,15cyclictest0-21swapper/210:50:392
1480899181,15cyclictest0-21swapper/209:07:532
1480899181,15cyclictest0-21swapper/208:29:072
1480899181,11cyclictest0-21swapper/211:36:122
1480899181,11cyclictest0-21swapper/211:36:122
1480499181,9cyclictest21074-21strings07:20:191
14804991811,3cyclictest0-21swapper/109:14:451
1481599173,4cyclictest21154-21ntp_states07:20:213
1481599171,7cyclictest3510-21diskmemload10:34:573
1481599171,6cyclictest39-21rcuc/311:50:253
1481599171,6cyclictest27545-21grep12:30:163
1481599171,6cyclictest26626-21cat07:35:033
1481599171,6cyclictest17551-21sendmail_mailst08:25:233
14815991712,3cyclictest1726-21interrupts07:50:203
14815991712,2cyclictest17545-21/usr/sbin/munin10:55:153
1481599171,13cyclictest0-21swapper/308:20:203
14815991710,4cyclictest5883-21missed_timers11:40:203
1481599170,6cyclictest27334-21needreboot11:15:203
1481599170,15cyclictest0-21swapper/308:00:183
1480899175,4cyclictest6073-21if_err_enp1s008:00:162
1480899174,2cyclictest30-21rcuc/207:30:022
1480899173,6cyclictest87650irq/129-enp1s0-09:30:152
1480899173,5cyclictest0-21swapper/210:01:072
1480899173,3cyclictest32-21ksoftirqd/208:20:012
1480899172,12cyclictest26825-21tr07:35:122
1480899171,7cyclictest0-21swapper/212:39:272
1480899171,6cyclictest9090-21df_inode10:35:152
1480899171,6cyclictest31502-21timerandwakeup10:10:252
1480899171,6cyclictest20078-21df11:00:132
1480899171,6cyclictest0-21swapper/210:16:242
1480899171,6cyclictest0-21swapper/209:15:172
1480899171,5cyclictest0-21swapper/211:22:582
1480899171,4cyclictest0-21swapper/209:47:222
14808991713,3cyclictest0-21swapper/208:51:542
14808991712,3cyclictest20364-21acpi12:15:122
1480899171,14cyclictest0-21swapper/208:55:182
14815991610,3cyclictest9082-21awk08:05:263
14815991610,3cyclictest9082-21awk08:05:263
1480899161,2cyclictest0-21swapper/207:45:122
14808991611,3cyclictest9711-21/usr/sbin/munin08:10:182
14808991611,3cyclictest29736-21cat07:40:182
14808991611,3cyclictest20699-21/usr/sbin/munin08:35:232
14808991610,3cyclictest960-21cpuspeed_turbos07:50:142
1480899159,3cyclictest20268-21cstates07:20:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional