You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-05 - 14:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Sep 05, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
311552253164,23sleep00-21swapper/007:06:030
312512242163,24sleep20-21swapper/207:07:182
312732235166,56sleep30-21swapper/307:07:353
312352232166,54sleep10-21swapper/107:07:051
662021480,3sleep23155299cyclictest09:50:232
1291421200,5sleep33155999cyclictest08:50:243
301172580,2sleep20-21swapper/210:45:132
141852530,2sleep20-21swapper/207:40:192
31552995148,2cyclictest0-21swapper/209:31:032
3155299419,27cyclictest909-21sssd_nss12:15:252
31538993910,13cyclictest7677-21munin-run09:55:010
3155299372,7cyclictest22392-40dnf09:14:242
3155299372,7cyclictest22392-40dnf09:14:242
3155299359,3cyclictest31469-21/usr/sbin/munin09:35:232
3154699351,28cyclictest0-21swapper/112:34:461
31552993410,3cyclictest30508-21/usr/sbin/munin12:00:132
31552993410,3cyclictest22684-21/usr/sbin/munin09:15:152
3153899335,13cyclictest13320-21munin-run12:35:010
3155299329,3cyclictest827-21runrttasks08:42:072
3155299325,6cyclictest909-21sssd_nss10:15:012
3155299321,7cyclictest579-21systemd-journal08:05:012
31552993212,3cyclictest25496-21/usr/sbin/munin10:35:232
3153899327,11cyclictest13052-21perf07:40:010
3155299319,3cyclictest827-21runrttasks12:32:262
3155299319,3cyclictest5091-21/usr/sbin/munin08:35:242
3155299314,4cyclictest18442-21cat07:50:162
3155299311,5cyclictest24154-21wc10:30:212
3155299311,4cyclictest87350irq/126-enp1s0-08:50:222
31552993112,3cyclictest25527-21smartctl08:05:252
31552993111,3cyclictest8358-21timerwakeupswit12:20:282
31552993111,3cyclictest21115-21sendmail_mailtr07:55:242
31552993110,3cyclictest29657-21latency_hist10:45:022
31552993110,3cyclictest18773-21cat11:30:262
3155299310,25cyclictest0-21swapper/211:52:272
3153899315,12cyclictest28446-21perf08:15:010
3155299308,4cyclictest16496-21/usr/sbin/munin11:25:252
3155299308,4cyclictest16496-21/usr/sbin/munin11:25:252
3155299308,3cyclictest10106-21/usr/sbin/munin10:00:182
31552993012,3cyclictest27990-21memory09:25:202
31552993012,3cyclictest1027-21dnf08:25:162
31552993011,3cyclictest18247-21date09:05:012
31552993011,2cyclictest1244-21sed12:05:192
3155299301,10cyclictest0-21swapper/211:57:112
31552993010,2cyclictest8539-21/usr/sbin/munin11:10:212
31552993010,2cyclictest3667-21if_err_enp1s008:30:192
3154699306,17cyclictest87550irq/128-enp1s0-09:55:151
3154699301,23cyclictest27227-21kworker/u8:210:51:421
3155299299,3cyclictest3570-21cpu09:45:132
3155299299,3cyclictest1537-21crond07:25:012
3155299299,3cyclictest12426-21timerandwakeup07:35:242
3155299298,3cyclictest15518-21pluginstate12:40:012
3155299295,3cyclictest5750irq/9-acpi08:20:192
3155299294,4cyclictest18449-21chrt10:17:162
3155299292,10cyclictest9292-21grep07:30:162
3155299291,5cyclictest10736-21/usr/sbin/munin11:15:282
3155299291,4cyclictest12400-21ls10:05:132
31552992912,3cyclictest26460-21/usr/sbin/munin08:10:202
31552992912,3cyclictest21782-21sed10:25:142
3155299291,14cyclictest0-21swapper/210:20:022
31552992911,3cyclictest32097-21/usr/sbin/munin10:50:242
3155299291,11cyclictest0-21swapper/208:47:392
31552992910,4cyclictest6686-21diskstats11:05:152
31552992910,3cyclictest32288-21fschecks_time07:10:162
3154699290,23cyclictest0-21swapper/110:48:031
31538992911,15cyclictest2681-21awk08:30:010
3155299289,4cyclictest6113-21/usr/sbin/munin11:05:012
3155299289,3cyclictest28838-21cstates08:15:112
3155299289,3cyclictest22443-21if_enp1s011:40:172
3155299289,2cyclictest24859-21missed_timers11:45:202
3155299289,2cyclictest11117-21latency_hist12:30:012
3155299281,5cyclictest0-21swapper/211:24:442
3155299281,3cyclictest12050irq/123-ahci[0010:00:012
3155299281,13cyclictest0-21swapper/208:57:592
3155299281,13cyclictest0-21swapper/207:29:032
3154699281,21cyclictest0-21swapper/110:11:131
3153899280,13cyclictest17112-21perf11:30:010
3153899280,13cyclictest17112-21perf11:30:010
3155999273,16cyclictest0-21swapper/310:11:553
3155999271,18cyclictest0-21swapper/310:15:503
3155299278,3cyclictest3698-21cat07:15:242
3155299278,3cyclictest25779-21cat09:20:182
3155299271,8cyclictest3380-21kernelversion12:10:192
3155299271,6cyclictest1294-21/usr/sbin/munin09:40:132
31552992712,3cyclictest0-21swapper/211:39:192
31552992710,3cyclictest4112-21sed11:00:022
3154699271,21cyclictest0-21swapper/112:26:171
3155999263,17cyclictest0-21swapper/309:18:403
3155999261,17cyclictest6629-21memory09:50:233
3155999261,17cyclictest30852-21sed10:45:193
3155999261,15cyclictest0-21swapper/310:30:273
3154699261,22cyclictest0-21swapper/109:50:261
3154699261,18cyclictest9-21rcu_preempt12:00:201
3154699261,12cyclictest7770-21irqstats12:20:211
3154699260,23cyclictest9501-21cpu08:45:121
3155999253,15cyclictest0-21swapper/310:03:273
3155999251,18cyclictest0-21swapper/311:41:163
3155999251,18cyclictest0-21swapper/311:32:103
3155299259,3cyclictest0-21swapper/209:05:592
31546992523,1cyclictest0-21swapper/111:11:421
3154699252,19cyclictest6113-21munin-run11:05:011
3154699252,17cyclictest25868-21kworker/u8:011:50:251
3154699251,21cyclictest28436-21perf08:15:011
3154699251,19cyclictest0-21swapper/110:20:101
3154699251,12cyclictest4704-21cat12:15:011
3154699251,11cyclictest14609-21ntp_states12:35:231
31538992511,2cyclictest997-21kworker/0:307:12:450
31538992510,2cyclictest22124-21kworker/0:109:14:310
31538992510,2cyclictest22124-21kworker/0:109:14:310
3155999241,7cyclictest39-21rcuc/310:40:153
3155999241,17cyclictest0-21swapper/311:35:073
3155999241,17cyclictest0-21swapper/311:04:413
3155999241,17cyclictest0-21swapper/310:54:213
3155999241,16cyclictest0-21swapper/312:01:303
3155999241,13cyclictest0-21swapper/309:10:193
3155999241,13cyclictest0-21swapper/309:10:193
3154699249,3cyclictest22160-21ntp_states10:25:211
3154699249,3cyclictest21481-21crond11:40:001
3154699249,3cyclictest11204-21cpuspeed_turbos07:35:131
3154699243,10cyclictest0-21swapper/109:50:011
3154699241,20cyclictest6354-21vmstat12:15:271
3154699241,20cyclictest2237-21expr10:55:141
3154699241,20cyclictest19761-21latency_hist07:55:011
3154699241,12cyclictest0-21swapper/109:25:271
3154699241,11cyclictest29657-21latency_hist10:45:021
3154699241,11cyclictest1770-21ntp_states08:25:231
3154699241,11cyclictest16865-21chrt11:27:101
3154699241,11cyclictest16865-21chrt11:27:101
3154699241,11cyclictest13646-21latency_hist08:55:011
3154699241,11cyclictest10453-21diskstats10:00:151
3153899248,4cyclictest23750-21kworker/0:308:07:430
3153899248,2cyclictest9830-21kworker/0:110:25:210
3153899241,11cyclictest18285-21kworker/0:009:15:550
3155999238,9cyclictest0-21swapper/312:09:233
3155999238,3cyclictest16455-21strings07:45:203
31559992317,3cyclictest27936-21grep08:10:253
3155999231,17cyclictest0-21swapper/310:26:533
3155999231,16cyclictest87350irq/126-enp1s0-10:08:513
3155999231,16cyclictest0-21swapper/312:35:433
3155999231,16cyclictest0-21swapper/311:25:053
3155999231,16cyclictest0-21swapper/311:25:053
3155999231,16cyclictest0-21swapper/311:08:073
3155999231,16cyclictest0-21swapper/310:37:263
3155999231,15cyclictest0-21swapper/309:48:583
3155999231,15cyclictest0-21swapper/309:21:283
3155999231,14cyclictest0-21swapper/310:55:423
3155999230,17cyclictest0-21swapper/312:16:133
3155999230,17cyclictest0-21swapper/311:45:143
3154699238,5cyclictest827-21runrttasks09:20:481
3154699234,3cyclictest15184-21kworker/u8:011:30:531
3154699232,19cyclictest0-21swapper/109:00:321
3154699231,5cyclictest939-21grep12:05:171
3154699231,20cyclictest0-21swapper/108:20:231
3154699231,19cyclictest7629-21df_inode08:40:141
3154699231,19cyclictest30631-21perf08:20:011
3154699231,14cyclictest0-21swapper/110:30:071
3154699231,10cyclictest8557-21sshd07:29:031
3154699231,10cyclictest1825-21df_inode07:15:031
3154699231,10cyclictest0-21swapper/111:46:311
3153899232,10cyclictest23585-21kworker/0:011:56:070
31538992311,3cyclictest5817-21kworker/0:311:08:150
3153899231,10cyclictest30894-21kworker/0:209:31:310
3153899231,10cyclictest23585-21kworker/0:012:00:230
3155999222,11cyclictest28658-21diskstats11:55:143
31559992217,3cyclictest10992-21grep08:45:253
3155999221,15cyclictest0-21swapper/308:17:513
3155999221,12cyclictest0-21swapper/312:31:043
3155999221,12cyclictest0-21swapper/312:21:413
3155999221,10cyclictest28068-21cut09:25:203
3155999220,16cyclictest87350irq/126-enp1s0-11:54:423
3155299227,3cyclictest0-21swapper/207:45:022
3154699229,2cyclictest0-21swapper/111:15:031
3154699228,4cyclictest0-21swapper/110:09:481
3154699228,3cyclictest23-21ksoftirqd/109:40:231
3154699223,9cyclictest0-21swapper/110:15:231
3154699222,19cyclictest0-21swapper/108:38:211
3154699222,11cyclictest0-21swapper/108:08:031
3154699221,20cyclictest0-21swapper/107:40:461
3154699221,19cyclictest21562-21awk09:10:231
3154699221,19cyclictest21562-21awk09:10:231
3154699221,19cyclictest0-21swapper/107:22:411
3154699221,19cyclictest0-21swapper/107:14:591
3154699221,10cyclictest27457-21latency_hist10:40:021
3154699221,10cyclictest0-21swapper/111:42:121
3154699221,10cyclictest0-21swapper/109:05:161
3154699221,10cyclictest0-21swapper/107:55:181
3154699220,20cyclictest0-21swapper/108:30:231
3154699220,13cyclictest23125-21tune2fs09:15:161
3153899226,2cyclictest9830-21kworker/0:110:25:010
3153899222,10cyclictest12361-21kworker/0:211:19:180
3153899221,9cyclictest3059-21kworker/0:009:55:590
3153899221,11cyclictest21311-21kworker/0:111:45:110
3153899221,10cyclictest8256-21kworker/0:111:11:110
3153899221,10cyclictest27281-21kworker/0:110:41:590
3153899220,10cyclictest26714-21kworker/0:009:28:270
3155999216,7cyclictest0-21swapper/311:23:153
3155999214,4cyclictest2672-21cpuspeed_turbos12:10:153
3155999212,10cyclictest603-21/usr/sbin/munin08:25:133
3155999211,7cyclictest20919-21ntp_offset07:55:213
31559992117,3cyclictest0-21swapper/309:05:273
3155999211,12cyclictest0-21swapper/310:20:433
3155999211,11cyclictest8357-21ntp_states08:40:223
3155999211,11cyclictest3346-21ntp_states07:15:203
31559992110,7cyclictest0-21swapper/309:30:223
3154699219,2cyclictest21-21rcuc/111:20:121
3154699216,3cyclictest30182-21/usr/sbin/munin09:30:191
3154699211,19cyclictest0-21swapper/111:59:431
3154699211,19cyclictest0-21swapper/107:46:081
3154699211,18cyclictest0-21swapper/108:03:541
3154699210,2cyclictest0-21swapper/107:30:261
3154699210,19cyclictest0-21swapper/111:09:121
3153899216,4cyclictest6536-21kworker/0:012:20:270
3153899212,9cyclictest31650-21kworker/0:010:49:110
3153899211,12cyclictest1932-21/usr/sbin/munin10:55:210
3153899211,11cyclictest4479-21/usr/sbin/munin07:20:170
3153899211,11cyclictest2897-21/usr/sbin/munin08:30:130
3153899211,11cyclictest10091-21sshd10:00:030
3153899211,10cyclictest24493-21kworker/0:209:21:270
3155999206,3cyclictest24031-21sh08:05:013
3155999203,4cyclictest808-21snmpd07:31:083
3155999202,10cyclictest16081-21/usr/sbin/munin09:00:213
3155999202,10cyclictest11072-21/usr/sbin/munin07:35:153
31559992018,1cyclictest0-21swapper/307:10:243
3155999201,7cyclictest14823-21missed_timers08:55:193
3155999201,5cyclictest15354-21cat07:45:013
3155999201,5cyclictest0-21swapper/309:44:583
3155999201,13cyclictest0-21swapper/312:27:493
3155999201,12cyclictest0-21swapper/309:58:293
3155999201,10cyclictest8539-21/usr/sbin/munin11:10:193
3154699201,16cyclictest15004-21pmu-power08:55:221
3154699200,18cyclictest0-21swapper/109:35:271
3153899209,2cyclictest12069-21kworker/0:010:09:230
3153899208,2cyclictest23671-21kworker/0:008:01:270
3153899208,11cyclictest0-21swapper/012:30:010
3153899207,11cyclictest0-21swapper/008:00:010
3153899206,4cyclictest16114-21kworker/0:210:15:230
3153899203,9cyclictest0-21swapper/011:30:030
3153899201,11cyclictest10522-21sshd07:31:030
3153899201,10cyclictest27281-21kworker/0:110:52:190
3155999191,9cyclictest0-21swapper/311:15:113
3155999191,6cyclictest6158-21ntpq08:35:223
3155999191,5cyclictest18834-21ps07:50:223
3155999191,11cyclictest0-21swapper/309:35:313
3155999191,10cyclictest30857-21/usr/sbin/munin08:20:223
3153899198,3cyclictest16561-21sed10:15:010
3153899198,3cyclictest16066-21sshd09:00:020
3153899197,4cyclictest27298-21sshd10:38:030
3153899193,4cyclictest17255-21kworker/0:107:50:450
3153899191,8cyclictest32711-21kworker/0:008:56:040
3153899191,15cyclictest23687-21ls11:45:010
3153899191,12cyclictest0-21swapper/011:03:020
3153899191,10cyclictest9471-21sshd08:45:020
3153899191,10cyclictest808-21snmpd08:17:520
3155999185,3cyclictest87350irq/126-enp1s0-08:30:143
3155999184,3cyclictest12050irq/123-ahci[0008:05:233
3155999182,13cyclictest41-21ksoftirqd/307:25:173
3153899188,3cyclictest3510-21/usr/sbin/munin09:45:180
3153899186,3cyclictest10559-21kworker/0:207:40:110
3153899184,4cyclictest32720-21/usr/sbin/munin12:05:250
3153899184,2cyclictest87350irq/126-enp1s0-12:19:370
3153899181,7cyclictest31778-21memory08:20:200
3153899177,2cyclictest2562-21/usr/sbin/munin12:10:170
3153899176,3cyclictest23302-21/usr/sbin/munin10:30:290
3153899171,6cyclictest27497-21cat11:50:250
3153899171,6cyclictest0-21swapper/011:37:350
3153899171,4cyclictest12934-21/usr/sbin/munin11:20:170
31538991713,2cyclictest2378-21ntp_states09:40:210
31538991712,3cyclictest15800-21df07:45:140
31538991712,3cyclictest15505-21latency_hist12:40:010
3153899171,13cyclictest31570-21grep09:35:120
3153899170,11cyclictest0-21swapper/008:40:180
83702160,2sleep00-21swapper/007:26:390
3155999160,5cyclictest0-21swapper/307:20:123
3153899161,6cyclictest0-21swapper/008:50:200
3153899161,14cyclictest0-21swapper/009:08:320
31538991611,3cyclictest5377-21expr08:35:140
31538991610,3cyclictest2285-21/usr/sbin/munin07:15:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional