You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-14 - 01:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Sep 13, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
226852238159,26sleep00-21swapper/007:06:040
228932202165,25sleep20-21swapper/207:08:472
227232202165,24sleep10-21swapper/107:06:331
210342201165,24sleep30-21swapper/307:05:113
2058321740,4sleep22308999cyclictest09:30:012
2718221300,1sleep30-21swapper/307:15:183
230899910097,2cyclictest0-21swapper/208:12:192
23089999289,2cyclictest0-21swapper/210:20:312
23089999289,2cyclictest0-21swapper/210:20:312
138472580,3sleep213851-21gltestperf11:40:162
20612570,5sleep121-21rcuc/112:25:261
23089995445,8cyclictest0-21swapper/212:01:332
140222530,4sleep123-21ksoftirqd/108:00:181
169522510,9sleep123-21ksoftirqd/110:35:011
286412490,3sleep230-21rcuc/207:20:142
23089994845,2cyclictest0-21swapper/208:50:562
23094994514,22cyclictest11015-21cat11:35:013
98922400,2sleep10-21swapper/111:30:211
98922400,2sleep10-21swapper/111:30:201
23094994012,11cyclictest12012-21munin-node09:10:113
23094993816,12cyclictest22092-21egrep12:00:123
23094993814,14cyclictest30834-21/usr/sbin/munin12:20:113
23094993614,18cyclictest28013-21/usr/sbin/munin11:00:253
23094993611,9cyclictest87350irq/126-enp1s0-11:40:153
2308399361,26cyclictest0-21swapper/107:35:541
2309499358,20cyclictest0-21swapper/311:20:233
23094993516,8cyclictest17993-21users09:20:253
23094993515,8cyclictest24587-21/usr/sbin/munin08:25:143
23094993512,14cyclictest11387-21/usr/sbin/munin07:55:233
23094993512,14cyclictest11387-21/usr/sbin/munin07:55:223
23094993511,14cyclictest25815-21/usr/sbin/munin10:55:163
85552340,18sleep20-21swapper/209:00:192
43002340,4sleep121-21rcuc/112:30:271
2309499346,15cyclictest0-21swapper/311:50:293
23094993416,6cyclictest8616-21munin-run11:30:013
23094993414,12cyclictest27402-21/usr/sbin/munin09:45:253
23094993413,10cyclictest12407-21timerandwakeup11:35:253
23094993411,15cyclictest25217-21users10:50:303
2309499339,16cyclictest11618-21kworker/u8:110:25:163
2309499339,13cyclictest0-21swapper/307:35:203
2309499339,10cyclictest827-21runrttasks07:49:503
23094993314,8cyclictest19819-21ps09:25:233
23094993312,17cyclictest1807-21tr12:25:233
23094993310,10cyclictest23780-21kworker/u8:009:40:143
2309499326,17cyclictest0-21swapper/312:35:113
23094993213,10cyclictest31560-21latency_hist09:55:013
23094993210,13cyclictest24077-21ntpq09:35:223
2309499319,11cyclictest15844-21netstat10:30:203
23094993122,5cyclictest20901-40dnf10:41:313
23094993119,8cyclictest32541-21awk07:30:003
23094993116,4cyclictest16248-21irqstats11:45:173
23094993114,5cyclictest24071-21sshd08:21:293
23094993113,15cyclictest87350irq/126-enp1s0-08:55:203
23094993112,16cyclictest8220-21head09:00:163
23094993110,9cyclictest0-21swapper/308:10:183
2308399311,21cyclictest0-21swapper/108:32:281
2309499309,7cyclictest827-21runrttasks12:30:213
23094993022,5cyclictest1975-21diskstats10:00:123
23094993014,8cyclictest32658-1kworker/3:0H07:10:243
23094993013,14cyclictest3835-21/usr/sbin/munin10:05:153
23094993013,13cyclictest15024-21/usr/sbin/munin09:15:183
23094993010,13cyclictest11785-21diskmemload11:13:043
2309499300,18cyclictest22487-21ntp_states10:45:223
2308399301,2cyclictest0-21swapper/108:17:581
2308399301,13cyclictest0-21swapper/111:59:021
2308399300,10cyclictest0-21swapper/112:23:341
2309499298,9cyclictest19833-21/usr/sbin/munin11:55:123
2309499298,6cyclictest24243-21/usr/sbin/munin12:05:203
23094992921,5cyclictest7299-21smart_sda10:10:283
23094992919,7cyclictest10771-21diskstats10:20:153
23094992919,7cyclictest10771-21diskstats10:20:153
23094992910,7cyclictest28641-21/usr/sbin/munin12:15:203
2308399290,22cyclictest0-21swapper/109:19:231
2308399290,10cyclictest0-21swapper/109:02:181
2307699291,13cyclictest16796-21munin-run10:35:010
23076992911,15cyclictest13023-21awk11:40:010
2309499288,11cyclictest0-21swapper/310:35:223
23094992821,4cyclictest28370-21/usr/sbin/munin08:30:273
23094992820,5cyclictest30285-21/usr/sbin/munin11:05:113
23094992820,5cyclictest29668-21/usr/sbin/munin08:35:183
23094992820,5cyclictest20544-21/usr/sbin/munin08:15:163
23094992820,5cyclictest14855-21smart_sda08:00:293
23094992820,5cyclictest10968-21/usr/sbin/munin09:05:243
23094992820,5cyclictest10968-21/usr/sbin/munin09:05:243
23094992819,6cyclictest32608-21/usr/sbin/munin09:55:183
23094992819,6cyclictest3067-21/usr/sbin/munin11:15:183
23094992817,7cyclictest32310-21sed08:40:233
23094992810,5cyclictest20806-21/usr/sbin/munin09:30:263
2308399281,11cyclictest0-21swapper/107:57:501
2308399281,11cyclictest0-21swapper/107:57:501
2308399281,11cyclictest0-21swapper/107:32:541
2308399281,10cyclictest0-21swapper/109:59:461
2308399280,12cyclictest0-21swapper/112:01:301
2308399280,12cyclictest0-21swapper/110:04:301
2308399280,10cyclictest0-21swapper/111:35:231
2307699288,16cyclictest28152-21awk07:20:020
23076992810,15cyclictest17419-21awk11:50:010
2309499278,17cyclictest0-21swapper/308:50:123
23094992720,4cyclictest29629-21/usr/sbin/munin07:20:253
23094992719,5cyclictest9564-21/usr/sbin/munin11:30:163
23094992718,6cyclictest992-21/usr/sbin/munin08:45:243
23094992718,6cyclictest4791-21/usr/sbin/munin07:40:153
23094992718,6cyclictest32759-21/usr/sbin/munin07:30:113
23094992718,6cyclictest15796-21/usr/sbin/munin08:05:143
23094992717,7cyclictest9260-21/usr/sbin/munin10:15:203
23094992716,8cyclictest27248-21/usr/sbin/munin12:10:183
23094992710,14cyclictest9194-21/usr/sbin/munin07:50:193
2308399272,13cyclictest808-21snmpd09:41:241
2308399271,11cyclictest0-21swapper/108:58:541
2308399271,11cyclictest0-21swapper/107:10:021
2308399271,10cyclictest0-21swapper/107:51:021
2308399270,5cyclictest0-21swapper/111:19:301
2308399270,17cyclictest0-21swapper/112:15:061
2308399270,11cyclictest0-21swapper/109:46:221
2308399270,10cyclictest0-21swapper/108:54:541
2308399261,9cyclictest0-21swapper/110:17:361
2308399261,9cyclictest0-21swapper/110:10:181
2308399261,9cyclictest0-21swapper/108:25:021
2308399261,11cyclictest0-21swapper/111:20:381
2308399261,11cyclictest0-21swapper/109:50:381
2308399261,11cyclictest0-21swapper/107:40:461
2308399260,11cyclictest0-21swapper/110:42:141
2308399260,11cyclictest0-21swapper/109:24:031
2308399260,11cyclictest0-21swapper/108:10:541
2308399260,10cyclictest0-21swapper/110:58:061
2307699261,12cyclictest31569-21munin-run09:55:000
23076992610,13cyclictest30007-21awk11:05:010
55692250,2sleep20-21swapper/207:40:172
2308399251,9cyclictest0-21swapper/111:09:461
2308399251,8cyclictest0-21swapper/108:09:061
2308399251,11cyclictest0-21swapper/110:49:461
2308399251,11cyclictest0-21swapper/109:31:511
2308399251,10cyclictest0-21swapper/108:47:341
2308399250,9cyclictest0-21swapper/111:54:141
2308399250,9cyclictest0-21swapper/107:45:301
2308399250,12cyclictest0-21swapper/112:13:581
2308399250,12cyclictest0-21swapper/109:06:521
2308399250,12cyclictest0-21swapper/109:06:521
23076992511,9cyclictest5952-21proc_pri07:40:220
2308399241,9cyclictest0-21swapper/110:50:541
2308399241,9cyclictest0-21swapper/110:35:101
2308399241,9cyclictest0-21swapper/110:23:061
2308399241,9cyclictest0-21swapper/110:23:061
2308399241,11cyclictest0-21swapper/112:37:241
2308399241,11cyclictest0-21swapper/107:27:421
2308399241,11cyclictest0-21swapper/107:22:531
2308399241,10cyclictest0-21swapper/108:37:241
2308399241,10cyclictest0-21swapper/108:21:401
23083992410,4cyclictest28140-21munin-run07:20:011
2308399240,8cyclictest0-21swapper/108:41:221
2308399240,12cyclictest0-21swapper/111:47:261
2308399240,12cyclictest0-21swapper/109:35:191
2308399240,10cyclictest0-21swapper/111:04:181
2308399231,11cyclictest0-21swapper/111:40:211
2308399231,10cyclictest0-21swapper/112:09:361
2308399230,9cyclictest0-21swapper/109:27:151
2307699231,6cyclictest0-21swapper/010:43:530
2308399221,18cyclictest4522-21hwlatdetect10:05:151
2308399220,16cyclictest0-21swapper/111:26:511
2308399220,10cyclictest0-21swapper/109:14:161
2307699227,13cyclictest22155-21munin-run08:20:010
2307699223,10cyclictest12625-21/usr/sbin/munin10:25:200
23076992210,4cyclictest26806-21fschecks_count12:10:150
2308999211,5cyclictest24576-21missed_timers10:50:232
2308399211,10cyclictest0-21swapper/110:25:241
2308399210,9cyclictest0-21swapper/111:10:221
2307699216,4cyclictest18169-21kworker/0:009:32:040
2307699213,15cyclictest9600-21awk09:05:020
23076992115,4cyclictest29134-21tr08:35:130
2307699211,4cyclictest0-21swapper/010:11:070
2307699211,10cyclictest11-21rcuc/012:35:250
2308999204,9cyclictest0-21swapper/208:09:462
2308999201,7cyclictest0-21swapper/209:32:582
2307699201,7cyclictest0-21swapper/011:05:480
23076992014,3cyclictest23036-21cut08:20:150
2307699201,3cyclictest0-21swapper/009:36:150
2307699201,11cyclictest32419-21/usr/sbin/munin11:10:190
2307699201,11cyclictest25198-21/usr/sbin/munin09:40:120
2307699201,10cyclictest13927-21sshd09:14:270
8212190,3sleep00-21swapper/012:25:130
2308999195,6cyclictest755-21rngd08:20:192
2308999193,8cyclictest0-21swapper/212:29:362
2308999192,9cyclictest0-21swapper/212:33:452
2308999191,15cyclictest20482-21smartctl10:40:262
2308999191,11cyclictest0-21swapper/209:55:202
2308999191,10cyclictest30272-21hddtemp_smartct09:50:212
2307699198,2cyclictest9221-21grep10:15:190
2307699197,3cyclictest0-21swapper/008:04:560
2307699193,3cyclictest0-21swapper/010:48:310
2307699191,4cyclictest0-21swapper/011:24:190
23076991914,3cyclictest24355-21if_enp1s010:50:210
23076991912,4cyclictest22638-21grep12:00:160
2307699191,15cyclictest11495-21sh10:20:210
2307699191,15cyclictest11495-21sh10:20:210
23076991911,2cyclictest11618-21kworker/u8:109:56:010
2307699191,12cyclictest0-21swapper/011:33:250
2307699191,12cyclictest0-21swapper/011:33:250
2307699191,11cyclictest0-21swapper/010:01:580
2307699191,10cyclictest14211-21/usr/sbin/munin09:15:250
2308999182,9cyclictest14493-21expr09:15:122
2308999181,8cyclictest12012-21/usr/sbin/munin09:10:172
2308999181,7cyclictest29527-21cat09:50:032
2308999181,6cyclictest808-21snmpd10:09:092
2308999181,5cyclictest9-21rcu_preempt11:18:012
2308999181,5cyclictest0-21swapper/211:32:062
2308999181,5cyclictest0-21swapper/211:32:062
2308999181,3cyclictest87650irq/129-enp1s0-12:05:262
23089991812,3cyclictest13588-21/usr/sbin/munin08:00:162
2308999181,15cyclictest0-21swapper/207:29:492
2307699189,2cyclictest0-21swapper/012:15:280
2307699186,3cyclictest4356-21pmu-power08:50:230
2307699182,4cyclictest25883-21runrttasks12:05:540
2307699181,6cyclictest0-21swapper/011:15:290
23076991812,4cyclictest30334-21sh07:25:010
23076991812,3cyclictest13896-21cut11:40:160
23076991811,4cyclictest3151-21cut07:35:160
85112170,2sleep00-21swapper/011:28:510
69522171,2sleep27028-21strings10:10:242
265762170,2sleep20-21swapper/208:30:022
245792170,2sleep00-21swapper/008:25:070
2308999178,3cyclictest0-21swapper/207:11:282
2308999178,2cyclictest32419-21/usr/sbin/munin11:10:262
2308999175,3cyclictest22164-21perf08:20:012
2308999174,6cyclictest32270-21timerwakeupswit12:20:262
2308999174,4cyclictest9396-21ps10:15:222
2308999174,3cyclictest0-21swapper/211:56:152
2308999171,9cyclictest0-21swapper/209:40:392
2308999171,7cyclictest24587-21/usr/sbin/munin08:25:232
2308999171,7cyclictest23201-21expr09:35:132
2308999171,7cyclictest0-21swapper/211:45:512
2308999171,6cyclictest18455-21cat09:25:012
2308999171,15cyclictest0-21swapper/211:29:232
2308999171,15cyclictest0-21swapper/207:51:322
23089991711,4cyclictest22992-21grep10:45:262
2308999171,14cyclictest11785-21diskmemload12:12:082
2308999171,13cyclictest5362-21memory11:20:192
23089991711,3cyclictest25396-21munin-node12:35:122
23089991711,3cyclictest18982-21idleruntime-cro10:40:012
23089991711,3cyclictest12625-21/usr/sbin/munin10:25:262
2307699178,2cyclictest23277-21/usr/sbin/munin07:10:250
2307699175,3cyclictest87550irq/128-enp1s0-08:30:190
2307699174,4cyclictest27576-21sshd10:57:290
2307699171,9cyclictest0-21swapper/010:05:080
2307699171,7cyclictest0-21swapper/009:22:370
2307699171,6cyclictest29364-21sh09:50:010
2307699171,4cyclictest18493-21latency11:50:210
23076991711,4cyclictest0-21swapper/008:40:250
23076991711,3cyclictest9298-21cut07:50:130
23076991711,3cyclictest7362-21grep07:45:150
2308999167,6cyclictest0-21swapper/208:45:142
2308999163,3cyclictest87650irq/129-enp1s0-11:35:122
2308999162,11cyclictest29161-21cut12:15:182
2308999161,7cyclictest0-21swapper/210:04:262
2308999161,6cyclictest30220-21/usr/sbin/munin11:05:122
2308999161,6cyclictest25815-21/usr/sbin/munin10:55:112
2308999161,6cyclictest11572-21sshd09:07:282
2308999161,6cyclictest11572-21sshd09:07:282
2308999161,3cyclictest87650irq/129-enp1s0-07:45:252
2308999161,2cyclictest0-21swapper/210:30:402
2308999161,2cyclictest0-21swapper/207:34:452
23089991612,3cyclictest0-21swapper/208:59:072
23089991612,3cyclictest0-21swapper/207:55:402
23089991612,3cyclictest0-21swapper/207:55:402
23089991611,3cyclictest30244-21swap08:35:252
23089991611,3cyclictest28685-21hddtemp_smartct11:00:162
23089991611,3cyclictest25649-21ntp_states07:15:022
23089991611,3cyclictest18163-21fschecks_time11:50:192
2307699164,4cyclictest992-21/usr/sbin/munin08:45:140
2307699164,3cyclictest11-21rcuc/012:20:250
2307699161,6cyclictest0-21swapper/012:32:450
2307699161,6cyclictest0-21swapper/008:05:180
2307699161,4cyclictest0-21swapper/010:35:140
23076991612,3cyclictest0-21swapper/009:08:120
23076991612,3cyclictest0-21swapper/009:08:120
23076991611,3cyclictest30919-21entropy07:25:140
23076991611,3cyclictest19663-21seq08:11:210
23076991611,3cyclictest1296-21/usr/sbin/munin07:30:170
23076991611,3cyclictest12558-21wc07:55:200
23076991611,3cyclictest12558-21wc07:55:200
2307699161,12cyclictest20824-21grep11:55:200
23076991610,4cyclictest18768-21tr09:25:130
23076991610,3cyclictest5758-21diskstats08:55:160
2308999159,3cyclictest3474-21wc07:35:192
2308999151,2cyclictest0-21swapper/208:44:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional