You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-06 - 22:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Fri Mar 06, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1893523100,7sleep341-21ksoftirqd/307:05:253
192472236162,24sleep20-21swapper/207:06:022
195542204167,25sleep10-21swapper/107:09:591
192352201164,25sleep00-21swapper/007:05:520
1990921760,5sleep01963899cyclictest12:00:250
2616921550,3sleep123-21ksoftirqd/112:15:191
312632770,3sleep30-21swapper/311:15:133
325922590,5sleep31965699cyclictest10:05:023
161022570,3sleep01963899cyclictest08:10:250
62572510,1sleep20-21swapper/209:05:012
1963899421,32cyclictest16394-21/usr/sbin/munin11:55:180
1963899412,29cyclictest32134-21/usr/sbin/munin08:50:180
1963899388,21cyclictest22162-21netstat08:25:200
1963899384,20cyclictest88150irq/126-enp1s0-11:00:020
1963899381,27cyclictest22779-21awk12:10:010
1963899380,27cyclictest26706-21unixbench_multi12:15:250
1963899372,20cyclictest11297-21unixbench_multi11:40:240
1963899371,22cyclictest3557-21/usr/sbin/munin07:45:230
1963899371,22cyclictest12482-21kworker/u8:209:25:210
19638993711,12cyclictest2384-21perf10:10:000
1963899367,21cyclictest23696-21sshd09:44:410
1963899361,31cyclictest2152-21pmu-power11:20:220
1963899361,26cyclictest26781-21/usr/sbin/munin11:05:230
1963899361,22cyclictest31769-21grep11:15:160
1963899361,21cyclictest27519-21sleep08:40:010
1963899361,21cyclictest27126-21/usr/sbin/munin07:25:240
1963899361,19cyclictest14913-21taskset09:24:460
1963899357,20cyclictest823-21snmpd10:37:300
1963899357,20cyclictest17976-21/usr/sbin/munin10:45:200
1963899351,26cyclictest25383-21latency_hist08:35:020
1963899351,25cyclictest24571-21/usr/sbin/munin11:00:260
1964399341,26cyclictest0-21swapper/110:14:291
1963899347,16cyclictest88150irq/126-enp1s0-11:35:160
1963899347,16cyclictest88150irq/126-enp1s0-11:35:160
1963899343,21cyclictest20405-21kernelversion09:35:170
1963899342,27cyclictest5856-21ntp_states10:15:200
1963899341,29cyclictest10148-21/usr/sbin/munin08:00:160
1963899341,25cyclictest30854-21swap12:25:250
1963899341,24cyclictest7949-21/usr/sbin/munin07:55:140
1963899341,20cyclictest27736-21/usr/sbin/munin08:40:230
1963899341,18cyclictest9192-21/usr/sbin/munin10:25:230
1963899341,16cyclictest21696-21tail10:50:290
1963899340,25cyclictest0-21swapper/010:40:150
1963899333,14cyclictest24707-21munin-run07:20:010
19638993316,13cyclictest11448-21tail09:15:170
1963899331,25cyclictest23958-21/usr/sbin/munin09:45:140
1963899331,24cyclictest7322-21chrt11:34:280
1963899331,24cyclictest4169-21/usr/sbin/munin09:00:180
1963899331,24cyclictest21953-21munin-node10:30:120
1963899331,24cyclictest21953-21munin-node10:30:120
1963899331,22cyclictest823-21snmpd11:12:380
1963899324,20cyclictest0-21swapper/011:45:150
1963899322,17cyclictest3606-1kworker/u9:210:20:250
1963899321,23cyclictest30947-21sshd07:30:410
1963899321,16cyclictest21852-21munin-node07:15:010
1963899320,24cyclictest0-21swapper/010:10:240
19656993124,2cyclictest0-21swapper/312:04:453
1963899313,20cyclictest30545-21/usr/sbin/munin10:00:150
1963899312,21cyclictest12050irq/123-ahci[0009:53:500
19638993114,14cyclictest25921-21needreboot07:20:180
1963899311,27cyclictest8580-21/usr/sbin/munin09:10:130
1963899311,27cyclictest3911-21/usr/sbin/munin11:25:160
1963899311,19cyclictest6382-21/usr/sbin/munin09:05:170
1963899303,24cyclictest28347-21/usr/sbin/munin09:55:150
1963899301,26cyclictest31791-21/usr/sbin/munin12:30:230
1963899301,26cyclictest31523-21/usr/sbin/munin07:35:250
1963899301,26cyclictest27392-21/usr/sbin/munin12:20:130
1963899301,26cyclictest1970-21/usr/sbin/munin08:55:110
1963899301,26cyclictest18949-21/usr/sbin/munin08:20:130
1963899301,17cyclictest15082-21latency11:50:180
1963899301,17cyclictest1335-21/usr/sbin/munin07:40:190
1963899300,19cyclictest88150irq/126-enp1s0-07:50:240
1964399292,20cyclictest14933-21crond09:25:011
19643992910,9cyclictest0-21swapper/112:30:411
1963899291,25cyclictest17793-21ntpq08:15:190
19638992912,14cyclictest1745-21expr12:35:120
1963899290,26cyclictest12360-21/usr/sbin/munin08:05:110
1963899290,17cyclictest0-21swapper/008:45:170
1963899290,13cyclictest88450irq/129-enp1s0-09:30:210
1964399281,9cyclictest17059-21kworker/1:112:25:121
1963899281,24cyclictest22998-21/usr/sbin/munin12:10:120
19643992721,3cyclictest25938-21munin-run09:50:011
1964399271,19cyclictest0-21swapper/110:03:131
1964399271,18cyclictest15519-21diskstats09:25:141
1964399270,18cyclictest0-21swapper/110:05:051
292512260,4sleep232-21ksoftirqd/207:30:032
1965199261,4cyclictest26326-21pmu-power12:15:222
1964399262,18cyclictest88350irq/128-enp1s0-10:37:561
19643992620,3cyclictest8761-21irqstats07:55:171
19643992620,3cyclictest28688-21df_inode09:55:121
19643992620,3cyclictest23543-21irqstats07:15:161
19643992620,3cyclictest15623-21ntp_states08:10:211
1964399261,6cyclictest0-21swapper/110:55:011
1964399261,16cyclictest0-21swapper/112:00:071
1964399261,13cyclictest8865-21smart_sda11:35:221
1964399261,13cyclictest8865-21smart_sda11:35:221
19656992510,6cyclictest4536-21fschecks_count09:00:163
1965199256,6cyclictest0-21swapper/210:20:252
1964399258,9cyclictest0-21swapper/111:42:331
19643992519,3cyclictest1971-21df_inode12:35:131
19643992518,4cyclictest6474-21ntp_states11:30:201
1964399251,18cyclictest0-21swapper/111:50:151
1964399251,15cyclictest0-21swapper/112:05:261
19643992510,13cyclictest29004-21kworker/u8:007:42:461
1964399250,17cyclictest31831-21kworker/u8:009:10:431
1965199241,5cyclictest0-21swapper/209:46:492
1964399246,11cyclictest0-21swapper/111:10:451
19643992420,3cyclictest0-21swapper/108:09:511
1964399241,17cyclictest0-21swapper/108:45:181
1964399241,16cyclictest0-21swapper/111:01:131
1964399241,11cyclictest4794-21users11:25:261
1964399241,11cyclictest10187-21uname10:25:201
19643992410,12cyclictest6455-21kworker/u8:010:22:151
1964399240,19cyclictest0-21swapper/109:31:121
1965699239,3cyclictest17772cut08:55:013
19656992318,3cyclictest14616-21/usr/sbin/munin11:50:143
1965699231,13cyclictest1627-21/usr/sbin/munin12:35:123
1965699231,10cyclictest24060-21ntp_states12:10:213
1965199235,5cyclictest15375-21kernelversion08:10:172
1965199235,15cyclictest0-21swapper/211:05:182
19643992319,3cyclictest0-21swapper/108:01:161
1964399231,11cyclictest0-21swapper/110:31:011
1964399231,11cyclictest0-21swapper/110:31:011
1964399231,11cyclictest0-21swapper/109:42:541
1964399231,11cyclictest0-21swapper/109:16:351
19643992310,11cyclictest24145-21kworker/u8:107:23:401
1965699222,4cyclictest8344-21idleruntime-cro09:10:013
19656992215,4cyclictest31094-21grep12:25:263
1965699221,2cyclictest88350irq/128-enp1s0-11:12:273
1965199226,4cyclictest32-21ksoftirqd/209:30:012
1965199222,6cyclictest2305-21df08:55:122
1965199221,8cyclictest2681-21ntpq12:35:202
1965199221,5cyclictest0-21swapper/210:39:052
1964399222,11cyclictest21886-21cat07:15:011
19643992219,2cyclictest0-21swapper/107:33:091
19643992218,3cyclictest0-21swapper/111:49:511
19643992217,3cyclictest26959-21cat11:05:131
1964399221,19cyclictest0-21swapper/110:55:091
1964399221,15cyclictest0-21swapper/111:17:001
1964399221,10cyclictest21237-21ls08:25:111
1964399220,20cyclictest0-21swapper/110:40:181
183982220,5sleep230-21rcuc/210:45:142
218732210,4sleep2311ktimersoftd/209:40:132
1965699213,8cyclictest0-21swapper/310:49:353
1965699212,11cyclictest20631-21ntp_states09:35:203
1965699211,5cyclictest0-21swapper/307:21:353
1965699211,4cyclictest0-21swapper/310:55:023
1965699211,4cyclictest0-21swapper/309:56:183
1965699211,12cyclictest0-21swapper/309:30:273
1965699211,11cyclictest29327-21/usr/sbin/munin07:30:203
1965699210,7cyclictest0-21swapper/310:14:523
1965199217,5cyclictest0-21swapper/210:03:462
1965199216,8cyclictest88150irq/126-enp1s0-12:25:132
1965199216,5cyclictest32-21ksoftirqd/211:00:232
1965199215,3cyclictest3342-21cpuspeed_turbos11:25:122
1965199214,5cyclictest24361-21awk11:00:012
1965199211,7cyclictest28696-21diskstats09:55:132
1965199211,6cyclictest0-21swapper/211:50:222
19651992111,2cyclictest1757-21awk08:55:002
1964399217,12cyclictest23-21ksoftirqd/108:40:011
1964399211,9cyclictest19905-21df09:35:121
1964399211,9cyclictest0-21swapper/111:56:271
1964399211,6cyclictest0-21swapper/111:20:361
19643992116,3cyclictest0-21swapper/108:57:461
1964399211,4cyclictest0-21swapper/112:20:281
1964399211,18cyclictest0-21swapper/107:28:481
1964399211,11cyclictest0-21swapper/110:46:051
1964399211,10cyclictest0-21swapper/112:11:111
1964399211,10cyclictest0-21swapper/109:50:221
1964399211,10cyclictest0-21swapper/109:05:261
1965699205,4cyclictest339-21kworker/u8:211:55:233
1965699203,6cyclictest0-21swapper/312:05:303
1965699201,4cyclictest0-21swapper/312:30:203
1965699201,4cyclictest0-21swapper/310:52:043
1965699201,4cyclictest0-21swapper/310:43:503
1965699201,16cyclictest26528-21cat12:15:253
1965699201,12cyclictest0-21swapper/310:38:133
1965699201,12cyclictest0-21swapper/309:46:093
1965699201,11cyclictest0-21swapper/310:05:143
1965699201,10cyclictest22828-21ntp_states09:40:213
1965699200,7cyclictest0-21swapper/311:05:583
1965199207,2cyclictest88350irq/128-enp1s0-10:15:172
1965199205,4cyclictest17657-21load08:15:172
1965199205,3cyclictest88150irq/126-enp1s0-10:40:142
1965199202,14cyclictest27720-21grep09:50:262
1965199201,5cyclictest0-21swapper/210:08:202
1965199201,4cyclictest88150irq/126-enp1s0-08:00:432
1965199201,10cyclictest0-21swapper/211:20:312
1965199201,10cyclictest0-21swapper/210:54:512
19643992017,2cyclictest0-21swapper/107:46:101
19643992016,3cyclictest0-21swapper/108:17:431
19643992014,4cyclictest5146-21/usr/sbin/munin09:00:211
1964399201,17cyclictest0-21swapper/108:22:081
19643992010,3cyclictest0-21swapper/110:15:341
1964399200,18cyclictest0-21swapper/108:31:591
1965699198,4cyclictest0-21swapper/310:29:423
1965699198,3cyclictest991-21/usr/sbin/munin11:20:133
1965699196,4cyclictest0-21swapper/309:23:503
1965699194,3cyclictest28977-21users12:20:253
1965699194,2cyclictest88150irq/126-enp1s0-11:37:153
1965699194,2cyclictest88150irq/126-enp1s0-11:37:153
1965699192,9cyclictest9802-21/usr/sbin/munin11:40:193
1965699191,9cyclictest5794-21/usr/sbin/munin10:15:193
1965699191,4cyclictest0-21swapper/309:28:023
1965699191,11cyclictest0-21swapper/309:12:443
1965699191,10cyclictest3206-21/usr/sbin/munin11:25:233
1965699191,10cyclictest24571-21/usr/sbin/munin11:00:153
1965699191,10cyclictest11834-21ntpq09:15:223
1965699191,10cyclictest0-21swapper/311:30:063
1965699191,10cyclictest0-21swapper/310:20:223
1965699191,10cyclictest0-21swapper/309:50:173
1965199196,3cyclictest88150irq/126-enp1s0-08:45:172
1965199195,3cyclictest88450irq/129-enp1s0-09:15:142
1965199195,3cyclictest88150irq/126-enp1s0-09:05:152
1965199194,4cyclictest5587-21cstates11:30:112
1965199194,3cyclictest88150irq/126-enp1s0-10:10:112
1965199192,5cyclictest0-21swapper/212:13:252
1965199192,4cyclictest19864-21wc07:10:122
1965199191,9cyclictest0-21swapper/211:42:432
1965199191,7cyclictest0-21swapper/212:00:242
1965199191,6cyclictest22731-21/usr/sbin/munin07:15:242
1965199191,5cyclictest28077-21df_inode08:40:122
1965199191,15cyclictest2406-21ntpq07:40:192
1965199190,10cyclictest0-21swapper/212:34:502
1965199190,10cyclictest0-21swapper/210:27:362
1964399191,9cyclictest0-21swapper/108:40:051
1964399191,16cyclictest0-21swapper/107:35:241
1964399190,9cyclictest0-21swapper/107:50:161
1965699182,13cyclictest21290-21tr08:25:113
1965699181,9cyclictest8349-21diskmemload10:30:393
1965699181,9cyclictest8349-21diskmemload10:30:383
1965699181,4cyclictest1335-21/usr/sbin/munin07:40:143
1965699181,3cyclictest17806-21ntpq08:15:203
1965699181,15cyclictest13881-21taskset11:49:023
1965699181,14cyclictest23966-21find07:15:223
1965199185,3cyclictest88150irq/126-enp1s0-07:45:222
1965199184,5cyclictest13464-21wc10:35:012
1965199184,5cyclictest13464-21wc10:35:012
1965199184,3cyclictest88150irq/126-enp1s0-09:20:182
1965199182,4cyclictest13037-21hddtemp_smartct08:05:162
1965199182,13cyclictest19711-21cpuspeed_turbos09:35:132
1965199181,6cyclictest0-21swapper/211:15:492
1965199181,5cyclictest8109-21expr07:55:112
1965199181,5cyclictest23349-21/usr/sbin/munin08:30:162
1965199181,5cyclictest0-21swapper/209:12:462
1965199181,3cyclictest0-21swapper/211:36:082
1965199181,3cyclictest0-21swapper/211:36:072
19651991813,3cyclictest25279-21diskstats07:20:142
1965199181,15cyclictest0-21swapper/211:10:292
1965199180,9cyclictest0-21swapper/212:20:372
1964399181,16cyclictest0-21swapper/108:51:321
1965699177,3cyclictest19842-21/usr/sbin/munin07:10:203
1965699175,8cyclictest11771-40dnf08:00:453
1965699171,6cyclictest0-21swapper/308:35:143
19656991712,3cyclictest31523-21/usr/sbin/munin07:35:213
1965699171,14cyclictest0-21swapper/308:55:113
19656991711,3cyclictest823-21snmpd08:22:033
19656991711,3cyclictest14555-21/usr/sbin/munin08:10:113
1965699170,15cyclictest0-21swapper/308:05:083
1965199179,5cyclictest0-21swapper/207:30:132
1965199171,6cyclictest19626-21cyclictest11:45:402
1965199171,6cyclictest16549-21cpuspeed_turbos11:55:132
1965199171,6cyclictest0-21swapper/209:30:202
19651991712,3cyclictest32444-21grep07:35:192
1965199171,15cyclictest0-21swapper/208:26:102
1965199171,14cyclictest0-21swapper/208:36:082
19651991710,4cyclictest0-21swapper/212:05:112
1965199170,4cyclictest0-21swapper/207:50:182
315622161,2sleep30-21swapper/308:45:443
1965699168,3cyclictest0-21swapper/307:50:223
1965699161,12cyclictest27736-21/usr/sbin/munin08:40:153
19656991610,4cyclictest0-21swapper/307:55:223
19656991610,3cyclictest29092-21latency_hist07:30:023
1965199161,6cyclictest19099-21tr08:20:122
1965699151,5cyclictest23413-21awk08:30:133
1965699151,11cyclictest825-21sshd07:46:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional