You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 21:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Sep 16, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22482239164,23sleep30-21swapper/307:08:533
20822234163,57sleep20-21swapper/207:06:422
22552204164,27sleep00-21swapper/007:08:580
21262203165,25sleep10-21swapper/107:07:191
86562540,3sleep121-21rcuc/108:35:131
195602490,2sleep10-21swapper/109:00:141
242699435,25cyclictest24765-40dnf11:39:330
242699379,26cyclictest0-21swapper/010:05:020
242699378,13cyclictest1241-21idleruntime-cro12:00:020
242699374,29cyclictest3722-21munin-run08:25:010
242699369,12cyclictest17553-21munin-run10:10:020
242699359,23cyclictest21641-21grep09:05:140
242699346,4cyclictest0-21swapper/010:25:500
242699344,3cyclictest12113-21/usr/sbin/munin07:30:190
242699344,3cyclictest12113-21/usr/sbin/munin07:30:190
242699343,3cyclictest86750irq/128-enp1s0-08:40:120
242699341,7cyclictest2375-21/usr/sbin/munin09:35:140
242699341,6cyclictest24280-21/usr/sbin/munin07:55:250
242699341,25cyclictest1697-21kworker/0:207:11:150
242699337,2cyclictest0-21swapper/009:50:200
242699336,25cyclictest0-21swapper/011:55:020
242699332,3cyclictest18198-21ntp_states08:55:230
242699332,3cyclictest1219-21ntp_states09:30:230
242699331,6cyclictest29552-21if_err_enp1s010:35:190
242699331,4cyclictest27963-21cat09:20:010
242699331,4cyclictest27278-21latency_hist08:05:010
242699331,3cyclictest29592-21cat09:20:260
242699322,5cyclictest15747-21ntp_states12:30:220
242699322,4cyclictest19685-21diskstats09:00:160
242699321,4cyclictest3777-21awk10:50:190
2426993211,18cyclictest7426-21/usr/sbin/munin11:00:260
2426993210,19cyclictest13116-21smart_sda11:10:260
243899311,14cyclictest0-21swapper/210:53:452
242699319,19cyclictest7253-21pluginstate08:30:250
242699316,2cyclictest86550irq/126-enp1s0-10:40:190
242699316,2cyclictest86550irq/126-enp1s0-10:40:190
242699312,4cyclictest8527-21irqstats07:20:180
242699312,3cyclictest10027-40dnf08:36:310
242699312,24cyclictest18843-21latency_hist12:40:010
2426993111,17cyclictest6274-21ntp_states10:55:230
2426993110,18cyclictest5522-21/usr/sbin/munin07:15:350
2426993110,17cyclictest1801-21cat12:00:150
242699310,22cyclictest0-21swapper/012:05:170
242699308,19cyclictest11847-21head12:20:280
242699307,19cyclictest10052-21sh12:20:010
242699302,4cyclictest16510-21/usr/sbin/munin07:40:370
242699301,5cyclictest0-21swapper/009:26:510
242699301,4cyclictest9919-21/usr/sbin/munin07:25:120
242699301,4cyclictest7154-21cut09:45:160
242699301,4cyclictest6995-21/usr/sbin/munin12:10:250
242699301,4cyclictest27510-21/usr/sbin/munin08:05:250
242699301,4cyclictest21106-21pmu-power10:15:240
242699301,3cyclictest581-21systemd-journal09:42:300
242699301,3cyclictest581-21systemd-journal09:42:300
242699301,25cyclictest784-21NetworkManager11:15:400
2426993012,15cyclictest25383-21seq09:10:490
2426993010,17cyclictest27490-21expr11:45:140
2426993010,17cyclictest27490-21expr11:45:130
2426993010,17cyclictest14303-21/usr/sbin/munin07:35:120
2426993010,17cyclictest10471-21cut11:05:200
242699300,4cyclictest0-21swapper/010:10:450
243399297,3cyclictest19384-21hddtemp_smartct07:45:191
2433992919,3cyclictest0-21swapper/110:15:231
2433992912,6cyclictest262342sleep110:28:411
242699298,18cyclictest29838-21tr08:10:140
242699296,3cyclictest13893-21tr08:45:250
242699292,3cyclictest22878-21/usr/sbin/munin07:55:020
242699291,4cyclictest738-21/usr/sbin/munin10:45:140
242699291,4cyclictest27282-21/usr/sbin/munin10:30:170
242699291,4cyclictest22947-21/usr/sbin/munin10:20:190
242699291,4cyclictest17015-21/usr/sbin/munin11:20:190
242699291,3cyclictest25389-21/usr/sbin/munin11:40:140
2426992912,14cyclictest14289-21seq12:28:040
2426992911,15cyclictest15052-21cpuspeed_turbos08:50:140
2426992910,16cyclictest19288-21cut11:25:200
242699282,4cyclictest19879-21/usr/sbin/munin07:45:300
242699281,4cyclictest20656-21acpi11:30:180
242699281,4cyclictest11503-21/usr/sbin/munin09:55:140
243399279,9cyclictest6660-21pmu-power07:15:351
242699271,4cyclictest4569-21gltestperf08:25:170
2426992711,13cyclictest32005-21cpuspeed_turbos08:15:140
243399261,9cyclictest86650irq/127-enp1s0-10:45:271
243399261,17cyclictest21668-21netstat11:30:251
176632260,3sleep230-21rcuc/212:35:192
244299257,7cyclictest0-21swapper/311:05:373
243899251,6cyclictest21143-21ps10:15:232
243399251,17cyclictest0-21swapper/111:35:191
243399251,16cyclictest0-21swapper/111:26:521
243399243,9cyclictest0-21swapper/110:20:051
243399242,16cyclictest86850irq/129-enp1s0-08:27:281
2433992414,3cyclictest14941-21/usr/sbin/munin08:50:191
243399241,17cyclictest0-21swapper/111:50:231
243399241,17cyclictest0-21swapper/109:12:471
243399241,14cyclictest507-21latency_hist10:45:011
243399241,14cyclictest507-21latency_hist10:45:011
219532240,2sleep123-21ksoftirqd/109:05:161
244299231,10cyclictest3304-21hddtemp_smartct07:10:163
2442992310,3cyclictest24243-21timerandwakeup11:35:253
243899238,3cyclictest0-21swapper/208:59:032
243899231,11cyclictest0-21swapper/209:21:052
243899231,10cyclictest0-21swapper/212:31:532
243399233,3cyclictest6933-21taskset10:56:431
243399232,6cyclictest5497-21/usr/sbin/munin09:40:231
243399232,6cyclictest5497-21/usr/sbin/munin09:40:231
2433992317,3cyclictest28772-21/usr/sbin/munin08:05:241
243399231,3cyclictest26837-21/usr/sbin/munin08:00:261
243399231,3cyclictest21-21rcuc/110:10:161
243399231,3cyclictest18049-21memory08:55:191
243399231,17cyclictest0-21swapper/109:37:181
243399231,16cyclictest0-21swapper/112:22:071
243399231,16cyclictest0-21swapper/107:24:201
243399231,15cyclictest0-21swapper/109:27:351
243399230,17cyclictest0-21swapper/109:30:231
48432220,8sleep339-21rcuc/312:05:263
244299224,8cyclictest0-21swapper/312:00:223
244299220,5cyclictest435-21kworker/u8:110:45:223
243899226,6cyclictest28774-21swap08:05:252
243899224,8cyclictest12003-21cpuspeed_turbos11:10:132
243899221,11cyclictest0-21swapper/209:48:552
243899221,11cyclictest0-21swapper/209:43:112
243899221,11cyclictest0-21swapper/209:43:112
243899221,10cyclictest0-21swapper/211:43:512
243899221,10cyclictest0-21swapper/207:53:132
2433992217,3cyclictest0-21swapper/112:15:281
2433992214,3cyclictest0-21swapper/111:15:111
2433992214,2cyclictest0-21swapper/110:54:551
2433992213,3cyclictest0-21swapper/112:00:261
2433992213,2cyclictest0-21swapper/108:16:531
2433992212,4cyclictest848-21sssd_nss07:52:271
243399221,16cyclictest0-21swapper/112:38:301
243399221,16cyclictest0-21swapper/107:44:201
243399221,15cyclictest0-21swapper/107:56:051
243399221,15cyclictest0-21swapper/107:27:161
243399221,14cyclictest88250irq/135-enp2s0-08:46:351
171622221,2sleep317185-21sshd10:05:323
133662220,5sleep339-21rcuc/311:10:263
244299219,3cyclictest23529-21basename09:10:013
244299213,10cyclictest28775-21/usr/sbin/munin10:35:153
244299212,11cyclictest14303-21/usr/sbin/munin07:35:133
244299211,7cyclictest17231-1kworker/u9:012:31:273
244299211,5cyclictest0-21swapper/310:02:113
2442992114,4cyclictest7197-21munin-plugin-st11:00:013
244299211,17cyclictest28186-21hddtemp_smartct08:05:183
244299211,11cyclictest26578-21/usr/sbin/munin10:30:263
243899217,3cyclictest0-21swapper/211:32:272
243899214,6cyclictest780-21rngd10:31:432
243899213,3cyclictest0-21swapper/208:03:352
243899211,9cyclictest0-21swapper/208:22:472
243899211,10cyclictest0-21swapper/208:46:012
243399215,3cyclictest0-21swapper/108:21:391
243399212,3cyclictest0-21swapper/107:12:331
243399211,8cyclictest0-21swapper/109:50:321
2433992117,3cyclictest23-21ksoftirqd/110:40:011
2433992115,3cyclictest23-21ksoftirqd/112:10:011
2433992113,3cyclictest0-21swapper/108:34:531
243399211,18cyclictest0-21swapper/112:30:311
243399211,16cyclictest0-21swapper/112:11:191
243399211,16cyclictest0-21swapper/110:31:001
243399211,15cyclictest0-21swapper/111:46:511
243399211,15cyclictest0-21swapper/111:46:511
243399211,15cyclictest0-21swapper/111:42:441
243399211,15cyclictest0-21swapper/109:46:151
243399211,15cyclictest0-21swapper/107:31:591
243399211,15cyclictest0-21swapper/107:31:591
244299208,3cyclictest12050irq/123-ahci[0012:40:003
244299204,9cyclictest0-21swapper/311:59:253
244299203,10cyclictest12201-21sshd12:24:293
244299202,3cyclictest0-21swapper/309:50:143
244299202,10cyclictest2375-21/usr/sbin/munin09:35:143
244299201,11cyclictest8090-21/usr/sbin/munin12:15:263
244299201,11cyclictest12741-21/usr/sbin/munin08:45:253
2442992010,2cyclictest0-21swapper/310:00:013
244299200,13cyclictest0-21swapper/312:28:053
243899209,3cyclictest86550irq/126-enp1s0-12:10:282
243899207,2cyclictest32-21ksoftirqd/207:59:272
243899206,3cyclictest0-21swapper/212:21:232
243899204,3cyclictest32352-21idleruntime11:55:172
243899202,5cyclictest0-21swapper/212:15:272
243899201,8cyclictest0-21swapper/209:19:192
2438992015,3cyclictest19746-21sh10:15:012
2438992014,3cyclictest31586-21mailstats09:25:242
243899200,8cyclictest0-21swapper/207:10:502
243899200,12cyclictest0-21swapper/211:15:232
243399205,2cyclictest86850irq/129-enp1s0-09:15:301
243399203,2cyclictest86750irq/128-enp1s0-11:05:391
2433992013,3cyclictest0-21swapper/111:01:251
243399201,18cyclictest0-21swapper/112:29:511
243399201,10cyclictest30107-21latency_hist09:25:001
243399200,6cyclictest0-21swapper/111:56:151
243399200,15cyclictest0-21swapper/111:12:311
243399200,15cyclictest0-21swapper/107:35:271
244299198,3cyclictest14634-21snmpd11:04:173
244299195,4cyclictest10260-21diskstats07:25:143
244299194,8cyclictest0-21swapper/311:16:393
244299191,8cyclictest6496-21diskstats08:30:163
244299191,5cyclictest0-21swapper/310:41:433
244299191,5cyclictest0-21swapper/310:41:433
244299191,4cyclictest0-21swapper/309:26:223
243899198,4cyclictest0-21swapper/209:00:192
243899198,2cyclictest0-21swapper/211:05:132
243899197,3cyclictest0-21swapper/208:43:232
243899196,3cyclictest0-21swapper/209:51:472
243899196,2cyclictest30259-21kworker/u8:209:13:042
243899196,2cyclictest12382-21kworker/2:009:05:272
243899195,3cyclictest0-21swapper/208:26:252
243899194,4cyclictest0-21swapper/211:54:312
243899194,3cyclictest0-21swapper/209:36:232
243899193,6cyclictest88250irq/135-enp2s0-10:25:132
243899191,9cyclictest50042cat12:05:282
243899191,7cyclictest0-21swapper/208:38:232
2438991913,3cyclictest86850irq/129-enp1s0-11:20:192
243899191,15cyclictest15846-21latency08:50:192
243399191,3cyclictest0-21swapper/110:01:011
2433991912,3cyclictest0-21swapper/111:21:081
243399191,17cyclictest0-21swapper/109:55:061
243399191,16cyclictest0-21swapper/110:05:201
141902191,2sleep20-21swapper/212:26:452
75082181,3sleep30-21swapper/312:10:373
244299188,2cyclictest0-21swapper/308:38:293
244299182,6cyclictest28316-21ntp_states11:45:223
244299182,6cyclictest28316-21ntp_states11:45:223
244299182,6cyclictest17778-21/usr/sbin/munin10:10:123
244299181,9cyclictest0-21swapper/309:44:343
244299181,9cyclictest0-21swapper/309:44:343
244299181,9cyclictest0-21swapper/309:10:393
244299181,7cyclictest3015-21/usr/sbin/munin10:50:153
244299181,7cyclictest20617-21cat11:30:013
244299181,7cyclictest0-21swapper/309:21:143
244299181,4cyclictest0-21swapper/307:31:363
244299181,4cyclictest0-21swapper/307:31:363
244299181,3cyclictest0-21swapper/311:33:053
2442991813,3cyclictest7835-21ntp_offset09:45:223
2442991813,3cyclictest17318-21cstates08:55:133
244299181,11cyclictest0-21swapper/309:30:253
244299180,10cyclictest3945-21/usr/sbin/munin08:25:243
243899188,3cyclictest0-21swapper/210:05:452
243899186,3cyclictest86550irq/126-enp1s0-10:22:312
243899186,3cyclictest0-21swapper/207:48:432
243899185,4cyclictest86850irq/129-enp1s0-11:00:272
243899185,3cyclictest86550irq/126-enp1s0-10:00:162
243899184,3cyclictest86550irq/126-enp1s0-10:45:172
243899183,3cyclictest88150irq/134-enp2s0-07:40:272
243899181,7cyclictest8683-21missed_timers07:20:202
243899181,5cyclictest32152-21processes10:40:252
243899181,5cyclictest32152-21processes10:40:252
243899181,5cyclictest0-21swapper/211:35:172
2438991812,4cyclictest921-21cat09:30:182
2438991812,3cyclictest27666-21fschecks_count11:45:152
2438991812,3cyclictest27666-21fschecks_count11:45:152
2438991812,3cyclictest2382-21cat12:00:192
2438991812,3cyclictest14930-21grep07:35:132
2438991811,4cyclictest32-21ksoftirqd/207:29:312
243899181,10cyclictest0-21swapper/209:57:042
243399181,8cyclictest29711-21/usr/sbin/munin08:10:171
243399181,16cyclictest0-21swapper/108:40:481
248132170,2sleep30-21swapper/307:56:503
244299171,7cyclictest29790-21cat08:10:133
244299171,6cyclictest0-21swapper/311:43:163
244299171,6cyclictest0-21swapper/307:50:273
244299171,5cyclictest0-21swapper/311:52:063
2442991712,3cyclictest26572-21gltestperf09:15:163
2442991712,3cyclictest17270-21netstat11:20:213
244299171,15cyclictest0-21swapper/310:16:593
244299171,13cyclictest25303-21/usr/sbin/munin08:00:193
244299171,13cyclictest24379-21/usr/sbin/munin10:25:213
2442991711,3cyclictest22259-21cat10:20:133
244299171,13cyclictest19338-21/usr/sbin/munin09:00:173
244299171,10cyclictest0-21swapper/308:50:213
243899175,3cyclictest30-21rcuc/207:15:182
243899174,5cyclictest18444-21/usr/sbin/munin11:25:172
243899172,6cyclictest30612-40dnf10:38:302
243899171,6cyclictest32721-21irqstats08:15:192
243899171,5cyclictest0-21swapper/210:55:162
2438991713,2cyclictest30-21rcuc/208:30:182
2438991710,4cyclictest848-21sssd_nss08:10:202
13602171,2sleep30-21swapper/308:18:303
244299169,4cyclictest8391-21sh07:20:173
244299161,3cyclictest10542-21/usr/sbin/munin08:40:123
2442991611,3cyclictest5250-21if_enp1s007:15:053
2442991611,3cyclictest16748-21expr07:40:153
2442991610,3cyclictest18958-21expr07:45:163
243899165,3cyclictest86550irq/126-enp1s0-07:30:132
243899165,3cyclictest86550irq/126-enp1s0-07:30:132
244299151,8cyclictest0-21swapper/308:20:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional