You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-12 - 19:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Nov 12, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
281882236163,23sleep30-21swapper/307:09:113
279272231154,25sleep00-21swapper/007:05:450
280892204167,25sleep20-21swapper/207:07:542
280462203166,24sleep10-21swapper/107:07:201
2836199106104,1cyclictest0-21swapper/210:10:192
28361999795,1cyclictest0-21swapper/210:15:382
28361997369,3cyclictest0-21swapper/210:25:202
3912650,2sleep00-21swapper/012:10:220
28361996258,2cyclictest0-21swapper/212:29:322
176082580,1sleep00-21swapper/009:10:150
133222540,1sleep00-21swapper/010:15:010
28361995343,8cyclictest0-21swapper/211:13:112
116252520,2sleep30-21swapper/310:10:143
30902510,2sleep00-21swapper/009:50:170
110712500,1sleep10-21swapper/112:35:171
327242490,4sleep121-21rcuc/107:15:271
255862490,3sleep230-21rcuc/210:40:212
255862490,3sleep230-21rcuc/210:40:212
2836199411,28cyclictest27676-21cstates12:00:142
144702410,1sleep30-21swapper/307:50:123
2836999401,29cyclictest0-21swapper/311:24:053
2836199391,19cyclictest8380-21cstates12:30:142
2835699381,3cyclictest0-21swapper/111:15:321
2836199371,26cyclictest15736-21/usr/sbin/munin10:20:202
2836199371,25cyclictest20756-21/usr/sbin/munin11:45:212
2836199371,25cyclictest20756-21/usr/sbin/munin11:45:212
2836199371,21cyclictest1041-21cpuspeed_turbos11:00:132
2836999366,2cyclictest7347-21awk11:15:013
2836999361,28cyclictest0-21swapper/308:52:053
2836999361,12cyclictest0-21swapper/310:25:213
2836999351,5cyclictest18331-21perf11:40:003
2836999351,27cyclictest0-21swapper/308:38:453
2836999351,21cyclictest780-21rngd08:14:113
2836999351,13cyclictest10815-21tr08:55:153
2836199354,22cyclictest27663-21/usr/sbin/munin08:20:152
2836199353,22cyclictest25290-21sed09:25:262
2836199351,25cyclictest11958-21/usr/sbin/munin11:25:162
2836199351,21cyclictest2667-21ntp_states12:15:222
2836999349,4cyclictest8804-21gltestperf12:30:163
2836999347,3cyclictest86550irq/126-enp1s0-12:05:133
2836999341,6cyclictest29719-21/usr/sbin/munin10:50:173
2836999341,6cyclictest17069-21diskmemload09:35:253
2836999341,6cyclictest17069-21diskmemload09:35:253
2836999340,28cyclictest0-21swapper/307:58:293
2836199349,13cyclictest86550irq/126-enp1s0-09:22:272
2836199343,22cyclictest86850irq/129-enp1s0-07:40:152
2836199343,22cyclictest2298-21latency_hist09:50:022
2836199343,18cyclictest19864-21fschecks_count09:15:172
2836199341,24cyclictest1253-21/usr/sbin/munin07:20:242
2836199341,23cyclictest3802-21/usr/sbin/munin12:20:232
2836999335,4cyclictest0-21swapper/311:40:243
2836999333,3cyclictest86550irq/126-enp1s0-10:45:013
2836999331,8cyclictest0-21swapper/308:42:323
2836999331,7cyclictest1602-21/usr/sbin/munin12:15:143
2836999331,24cyclictest0-21swapper/312:28:323
28369993311,3cyclictest836-21runrttasks07:37:483
28369993310,3cyclictest30693-21cat07:15:013
2836199335,19cyclictest6931-21/usr/sbin/munin10:00:202
2836199332,3cyclictest28486-21chrt10:47:272
2836199332,22cyclictest10694-21/usr/sbin/munin08:55:292
2836199331,6cyclictest31767-21/usr/sbin/munin12:10:262
2836199331,24cyclictest0-21swapper/208:11:142
2836199331,22cyclictest4098-21/usr/sbin/munin08:40:182
2836199331,20cyclictest818-21cat11:00:022
2836199331,17cyclictest15104-21tail07:50:172
2836999321,25cyclictest0-21swapper/310:04:273
2836999321,20cyclictest0-21swapper/312:13:223
2836999321,18cyclictest0-21swapper/309:09:013
2836999321,13cyclictest0-21swapper/309:51:373
2836999321,13cyclictest0-21swapper/308:28:273
2836999321,12cyclictest12861-21grep07:45:163
28369993210,4cyclictest639-21dump-pmu-power11:00:003
28369993210,3cyclictest836-21runrttasks12:39:333
2836999320,21cyclictest0-21swapper/311:19:133
2836199328,21cyclictest16689-21df11:35:142
2836199325,15cyclictest28292-21/usr/sbin/munin09:35:132
2836199325,15cyclictest28292-21/usr/sbin/munin09:35:132
28361993219,3cyclictest32275-40dnf09:42:462
2836199321,3cyclictest30914-21grep07:15:022
2836199321,23cyclictest0-21swapper/210:05:502
2836999311,5cyclictest2285-21cat11:00:253
2836999311,4cyclictest3174-21acpi11:05:143
2836999311,4cyclictest17300-21/usr/sbin/munin09:10:183
28369993113,3cyclictest22319-21/usr/sbin/munin10:35:243
2836999311,14cyclictest0-21swapper/307:31:513
2836999311,14cyclictest0-21swapper/307:31:513
2836999311,13cyclictest780-21rngd07:44:433
2836999311,13cyclictest0-21swapper/309:16:113
2836999311,13cyclictest0-21swapper/307:28:033
2836999311,12cyclictest26092-21/usr/sbin/munin09:30:213
2836999311,12cyclictest0-21swapper/309:46:333
2836999310,22cyclictest0-21swapper/311:50:483
2836199316,22cyclictest848-21sssd_nss09:50:172
2836199314,16cyclictest21065-21/usr/sbin/munin08:05:202
2836199313,3cyclictest12050irq/123-ahci[0007:50:022
2836199312,25cyclictest15424-21smart_sda11:30:252
2836199311,4cyclictest8832-21df08:50:152
2836199311,26cyclictest5792-21tr07:30:122
2836199311,26cyclictest5792-21tr07:30:122
2836199311,19cyclictest17390-21wc09:10:142
2836199310,23cyclictest0-21swapper/211:05:202
2836999309,3cyclictest24701-21cstates10:40:123
2836999303,4cyclictest0-21swapper/310:30:133
28369993013,3cyclictest6966-21hddtemp_smartct08:45:143
28369993012,3cyclictest27352-21/usr/sbin/munin12:00:203
28369993011,3cyclictest2801-21unixbench_multi07:20:263
28369993010,3cyclictest14876-21timerandwakeup10:15:273
28369993010,3cyclictest13064-21pmu-power11:25:213
2836199303,15cyclictest32065-21/usr/sbin/munin08:30:152
2836199303,15cyclictest32065-21/usr/sbin/munin08:30:152
2836199302,24cyclictest26418-21smart_sda11:55:282
2836199301,4cyclictest20125-21/usr/sbin/munin10:30:182
2836199301,25cyclictest18174-21grep07:55:292
2836199301,20cyclictest30634-21ntp_states12:05:222
2836199301,19cyclictest22688-21sshd10:35:152
28361993010,17cyclictest16185-21open_files09:05:232
2836999291,4cyclictest22518-21cat08:05:283
28369992912,3cyclictest22055-21sed11:45:273
28369992912,3cyclictest22055-21sed11:45:273
2836999291,15cyclictest0-21swapper/309:27:323
2836999291,15cyclictest0-21swapper/308:18:333
28369992911,3cyclictest27605-21wc10:45:183
2836199296,20cyclictest848-21sssd_nss09:00:202
2836199294,22cyclictest848-21sssd_nss08:35:142
2836199292,3cyclictest8781-21memory07:35:202
2836199292,2cyclictest19624-21ntp_states11:40:222
2836199291,4cyclictest11484-21/usr/sbin/munin12:35:232
2836199291,3cyclictest5468-21/usr/sbin/munin09:55:172
2836199291,3cyclictest30034-21/usr/sbin/munin10:50:232
28361992910,16cyclictest26923-21unixbench-2d08:15:242
2835199292,12cyclictest24307-21perf10:40:010
2836999282,11cyclictest18855-21/usr/sbin/munin08:00:213
2836999281,5cyclictest15736-21/usr/sbin/munin10:20:203
28369992811,4cyclictest4870-21ntp_states12:20:223
2836999280,19cyclictest0-21swapper/309:01:333
2836199289,2cyclictest0-21swapper/211:50:142
2836199281,4cyclictest7362-21ntp_states08:45:202
2836199281,3cyclictest7567-21/usr/sbin/munin11:15:262
2836199281,3cyclictest26260-21/usr/sbin/munin09:30:132
2836199281,3cyclictest18855-21/usr/sbin/munin08:00:212
2836199281,3cyclictest10579-21/usr/sbin/munin11:20:192
2836199281,2cyclictest29868-21/usr/sbin/munin08:25:172
28361992811,14cyclictest4013-21fschecks_time07:25:182
2835699281,11cyclictest26313-21taskset10:42:501
2835699281,11cyclictest26313-21taskset10:42:501
2835699281,10cyclictest0-21swapper/108:39:141
2835699280,11cyclictest0-21swapper/108:55:201
78712270,4sleep07-21ksoftirqd/011:15:150
2836999279,3cyclictest0-21swapper/308:22:213
2836999272,7cyclictest0-21swapper/307:16:343
28369992712,3cyclictest0-21swapper/311:33:323
28369992710,4cyclictest25489-21df11:55:183
28369992710,3cyclictest5775-21ntp_kernel_pll_09:55:203
2836199271,4cyclictest28716-21/usr/sbin/munin07:10:152
2835699270,1cyclictest0-21swapper/111:30:161
2835199271,13cyclictest8273-21munin-run08:50:010
2835199270,12cyclictest13946-21perf11:30:010
2836999261,6cyclictest31911-21timerwakeupswit09:40:253
2836999261,12cyclictest0-21swapper/310:05:273
2835699261,17cyclictest3303-21/usr/sbin/munin09:50:191
2835699261,17cyclictest1047-21taskset12:11:121
2835699261,10cyclictest7062-21ntp_states12:25:221
2835199267,16cyclictest32469-21munin-run09:45:010
2836999251,10cyclictest0-21swapper/309:24:513
28356992514,5cyclictest0-21swapper/112:01:301
28356992511,11cyclictest29364-21kernelversion07:10:191
2836999245,3cyclictest0-21swapper/308:30:533
2836999245,3cyclictest0-21swapper/308:30:533
2835699241,9cyclictest0-21swapper/108:15:441
2835699241,15cyclictest0-21swapper/109:48:481
2835699240,9cyclictest0-21swapper/107:50:021
2835199246,6cyclictest0-21swapper/012:28:030
2835699231,17cyclictest0-21swapper/110:39:251
2835699231,17cyclictest0-21swapper/110:29:411
2835699231,17cyclictest0-21swapper/108:32:541
2835699231,17cyclictest0-21swapper/108:32:541
2835699231,17cyclictest0-21swapper/108:11:461
2835699231,16cyclictest0-21swapper/108:47:101
2835699231,14cyclictest0-21swapper/112:21:291
2835699231,14cyclictest0-21swapper/109:58:321
2835699231,10cyclictest0-21swapper/110:09:581
2835699230,8cyclictest0-21swapper/111:10:021
28351992316,4cyclictest7617-21/usr/sbin/munin10:00:190
2835199230,11cyclictest23671-21perf09:25:010
2835699228,11cyclictest17408-21grep09:10:141
2835699226,8cyclictest0-21swapper/111:51:581
2835699222,11cyclictest8214-21diskstats07:35:151
2835699222,11cyclictest11687-21diskstats10:10:161
2835699221,18cyclictest21707-21uptime10:30:281
2835699221,16cyclictest0-21swapper/111:55:451
2835699221,16cyclictest0-21swapper/111:44:041
2835699221,15cyclictest0-21swapper/111:14:281
2835699221,15cyclictest0-21swapper/107:33:181
2835699221,15cyclictest0-21swapper/107:33:181
2835699221,15cyclictest0-21swapper/107:29:271
2835699221,12cyclictest1-21systemd10:20:101
2835699220,10cyclictest0-21swapper/109:27:561
2835199229,5cyclictest0-21swapper/011:50:270
2835199221,4cyclictest0-21swapper/011:41:480
2835199221,4cyclictest0-21swapper/010:51:480
19192220,4sleep12835699cyclictest12:15:141
2835699214,9cyclictest0-21swapper/109:18:261
28356992113,3cyclictest0-21swapper/109:23:161
2835699211,18cyclictest0-21swapper/110:55:481
2835699211,17cyclictest30770-21cstates09:40:141
2835699211,16cyclictest0-21swapper/108:40:471
2835699211,15cyclictest0-21swapper/110:47:131
2835699211,15cyclictest0-21swapper/109:31:451
2835699211,15cyclictest0-21swapper/109:00:451
2835699211,15cyclictest0-21swapper/108:20:431
2835699211,15cyclictest0-21swapper/107:50:341
2835699211,15cyclictest0-21swapper/107:44:251
2835199218,3cyclictest13859-21kworker/0:211:35:180
2835199216,6cyclictest0-21swapper/009:29:290
2835199212,6cyclictest29305-21kworker/0:108:39:360
2835199211,9cyclictest636-21kworker/0:007:22:480
2835199211,12cyclictest0-21swapper/010:16:280
2835199211,10cyclictest0-21swapper/012:04:050
2835199211,10cyclictest0-21swapper/011:08:480
2835699202,4cyclictest7602-21hddtemp_smartct10:00:181
2835699201,8cyclictest0-21swapper/111:20:251
28356992014,3cyclictest1907-21sed07:20:171
28356992013,3cyclictest0-21swapper/108:50:101
2835699201,2cyclictest0-21swapper/112:30:321
28356992012,3cyclictest0-21swapper/111:48:001
28356992012,3cyclictest0-21swapper/111:48:001
28356992012,2cyclictest0-21swapper/108:25:091
2835699201,14cyclictest0-21swapper/112:08:421
2835699201,11cyclictest13547-21users11:25:261
2835699201,10cyclictest3069-21cat11:05:021
2835699200,7cyclictest0-21swapper/108:06:241
2835699200,15cyclictest0-21swapper/108:03:251
2835199206,5cyclictest0-21swapper/010:21:340
2835199206,4cyclictest0-21swapper/012:21:270
2835199201,4cyclictest22189-21wc11:45:290
2835199201,4cyclictest22189-21wc11:45:290
28351992011,2cyclictest11-21rcuc/010:25:190
2835199201,10cyclictest23267-21/usr/sbin/munin08:10:250
2835199201,10cyclictest0-21swapper/012:15:540
2835199200,11cyclictest24302-21kworker/0:110:45:370
2835699199,2cyclictest29873-21/usr/sbin/munin10:50:201
2835699193,3cyclictest15446-21diskstats09:05:161
2835699191,9cyclictest0-21swapper/109:38:371
2835699191,9cyclictest0-21swapper/109:38:371
28356991913,2cyclictest0-21swapper/111:38:591
2835699191,10cyclictest18097-21timerwakeupswit07:55:281
2835699190,17cyclictest0-21swapper/110:15:061
2835199199,3cyclictest10066-21/usr/sbin/munin07:40:190
2835199193,5cyclictest0-21swapper/011:57:400
2835199193,4cyclictest7-21ksoftirqd/011:10:210
28351991913,3cyclictest5444-21munin-run07:30:010
2835199191,10cyclictest19497-21/usr/sbin/munin09:15:220
2835199191,10cyclictest17069-21diskmemload12:36:480
2835199191,10cyclictest14634-21snmpd11:03:350
2835199191,10cyclictest14634-21snmpd07:12:520
2835199191,10cyclictest12264-21/usr/sbin/munin07:45:170
2835199191,10cyclictest0-21swapper/012:06:480
2835199191,10cyclictest0-21swapper/010:31:470
2835199190,8cyclictest0-21swapper/010:59:230
2835199188,3cyclictest0-21swapper/011:30:280
2835199187,3cyclictest32691-21/usr/sbin/munin09:45:140
2835199186,5cyclictest0-21swapper/012:34:250
2835199184,4cyclictest4132-21cpu08:40:120
2835199181,9cyclictest0-21swapper/009:57:000
2835199181,6cyclictest26250-21tr09:30:130
28351991813,2cyclictest23031-21latency_hist08:10:020
2835199181,10cyclictest0-21swapper/009:35:190
2835199181,10cyclictest0-21swapper/009:35:190
15312181,2sleep00-21swapper/008:33:390
15312181,2sleep00-21swapper/008:33:390
2835199173,2cyclictest0-21swapper/008:55:220
2835199171,7cyclictest0-21swapper/008:29:450
2835199171,3cyclictest9134-21/usr/sbin/munin10:05:210
2835199171,2cyclictest88150irq/134-enp2s0-08:51:120
28351991712,3cyclictest31398-21sed07:15:050
28351991711,4cyclictest7-21ksoftirqd/007:35:170
28351991711,3cyclictest24719-21expr10:40:130
28351991711,3cyclictest24719-21expr10:40:120
28351991711,3cyclictest10117-21entropy11:20:150
296062160,2sleep00-21swapper/008:24:480
2835199169,4cyclictest6322-21gltestperf07:30:150
2835199169,4cyclictest6322-21gltestperf07:30:150
2835199161,6cyclictest0-21swapper/008:15:170
2835199161,14cyclictest0-21swapper/009:03:290
28351991611,3cyclictest17804-21grep07:55:260
28351991611,3cyclictest16109-21seq07:51:030
28351991610,3cyclictest19585-21if_enp1s008:00:170
2835199159,4cyclictest848-21sssd_nss09:05:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional