You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 17:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Nov 18, 2025 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291922238162,24sleep20-21swapper/207:05:512
291782236165,58sleep30-21swapper/307:05:403
293092204167,25sleep10-21swapper/107:07:231
292052201164,24sleep00-21swapper/007:06:020
2567521630,4sleep3401ktimersoftd/310:40:013
1220821070,1sleep30-21swapper/308:55:153
29618999995,3cyclictest0-21swapper/210:06:482
29618996562,2cyclictest0-21swapper/208:56:202
75562640,5sleep232-21ksoftirqd/211:10:192
15062610,3sleep30-21swapper/309:45:013
78672590,1sleep10-21swapper/112:25:171
93772560,4sleep121-21rcuc/111:15:161
2960899331,15cyclictest5774-21munin-run09:55:020
2960899326,12cyclictest24296-21munin-run08:10:010
2961299311,18cyclictest21824-21ntp_states09:15:231
48192290,7sleep3401ktimersoftd/311:05:193
2960899291,13cyclictest19898-21awk08:00:020
2960899284,11cyclictest10816-21idleruntime-cro11:20:000
29608992711,13cyclictest23998-21munin-run11:50:010
2961299265,11cyclictest28739-21tr12:00:141
2961299261,3cyclictest20457-21df_inode08:00:151
2961299260,17cyclictest0-21swapper/108:40:261
2961299252,3cyclictest87950irq/132-enp2s0-10:20:051
2961299252,16cyclictest0-21swapper/107:55:251
2961299251,3cyclictest1272-21mailstats10:55:291
29612992513,3cyclictest0-21swapper/108:46:001
2961299241,20cyclictest3577-21perf09:50:011
2961299232,19cyclictest0-21swapper/110:25:081
29612992314,3cyclictest0-21swapper/107:10:291
2961299231,3cyclictest15428-21switchtime09:00:261
2961299231,19cyclictest32098-21cat08:25:221
2961299231,19cyclictest28182-21seq11:57:261
2961299231,19cyclictest15115-21taskset11:29:051
29612992310,3cyclictest28342-21diskstats10:45:161
2960899230,11cyclictest13018-21perf11:25:010
2962399224,9cyclictest0-21swapper/310:25:353
29623992217,3cyclictest26639-21grep09:25:263
29623992217,3cyclictest26639-21grep09:25:263
2961899221,6cyclictest0-21swapper/211:17:242
2961899221,11cyclictest16489-21ntp_states11:30:202
2961299229,3cyclictest0-21swapper/107:19:391
2961299229,3cyclictest0-21swapper/107:19:391
2961299222,15cyclictest0-21swapper/107:35:451
29612992214,3cyclictest0-21swapper/108:16:351
2961299221,19cyclictest0-21swapper/112:05:161
2961299221,16cyclictest0-21swapper/109:21:051
2961299221,15cyclictest0-21swapper/108:52:171
2961299221,12cyclictest23934-21df_inode10:35:151
2961299220,5cyclictest0-21swapper/112:13:551
2961299220,16cyclictest0-21swapper/109:14:591
2960899223,10cyclictest4440-21/usr/sbin/munin11:05:280
2960899221,4cyclictest25095-21grep10:35:290
23042220,5sleep230-21rcuc/207:20:022
23042220,5sleep230-21rcuc/207:20:012
2962399212,11cyclictest14804-21/usr/sbin/munin10:15:273
2962399211,11cyclictest580-21/usr/sbin/munin12:10:153
2962399211,11cyclictest29885-21sendmail_mailtr12:00:263
2962399211,11cyclictest2534-21/usr/sbin/munin07:20:163
2962399211,11cyclictest14634-21snmpd10:23:143
2961899218,10cyclictest581-21systemd-journal08:55:012
2961899213,15cyclictest9080-21grep10:00:202
2961899211,6cyclictest0-21swapper/209:23:002
2961899211,17cyclictest15872-21ntp_states10:15:242
2961299215,9cyclictest0-21swapper/109:32:071
2961299212,17cyclictest0-21swapper/110:02:321
29612992117,3cyclictest0-21swapper/111:05:461
29612992113,2cyclictest0-21swapper/109:05:221
29612992113,2cyclictest0-21swapper/107:30:341
2961299211,2cyclictest0-21swapper/107:53:431
29612992112,3cyclictest0-21swapper/107:45:541
2961299211,18cyclictest0-21swapper/111:00:241
2961299211,18cyclictest0-21swapper/110:15:011
2961299211,15cyclictest0-21swapper/108:56:171
2961299211,15cyclictest0-21swapper/108:30:411
2961299211,15cyclictest0-21swapper/108:22:341
2961299211,15cyclictest0-21swapper/108:22:341
2961299211,15cyclictest0-21swapper/108:11:341
2961299211,15cyclictest0-21swapper/107:43:241
29612992111,3cyclictest0-21swapper/109:29:561
29612992111,3cyclictest0-21swapper/109:29:561
2961299211,12cyclictest0-21swapper/110:17:491
29612992111,2cyclictest0-21swapper/110:07:281
2961299211,10cyclictest6344-21diskstats09:55:141
2961299211,10cyclictest2881-21diskstats07:20:131
2961299211,10cyclictest247052fschecks_time11:50:171
2960899219,3cyclictest25160-21/usr/sbin/munin09:25:260
2960899219,3cyclictest25160-21/usr/sbin/munin09:25:260
2960899212,5cyclictest12735-1kworker/u9:212:32:550
2960899211,11cyclictest31128-21/usr/sbin/munin08:25:190
2960899211,11cyclictest30837-21/usr/sbin/munin12:05:260
2960899211,11cyclictest24226-21/usr/sbin/munin11:50:230
195162211,2sleep214634-21snmpd11:39:102
195162211,2sleep214634-21snmpd11:39:102
2962399205,8cyclictest86550irq/126-enp1s0-09:55:263
2962399201,7cyclictest0-21swapper/311:29:423
29623992015,3cyclictest18331-21diskmemload10:05:443
29623992011,3cyclictest0-21swapper/311:21:333
2962399201,11cyclictest9054-21chrt12:27:493
2962399201,11cyclictest8832-21/usr/sbin/munin11:15:213
2962399201,11cyclictest14634-21snmpd11:03:233
2962399201,10cyclictest28688-21latency_hist08:20:023
2962399201,10cyclictest11339-21taskset12:33:563
2961899207,4cyclictest4860-21tr07:25:132
2961899202,4cyclictest12727-21/usr/sbin/munin12:35:262
2961899201,5cyclictest0-21swapper/210:49:422
2961899201,16cyclictest17675-21hddtemp_smartct10:20:162
2961899201,12cyclictest0-21swapper/207:40:022
2961299207,4cyclictest935-21sed09:40:271
2961299205,2cyclictest0-21swapper/111:35:201
2961299205,2cyclictest0-21swapper/111:35:201
2961299202,3cyclictest0-21swapper/111:42:431
29612992016,3cyclictest0-21swapper/110:43:521
2961299201,1cyclictest0-21swapper/111:45:151
2961299201,18cyclictest0-21swapper/111:10:311
2961299201,17cyclictest0-21swapper/110:50:491
2961299201,17cyclictest0-21swapper/108:37:371
2961299201,11cyclictest0-21swapper/109:35:211
29612992010,7cyclictest13506-21seq12:38:321
2961299200,2cyclictest86850irq/129-enp1s0-12:31:471
2961299200,15cyclictest0-21swapper/108:09:381
2961299200,15cyclictest0-21swapper/107:26:321
2960899209,3cyclictest28624-21/usr/sbin/munin12:00:210
2960899209,3cyclictest16361-21/usr/sbin/munin09:05:220
2960899207,5cyclictest0-21swapper/010:15:390
2960899206,4cyclictest0-21swapper/012:25:050
2960899205,6cyclictest0-21swapper/009:12:500
2960899201,11cyclictest30197-21/usr/sbin/munin10:50:190
2960899201,11cyclictest17710-21cpuspeed_turbos11:35:130
2960899201,11cyclictest17710-21cpuspeed_turbos11:35:130
2960899201,11cyclictest14634-21snmpd08:37:490
2960899201,11cyclictest14498-21smartctl11:25:280
2960899201,11cyclictest0-21swapper/012:35:190
113152201,2sleep20-21swapper/212:33:382
58552194,11sleep15942-21latency12:20:201
2962399198,3cyclictest7558-21/usr/sbin/munin08:45:193
2962399193,8cyclictest86550irq/126-enp1s0-10:11:273
2962399193,3cyclictest86550irq/126-enp1s0-09:30:213
2962399192,9cyclictest0-21swapper/310:55:203
2962399191,6cyclictest0-21swapper/309:19:093
2962399191,3cyclictest1943-21df_inode09:45:153
29623991913,3cyclictest8814-21gltestperf10:00:183
29623991913,3cyclictest20893-21ntp_states11:40:233
2962399191,11cyclictest842-21lldpd12:08:403
2962399191,10cyclictest29562-21/usr/sbin/munin09:35:183
2962399191,10cyclictest24129-21ps09:20:243
2962399191,10cyclictest0-21swapper/310:43:403
2961899195,11cyclictest2750-21tr09:45:242
2961899191,9cyclictest590-21expr09:40:262
2961899191,8cyclictest0-21swapper/211:51:342
2961899191,7cyclictest0-21swapper/210:40:012
29618991913,3cyclictest23587-21smartctl08:05:272
29612991915,3cyclictest0-21swapper/111:20:101
2961299191,2cyclictest0-21swapper/109:50:491
2961299191,17cyclictest0-21swapper/110:30:421
2961299191,17cyclictest0-21swapper/110:30:421
2961299190,2cyclictest0-21swapper/112:16:061
2961299190,17cyclictest0-21swapper/111:30:171
2960899191,9cyclictest0-21swapper/010:28:330
29608991912,4cyclictest13141-21seq07:43:160
2960899191,10cyclictest31762-21/usr/sbin/munin09:40:300
2960899190,12cyclictest0-21swapper/009:35:290
226222190,2sleep00-21swapper/009:18:510
171112191,3sleep00-21swapper/011:31:290
135292191,14sleep013802-21cstates07:45:130
109332190,2sleep00-21swapper/007:38:070
2962399188,3cyclictest14634-21snmpd11:12:383
2962399185,4cyclictest31601-21fschecks_time08:25:183
2962399182,7cyclictest941-21/usr/sbin/munin08:30:233
2962399182,7cyclictest28922-21/usr/sbin/munin08:20:173
2962399182,7cyclictest28922-21/usr/sbin/munin08:20:163
2962399182,6cyclictest20672-21perl08:00:173
2962399181,9cyclictest14261-21runrttasks07:45:173
2962399181,7cyclictest0-21swapper/311:31:323
2962399181,6cyclictest0-21swapper/312:35:313
2962399181,5cyclictest18342-21perf09:10:003
29623991813,3cyclictest18564-21/usr/sbin/munin09:10:193
29623991811,4cyclictest5796-21sh07:25:233
29623991811,4cyclictest22635-21fgrep10:30:263
29623991811,4cyclictest22635-21fgrep10:30:253
2962399181,10cyclictest0-21swapper/311:37:513
2962399181,10cyclictest0-21swapper/311:37:513
2962399181,10cyclictest0-21swapper/309:50:313
2961899188,2cyclictest86550irq/126-enp1s0-11:06:282
2961899188,2cyclictest19893-21awk08:00:012
2961899185,4cyclictest15620-21unixbench-2d09:00:272
2961899181,8cyclictest5998-21/usr/sbin/munin09:55:172
2961899181,7cyclictest780-21rngd12:11:222
2961899181,7cyclictest0-21swapper/210:53:052
2961899181,7cyclictest0-21swapper/210:31:542
2961899181,7cyclictest0-21swapper/210:31:542
2961899181,7cyclictest0-21swapper/209:25:162
2961899181,7cyclictest0-21swapper/209:25:162
2961899181,6cyclictest29955-21grep07:10:172
29618991812,3cyclictest20116-21/usr/sbin/munin08:00:192
29618991811,4cyclictest7930-21tail12:25:172
2961899181,14cyclictest26918-21tune2fs11:55:162
2960899185,3cyclictest0-21swapper/011:00:490
2960899184,7cyclictest21079-21kworker/u8:011:13:430
2960899181,9cyclictest18331-21diskmemload11:40:240
2960899181,8cyclictest1946-21sshd07:15:350
2960899181,8cyclictest1946-21sshd07:15:350
2960899181,6cyclictest18309-21switchtime10:20:250
29608991813,3cyclictest16113-21cat09:04:570
2960899181,14cyclictest18331-21diskmemload10:07:500
2960899181,14cyclictest0-21swapper/009:55:180
2960899181,11cyclictest0-21swapper/012:20:080
2960899181,10cyclictest0-21swapper/011:55:080
2960899180,5cyclictest0-21swapper/009:33:120
2962399179,5cyclictest848-21sssd_nss07:40:133
2962399176,3cyclictest86550irq/126-enp1s0-09:00:253
2962399173,9cyclictest31778-21/usr/sbin/munin07:15:013
2962399173,4cyclictest0-21swapper/312:20:273
2962399171,9cyclictest0-21swapper/312:17:133
2962399171,7cyclictest24226-21/usr/sbin/munin11:50:183
2962399171,6cyclictest22485-21cut11:45:183
2962399171,6cyclictest0-21swapper/311:58:353
2962399171,4cyclictest9832-21cpuspeed_turbos08:50:143
29623991713,2cyclictest0-21swapper/308:40:203
2962399171,2cyclictest22676-21entropy08:05:183
29623991712,3cyclictest8422-21unixbench_multi07:30:253
29623991712,3cyclictest10173-21ntp_kernel_pll_07:35:213
2962399171,15cyclictest0-21swapper/310:50:203
2962399171,13cyclictest3161-21/usr/sbin/munin08:35:163
2962399171,13cyclictest27992-21/usr/sbin/munin10:45:293
2961899179,3cyclictest0-21swapper/207:20:592
2961899173,11cyclictest16769-21ntpq07:50:202
2961899172,6cyclictest0-21swapper/211:25:192
2961899171,9cyclictest0-21swapper/210:56:332
2961899171,9cyclictest0-21swapper/209:33:452
2961899171,7cyclictest24526-21/usr/sbin/munin08:10:252
2961899171,7cyclictest14634-21snmpd07:47:252
2961899171,6cyclictest8192-21/usr/sbin/munin07:30:232
2961899171,6cyclictest0-21swapper/211:42:002
2961899171,6cyclictest0-21swapper/210:40:232
2961899171,6cyclictest0-21swapper/209:38:432
29618991712,3cyclictest3562-21latency_hist09:50:022
29618991712,3cyclictest30-21rcuc/208:39:132
29618991712,3cyclictest22713-21hostname11:45:192
29618991712,3cyclictest1664-21sort08:30:172
29618991712,3cyclictest16361-21/usr/sbin/munin09:05:232
29618991712,3cyclictest11337-21expr11:20:142
2961899171,14cyclictest0-21swapper/210:25:092
29618991711,3cyclictest31107-21cut08:25:012
2961899170,3cyclictest86550irq/126-enp1s0-10:13:242
2960899174,4cyclictest3986-21sed07:20:250
2960899172,12cyclictest15719-21/usr/sbin/munin07:50:230
2960899171,9cyclictest32709-21kworker/u8:012:10:260
2960899171,9cyclictest0-21swapper/012:16:360
2960899171,7cyclictest4731-21/usr/sbin/munin07:25:160
2960899171,6cyclictest0-21swapper/008:50:240
2960899171,4cyclictest14634-21snmpd07:32:230
2960899171,4cyclictest12608-21/usr/sbin/munin10:10:190
2960899171,3cyclictest86550irq/126-enp1s0-10:41:430
29608991712,3cyclictest30812-21netstat07:10:230
29608991711,3cyclictest26974-21expr08:15:150
29608991711,3cyclictest22024-21gltestperf10:30:180
29608991711,3cyclictest22024-21gltestperf10:30:170
2960899171,10cyclictest0-21swapper/010:45:250
2960899170,8cyclictest361-21kworker/u8:109:22:030
286012171,2sleep20-21swapper/208:19:032
308392161,2sleep20-21swapper/208:24:292
2962399161,14cyclictest0-21swapper/307:55:043
29623991611,3cyclictest439-21expr07:15:173
29623991611,3cyclictest439-21expr07:15:173
29623991611,3cyclictest25655-21pmu-power08:10:253
2961899161,7cyclictest0-21swapper/209:15:362
2961899161,6cyclictest28746-21cpuspeed_turbos12:00:142
2961899161,6cyclictest0-21swapper/211:00:232
2961899161,5cyclictest0-21swapper/212:20:132
2961899161,5cyclictest0-21swapper/212:15:392
29618991612,2cyclictest5361-21/usr/sbin/munin08:40:172
2961899161,13cyclictest31649-21irqstats12:05:202
29618991611,3cyclictest18598-21cpu09:10:132
29618991610,3cyclictest31266-21grep08:25:152
2961899160,10cyclictest0-21swapper/207:35:182
2960899168,6cyclictest361-21kworker/u8:108:40:080
2960899165,3cyclictest0-21swapper/009:45:100
2960899161,9cyclictest0-21swapper/008:55:240
2960899161,4cyclictest28922-21/usr/sbin/munin08:20:260
2960899161,4cyclictest28922-21/usr/sbin/munin08:20:260
29608991612,3cyclictest0-21swapper/010:00:030
29608991611,3cyclictest7756-21cat08:45:150
29608991611,3cyclictest21884-21seq08:02:310
29608991610,3cyclictest32555-21cpuspeed_turbos10:55:140
262672161,2sleep00-21swapper/008:12:150
175682160,2sleep30-21swapper/307:53:393
29618991510,3cyclictest8184-21grep08:45:172
2960899151,11cyclictest941-21/usr/sbin/munin08:30:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional