You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 16:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Nov 30, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
42862237163,24sleep10-21swapper/107:08:081
42702235163,24sleep00-21swapper/007:07:550
42572234160,24sleep20-21swapper/207:07:462
42312229163,53sleep30-21swapper/307:07:263
1277921860,4sleep0451999cyclictest11:05:280
4530991071,104cyclictest0-21swapper/208:07:522
4530999188,2cyclictest0-21swapper/209:43:592
167232490,1sleep20-21swapper/207:35:142
47002480,1sleep10-21swapper/109:35:121
214562400,4sleep10-21swapper/110:10:291
4519993811,12cyclictest4882-21perf10:50:010
290252360,4sleep1452699cyclictest09:15:211
50302350,4sleep0451999cyclictest12:00:270
451999339,11cyclictest20581-21awk07:45:010
453699321,12cyclictest0-21swapper/307:26:373
453699311,12cyclictest0-21swapper/310:54:533
453699311,12cyclictest0-21swapper/307:33:113
453699301,13cyclictest0-21swapper/311:39:493
453699301,12cyclictest0-21swapper/310:55:573
453699300,13cyclictest0-21swapper/309:54:533
453699300,13cyclictest0-21swapper/309:54:533
453699291,14cyclictest0-21swapper/308:31:413
453699291,13cyclictest0-21swapper/308:47:253
3412290,4sleep339-21rcuc/309:25:153
453699281,13cyclictest0-21swapper/308:59:413
453699271,12cyclictest0-21swapper/312:36:353
453699271,12cyclictest0-21swapper/312:16:453
453699271,12cyclictest0-21swapper/311:22:133
453699271,12cyclictest0-21swapper/310:26:293
453699271,12cyclictest0-21swapper/309:39:413
453699271,11cyclictest0-21swapper/309:16:533
451999270,12cyclictest17430-21crond10:05:010
453699261,13cyclictest0-21swapper/309:45:333
453699261,12cyclictest25604-21diskmemload09:30:293
453699261,12cyclictest0-21swapper/310:01:093
452699261,21cyclictest23411-21munin-run09:05:011
4526992612,12cyclictest27013-21kworker/u8:108:44:111
451999260,12cyclictest11-21rcuc/009:35:000
221132260,1sleep20-21swapper/210:15:122
453699251,17cyclictest0-21swapper/312:10:593
453699251,14cyclictest0-21swapper/311:00:533
453699251,14cyclictest0-21swapper/311:00:533
453699251,13cyclictest0-21swapper/311:12:453
453699251,13cyclictest0-21swapper/310:39:413
453699251,13cyclictest0-21swapper/310:14:533
453699251,12cyclictest0-21swapper/309:57:493
453699251,11cyclictest0-21swapper/312:21:253
453699251,11cyclictest0-21swapper/311:50:533
453699251,11cyclictest0-21swapper/310:20:453
453699251,10cyclictest0-21swapper/307:35:093
452699252,4cyclictest29067-21munin-run11:45:011
452699251,3cyclictest15387-21ls12:25:201
4526992510,6cyclictest674-21systemd-journal11:50:531
453699241,12cyclictest0-21swapper/307:43:413
453699241,11cyclictest0-21swapper/309:22:293
453699241,11cyclictest0-21swapper/308:16:533
453699240,12cyclictest0-21swapper/310:30:213
453699240,10cyclictest0-21swapper/311:16:373
452699244,18cyclictest0-21swapper/111:46:221
452699243,13cyclictest0-21swapper/107:33:471
452699241,17cyclictest0-21swapper/110:53:351
451999247,14cyclictest31588-21perf08:10:010
4519992410,11cyclictest27821-21awk09:15:010
453699231,13cyclictest0-21swapper/307:24:063
453699231,12cyclictest0-21swapper/308:11:093
453699231,11cyclictest0-21swapper/310:49:333
453099233,12cyclictest24221-21kworker/2:212:28:192
453099231,4cyclictest0-21swapper/210:11:272
452699231,4cyclictest24077-21open_files07:50:271
452699231,2cyclictest15239-21perf10:00:011
452699231,16cyclictest0-21swapper/112:30:031
452699231,10cyclictest0-21swapper/111:05:101
451999231,3cyclictest86550irq/126-enp1s0-09:41:380
4519992310,10cyclictest7-21ksoftirqd/010:30:450
140412230,12sleep230-21rcuc/207:30:012
453699222,5cyclictest0-21swapper/310:19:533
453699221,13cyclictest0-21swapper/311:27:573
453699221,13cyclictest0-21swapper/309:00:453
453699221,12cyclictest0-21swapper/312:28:133
453699221,12cyclictest0-21swapper/312:07:093
453699221,12cyclictest0-21swapper/312:07:093
453699221,12cyclictest0-21swapper/309:06:373
453699221,12cyclictest0-21swapper/308:41:573
4530992217,3cyclictest20067-21grep07:40:282
4530992212,3cyclictest0-21swapper/210:02:322
452699227,3cyclictest0-21swapper/109:49:151
452699224,8cyclictest88050irq/133-enp2s0-11:30:211
452699222,15cyclictest0-21swapper/111:12:311
452699222,15cyclictest0-21swapper/110:06:041
452699221,9cyclictest0-21swapper/108:14:081
452699221,4cyclictest0-21swapper/107:58:141
452699221,19cyclictest0-21swapper/108:35:201
452699221,18cyclictest7621-21cstates10:55:131
452699221,16cyclictest0-21swapper/111:58:301
452699221,16cyclictest0-21swapper/110:28:181
452699221,15cyclictest0-21swapper/110:33:171
452699221,10cyclictest780-21rngd10:15:351
452699221,10cyclictest7368-21sort09:40:151
452699221,10cyclictest0-21swapper/110:02:111
452699221,10cyclictest0-21swapper/108:47:281
4526992210,3cyclictest25329-21ntp_states10:20:231
4526992210,10cyclictest23-21ksoftirqd/112:15:131
451999227,6cyclictest0-21swapper/012:18:420
4519992211,3cyclictest17460-21ntp_offset07:35:220
184152220,8sleep121-21rcuc/111:20:141
453699212,11cyclictest14634-21snmpd11:07:463
453699211,14cyclictest0-21swapper/311:33:473
453699211,13cyclictest0-21swapper/308:36:373
453699211,12cyclictest4725-21/usr/sbin/munin07:10:133
453699211,12cyclictest0-21swapper/307:48:533
453699211,11cyclictest0-21swapper/312:00:293
453699211,11cyclictest0-21swapper/309:44:293
453699211,11cyclictest0-21swapper/308:06:293
453099218,3cyclictest19330-21expr08:55:132
453099211,9cyclictest0-21swapper/212:38:042
453099211,8cyclictest0-21swapper/209:10:232
453099211,7cyclictest0-21swapper/209:55:532
453099211,4cyclictest0-21swapper/211:20:092
453099211,3cyclictest0-21swapper/212:11:552
452699216,3cyclictest17639-21unixbench_singl11:15:271
452699214,9cyclictest19273-21diskstats12:35:161
452699212,3cyclictest0-21swapper/107:41:371
4526992117,3cyclictest0-21swapper/112:02:061
4526992117,3cyclictest0-21swapper/108:50:371
4526992117,3cyclictest0-21swapper/108:50:371
4526992114,3cyclictest0-21swapper/110:49:411
452699211,19cyclictest0-21swapper/112:06:021
452699211,19cyclictest0-21swapper/112:06:021
452699211,18cyclictest0-21swapper/108:00:291
452699211,11cyclictest21408-21memory11:25:191
452699210,15cyclictest0-21swapper/111:00:081
452699210,15cyclictest0-21swapper/111:00:081
451999213,10cyclictest28036-21/usr/sbin/munin09:15:200
451999211,4cyclictest11477-21munin-run11:05:020
451999211,4cyclictest11477-21munin-run11:05:020
451999211,11cyclictest5352-21grep08:20:260
451999211,11cyclictest31449-21ps09:20:220
451999211,11cyclictest23619-21grep10:15:260
451999211,11cyclictest19223-21/usr/sbin/munin08:55:240
451999211,10cyclictest22000-21grep11:25:270
451999210,5cyclictest0-21swapper/010:12:340
180872211,3sleep10-21swapper/107:36:311
453699201,11cyclictest25838-21/usr/sbin/munin09:10:143
453699201,11cyclictest0-21swapper/308:01:333
453699201,11cyclictest0-21swapper/307:56:213
453699201,11cyclictest0-21swapper/307:54:213
453699201,10cyclictest0-21swapper/311:46:533
453699201,10cyclictest0-21swapper/308:50:233
453699201,10cyclictest0-21swapper/308:50:233
453099208,3cyclictest29378-21awk08:05:012
453099206,8cyclictest0-21swapper/212:33:162
453099201,6cyclictest0-21swapper/210:56:072
452699209,3cyclictest31706-21cat08:10:021
452699202,17cyclictest0-21swapper/108:21:071
4526992017,2cyclictest0-21swapper/108:31:341
4526992016,3cyclictest0-21swapper/112:10:491
4526992013,2cyclictest0-21swapper/110:36:431
452699201,2cyclictest258322sleep109:10:081
452699201,18cyclictest0-21swapper/109:21:281
452699201,17cyclictest0-21swapper/108:25:411
452699201,11cyclictest22385-21users07:45:271
452699201,10cyclictest4861-21cut07:10:121
452699201,10cyclictest0-21swapper/112:20:191
4526992010,3cyclictest21188-21latency_hist09:00:021
451999208,3cyclictest0-21swapper/011:30:580
451999207,7cyclictest0-21swapper/011:35:580
451999206,5cyclictest0-21swapper/011:46:300
451999206,5cyclictest0-21swapper/011:16:080
451999206,4cyclictest0-21swapper/009:55:530
451999205,6cyclictest0-21swapper/010:42:260
451999201,7cyclictest13237-21head08:40:170
4519992013,4cyclictest11592-21fschecks_time09:50:150
4519992013,4cyclictest11592-21fschecks_time09:50:150
451999201,11cyclictest14634-21snmpd07:22:030
451999201,11cyclictest12008-21/usr/sbin/munin07:25:230
451999201,10cyclictest18255-21tr10:05:180
451999201,10cyclictest0-21swapper/009:29:370
451999200,9cyclictest0-21swapper/007:55:190
221182200,3sleep20-21swapper/211:25:462
183702200,2sleep00-21swapper/012:30:590
93432191,3sleep20-21swapper/207:17:052
78332191,2sleep00-21swapper/008:27:560
453699198,3cyclictest0-21swapper/311:42:133
4536991914,3cyclictest2659-21idleruntime-cro10:45:003
453699191,14cyclictest0-21swapper/312:30:213
453699191,10cyclictest8680-21ntp_states07:15:243
453699191,10cyclictest17669-21/usr/sbin/munin10:05:213
453099193,3cyclictest0-21swapper/209:30:382
453099191,7cyclictest0-21swapper/210:48:182
453099191,6cyclictest0-21swapper/209:35:262
4530991914,3cyclictest25613-21grep07:55:162
453099191,12cyclictest0-21swapper/210:20:232
453099191,11cyclictest0-21swapper/212:18:512
452699196,4cyclictest8531-21memory07:15:201
4526991915,3cyclictest0-21swapper/107:26:431
4526991914,3cyclictest12558-21grep09:50:251
4526991914,3cyclictest12558-21grep09:50:251
452699191,2cyclictest0-21swapper/109:27:111
452699191,16cyclictest0-21swapper/109:31:461
452699191,16cyclictest0-21swapper/107:20:221
452699191,14cyclictest0-21swapper/111:36:001
452699191,10cyclictest2396-21chrt10:41:401
451999198,3cyclictest20546-21sshd12:35:410
451999196,4cyclictest0-21swapper/009:45:080
451999196,4cyclictest0-21swapper/009:35:510
451999191,3cyclictest5920-21cat12:05:140
451999191,3cyclictest5920-21cat12:05:140
451999191,2cyclictest0-21swapper/011:50:220
451999191,10cyclictest3178-21sshd11:58:400
451999191,10cyclictest21444-21cpu09:00:120
451999191,10cyclictest14827-21/usr/sbin/munin08:45:190
451999191,10cyclictest14504-21gltestperf11:10:180
270832190,2sleep20-21swapper/211:40:072
17732191,3sleep01974-21df08:15:160
96682181,2sleep00-21swapper/012:12:040
453699181,3cyclictest0-21swapper/308:24:213
453699181,10cyclictest0-21swapper/308:25:223
453099182,6cyclictest30636-21munin-run10:35:012
453099182,13cyclictest23918-21grep07:50:202
453099181,7cyclictest32-21ksoftirqd/212:21:452
453099181,7cyclictest17014-21memory11:15:202
453099181,7cyclictest0-21swapper/211:08:212
4530991814,2cyclictest32-21ksoftirqd/211:40:012
4530991813,3cyclictest9618-21cpuspeed_turbos11:00:132
4530991813,3cyclictest9618-21cpuspeed_turbos11:00:132
453099181,15cyclictest0-21swapper/209:20:312
453099181,15cyclictest0-21swapper/209:00:162
453099180,9cyclictest25604-21diskmemload09:49:492
453099180,12cyclictest0-21swapper/210:35:282
452699181,3cyclictest0-21swapper/109:06:001
452699181,3cyclictest0-21swapper/108:15:341
451999185,4cyclictest0-21swapper/011:43:110
451999181,9cyclictest0-21swapper/007:15:170
451999181,4cyclictest8236-21/usr/sbin/munin08:30:200
451999181,10cyclictest32608-21kworker/dying10:50:280
451999180,10cyclictest0-21swapper/010:20:150
32292180,2sleep20-21swapper/211:59:222
164012180,2sleep00-21swapper/012:28:540
153802180,4sleep011-21rcuc/007:30:240
128482181,2sleep20-21swapper/209:52:452
128482181,2sleep20-21swapper/209:52:452
63902171,2sleep00-21swapper/007:11:170
4536991712,3cyclictest2893-21grep11:55:283
453099175,3cyclictest86850irq/129-enp1s0-08:45:122
453099173,5cyclictest0-21swapper/209:25:202
453099172,12cyclictest1409-21cat08:15:012
453099171,7cyclictest14640-21cat08:45:012
453099171,7cyclictest0-21swapper/211:52:332
453099171,7cyclictest0-21swapper/211:10:492
453099171,7cyclictest0-21swapper/210:28:072
453099171,7cyclictest0-21swapper/208:53:582
453099171,7cyclictest0-21swapper/208:53:582
453099171,7cyclictest0-21swapper/207:30:402
453099171,6cyclictest25618-21idleruntime-cro09:10:012
453099171,6cyclictest10401-21cat08:35:012
453099171,6cyclictest0-21swapper/209:16:102
453099171,5cyclictest0-21swapper/210:50:502
4530991712,3cyclictest7279-21grep12:05:272
4530991712,3cyclictest7279-21grep12:05:272
4530991712,3cyclictest23019-21expr11:30:132
4530991712,3cyclictest17958-21expr10:05:152
453099171,15cyclictest0-21swapper/210:40:322
453099171,15cyclictest0-21swapper/208:37:472
453099171,15cyclictest0-21swapper/207:12:182
4530991711,3cyclictest3834-21/usr/sbin/munin08:20:242
4530991711,3cyclictest30333-21ntp_kernel_pll_11:45:212
451999177,3cyclictest23631-21/usr/sbin/munin09:05:230
451999174,7cyclictest17971-21kworker/u8:011:20:050
451999173,6cyclictest86550irq/126-enp1s0-10:27:420
451999171,6cyclictest11926-21sshd08:35:260
451999171,5cyclictest0-21swapper/012:21:160
451999171,14cyclictest0-21swapper/008:13:110
451999171,14cyclictest0-21swapper/008:00:100
4519991711,3cyclictest31019-21tr10:35:150
451999171,10cyclictest0-21swapper/010:55:190
179692170,2sleep00-21swapper/008:50:200
179692170,2sleep00-21swapper/008:50:200
453099161,5cyclictest0-21swapper/212:00:092
4530991612,3cyclictest0-21swapper/208:17:142
4530991611,3cyclictest6038-21/usr/sbin/munin08:25:272
4530991611,3cyclictest10630-21irqstats07:20:232
451999161,8cyclictest0-21swapper/007:50:290
218992161,12sleep222016-21rt-features07:45:232
4519991510,3cyclictest22898-21date07:50:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional