You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-28 - 17:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 28, 2025 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
269612238163,24sleep20-21swapper/207:06:112
270392227198,19sleep10-21swapper/107:07:111
271262202166,24sleep00-21swapper/007:08:190
270052200165,23sleep30-21swapper/307:06:443
48121660,4sleep02734499cyclictest07:20:130
2176521620,3sleep32736199cyclictest11:50:023
856421530,5sleep22735499cyclictest08:50:222
204021440,6sleep12735099cyclictest08:35:231
226672910,3sleep22735499cyclictest11:50:162
179172530,1sleep00-21swapper/011:40:140
120642500,4sleep339-21rcuc/309:00:143
295092430,1sleep00-21swapper/010:50:420
176592430,4sleep217661-21latency_hist08:00:002
27344993611,12cyclictest23268-21awk10:40:010
2736199318,14cyclictest835-21runrttasks11:43:073
2736199306,5cyclictest3222-21/usr/sbin/munin12:20:153
2736199292,12cyclictest751-21rngd09:15:143
224762290,4sleep2311ktimersoftd/208:10:122
2736199281,8cyclictest27043-21/usr/sbin/munin12:00:203
2735099281,14cyclictest0-21swapper/112:09:101
2734499281,14cyclictest5678-21munin-run10:00:020
83232270,4sleep12735099cyclictest07:35:281
2736199273,9cyclictest576-21grep12:15:153
27361992715,8cyclictest3302-40dnf09:52:573
2736199269,10cyclictest22141-21cat11:50:123
2736199261,11cyclictest971-21ntp_states11:00:233
27361992610,8cyclictest5904-21/usr/sbin/munin10:00:283
2735099262,11cyclictest19487-21munin-run11:45:001
2735099261,17cyclictest0-21swapper/109:53:471
2735099261,14cyclictest0-21swapper/111:46:261
242982260,3sleep30-21swapper/310:40:183
241582260,2sleep00-21swapper/011:55:120
2736199254,11cyclictest20062-21/usr/sbin/munin08:05:253
2736199251,11cyclictest0-21swapper/307:50:553
2736199250,11cyclictest23514-21gltestperf09:25:163
27350992512,6cyclictest21015-21kworker/u8:010:46:001
27350992512,6cyclictest21015-21kworker/u8:010:46:001
2736199247,5cyclictest4682-21diskstats11:10:143
2736199244,13cyclictest16515-21kworker/3:208:53:343
2736199243,3cyclictest0-21swapper/307:59:353
2736199242,7cyclictest87250irq/129-enp1s0-11:25:253
2736199241,10cyclictest28101-21perl07:10:153
2736199241,10cyclictest0-21swapper/308:26:153
27361992410,2cyclictest88650irq/133-enp2s0-07:42:093
2736199240,11cyclictest0-21swapper/312:13:503
2735099241,16cyclictest0-21swapper/109:59:501
2735099241,15cyclictest87150irq/128-enp1s0-12:39:401
2735099241,14cyclictest0-21swapper/110:11:451
2735099240,3cyclictest0-21swapper/111:10:241
2735099240,17cyclictest0-21swapper/111:28:081
2734499247,15cyclictest30500-21awk12:10:020
2734499240,13cyclictest29252-21perf09:40:010
314352230,3sleep09-21rcu_preempt07:15:200
2736199237,3cyclictest8783-21kworker/u8:011:22:493
2736199235,8cyclictest0-21swapper/312:05:143
2736199234,16cyclictest3769-21taskset09:55:133
2736199233,9cyclictest29996-21grep09:40:163
2736199233,3cyclictest22658-21awk09:25:013
2736199231,7cyclictest0-21swapper/312:26:083
2736199231,7cyclictest0-21swapper/310:49:073
2736199231,7cyclictest0-21swapper/310:49:073
2736199231,14cyclictest32635-21/usr/sbin/munin07:20:203
2736199230,9cyclictest11378-21/usr/sbin/munin10:10:243
2736199230,12cyclictest0-21swapper/310:25:193
2735099237,4cyclictest0-21swapper/111:34:021
2735099231,19cyclictest24437-21df11:55:121
2735099231,14cyclictest30361-21cat09:40:181
2735099231,13cyclictest0-21swapper/111:20:211
2735099231,10cyclictest0-21swapper/112:14:481
2735099231,10cyclictest0-21swapper/111:37:261
2735099231,10cyclictest0-21swapper/110:30:071
2735099231,10cyclictest0-21swapper/109:39:111
2734499231,12cyclictest0-21swapper/008:32:170
27344992310,2cyclictest87150irq/128-enp1s0-10:00:180
209092231,3sleep121021-21df_inode09:20:141
132162231,18sleep013230-21munin-run07:50:010
2736199229,3cyclictest16515-21kworker/3:208:35:143
2736199227,3cyclictest0-21swapper/308:48:343
2736199225,4cyclictest12050irq/123-ahci[0010:20:203
2736199222,8cyclictest0-21swapper/309:06:163
2736199221,9cyclictest31662-21/usr/sbin/munin09:45:213
2736199221,7cyclictest0-21swapper/309:34:343
2736199221,7cyclictest0-21swapper/309:34:343
2736199221,5cyclictest0-21swapper/311:55:163
2736199221,10cyclictest15662-21diskstats11:35:143
2736199221,10cyclictest0-21swapper/308:32:483
2736199220,13cyclictest8791-21smartctl10:05:173
2736199220,11cyclictest7777-21sed12:30:163
2735099228,4cyclictest0-21swapper/109:19:571
2735099221,5cyclictest29592-21cat07:15:021
27350992214,2cyclictest0-21swapper/107:18:051
2735099221,3cyclictest0-21swapper/110:04:301
27350992213,3cyclictest0-21swapper/108:03:501
2735099221,18cyclictest7802-21expr08:50:121
2735099221,18cyclictest26135-21seq08:16:331
2735099221,18cyclictest26135-21seq08:16:331
2735099221,18cyclictest16916-21smartctl07:55:251
2735099221,14cyclictest0-21swapper/112:15:561
2735099221,12cyclictest7775-21gltestperf12:30:171
2735099221,11cyclictest5548-21cstates08:45:121
2735099221,11cyclictest26428-21sed09:30:251
2735099221,11cyclictest26428-21sed09:30:251
2735099221,10cyclictest0-21swapper/112:28:181
2734499222,10cyclictest23489-21users11:50:260
2734499221,10cyclictest25911-21kworker/0:012:00:340
2734499220,13cyclictest0-21swapper/011:15:140
203982221,3sleep220409-21diskstats08:05:152
2736199216,8cyclictest0-21swapper/307:31:263
2736199216,12cyclictest11959-21chrt07:45:163
2736199215,3cyclictest0-21swapper/308:10:253
2736199213,10cyclictest3307-21proc_pri11:05:243
2736199212,17cyclictest22772-21grep10:35:273
2736199211,9cyclictest0-21swapper/307:36:473
2736199211,8cyclictest32403-21latency_hist07:20:023
2736199211,7cyclictest0-21swapper/308:16:533
2736199211,7cyclictest0-21swapper/308:16:533
2736199211,11cyclictest10059-21cat12:35:193
2736199210,14cyclictest0-21swapper/309:39:503
2736199210,13cyclictest28955-21ntp_states10:50:233
2735499211,4cyclictest16053-21diskmemload09:35:352
27354992114,4cyclictest24876-21if_err_enp1s011:55:162
27354992111,2cyclictest4738-21awk12:25:012
2735099219,3cyclictest23215-21switchtime11:50:231
2735099218,11cyclictest30420-21kworker/u8:010:57:141
2735099218,11cyclictest12202-21kworker/u8:010:25:141
2735099217,3cyclictest0-21swapper/110:18:131
27350992113,6cyclictest0-21swapper/110:38:391
2735099211,10cyclictest0-21swapper/110:20:311
2735099210,19cyclictest0-21swapper/107:50:211
2734499213,10cyclictest0-21swapper/009:40:160
2734499211,8cyclictest0-21swapper/011:00:190
2734499211,6cyclictest0-21swapper/012:12:530
256362210,4sleep011-21rcuc/009:30:170
256362210,4sleep011-21rcuc/009:30:170
177682210,4sleep22735499cyclictest11:40:132
163662211,16sleep216618-21df09:10:132
2736199208,2cyclictest0-21swapper/308:41:123
2736199207,5cyclictest0-21swapper/310:17:333
2736199204,7cyclictest31347-21smart_sda10:55:233
2736199203,14cyclictest13119-21/usr/sbin/munin11:30:173
2736199200,13cyclictest86950irq/126-enp1s0-10:32:503
2735499201,6cyclictest0-21swapper/211:00:142
27354992014,4cyclictest15346-21tail10:20:162
2735499201,11cyclictest6197-21memory08:45:192
2735099208,3cyclictest8499-21ls11:20:021
2735099208,3cyclictest26966-21kworker/u8:007:28:131
2735099207,3cyclictest23547-21hddtemp_smartct09:25:151
2735099205,6cyclictest0-21swapper/109:49:191
2735099202,7cyclictest21-21rcuc/107:30:191
27350992016,3cyclictest0-21swapper/109:14:081
27350992016,3cyclictest0-21swapper/108:24:211
27350992012,4cyclictest2736-21grep11:05:181
2735099201,18cyclictest0-21swapper/108:27:271
2735099201,17cyclictest0-21swapper/108:55:301
2735099201,17cyclictest0-21swapper/107:40:161
2735099201,16cyclictest16053-21diskmemload12:20:101
2735099201,11cyclictest29744-21dnf10:53:431
2735099201,10cyclictest25612-21cat10:45:021
2735099201,10cyclictest1908-21latency_hist11:05:021
2734499206,3cyclictest3696-21grep11:05:270
2734499202,10cyclictest457-21/usr/sbin/munin12:15:160
27344992014,3cyclictest9199-21cpuspeed_turbos07:40:120
2734499201,16cyclictest6644-21munin-run07:35:020
27344992011,2cyclictest87150irq/128-enp1s0-10:21:440
276472190,4sleep230-21rcuc/207:10:132
2736199194,7cyclictest751-21rngd07:25:103
2736199193,7cyclictest751-21rngd08:04:333
2736199191,6cyclictest0-21swapper/309:10:133
2736199191,13cyclictest86950irq/126-enp1s0-11:15:183
2736199190,6cyclictest26944-21expr08:20:143
2735499196,4cyclictest21835-21cat11:50:022
2735499191,8cyclictest12278-21munin-run10:15:012
2735499191,8cyclictest0-21swapper/211:23:412
2735499191,5cyclictest0-21swapper/212:11:272
2735499191,5cyclictest0-21swapper/208:35:252
27354991914,3cyclictest19154-21egrep10:30:112
27354991911,5cyclictest19324-21grep08:00:272
2735099197,9cyclictest10078-21kworker/u8:207:20:131
2735099195,5cyclictest0-21swapper/112:01:181
2735099193,8cyclictest0-21swapper/110:05:141
2735099192,8cyclictest904-21sssd_nss08:40:151
2735099191,10cyclictest22925-21hddtemp_smartct08:10:161
27350991910,3cyclictest0-21swapper/108:05:281
2734499193,3cyclictest87150irq/128-enp1s0-09:10:180
2734499192,5cyclictest2087-21perl09:50:160
2734499191,5cyclictest0-21swapper/010:08:350
27344991915,2cyclictest0-21swapper/010:18:150
2734499191,15cyclictest26830-21cstates08:20:130
2734499191,10cyclictest4953-21/usr/sbin/munin12:25:230
2734499191,10cyclictest27547-21/usr/sbin/munin07:10:170
2734499191,10cyclictest0-21swapper/011:11:340
26182190,3sleep00-21swapper/008:37:030
129442190,3sleep10-21swapper/107:46:331
2736199184,6cyclictest41-21ksoftirqd/308:56:083
2735499184,4cyclictest7263-21cpuspeed_turbos12:30:132
2735499181,7cyclictest0-21swapper/212:09:282
2735499181,7cyclictest0-21swapper/209:34:162
2735499181,7cyclictest0-21swapper/209:34:162
2735499181,6cyclictest0-21swapper/211:30:152
27354991812,3cyclictest15313-21/usr/sbin/munin11:35:202
27354991811,4cyclictest4701-21/usr/sbin/munin11:10:162
2735499181,14cyclictest10006-21cut12:35:182
2735499181,11cyclictest0-21swapper/210:29:452
2735099181,16cyclictest0-21swapper/109:01:591
2735099181,16cyclictest0-21swapper/108:32:531
27350991811,4cyclictest0-21swapper/109:05:201
2734499187,5cyclictest87150irq/128-enp1s0-08:10:190
2734499186,2cyclictest11-21rcuc/012:20:130
2734499184,7cyclictest1333-21cat09:50:020
2734499182,5cyclictest15659-21diskstats11:35:130
2734499181,9cyclictest14086-21/usr/sbin/munin09:05:190
2734499181,4cyclictest86950irq/126-enp1s0-10:33:500
2734499181,4cyclictest19819-21cpuspeed_turbos11:45:100
27344991812,4cyclictest10078-21kworker/u8:207:52:440
27344991812,3cyclictest10321-21cpu10:10:140
27344991811,4cyclictest20786-21grep09:20:140
2734499181,12cyclictest0-21swapper/007:55:190
2734499181,10cyclictest0-21swapper/009:28:470
231632180,3sleep20-21swapper/210:38:572
131082180,3sleep00-21swapper/011:30:040
87772171,2sleep00-21swapper/012:30:490
63512171,2sleep20-21swapper/207:31:282
299822170,4sleep230-21rcuc/207:15:032
273882170,2sleep00-21swapper/010:46:420
273882170,2sleep00-21swapper/010:46:420
2735499176,3cyclictest87250irq/129-enp1s0-10:40:242
2735499175,3cyclictest12050irq/123-ahci[0008:20:012
2735499175,3cyclictest12050irq/123-ahci[0008:20:012
2735499175,3cyclictest0-21swapper/210:49:552
2735499175,3cyclictest0-21swapper/210:49:552
2735499172,5cyclictest0-21swapper/209:53:202
2735499171,9cyclictest0-21swapper/211:26:412
2735499171,9cyclictest0-21swapper/210:05:242
2735499171,6cyclictest0-21swapper/209:43:222
2735499171,6cyclictest0-21swapper/209:20:062
2735499171,6cyclictest0-21swapper/208:55:252
2735499171,6cyclictest0-21swapper/207:20:132
2735499171,3cyclictest0-21swapper/209:26:182
2735499171,15cyclictest0-21swapper/207:26:222
27354991711,4cyclictest28225-21sh10:50:152
2735499171,14cyclictest0-21swapper/208:30:072
27354991711,3cyclictest4103-21idleruntime-cro11:10:022
27354991711,3cyclictest28619-21crond08:25:012
2734499174,4cyclictest7859-21strings07:35:230
2734499172,3cyclictest87150irq/128-enp1s0-11:29:200
2734499171,8cyclictest24993-21grep10:40:250
2734499171,3cyclictest0-21swapper/010:55:220
2734499171,3cyclictest0-21swapper/010:25:350
27344991712,3cyclictest19839-21munin-run08:05:010
27344991712,3cyclictest18477-21/usr/sbin/munin09:15:230
27344991712,3cyclictest12211-21df09:00:140
2734499171,14cyclictest9313-21grep11:20:150
2734499170,5cyclictest0-21swapper/008:45:180
130502170,2sleep20-21swapper/207:47:532
78282161,2sleep20-21swapper/210:04:302
306402161,2sleep20-21swapper/208:27:512
2735499162,5cyclictest801-21snmpd09:02:332
2735499162,5cyclictest4769-21ntp_states09:55:212
2735499162,11cyclictest8076-21grep11:15:262
2735499161,7cyclictest0-21swapper/212:02:542
2735499161,7cyclictest0-21swapper/210:15:572
2735499161,6cyclictest31662-21/usr/sbin/munin09:45:232
2735499161,5cyclictest0-21swapper/212:18:442
27354991611,3cyclictest5248-21expr12:25:152
27354991611,3cyclictest30083-21/usr/sbin/munin10:55:152
27354991611,3cyclictest10452-21cat07:40:232
27354991610,4cyclictest0-21swapper/207:50:192
27354991610,3cyclictest6985-21tr07:35:132
27354991610,3cyclictest5052-21latency_hist08:45:022
2735499160,14cyclictest0-21swapper/209:15:232
2734499164,5cyclictest0-21swapper/012:35:060
2734499162,12cyclictest3087-21/usr/sbin/munin08:40:220
2734499161,2cyclictest0-21swapper/008:05:140
27344991612,3cyclictest0-21swapper/008:15:220
27344991612,3cyclictest0-21swapper/008:15:210
2734499161,12cyclictest2475-21/usr/sbin/munin07:25:240
27344991610,4cyclictest0-21swapper/008:55:230
2735499154,3cyclictest87250irq/129-enp1s0-09:05:152
2734499153,9cyclictest7580-21cpuspeed_turbos08:50:110
2734499152,2cyclictest86950irq/126-enp1s0-08:25:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional