You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-22 - 14:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Mon Dec 22, 2025 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
121792233159,24sleep10-21swapper/107:06:291
123372224194,20sleep20-21swapper/207:08:312
104972220191,19sleep00-21swapper/007:05:100
104942200165,24sleep30-21swapper/307:05:103
36032570,2sleep00-21swapper/009:15:020
1255999350,15cyclictest0-21swapper/310:21:483
1255999331,25cyclictest0-21swapper/307:49:573
1255999331,24cyclictest0-21swapper/307:40:433
1255999330,25cyclictest0-21swapper/307:22:273
1255999321,21cyclictest0-21swapper/308:51:533
1255999321,21cyclictest0-21swapper/308:51:533
1255999321,14cyclictest0-21swapper/312:11:423
1255999321,14cyclictest0-21swapper/311:55:153
1255999321,12cyclictest0-21swapper/308:10:363
1255999311,6cyclictest0-21swapper/311:09:013
1255999311,3cyclictest0-21swapper/311:02:443
1255999311,14cyclictest0-21swapper/311:10:393
1255999311,13cyclictest0-21swapper/310:31:003
1255999310,14cyclictest0-21swapper/311:41:153
1255999301,14cyclictest0-21swapper/312:35:023
1255999301,13cyclictest0-21swapper/308:35:223
1255999301,12cyclictest0-21swapper/309:23:153
1255999300,13cyclictest0-21swapper/307:27:473
1254699301,25cyclictest16309-21munin-run12:15:011
1254699301,23cyclictest0-21swapper/111:40:111
1254699301,15cyclictest0-21swapper/109:56:321
1255999291,18cyclictest0-21swapper/308:20:083
1255999291,12cyclictest0-21swapper/310:28:123
1255999291,12cyclictest0-21swapper/307:31:073
1255999291,11cyclictest0-21swapper/309:52:033
1255999291,11cyclictest0-21swapper/307:59:243
1255999291,11cyclictest0-21swapper/307:18:253
1255999291,10cyclictest15413-21/usr/sbin/munin09:40:173
1255999290,12cyclictest0-21swapper/311:35:073
1255999290,12cyclictest0-21swapper/310:05:183
1255999290,12cyclictest0-21swapper/310:05:183
1254699291,19cyclictest18496-21latency_hist12:20:011
1255999281,13cyclictest0-21swapper/311:47:473
1255999281,13cyclictest0-21swapper/309:37:093
1255999281,12cyclictest0-21swapper/312:24:353
1255999281,12cyclictest0-21swapper/308:41:183
1255999281,12cyclictest0-21swapper/308:32:203
1255999281,12cyclictest0-21swapper/308:17:483
1255999281,11cyclictest0-21swapper/309:27:553
1255999280,12cyclictest0-21swapper/310:50:443
1255999280,11cyclictest0-21swapper/311:50:033
1254699283,12cyclictest27012-21munin-run08:55:011
1254699283,12cyclictest27012-21munin-run08:55:011
1254699281,15cyclictest0-21swapper/110:50:011
1254699280,26cyclictest0-21swapper/112:20:241
1254299289,16cyclictest21050-21awk09:55:000
1255999279,8cyclictest0-21swapper/309:30:213
1255999271,5cyclictest0-21swapper/310:56:183
1255999271,13cyclictest0-21swapper/310:44:203
1255999271,13cyclictest0-21swapper/310:19:483
1255999271,12cyclictest0-21swapper/311:29:393
1255999271,12cyclictest0-21swapper/310:35:043
1255999271,12cyclictest0-21swapper/307:38:193
1255999271,11cyclictest0-21swapper/312:16:083
1255999271,11cyclictest0-21swapper/310:10:443
1255999271,11cyclictest0-21swapper/309:06:593
1255999271,11cyclictest0-21swapper/308:29:403
1255999270,11cyclictest0-21swapper/310:00:553
1255999270,10cyclictest0-21swapper/312:34:363
1255999270,10cyclictest0-21swapper/312:34:363
1254699278,6cyclictest0-21swapper/110:38:141
1254699271,5cyclictest0-21swapper/111:07:511
1254699271,11cyclictest0-21swapper/110:01:531
167402260,4sleep230-21rcuc/207:15:242
1255999261,12cyclictest0-21swapper/309:15:453
1255999261,10cyclictest0-21swapper/311:33:553
1255999260,12cyclictest0-21swapper/311:24:593
1255999260,12cyclictest0-21swapper/311:24:593
1254699261,21cyclictest1715-21latency07:55:181
1254299269,15cyclictest29854-21perf10:15:010
234192251,2sleep00-21swapper/008:45:170
1255999251,2cyclictest0-21swapper/308:56:393
1255999251,13cyclictest0-21swapper/308:07:523
1255999251,12cyclictest0-21swapper/312:29:563
1255999251,12cyclictest0-21swapper/309:04:033
1255999251,10cyclictest0-21swapper/309:13:233
1255999250,12cyclictest0-21swapper/307:53:153
1255399256,3cyclictest12050irq/123-ahci[0010:40:012
12553992512,3cyclictest5045-21awk08:05:022
1254699253,10cyclictest12947-21wc12:00:271
1254699251,21cyclictest9762-21sh10:40:221
1254699251,21cyclictest23906-21date11:15:011
1254699251,21cyclictest21130-21cat09:55:021
1254699251,21cyclictest18255-21sort07:20:171
1254699251,21cyclictest1723-21expr09:10:141
1254699251,21cyclictest0-21swapper/108:05:151
1254699251,11cyclictest800-21sshd07:25:121
1254699251,11cyclictest26702-21fschecks_count11:20:151
1254699251,11cyclictest26702-21fschecks_count11:20:151
1254699251,10cyclictest26586-21memory10:05:191
1254699251,10cyclictest26586-21memory10:05:191
1254699251,10cyclictest2451-21/usr/sbin/munin10:25:161
12546992510,3cyclictest87150irq/128-enp1s0-11:00:101
12546992510,3cyclictest14057-21users09:35:241
1255999241,9cyclictest0-21swapper/312:06:033
1255999241,12cyclictest0-21swapper/310:47:083
1255999240,11cyclictest0-21swapper/312:00:333
1255999240,10cyclictest0-21swapper/308:45:153
1255399245,10cyclictest5681-21if_enp2s011:45:192
1254699249,3cyclictest0-21swapper/110:55:141
1254699244,9cyclictest1228-21diskmemload11:16:371
1254699242,8cyclictest1228-21diskmemload10:31:421
12546992415,7cyclictest1110-21/usr/sbin/munin10:20:251
1254699241,3cyclictest30461-21awk11:30:021
1254699241,21cyclictest0-21swapper/107:40:571
1254699241,20cyclictest10122-21irqstats11:55:191
1254699241,11cyclictest7928-21irqstats11:50:191
1254699241,11cyclictest16887-21cpu09:45:131
1254699241,10cyclictest19708-21smart_sda08:35:261
1254699241,10cyclictest0-21swapper/112:35:231
189822230,2sleep00-21swapper/009:50:010
1255999231,9cyclictest0-21swapper/311:17:473
1255999231,10cyclictest0-21swapper/307:13:163
1255999230,8cyclictest0-21swapper/309:55:213
1255399236,9cyclictest0-21swapper/212:29:162
1255399236,4cyclictest14450-21awk09:40:012
1254699239,2cyclictest0-21swapper/108:20:131
1254699238,4cyclictest25363-21kworker/u8:010:52:141
1254699238,3cyclictest25088-21latency_hist12:35:011
1254699238,3cyclictest25088-21latency_hist12:35:011
12546992316,4cyclictest14710-21munin-plugin-st07:15:001
1254699231,20cyclictest0-21swapper/112:09:011
1254699231,11cyclictest31651-21users10:15:261
1254699231,10cyclictest6866-40dnf11:49:581
1254699231,10cyclictest11043-21if_err_enp1s009:30:191
1254699230,9cyclictest0-21swapper/108:25:311
12542992316,4cyclictest32707-21ntp_states09:05:220
98382220,3sleep29840-21pmu-power10:40:232
159652222,3sleep115984-21diskstats07:15:161
1255999221,13cyclictest0-21swapper/308:01:403
1255399225,3cyclictest12050irq/123-ahci[0008:00:022
1255399223,4cyclictest7602-21grep08:10:142
1255399222,4cyclictest0-21swapper/210:17:032
1254699228,3cyclictest25482-21ntp_states07:35:191
1254699226,14cyclictest21282-21kworker/u8:207:30:031
1254699222,18cyclictest0-21swapper/109:03:251
1254699222,14cyclictest87150irq/128-enp1s0-09:17:291
1254699221,19cyclictest0-21swapper/109:09:051
1254699221,18cyclictest647-21cstates11:35:161
1254699221,11cyclictest0-21swapper/108:30:231
1254699221,10cyclictest0-21swapper/110:10:211
1254699221,10cyclictest0-21swapper/109:25:171
1254299222,12cyclictest0-21swapper/012:35:160
1255399218,2cyclictest22004-21awk07:30:012
1255399216,3cyclictest4481-21cstates10:30:132
1255399211,9cyclictest15814-21pmu-power09:40:242
1255399211,6cyclictest2036-21sed11:35:272
1255399211,6cyclictest1828-21diskstats09:10:142
1255399211,6cyclictest17938-21ntp_states09:45:232
1255399211,6cyclictest0-21swapper/212:20:232
1255399211,4cyclictest0-21swapper/209:57:162
1255399211,12cyclictest0-21swapper/211:04:122
1255399211,10cyclictest10636-21needreboot08:15:202
1254699211,5cyclictest0-21swapper/112:25:321
1254699211,2cyclictest0-21swapper/108:48:461
1254699211,18cyclictest0-21swapper/108:10:311
1254699211,18cyclictest0-21swapper/108:00:361
1254699211,10cyclictest31266-21gltestperf11:30:161
1254699211,10cyclictest15280-21grep09:40:171
1254699211,10cyclictest0-21swapper/109:20:141
1254699211,10cyclictest0-21swapper/107:50:061
1254299216,2cyclictest2447-21kworker/0:111:55:170
1254299215,14cyclictest20428-21perf08:40:010
1254299214,10cyclictest12530-21cyclictest07:16:320
1254299211,5cyclictest0-21swapper/010:21:340
12542992114,4cyclictest6532-21switchtime08:05:250
1254299211,17cyclictest16232-21head09:40:260
1255999200,10cyclictest0-21swapper/309:46:473
1255399202,5cyclictest25409-21strings07:35:182
1255399202,5cyclictest0-21swapper/211:40:122
1255399201,7cyclictest0-21swapper/212:31:232
1255399201,7cyclictest0-21swapper/212:31:232
1255399201,7cyclictest0-21swapper/211:51:382
1255399201,7cyclictest0-21swapper/211:31:222
1255399201,6cyclictest24667-21fschecks_time11:15:162
1255399201,6cyclictest1228-21diskmemload09:30:022
1255399201,6cyclictest0-21swapper/210:10:222
1255399201,5cyclictest6813-21sed08:05:272
1255399201,10cyclictest835-21runrttasks08:26:052
1255399201,10cyclictest801-21snmpd11:13:202
1255399201,10cyclictest30243-21irqstats09:00:182
1255399201,10cyclictest23216-21needreboot07:30:212
1255399201,10cyclictest16180-21sshd12:13:322
1254699201,18cyclictest0-21swapper/108:58:011
1254699201,18cyclictest0-21swapper/108:41:461
1254699200,19cyclictest0-21swapper/107:49:201
1254299209,3cyclictest30678-21/usr/sbin/munin11:30:170
1254299202,10cyclictest19700-21/usr/sbin/munin11:05:210
1254299202,10cyclictest17491-21/usr/sbin/munin11:00:130
1254299201,11cyclictest32262-21sshd07:50:230
1254299201,11cyclictest27234-21/usr/sbin/munin08:55:230
1254299201,11cyclictest21898-21/usr/sbin/munin11:10:240
1254299201,11cyclictest15295-21/usr/sbin/munin10:55:280
1254299201,10cyclictest413-21/usr/sbin/munin11:35:260
1254299201,10cyclictest17834-21/usr/sbin/munin07:20:200
136472191,3sleep20-21swapper/208:22:552
1255399193,3cyclictest28789-21sed08:55:262
1255399192,9cyclictest19067-21/usr/sbin/munin09:50:132
1255399191,9cyclictest86950irq/126-enp1s0-09:20:202
1255399191,9cyclictest4742-21ntp_states09:15:212
1255399191,7cyclictest0-21swapper/208:45:162
1255399191,6cyclictest26001-21expr10:05:142
1255399191,6cyclictest26001-21expr10:05:142
1255399191,6cyclictest0-21swapper/212:00:282
1255399191,6cyclictest0-21swapper/210:28:452
1255399191,4cyclictest26312-21head12:35:222
12553991914,3cyclictest9403-21cpuspeed_turbos11:55:142
12553991913,3cyclictest23462-21/usr/sbin/munin10:00:152
1255399191,15cyclictest27349-21ntp_states11:20:212
1255399191,15cyclictest27349-21ntp_states11:20:212
1254699191,2cyclictest0-21swapper/108:17:061
1254299197,2cyclictest21719-21kworker/0:007:37:110
1254299193,4cyclictest87150irq/128-enp1s0-09:25:170
1254299193,4cyclictest0-21swapper/011:48:170
1254299191,9cyclictest25351-21smart_sda11:15:250
1254299191,6cyclictest15146-21/usr/sbin/munin12:10:170
12542991914,3cyclictest6272-21munin-run10:35:010
1254299191,10cyclictest24526-21ntp_states10:00:210
1254299191,10cyclictest23125-21/usr/sbin/munin12:30:190
1254299191,10cyclictest23125-21/usr/sbin/munin12:30:190
1255399184,3cyclictest87250irq/129-enp1s0-10:45:202
1255399184,3cyclictest86950irq/126-enp1s0-10:20:232
1255399184,3cyclictest86950irq/126-enp1s0-07:50:182
1255399184,3cyclictest18834-21fschecks_count08:35:172
1255399184,3cyclictest12050irq/123-ahci[0008:35:002
1255399183,9cyclictest0-21swapper/207:20:352
1255399183,3cyclictest25153-21grep08:50:142
1255399183,3cyclictest25153-21grep08:50:142
1255399182,8cyclictest87250irq/129-enp1s0-07:47:422
1255399181,9cyclictest0-21swapper/211:06:042
1255399181,6cyclictest0-21swapper/207:40:142
1255399181,5cyclictest835-21runrttasks10:56:062
1255399181,5cyclictest1228-21diskmemload10:50:172
1255399181,5cyclictest0-21swapper/212:07:262
1255399181,4cyclictest14248-21sed07:10:272
12553991812,3cyclictest835-21runrttasks09:06:262
12553991812,3cyclictest10822-21fschecks_time09:30:162
1255399181,10cyclictest0-21swapper/212:17:322
1254299181,9cyclictest801-21snmpd12:03:290
1254299181,8cyclictest0-21swapper/009:30:020
1254299181,5cyclictest26701-21fschecks_count11:20:150
1254299181,5cyclictest26701-21fschecks_count11:20:150
1254299181,4cyclictest0-21swapper/011:50:130
1254299181,3cyclictest0-21swapper/009:55:150
1254299181,2cyclictest0-21swapper/010:15:550
12542991811,4cyclictest2866-21cpuspeed_turbos11:40:140
1254299181,11cyclictest0-21swapper/012:08:260
126882171,2sleep00-21swapper/010:47:540
1255399171,5cyclictest1228-21diskmemload11:25:072
1255399171,4cyclictest0-21swapper/208:40:222
1254299175,3cyclictest11980-21expr08:20:140
1254299173,8cyclictest0-21swapper/009:15:090
1254299171,8cyclictest5345-21kworker/u8:209:35:050
1254299171,8cyclictest22136-21latency_hist07:30:010
1254299171,7cyclictest0-21swapper/011:28:410
1254299171,6cyclictest3048-21/usr/sbin/munin08:00:180
1254299171,5cyclictest25688-21cpu10:05:120
1254299171,5cyclictest25688-21cpu10:05:120
1254299171,4cyclictest2105-21/usr/sbin/munin10:25:160
12542991712,3cyclictest8068-21unixbench_singl10:35:250
12542991712,3cyclictest17753-21rt-features12:15:240
1254299171,15cyclictest0-21swapper/008:17:410
1254299171,10cyclictest0-21swapper/010:52:270
223702161,2sleep00-21swapper/008:42:040
1254299168,3cyclictest0-21swapper/007:45:160
1254299165,3cyclictest87150irq/128-enp1s0-08:30:180
1254299161,9cyclictest0-21swapper/008:50:170
1254299161,9cyclictest0-21swapper/008:50:170
1254299161,7cyclictest801-21snmpd10:43:150
1254299161,6cyclictest0-21swapper/007:40:130
12542991612,3cyclictest0-21swapper/008:25:590
12542991612,3cyclictest0-21swapper/008:10:090
12542991612,2cyclictest13802-21ntp_offset07:10:220
12542991611,3cyclictest31459-21cut09:05:020
1254299161,10cyclictest0-21swapper/009:20:320
12542991610,4cyclictest20925-21/usr/sbin/munin12:25:140
12542991610,4cyclictest19641-21memory12:20:170
1254299160,15cyclictest0-21swapper/007:31:510
1254299153,6cyclictest87150irq/128-enp1s0-07:55:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional