You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-20 - 01:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sun Jul 19, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
76082248163,21sleep20-21swapper/207:09:002
76732203168,23sleep30-21swapper/307:09:493
75962201163,26sleep00-21swapper/007:08:500
73842201165,24sleep10-21swapper/107:06:051
1853221580,5sleep3779199cyclictest07:30:233
2609021410,4sleep2778799cyclictest10:15:162
77092540,2sleep30-21swapper/310:45:273
185702540,4sleep081ktimersoftd/007:30:240
778099489,20cyclictest6536-21cat12:00:011
164352430,2sleep20-21swapper/209:55:012
778099421,30cyclictest0-21swapper/111:05:171
778099421,29cyclictest28844-21latency_hist09:10:011
778099391,26cyclictest12485-21latency_hist11:00:011
778099381,27cyclictest19696-21latency_hist12:30:001
778099380,30cyclictest0-21swapper/109:40:121
778099379,20cyclictest6275-21sed08:15:231
778099377,18cyclictest9171-21expr12:05:131
778099361,23cyclictest12450-21diskstats09:45:151
778099351,3cyclictest25314-21smart_sda07:45:251
778099351,3cyclictest2476-21awk08:10:011
778099351,3cyclictest23119-21smart_sda07:40:251
778099351,3cyclictest14700-21awk11:05:011
778099351,31cyclictest1-21systemd11:30:011
778099351,27cyclictest21417-21cat11:20:011
778099342,3cyclictest23095-21/usr/sbin/munin10:10:251
778099341,6cyclictest20552-21kworker/u8:110:54:011
778099341,3cyclictest22250-21perf08:55:011
778099339,16cyclictest24292-21cat12:40:021
778099338,21cyclictest16781-21load08:40:191
778099333,3cyclictest87650irq/129-enp1s0-10:45:241
778099331,6cyclictest28503-21fschecks_count11:35:141
778099331,5cyclictest13063-21/usr/sbin/munin07:20:141
778099331,3cyclictest5917-21tune2fs09:30:151
778099331,23cyclictest0-21swapper/112:30:271
778099331,20cyclictest0-21swapper/111:20:231
778099324,2cyclictest2112-21sh11:50:011
778099322,26cyclictest2987-21smart_sda10:35:241
778099322,26cyclictest27265-21munin-run10:20:011
778099321,4cyclictest808-21snmpd08:12:231
778099321,3cyclictest27906-21fschecks_count10:20:151
778099321,3cyclictest2374-21smart_sda09:20:241
778099321,28cyclictest3956-21gltestperf09:25:161
7780993210,19cyclictest20680-21perf10:05:011
7787993128,2cyclictest0-21swapper/211:47:312
778099314,24cyclictest17807-21/usr/sbin/munin07:30:161
778099311,7cyclictest808-21snmpd08:58:141
778099311,5cyclictest32495-21sshd08:03:311
778099311,3cyclictest27774-21unixbench_multi07:50:261
778099311,3cyclictest18614-21fschecks_time08:45:151
778099311,3cyclictest17389-21cat11:10:121
778099311,3cyclictest17389-21cat11:10:121
778099311,3cyclictest15843-21cat08:40:011
7780993113,15cyclictest18047-21grep09:55:261
778099311,27cyclictest9775-21grep09:40:011
326142310,4sleep3779199cyclictest10:30:183
778099309,18cyclictest15263-21/usr/sbin/munin07:25:251
778099309,17cyclictest7557-21kernelversion12:00:181
778099303,2cyclictest23-21ksoftirqd/111:40:221
778099301,6cyclictest0-21swapper/109:50:171
778099301,3cyclictest790-21in:imjournal09:20:021
778099301,3cyclictest26638-21chrt09:05:001
778099301,26cyclictest9055-21latency_hist08:25:011
778099301,26cyclictest13457-21perf08:35:011
7780993011,16cyclictest324-21cat10:30:191
777499307,11cyclictest15662-21basename08:40:010
777499301,14cyclictest14699-21munin-run11:05:010
778099297,19cyclictest30755-21ntpq10:25:211
778099292,3cyclictest0-21swapper/110:05:181
778099291,4cyclictest14366-21/usr/sbin/munin12:15:211
778099291,3cyclictest5903-21awk10:45:011
778099291,3cyclictest29577-21/usr/sbin/munin07:55:231
778099291,3cyclictest0-21swapper/109:10:171
778099291,2cyclictest0-21swapper/111:30:151
778099290,3cyclictest12755-21taskset12:10:331
779199285,10cyclictest12104-21/usr/sbin/munin09:45:243
778099281,4cyclictest15884-21/usr/sbin/munin12:20:141
778099281,4cyclictest15884-21/usr/sbin/munin12:20:141
778099281,4cyclictest12839-21turbostat07:20:011
7780992810,15cyclictest8653-21smartctl07:10:171
778799273,14cyclictest11514-40dnf09:40:332
778099271,4cyclictest9992-21/usr/sbin/munin08:25:181
778099271,3cyclictest3013-21hddtemp_smartct11:50:171
777499271,12cyclictest16902-21awk11:10:010
779199264,11cyclictest0-21swapper/312:00:193
778799260,11cyclictest0-21swapper/212:10:402
778099261,3cyclictest19673-21/usr/sbin/munin07:35:121
778099261,3cyclictest19673-21/usr/sbin/munin07:35:121
777499269,14cyclictest9007-21sshd08:24:310
777499263,11cyclictest27489-21/usr/sbin/munin10:20:160
779199251,4cyclictest25686-21turbostat11:30:013
90132240,2sleep20-21swapper/209:35:272
779199241,11cyclictest17732-21/usr/sbin/munin12:25:263
779199241,10cyclictest1382-21smart_sda11:45:243
778799241,12cyclictest0-21swapper/212:33:472
777499245,9cyclictest14675-21sshd07:20:310
777499241,4cyclictest28847-21diskmemload10:11:030
777499241,16cyclictest0-21swapper/012:11:060
777499241,11cyclictest6299-21sshd11:57:310
777499241,11cyclictest25292-21/usr/sbin/munin10:15:250
779199237,6cyclictest0-21swapper/311:55:133
777499231,4cyclictest0-21swapper/011:25:350
777499231,3cyclictest28847-21diskmemload10:07:160
777499231,3cyclictest21512-21/usr/sbin/munin11:20:260
779199227,3cyclictest26895-21awk10:15:253
778799225,4cyclictest14062-21crond09:50:012
777499229,3cyclictest32551-21switchtime09:15:250
777499227,4cyclictest20994-21ntp_states12:30:220
777499227,12cyclictest24097-21munin-run12:40:010
777499224,10cyclictest30301-21/usr/sbin/munin11:40:120
777499221,12cyclictest0-21swapper/011:30:250
779199219,3cyclictest30301-21/usr/sbin/munin11:40:163
779199213,9cyclictest0-21swapper/308:17:313
7791992115,3cyclictest4822-21tail09:25:253
779199211,11cyclictest839-21lldpd10:52:103
779199211,11cyclictest15874-21/usr/sbin/munin08:40:233
779199211,10cyclictest29581-21unixbench-2d11:35:253
779199211,10cyclictest17090-21grep12:20:253
779199211,10cyclictest17090-21grep12:20:253
778799216,5cyclictest3827-21grep11:50:272
778799214,9cyclictest0-21swapper/210:29:042
778799210,5cyclictest29843-21kworker/u8:209:55:372
777499215,4cyclictest18185-21ntpq11:10:200
777499215,4cyclictest18185-21ntpq11:10:200
777499215,3cyclictest10280-21head08:25:210
777499214,4cyclictest5435-21tail10:40:260
777499211,5cyclictest0-21swapper/010:33:120
777499211,10cyclictest24668-21/usr/sbin/munin09:00:120
779199206,4cyclictest18461-21sh10:00:013
779199202,6cyclictest0-21swapper/309:40:333
779199201,4cyclictest14894-21sed12:15:263
779199201,3cyclictest20646-21grep07:35:193
779199201,3cyclictest20646-21grep07:35:193
779199201,11cyclictest839-21lldpd08:23:373
779199201,11cyclictest4528-21grep10:40:163
779199201,10cyclictest31034-21timerandwakeup10:25:253
778799205,3cyclictest87650irq/129-enp1s0-10:50:262
778799205,3cyclictest87650irq/129-enp1s0-08:30:222
778799203,9cyclictest28847-21diskmemload11:59:552
778799202,9cyclictest29158-21ntp_offset11:35:202
778799202,2cyclictest87650irq/129-enp1s0-11:14:082
778799202,2cyclictest87650irq/129-enp1s0-11:14:082
778799202,10cyclictest23825-21turbostat07:45:012
778799201,7cyclictest0-21swapper/210:58:422
778799201,5cyclictest0-21swapper/211:23:192
778799201,12cyclictest0-21swapper/211:40:392
778799201,10cyclictest7648-21memory12:00:192
778799201,10cyclictest19441-21awk07:35:012
778799201,10cyclictest15065-21cpuspeed_turbos11:05:132
777499206,3cyclictest11-21rcuc/010:55:260
777499205,3cyclictest29096-21head11:35:190
777499201,10cyclictest8349-21sshd12:00:310
777499201,10cyclictest18512-21/usr/sbin/munin12:25:180
777499201,10cyclictest17299-21sshd12:22:310
777499201,10cyclictest17299-21sshd12:22:310
777499201,10cyclictest11482-21/usr/sbin/munin08:30:270
777499201,10cyclictest0-21swapper/009:40:320
779199195,3cyclictest365-21sed09:15:273
779199194,6cyclictest39-21rcuc/311:00:243
779199194,12cyclictest39-21rcuc/307:20:203
779199192,14cyclictest22120-21grep11:20:163
7791991912,4cyclictest22047-21awk10:05:213
779199191,11cyclictest808-21snmpd12:30:523
779199191,10cyclictest0-21swapper/309:22:053
778799196,3cyclictest24507-21if_err_enp2s011:25:182
778799195,3cyclictest5730-21chrt08:15:172
778799194,3cyclictest87650irq/129-enp1s0-11:30:202
778799194,3cyclictest87650irq/129-enp1s0-10:05:182
778799192,3cyclictest87350irq/126-enp1s0-09:27:102
778799191,7cyclictest24099-21perf12:40:012
7787991912,6cyclictest32-21ksoftirqd/210:45:482
778799191,10cyclictest28847-21diskmemload10:37:232
778799191,10cyclictest0-21swapper/211:19:072
778799191,10cyclictest0-21swapper/208:35:182
777499197,3cyclictest0-21swapper/011:53:380
777499197,3cyclictest0-21swapper/009:22:450
777499195,3cyclictest25346-21switchtime07:45:260
777499195,2cyclictest26250-21/usr/sbin/munin07:50:190
777499194,3cyclictest12863-21if_enp2s009:45:180
777499194,3cyclictest0-21swapper/010:49:270
777499193,3cyclictest29441-21strings07:55:200
777499192,14cyclictest5024-21tr08:15:110
777499191,9cyclictest0-21swapper/010:39:520
777499191,6cyclictest0-21swapper/009:10:580
777499191,5cyclictest31247-21grep08:00:150
777499191,5cyclictest10154-21taskset10:53:220
7774991913,3cyclictest11402-21uniq07:15:160
777499191,10cyclictest28847-21diskmemload11:45:200
777499191,10cyclictest0-21swapper/009:26:290
777499190,4cyclictest6598-21open_inodes09:30:220
777499190,11cyclictest0-21swapper/009:54:560
139452191,2sleep20-21swapper/211:00:242
779199186,4cyclictest0-21swapper/309:05:243
779199184,5cyclictest755-21rngd08:26:163
779199183,6cyclictest39-21rcuc/309:10:153
779199183,11cyclictest3291-21tail08:10:153
779199182,8cyclictest773-21NetworkManager07:48:073
779199182,6cyclictest5845-21df_inode09:30:143
779199182,6cyclictest28225-21munin-run07:55:013
779199181,7cyclictest0-21swapper/311:06:533
779199181,6cyclictest4306-21sh11:55:013
779199181,6cyclictest0-21swapper/309:35:193
779199181,5cyclictest18174-21ntpq11:10:193
779199181,5cyclictest18174-21ntpq11:10:183
779199181,4cyclictest0-21swapper/312:09:293
779199181,2cyclictest14824-21pmu-power08:35:223
7791991812,3cyclictest808-21snmpd07:43:003
779199181,15cyclictest0-21swapper/308:33:413
779199181,10cyclictest0-21swapper/312:14:533
779199181,10cyclictest0-21swapper/311:19:453
779199180,11cyclictest0-21swapper/308:45:123
778799185,4cyclictest29712-21/usr/sbin/munin07:55:252
778799183,8cyclictest0-21swapper/212:06:152
778799183,4cyclictest994-21grep10:30:262
778799183,3cyclictest16941-21ntp_states08:40:222
778799182,4cyclictest31397-21tr09:15:132
778799181,9cyclictest827-21runrttasks08:05:072
778799181,9cyclictest17108-21unixbench_singl12:20:252
778799181,9cyclictest17108-21unixbench_singl12:20:252
778799181,9cyclictest0-21swapper/210:13:312
778799181,6cyclictest28847-21diskmemload09:10:132
778799181,6cyclictest24711-21grep07:45:172
778799181,5cyclictest5505-21/usr/sbin/munin09:30:172
778799181,4cyclictest11859-21strings07:15:212
778799181,11cyclictest0-21swapper/212:15:312
778799181,10cyclictest0-21swapper/210:02:492
777499189,3cyclictest0-21swapper/010:04:100
777499185,2cyclictest87550irq/128-enp1s0-12:05:220
777499181,8cyclictest87550irq/128-enp1s0-10:25:270
777499181,5cyclictest7975-21/usr/sbin/munin07:10:190
777499181,5cyclictest26879-21/usr/sbin/munin09:05:160
777499181,5cyclictest19762-21sshd08:46:310
777499181,5cyclictest19314-21/usr/sbin/munin11:15:150
777499181,5cyclictest17438-21unixbench_singl08:40:270
777499181,4cyclictest23982-21grep08:55:270
777499181,14cyclictest8058-21diskstats09:35:160
779199175,6cyclictest0-21swapper/307:55:213
779199175,4cyclictest24075-21cat10:10:193
779199175,3cyclictest20276-21cat08:50:113
779199174,4cyclictest909-21sssd_nss10:35:253
779199173,4cyclictest15127-21kworker/u8:212:35:173
779199172,12cyclictest18696-21/usr/sbin/munin10:00:133
779199171,9cyclictest0-21swapper/309:52:093
779199171,7cyclictest27040-21sed11:30:233
779199171,6cyclictest10870-21/usr/sbin/munin07:15:143
779199171,6cyclictest0-21swapper/310:20:203
779199171,14cyclictest0-21swapper/308:07:213
7791991711,3cyclictest9956-21sleep07:15:013
779199171,13cyclictest401ktimersoftd/308:00:123
779199170,9cyclictest0-21swapper/308:55:133
778799173,4cyclictest7238-21grep08:20:132
778799173,3cyclictest87350irq/126-enp1s0-07:10:262
778799173,3cyclictest30-21rcuc/207:35:192
778799173,3cyclictest30-21rcuc/207:35:192
778799173,3cyclictest12050irq/123-ahci[0007:55:012
778799173,2cyclictest87550irq/128-enp1s0-08:55:212
778799172,8cyclictest0-21swapper/209:05:032
778799172,4cyclictest0-21swapper/210:40:142
778799171,6cyclictest18203-21grep08:45:132
778799171,6cyclictest13413-21diskstats07:20:152
778799171,6cyclictest0-21swapper/208:25:142
778799171,5cyclictest21620-21cat08:50:232
778799171,5cyclictest0-21swapper/212:25:592
777499171,5cyclictest21854-21/usr/sbin/munin07:40:140
777499171,3cyclictest388-21/usr/sbin/munin08:05:240
777499171,10cyclictest0-21swapper/009:58:210
777499170,6cyclictest0-21swapper/012:19:460
779199165,3cyclictest0-21swapper/307:25:173
779199161,6cyclictest26644-21latency_hist09:05:013
779199161,3cyclictest10521-21/usr/sbin/munin10:55:123
778799164,3cyclictest24799-21cpuspeed_turbos09:00:122
778799162,3cyclictest16253-21needreboot07:25:192
778799161,5cyclictest0-21swapper/208:00:212
778799161,4cyclictest27489-21/usr/sbin/munin10:20:142
7787991611,3cyclictest3755-21ntp_states08:10:212
778799160,2cyclictest87550irq/128-enp1s0-09:20:262
777499166,1cyclictest0-21swapper/008:50:240
7774991610,3cyclictest3455-21if_err_enp1s008:10:170
213392161,2sleep00-21swapper/007:36:310
213392161,2sleep00-21swapper/007:36:310
777499159,3cyclictest15357-21cpuspeed_turbos07:25:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional