You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 18:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Dec 27, 2025 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
103522241163,24sleep20-21swapper/207:09:242
82392203165,26sleep30-21swapper/307:05:023
102742200164,24sleep10-21swapper/107:08:221
84362199161,26sleep00-21swapper/007:05:030
2408521770,3sleep21049699cyclictest11:20:012
3204121400,4sleep01048699cyclictest11:35:240
3204121400,4sleep01048699cyclictest11:35:240
1293321400,3sleep31050299cyclictest07:15:023
302642630,4sleep121-21rcuc/109:05:161
186202520,4sleep232-21ksoftirqd/207:25:162
169842520,4sleep116988-21gltestperf08:35:141
203212460,6sleep011-21rcuc/009:55:230
1048699411,27cyclictest9012-21cstates10:45:130
1048699391,27cyclictest26422-21/usr/sbin/munin11:25:280
1048699391,25cyclictest11990-21/usr/sbin/munin08:25:140
1048699391,21cyclictest13423-21runrttasks12:07:320
1048699378,20cyclictest25177-21/usr/sbin/munin08:55:250
1048699374,25cyclictest5750irq/9-acpi08:40:250
1048699371,24cyclictest1554-21sort09:15:010
1048699371,24cyclictest1554-21sort09:15:010
1048699371,23cyclictest297062expr09:05:120
1048699366,20cyclictest22637-21/usr/sbin/munin12:30:130
1048699361,22cyclictest6630-21/usr/sbin/munin10:40:130
1048699361,21cyclictest31550-21diskmemload09:42:250
1048699360,26cyclictest0-21swapper/009:35:250
1048699351,31cyclictest5049-21/usr/sbin/munin11:50:230
1048699351,31cyclictest22030-21/usr/sbin/munin11:15:220
1048699351,24cyclictest23307-21runrttasks10:04:140
1048699351,21cyclictest28952-21df11:30:130
1048699350,32cyclictest1687-21/usr/sbin/munin11:40:200
1048699345,21cyclictest87150irq/128-enp1s0-11:55:170
1048699342,22cyclictest23609-21/usr/sbin/munin10:05:200
1048699342,19cyclictest29968-21latency_hist10:20:010
1048699341,30cyclictest31980-21sshd10:22:410
1048699341,30cyclictest31980-21sshd10:22:410
1048699341,29cyclictest6012-21/usr/sbin/munin09:25:230
1048699341,24cyclictest21909-21pmu-power08:45:220
1048699341,21cyclictest835-21runrttasks11:05:380
1048699341,19cyclictest12919-21cat07:15:020
1048699335,19cyclictest3204-21/usr/sbin/munin08:05:140
1048699332,21cyclictest11213-21cstates10:50:120
1048699331,22cyclictest4750-21taskset11:49:150
1048699331,22cyclictest15428-21/usr/sbin/munin11:00:190
1048699331,19cyclictest18249-21/usr/sbin/munin12:20:280
1048699331,14cyclictest0-21swapper/007:28:570
1048699330,17cyclictest0-21swapper/010:59:090
1048699322,26cyclictest21090-21smart_sda11:10:240
1048699321,28cyclictest13855-21/usr/sbin/munin12:10:180
1048699321,27cyclictest5671-21sshd09:23:410
1048699321,23cyclictest0-21swapper/009:47:430
1048699321,21cyclictest86950irq/126-enp1s0-12:15:270
1048699321,21cyclictest0-21swapper/012:35:130
1048699321,20cyclictest13582-21/usr/sbin/munin07:15:170
1048699321,19cyclictest17234-21expr09:50:130
1049199311,21cyclictest0-21swapper/110:51:191
1048699312,19cyclictest87250irq/129-enp1s0-09:30:230
1048699312,18cyclictest9785-21/usr/sbin/munin08:20:140
1048699311,27cyclictest22977-21/usr/sbin/munin08:50:140
1048699311,23cyclictest0-21swapper/008:02:570
1048699311,23cyclictest0-21swapper/008:02:570
1048699311,20cyclictest16142-21forks07:20:160
1048699311,19cyclictest28957-21/usr/sbin/munin07:50:180
1048699310,20cyclictest0-21swapper/010:10:180
1050299301,14cyclictest0-21swapper/308:23:333
1048699302,24cyclictest15450-21smart_sda08:30:250
1048699301,26cyclictest5398-21/usr/sbin/munin08:10:150
1048699301,26cyclictest22365-21/usr/sbin/munin07:35:210
1048699301,26cyclictest16384-21/usr/sbin/munin08:35:140
10486993010,17cyclictest24669-21grep07:40:120
1050299291,13cyclictest0-21swapper/311:32:303
1050299291,13cyclictest0-21swapper/308:48:133
1048699297,14cyclictest21231-21ntpq07:30:200
1048699294,16cyclictest0-21swapper/010:30:230
1048699292,23cyclictest904-21sssd_nss08:15:160
1048699292,23cyclictest27721-21/usr/sbin/munin09:00:140
1048699291,25cyclictest9444-21/usr/sbin/munin12:00:190
1048699291,25cyclictest31152-21/usr/sbin/munin07:55:220
1048699291,25cyclictest26764-21/usr/sbin/munin07:45:180
1048699291,25cyclictest24222-21/usr/sbin/munin11:20:200
1048699291,25cyclictest1969-21diskstats09:15:150
1048699291,19cyclictest0-21swapper/010:28:560
1050299281,12cyclictest0-21swapper/307:27:353
1049199281,17cyclictest24876-21gltestperf11:20:181
1048699281,24cyclictest5018-21/usr/sbin/munin10:35:160
1048699281,24cyclictest20443-21/usr/sbin/munin12:25:130
1050299273,9cyclictest0-21swapper/309:53:413
10502992710,2cyclictest0-21swapper/310:15:153
1050299270,19cyclictest0-21swapper/310:14:373
10496992724,2cyclictest0-21swapper/209:53:302
1050299261,10cyclictest16503-21chrt11:00:233
1050299261,10cyclictest0-21swapper/311:48:563
1049199261,17cyclictest6360-21diskstats09:25:131
220352250,5sleep11049199cyclictest12:25:251
1050299256,8cyclictest10331-21cat09:35:023
1050299256,3cyclictest0-21swapper/308:59:533
1050299251,10cyclictest0-21swapper/312:08:423
1050299251,10cyclictest0-21swapper/308:17:333
1049199251,21cyclictest26315-21grep12:35:261
1049199251,16cyclictest13656-21runrttasks08:26:141
1049199250,18cyclictest0-21swapper/111:52:311
294292240,5sleep129428-21fschecks_time07:50:151
1050299247,3cyclictest8430-40dnf10:42:563
1050299246,6cyclictest24187-21perl10:05:173
1050299245,16cyclictest87150irq/128-enp1s0-08:00:153
1050299245,16cyclictest87150irq/128-enp1s0-08:00:153
1050299244,9cyclictest21422-21needreboot12:25:183
1050299242,15cyclictest751-21rngd09:00:053
10502992413,5cyclictest87250irq/129-enp1s0-09:15:233
1050299241,17cyclictest0-21swapper/308:35:183
1049199241,3cyclictest8430-40dnf10:42:551
1049199241,3cyclictest10185-21perf09:35:011
146262230,4sleep31050299cyclictest10:55:273
1050299237,3cyclictest0-21swapper/311:14:573
1050299235,3cyclictest0-21swapper/308:51:333
1050299234,3cyclictest0-21swapper/307:59:033
1050299233,10cyclictest3856-21cpu09:20:133
1050299231,8cyclictest0-21swapper/309:46:293
1050299231,2cyclictest88750irq/134-enp2s0-08:30:273
1050299231,10cyclictest0-21swapper/307:49:173
1049699239,5cyclictest0-21swapper/212:30:252
1049699232,9cyclictest14388-40dnf09:42:432
10491992317,3cyclictest30879-21hddtemp_smartct10:20:201
10491992317,3cyclictest30879-21hddtemp_smartct10:20:201
1049199231,16cyclictest0-21swapper/111:28:431
1049199231,16cyclictest0-21swapper/107:38:031
1050299228,4cyclictest577-21systemd-journal07:20:153
1050299227,3cyclictest31550-21diskmemload11:35:263
1050299227,3cyclictest31550-21diskmemload11:35:263
1050299225,6cyclictest19725-21grep09:55:163
1050299225,3cyclictest7268-21sensors_temp09:25:233
1050299225,3cyclictest14580-21sh09:45:013
1050299224,9cyclictest20227-21awk12:25:013
1050299224,14cyclictest88650irq/133-enp2s0-09:08:013
1050299223,10cyclictest0-21swapper/312:35:013
1050299221,9cyclictest294312grep07:50:153
1050299221,11cyclictest0-21swapper/309:38:073
1049699221,6cyclictest0-21swapper/212:16:292
10491992218,3cyclictest0-21swapper/111:35:171
10491992218,3cyclictest0-21swapper/111:35:171
1049199221,3cyclictest19947-21munin-run07:30:011
1049199221,18cyclictest3289-21cut11:45:181
1049199221,18cyclictest29369-21timerandwakeup10:15:251
1049199221,18cyclictest12790-21sed12:05:231
1049199221,16cyclictest0-21swapper/109:13:301
1049199221,16cyclictest0-21swapper/109:13:301
1049199221,15cyclictest0-21swapper/108:22:361
1049199220,15cyclictest10086-21/usr/sbin/munin10:45:241
92042210,2sleep10-21swapper/108:15:371
145722211,2sleep114578-21latency_hist09:45:021
1050299219,3cyclictest751-21rngd12:00:163
1050299217,3cyclictest0-21swapper/307:31:213
1050299215,7cyclictest801-21snmpd08:27:533
1050299213,9cyclictest0-21swapper/312:35:263
1050299213,9cyclictest0-21swapper/312:15:223
1050299212,2cyclictest88750irq/134-enp2s0-07:36:533
10502992113,3cyclictest10552-21kworker/u8:011:07:123
1050299211,12cyclictest553-21/usr/sbin/munin11:40:193
10502992111,2cyclictest12050irq/123-ahci[0010:55:023
1050299211,10cyclictest0-21swapper/311:26:333
1049699217,3cyclictest0-21swapper/208:55:052
1049699211,6cyclictest0-21swapper/212:12:112
1049699211,6cyclictest0-21swapper/209:23:372
1049199217,6cyclictest2299-21systemd-journal11:41:001
1049199216,3cyclictest2132-21grep08:00:231
1049199216,3cyclictest2132-21grep08:00:231
10491992113,2cyclictest0-21swapper/110:55:021
1049199211,2cyclictest23158-21cstates08:50:131
10491992112,2cyclictest0-21swapper/107:10:121
1049199211,15cyclictest0-21swapper/108:56:291
1049199211,14cyclictest0-21swapper/111:10:091
1049199211,14cyclictest0-21swapper/107:33:011
1049199211,14cyclictest0-21swapper/107:23:151
1049199211,11cyclictest10358-21memory12:00:191
1049199210,6cyclictest0-21swapper/112:13:391
1049199210,3cyclictest4549-21expr10:35:131
1049199210,15cyclictest0-21swapper/109:04:481
1049199210,15cyclictest0-21swapper/108:33:141
1049199210,14cyclictest87250irq/129-enp1s0-09:58:091
177622200,4sleep21049699cyclictest08:35:232
1050299207,3cyclictest0-21swapper/309:14:413
1050299207,3cyclictest0-21swapper/309:14:413
1050299206,3cyclictest0-21swapper/307:41:453
1050299205,3cyclictest3050-21irqstats10:30:193
1050299204,3cyclictest0-21swapper/308:41:113
1050299203,9cyclictest0-21swapper/307:15:173
1050299203,14cyclictest7249-21/usr/sbin/munin11:55:173
1050299202,8cyclictest0-21swapper/310:45:163
1050299202,6cyclictest0-21swapper/312:10:383
10496992017,2cyclictest0-21swapper/211:22:472
1049699201,5cyclictest0-21swapper/209:11:272
1049699201,5cyclictest0-21swapper/209:11:272
1049199208,11cyclictest4760-21kworker/u8:211:57:121
10491992016,3cyclictest0-21swapper/109:17:011
10491992014,1cyclictest0-21swapper/108:40:151
1049199201,3cyclictest18634-21sshd09:50:451
10491992013,2cyclictest0-21swapper/110:08:031
1049199201,14cyclictest0-21swapper/111:06:531
1049199201,14cyclictest0-21swapper/111:01:171
1049199201,14cyclictest0-21swapper/110:12:071
1049199201,14cyclictest0-21swapper/108:46:411
1049199201,14cyclictest0-21swapper/108:13:031
1049199201,11cyclictest29527-21memory11:30:191
1049199201,11cyclictest15873-21ntp_states09:45:221
224882191,2sleep20-21swapper/208:46:482
1050299199,2cyclictest12050irq/123-ahci[0011:55:023
1050299192,9cyclictest0-21swapper/311:15:203
1050299191,8cyclictest0-21swapper/310:27:133
1050299191,6cyclictest0-21swapper/310:39:573
1050299191,3cyclictest21814-21cut10:00:153
10502991910,5cyclictest31550-21diskmemload10:20:273
10502991910,5cyclictest31550-21diskmemload10:20:273
1050299190,5cyclictest3809-1kworker/3:2H11:20:193
1049699197,3cyclictest32-21ksoftirqd/211:35:172
1049699197,3cyclictest32-21ksoftirqd/211:35:172
1049699191,8cyclictest9167-21df10:45:132
1049699191,7cyclictest17982-21diskstats11:05:132
1049699191,6cyclictest0-21swapper/211:32:262
1049699191,6cyclictest0-21swapper/207:10:182
10496991914,3cyclictest4379-21grep11:45:292
10496991913,3cyclictest11993-21df12:05:142
1049699191,12cyclictest0-21swapper/210:05:172
1049199198,10cyclictest19499-21sendmail_mailtr12:20:261
1049199194,13cyclictest23-21ksoftirqd/112:30:121
10491991915,3cyclictest0-21swapper/110:30:441
10491991912,2cyclictest0-21swapper/109:36:461
1049199191,17cyclictest0-21swapper/107:49:261
1049199191,14cyclictest0-21swapper/109:22:171
1049199191,14cyclictest0-21swapper/108:07:171
10491991911,3cyclictest0-21swapper/110:03:181
1049199191,10cyclictest18022-21idleruntime-cro12:20:011
1049199190,14cyclictest0-21swapper/107:43:401
38882181,2sleep20-21swapper/210:31:022
321102181,2sleep20-21swapper/210:24:242
321102181,2sleep20-21swapper/210:24:242
1050299181,3cyclictest88650irq/133-enp2s0-08:07:013
1050299181,3cyclictest0-21swapper/308:14:173
1049699188,3cyclictest0-21swapper/208:40:422
1049699186,4cyclictest86950irq/126-enp1s0-10:55:012
1049699184,3cyclictest87250irq/129-enp1s0-09:35:182
1049699181,7cyclictest31550-21diskmemload10:12:392
1049699181,7cyclictest0-21swapper/212:39:222
1049699181,5cyclictest9758-21cstates12:00:132
1049699181,5cyclictest6189-21head11:50:222
1049699181,5cyclictest16416-21strings11:00:212
1049699181,4cyclictest0-21swapper/212:28:202
1049699181,3cyclictest7693-21ntp_states10:40:212
10496991813,3cyclictest4599-21grep10:35:132
10496991811,4cyclictest7883-21tail11:55:162
1049699181,11cyclictest0-21swapper/209:57:142
1049199181,9cyclictest32399-21/usr/sbin/munin10:25:181
1049199181,2cyclictest0-21swapper/111:15:201
1049199180,13cyclictest0-21swapper/107:18:331
327572171,2sleep20-21swapper/210:25:162
1049699176,3cyclictest86950irq/126-enp1s0-08:20:192
1049699174,4cyclictest690-21tr11:40:122
1049699172,4cyclictest0-21swapper/211:10:302
1049699171,7cyclictest22311-21chrt10:00:192
1049699171,7cyclictest0-21swapper/210:19:522
1049699171,7cyclictest0-21swapper/207:55:192
1049699171,6cyclictest29364-21awk09:05:012
1049699171,6cyclictest0-21swapper/209:25:132
1049699171,5cyclictest986-1kworker/u9:109:17:132
1049699171,4cyclictest0-21swapper/209:30:132
1049699171,3cyclictest763-21avahi-daemon07:39:382
1049699171,15cyclictest0-21swapper/208:50:192
10496991711,4cyclictest28069-21switchtime07:45:262
1049699171,13cyclictest15939-21ls09:45:232
1049699171,10cyclictest0-21swapper/210:56:382
1049699171,10cyclictest0-21swapper/207:51:272
10491991711,3cyclictest32503-21sort07:55:241
1049699166,2cyclictest17671-1kworker/2:1H08:25:072
1049699164,3cyclictest2137-21pmu-power08:00:232
1049699164,3cyclictest2137-21pmu-power08:00:232
1049699164,3cyclictest15182-21needreboot08:30:212
1049699163,3cyclictest30533-21ls09:05:192
1049699161,6cyclictest5398-21/usr/sbin/munin08:10:272
1049699161,6cyclictest0-21swapper/212:21:582
1049699161,5cyclictest8584-21needreboot08:15:202
1049699161,2cyclictest0-21swapper/211:27:142
1049699161,14cyclictest0-21swapper/208:06:062
1049699161,14cyclictest0-21swapper/207:30:102
1049699161,14cyclictest0-21swapper/207:16:072
10496991611,3cyclictest24720-21cut07:40:122
10496991510,3cyclictest15838-21cpu07:20:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional