You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 12:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Fri Dec 05, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
151852233160,23sleep30-21swapper/319:07:163
150672231162,55sleep10-21swapper/119:05:461
153932202164,25sleep20-21swapper/219:09:572
151332202163,26sleep00-21swapper/019:06:370
2294621310,5sleep31549999cyclictest21:50:193
320452590,4sleep341-21ksoftirqd/322:10:263
198992560,3sleep01547899cyclictest20:30:230
106492530,3sleep339-21rcuc/320:10:143
125292490,3sleep012533-21latency22:40:200
125292490,3sleep012533-21latency22:40:200
1548699431,9cyclictest0-21swapper/123:14:431
1548699421,8cyclictest0-21swapper/123:34:471
1547899401,25cyclictest22452-21munin-run23:05:010
1548699381,8cyclictest0-21swapper/100:11:331
1548699361,7cyclictest0-21swapper/121:42:291
1548699351,6cyclictest0-21swapper/119:28:111
1548699341,13cyclictest0-21swapper/122:20:231
1548699341,13cyclictest0-21swapper/120:25:251
1548699331,9cyclictest0-21swapper/122:15:311
234152310,2sleep30-21swapper/320:40:013
234152310,2sleep30-21swapper/320:40:013
1548699311,11cyclictest0-21swapper/119:41:451
1548699301,8cyclictest0-21swapper/122:46:351
1548699301,8cyclictest0-21swapper/121:19:091
1548699301,8cyclictest0-21swapper/121:06:191
1548699301,8cyclictest0-21swapper/100:26:571
1548699301,12cyclictest0-21swapper/122:39:431
1548699301,12cyclictest0-21swapper/122:06:031
1548699301,11cyclictest0-21swapper/122:59:491
1548699301,10cyclictest1532-21awk21:00:311
1547899302,13cyclictest24956-21idleruntime-cro19:30:020
15478993020,3cyclictest14907-21kworker/0:321:35:290
15478993020,3cyclictest14907-21kworker/0:321:35:290
1548699291,8cyclictest0-21swapper/100:08:211
1548699291,7cyclictest0-21swapper/121:36:211
1548699291,7cyclictest0-21swapper/121:36:211
1548699291,5cyclictest0-21swapper/123:46:231
1548699291,5cyclictest0-21swapper/123:46:231
1548699291,18cyclictest4218-21diskmemload23:43:031
1548699291,14cyclictest0-21swapper/121:13:351
1548699291,11cyclictest0-21swapper/123:53:491
1548699291,10cyclictest0-21swapper/122:01:051
1548699290,9cyclictest0-21swapper/100:37:251
1548699290,13cyclictest0-21swapper/121:45:211
1548699281,8cyclictest0-21swapper/100:32:211
1548699281,12cyclictest0-21swapper/120:18:491
1548699280,8cyclictest0-21swapper/121:31:091
1547899281,12cyclictest25322-21awk00:25:010
1547899280,13cyclictest14572-21idleruntime-cro20:20:010
162032270,4sleep121-21rcuc/122:50:131
1548699271,15cyclictest0-21swapper/123:27:371
1548699271,14cyclictest0-21swapper/123:22:111
1548699271,10cyclictest0-21swapper/123:01:571
1548699271,10cyclictest0-21swapper/119:57:171
1548699270,6cyclictest0-21swapper/123:58:451
1548699268,5cyclictest16751-21ntp_states19:10:211
1548699263,4cyclictest12306-21kworker/u8:200:02:351
1548699262,13cyclictest974-21users19:45:271
1548699261,17cyclictest8286-21runrttasks20:05:131
1548699261,12cyclictest0-21swapper/122:31:151
1548699261,12cyclictest0-21swapper/122:26:311
1548699261,11cyclictest0-21swapper/119:20:131
1548699261,11cyclictest0-21swapper/100:21:231
1548699260,17cyclictest0-21swapper/121:20:191
1547899260,12cyclictest3298-21perf23:35:000
1549999251,12cyclictest10183-21munin-run20:10:013
1549299259,4cyclictest16114-21users00:00:302
1548699251,8cyclictest0-21swapper/121:53:431
1548699251,10cyclictest0-21swapper/122:40:051
1548699251,10cyclictest0-21swapper/122:40:051
1548699251,10cyclictest0-21swapper/119:39:091
1548699251,10cyclictest0-21swapper/119:34:291
1548699250,10cyclictest0-21swapper/121:56:271
1548699250,10cyclictest0-21swapper/120:55:251
1547899254,3cyclictest24734-21fschecks_time21:55:180
1549999249,2cyclictest0-21swapper/300:07:353
15499992413,4cyclictest16319-21latency21:35:183
15499992413,4cyclictest16319-21latency21:35:183
1549299241,6cyclictest0-21swapper/222:21:472
1549299241,3cyclictest0-21swapper/222:36:032
1548699241,9cyclictest0-21swapper/121:25:571
1548699241,3cyclictest0-21swapper/100:18:431
1548699241,18cyclictest0-21swapper/120:23:361
1548699241,11cyclictest0-21swapper/120:33:331
1548699241,10cyclictest0-21swapper/123:15:531
1548699241,10cyclictest0-21swapper/120:03:091
1548699240,9cyclictest0-21swapper/120:45:231
1548699240,11cyclictest0-21swapper/120:37:331
1548699240,11cyclictest0-21swapper/120:37:331
1547899241,6cyclictest0-21swapper/000:10:470
1549999231,1cyclictest0-21swapper/320:27:293
1549999231,16cyclictest0-21swapper/300:27:233
15499992311,2cyclictest0-21swapper/323:00:183
1549299235,5cyclictest768-21in:imjournal21:10:012
1549299231,8cyclictest11664-21grep21:25:182
1548699230,13cyclictest0-21swapper/122:11:371
11222230,5sleep230-21rcuc/222:15:142
269832220,2sleep00-21swapper/000:25:300
1549999229,3cyclictest25115-21cat21:55:233
1549299221,6cyclictest581-21systemd-journal20:55:282
1549299221,5cyclictest0-21swapper/223:49:082
1549299221,5cyclictest0-21swapper/223:49:082
1549299221,11cyclictest0-21swapper/221:20:162
1548699228,2cyclictest23984-21sed23:05:271
1548699221,2cyclictest1609-21/usr/sbin/munin19:50:181
1548699221,18cyclictest11545-21sed20:10:241
1548699221,13cyclictest0-21swapper/120:50:091
1548699220,16cyclictest0-21swapper/119:17:231
1547899221,8cyclictest2730650kworker/u8:000:38:160
1547899221,6cyclictest0-21swapper/021:50:240
1547899221,6cyclictest0-21swapper/021:49:280
15478992211,3cyclictest836-21runrttasks00:08:340
1547899220,5cyclictest0-21swapper/023:28:120
1549999218,3cyclictest7667-21grep22:30:193
1549999217,3cyclictest28981-21find00:30:303
15499992114,4cyclictest13764-21fschecks_time21:30:213
15499992112,4cyclictest0-21swapper/322:37:493
1549299216,4cyclictest23287-21head23:05:192
1549299213,4cyclictest8337-21ntpq22:30:262
1549299211,9cyclictest0-21swapper/219:30:152
1549299211,7cyclictest3806-21/usr/sbin/munin19:55:202
1549299211,6cyclictest28985-21df22:05:152
1549299211,6cyclictest0-21swapper/200:12:262
1549299211,6cyclictest0-21swapper/200:08:542
1549299211,2cyclictest86850irq/129-enp1s0-22:49:402
1548699211,8cyclictest0-21swapper/123:35:071
1548699211,11cyclictest24284-21hddtemp_smartct20:40:181
1547899218,6cyclictest0-21swapper/020:10:070
1547899218,5cyclictest2016-21crond21:05:010
1547899211,4cyclictest7-21ksoftirqd/021:10:010
15478992110,3cyclictest11348-21kworker/0:022:50:170
1549999208,3cyclictest0-21swapper/322:25:233
1549999208,2cyclictest39-21rcuc/321:45:243
1549999204,9cyclictest2598-21sshd22:19:293
1549999202,6cyclictest15667-21pmu-power00:00:253
1549999202,2cyclictest88050irq/133-enp2s0-22:03:523
1549999201,2cyclictest31080-21unixbench_multi19:40:253
15499992012,2cyclictest1603-21kworker/3:220:40:293
1549999201,12cyclictest0-21swapper/322:07:053
1549999201,11cyclictest842-21lldpd00:10:143
1549999201,10cyclictest6885-21proc_pri23:40:233
1549999201,10cyclictest13375-21ntp_states23:55:243
1549999201,10cyclictest12282-21tail22:40:193
1549999201,10cyclictest12282-21tail22:40:183
1549999201,10cyclictest0-21swapper/323:14:013
15499992010,3cyclictest15961-21pmu-power20:20:253
1549299201,6cyclictest13246-21/usr/sbin/munin21:30:222
1549299201,6cyclictest0-21swapper/223:35:542
1549299201,5cyclictest0-21swapper/221:12:422
1549299201,4cyclictest31571-21idleruntime22:10:192
1549299201,4cyclictest0-21swapper/223:34:492
15492992014,3cyclictest836-21runrttasks00:35:272
1549299201,2cyclictest86850irq/129-enp1s0-21:58:402
1547899208,4cyclictest0-21swapper/000:00:160
1547899208,3cyclictest4218-21diskmemload22:05:010
1547899206,5cyclictest0-21swapper/022:28:160
1547899205,5cyclictest7685-21kworker/0:120:04:100
1547899202,8cyclictest7-21ksoftirqd/021:33:050
1547899202,5cyclictest22552-1kworker/u9:122:24:030
1547899202,10cyclictest6646-21/usr/sbin/munin21:15:260
1547899202,10cyclictest21137-21/usr/sbin/munin00:15:240
1547899201,5cyclictest20146-21kworker/u8:223:11:450
1547899201,4cyclictest0-21swapper/023:09:070
1547899201,11cyclictest27082-21/usr/sbin/munin23:15:140
1547899201,11cyclictest0-21swapper/021:27:100
1547899201,10cyclictest29497-21sshd00:32:290
1547899201,10cyclictest0-21swapper/021:41:350
1549999199,3cyclictest608-21users23:25:263
1549999199,2cyclictest2598-21needreboot19:50:193
1549999198,4cyclictest768-21in:imjournal20:45:283
1549999195,7cyclictest4283-21timerandwakeup22:20:273
1549999191,9cyclictest17650-21latency_hist19:15:013
1549999191,6cyclictest3888-21grep19:55:143
1549999191,6cyclictest18614-21df22:55:153
1549999191,4cyclictest0-21swapper/321:14:163
15499991913,3cyclictest32684-21memory19:45:203
1549999191,2cyclictest0-21swapper/323:05:153
15499991912,2cyclictest0-21swapper/321:05:193
1549999191,15cyclictest30543-21grep00:35:203
1549999191,10cyclictest7911-21/usr/sbin/munin23:45:193
1549999191,10cyclictest7911-21/usr/sbin/munin23:45:193
1549999191,10cyclictest0-21swapper/321:15:083
1549299197,3cyclictest234132cat20:40:012
1549299197,3cyclictest234132cat20:40:012
1549299193,2cyclictest86850irq/129-enp1s0-23:28:162
1549299192,7cyclictest19848-21/usr/sbin/munin21:45:152
1549299191,9cyclictest0-21swapper/222:41:082
1549299191,9cyclictest0-21swapper/222:41:072
1549299191,7cyclictest6038-21cstates23:40:142
1549299191,6cyclictest7088-21tune2fs21:15:162
1549299191,6cyclictest1352-21latency_hist19:50:012
1549299191,5cyclictest27373-21/usr/sbin/munin19:35:122
1549299191,5cyclictest22320-21sshd23:03:282
1549299191,5cyclictest17702-21cat19:15:022
1549299191,5cyclictest0-21swapper/221:36:102
1549299191,5cyclictest0-21swapper/221:36:102
1549299191,5cyclictest0-21swapper/221:01:542
1549299191,4cyclictest0-21swapper/200:20:142
1549299191,3cyclictest0-21swapper/220:07:222
1549299191,2cyclictest86850irq/129-enp1s0-22:26:282
1549299190,2cyclictest86850irq/129-enp1s0-23:11:472
1547899198,3cyclictest3511-21/usr/sbin/munin23:35:190
1547899197,4cyclictest86550irq/126-enp1s0-20:55:140
1547899195,5cyclictest14634-21snmpd19:40:200
1547899195,3cyclictest7685-21kworker/0:120:45:160
1547899194,3cyclictest14634-21snmpd23:47:030
1547899194,3cyclictest14634-21snmpd23:47:030
1547899193,14cyclictest22757-21awk19:25:000
1547899191,9cyclictest14634-21snmpd23:50:460
1547899191,4cyclictest0-21swapper/022:16:080
1547899191,10cyclictest4218-21diskmemload21:13:500
1547899191,10cyclictest18576-21/usr/sbin/munin19:15:170
1547899191,10cyclictest12310-21/usr/sbin/munin23:55:280
1547899191,10cyclictest0-21swapper/023:40:080
1547899191,10cyclictest0-21swapper/023:22:580
1547899191,10cyclictest0-21swapper/022:37:490
1547899191,10cyclictest0-21swapper/022:30:170
1547899191,10cyclictest0-21swapper/022:06:460
1547899191,10cyclictest0-21swapper/021:21:590
1549999187,2cyclictest0-21swapper/300:20:243
1549999184,9cyclictest0-21swapper/322:50:203
1549999182,8cyclictest9064-21cstates21:20:143
1549999182,3cyclictest0-21swapper/323:20:183
1549999181,9cyclictest15337-21unixbench-2d22:45:283
1549999181,7cyclictest0-21swapper/323:15:473
1549999181,7cyclictest0-21swapper/321:27:323
15499991813,3cyclictest19461-21expr20:30:193
15499991812,3cyclictest12931-21cat20:15:133
1549999181,16cyclictest0-21swapper/320:01:443
1549999181,15cyclictest0-21swapper/319:32:153
15499991811,4cyclictest836-21runrttasks23:50:243
1549299183,3cyclictest26125-21gltestperf00:25:212
1549299182,6cyclictest10108-21/usr/sbin/munin23:50:242
1549299181,6cyclictest26561-21cpuspeed_turbos22:00:122
1549299181,6cyclictest16080-21/usr/sbin/munin22:50:202
1549299181,2cyclictest86850irq/129-enp1s0-23:15:162
1549299180,5cyclictest0-21swapper/200:16:252
1547899185,4cyclictest7-21ksoftirqd/022:10:180
1547899183,9cyclictest0-21swapper/022:45:220
15478991814,3cyclictest7-21ksoftirqd/020:45:020
1547899181,3cyclictest25532-21entropy19:30:150
15478991812,3cyclictest15751-21awk19:10:120
15478991811,4cyclictest1-21systemd20:40:020
15478991811,4cyclictest1-21systemd20:40:020
1547899181,12cyclictest0-21swapper/019:35:190
1547899181,10cyclictest0-21swapper/022:56:330
1549999171,4cyclictest30352-21tr20:55:153
1549999171,3cyclictest0-21swapper/323:35:143
1549999171,3cyclictest0-21swapper/323:30:153
15499991712,3cyclictest592-21tail21:00:223
15499991712,3cyclictest29313-21switchtime20:50:283
15499991712,3cyclictest24011-21grep19:25:203
15499991712,3cyclictest21825-21ntp_offset19:20:193
15499991712,3cyclictest21138-21acpi00:15:133
1549999171,15cyclictest0-21swapper/321:42:323
1549999171,15cyclictest0-21swapper/319:17:463
1549299172,6cyclictest88150irq/134-enp2s0-20:15:152
1549299171,7cyclictest0-21swapper/223:55:152
1549299171,6cyclictest3117-21grep19:50:252
1549299171,4cyclictest22041-21/usr/sbin/munin21:50:192
1549299171,4cyclictest19218-21/usr/sbin/munin20:30:202
1549299171,4cyclictest14811-21/usr/sbin/munin20:20:122
1549299171,3cyclictest19323-21if_enp2s019:15:162
1549299171,3cyclictest18627-21/usr/sbin/munin22:55:172
1549299171,3cyclictest178362ls21:40:122
15492991712,3cyclictest26392-21gltestperf20:45:162
15492991710,4cyclictest24545-21grep19:25:252
1547899173,4cyclictest0-21swapper/019:45:480
1547899172,12cyclictest18511-21grep20:25:280
15499991612,2cyclictest836-21runrttasks19:35:063
1549299161,6cyclictest0-21swapper/223:20:102
1549299161,3cyclictest0-21swapper/220:52:442
1549299161,2cyclictest0-21swapper/220:44:532
15492991612,3cyclictest0-21swapper/219:21:022
1549299161,14cyclictest0-21swapper/220:01:332
15492991610,3cyclictest27996-21expr00:30:142
1547899168,1cyclictest0-21swapper/019:55:230
1547899161,6cyclictest0-21swapper/020:05:180
1547899161,14cyclictest0-21swapper/020:50:230
15478991611,3cyclictest3551-21seq19:54:510
1549299153,3cyclictest0-21swapper/220:25:152
15492991510,3cyclictest30042-21grep19:40:152
15492991510,3cyclictest10630-21expr20:10:142
1547899159,3cyclictest14987-21cat20:20:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional