You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-06 - 00:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 05, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
263132239162,25sleep00-21swapper/007:06:040
262892239164,25sleep10-21swapper/107:05:441
264662235164,57sleep30-21swapper/307:08:033
263722205168,24sleep20-21swapper/207:06:492
130821720,4sleep12671699cyclictest08:35:241
1282721670,3sleep22672299cyclictest09:00:262
26722999087,2cyclictest0-21swapper/208:49:552
111742790,3sleep39-21rcu_preempt07:45:173
74332730,4sleep230-21rcuc/210:05:022
26722997257,8cyclictest0-21swapper/211:57:342
89552680,1sleep30-21swapper/312:35:153
89552680,1sleep30-21swapper/312:35:153
268502630,4sleep02671299cyclictest12:00:240
312912600,4sleep230-21rcuc/209:45:142
181502590,3sleep12671699cyclictest09:15:141
181502590,3sleep12671699cyclictest09:15:141
306102580,3sleep030613-21fschecks_time12:10:150
204392580,4sleep12671699cyclictest09:20:161
123852510,3sleep339-21rcuc/310:15:183
2671699384,7cyclictest0-21swapper/110:02:461
2671699361,9cyclictest9441-21munin-run10:10:001
26716993616,4cyclictest19996-21memory11:45:231
2671699361,2cyclictest25634-21grep10:45:161
26716993610,5cyclictest647-21kworker/u8:009:08:481
2671699357,3cyclictest18471-21users11:40:291
2671699351,18cyclictest836-21runrttasks07:20:441
2671699343,15cyclictest86750irq/128-enp1s0-09:35:381
2671699342,3cyclictest25786-21munin-run08:20:001
2671699342,26cyclictest0-21swapper/107:10:241
2671699341,29cyclictest1429-21perl09:50:151
2671699331,10cyclictest31845-21cat07:20:011
2671299335,12cyclictest23254-21idleruntime-cro11:55:010
2671699322,13cyclictest16655-21ls11:40:021
2671699321,10cyclictest11321-21ntp_kernel_pll_11:25:211
2671699310,18cyclictest0-21swapper/108:52:551
2671299316,11cyclictest0-21swapper/008:05:000
2672799291,11cyclictest0-21swapper/310:20:143
319702280,3sleep121-21rcuc/111:00:151
2671699285,14cyclictest780-21rngd11:59:321
2671699281,15cyclictest0-21swapper/112:35:441
2671699281,15cyclictest0-21swapper/112:35:441
26716992810,3cyclictest3210-21timerandwakeup07:25:271
26716992713,2cyclictest0-21swapper/112:09:491
2671699271,17cyclictest0-21swapper/110:44:131
2671699270,8cyclictest86550irq/126-enp1s0-07:50:251
2671299277,17cyclictest25441-21munin-run12:00:010
2671299271,13cyclictest31413-21idleruntime-cro11:00:010
2672799260,12cyclictest23986-21kworker/u8:209:55:363
2671299267,16cyclictest12587-21idleruntime-cro07:50:010
2671299260,12cyclictest16664-21perf11:40:010
2671699251,5cyclictest4383-21diskstats07:30:151
2671299256,16cyclictest8203-21munin-run07:40:010
2671299251,13cyclictest30199-21idleruntime-cro08:30:010
124122250,1sleep00-21swapper/011:30:010
2672299241,12cyclictest0-21swapper/210:10:102
26716992418,3cyclictest7152-21memory07:35:191
2671699241,11cyclictest0-21swapper/112:13:421
2671299242,10cyclictest5679-21perf11:15:000
2671699238,3cyclictest0-21swapper/108:07:331
26716992310,2cyclictest0-21swapper/111:10:191
2671299231,11cyclictest30821-21awk09:45:010
2672799221,7cyclictest0-21swapper/310:46:353
2672799221,5cyclictest0-21swapper/310:25:293
2672799221,5cyclictest0-21swapper/310:25:293
2672299221,11cyclictest780-21rngd07:50:202
2672299221,11cyclictest0-21swapper/211:50:212
2671699227,3cyclictest27584-21runrttasks12:04:181
2671699221,5cyclictest6570-21hddtemp_smartct11:15:191
26716992214,3cyclictest0-21swapper/112:15:171
2671699221,18cyclictest13555-21ntp_states11:30:241
2671699221,18cyclictest13555-21ntp_states11:30:231
2671699221,15cyclictest0-21swapper/109:58:041
2671699221,14cyclictest0-21swapper/110:14:291
2671299226,2cyclictest86550irq/126-enp1s0-11:46:360
2671299224,9cyclictest0-21swapper/007:28:170
222532220,2sleep20-21swapper/209:25:142
2672799213,4cyclictest0-21swapper/308:42:123
2672799212,7cyclictest0-21swapper/310:55:493
2672799211,2cyclictest0-21swapper/309:11:023
2672299217,3cyclictest9177-21grep08:55:112
2672299212,2cyclictest86550irq/126-enp1s0-10:57:032
2672299211,12cyclictest0-21swapper/212:07:482
2671699217,11cyclictest15655-21acpi09:10:131
2671699216,4cyclictest0-21swapper/110:20:251
2671699214,9cyclictest780-21rngd08:42:111
2671699212,10cyclictest3467-21munin-run11:10:001
2671699211,3cyclictest21278-21/usr/sbin/munin11:50:141
2671699211,15cyclictest0-21swapper/112:20:031
2671699211,10cyclictest0-21swapper/107:58:451
2671299211,11cyclictest9670-21/usr/sbin/munin10:10:160
59502201,3sleep00-21swapper/012:25:540
2672799204,3cyclictest0-21swapper/311:07:123
2672799203,9cyclictest19423-21/usr/sbin/munin08:05:253
2672799201,8cyclictest14634-21snmpd08:35:223
2672799201,11cyclictest26914-21/usr/sbin/munin07:10:173
2672799201,11cyclictest21278-21/usr/sbin/munin11:50:153
2672299202,9cyclictest30-21rcuc/209:40:012
2672299201,9cyclictest0-21swapper/207:34:252
26722992015,2cyclictest30-21rcuc/212:30:012
2672299201,4cyclictest0-21swapper/209:58:482
26722992014,3cyclictest8429-21df_inode11:20:152
26722992013,4cyclictest18454-21fgrep08:00:262
2672299201,11cyclictest15080-21grep07:55:152
2671699207,3cyclictest0-21swapper/112:25:261
2671699206,3cyclictest15415-21diskmemload11:20:191
26716992017,2cyclictest0-21swapper/108:24:171
26716992016,3cyclictest0-21swapper/112:32:201
26716992016,3cyclictest0-21swapper/110:55:191
26716992016,3cyclictest0-21swapper/109:04:531
26716992013,2cyclictest0-21swapper/110:18:241
2671699201,18cyclictest0-21swapper/110:36:151
2671699201,18cyclictest0-21swapper/108:56:151
2671699201,18cyclictest0-21swapper/108:33:481
2671699201,18cyclictest0-21swapper/107:42:141
2671699201,10cyclictest28235-21cpu08:25:131
2671699201,10cyclictest27925-21tail10:50:181
2671299206,5cyclictest0-21swapper/009:25:390
2671299206,4cyclictest0-21swapper/011:03:190
2671299205,3cyclictest28496-21kworker/u8:110:50:100
2671299201,4cyclictest11573-21df09:00:120
26712992014,3cyclictest13845-21awk10:20:000
2671299201,11cyclictest6516-21/usr/sbin/munin12:30:270
2671299201,11cyclictest13447-21/usr/sbin/munin09:05:160
2671299201,10cyclictest0-21swapper/011:40:330
2672799198,8cyclictest12185-21sshd11:29:193
2672799198,2cyclictest0-21swapper/311:17:023
2672799195,5cyclictest0-21swapper/307:15:243
2672799191,6cyclictest0-21swapper/310:45:023
2672799191,4cyclictest0-21swapper/309:40:293
26727991913,4cyclictest5518-21sed07:30:323
2672799191,10cyclictest12485-21/usr/sbin/munin11:30:143
2672799191,10cyclictest12485-21/usr/sbin/munin11:30:143
2672799190,10cyclictest0-21swapper/310:35:163
2672299198,3cyclictest86550irq/126-enp1s0-09:30:172
2672299195,3cyclictest86850irq/129-enp1s0-10:26:542
2672299195,3cyclictest86850irq/129-enp1s0-10:26:542
2672299194,9cyclictest0-21swapper/210:06:372
2672299193,3cyclictest86550irq/126-enp1s0-10:15:292
2672299191,8cyclictest581-21systemd-journal08:45:012
2672299191,8cyclictest1610-21grep11:05:172
2672299191,11cyclictest0-21swapper/209:16:092
2672299191,11cyclictest0-21swapper/209:16:092
26716991916,2cyclictest0-21swapper/109:25:241
2671699191,3cyclictest0-21swapper/109:46:051
26716991913,3cyclictest5301-21grep08:45:161
26716991913,3cyclictest18477-21sensors_temp08:00:251
2671699191,17cyclictest0-21swapper/107:47:561
2671699191,11cyclictest20285-40dnf10:33:241
2671299199,5cyclictest0-21swapper/010:04:560
2671299199,2cyclictest5894-21/usr/sbin/munin11:15:170
2671299198,6cyclictest0-21swapper/007:44:560
2671299198,3cyclictest17235-21users09:10:270
2671299195,5cyclictest0-21swapper/010:28:410
2671299195,5cyclictest0-21swapper/010:28:410
2671299194,5cyclictest0-21swapper/009:39:400
2671299192,10cyclictest32160-21sshd08:32:180
2671299191,9cyclictest4037-21/usr/sbin/munin07:30:150
2671299191,6cyclictest0-21swapper/010:45:050
2671299191,11cyclictest0-21swapper/012:21:250
2671299191,11cyclictest0-21swapper/009:45:210
2671299191,10cyclictest0-21swapper/009:55:180
167622190,2sleep00-21swapper/007:57:180
2672799189,7cyclictest4346-21gltestperf11:10:183
2672799185,3cyclictest0-21swapper/307:35:063
2672799181,7cyclictest4981-21gltestperf12:25:153
2672799181,7cyclictest26636-21/usr/sbin/munin09:35:143
2672799181,7cyclictest0-21swapper/311:43:093
2672799181,4cyclictest14634-21snmpd08:48:363
2672799181,3cyclictest86850irq/129-enp1s0-12:15:083
2672799181,3cyclictest0-21swapper/308:54:403
26727991813,3cyclictest9030-21gltestperf07:40:183
26727991813,3cyclictest29121-21grep12:05:243
26727991813,3cyclictest28297-21ntpq10:50:233
26727991812,3cyclictest19419-21grep09:15:283
26727991812,3cyclictest19419-21grep09:15:283
2672799181,15cyclictest0-21swapper/308:10:363
2672799181,14cyclictest17333-21cpuspeed_turbos08:00:153
2672799181,10cyclictest0-21swapper/310:01:113
2672299189,7cyclictest25788-21awk08:20:012
2672299187,9cyclictest9681-21mailstats07:40:252
2672299185,5cyclictest0-21swapper/212:16:492
2672299185,3cyclictest32-21ksoftirqd/209:41:322
2672299185,10cyclictest30-21rcuc/210:45:262
2672299184,3cyclictest0-21swapper/211:15:202
2672299182,7cyclictest19081-21/usr/sbin/munin11:45:272
2672299181,8cyclictest6848-21/usr/sbin/munin08:50:232
2672299181,8cyclictest24821-21munin-run10:45:022
2672299181,7cyclictest22010-21awk09:25:012
2672299181,7cyclictest0-21swapper/212:36:422
2672299181,7cyclictest0-21swapper/212:36:412
2672299181,7cyclictest0-21swapper/212:12:592
2672299181,7cyclictest0-21swapper/211:11:512
2672299181,7cyclictest0-21swapper/208:28:182
26722991815,2cyclictest32-21ksoftirqd/208:10:182
26722991813,3cyclictest10752-21tr07:45:132
2672299181,15cyclictest0-21swapper/212:20:102
2672299181,15cyclictest0-21swapper/208:05:482
2672299181,10cyclictest0-21swapper/211:35:332
2671699181,8cyclictest17860-21vmstat10:25:291
2671699181,8cyclictest17860-21vmstat10:25:291
2671699181,4cyclictest21685-21awk08:10:131
2671699181,16cyclictest0-21swapper/109:30:081
26716991811,4cyclictest30117-21/usr/sbin/munin09:40:261
2671299187,8cyclictest0-21swapper/010:35:260
2671299187,3cyclictest4648-21/usr/sbin/munin08:45:150
2671299187,3cyclictest0-21swapper/009:17:270
2671299187,3cyclictest0-21swapper/009:17:270
2671299186,3cyclictest0-21swapper/010:22:200
2671299186,3cyclictest0-21swapper/010:08:410
2671299186,3cyclictest0-21swapper/009:53:580
2671299184,5cyclictest12200-21kworker/u8:011:30:290
2671299184,5cyclictest12200-21kworker/u8:011:30:290
2671299184,4cyclictest20081-21sshd10:30:420
2671299183,12cyclictest904-21sshd08:35:180
2671299181,9cyclictest24431-21/usr/sbin/munin09:30:240
2671299181,9cyclictest0-21swapper/010:43:220
26712991813,2cyclictest836-21runrttasks07:10:170
26712991812,3cyclictest29034-21pmu-power12:05:240
2671299181,15cyclictest0-21swapper/008:11:330
2671299181,14cyclictest20949-21tail08:05:270
2671299181,10cyclictest8088-21/usr/sbin/munin11:20:240
210252181,2sleep30-21swapper/311:49:563
12682181,2sleep20-21swapper/207:21:032
103622181,2sleep00-21swapper/012:36:100
103622181,2sleep00-21swapper/012:36:100
9622171,2sleep20-21swapper/211:01:312
2672799174,4cyclictest23835-21grep11:55:153
2672799174,3cyclictest86850irq/129-enp1s0-11:20:153
2672799172,12cyclictest22235-21/usr/sbin/munin09:25:183
2672799171,9cyclictest0-21swapper/308:15:203
2672799171,7cyclictest4027-21NetworkManager12:24:243
2672799171,7cyclictest32157-21taskset08:32:163
2672799171,7cyclictest27252-21grep08:20:263
2672799171,7cyclictest15415-21diskmemload12:00:233
2672799171,7cyclictest15415-21diskmemload09:33:053
2672799171,7cyclictest14634-21snmpd09:00:373
2672799171,7cyclictest14022-21fschecks_time09:05:193
2672799171,7cyclictest0-21swapper/312:10:173
2672799171,6cyclictest15750-21ntp_states11:35:243
2672799171,6cyclictest0-21swapper/309:50:213
2672799171,5cyclictest0-21swapper/310:10:163
26727991712,3cyclictest9039-21/usr/sbin/munin08:55:213
26727991712,3cyclictest6698-21cstates12:30:173
26727991712,3cyclictest3422-21uptime07:25:293
26727991712,3cyclictest29016-21munin-node11:00:153
26727991712,3cyclictest20058-21vmstat10:30:273
26727991712,3cyclictest15303-21cat07:55:153
26727991712,3cyclictest0-21swapper/309:24:243
26727991712,2cyclictest28219-21/usr/sbin/munin08:25:233
26727991711,4cyclictest13410-21grep07:50:193
2672799170,11cyclictest0-21swapper/309:45:263
2672299176,3cyclictest86550irq/126-enp1s0-09:52:202
2672299176,3cyclictest30-21rcuc/207:35:142
2672299175,7cyclictest26633-21kworker/2:112:30:062
2672299171,6cyclictest0-21swapper/211:40:082
2672299171,5cyclictest11545-21smart_sda11:25:262
26722991713,3cyclictest0-21swapper/210:50:062
26722991712,3cyclictest836-21runrttasks11:34:362
26722991712,3cyclictest836-21runrttasks11:34:362
26722991712,3cyclictest27026-21expr07:10:142
26722991712,3cyclictest1920-21cat07:25:142
26722991712,3cyclictest16723-21ntpq09:10:232
26722991711,3cyclictest13448-21acpi09:05:142
2672299171,11cyclictest0-21swapper/208:20:162
2671299172,6cyclictest842-21lldpd11:06:010
2671299171,7cyclictest0-21swapper/008:15:240
2671299171,6cyclictest5230-1kworker/u9:108:22:140
2671299171,15cyclictest0-21swapper/008:41:170
2671299171,15cyclictest0-21swapper/007:21:430
26712991711,3cyclictest9039-21/usr/sbin/munin08:55:110
26712991711,3cyclictest20231-21expr09:20:140
16942170,2sleep00-21swapper/012:17:390
140462171,2sleep20-21swapper/210:20:042
2672799169,4cyclictest0-21swapper/310:05:253
26727991611,3cyclictest32294-21expr07:20:173
2672299161,7cyclictest0-21swapper/212:00:032
2672299161,5cyclictest0-21swapper/208:30:272
26722991612,3cyclictest0-21swapper/210:30:072
26722991611,3cyclictest29191-21if_enp1s007:15:032
2672299160,7cyclictest32708-21cpuspeed_turbos08:35:142
2671299165,3cyclictest6848-21/usr/sbin/munin08:50:160
2671299162,11cyclictest13459-21tail07:50:180
26712991610,3cyclictest31336-21unixbench_singl07:15:270
2672299151,6cyclictest32-21ksoftirqd/210:35:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional