You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-18 - 04:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 18, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
44642239164,24sleep30-21swapper/319:05:473
46782237162,24sleep10-21swapper/119:08:311
46682234163,58sleep20-21swapper/219:08:242
45562201162,26sleep00-21swapper/019:06:560
3171021720,4sleep3490299cyclictest00:35:173
1467421460,4sleep3490299cyclictest23:50:133
4897991121,2cyclictest0-21swapper/222:30:372
4897997370,2cyclictest0-21swapper/221:44:502
4897996561,3cyclictest0-21swapper/223:13:132
64222560,4sleep230-21rcuc/219:10:252
12992550,3sleep339-21rcuc/321:35:253
490299401,27cyclictest6770-21latency_hist20:30:023
490299401,27cyclictest6770-21latency_hist20:30:023
4885994010,14cyclictest16361-21munin-run23:55:020
490299392,33cyclictest26028-21/usr/sbin/munin00:20:203
490299392,23cyclictest2170-21/usr/sbin/munin23:15:213
490299374,19cyclictest780-21rngd20:32:423
490299371,24cyclictest19706-21memory00:00:203
490299367,19cyclictest29512-21/usr/sbin/munin23:00:173
490299364,20cyclictest780-21rngd20:54:023
490299362,24cyclictest21764-21/usr/sbin/munin19:50:193
490299361,23cyclictest21457-21latency_hist22:35:023
490299351,24cyclictest13596-21/usr/sbin/munin20:45:243
490299351,22cyclictest836-21runrttasks23:08:463
490299351,22cyclictest780-21rngd20:14:123
490299351,22cyclictest0-21swapper/323:58:593
490299351,20cyclictest780-21rngd19:49:163
490299345,15cyclictest86850irq/129-enp1s0-21:00:183
490299343,20cyclictest780-21rngd19:34:573
490299341,23cyclictest848-21sssd_nss22:20:253
490299341,21cyclictest7331-21cpuspeed_turbos21:55:133
488599347,12cyclictest32556-21perf20:15:010
490299331,28cyclictest23964-21/usr/sbin/munin19:55:263
490299331,21cyclictest29658-21sshd21:25:553
490299331,21cyclictest29658-21sshd21:25:553
490299331,18cyclictest0-21swapper/322:15:193
4902993311,19cyclictest3010-21ethtool21:40:173
490299325,16cyclictest780-21rngd23:23:043
490299322,27cyclictest2602-21/usr/sbin/munin20:20:213
490299322,26cyclictest24269-21smart_sda21:10:213
490299321,28cyclictest5960-21sed21:50:203
490299321,23cyclictest0-21swapper/319:11:383
490299321,21cyclictest848-21sssd_nss23:10:213
4902993211,18cyclictest21676-21cpu00:10:143
490299312,23cyclictest4850-21perf21:50:023
490299312,15cyclictest86550irq/126-enp1s0-22:50:133
490299311,7cyclictest0-21swapper/322:57:563
490299311,27cyclictest26004-21/usr/sbin/munin21:15:183
490299311,20cyclictest27409-21/usr/sbin/munin21:20:153
490299311,19cyclictest29250-21fschecks_count00:30:113
490299311,18cyclictest14634-21snmpd20:16:483
490299311,16cyclictest32272-21cat21:35:023
490299311,14cyclictest0-21swapper/323:32:523
490299311,13cyclictest0-21swapper/300:29:243
4902993111,17cyclictest11769-21sort20:40:173
490299302,25cyclictest12578-21/usr/sbin/munin23:40:213
490299302,25cyclictest12578-21/usr/sbin/munin23:40:213
490299301,26cyclictest27222-21/usr/sbin/munin20:00:213
490299301,26cyclictest24891-21/usr/sbin/munin00:15:223
490299301,26cyclictest14233-21sshd22:14:553
490299301,22cyclictest0-21swapper/320:36:023
4902993012,15cyclictest9131-21proc_pri19:15:263
490299301,11cyclictest0-21swapper/319:28:233
490299292,5cyclictest0-21swapper/322:02:463
490299292,23cyclictest29001-21grep20:05:173
490299291,25cyclictest10237-21cpuspeed_turbos19:20:133
490299291,25cyclictest10237-21cpuspeed_turbos19:20:123
490299291,24cyclictest13797-40dnf23:45:313
490299291,24cyclictest12149-21latency_hist22:10:023
4902992911,15cyclictest16982-21expr19:35:143
489799290,11cyclictest86550irq/126-enp1s0-21:10:152
489099293,13cyclictest24253-40dnf22:44:571
490299283,13cyclictest0-21swapper/323:27:543
490299281,5cyclictest0-21swapper/322:26:143
490299281,24cyclictest9378-21/usr/sbin/munin23:35:293
490299281,24cyclictest22383-21/usr/sbin/munin22:40:103
490299281,23cyclictest19583-21tail19:40:193
490299281,23cyclictest19547-21grep20:55:293
490299281,12cyclictest0-21swapper/322:37:043
490299271,6cyclictest0-21swapper/300:05:313
489799279,1cyclictest0-21swapper/221:50:262
4897992711,7cyclictest2170-21/usr/sbin/munin23:15:202
490299261,21cyclictest22165-21latency_hist21:05:033
489799269,7cyclictest11399-21/usr/sbin/munin20:40:172
4897992612,5cyclictest86850irq/129-enp1s0-19:50:142
4897992610,3cyclictest15651-21/usr/sbin/munin23:50:232
489099260,17cyclictest0-21swapper/122:25:311
488599261,13cyclictest27095-21munin-run22:55:020
490299251,11cyclictest0-21swapper/322:46:003
489799259,8cyclictest30347-21kernelversion23:00:172
489799259,7cyclictest2104-21cat21:40:022
489799258,4cyclictest12279-21/usr/sbin/munin22:10:152
489799251,13cyclictest0-21swapper/221:22:372
489799251,12cyclictest0-21swapper/200:35:072
489799251,10cyclictest0-21swapper/200:14:412
4890992518,4cyclictest3059-21irqstats21:40:171
4890992518,4cyclictest25464-21/usr/sbin/munin22:50:101
489099251,17cyclictest22896-21smartctl00:10:251
489099251,16cyclictest28577-21ps21:20:231
488599257,15cyclictest22802-21awk21:10:010
488599251,12cyclictest6954-21munin-run23:30:010
489799248,8cyclictest25454-21df_inode22:50:092
489799247,7cyclictest10152-21date19:20:012
489799247,7cyclictest10152-21date19:20:012
4897992412,3cyclictest86550irq/126-enp1s0-19:25:182
489799241,11cyclictest0-21swapper/200:07:512
489799241,10cyclictest0-21swapper/220:06:072
489099243,14cyclictest86750irq/128-enp1s0-23:37:541
489099242,7cyclictest30458-21kworker/u8:119:43:321
4890992416,2cyclictest0-21swapper/123:15:131
489099241,4cyclictest29289-21munin-run23:00:011
489099241,17cyclictest0-21swapper/121:38:441
489799239,4cyclictest32090-21sshd21:33:562
489799235,6cyclictest19375-21gltestperf00:00:162
489799235,2cyclictest22004-21kworker/2:020:32:572
489799233,7cyclictest0-21swapper/200:25:112
489799231,8cyclictest9946-21munin-run19:20:012
4897992311,5cyclictest86550irq/126-enp1s0-22:00:282
489799230,13cyclictest32-21ksoftirqd/223:08:412
489099237,8cyclictest11049-21users22:00:281
489099231,9cyclictest29903-21ntp_states00:30:171
489099231,17cyclictest0-21swapper/123:00:231
489099231,17cyclictest0-21swapper/121:07:451
489099231,16cyclictest0-21swapper/119:55:381
489099231,15cyclictest22783-21diskmemload23:45:241
489099231,14cyclictest0-21swapper/121:30:361
488599231,15cyclictest20532-21/usr/sbin/munin21:00:190
242502231,3sleep00-21swapper/022:44:560
489799229,5cyclictest86550irq/126-enp1s0-22:55:252
489799227,4cyclictest16928-21df_inode23:55:142
489799225,7cyclictest10446-21ntp_states23:35:222
489799223,11cyclictest22783-21diskmemload23:48:002
489799221,9cyclictest0-21swapper/221:46:082
4897992213,2cyclictest86850irq/129-enp1s0-23:25:002
489799221,10cyclictest0-21swapper/220:00:292
4890992214,3cyclictest0-21swapper/123:09:561
489099221,3cyclictest15645-21mailstats23:50:231
489099221,16cyclictest0-21swapper/122:47:281
489099221,16cyclictest0-21swapper/120:36:121
489099221,16cyclictest0-21swapper/120:27:531
489099221,16cyclictest0-21swapper/120:27:531
489099221,12cyclictest7054-21munin-run19:15:001
489099221,11cyclictest14532-21cat19:30:021
489099220,16cyclictest0-21swapper/123:20:251
488599223,7cyclictest20798-40dnf19:43:330
488599223,6cyclictest19041-1kworker/u9:221:30:270
489799216,5cyclictest88250irq/135-enp2s0-20:23:272
489799215,7cyclictest32-21ksoftirqd/221:55:142
489799213,8cyclictest32767-21/usr/sbin/munin20:15:222
489799213,8cyclictest25898-21hwlatdetect21:15:162
489799213,8cyclictest0-21swapper/223:30:152
489799212,2cyclictest13513-21runrttasks23:44:452
489799212,2cyclictest13513-21runrttasks23:44:452
489799212,10cyclictest17021-21df_inode22:20:152
489799211,9cyclictest0-21swapper/220:48:392
489799211,12cyclictest0-21swapper/221:29:592
489799211,12cyclictest0-21swapper/221:29:582
489799210,12cyclictest15795-21/usr/sbin/munin20:50:132
489099219,3cyclictest5410-21df21:50:141
489099215,3cyclictest30343-21dump-pmu-power20:10:001
489099211,8cyclictest0-21swapper/121:45:301
4890992114,2cyclictest0-21swapper/120:52:101
4890992114,2cyclictest0-21swapper/119:23:531
4890992114,2cyclictest0-21swapper/119:23:521
489099211,3cyclictest0-21swapper/121:17:521
4890992113,6cyclictest7891-21fschecks_count19:15:041
4890992113,3cyclictest0-21swapper/121:29:271
4890992113,3cyclictest0-21swapper/121:29:271
489099211,18cyclictest18792-21/usr/sbin/munin00:00:261
489099211,17cyclictest22783-21diskmemload23:31:011
489099211,15cyclictest0-21swapper/123:29:101
489099211,15cyclictest0-21swapper/122:19:251
489099211,15cyclictest0-21swapper/122:11:401
489099211,15cyclictest0-21swapper/120:58:181
489099211,15cyclictest0-21swapper/120:42:171
489099211,15cyclictest0-21swapper/120:15:281
489099211,15cyclictest0-21swapper/119:46:371
489099211,15cyclictest0-21swapper/119:39:111
489099211,15cyclictest0-21swapper/100:07:381
489099211,14cyclictest0-21swapper/119:30:081
489099211,11cyclictest23730-21crond19:55:011
489099210,15cyclictest0-21swapper/121:05:011
488599216,5cyclictest0-21swapper/021:17:140
488599215,5cyclictest0-21swapper/023:32:090
488599212,12cyclictest2858950sshd00:29:550
488599210,10cyclictest0-21swapper/023:43:010
488599210,10cyclictest0-21swapper/023:43:010
488599210,10cyclictest0-21swapper/019:40:020
312082210,1sleep10-21swapper/120:10:201
68162201,2sleep20-21swapper/223:28:242
489799207,3cyclictest0-21swapper/220:13:032
489799206,5cyclictest16206-21chrt22:17:032
489799204,6cyclictest23829-21/usr/sbin/munin00:15:272
489799204,2cyclictest30-21rcuc/220:57:592
4897992012,2cyclictest88150irq/134-enp2s0-19:57:592
489099205,7cyclictest0-21swapper/122:06:041
489099202,4cyclictest14666-21ntp_states20:45:221
4890992015,3cyclictest32252-21rt-features00:35:231
489099201,3cyclictest0-21swapper/123:12:051
489099201,15cyclictest0-21swapper/123:58:401
489099201,15cyclictest0-21swapper/123:42:021
489099201,15cyclictest0-21swapper/123:42:021
489099201,15cyclictest0-21swapper/122:35:491
489099201,15cyclictest0-21swapper/122:22:171
489099201,15cyclictest0-21swapper/100:16:251
489099201,14cyclictest0-21swapper/100:29:491
489099200,15cyclictest0-21swapper/122:33:321
489099200,15cyclictest0-21swapper/120:21:581
489099200,11cyclictest0-21swapper/100:20:091
488599209,3cyclictest24790-21sshd22:48:550
488599207,4cyclictest5197-21cpuspeed_turbos21:50:140
488599207,3cyclictest0-21swapper/000:24:260
488599205,5cyclictest0-21swapper/023:55:470
488599205,4cyclictest0-21swapper/000:16:480
488599204,8cyclictest22783-21diskmemload22:09:230
488599202,9cyclictest14634-21snmpd20:35:150
488599202,6cyclictest0-21swapper/021:29:060
488599202,6cyclictest0-21swapper/021:29:060
488599202,10cyclictest21764-21/usr/sbin/munin19:50:110
4885992014,3cyclictest21546-21munin-run19:50:020
488599201,3cyclictest7600-21df21:55:140
488599201,11cyclictest31771-21sshd23:05:550
488599201,11cyclictest0-21swapper/023:04:580
488599201,10cyclictest22783-21diskmemload22:55:250
488599201,10cyclictest11351-21kworker/u8:022:10:160
488599201,10cyclictest0-21swapper/000:01:540
489799195,8cyclictest0-21swapper/222:06:422
489799195,2cyclictest14568-21/usr/sbin/munin19:30:262
489799192,3cyclictest0-21swapper/219:46:432
489799191,2cyclictest86850irq/129-enp1s0-21:10:002
489799191,11cyclictest0-21swapper/200:34:232
489799191,11cyclictest0-21swapper/200:20:042
489799191,10cyclictest0-21swapper/222:47:162
489099199,2cyclictest23344-21df21:10:101
489099191,2cyclictest22783-21diskmemload21:56:141
4890991912,2cyclictest0-21swapper/120:03:121
488599197,4cyclictest0-21swapper/022:30:260
488599194,6cyclictest0-21swapper/000:08:230
488599192,10cyclictest28371-21/usr/sbin/munin20:05:210
488599191,6cyclictest0-21swapper/023:36:400
488599191,5cyclictest0-21swapper/021:48:250
488599191,3cyclictest0-21swapper/022:04:100
4885991913,3cyclictest14634-21snmpd19:30:020
488599191,10cyclictest27409-21/usr/sbin/munin21:20:210
488599191,10cyclictest0-21swapper/021:39:190
488599190,5cyclictest0-21swapper/020:20:170
32062191,15sleep03233-21ntp_states23:15:230
489799184,7cyclictest311ktimersoftd/222:42:152
4897991812,3cyclictest20321-21grep21:00:182
489799181,11cyclictest0-21swapper/222:25:472
489099181,16cyclictest0-21swapper/120:30:011
488599188,3cyclictest5084-21/usr/sbin/munin19:10:180
488599184,3cyclictest0-21swapper/023:48:240
488599183,4cyclictest0-21swapper/022:21:590
488599182,2cyclictest0-21swapper/022:29:100
488599181,9cyclictest2249-21/usr/sbin/munin21:40:220
488599181,9cyclictest0-21swapper/023:21:440
488599181,9cyclictest0-21swapper/000:14:000
488599181,5cyclictest0-21swapper/019:15:230
488599181,3cyclictest0-21swapper/023:10:540
219052181,2sleep00-21swapper/022:38:010
217132181,3sleep20-21swapper/222:35:332
489799174,3cyclictest30-21rcuc/219:37:542
489799174,2cyclictest0-21swapper/220:35:252
489799171,8cyclictest0-21swapper/220:25:232
489799171,8cyclictest0-21swapper/220:25:232
4897991712,3cyclictest18962-21/usr/sbin/munin19:40:152
488599175,3cyclictest25074-21ls19:55:230
488599173,11cyclictest12960-21head20:40:280
488599171,6cyclictest0-21swapper/020:30:280
488599171,6cyclictest0-21swapper/000:36:540
488599171,6cyclictest0-21swapper/000:30:290
4885991712,3cyclictest19919-21seq20:59:410
488599171,10cyclictest0-21swapper/022:15:520
4885991710,4cyclictest6042-21fgrep20:25:250
4885991710,4cyclictest6042-21fgrep20:25:250
488599170,4cyclictest0-21swapper/021:12:490
152722171,2sleep00-21swapper/020:46:260
488599162,9cyclictest0-21swapper/020:15:130
488599161,6cyclictest27801-21sshd20:00:550
488599161,6cyclictest10169-21/usr/sbin/munin19:20:180
488599161,6cyclictest10169-21/usr/sbin/munin19:20:180
4885991611,3cyclictest836-21runrttasks20:50:150
163932161,2sleep00-21swapper/019:33:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional