You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 13:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 12, 2025 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
98602239164,25sleep20-21swapper/219:07:102
98192234162,21sleep30-21swapper/319:06:393
98742231163,54sleep10-21swapper/119:07:211
97392211179,21sleep00-21swapper/019:05:340
101799910197,3cyclictest0-21swapper/219:18:092
85602690,2sleep00-21swapper/020:15:240
237932650,1sleep00-21swapper/000:30:270
237532600,3sleep232-21ksoftirqd/220:50:232
222892600,2sleep20-21swapper/200:30:012
307592470,4sleep230-21rcuc/222:20:192
10184994327,3cyclictest0-21swapper/320:31:223
10184994327,3cyclictest0-21swapper/320:31:223
10184994025,10cyclictest0-21swapper/321:59:373
10184993927,3cyclictest0-21swapper/320:51:243
10184993831,3cyclictest0-21swapper/319:21:243
10184993816,9cyclictest0-21swapper/320:56:513
10184993726,3cyclictest0-21swapper/323:14:393
10184993722,7cyclictest0-21swapper/319:54:083
10184993717,4cyclictest0-21swapper/322:11:013
10184993616,3cyclictest0-21swapper/321:13:533
1018499361,20cyclictest0-21swapper/323:55:473
1018499361,14cyclictest0-21swapper/319:30:183
1018499361,12cyclictest19874-40dnf20:40:323
1018499351,32cyclictest0-21swapper/321:03:393
1018499351,15cyclictest0-21swapper/323:45:213
1018499351,15cyclictest0-21swapper/323:45:213
1018499341,15cyclictest0-21swapper/322:16:153
1018499339,7cyclictest0-21swapper/319:56:543
10184993321,3cyclictest0-21swapper/322:30:233
10184993315,3cyclictest0-21swapper/322:37:173
10184993314,3cyclictest0-21swapper/300:03:053
1018499331,15cyclictest13971-40dnf21:41:233
1018499331,14cyclictest0-21swapper/320:18:413
278572320,2sleep121-21rcuc/122:15:161
1018499328,8cyclictest0-21swapper/319:43:363
10184993221,2cyclictest0-21swapper/300:36:493
10184993215,3cyclictest0-21swapper/323:29:093
10184993212,9cyclictest0-21swapper/320:09:553
1018499321,19cyclictest0-21swapper/321:37:333
1018499321,15cyclictest0-21swapper/321:34:373
1018499321,15cyclictest0-21swapper/300:14:133
1018499321,12cyclictest0-21swapper/300:15:333
10184993119,3cyclictest0-21swapper/320:39:483
10184993111,9cyclictest0-21swapper/320:28:563
1018499311,14cyclictest0-21swapper/323:20:253
1018499311,13cyclictest0-21swapper/323:03:573
1018499311,12cyclictest0-21swapper/321:25:493
1018499310,24cyclictest0-21swapper/323:53:073
1017499311,5cyclictest86550irq/126-enp1s0-21:25:151
22122300,2sleep10-21swapper/122:30:131
1018499309,3cyclictest0-21swapper/300:24:533
1018499307,3cyclictest0-21swapper/323:09:433
10184993016,3cyclictest0-21swapper/319:37:463
1018499301,27cyclictest0-21swapper/320:23:363
1018499301,18cyclictest0-21swapper/323:40:573
1018499301,18cyclictest0-21swapper/323:40:573
1018499301,18cyclictest0-21swapper/322:29:013
1018499301,12cyclictest0-21swapper/320:01:043
1018499300,20cyclictest0-21swapper/322:49:453
10184992918,3cyclictest0-21swapper/322:52:353
10184992912,7cyclictest0-21swapper/321:08:273
1018499291,15cyclictest0-21swapper/323:17:093
1018499291,13cyclictest5432-21diskstats20:10:153
1018499291,13cyclictest0-21swapper/321:23:333
1018499291,12cyclictest41-21ksoftirqd/300:09:393
10184992910,9cyclictest0-21swapper/319:19:213
1018499281,12cyclictest0-21swapper/322:07:153
1018499280,19cyclictest0-21swapper/321:45:213
1018499276,3cyclictest0-21swapper/321:15:353
1018499271,17cyclictest0-21swapper/321:53:093
10184992711,3cyclictest14634-21snmpd20:45:373
1018499271,10cyclictest0-21swapper/319:46:103
1018499271,10cyclictest0-21swapper/300:28:573
10174992721,3cyclictest4514-21munin-run23:50:011
10174992721,3cyclictest4514-21munin-run23:50:011
10174992710,3cyclictest6329-21users23:50:291
103652261,2sleep010408-21smart_sda00:00:260
1018499267,3cyclictest0-21swapper/322:59:453
1018499265,4cyclictest0-21swapper/300:30:073
10184992615,3cyclictest0-21swapper/323:30:413
1018499261,12cyclictest0-21swapper/322:23:533
1018499261,11cyclictest0-21swapper/322:45:013
1018499260,12cyclictest0-21swapper/322:03:113
1017499261,6cyclictest32596-21/usr/sbin/munin21:10:231
1017499261,18cyclictest0-21swapper/100:33:081
1017499261,15cyclictest21436-21diskstats22:00:161
1016999264,10cyclictest10504-21perf22:50:010
1018499251,12cyclictest0-21swapper/319:14:513
1017499259,4cyclictest9247-21munin-run20:20:011
1017499257,10cyclictest0-21swapper/123:21:281
1017499256,10cyclictest7280-21df_inode23:55:151
1017499251,4cyclictest17867-21grep21:50:271
1017499251,17cyclictest0-21swapper/100:05:141
1017499251,11cyclictest11455-21perf20:25:011
239972240,1sleep20-21swapper/222:05:182
1018499247,8cyclictest0-21swapper/319:25:483
1018499247,8cyclictest0-21swapper/319:25:483
1018499240,20cyclictest12467-21munin-node23:35:173
1017499242,16cyclictest0-21swapper/100:38:401
1017499241,20cyclictest22125-21munin-run00:30:011
1017499241,20cyclictest21528-21ntp_states20:45:221
1017499241,16cyclictest0-21swapper/100:10:201
1017499241,12cyclictest0-21swapper/123:28:521
1017499240,14cyclictest0-21swapper/121:19:111
1017499237,9cyclictest0-21swapper/122:42:021
1017499232,19cyclictest0-21swapper/119:33:581
1017499232,11cyclictest24209-21memory22:05:201
10174992317,3cyclictest23336-21hddtemp_smartct20:50:171
10174992315,3cyclictest0-21swapper/123:31:591
1017499231,19cyclictest29999-21cpuspeed_turbos22:20:131
1017499231,15cyclictest0-21swapper/123:15:151
1017499231,12cyclictest29452-21kworker/u8:222:52:191
1017499231,11cyclictest0-21swapper/121:47:121
1016999236,8cyclictest2443-21kworker/0:320:05:180
269612221,17sleep227086-21users22:10:282
1017999221,4cyclictest0-21swapper/200:10:302
1017499228,6cyclictest0-21swapper/121:36:111
1017499227,4cyclictest0-21swapper/100:19:221
1017499221,19cyclictest0-21swapper/120:00:271
1017499221,19cyclictest0-21swapper/119:40:191
1017499221,15cyclictest0-21swapper/122:45:331
1017499221,11cyclictest0-21swapper/122:56:421
1017499221,10cyclictest0-21swapper/123:44:151
1017499221,10cyclictest0-21swapper/123:44:151
1017499220,17cyclictest0-21swapper/122:37:521
1017499220,16cyclictest0-21swapper/100:04:181
1016999229,6cyclictest15294-21kworker/0:000:35:260
182892210,4sleep230-21rcuc/223:05:192
1017999211,8cyclictest17658-21users20:35:282
1017999211,10cyclictest6733-21ls23:55:012
1017999211,10cyclictest32595-21sshd21:10:232
1017499214,4cyclictest0-21swapper/121:56:231
1017499212,17cyclictest0-21swapper/119:26:561
1017499212,17cyclictest0-21swapper/119:26:561
10174992117,3cyclictest0-21swapper/122:27:191
10174992117,3cyclictest0-21swapper/121:41:421
10174992117,3cyclictest0-21swapper/120:08:211
1017499211,18cyclictest0-21swapper/123:06:081
1017499211,15cyclictest0-21swapper/123:38:271
1017499211,11cyclictest20234-21latency_hist20:45:021
1017499211,11cyclictest17110-21awk23:05:011
1017499211,11cyclictest13800-21cat20:30:011
1017499211,10cyclictest17967-21basename00:20:111
1017499211,10cyclictest0-21swapper/123:12:441
10174992110,3cyclictest10388-21cpu19:10:111
1016999211,7cyclictest0-21swapper/022:25:590
1016999211,17cyclictest22125-21munin-run00:30:010
1016999211,10cyclictest3188-21smart_sda22:30:230
66442201,2sleep00-21swapper/023:54:230
1017999206,3cyclictest86550irq/126-enp1s0-21:55:272
1017999205,4cyclictest26414-21latency_hist19:45:022
1017999204,8cyclictest86550irq/126-enp1s0-21:45:142
1017999204,3cyclictest0-21swapper/220:29:592
1017999202,7cyclictest0-21swapper/223:42:262
1017999202,7cyclictest0-21swapper/223:42:262
1017999201,6cyclictest0-21swapper/219:35:542
1017999201,5cyclictest23268-21unixbench_singl23:15:272
10179992014,3cyclictest20267-21grep19:30:152
1017999201,3cyclictest0-21swapper/221:55:012
1017999201,10cyclictest0-21swapper/222:55:492
1017499202,17cyclictest0-21swapper/122:12:531
1017499202,17cyclictest0-21swapper/121:20:401
10174992016,3cyclictest0-21swapper/121:30:231
10174992016,3cyclictest0-21swapper/119:21:061
1017499201,12cyclictest0-21swapper/120:35:271
1017499201,10cyclictest31231-21latency_hist21:10:021
1017499201,10cyclictest26491-21chrt20:55:481
1017499200,10cyclictest28758-21cpuspeed_turbos19:50:141
1016999206,8cyclictest0-21swapper/023:56:080
1016999206,2cyclictest15482-21needreboot21:45:190
1016999204,2cyclictest2864-21grep21:15:280
1016999202,10cyclictest20465-21/usr/sbin/munin20:45:200
1016999201,6cyclictest17605-21kworker/0:100:20:170
10169992011,3cyclictest86750irq/128-enp1s0-22:22:180
1016999201,11cyclictest26108-21/usr/sbin/munin23:25:220
1016999201,11cyclictest15760-21/usr/sbin/munin00:15:280
1016999201,11cyclictest0-21swapper/023:32:370
1016999201,11cyclictest0-21swapper/023:12:280
1016999201,10cyclictest9726-21sshd21:33:210
1016999201,10cyclictest5021-21tail21:20:290
1016999201,10cyclictest357-21sed21:10:260
1016999201,10cyclictest2322-21perf23:45:010
1016999201,10cyclictest2322-21perf23:45:010
278592190,3sleep20-21swapper/223:27:242
27402191,3sleep20-21swapper/222:30:192
241132191,3sleep20-21swapper/200:32:352
191272191,3sleep00-21swapper/023:08:060
185772190,1sleep20-21swapper/220:40:162
141862191,3sleep00-21swapper/021:44:080
1017999197,3cyclictest86550irq/126-enp1s0-00:15:242
1017999193,4cyclictest6343-21cpu22:40:152
1017999192,3cyclictest0-21swapper/223:45:182
1017999192,3cyclictest0-21swapper/223:45:182
1017999191,6cyclictest0-21swapper/223:03:062
1017999191,5cyclictest0-21swapper/222:03:372
1017999191,5cyclictest0-21swapper/219:22:272
1017999191,15cyclictest971-21expr20:00:152
1017999191,15cyclictest11397-21hddtemp_smartct22:50:192
10179991911,4cyclictest848-21sssd_nss00:20:232
1017999191,11cyclictest0-21swapper/223:57:422
1017999191,10cyclictest9864-21turbostat21:35:012
1017499191,9cyclictest28283-21/usr/sbin/munin21:00:261
10174991915,3cyclictest0-21swapper/119:47:471
1017499191,3cyclictest0-21swapper/119:15:121
1017499191,17cyclictest0-21swapper/119:55:591
1017499191,17cyclictest0-21swapper/119:38:571
1017499190,18cyclictest0-21swapper/120:31:581
1017499190,18cyclictest0-21swapper/120:31:581
1016999196,3cyclictest86550irq/126-enp1s0-00:10:280
1016999194,3cyclictest7259-21grep21:25:280
1016999193,13cyclictest11-21rcuc/019:25:270
1016999193,13cyclictest11-21rcuc/019:25:270
1016999192,8cyclictest10473-21strings20:20:220
1016999192,13cyclictest14634-21snmpd20:00:270
1016999191,6cyclictest12938-21chrt20:25:240
1016999191,4cyclictest0-21swapper/021:53:500
1016999191,10cyclictest26099-21tail22:10:160
1016999191,10cyclictest23911-21/usr/sbin/munin23:20:260
1016999191,10cyclictest21844-21awk22:00:180
1016999191,10cyclictest14634-21snmpd00:06:390
1016999191,10cyclictest10722-21/usr/sbin/munin22:50:270
321782180,2sleep20-21swapper/223:36:232
155542181,3sleep00-21swapper/020:31:270
155542181,3sleep00-21swapper/020:31:270
118072181,2sleep00-21swapper/021:36:490
102872181,2sleep20-21swapper/222:47:342
1017999181,7cyclictest0-21swapper/221:40:162
1017999181,6cyclictest0-21swapper/220:18:072
1017999181,5cyclictest27683-21sendmail_mailqu19:45:272
1017999181,3cyclictest0-21swapper/221:05:202
10179991813,2cyclictest29906-21smart_sda19:50:262
10179991812,3cyclictest9321-21cstates00:00:142
10179991812,3cyclictest29855-21cut22:20:012
10179991811,4cyclictest26035-21date23:25:012
1017999181,11cyclictest0-21swapper/222:39:032
1017999181,11cyclictest0-21swapper/222:25:422
1017499180,17cyclictest0-21swapper/120:14:411
1016999188,2cyclictest14017-21kworker/0:021:55:190
1016999187,8cyclictest29737-21fschecks_time21:05:150
1016999187,2cyclictest7-21ksoftirqd/019:35:200
1016999185,3cyclictest5230-1kworker/u9:122:37:150
1016999185,3cyclictest31581-21ntp_states23:35:270
1016999181,9cyclictest14634-21snmpd19:54:580
1016999181,8cyclictest23274-21grep23:15:260
1016999181,7cyclictest0-21swapper/022:55:260
1016999181,7cyclictest0-21swapper/019:10:260
1016999181,10cyclictest24967-21sshd22:06:220
1016999181,10cyclictest0-21swapper/023:03:040
1016999180,7cyclictest0-21swapper/020:53:240
70392170,1sleep00-21swapper/020:15:010
112122171,2sleep20-21swapper/220:22:102
1017999176,3cyclictest86550irq/126-enp1s0-21:20:192
1017999175,3cyclictest86850irq/129-enp1s0-21:00:192
1017999171,9cyclictest0-21swapper/220:05:272
1017999171,6cyclictest24542-21/usr/sbin/munin00:35:222
1017999171,6cyclictest0-21swapper/223:31:102
1017999171,6cyclictest0-21swapper/221:17:042
1017999171,6cyclictest0-21swapper/220:59:432
1017999171,5cyclictest0-21swapper/221:27:242
1017999171,4cyclictest0-21swapper/220:12:022
1017999171,13cyclictest14788-21memory20:30:192
1017999171,13cyclictest14788-21memory20:30:192
10179991710,4cyclictest19992-21fschecks_time23:10:192
1016999174,3cyclictest0-21swapper/019:30:050
1016999172,12cyclictest24838-21gltestperf19:40:200
1016999171,9cyclictest0-21swapper/019:19:510
1016999171,7cyclictest0-21swapper/019:58:020
1016999171,6cyclictest28343-21tail22:15:190
1016999171,6cyclictest25853-21head20:55:200
1016999171,6cyclictest0-21swapper/022:40:510
1016999171,5cyclictest27051-21grep19:45:190
10169991711,3cyclictest16672-21grep20:35:170
1016999170,16cyclictest0-21swapper/019:20:170
1017999169,4cyclictest18700-21ntp_kernel_pll_19:25:222
1017999169,4cyclictest18700-21ntp_kernel_pll_19:25:222
1017999161,9cyclictest10157-21cyclictest21:36:202
1017999161,6cyclictest32013-21ps19:55:262
1017999161,3cyclictest13210-21chrt00:08:382
10179991611,3cyclictest20468-21acpi20:45:132
10179991611,3cyclictest11154-21if_err_enp2s019:10:172
1016999165,3cyclictest86750irq/128-enp1s0-23:45:150
1016999165,3cyclictest86750irq/128-enp1s0-23:45:150
1016999161,5cyclictest18288-21egrep20:40:140
1016999161,3cyclictest28354-21cat21:00:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional