You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-04 - 01:45

x86 Intel Xeon E3-1230 @3200 MHz, Linux 5.10.27-rt36 (Profile)

Latency plot of system in rack #b, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot2.osadl.org (updated Wed Aug 25, 2021 01:28:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2173317211093,6sleep30-21swapper/319:06:583
23128362810,0sleep6511rcuc/623:55:266
217331927874,3sleep50-21swapper/519:07:005
22104002730,0sleep10-21swapper/120:25:001
217329426343,6sleep70-21swapper/719:06:387
217347126249,6sleep20-21swapper/219:08:572
217336325937,7sleep40-21swapper/419:07:314
217348525848,6sleep60-21swapper/619:09:096
217328825848,6sleep10-21swapper/119:06:321
217340525646,6sleep00-21swapper/019:08:070
217585199255,3cyclictest141rcu_preempt22:03:006
217585299192,9cyclictest0-21swapper/722:50:017
2175852991914,3cyclictest2217522-21munin-run20:40:007
217584899189,1cyclictest0-21swapper/320:25:013
217585299179,2cyclictest2235876-21smart_sdc21:15:247
217585099170,16cyclictest0-21swapper/523:31:415
217584899170,10cyclictest0-21swapper/321:40:113
217584899170,10cyclictest0-21swapper/321:40:113
2175851991613,2cyclictest2217517-21turbostat20:40:006
217585099165,8cyclictest141rcu_preempt22:59:005
217584899160,11cyclictest2288007-21awk23:05:183
217584899160,11cyclictest2288007-21awk23:05:183
217584899160,10cyclictest0-21swapper/300:02:173
217585199155,7cyclictest141rcu_preempt19:30:016
217584999154,7cyclictest40-21ksoftirqd/419:32:474
217584899158,3cyclictest1439-21snmpd21:57:013
217584899158,2cyclictest2213625-21diskstats20:30:143
217584899150,2cyclictest0-21swapper/323:20:123
217584799157,8cyclictest0-21swapper/222:50:002
217584799153,4cyclictest0-21swapper/219:32:482
217584599150,1cyclictest0-21swapper/021:40:170
217584599150,1cyclictest0-21swapper/021:40:170
217585199146,3cyclictest0-21swapper/619:54:496
2175851991413,1cyclictest0-21swapper/619:56:226
217585199140,8cyclictest2297533-21gltestperf23:25:156
217585099142,2cyclictest141rcu_preempt22:27:005
217584999146,3cyclictest0-21swapper/420:53:324
2175849991413,1cyclictest0-21swapper/420:06:254
2175849991411,2cyclictest0-21swapper/421:05:144
217584799140,5cyclictest0-21swapper/220:02:322
217584699142,6cyclictest964-21NetworkManager00:00:451
217584599145,6cyclictest2241498-21munin-run21:30:000
217584599142,5cyclictest0-21swapper/021:11:570
217585299135,5cyclictest0-21swapper/720:15:167
217585299135,5cyclictest0-21swapper/720:15:157
217585299130,2cyclictest0-21swapper/722:20:007
217585299130,12cyclictest0-21swapper/720:15:007
217585199135,2cyclictest141rcu_preempt22:36:006
217585199132,2cyclictest141rcu_preempt00:33:026
217585199132,2cyclictest141rcu_preempt00:33:016
2175851991312,1cyclictest2313357-21kworker/6:100:24:406
217585099135,2cyclictest141rcu_preempt00:04:025
217585099132,2cyclictest141rcu_preempt23:01:025
217585099130,10cyclictest949-21systemd-machine20:28:245
217584999137,4cyclictest40-21ksoftirqd/420:50:004
217584999136,5cyclictest40-21ksoftirqd/421:40:004
217584999136,5cyclictest40-21ksoftirqd/420:55:014
217584999132,6cyclictest40-21ksoftirqd/419:21:274
217584999132,5cyclictest0-21swapper/400:36:334
217584999130,12cyclictest0-21swapper/423:58:354
217584899138,3cyclictest2287069-21latency_hist23:05:003
217584899135,3cyclictest0-21swapper/322:49:393
217584699132,6cyclictest2198315-21munin-run20:00:001
217584599134,4cyclictest0-21swapper/023:17:150
217584599130,8cyclictest0-21swapper/020:23:200
217585299128,3cyclictest2299888-21diskstats23:30:137
217585299122,6cyclictest0-21swapper/721:51:257
217585299121,10cyclictest1439-21snmpd21:27:517
217585299121,10cyclictest1439-21snmpd19:40:497
217585199128,3cyclictest2210819-21cat20:25:116
217585199125,1cyclictest52-21ksoftirqd/620:49:016
217585199124,5cyclictest141rcu_preempt19:48:016
217585199122,9cyclictest2200644-21pgrep20:03:296
217585199120,8cyclictest725-21systemd-journal20:10:006
217585199120,3cyclictest2317224-21/usr/sbin/munin00:05:226
217585099128,3cyclictest2294751-21cpuspeed_turbos23:20:115
217585099128,3cyclictest2231478-21unixbench_multi21:05:285
217585099124,5cyclictest0-21swapper/521:02:265
217585099123,6cyclictest2201988-21meminfo20:05:195
217585099120,11cyclictest2291080-21sed23:10:295
217584999127,3cyclictest40-21ksoftirqd/419:53:014
217584999126,4cyclictest40-21ksoftirqd/423:04:004
217584999126,4cyclictest40-21ksoftirqd/422:25:014
217584999122,9cyclictest2188760-21kworker/4:220:29:514
2175849991210,1cyclictest2188760-21kworker/4:220:36:574
2175849991210,1cyclictest0-21swapper/422:17:284
217584999120,3cyclictest2263308-21latency_hist22:15:024
217584899128,3cyclictest1439-21snmpd19:44:103
217584899121,2cyclictest0-21swapper/320:10:203
217584899120,3cyclictest2322644-21sshd00:15:463
217584899120,11cyclictest215724-21runrttasks20:28:543
217584799121,2cyclictest0-21swapper/220:41:302
217584799121,2cyclictest0-21swapper/219:20:162
217584699128,1cyclictest0-21swapper/122:05:541
217584699125,5cyclictest0-21swapper/120:17:341
217584699125,5cyclictest0-21swapper/120:17:341
217584699122,9cyclictest2261972-21cat22:10:201
217584699121,9cyclictest0-21swapper/121:59:001
217584699121,9cyclictest0-21swapper/100:30:011
217584699121,2cyclictest0-21swapper/121:45:241
217584699121,10cyclictest2205559-21latency_hist20:15:001
217584699120,11cyclictest2202725-21sed20:05:271
217584699120,10cyclictest2301391-21munin-run23:35:001
217584699120,10cyclictest2237285-21cpuspeed_turbos21:20:111
217584599128,2cyclictest0-21swapper/000:23:010
217584599121,6cyclictest0-21swapper/023:50:170
217584599121,10cyclictest1439-21snmpd20:30:590
217584599120,9cyclictest725-21systemd-journal00:31:080
217584599120,9cyclictest725-21systemd-journal00:31:080
217584599120,3cyclictest2320123-21/usr/sbin/munin00:10:320
217584599120,3cyclictest2301706-21latency_hist23:35:010
217584599120,3cyclictest2272445-21taskset22:32:390
217584599120,3cyclictest2205924-21/usr/sbin/munin20:15:120
217584599120,3cyclictest2205924-21/usr/sbin/munin20:15:120
217584599120,11cyclictest2236934-21cat21:20:000
217584599120,11cyclictest0-21swapper/021:05:200
21995322110,0sleep40-21swapper/420:00:204
217585299118,2cyclictest2197486-21smart_sdc19:55:267
217585299118,2cyclictest0-21swapper/723:20:117
217585299118,2cyclictest0-21swapper/723:05:147
217585299118,2cyclictest0-21swapper/723:05:147
217585299118,2cyclictest0-21swapper/721:20:057
217585299117,3cyclictest2331064-21diskstats00:35:177
217585299115,5cyclictest0-21swapper/723:38:347
217585299115,5cyclictest0-21swapper/700:12:597
217585299112,8cyclictest0-21swapper/719:20:217
217585299112,1cyclictest0-21swapper/721:55:517
217585299111,9cyclictest2292753-21grep23:15:177
217585299111,9cyclictest2182798-21meminfo19:25:197
217585299111,9cyclictest0-21swapper/722:00:147
217585299111,5cyclictest0-21swapper/722:21:447
217585299110,9cyclictest0-21swapper/722:10:117
217585299110,8cyclictest725-21systemd-journal21:10:007
217585299110,6cyclictest2177056-21df_inode19:15:127
217585299110,3cyclictest2317243-21/usr/sbin/munin00:05:227
217585299110,3cyclictest2192548-21ntpq19:45:217
217585299110,10cyclictest2323554-21cpuspeed_turbos00:20:117
217585299110,10cyclictest2271305-21irqstats22:30:177
217585299110,10cyclictest2268980-21latency22:25:197
217585299110,10cyclictest2244248-21/usr/sbin/munin21:35:137
217585299110,10cyclictest2186465-21latency_hist19:35:017
217585199118,2cyclictest2248483-21runrttasks21:42:416
217585199118,2cyclictest2248483-21runrttasks21:42:416
217585199118,2cyclictest2190934-21seq19:42:286
217585199118,2cyclictest0-21swapper/621:05:006
217585199118,2cyclictest0-21swapper/620:55:246
217585199118,2cyclictest0-21swapper/619:39:016
217585199118,2cyclictest0-21swapper/619:15:106
217585199117,3cyclictest2292517-21/usr/sbin/munin23:15:136
217585199117,3cyclictest2271230-21cat22:30:146
217585199117,1cyclictest919-17auditd23:10:466
217585199116,2cyclictest725-21systemd-journal20:21:276
217585199114,6cyclictest2224610-21kworker/6:222:13:466
217585199112,4cyclictest0-21swapper/623:05:186
217585199112,4cyclictest0-21swapper/623:05:186
217585199112,1cyclictest141rcu_preempt23:23:006
217585199111,9cyclictest0-21swapper/621:30:226
217585199111,9cyclictest0-21swapper/619:20:256
217585199111,2cyclictest0-21swapper/621:55:146
217585199111,2cyclictest0-21swapper/621:50:146
2175851991110,1cyclictest1773844-21kworker/6:219:12:266
2175851991110,1cyclictest0-21swapper/622:06:396
217585199110,9cyclictest0-21swapper/620:32:016
217585199110,2cyclictest2303825-21date23:40:006
217585199110,2cyclictest0-21swapper/621:25:166
217585199110,10cyclictest2307394-21missed_timers23:45:206
217585199110,10cyclictest2306296-21sed23:45:006
217585199110,10cyclictest2277735-21cat22:45:006
217585199110,10cyclictest215724-21runrttasks21:17:056
217585199110,10cyclictest0-21swapper/623:00:246
217585099119,1cyclictest0-21swapper/521:15:255
217585099118,2cyclictest2331239-21irqstats00:35:215
217585099118,2cyclictest2187933-21/usr/sbin/munin19:35:295
217585099117,3cyclictest2309612-21awk23:50:185
217585099114,5cyclictest2294410-21sed23:20:005
217585099114,5cyclictest0-21swapper/520:23:275
217585099111,9cyclictest0-21swapper/519:55:285
217585099111,8cyclictest0-21swapper/521:50:105
217585099111,1cyclictest141rcu_preempt23:42:015
217585099110,9cyclictest0-21swapper/521:55:145
217585099110,2cyclictest2308557-21sh23:50:015
217585099110,2cyclictest2277245-21runrttasks22:42:555
217585099110,2cyclictest2247571-21missed_timers21:40:205
217585099110,2cyclictest2247571-21missed_timers21:40:195
217585099110,2cyclictest2177247-21kernelversion19:15:185
217585099110,10cyclictest2318573-21/usr/sbin/munin00:10:245
217585099110,10cyclictest2302886-21ps23:35:265
217585099110,10cyclictest2264747-21find22:15:275
217585099110,10cyclictest2257019-21latency22:00:195
217585099110,10cyclictest2245252-21/usr/sbin/munin21:35:205
217585099110,10cyclictest2224693-21perf20:55:005
217585099110,10cyclictest2203876-21cat20:10:135
217585099110,10cyclictest0-21swapper/520:57:325
217584999118,2cyclictest2280739-21smartctl22:50:144
217584999118,2cyclictest2248284-21users21:40:284
217584999118,2cyclictest2248284-21users21:40:284
217584999118,2cyclictest0-21swapper/423:24:574
217584999118,2cyclictest0-21swapper/423:05:274
217584999118,2cyclictest0-21swapper/423:05:274
217584999118,2cyclictest0-21swapper/421:46:264
217584999118,2cyclictest0-21swapper/419:55:274
217584999117,3cyclictest2311208-21sed23:55:004
217584999117,3cyclictest2279710-21chrt22:48:574
217584999113,7cyclictest2304695-21/usr/sbin/munin23:40:174
217584999112,8cyclictest2222292-21kworker/4:021:17:584
217584999111,9cyclictest0-21swapper/422:40:144
217584999111,9cyclictest0-21swapper/400:30:244
217584999111,9cyclictest0-21swapper/400:30:234
217584999111,1cyclictest2242271-21expr21:30:114
217584999110,9cyclictest2271796-21smart_sdc22:30:244
217584999110,9cyclictest2176796-21tr19:15:114
217584999110,9cyclictest0-21swapper/420:30:174
217584999110,2cyclictest2321490-21diskstats00:15:144
217584999110,2cyclictest2318209-21munin-run00:09:594
217584999110,2cyclictest2206049-21/usr/sbin/munin20:15:104
217584999110,2cyclictest2206049-21/usr/sbin/munin20:15:104
217584999110,2cyclictest0-21swapper/420:10:254
217584999110,10cyclictest2299360-21/usr/sbin/munin23:30:274
217584999110,10cyclictest2256985-21grep22:00:194
217584999110,10cyclictest2218281-21cstates20:40:124
217584999110,10cyclictest2186724-21chrt19:35:114
217584899118,2cyclictest0-21swapper/323:10:213
217584899118,2cyclictest0-21swapper/322:25:213
217584899118,2cyclictest0-21swapper/322:10:003
217584899118,2cyclictest0-21swapper/321:30:003
217584899118,2cyclictest0-21swapper/319:55:233
217584899118,2cyclictest0-21swapper/319:20:313
217584899118,2cyclictest0-21swapper/300:30:243
217584899118,2cyclictest0-21swapper/300:30:233
217584899115,5cyclictest0-21swapper/323:35:523
217584899111,9cyclictest0-21swapper/323:55:113
217584899111,9cyclictest0-21swapper/319:30:113
217584899111,8cyclictest0-21swapper/322:40:023
217584899111,7cyclictest1212-21systemd-logind22:55:003
217584899110,2cyclictest2331588-21ntp_states00:35:273
217584899110,2cyclictest2317503-21sed00:05:263
217584899110,2cyclictest2244129-21latency_hist21:35:003
217584899110,2cyclictest2216395-21missed_timers20:35:233
217584899110,2cyclictest215724-21runrttasks21:38:153
217584899110,2cyclictest1212-21systemd-logind23:45:013
217584899110,2cyclictest0-21swapper/323:50:223
217584899110,2cyclictest0-21swapper/323:45:193
217584899110,2cyclictest0-21swapper/323:15:243
217584899110,2cyclictest0-21swapper/320:15:143
217584899110,2cyclictest0-21swapper/320:15:133
217584899110,10cyclictest2271684-21awk22:30:213
217584899110,10cyclictest2249729-21taskset21:45:163
217584799118,2cyclictest2187788-21ntp_states19:35:262
217584799118,2cyclictest1439-21snmpd23:30:332
217584799118,2cyclictest0-21swapper/223:50:122
217584799118,2cyclictest0-21swapper/223:15:142
217584799118,2cyclictest0-21swapper/222:55:162
217584799118,2cyclictest0-21swapper/222:00:152
217584799118,2cyclictest0-21swapper/221:47:282
217584799118,2cyclictest0-21swapper/221:40:192
217584799118,2cyclictest0-21swapper/221:40:192
217584799118,2cyclictest0-21swapper/220:25:302
217584799118,2cyclictest0-21swapper/200:15:112
217584799118,2cyclictest0-21swapper/200:10:002
217584799117,3cyclictest2262486-21cat22:10:272
217584799117,3cyclictest2208320-21/usr/sbin/munin20:20:232
217584799115,5cyclictest0-21swapper/223:03:122
217584799111,9cyclictest0-21swapper/223:35:292
217584799111,9cyclictest0-21swapper/223:23:002
217584799111,9cyclictest0-21swapper/222:30:132
217584799111,9cyclictest0-21swapper/220:10:002
217584799111,9cyclictest0-21swapper/200:35:322
217584799111,2cyclictest0-21swapper/222:25:192
217584799110,9cyclictest0-21swapper/220:10:242
217584799110,3cyclictest725-21systemd-journal21:30:092
217584799110,3cyclictest2305193-21ntp_states23:40:232
217584799110,2cyclictest0-21swapper/223:55:142
217584799110,2cyclictest0-21swapper/223:45:142
217584799110,2cyclictest0-21swapper/222:20:002
217584799110,2cyclictest0-21swapper/221:25:152
217584799110,2cyclictest0-21swapper/220:45:122
217584799110,1cyclictest725-21systemd-journal19:50:002
217584799110,10cyclictest2328373-21cpuspeed_turbos00:30:102
217584799110,10cyclictest2328373-21cpuspeed_turbos00:30:102
217584799110,10cyclictest2234458-21cut21:15:002
217584799110,10cyclictest2223158-21expr20:50:122
217584799110,10cyclictest2178411-21interrupts19:16:022
217584799110,10cyclictest0-21swapper/222:20:102
217584699118,2cyclictest2293048-21meminfo23:15:221
217584699118,2cyclictest2288933-21unixbench-2d23:05:291
217584699118,2cyclictest2288933-21unixbench-2d23:05:281
217584699118,2cyclictest0-21swapper/123:50:131
217584699118,2cyclictest0-21swapper/122:00:221
217584699118,2cyclictest0-21swapper/121:30:261
217584699118,2cyclictest0-21swapper/120:04:051
217584699117,3cyclictest2265752-21latency_hist22:20:011
217584699112,4cyclictest0-21swapper/122:47:061
217584699111,9cyclictest0-21swapper/122:43:441
217584699111,9cyclictest0-21swapper/122:23:091
217584699111,9cyclictest0-21swapper/119:15:121
217584699111,5cyclictest0-21swapper/120:38:111
217584699111,1cyclictest0-21swapper/121:35:121
217584699110,9cyclictest2331085-21fw_forwarded_lo00:35:191
217584699110,6cyclictest964-21NetworkManager19:53:541
217584699110,2cyclictest0-21swapper/120:25:211
217584699110,2cyclictest0-21swapper/100:05:131
217584699110,10cyclictest2268759-21forks22:25:161
217584699110,10cyclictest2223767-21pmu-power20:50:251
217584699110,10cyclictest0-21swapper/120:45:021
217584699110,10cyclictest0-21swapper/120:41:121
217584599118,2cyclictest2191508-21/usr/sbin/munin19:45:140
217584599118,2cyclictest2189137-21cat19:40:110
217584599118,2cyclictest0-21swapper/023:42:100
217584599118,2cyclictest0-21swapper/023:35:180
217584599118,2cyclictest0-21swapper/022:55:540
217584599118,2cyclictest0-21swapper/000:15:130
217584599118,2cyclictest0-21swapper/000:00:160
217584599111,9cyclictest0-21swapper/020:55:010
217584599111,9cyclictest0-21swapper/019:32:000
217584599110,2cyclictest2275391-21/usr/sbin/munin22:40:310
217584599110,2cyclictest0-21swapper/023:50:000
217584599110,2cyclictest0-21swapper/023:25:010
217584599110,2cyclictest0-21swapper/021:24:410
217584599110,1cyclictest0-21swapper/022:12:010
217584599110,10cyclictest2325734-21chrt00:25:020
217584599110,10cyclictest2266570-21expr22:20:170
217584599110,10cyclictest2225586-21/usr/sbin/munin20:55:160
217584599110,10cyclictest2197346-21ntp_states19:55:220
217584599110,10cyclictest2181940-21/usr/sbin/munin19:25:110
217584599110,10cyclictest2176535-21/usr/sbin/munin19:15:290
217585299108,1cyclictest2306994-21expr23:45:137
217585299108,1cyclictest0-21swapper/723:00:187
217585299108,1cyclictest0-21swapper/722:50:277
217585299108,1cyclictest0-21swapper/722:40:317
217585299108,1cyclictest0-21swapper/720:25:117
217585299108,1cyclictest0-21swapper/720:00:207
217585299108,1cyclictest0-21swapper/700:30:017
217585299108,1cyclictest0-21swapper/700:30:017
217585299105,5cyclictest0-21swapper/720:50:447
217585299103,1cyclictest2282554-21/usr/sbin/munin22:55:137
217585299102,3cyclictest0-21swapper/719:14:517
217585299101,2cyclictest2321179-21expr00:15:127
217585299100,9cyclictest2314902-21smart_sdc00:00:267
217585299100,9cyclictest2305819-21chrt23:40:457
217585299100,9cyclictest2259657-21/usr/sbin/munin22:05:207
217585299100,9cyclictest2243316-21/usr/sbin/munin21:30:267
217585299100,9cyclictest2232823-21diskstats21:10:167
217585299100,9cyclictest2228495-21ntp_states21:00:227
217585299100,9cyclictest2225054-21/usr/sbin/munin20:55:187
217585299100,9cyclictest2166350-21kworker/7:319:39:157
217585299100,9cyclictest0-21swapper/723:55:167
217585299100,9cyclictest0-21swapper/721:45:167
217585299100,9cyclictest0-21swapper/721:40:217
217585299100,9cyclictest0-21swapper/721:40:217
217585299100,9cyclictest0-21swapper/720:40:137
217585299100,3cyclictest2202704-21sed20:05:267
217585299100,2cyclictest0-21swapper/719:50:147
217585299100,1cyclictest2326285-21fw_conntrack00:25:147
217585299100,1cyclictest2309974-21ntp_states23:50:227
217585299100,1cyclictest2297695-21cut23:25:187
217585299100,1cyclictest2209166-21mdstat20:20:227
217585299100,1cyclictest0-21swapper/722:35:197
217585199108,2cyclictest2266346-21df_inode22:20:116
217585199108,2cyclictest2224610-21kworker/6:223:34:016
217585199108,1cyclictest2310279-21timerandwakeup23:50:266
217585199108,1cyclictest0-21swapper/622:55:116
217585199108,1cyclictest0-21swapper/600:15:206
217585199107,2cyclictest2264486-21ntp_offset22:15:226
217585199107,2cyclictest2234080-21ssh21:12:016
217585199107,2cyclictest0-21swapper/620:10:286
217585199104,3cyclictest919-17auditd00:28:176
217585199100,9cyclictest2319143-21hddtemp_smartct00:10:206
217585199100,9cyclictest2269912-21sshd22:25:566
217585199100,9cyclictest2230958-21ntp_states21:05:226
217585199100,9cyclictest2218407-21diskstats20:40:126
217585199100,9cyclictest2207906-21seq20:19:306
217585199100,9cyclictest2207906-21seq20:19:306
217585199100,9cyclictest2182799-21meminfo19:25:196
217585199100,9cyclictest0-21swapper/622:50:136
217585199100,9cyclictest0-21swapper/621:22:236
217585199100,9cyclictest0-21swapper/600:36:346
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional