You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-17 - 02:25

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #b, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rackbslot1.osadl.org (updated Sun Jan 16, 2022 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1034625745,7sleep30-21swapper/307:07:263
1035525633,7sleep10-21swapper/107:07:321
1034325140,7sleep00-21swapper/007:07:230
1018024836,7sleep20-21swapper/207:06:012
189432450,0sleep30-21swapper/307:20:233
10788992727,0cyclictest10817-21kworker/2:212:18:112
157852190,0sleep30-21swapper/311:45:173
10795991913,5cyclictest26380-21awk11:10:153
10795991711,5cyclictest1350-21isc-worker000308:43:503
10788991717,0cyclictest31760-21kworker/2:207:45:452
10788991717,0cyclictest14124-21kworker/2:111:45:272
1079599160,11cyclictest23600-31munin-graph09:15:553
10788991514,0cyclictest23066-21snmpd08:20:132
10782991413,0cyclictest20477-21kworker/1:308:19:131
1079599137,1cyclictest0-21swapper/312:13:513
10788991311,1cyclictest23066-21snmpd07:29:312
10782991312,0cyclictest23066-21snmpd07:20:041
10782991311,1cyclictest23066-21snmpd07:54:351
10775991311,1cyclictest19602-31munin-html11:50:320
1079599126,0cyclictest23066-21snmpd09:13:453
10795991210,1cyclictest8188-21entropy10:40:133
10788991210,1cyclictest19871-31munin-graph08:15:512
1078899120,11cyclictest8912-21/usr/sbin/munin11:35:152
1078899120,11cyclictest4783-21ping_192.168.1710:35:022
1078899120,11cyclictest19262-31munin-html07:20:472
10782991212,0cyclictest9991-21kworker/1:309:59:451
10782991211,0cyclictest23066-21snmpd10:07:351
1078299120,1cyclictest14224-31munin-graph12:35:411
10775991210,1cyclictest23066-21snmpd10:53:540
10775991210,1cyclictest23066-21snmpd09:19:520
1079599119,1cyclictest0-21swapper/308:26:503
1079599115,1cyclictest23066-21snmpd08:10:583
1079599114,1cyclictest775-21polkitd07:29:003
1079599110,10cyclictest0-21swapper/312:30:003
1079599110,10cyclictest0-21swapper/312:05:573
1079599110,0cyclictest712-31munin-graph10:25:513
1078899119,1cyclictest0-21swapper/209:15:552
1078899110,10cyclictest0-21swapper/212:31:172
1078899110,10cyclictest0-21swapper/211:56:232
1078299119,1cyclictest28101-21kworker/1:010:41:401
1078299119,1cyclictest11090-21kworker/1:112:17:011
1078299119,1cyclictest0-21swapper/112:30:121
1078299119,1cyclictest0-21swapper/107:10:501
10782991110,0cyclictest32720-21kworker/1:208:56:071
10782991110,0cyclictest32720-21kworker/1:208:45:221
10782991110,0cyclictest28101-21kworker/1:010:36:291
10782991110,0cyclictest16001-21kworker/1:211:00:251
10782991110,0cyclictest11090-21kworker/1:111:40:261
1078299110,10cyclictest0-21swapper/110:25:041
1077599119,1cyclictest0-21swapper/012:09:160
1077599119,1cyclictest0-21swapper/011:05:000
1077599119,1cyclictest0-21swapper/007:50:160
10775991110,1cyclictest23066-21snmpd07:26:030
1077599110,1cyclictest0-21swapper/012:19:360
1077599110,10cyclictest0-21swapper/012:29:560
1077599110,10cyclictest0-21swapper/011:55:030
1077599110,10cyclictest0-21swapper/009:00:220
1079599108,1cyclictest23066-21snmpd11:58:353
1079599107,2cyclictest0-21swapper/309:04:373
1079599100,9cyclictest0-21swapper/310:38:463
1079599100,1cyclictest0-21swapper/309:25:183
1079599100,0cyclictest0-21swapper/308:15:243
1078899109,0cyclictest23066-21snmpd07:43:072
1078899108,1cyclictest9976-31munin-graph10:40:512
1078899108,1cyclictest0-21swapper/209:31:442
1078899108,1cyclictest0-21swapper/208:50:242
1078899100,9cyclictest0-21swapper/212:26:072
1078899100,9cyclictest0-21swapper/211:54:182
1078899100,9cyclictest0-21swapper/208:26:592
1078899100,1cyclictest0-21swapper/208:38:562
1078299109,1cyclictest28101-21kworker/1:010:31:161
1078299109,1cyclictest23580-21kworker/1:008:38:261
1078299109,1cyclictest18353-21kworker/1:209:13:261
1078299109,1cyclictest10433-21kworker/1:110:50:111
1078299109,0cyclictest27252-21kworker/1:009:30:271
1078299109,0cyclictest23066-21snmpd09:01:291
1078299109,0cyclictest20477-21kworker/1:308:21:061
1078299108,1cyclictest28413-21kworker/1:311:19:281
1078299108,1cyclictest10621-21kworker/1:207:18:521
1078299108,1cyclictest0-21swapper/112:05:491
1078299108,1cyclictest0-21swapper/111:11:341
1078299108,1cyclictest0-21swapper/110:18:551
10782991010,0cyclictest28413-21kworker/1:311:26:351
10782991010,0cyclictest11090-21kworker/1:111:56:031
1077599109,0cyclictest0-21swapper/007:30:540
1077599108,1cyclictest0-21swapper/011:10:200
1077599108,1cyclictest0-21swapper/010:43:500
1077599108,1cyclictest0-21swapper/008:08:020
1077599108,1cyclictest0-21swapper/007:15:340
1077599108,1cyclictest0-21swapper/007:14:070
1077599100,9cyclictest0-21swapper/012:24:460
1077599100,9cyclictest0-21swapper/012:24:460
1077599100,9cyclictest0-21swapper/011:19:030
1077599100,9cyclictest0-21swapper/010:06:340
1077599100,9cyclictest0-21swapper/008:40:300
1077599100,9cyclictest0-21swapper/008:19:500
1077599100,1cyclictest0-21swapper/010:16:530
107959997,1cyclictest0-21swapper/307:10:283
107959990,8cyclictest0-21swapper/312:27:163
107959990,8cyclictest0-21swapper/310:14:523
107959990,1cyclictest0-21swapper/311:04:553
107889997,1cyclictest0-21swapper/210:23:532
107889997,1cyclictest0-21swapper/209:09:472
107889997,1cyclictest0-21swapper/208:03:062
107889997,1cyclictest0-21swapper/207:30:382
107889990,8cyclictest0-21swapper/212:20:572
107889990,8cyclictest0-21swapper/212:20:562
107889990,1cyclictest0-21swapper/208:56:232
107889990,1cyclictest0-21swapper/208:32:002
107829999,0cyclictest8773-21kworker/1:108:12:161
107829999,0cyclictest32720-21kworker/1:208:52:091
107829999,0cyclictest321-21kworker/1:109:35:381
107829999,0cyclictest23066-21snmpd10:46:031
107829999,0cyclictest19679-21kworker/1:007:27:131
107829999,0cyclictest16001-21kworker/1:210:58:511
107829999,0cyclictest16001-21kworker/1:210:58:501
107829999,0cyclictest11090-21kworker/1:112:21:531
107829999,0cyclictest11090-21kworker/1:112:21:531
107829999,0cyclictest11090-21kworker/1:112:13:331
107829998,1cyclictest26663-21kworker/1:108:29:571
107829998,1cyclictest25645-21kworker/1:307:41:141
107829998,0cyclictest28101-21kworker/1:010:24:211
107829998,0cyclictest23066-21snmpd09:29:011
107829998,0cyclictest18353-21kworker/1:209:20:131
107829998,0cyclictest11090-21kworker/1:112:03:091
107829997,1cyclictest9565-21kworker/1:009:49:311
107829997,1cyclictest2468-21pluginstate08:40:461
107829997,1cyclictest0-21swapper/107:55:231
107829990,1cyclictest0-21swapper/109:09:501
107759999,0cyclictest23066-21snmpd09:37:450
107759998,0cyclictest0-21swapper/012:35:060
107759997,1cyclictest0-21swapper/011:03:130
107759997,1cyclictest0-21swapper/010:56:270
107759997,1cyclictest0-21swapper/010:56:270
107759997,1cyclictest0-21swapper/007:23:580
107759990,5cyclictest21295-21idleruntime-cro09:15:000
107759990,1cyclictest0-21swapper/010:31:350
107759990,1cyclictest0-21swapper/009:53:180
107759990,1cyclictest0-21swapper/007:36:240
107959987,1cyclictest23066-21snmpd07:18:553
107959987,0cyclictest0-21swapper/307:35:013
107959986,1cyclictest23066-21snmpd10:56:393
107959986,1cyclictest23066-21snmpd10:56:393
107959986,1cyclictest0-21swapper/311:16:523
107959986,1cyclictest0-21swapper/309:32:053
107889986,1cyclictest0-21swapper/211:40:062
107889986,1cyclictest0-21swapper/211:25:342
107889986,1cyclictest0-21swapper/210:07:052
107889986,1cyclictest0-21swapper/209:41:252
107889986,1cyclictest0-21swapper/209:23:502
107889986,1cyclictest0-21swapper/208:06:392
107889980,7cyclictest0-21swapper/210:48:262
107889980,7cyclictest0-21swapper/210:28:342
107889980,1cyclictest0-21swapper/211:03:072
107889980,1cyclictest0-21swapper/210:58:072
107889980,1cyclictest0-21swapper/210:58:062
107889980,1cyclictest0-21swapper/210:02:352
107829988,0cyclictest8773-21kworker/1:108:00:391
107829988,0cyclictest28413-21kworker/1:311:21:131
107829988,0cyclictest25645-21kworker/1:307:39:311
107829988,0cyclictest25118-21kworker/1:011:07:411
107829988,0cyclictest23580-21kworker/1:008:31:511
107829988,0cyclictest23066-21snmpd07:48:201
107829988,0cyclictest15861-21kworker/1:110:12:231
107829988,0cyclictest15861-21kworker/1:110:01:391
107829988,0cyclictest11090-21kworker/1:111:47:231
107829987,0cyclictest9991-21kworker/1:309:53:081
107829987,0cyclictest23066-21snmpd12:25:391
107829986,1cyclictest24-21ksoftirqd/109:15:011
107759987,1cyclictest23066-21snmpd10:36:570
107759986,1cyclictest23066-21snmpd09:05:300
107759986,1cyclictest16137-21/usr/share/muni09:59:590
107759986,1cyclictest0-21swapper/012:04:060
107759986,1cyclictest0-21swapper/011:25:590
107759986,1cyclictest0-21swapper/010:14:090
107759986,1cyclictest0-21swapper/009:31:120
107759986,1cyclictest0-21swapper/008:32:540
107759980,1cyclictest0-21swapper/008:23:320
3903270,1sleep33612-31munin-html10:30:503
107959975,1cyclictest23066-21snmpd07:54:023
107959975,1cyclictest11428-31munin-html08:55:453
107959975,1cyclictest0-21swapper/309:50:593
107959975,1cyclictest0-21swapper/309:21:353
107959975,1cyclictest0-21swapper/308:32:113
107959975,1cyclictest0-21swapper/307:46:093
107959970,6cyclictest0-21swapper/312:01:073
107959970,6cyclictest0-21swapper/310:06:573
107959970,6cyclictest0-21swapper/307:42:183
107889977,0cyclictest23066-21snmpd10:11:562
107889976,1cyclictest0-21swapper/210:17:262
107889976,0cyclictest23066-21snmpd08:42:542
107889975,1cyclictest8012-31munin-graph07:55:592
107889975,1cyclictest33-21ksoftirqd/208:49:592
107889975,1cyclictest13193-31munin-graph07:10:572
107889975,1cyclictest0-21swapper/212:10:372
107889974,2cyclictest15608-21ntp_states07:15:162
107889970,1cyclictest0-21swapper/211:32:492
107829977,0cyclictest11090-21kworker/1:111:50:511
107829976,1cyclictest24-21ksoftirqd/107:30:001
107829976,1cyclictest23066-21snmpd11:31:561
107829974,2cyclictest8773-21kworker/1:108:05:221
107759976,1cyclictest23066-21snmpd12:13:300
107759975,1cyclictest31954-21/usr/sbin/munin08:35:480
107759975,1cyclictest23066-21snmpd12:34:190
107759975,1cyclictest23066-21snmpd08:14:110
107759975,1cyclictest11945-21ntp_states10:45:190
107759975,1cyclictest0-21swapper/010:28:210
107759975,1cyclictest0-21swapper/009:24:240
107759975,1cyclictest0-21swapper/008:53:150
107759975,1cyclictest0-21swapper/007:55:550
107959965,1cyclictest23066-21snmpd11:25:283
107959965,1cyclictest23066-21snmpd10:51:453
107959964,1cyclictest830-21gdbus07:34:593
107959964,1cyclictest23066-21snmpd11:42:203
107959964,1cyclictest11620-21awk10:45:163
107959964,1cyclictest0-21swapper/312:21:473
107959964,1cyclictest0-21swapper/312:21:473
107959964,1cyclictest0-21swapper/311:51:453
107959964,1cyclictest0-21swapper/311:34:093
107959964,1cyclictest0-21swapper/311:21:433
107959964,1cyclictest0-21swapper/308:03:553
107959963,2cyclictest23066-21snmpd09:35:013
107959963,2cyclictest23066-21snmpd08:22:413
107959962,1cyclictest91rcu_preempt10:20:003
107959960,5cyclictest0-21swapper/310:17:373
107959960,1cyclictest0-21swapper/310:02:073
107889965,0cyclictest23066-21snmpd10:50:272
107889964,1cyclictest33-21ksoftirqd/209:30:002
107889964,1cyclictest0-21swapper/209:53:522
107889964,1cyclictest0-21swapper/209:11:422
107889964,1cyclictest0-21swapper/209:04:552
107889964,1cyclictest0-21swapper/207:37:352
107889963,2cyclictest752-21dbus-daemon09:55:502
107889962,3cyclictest0-21swapper/212:00:432
107889960,2cyclictest0-21swapper/211:20:012
107829966,0cyclictest10710-21kworker/1:011:39:031
107829965,1cyclictest23066-21snmpd09:41:071
107759966,0cyclictest23066-21snmpd09:42:310
107759965,0cyclictest23066-21snmpd08:00:140
107759965,0cyclictest0-21swapper/011:30:500
107759964,1cyclictest0-21swapper/011:43:160
107759964,1cyclictest0-21swapper/010:23:400
107759964,1cyclictest0-21swapper/008:27:350
107759963,2cyclictest4094-21cat08:45:140
107759963,2cyclictest13961-21awk09:55:190
107959953,1cyclictest6325-21/usr/sbin/munin08:50:133
107959953,1cyclictest23614-21rndc11:05:153
107959953,1cyclictest23066-21snmpd09:55:193
107959953,1cyclictest23066-21snmpd08:35:143
107959952,2cyclictest8301-21/usr/sbin/munin11:35:133
107959952,2cyclictest6878-21/usr/share/muni09:45:053
107959952,2cyclictest5981-21/usr/share/muni07:55:123
107959952,2cyclictest3561-21/usr/share/muni08:45:043
107959952,2cyclictest32382-21/usr/share/muni12:15:053
107959952,2cyclictest11737-21/usr/share/muni12:35:053
107959952,2cyclictest0-21swapper/309:40:063
107959951,3cyclictest23066-21snmpd08:05:473
107959950,1cyclictest15331-21munin-update09:05:013
107889953,1cyclictest3601-21munin-limits10:30:442
107889953,1cyclictest29437-21cat11:15:152
107889953,1cyclictest26322-21cat11:10:152
107889953,1cyclictest23066-21snmpd12:05:092
107889953,1cyclictest23066-21snmpd11:05:092
107889953,1cyclictest23066-21snmpd08:10:072
107889953,1cyclictest0-21swapper/207:50:482
107889952,2cyclictest8089-21cat09:45:182
107889952,2cyclictest0-21swapper/212:35:042
107889952,2cyclictest0-21swapper/209:35:012
107759953,1cyclictest9135-21cat11:35:160
107759953,1cyclictest7988-21cat09:45:180
107759953,1cyclictest626-21cat07:45:190
107759953,1cyclictest31673-21/usr/share/muni11:20:010
107759953,1cyclictest0-21swapper/007:40:040
107759952,2cyclictest28349-21tr09:25:200
107759952,2cyclictest14389-21/usr/share/muni11:45:060
107759952,2cyclictest0-21swapper/008:55:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional