You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-04 - 04:04

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #b, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rackbslot1.osadl.org (updated Thu Aug 04, 2022 12:44:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2352599407405,1cyclictest20703-31munin-html07:55:311
2352599406401,4cyclictest9561-31munin-html08:30:301
23525993960,1cyclictest30874-31munin-html10:00:321
2353799395393,1cyclictest16132-31munin-html09:35:323
2353799393387,5cyclictest11981-31munin-html07:40:333
2353099392390,1cyclictest16122-31munin-html09:35:322
2353099389383,5cyclictest11985-31munin-html07:40:322
2353799383378,4cyclictest6589-31munin-html08:25:333
2352599381379,1cyclictest0-21swapper/107:40:461
2353099380375,4cyclictest6581-31munin-html08:25:332
2352599380378,1cyclictest0-21swapper/108:26:101
2353799379377,1cyclictest0-21swapper/308:13:013
2351999378376,1cyclictest0-21swapper/008:07:350
2353799377375,1cyclictest0-21swapper/310:36:243
2353099376374,1cyclictest0-21swapper/208:13:012
2353099374370,3cyclictest19551-31munin-html10:35:332
2351999374372,1cyclictest0-21swapper/009:51:310
2351999368366,1cyclictest15535-31munin-html08:40:310
2351999365363,1cyclictest27518-31munin-html09:00:310
2353799364363,0cyclictest20943-31munin-html07:55:313
2353799364359,4cyclictest9575-31munin-html08:30:303
2352599364362,1cyclictest0-21swapper/108:07:351
2353099362360,1cyclictest20715-31munin-html07:55:312
2353099360355,4cyclictest9566-31munin-html08:30:312
2352599360358,1cyclictest0-21swapper/109:51:311
2353799359357,1cyclictest31087-31munin-html10:00:333
2351999357355,1cyclictest0-21swapper/009:56:280
2353099355353,1cyclictest31093-31munin-html10:00:332
2353799354353,0cyclictest0-21swapper/308:41:283
2352599354352,1cyclictest15460-31munin-html08:40:321
2353799353351,1cyclictest0-21swapper/309:16:113
2351999352350,1cyclictest0-21swapper/010:11:590
2351999352350,1cyclictest0-21swapper/008:22:020
2353099351349,1cyclictest0-21swapper/208:41:282
2352599351349,1cyclictest27397-31munin-html09:00:311
2351999351349,1cyclictest0-21swapper/008:16:180
2353099350348,1cyclictest0-21swapper/209:16:112
2352599343341,1cyclictest0-21swapper/109:56:281
2352599339337,1cyclictest0-21swapper/108:22:021
2352599338336,1cyclictest0-21swapper/110:12:001
2352599338336,1cyclictest0-21swapper/108:16:191
2351999336334,1cyclictest0-21swapper/008:10:290
2351999334331,2cyclictest18856-31munin-html09:40:310
2351999329327,1cyclictest0-21swapper/009:15:300
2352599326321,4cyclictest29974-31munin-html08:10:291
2353799322320,1cyclictest0-21swapper/308:07:353
2352599320317,2cyclictest18752-31munin-html09:40:311
2353099319317,1cyclictest0-21swapper/208:07:352
2353799318316,1cyclictest0-21swapper/309:51:313
2352599318313,4cyclictest4144-31munin-html09:15:301
2353099315313,1cyclictest0-21swapper/209:51:312
2351999311309,1cyclictest0-21swapper/009:36:320
2353799309307,1cyclictest27515-31munin-html09:00:313
2353099306304,1cyclictest27500-31munin-html09:00:312
2351999305303,1cyclictest9971-31munin-html09:25:300
2353799301299,1cyclictest0-21swapper/309:56:273
2351999301299,1cyclictest0-21swapper/010:06:410
2353099298296,1cyclictest0-21swapper/209:56:272
2353799297295,1cyclictest0-21swapper/308:22:033
2353799296294,1cyclictest0-21swapper/310:11:593
2353799296294,1cyclictest0-21swapper/308:16:183
2352599296295,0cyclictest0-21swapper/109:36:321
2353099292291,0cyclictest0-21swapper/208:22:022
2353099292290,1cyclictest0-21swapper/208:16:182
2353099292288,3cyclictest4759-31munin-html10:10:332
2352599291289,1cyclictest10158-31munin-html09:25:301
2352599288286,1cyclictest0-21swapper/110:06:401
2353799279275,3cyclictest18765-31munin-html09:40:313
2353099274272,1cyclictest18741-31munin-html09:40:312
2351999274272,1cyclictest0-21swapper/009:07:440
2351999274272,1cyclictest0-21swapper/008:31:370
2352599261259,1cyclictest0-21swapper/109:07:451
2353799250248,1cyclictest10157-31munin-html09:25:303
2351999250248,1cyclictest0-21swapper/010:01:450
2351999248242,5cyclictest11991-31munin-html07:40:320
2353799246244,1cyclictest0-21swapper/310:06:403
2353099246244,1cyclictest10131-31munin-html09:25:302
2353099242240,1cyclictest0-21swapper/210:06:402
2351999235233,1cyclictest0-21swapper/008:25:340
2351999232230,1cyclictest0-21swapper/010:36:250
2351999220218,1cyclictest20941-31munin-html07:55:310
2353799219217,1cyclictest0-21swapper/309:07:443
2352599219215,3cyclictest19705-31munin-html10:35:331
2353099215213,1cyclictest0-21swapper/209:07:442
2336426037,7sleep00-21swapper/007:09:390
96882590,1sleep39687-21df10:20:113
2300425733,7sleep10-21swapper/107:06:191
2259225732,11sleep30-21swapper/307:05:253
20282560,0sleep10-21swapper/107:25:131
2317325139,7sleep20-21swapper/207:07:572
70092270,0sleep00-21swapper/007:35:050
23530991513,1cyclictest17929-21snmpd12:20:432
23525991513,1cyclictest17929-21snmpd11:55:261
23537991412,1cyclictest17929-21snmpd11:20:103
2353799130,11cyclictest29452-21irqstats10:55:003
2353099130,1cyclictest20281-31munin-html11:30:332
23525991311,1cyclictest29428-31munin-graph07:15:391
23530991210,1cyclictest3737-21isc-worker000111:25:482
23530991210,1cyclictest14822-21munin-limits07:45:282
23525991211,0cyclictest17929-21snmpd08:51:101
23525991210,1cyclictest3094-21/usr/share/muni11:05:051
23525991210,1cyclictest11875-31munin-html12:10:281
2352599120,1cyclictest26289-31munin-graph11:40:371
2352599120,11cyclictest0-21swapper/111:18:011
23519991210,1cyclictest0-21swapper/012:25:430
2351999120,1cyclictest2848-31munin-html11:55:300
2353799119,1cyclictest0-21swapper/309:45:153
2353799119,1cyclictest0-21swapper/309:11:033
23537991110,0cyclictest17929-21snmpd07:35:533
2353799110,1cyclictest0-21swapper/312:30:463
2353799110,1cyclictest0-21swapper/307:10:253
2353799110,10cyclictest0-21swapper/312:05:063
2353799110,10cyclictest0-21swapper/311:30:233
23530991110,0cyclictest17929-21snmpd11:01:342
2353099110,1cyclictest0-21swapper/208:55:432
2353099110,0cyclictest0-21swapper/211:55:052
2352599119,1cyclictest16707-31munin-html10:30:321
2352599110,10cyclictest0-21swapper/111:00:541
2351999119,1cyclictest0-21swapper/009:30:220
2351999119,1cyclictest0-21swapper/008:03:010
239442100,1sleep30-21swapper/310:45:063
2353799108,1cyclictest17929-21snmpd07:53:533
2353799108,1cyclictest0-21swapper/311:37:183
2353799108,1cyclictest0-21swapper/311:12:563
2353799100,9cyclictest0-21swapper/312:21:243
2353799100,1cyclictest0-21swapper/310:31:453
2353799100,1cyclictest0-21swapper/308:50:223
2353799100,1cyclictest0-21swapper/308:37:173
2352599108,1cyclictest0-21swapper/107:48:541
2352599100,9cyclictest0-21swapper/112:22:151
2352599100,9cyclictest0-21swapper/111:31:141
2352599100,1cyclictest0-21swapper/110:29:041
2352599100,1cyclictest0-21swapper/108:35:051
2352599100,1cyclictest0-21swapper/108:04:531
2351999109,0cyclictest0-21swapper/007:20:520
2351999108,1cyclictest0-21swapper/012:13:550
2351999108,1cyclictest0-21swapper/012:00:030
2351999108,1cyclictest0-21swapper/011:25:200
2351999108,1cyclictest0-21swapper/007:46:030
2351999100,1cyclictest0-21swapper/011:46:080
2351999100,1cyclictest0-21swapper/008:55:480
2351999100,0cyclictest0-21swapper/007:50:050
235379997,1cyclictest2865-31munin-graph11:55:443
235379997,1cyclictest0-21swapper/311:51:123
235379990,1cyclictest0-21swapper/310:42:553
235309997,1cyclictest17929-21snmpd11:16:292
235259998,0cyclictest0-21swapper/111:45:071
235259998,0cyclictest0-21swapper/111:20:061
235259998,0cyclictest0-21swapper/107:20:201
235259997,1cyclictest0-21swapper/111:35:451
235259997,1cyclictest0-21swapper/107:37:461
235259990,8cyclictest0-21swapper/109:34:121
235259990,1cyclictest0-21swapper/110:59:061
235199998,0cyclictest7-21ksoftirqd/008:55:010
235199997,1cyclictest0-21swapper/012:35:230
235199997,1cyclictest0-21swapper/007:34:160
235199997,1cyclictest0-21swapper/007:29:540
235199990,8cyclictest0-21swapper/012:32:580
6843280,0sleep20-21swapper/207:35:032
235379988,0cyclictest17929-21snmpd08:01:323
235379987,0cyclictest0-21swapper/309:33:383
235379986,1cyclictest0-21swapper/312:35:373
235379986,1cyclictest0-21swapper/311:44:163
235379986,1cyclictest0-21swapper/308:58:363
235379986,1cyclictest0-21swapper/307:19:283
235379980,7cyclictest0-21swapper/311:25:513
235379980,7cyclictest0-21swapper/311:04:323
235379980,1cyclictest0-21swapper/308:47:373
235259986,1cyclictest24-21ksoftirqd/108:45:011
235259986,1cyclictest17929-21snmpd10:48:131
235259986,1cyclictest0-21swapper/107:52:371
235259980,1cyclictest0-21swapper/109:47:551
235259980,1cyclictest0-21swapper/109:13:211
235259980,1cyclictest0-21swapper/108:59:081
235199986,1cyclictest0-21swapper/011:53:060
235199986,1cyclictest0-21swapper/011:39:130
235199980,1cyclictest0-21swapper/007:16:110
235199980,1cyclictest0-21swapper/007:11:300
23539270,1sleep20-21swapper/207:10:012
235379976,0cyclictest17929-21snmpd09:20:223
235379975,1cyclictest27831-21awk11:45:103
235379975,1cyclictest17929-21snmpd12:02:123
235379975,1cyclictest0-21swapper/312:25:543
235379975,1cyclictest0-21swapper/311:15:033
235379975,1cyclictest0-21swapper/307:48:413
235379975,1cyclictest0-21swapper/307:32:523
235379970,6cyclictest0-21swapper/311:08:453
235309976,0cyclictest0-21swapper/211:40:142
235309975,1cyclictest0-21swapper/212:09:372
235309975,1cyclictest0-21swapper/210:53:152
235309975,1cyclictest0-21swapper/208:52:382
235309975,1cyclictest0-21swapper/207:53:512
235309975,1cyclictest0-21swapper/207:17:512
235309974,2cyclictest24596-21gdbus10:44:592
235309970,6cyclictest0-21swapper/212:32:232
235309970,6cyclictest0-21swapper/212:27:122
235309970,6cyclictest0-21swapper/212:12:022
235309970,6cyclictest0-21swapper/211:35:232
235309970,6cyclictest0-21swapper/210:34:212
235309970,6cyclictest0-21swapper/209:11:502
235309970,6cyclictest0-21swapper/208:46:492
235309970,6cyclictest0-21swapper/208:38:352
235259976,1cyclictest20101-21kworker/1:010:43:121
235259976,1cyclictest17929-21snmpd10:54:581
235259976,0cyclictest4464-21kworker/1:309:20:221
235259975,1cyclictest0-21swapper/112:39:121
235259975,1cyclictest0-21swapper/112:03:331
235259970,1cyclictest0-21swapper/111:52:051
235199976,1cyclictest17929-21snmpd10:41:120
235199975,1cyclictest7-21ksoftirqd/011:25:000
235199975,1cyclictest7-21ksoftirqd/011:05:000
235199975,1cyclictest7-21ksoftirqd/010:25:010
235199975,1cyclictest17929-21snmpd12:06:540
235199975,1cyclictest11494-21/usr/share/muni10:25:010
235199975,1cyclictest0-21swapper/012:20:510
235199975,1cyclictest0-21swapper/009:20:500
235199975,1cyclictest0-21swapper/008:47:150
235199972,4cyclictest29224-21perf10:55:000
235199970,6cyclictest0-21swapper/008:39:200
235199970,1cyclictest0-21swapper/010:57:420
8688260,1sleep20-21swapper/211:13:422
6034260,1sleep20-21swapper/212:01:232
235379965,1cyclictest17929-21snmpd07:29:413
235379964,1cyclictest17929-21snmpd12:13:123
235379963,2cyclictest752-21dbus-daemon10:54:053
235379963,2cyclictest7055-21awk10:15:163
235309965,1cyclictest17929-21snmpd10:45:062
235309965,1cyclictest17929-21snmpd07:31:532
235309965,0cyclictest17929-21snmpd08:02:232
235309964,1cyclictest24596-21gdbus11:55:002
235309964,1cyclictest0-21swapper/212:19:382
235309963,2cyclictest752-21dbus-daemon10:57:142
235309963,2cyclictest5514-21/usr/share/muni10:15:072
235309963,2cyclictest17929-21snmpd09:24:172
235309963,2cyclictest1303-21lldpd10:20:312
235309963,2cyclictest1266-21isc-worker000007:25:082
235309963,2cyclictest11875-21grep10:25:072
235309960,5cyclictest0-21swapper/207:22:032
235259964,1cyclictest24594-21polkitd12:31:001
235259964,1cyclictest0-21swapper/112:17:261
235259964,1cyclictest0-21swapper/111:11:221
235259963,2cyclictest4466-21cpu07:30:091
235259963,2cyclictest24518-21awk07:10:101
235199966,0cyclictest17929-21snmpd09:11:430
235199964,1cyclictest752-21dbus-daemon09:49:000
235199964,1cyclictest17929-21snmpd11:11:110
235199964,1cyclictest0-21swapper/011:18:420
235199963,2cyclictest20000-21ping11:30:240
235199963,2cyclictest17929-21snmpd11:41:110
235199963,2cyclictest17929-21snmpd10:18:240
235199963,2cyclictest1303-21lldpd10:30:110
235199962,1cyclictest0-21swapper/010:45:060
235379953,1cyclictest777-21ntpd10:25:093
235379953,1cyclictest31700-21cat07:20:143
235379950,2cyclictest12886-21sensors_volt12:15:043
235309953,1cyclictest3113-21open_inodes11:05:012
235309953,1cyclictest27096-21ntpq11:45:032
235309953,1cyclictest19873-21/usr/sbin/munin09:45:102
235309953,1cyclictest17929-21snmpd12:35:492
235309953,1cyclictest13598-21cat11:20:152
235309952,2cyclictest12228-21expr09:30:132
235259953,1cyclictest6761-21ls10:15:131
235259953,1cyclictest24594-21polkitd12:05:001
235259952,2cyclictest8508-21/usr/share/muni10:20:091
235259952,2cyclictest18623-21/usr/share/muni12:25:141
235199953,1cyclictest14122-21cat12:15:140
235199950,1cyclictest32425-21/usr/share/muni11:00:010
15110250,1sleep10-21swapper/111:25:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional