You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-06-27 - 02:08

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #b, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackbslot1.osadl.org (updated Sun Jun 26, 2022 12:44:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
303192600,0sleep00-21swapper/007:19:080
2348025746,7sleep00-21swapper/007:06:110
2346925038,7sleep10-21swapper/107:06:041
165592500,1sleep216561-21idleruntime-cro08:45:002
2348924937,7sleep30-21swapper/307:06:173
2374424835,8sleep20-21swapper/207:08:462
249292450,0sleep20-21swapper/209:50:282
243422440,0sleep10-21swapper/112:30:341
24012991614,1cyclictest17929-21snmpd12:37:300
24012991614,1cyclictest17929-21snmpd12:09:490
24017991514,0cyclictest1681-21kworker/1:008:24:171
24017991513,1cyclictest3224-21kworker/1:011:07:331
24017991513,1cyclictest17929-21snmpd10:35:001
24029991413,0cyclictest17929-21snmpd09:30:253
24025991412,1cyclictest17929-21snmpd11:51:532
24025991412,1cyclictest17929-21snmpd07:15:152
24017991413,0cyclictest997-21kworker/1:107:44:351
24017991413,0cyclictest997-21kworker/1:107:44:351
24029991312,0cyclictest17929-21snmpd08:16:563
24029991311,1cyclictest17929-21snmpd10:54:373
24025991311,1cyclictest17929-21snmpd12:10:572
24017991312,0cyclictest19673-21kworker/1:210:01:511
24017991311,1cyclictest17778-31munin-graph11:25:361
24029991211,1cyclictest17929-21snmpd09:25:083
2402999120,11cyclictest0-21swapper/309:45:013
24017991211,0cyclictest5530-21kworker/1:010:27:311
24017991211,0cyclictest23790-21kworker/1:107:15:171
24017991211,0cyclictest17929-21snmpd11:58:291
24017991211,0cyclictest17929-21snmpd10:48:521
24017991210,1cyclictest884-21kworker/1:212:04:351
24017991210,1cyclictest29704-21kworker/1:111:02:011
24017991210,1cyclictest0-21swapper/112:20:391
24017991210,1cyclictest0-21swapper/111:30:171
2401799120,1cyclictest91rcu_preempt10:19:591
24012991210,1cyclictest17929-21snmpd11:36:410
24029991110,1cyclictest17929-21snmpd10:17:213
2402999110,10cyclictest0-21swapper/309:40:203
2402599110,10cyclictest0-21swapper/208:20:182
2402599110,10cyclictest0-21swapper/207:45:342
2402599110,10cyclictest0-21swapper/207:20:332
2401799119,1cyclictest31551-21kworker/1:209:21:591
2401799119,1cyclictest15972-21kworker/1:307:55:191
2401799119,1cyclictest0-21swapper/108:25:241
2401799116,1cyclictest24-21ksoftirqd/111:50:011
24017991111,0cyclictest21878-21kworker/1:112:35:471
24017991110,0cyclictest997-21kworker/1:107:34:211
24017991110,0cyclictest9871-21kworker/1:112:13:151
24017991110,0cyclictest4681-21kworker/1:308:36:241
24017991110,0cyclictest29172-31munin-graph10:50:431
24017991110,0cyclictest24791-21kworker/1:108:19:151
24017991110,0cyclictest23790-21kworker/1:107:13:121
24017991110,0cyclictest15432-21kworker/1:012:26:571
2401799110,10cyclictest0-21swapper/109:55:201
2401799110,10cyclictest0-21swapper/109:40:101
2401799110,10cyclictest0-21swapper/109:14:001
2401299119,1cyclictest13626-31munin-html09:30:320
2401299110,10cyclictest0-21swapper/010:33:010
2402999109,0cyclictest17929-21snmpd11:46:413
2402999108,1cyclictest0-21swapper/310:35:433
2402999100,0cyclictest0-21swapper/308:40:253
2402599108,1cyclictest0-21swapper/210:00:342
2402599108,1cyclictest0-21swapper/207:35:252
2402599100,9cyclictest0-21swapper/211:05:382
2402599100,9cyclictest0-21swapper/209:58:182
2401799109,1cyclictest31551-21kworker/1:209:19:451
2401799109,1cyclictest19673-21kworker/1:209:53:001
2401799109,0cyclictest24238-21kworker/1:011:42:031
2401799109,0cyclictest19673-21kworker/1:209:46:151
2401799108,1cyclictest4681-21kworker/1:308:31:231
2401799108,1cyclictest24791-21kworker/1:108:08:511
2401799108,1cyclictest24238-21kworker/1:011:47:351
2401799108,1cyclictest17929-21snmpd10:43:271
2401799108,1cyclictest0-21swapper/111:17:411
2401799108,1cyclictest0-21swapper/110:21:491
2401799108,1cyclictest0-21swapper/110:06:381
24017991010,0cyclictest997-21kworker/1:107:35:541
24017991010,0cyclictest884-21kworker/1:212:08:031
24017991010,0cyclictest19091-21kworker/1:009:07:571
24017991010,0cyclictest19091-21kworker/1:009:01:001
2401799100,1cyclictest0-21swapper/109:37:351
2401799100,1cyclictest0-21swapper/107:21:571
2401299109,0cyclictest0-21swapper/010:05:240
2401299108,1cyclictest0-21swapper/007:53:200
2401299100,9cyclictest0-21swapper/010:44:090
2401299100,9cyclictest0-21swapper/010:37:510
2401299100,1cyclictest0-21swapper/010:14:360
240259998,0cyclictest17929-21snmpd07:51:592
240259997,1cyclictest8817-31munin-graph11:10:382
240259997,1cyclictest33-21ksoftirqd/209:05:012
240259990,8cyclictest0-21swapper/211:58:452
240179999,0cyclictest231ktimersoftd/108:09:591
240179999,0cyclictest19091-21kworker/1:008:52:001
240179999,0cyclictest19091-21kworker/1:008:47:191
240179998,0cyclictest8833-21kworker/1:111:11:111
240179998,0cyclictest19091-21kworker/1:008:57:331
240179997,1cyclictest31551-21kworker/1:209:33:571
240179997,1cyclictest24-21ksoftirqd/107:50:011
240179997,1cyclictest0-21swapper/111:36:441
240179997,1cyclictest0-21swapper/109:25:171
240179997,1cyclictest0-21swapper/107:47:461
240179997,1cyclictest0-21swapper/107:27:161
240129998,0cyclictest9493-31munin-graph07:35:330
240129998,0cyclictest0-21swapper/011:15:090
240129997,1cyclictest0-21swapper/011:06:550
240129997,1cyclictest0-21swapper/010:49:190
240129997,1cyclictest0-21swapper/010:23:580
240129997,1cyclictest0-21swapper/008:48:130
240129997,1cyclictest0-21swapper/008:03:110
240129990,8cyclictest0-21swapper/007:44:080
240129990,8cyclictest0-21swapper/007:44:080
240129990,1cyclictest0-21swapper/008:05:550
240299987,0cyclictest0-21swapper/312:15:013
240299986,1cyclictest17929-21snmpd10:25:183
240299986,1cyclictest17929-21snmpd08:57:493
240299986,1cyclictest0-21swapper/311:03:003
240299986,1cyclictest0-21swapper/310:30:523
240299985,2cyclictest752-21dbus-daemon08:15:003
240299980,7cyclictest0-21swapper/308:36:343
240259987,0cyclictest17929-21snmpd08:36:572
240259987,0cyclictest17929-21snmpd08:26:052
240259986,1cyclictest33-21ksoftirqd/210:20:002
240259986,1cyclictest33-21ksoftirqd/209:10:012
240259986,1cyclictest17929-21snmpd09:31:172
240259986,1cyclictest0-21swapper/210:05:232
240259986,1cyclictest0-21swapper/208:55:302
240259982,3cyclictest752-21dbus-daemon07:10:012
240259980,7cyclictest0-21swapper/211:41:572
240179988,0cyclictest4681-21kworker/1:308:41:571
240179988,0cyclictest29704-21kworker/1:110:58:331
240179987,0cyclictest9292-21kworker/1:311:23:091
240179986,1cyclictest24-21ksoftirqd/112:15:001
240129986,1cyclictest17929-21snmpd08:26:420
240129986,1cyclictest0-21swapper/010:00:330
240129986,1cyclictest0-21swapper/009:09:130
240129986,1cyclictest0-21swapper/008:52:060
240129986,1cyclictest0-21swapper/008:37:140
240129980,1cyclictest0-21swapper/012:32:110
240129980,1cyclictest0-21swapper/008:33:220
240129980,1cyclictest0-21swapper/007:28:380
5858270,1sleep30-21swapper/307:30:173
28039270,1sleep30-21swapper/309:00:483
240299976,0cyclictest0-21swapper/311:50:183
240299975,1cyclictest17929-21snmpd12:21:183
240299975,1cyclictest17929-21snmpd12:10:133
240299975,1cyclictest17929-21snmpd08:27:513
240299975,1cyclictest0-21swapper/311:38:413
240299970,6cyclictest0-21swapper/310:59:463
240299970,6cyclictest0-21swapper/310:42:013
240299970,6cyclictest0-21swapper/310:09:143
240299970,6cyclictest0-21swapper/308:47:523
240299970,6cyclictest0-21swapper/307:54:053
240299970,6cyclictest0-21swapper/307:42:573
240299970,6cyclictest0-21swapper/307:42:563
240299970,1cyclictest0-21swapper/310:21:213
240259976,0cyclictest17929-21snmpd12:30:482
240259975,1cyclictest33-21ksoftirqd/211:35:172
240259975,1cyclictest33-21ksoftirqd/211:20:002
240259975,1cyclictest33-21ksoftirqd/209:19:592
240259975,1cyclictest0-21swapper/212:18:562
240259975,1cyclictest0-21swapper/210:28:492
240259975,1cyclictest0-21swapper/209:48:472
240259975,1cyclictest0-21swapper/209:27:182
240259975,1cyclictest0-21swapper/208:33:422
240259975,1cyclictest0-21swapper/208:10:462
240259975,1cyclictest0-21swapper/208:04:562
240259974,2cyclictest0-21swapper/210:20:322
240259970,6cyclictest0-21swapper/211:45:502
240259970,6cyclictest0-21swapper/211:29:122
240259970,6cyclictest0-21swapper/210:33:392
240259970,6cyclictest0-21swapper/209:00:012
240259970,1cyclictest101rcu_sched08:00:002
240179977,0cyclictest5530-21kworker/1:010:34:271
240179977,0cyclictest21500-21kworker/1:008:02:151
240179975,1cyclictest24-21ksoftirqd/110:15:001
240129976,0cyclictest0-21swapper/010:55:270
240129976,0cyclictest0-21swapper/009:44:250
240129975,1cyclictest7-21ksoftirqd/009:14:590
240129975,1cyclictest17929-21snmpd11:46:500
240129975,1cyclictest17929-21snmpd09:22:590
240129975,1cyclictest0-21swapper/012:15:520
240129975,1cyclictest0-21swapper/011:34:320
240129975,1cyclictest0-21swapper/011:28:040
240129975,1cyclictest0-21swapper/011:21:370
240129975,1cyclictest0-21swapper/007:22:000
240129975,1cyclictest0-21swapper/007:14:240
240129970,6cyclictest0-21swapper/011:51:480
240129970,6cyclictest0-21swapper/008:17:330
240129970,4cyclictest91rcu_preempt12:25:000
240129970,1cyclictest0-21swapper/010:16:520
240129970,1cyclictest0-21swapper/009:47:090
27806260,1sleep20-21swapper/208:08:172
26272260,1sleep30-21swapper/310:46:203
24803260,1sleep30-21swapper/308:03:093
240299966,0cyclictest17929-21snmpd07:23:283
240299965,0cyclictest17929-21snmpd11:15:253
240299964,1cyclictest87950irq/34-eno2-rx-08:20:303
240299964,1cyclictest6347-21sadc11:10:003
240299964,1cyclictest28575-31munin-html09:55:313
240299964,1cyclictest25209-21/usr/sbin/munin09:50:313
240299964,1cyclictest24594-21polkitd12:29:003
240299964,1cyclictest24594-21polkitd09:04:593
240299964,1cyclictest22087-21/usr/share/muni12:30:003
240299964,1cyclictest17929-21snmpd07:35:583
240299964,1cyclictest0-21swapper/312:08:233
240299964,1cyclictest0-21swapper/311:24:483
240299964,1cyclictest0-21swapper/310:11:493
240299964,1cyclictest0-21swapper/308:09:263
240299964,1cyclictest0-21swapper/307:19:513
240299963,2cyclictest752-21dbus-daemon07:12:003
240299963,2cyclictest4287-21ping09:15:213
240299963,2cyclictest0-21swapper/311:30:013
240259966,0cyclictest27741-21awk10:50:102
240259965,0cyclictest0-21swapper/208:45:192
240259964,1cyclictest33-21ksoftirqd/212:25:002
240259964,1cyclictest22413-21cat10:40:152
240259964,1cyclictest0-21swapper/211:17:252
240259964,1cyclictest0-21swapper/209:44:052
240259963,2cyclictest31707-21ntpq10:55:182
240259963,2cyclictest24596-21gdbus10:40:002
240259963,2cyclictest17929-21snmpd12:27:062
240259963,2cyclictest17929-21snmpd10:11:062
240259962,3cyclictest0-21swapper/209:35:312
240259961,1cyclictest91rcu_preempt12:00:122
240259961,1cyclictest25088-21/usr/share/muni12:35:002
240259960,5cyclictest453-21/usr/share/muni11:00:012
240259960,5cyclictest0-21swapper/211:34:122
240259960,3cyclictest26673-21munin-update10:50:002
240259960,1cyclictest91rcu_preempt07:40:012
240259960,1cyclictest91rcu_preempt07:40:002
240259960,1cyclictest851-31munin-html08:15:302
240259960,1cyclictest0-21swapper/209:22:272
240259960,1cyclictest0-21swapper/207:27:292
240129966,0cyclictest17929-21snmpd11:12:220
240129966,0cyclictest0-21swapper/009:04:410
240129965,1cyclictest7-21ksoftirqd/008:45:000
240129964,1cyclictest7-21ksoftirqd/012:25:000
240129964,1cyclictest7-21ksoftirqd/009:15:000
240129964,1cyclictest7-21ksoftirqd/008:25:000
240129964,1cyclictest17929-21snmpd11:01:580
240129964,1cyclictest16590-21users09:35:260
240129964,1cyclictest0-21swapper/009:26:490
240129963,2cyclictest752-21dbus-daemon07:59:590
240129963,2cyclictest24160-21awk09:50:100
240129963,2cyclictest17929-21snmpd10:51:280
240129963,2cyclictest17929-21snmpd08:59:490
240129963,2cyclictest0-21swapper/012:00:000
240129963,2cyclictest0-21swapper/008:10:000
240129960,3cyclictest91rcu_preempt10:25:320
240299953,1cyclictest777-21ntpd11:55:093
240299953,1cyclictest32139-21cat09:10:063
240299953,1cyclictest20078-21cat08:50:063
240299953,1cyclictest1996-21cat07:25:093
240299953,1cyclictest17929-21snmpd12:35:093
240299953,1cyclictest17929-21snmpd11:10:163
240299953,1cyclictest17929-21snmpd10:00:263
240299953,1cyclictest17929-21snmpd09:20:343
240299953,1cyclictest15900-21cat09:35:153
240299953,1cyclictest15457-31munin-html07:45:313
240299952,2cyclictest4110-21/usr/share/muni12:00:013
240299952,2cyclictest24502-21/usr/share/muni11:40:013
240299952,2cyclictest0-21swapper/307:55:163
240299950,4cyclictest15816-21users11:25:053
240299950,2cyclictest7713-21/usr/share/muni08:30:153
240259953,1cyclictest17929-21snmpd07:33:372
240259952,2cyclictest24596-21gdbus12:05:002
240259952,2cyclictest24596-21gdbus08:50:002
240129953,1cyclictest24594-21polkitd07:45:010
240129953,1cyclictest1303-21lldpd11:55:260
240129952,2cyclictest4773-21isc-worker000007:30:070
240129952,2cyclictest24502-21/usr/share/muni11:40:140
240129952,2cyclictest0-21swapper/012:10:080
240129951,3cyclictest26526-21vmstat09:55:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional