You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2022-01-23 - 19:14

Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Linux 4.16.15-rt7 (Profile)

Latency plot of system in rack #b, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of all highest latencies:
System rackbslot1.osadl.org (updated Sun Jan 23, 2022 12:44:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
67512690,0sleep10-21swapper/107:50:081
1347325535,15sleep00-21swapper/007:07:400
1252125131,7sleep10-21swapper/107:05:241
1327625039,7sleep30-21swapper/307:05:533
1327524736,7sleep20-21swapper/207:05:532
84382440,0sleep30-21swapper/307:51:553
185112180,0sleep00-21swapper/008:10:060
13872991614,1cyclictest23066-21snmpd09:52:371
1387899159,5cyclictest1349-21isc-worker000209:13:402
1387899150,1cyclictest0-21swapper/209:50:332
1387899140,13cyclictest0-21swapper/210:05:052
13872991413,0cyclictest2610-21kworker/1:012:15:291
13872991413,0cyclictest231ktimersoftd/107:30:221
13872991413,0cyclictest23066-21snmpd08:32:311
13885991313,0cyclictest23066-21snmpd10:05:043
1387899137,1cyclictest23066-21snmpd10:41:082
1387899137,1cyclictest0-21swapper/210:31:342
13878991311,1cyclictest23066-21snmpd12:19:292
1387899130,8cyclictest0-21swapper/209:36:302
1387899130,8cyclictest0-21swapper/208:08:102
13872991313,0cyclictest23066-21snmpd08:50:471
13872991312,1cyclictest23066-21snmpd11:26:491
13872991312,0cyclictest23066-21snmpd07:21:391
13885991211,0cyclictest23066-21snmpd07:10:273
1388599120,1cyclictest32043-31munin-graph08:30:473
1388599120,11cyclictest0-21swapper/310:20:363
1387899126,1cyclictest0-21swapper/211:20:482
13878991210,1cyclictest0-21swapper/208:35:192
13872991212,0cyclictest5688-21kworker/1:212:25:531
13872991211,1cyclictest5120-21kworker/1:111:55:111
13872991211,1cyclictest21052-21kworker/1:008:15:021
13872991211,0cyclictest32741-21kworker/1:008:35:411
13872991211,0cyclictest17896-21kworker/1:009:06:321
13872991210,1cyclictest23066-21snmpd10:26:201
13867991210,1cyclictest23066-21snmpd08:21:280
13867991210,1cyclictest23066-21snmpd07:35:020
1386799120,11cyclictest0-21swapper/010:55:210
1388599119,1cyclictest32436-21/usr/share/muni07:40:153
13885991111,0cyclictest23066-21snmpd10:13:193
1388599110,1cyclictest0-21swapper/311:05:193
1388599110,10cyclictest0-21swapper/312:15:143
1388599110,10cyclictest0-21swapper/308:39:323
1387899119,1cyclictest3838-31munin-graph10:25:352
1387899119,1cyclictest14926-21/usr/share/muni12:35:052
1387899119,1cyclictest0-21swapper/212:00:412
1387899119,1cyclictest0-21swapper/208:56:072
1387899119,1cyclictest0-21swapper/207:10:422
13878991110,0cyclictest23066-21snmpd11:51:112
13878991110,0cyclictest23066-21snmpd09:09:262
1387899110,10cyclictest0-21swapper/212:05:032
1387899110,10cyclictest0-21swapper/211:45:212
1387899110,10cyclictest0-21swapper/211:45:212
1387299119,1cyclictest5120-21kworker/1:111:35:371
1387299119,1cyclictest21052-21kworker/1:008:20:151
1387299119,1cyclictest13902-21kworker/1:107:10:441
1387299119,1cyclictest0-21swapper/111:45:361
1387299119,1cyclictest0-21swapper/111:45:361
1387299119,1cyclictest0-21swapper/110:05:011
1387299119,1cyclictest0-21swapper/110:00:201
13872991110,1cyclictest5688-21kworker/1:212:20:411
13872991110,1cyclictest5120-21kworker/1:112:05:151
13872991110,0cyclictest28414-21kworker/1:011:16:431
13872991110,0cyclictest0-21swapper/110:55:331
1387299110,10cyclictest0-21swapper/111:06:031
1386799110,1cyclictest0-21swapper/012:30:170
1386799110,10cyclictest0-21swapper/011:35:140
1388599100,9cyclictest0-21swapper/312:36:043
1388599100,1cyclictest0-21swapper/307:45:553
1387899108,1cyclictest0-21swapper/212:23:562
1387899108,1cyclictest0-21swapper/211:56:202
1387899108,1cyclictest0-21swapper/210:24:272
1387899108,1cyclictest0-21swapper/209:21:102
1387899100,9cyclictest0-21swapper/211:27:162
1387899100,9cyclictest0-21swapper/209:46:312
1387899100,1cyclictest0-21swapper/208:41:352
1387299109,1cyclictest5120-21kworker/1:112:00:031
1387299109,1cyclictest23066-21snmpd08:13:551
1387299109,0cyclictest32741-21kworker/1:008:40:521
1387299109,0cyclictest0-21swapper/107:30:001
1387299101,1cyclictest101rcu_sched11:45:001
13872991010,0cyclictest27154-21kworker/1:109:22:091
13872991010,0cyclictest15787-21kworker/1:010:45:411
1387299100,9cyclictest0-21swapper/109:46:071
1387299100,1cyclictest0-21swapper/112:31:471
1387299100,0cyclictest0-21swapper/111:20:051
1386799109,0cyclictest0-21swapper/008:35:410
1386799109,0cyclictest0-21swapper/007:40:480
1386799108,1cyclictest0-21swapper/010:06:340
1386799102,4cyclictest8725-21awk12:25:000
1386799101,1cyclictest101rcu_sched07:35:010
1386799100,9cyclictest0-21swapper/012:01:040
1386799100,9cyclictest0-21swapper/009:39:380
1386799100,1cyclictest0-21swapper/012:14:290
1386799100,1cyclictest0-21swapper/010:28:330
684290,1sleep10-21swapper/109:28:001
26645290,1sleep00-21swapper/010:10:200
138859997,1cyclictest23066-21snmpd10:01:313
138859997,1cyclictest0-21swapper/312:06:113
138859997,1cyclictest0-21swapper/310:36:353
138859990,8cyclictest0-21swapper/311:19:223
138859990,8cyclictest0-21swapper/310:56:173
138859990,1cyclictest0-21swapper/307:24:463
138789997,1cyclictest0-21swapper/211:39:322
138789997,1cyclictest0-21swapper/211:30:202
138789997,1cyclictest0-21swapper/210:35:152
138789997,1cyclictest0-21swapper/208:46:372
138789990,8cyclictest0-21swapper/207:43:182
138789990,1cyclictest0-21swapper/209:31:392
138729999,0cyclictest26092-21kworker/1:107:36:551
138729997,1cyclictest26308-21kworker/1:108:27:011
138729997,1cyclictest26092-21kworker/1:107:40:031
138729997,1cyclictest24-21ksoftirqd/107:15:011
138729997,1cyclictest0-21swapper/110:43:551
138729990,8cyclictest0-21swapper/109:14:581
138679997,1cyclictest0-21swapper/009:02:000
138679997,1cyclictest0-21swapper/008:55:230
138679990,8cyclictest0-21swapper/009:22:310
138679990,1cyclictest101rcu_sched11:55:000
138679990,0cyclictest0-21swapper/007:13:000
26148280,6sleep326195-21uname09:15:223
18029280,6sleep015935-21munin-update09:55:270
138859987,0cyclictest0-21swapper/307:17:003
138859986,1cyclictest23066-21snmpd11:37:343
138789986,1cyclictest23066-21snmpd10:00:132
138789986,1cyclictest23066-21snmpd08:01:312
138789986,1cyclictest0-21swapper/209:43:562
138789986,1cyclictest0-21swapper/209:25:412
138789986,1cyclictest0-21swapper/208:17:222
138789980,7cyclictest0-21swapper/208:27:342
138729988,0cyclictest19071-21kworker/1:110:52:171
138729988,0cyclictest17388-21kworker/1:208:07:371
138729987,1cyclictest32741-21kworker/1:008:47:391
138729987,1cyclictest1009-21kworker/1:209:32:421
138729987,0cyclictest24-21ksoftirqd/112:10:001
138729986,1cyclictest24-21ksoftirqd/110:15:321
138729986,1cyclictest24-21ksoftirqd/109:40:011
138729986,1cyclictest0-21swapper/109:01:341
138729980,7cyclictest0-21swapper/107:49:041
138679987,0cyclictest0-21swapper/009:15:040
138679986,1cyclictest23066-21snmpd08:53:250
138679986,1cyclictest0-21swapper/011:31:500
138679986,1cyclictest0-21swapper/010:01:550
138679986,1cyclictest0-21swapper/008:31:300
138679980,7cyclictest0-21swapper/010:53:540
138679980,7cyclictest0-21swapper/009:48:500
29025270,0sleep20-21swapper/211:10:052
138859975,1cyclictest7949-31munin-html12:20:333
138859975,1cyclictest7222-21/usr/share/muni09:40:013
138859975,1cyclictest23066-21snmpd09:11:293
138859975,1cyclictest23066-21snmpd07:30:113
138859975,1cyclictest2084-31munin-html12:10:333
138859975,1cyclictest0-21swapper/311:44:333
138859975,1cyclictest0-21swapper/309:29:353
138859970,6cyclictest0-21swapper/310:54:503
138859970,6cyclictest0-21swapper/309:46:113
138859970,6cyclictest0-21swapper/308:06:053
138859970,1cyclictest0-21swapper/312:31:033
138859970,1cyclictest0-21swapper/311:24:323
138859970,1cyclictest0-21swapper/309:23:353
138789975,1cyclictest15873-21awk11:40:212
138789975,1cyclictest0-21swapper/212:30:432
138789975,1cyclictest0-21swapper/210:12:002
138789975,1cyclictest0-21swapper/209:00:202
138789975,1cyclictest0-21swapper/207:45:252
138789975,1cyclictest0-21swapper/207:35:342
138789975,1cyclictest0-21swapper/207:21:122
138729977,0cyclictest9347-21kworker/1:108:58:031
138729976,1cyclictest26150-21awk09:15:211
138729976,1cyclictest24748-21kworker/1:310:14:091
138729976,0cyclictest0-21swapper/110:20:101
138729976,0cyclictest0-21swapper/109:38:221
138729975,1cyclictest24-21ksoftirqd/111:30:011
138729975,1cyclictest24-21ksoftirqd/110:35:001
138729975,1cyclictest24-21ksoftirqd/107:55:011
138729975,1cyclictest0-21swapper/110:31:001
138729970,6cyclictest0-21swapper/111:04:351
138729970,6cyclictest0-21swapper/109:58:521
138729970,6cyclictest0-21swapper/108:04:051
138679976,0cyclictest0-21swapper/012:05:250
138679975,1cyclictest23066-21snmpd10:34:320
138679975,1cyclictest17206-21/usr/share/muni07:15:070
138679975,1cyclictest0-21swapper/012:39:210
138679975,1cyclictest0-21swapper/009:29:370
138679975,1cyclictest0-21swapper/008:49:250
138679975,1cyclictest0-21swapper/008:40:130
138679975,1cyclictest0-21swapper/008:18:050
138679975,1cyclictest0-21swapper/008:00:000
138679974,1cyclictest7-21ksoftirqd/011:45:000
138679971,4cyclictest17875-21kworker/0:108:25:280
138679971,1cyclictest101rcu_sched11:30:000
138679970,6cyclictest0-21swapper/011:11:590
138679970,1cyclictest19120-21ntpdc11:45:270
138679970,1cyclictest19120-21ntpdc11:45:270
138679970,1cyclictest0-21swapper/009:44:470
138679970,1cyclictest0-21swapper/007:59:220
22173260,1sleep30-21swapper/308:15:153
18002260,1sleep30-21swapper/309:01:203
16058260,1sleep30-21swapper/310:46:003
138859965,1cyclictest9365-21sadc08:50:003
138859965,1cyclictest23066-21snmpd11:57:323
138859965,1cyclictest23066-21snmpd11:45:583
138859965,1cyclictest23066-21snmpd11:45:583
138859965,1cyclictest23066-21snmpd08:28:583
138859964,1cyclictest87650irq/28-eno1-rx-08:55:303
138859964,1cyclictest23066-21snmpd09:33:093
138859964,1cyclictest0-21swapper/311:31:193
138859964,1cyclictest0-21swapper/311:27:563
138859964,1cyclictest0-21swapper/310:41:453
138859964,1cyclictest0-21swapper/310:34:003
138859964,1cyclictest0-21swapper/309:56:033
138859964,1cyclictest0-21swapper/309:37:003
138859963,2cyclictest752-21dbus-daemon07:36:343
138859960,1cyclictest0-21swapper/311:11:273
138789964,1cyclictest87950irq/34-eno2-rx-09:15:302
138789964,1cyclictest1349-21isc-worker000211:19:212
138789963,2cyclictest25852-21grep08:20:252
138789963,2cyclictest23066-21snmpd10:50:172
138789963,2cyclictest10881-31munin-html07:55:342
138789963,2cyclictest0-21swapper/210:55:012
138789960,2cyclictest188872expr07:15:202
138729965,1cyclictest23066-21snmpd12:36:301
138729964,1cyclictest0-21swapper/111:12:111
138729963,2cyclictest20787-21awk11:50:051
138679965,1cyclictest23066-21snmpd11:16:510
138679965,1cyclictest23066-21snmpd07:25:060
138679965,0cyclictest23066-21snmpd12:26:420
138679964,1cyclictest7-21ksoftirqd/012:20:000
138679964,1cyclictest0-21swapper/009:06:510
138679963,2cyclictest23066-21snmpd09:54:370
138679962,3cyclictest23943-21cpuspeed_turbos11:55:070
138679962,1cyclictest91rcu_preempt11:20:400
138679961,1cyclictest91rcu_preempt10:35:000
138679961,1cyclictest91rcu_preempt10:15:010
138679961,1cyclictest101rcu_sched09:35:000
138679960,1cyclictest2830-21munin-update07:45:010
138679960,1cyclictest23293-21munin-run07:25:010
138859953,1cyclictest9914-21cat08:50:063
138859953,1cyclictest9241-21cat12:25:053
138859953,1cyclictest775-21polkitd10:15:013
138859953,1cyclictest3944-21cat08:40:063
138859953,1cyclictest24229-21cat11:00:193
138859953,1cyclictest23066-21snmpd12:00:083
138859953,1cyclictest23066-21snmpd07:55:083
138859953,1cyclictest22194-31munin-html11:50:293
138859953,1cyclictest13494-21cat09:50:063
138859952,2cyclictest569-21systemd-journal10:25:073
138859952,2cyclictest24054-21/usr/share/muni08:20:133
138859952,2cyclictest23549-21sshd07:25:023
138859952,2cyclictest19681-21expr09:05:123
138859952,2cyclictest14067-31munin-graph08:00:363
138859950,4cyclictest18079-21/usr/share/muni08:10:013
138789953,1cyclictest752-21dbus-daemon07:30:012
138789953,1cyclictest26091-21cat11:05:052
138789953,1cyclictest24984-21awk07:25:192
138789953,1cyclictest23066-21snmpd10:15:012
138789953,1cyclictest18894-21awk08:10:102
138789953,1cyclictest0-21swapper/207:50:032
138789952,2cyclictest9620-21/usr/share/muni08:50:122
138789952,2cyclictest30029-21/usr/share/muni08:30:052
138789952,2cyclictest22746-21/usr/share/muni11:00:092
138789952,2cyclictest13769-21/usr/share/muni10:45:162
138789952,2cyclictest0-21swapper/212:10:012
138789952,2cyclictest0-21swapper/209:55:022
138789951,3cyclictest10234-21wc12:25:182
138679952,2cyclictest15123-21df08:05:010
138679951,2cyclictest91rcu_preempt10:20:070
138679951,1cyclictest91rcu_preempt10:40:000
138679950,4cyclictest21388-21/usr/share/muni09:10:130
138679950,2cyclictest6055-21/usr/share/muni07:50:010
138679950,1cyclictest91rcu_preempt10:45:040
138679950,1cyclictest25791-21/usr/share/muni11:05:000
138679950,1cyclictest22739-21/usr/share/muni11:00:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional