You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-22 - 22:25

x86 Intel Xeon E3-1270 v2 @3500 MHz, Linux 4.19.25-rt16 (Profile)

Latency plot of system in rack #3, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Wed Mar 22, 2023 12:43:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1031025026,9sleep20-21swapper/207:03:032
1004425039,8sleep30-21swapper/306:59:283
1025624824,9sleep70-21swapper/707:02:207
1002224634,8sleep10-21swapper/106:59:101
322872440,0sleep20-21swapper/210:38:032
111742430,0sleep70-21swapper/708:54:307
1002724129,8sleep60-21swapper/606:59:156
1007224028,8sleep40-21swapper/406:59:534
1006424028,8sleep50-21swapper/506:59:465
1009723927,8sleep00-21swapper/007:00:130
8072200,0sleep60-21swapper/609:43:536
8072200,0sleep60-21swapper/609:43:536
10476991712,3cyclictest57-21ksoftirqd/509:59:305
10476991712,3cyclictest57-21ksoftirqd/509:59:305
10480991512,2cyclictest7226-21memory09:48:547
1047699159,4cyclictest57-21ksoftirqd/510:47:475
1047699159,4cyclictest57-21ksoftirqd/510:47:475
1047699155,1cyclictest57-21ksoftirqd/509:20:325
1047699155,1cyclictest57-21ksoftirqd/509:20:325
10476991512,3cyclictest57-21ksoftirqd/510:32:585
1047399151,5cyclictest692-21dbus-daemon11:29:473
1047399151,5cyclictest692-21dbus-daemon11:29:473
1046799151,2cyclictest0-21swapper/209:41:562
1046799151,2cyclictest0-21swapper/209:41:562
1046799151,2cyclictest0-21swapper/209:41:562
10463991513,1cyclictest9-21ksoftirqd/011:44:590
10463991513,1cyclictest9-21ksoftirqd/011:44:590
134132140,0sleep00-21swapper/009:53:480
134132140,0sleep00-21swapper/009:53:480
10480991412,1cyclictest12940-21memory08:58:477
10480991412,1cyclictest12940-21memory08:58:477
1047699143,3cyclictest57-21ksoftirqd/511:41:495
10476991412,2cyclictest57-21ksoftirqd/511:16:405
10476991412,2cyclictest57-21ksoftirqd/511:16:405
10476991411,1cyclictest57-21ksoftirqd/509:04:525
10476991411,1cyclictest57-21ksoftirqd/509:04:525
1047699140,13cyclictest27432-21kworker/5:109:39:105
1047699140,13cyclictest27432-21kworker/5:109:39:105
1047699140,13cyclictest27432-21kworker/5:109:39:105
1047399142,5cyclictest29467-21systemd10:34:053
1047399142,5cyclictest25304-21systemd09:10:323
1047399142,5cyclictest25304-21systemd09:10:323
1047399142,4cyclictest24813-21systemd12:16:323
1047399142,4cyclictest24813-21systemd12:16:323
1047399141,7cyclictest0-21swapper/312:08:433
1047399141,7cyclictest0-21swapper/312:08:433
1047399141,5cyclictest692-21dbus-daemon12:23:443
1047399141,5cyclictest692-21dbus-daemon12:23:443
1047399141,4cyclictest0-21swapper/311:49:523
1047399141,4cyclictest0-21swapper/311:49:523
1047399141,4cyclictest0-21swapper/310:38:483
1047399141,4cyclictest0-21swapper/310:38:483
1047399140,13cyclictest12489-21kworker/3:309:31:523
1047399140,13cyclictest12489-21kworker/3:309:31:523
81392130,0sleep10-21swapper/110:43:541
81392130,0sleep10-21swapper/110:43:541
20612130,0sleep20-21swapper/210:38:512
20612130,0sleep20-21swapper/210:38:512
1047699135,1cyclictest57-21ksoftirqd/511:19:035
1047699135,1cyclictest57-21ksoftirqd/511:19:035
1047699135,1cyclictest57-21ksoftirqd/510:35:135
1047699133,1cyclictest57-21ksoftirqd/511:26:315
1047699133,1cyclictest57-21ksoftirqd/511:26:315
1047399132,5cyclictest9338-21systemd12:30:183
1047399132,5cyclictest9338-21systemd12:30:183
1047399132,5cyclictest7425-21systemd10:16:513
1047399132,5cyclictest7425-21systemd10:16:513
1047399132,5cyclictest25237-21systemd11:27:363
1047399132,5cyclictest25237-21systemd11:27:363
1047399132,5cyclictest19100-21systemd10:26:333
1047399132,4cyclictest20135-21http_loadtime07:18:453
1047399132,4cyclictest20135-21http_loadtime07:18:453
1047399132,4cyclictest14203-21memory11:43:483
1047399132,4cyclictest14203-21memory11:43:483
1047399132,2cyclictest28609-21smart_sda12:18:583
1047399132,2cyclictest28609-21smart_sda12:18:583
1047399132,2cyclictest22229-21systemd10:56:263
1047399132,2cyclictest22229-21systemd10:56:263
1047399132,2cyclictest22229-21systemd10:56:263
1047399132,2cyclictest17011-21systemd11:19:113
1047399132,2cyclictest17011-21systemd11:19:113
1047399132,2cyclictest12607-21idleruntime-cro11:43:293
1047399131,6cyclictest0-21swapper/311:08:523
1047399131,5cyclictest0-21swapper/309:03:513
1047399131,5cyclictest0-21swapper/309:03:513
1047399131,4cyclictest1-21systemd09:57:143
1047399131,4cyclictest1-21systemd09:57:143
1047399131,4cyclictest0-21swapper/310:21:503
1047399131,4cyclictest0-21swapper/310:21:503
1047399131,4cyclictest0-21swapper/309:46:003
1047399131,4cyclictest0-21swapper/309:46:003
1047399131,2cyclictest32288-21systemd11:04:253
1047399131,2cyclictest32288-21systemd11:04:253
1047399130,5cyclictest0-21swapper/310:58:443
10467991312,1cyclictest33-21ksoftirqd/211:38:562
10467991312,1cyclictest33-21ksoftirqd/211:05:382
10467991312,1cyclictest33-21ksoftirqd/211:05:382
1046799131,1cyclictest0-21swapper/209:30:582
1046799131,1cyclictest0-21swapper/209:30:582
1046799130,1cyclictest0-21swapper/211:28:342
1046799130,1cyclictest0-21swapper/211:28:342
1046799130,1cyclictest0-21swapper/208:48:522
113652120,0sleep20-21swapper/207:03:432
113652120,0sleep20-21swapper/207:03:432
1047699127,2cyclictest57-21ksoftirqd/510:48:375
1047699127,2cyclictest57-21ksoftirqd/510:48:375
1047699122,2cyclictest57-21ksoftirqd/510:40:165
1047699122,2cyclictest57-21ksoftirqd/510:40:165
10476991211,1cyclictest57-21ksoftirqd/509:33:555
10476991211,1cyclictest57-21ksoftirqd/509:33:555
10476991210,1cyclictest57-21ksoftirqd/512:19:315
10476991210,1cyclictest57-21ksoftirqd/512:19:315
10476991210,1cyclictest57-21ksoftirqd/512:03:295
10476991210,1cyclictest57-21ksoftirqd/512:03:295
10476991210,1cyclictest57-21ksoftirqd/511:53:445
10476991210,1cyclictest57-21ksoftirqd/509:13:475
10476991210,1cyclictest57-21ksoftirqd/509:13:475
1047699121,2cyclictest131rcu_sched11:29:025
1047699121,2cyclictest131rcu_sched11:29:025
1047699120,2cyclictest131rcu_sched12:05:235
1047699120,2cyclictest131rcu_sched12:05:235
1047399122,4cyclictest28951-21missed_timers09:13:563
1047399122,4cyclictest28951-21missed_timers09:13:563
1047399122,2cyclictest4478-21systemctl11:37:503
1047399122,2cyclictest4478-21systemctl11:37:503
1047399122,2cyclictest27977-21irqstats08:28:463
1047399122,2cyclictest26901-21timerandwakeup07:28:513
1047399122,2cyclictest26901-21timerandwakeup07:28:513
1047399122,2cyclictest24633-21systemd09:37:303
1047399122,2cyclictest24633-21systemd09:37:303
1047399122,2cyclictest10614-21grepconf.sh09:25:033
1047399122,2cyclictest10614-21grepconf.sh09:25:033
1047399121,7cyclictest0-21swapper/308:18:443
1047399121,7cyclictest0-21swapper/308:18:443
1047399121,6cyclictest27980-21sed09:38:573
1047399121,6cyclictest27980-21sed09:38:573
1047399121,6cyclictest27980-21sed09:38:573
1047399121,6cyclictest10960-21cpuspeed_turbos07:03:403
1047399121,6cyclictest10960-21cpuspeed_turbos07:03:403
1047399121,5cyclictest29970-21nfsd411:53:503
1047399121,5cyclictest2675-21iostat_ios09:18:543
1047399121,5cyclictest2675-21iostat_ios09:18:543
1047399121,4cyclictest692-21dbus-daemon10:45:003
1047399121,4cyclictest692-21dbus-daemon10:45:003
1047399121,4cyclictest13335-21sshd11:17:163
1047399121,4cyclictest13335-21sshd11:17:163
1047399121,4cyclictest1-21systemd12:02:373
1047399121,4cyclictest1-21systemd12:02:373
1047399121,2cyclictest30067-21sshd10:08:413
1047399121,2cyclictest30067-21sshd10:08:413
1047399121,2cyclictest15689-21cpuspeed_turbos08:08:423
1047399120,5cyclictest0-21swapper/310:03:423
1047399120,5cyclictest0-21swapper/310:03:423
1047399120,3cyclictest0-21swapper/309:58:483
1047399120,3cyclictest0-21swapper/309:58:483
1046799127,3cyclictest33-21ksoftirqd/212:02:102
1046799127,3cyclictest33-21ksoftirqd/212:02:102
1046799125,1cyclictest33-21ksoftirqd/209:23:142
1046799125,1cyclictest33-21ksoftirqd/209:23:142
1046799124,2cyclictest33-21ksoftirqd/211:55:032
1046799124,2cyclictest33-21ksoftirqd/210:46:022
1046799124,2cyclictest33-21ksoftirqd/210:46:022
1046799124,2cyclictest33-21ksoftirqd/210:26:092
10467991211,1cyclictest33-21ksoftirqd/211:36:442
10467991211,1cyclictest33-21ksoftirqd/211:36:442
10467991211,1cyclictest33-21ksoftirqd/211:23:492
10467991211,1cyclictest33-21ksoftirqd/211:23:492
10467991211,1cyclictest33-21ksoftirqd/211:10:442
10467991211,1cyclictest33-21ksoftirqd/211:02:402
1046799120,1cyclictest0-21swapper/209:10:292
1046799120,1cyclictest0-21swapper/209:10:292
1046799120,0cyclictest0-21swapper/211:19:392
1046799120,0cyclictest0-21swapper/211:19:392
10463991210,1cyclictest9-21ksoftirqd/011:53:500
10463991210,1cyclictest9-21ksoftirqd/010:38:290
10480991110,1cyclictest73-21ksoftirqd/711:03:537
10480991110,1cyclictest73-21ksoftirqd/711:03:537
1048099110,11cyclictest12277-21kworker/7:209:27:317
1048099110,11cyclictest12277-21kworker/7:209:27:317
1048099110,10cyclictest8203-21kworker/7:010:18:177
1048099110,10cyclictest8203-21kworker/7:010:18:177
1047799119,2cyclictest0-21swapper/609:33:586
1047799119,2cyclictest0-21swapper/609:33:586
1047799110,10cyclictest0-21swapper/610:43:336
1047799110,10cyclictest0-21swapper/610:43:336
1047699119,1cyclictest57-21ksoftirqd/511:44:545
1047699119,1cyclictest57-21ksoftirqd/511:44:545
1047699119,1cyclictest57-21ksoftirqd/511:08:525
1047699119,1cyclictest57-21ksoftirqd/510:11:415
1047699119,1cyclictest57-21ksoftirqd/510:11:415
1047699119,1cyclictest57-21ksoftirqd/510:07:305
1047699119,1cyclictest57-21ksoftirqd/510:07:305
1047699119,1cyclictest57-21ksoftirqd/509:57:445
1047699119,1cyclictest57-21ksoftirqd/509:57:445
1047699119,1cyclictest57-21ksoftirqd/509:46:415
1047699119,1cyclictest57-21ksoftirqd/509:46:415
1047699119,1cyclictest57-21ksoftirqd/508:48:305
1047699119,1cyclictest57-21ksoftirqd/508:48:305
1047699112,1cyclictest57-21ksoftirqd/510:23:015
1047699112,1cyclictest57-21ksoftirqd/510:23:015
10476991110,1cyclictest57-21ksoftirqd/509:51:025
10476991110,1cyclictest57-21ksoftirqd/509:09:045
10476991110,1cyclictest57-21ksoftirqd/509:09:045
10476991110,1cyclictest15-21kswork10:58:465
1047699111,3cyclictest57-21ksoftirqd/512:09:265
1047699111,3cyclictest57-21ksoftirqd/512:09:265
1047699111,1cyclictest57-21ksoftirqd/511:06:565
1047699111,1cyclictest57-21ksoftirqd/511:06:565
1047599111,4cyclictest28860-21sshd11:03:004
1047399119,1cyclictest21496-21diskstats10:28:513
1047399111,5cyclictest31889-21unixbench-2d08:33:513
1047399111,4cyclictest918-21grep08:38:433
1047399111,4cyclictest15311-21mailstats07:08:593
1047399111,4cyclictest14302-21sshd10:49:003
1047399111,4cyclictest14302-21sshd10:49:003
1047399111,4cyclictest13539-21smartctl08:03:453
1046799118,2cyclictest33-21ksoftirqd/211:44:462
1046799118,2cyclictest33-21ksoftirqd/211:44:462
1046799114,1cyclictest33-21ksoftirqd/212:23:002
1046799114,1cyclictest33-21ksoftirqd/212:23:002
1046799114,1cyclictest33-21ksoftirqd/210:32:062
10467991110,1cyclictest33-21ksoftirqd/212:23:542
10467991110,1cyclictest33-21ksoftirqd/212:23:542
10467991110,1cyclictest33-21ksoftirqd/211:13:532
10467991110,1cyclictest33-21ksoftirqd/211:13:532
10467991110,1cyclictest33-21ksoftirqd/210:18:542
10467991110,1cyclictest33-21ksoftirqd/210:18:542
10467991110,1cyclictest33-21ksoftirqd/209:03:532
10467991110,1cyclictest33-21ksoftirqd/209:03:532
1046799111,3cyclictest0-21swapper/212:11:112
1046799111,3cyclictest0-21swapper/212:11:112
1046799111,3cyclictest0-21swapper/211:52:142
1046799111,3cyclictest0-21swapper/211:52:142
1046799111,3cyclictest0-21swapper/210:56:492
1046799111,3cyclictest0-21swapper/210:56:492
1046799111,3cyclictest0-21swapper/210:56:492
1046799111,1cyclictest33-21ksoftirqd/212:30:022
1046799111,1cyclictest33-21ksoftirqd/212:30:022
1046399119,1cyclictest9-21ksoftirqd/012:03:050
1046399119,1cyclictest9-21ksoftirqd/012:03:050
1048099109,1cyclictest27003-21kworker/7:310:21:077
1048099109,1cyclictest27003-21kworker/7:310:21:077
1048099107,2cyclictest15258-21processes07:08:597
1047699108,1cyclictest57-21ksoftirqd/512:31:125
1047699108,1cyclictest57-21ksoftirqd/512:31:125
1047699108,1cyclictest57-21ksoftirqd/511:48:405
1047699108,1cyclictest57-21ksoftirqd/511:48:405
1047699108,1cyclictest57-21ksoftirqd/510:54:175
1047699108,1cyclictest57-21ksoftirqd/510:54:175
1047699108,1cyclictest57-21ksoftirqd/510:54:175
1047699108,1cyclictest57-21ksoftirqd/509:28:225
1047699108,1cyclictest57-21ksoftirqd/509:28:225
1047699103,6cyclictest692-21dbus-daemon09:31:045
1047699103,6cyclictest692-21dbus-daemon09:31:045
1047699101,1cyclictest57-21ksoftirqd/512:14:005
1047699101,1cyclictest57-21ksoftirqd/512:14:005
1047699101,1cyclictest57-21ksoftirqd/510:24:125
1047599100,9cyclictest17432-21kworker/4:010:03:374
1047599100,9cyclictest17432-21kworker/4:010:03:374
1047599100,4cyclictest31136-21sshd09:15:094
1047599100,4cyclictest31136-21sshd09:15:094
1047399101,7cyclictest0-21swapper/308:53:453
1047399101,6cyclictest0-21swapper/308:49:003
1047399101,6cyclictest0-21swapper/308:43:563
1047399101,6cyclictest0-21swapper/308:43:563
1047399101,2cyclictest0-21swapper/312:03:393
1047399101,2cyclictest0-21swapper/312:03:393
1046799105,3cyclictest33-21ksoftirqd/210:48:432
1046799105,3cyclictest33-21ksoftirqd/210:48:432
1046799105,3cyclictest33-21ksoftirqd/209:43:472
1046799105,3cyclictest33-21ksoftirqd/209:43:472
1046799105,3cyclictest33-21ksoftirqd/209:36:082
1046799105,3cyclictest33-21ksoftirqd/209:36:082
1046799104,2cyclictest33-21ksoftirqd/207:10:092
1046799103,1cyclictest33-21ksoftirqd/207:37:412
1046799101,3cyclictest33-21ksoftirqd/210:08:462
1046799101,3cyclictest33-21ksoftirqd/210:08:462
1046799101,3cyclictest0-21swapper/210:03:222
1046799101,3cyclictest0-21swapper/210:03:222
1046799101,1cyclictest0-21swapper/210:15:202
1046799101,1cyclictest0-21swapper/210:15:202
1046799101,1cyclictest0-21swapper/209:13:582
1046799101,1cyclictest0-21swapper/209:13:582
1046799100,3cyclictest0-21swapper/212:17:152
1046799100,3cyclictest0-21swapper/212:17:152
1046799100,1cyclictest0-21swapper/210:05:562
1046799100,1cyclictest0-21swapper/210:05:562
1046499100,1cyclictest0-21swapper/110:10:291
1046499100,1cyclictest0-21swapper/110:10:291
1046399109,1cyclictest9-21ksoftirqd/009:23:290
1046399109,1cyclictest9-21ksoftirqd/009:23:290
1046399104,3cyclictest28580-21sshd11:02:400
11605290,0sleep20-21swapper/207:58:512
104809997,1cyclictest73-21ksoftirqd/711:34:457
104809997,1cyclictest73-21ksoftirqd/711:34:457
104809997,1cyclictest73-21ksoftirqd/711:32:267
104809997,1cyclictest73-21ksoftirqd/711:32:267
104809997,1cyclictest73-21ksoftirqd/710:31:067
104809994,3cyclictest10175-21sshd12:31:237
104809994,3cyclictest10175-21sshd12:31:237
104809994,3cyclictest0-21swapper/710:56:197
104809994,3cyclictest0-21swapper/710:56:197
104809994,3cyclictest0-21swapper/710:56:197
104809994,3cyclictest0-21swapper/710:10:497
104809994,3cyclictest0-21swapper/710:10:497
104809993,3cyclictest17815-21sshd09:31:317
104809993,3cyclictest17815-21sshd09:31:317
104809990,9cyclictest0-21swapper/711:00:297
104779990,9cyclictest0-21swapper/611:47:546
104779990,9cyclictest0-21swapper/611:47:546
104779990,9cyclictest0-21swapper/607:58:476
104769998,1cyclictest57-21ksoftirqd/508:23:445
104769998,1cyclictest57-21ksoftirqd/508:23:445
104769998,1cyclictest57-21ksoftirqd/508:23:295
104769998,1cyclictest57-21ksoftirqd/508:23:295
104769998,1cyclictest57-21ksoftirqd/507:58:435
104769997,1cyclictest57-21ksoftirqd/512:24:285
104769997,1cyclictest57-21ksoftirqd/512:24:285
104769997,1cyclictest57-21ksoftirqd/507:48:295
104769997,1cyclictest57-21ksoftirqd/507:48:295
104769997,1cyclictest57-21ksoftirqd/507:28:295
104769995,3cyclictest57-21ksoftirqd/508:43:255
104769991,1cyclictest57-21ksoftirqd/511:34:245
104769991,1cyclictest57-21ksoftirqd/511:34:245
104759997,1cyclictest49-21ksoftirqd/407:43:294
104759997,1cyclictest49-21ksoftirqd/407:43:294
104759992,6cyclictest869-21polkitd11:53:414
104759990,9cyclictest27947-21kworker/4:310:10:004
104759990,9cyclictest27947-21kworker/4:310:10:004
104759990,9cyclictest0-21swapper/412:15:174
104759990,9cyclictest0-21swapper/412:15:174
104759990,9cyclictest0-21swapper/410:29:014
104759990,9cyclictest0-21swapper/409:20:364
104759990,9cyclictest0-21swapper/409:20:364
104759990,0cyclictest0-21swapper/411:59:024
104759990,0cyclictest0-21swapper/411:59:024
104759990,0cyclictest0-21swapper/409:38:434
104759990,0cyclictest0-21swapper/409:38:434
104759990,0cyclictest0-21swapper/409:38:434
104739999,0cyclictest0-21swapper/307:33:513
104739992,4cyclictest9507-21sshd09:50:463
104739991,6cyclictest0-21swapper/307:53:503
104739991,5cyclictest24061-21timerandwakeup07:23:543
104679999,0cyclictest321ktimersoftd/208:58:452
104679999,0cyclictest321ktimersoftd/208:58:452
104679998,1cyclictest33-21ksoftirqd/208:03:432
104679998,1cyclictest33-21ksoftirqd/207:14:022
104679998,1cyclictest33-21ksoftirqd/207:14:022
104679994,3cyclictest33-21ksoftirqd/208:40:192
104679994,3cyclictest33-21ksoftirqd/208:28:422
104679994,3cyclictest33-21ksoftirqd/207:49:012
104679991,3cyclictest0-21swapper/209:53:402
104679991,3cyclictest0-21swapper/209:53:402
104679991,2cyclictest0-21swapper/209:48:472
104679991,1cyclictest0-21swapper/212:03:512
104679991,1cyclictest0-21swapper/212:03:512
104679991,1cyclictest0-21swapper/209:24:002
104679991,1cyclictest0-21swapper/209:24:002
104679991,1cyclictest0-21swapper/208:17:252
104679991,1cyclictest0-21swapper/208:17:252
104679991,1cyclictest0-21swapper/207:43:312
104679991,1cyclictest0-21swapper/207:43:312
104679991,1cyclictest0-21swapper/207:24:362
104679990,4cyclictest0-21swapper/207:18:422
104679990,4cyclictest0-21swapper/207:18:422
104649999,0cyclictest0-21swapper/110:14:351
104649999,0cyclictest0-21swapper/110:14:351
104649999,0cyclictest0-21swapper/110:06:141
104649999,0cyclictest0-21swapper/110:06:141
104649997,1cyclictest25-21ksoftirqd/110:37:031
104649990,9cyclictest0-21swapper/112:27:391
104649990,9cyclictest0-21swapper/112:27:391
104649990,9cyclictest0-21swapper/111:37:221
104649990,9cyclictest0-21swapper/111:37:221
104649990,9cyclictest0-21swapper/110:18:511
104649990,9cyclictest0-21swapper/110:18:511
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional