You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-01 - 18:02

x86 Intel Xeon E3-1270 v2 @3500 MHz, Linux 4.19.25-rt16 (Profile)

Latency plot of system in rack #3, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack3slot0.osadl.org (updated Fri Jul 01, 2022 12:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
227382730,1sleep233-21ksoftirqd/212:02:182
304922480,0sleep30-21swapper/312:09:483
304922480,0sleep30-21swapper/312:09:483
275232440,0sleep70-21swapper/710:59:537
218102390,0sleep20-21swapper/208:24:532
218102390,0sleep20-21swapper/208:24:532
289452370,0sleep20-21swapper/209:54:532
289452370,0sleep20-21swapper/209:54:532
253982370,0sleep70-21swapper/712:04:477
253982370,0sleep70-21swapper/712:04:477
74182306,8sleep00-21swapper/007:04:210
71382305,8sleep40-21swapper/407:00:324
710822918,8sleep50-21swapper/507:00:095
71052296,8sleep20-21swapper/207:00:062
50482243,17sleep30-21swapper/306:59:433
737122312,8sleep70-21swapper/707:03:397
74102219,9sleep10-21swapper/107:04:141
7564992019,1cyclictest2146-21kworker/3:111:18:543
7564992018,1cyclictest22803-21kworker/3:012:02:343
7564992018,1cyclictest12520-21kworker/3:310:16:303
7564991918,1cyclictest7316-21kworker/3:211:38:083
7564991918,1cyclictest3576-21kworker/3:011:46:383
7564991918,1cyclictest18598-21kworker/3:107:59:543
7564991918,1cyclictest18598-21kworker/3:107:54:523
7564991918,1cyclictest12706-21kworker/3:011:20:583
7564991918,1cyclictest12706-21kworker/3:011:20:583
7564991918,0cyclictest27941-21kworker/3:309:30:123
7564991918,0cyclictest27941-21kworker/3:309:30:123
7564991917,1cyclictest9889-21kworker/3:210:56:323
7564991917,1cyclictest21593-21kworker/3:110:44:443
7564991917,1cyclictest21593-21kworker/3:110:44:443
7564991917,1cyclictest17846-21kworker/3:312:23:023
7564991917,1cyclictest17846-21kworker/3:312:23:023
7564991917,1cyclictest17846-21kworker/3:312:16:263
7564991917,1cyclictest17846-21kworker/3:312:16:263
7564991917,1cyclictest15709-21kworker/3:309:52:343
7564991916,2cyclictest13751-21kworker/3:209:21:323
7564991916,2cyclictest13751-21kworker/3:209:21:323
71582198,8sleep60-21swapper/607:00:506
7564991817,0cyclictest9889-21kworker/3:211:05:323
7564991817,0cyclictest2688-21kworker/3:010:03:083
7564991817,0cyclictest2688-21kworker/3:010:03:083
7564991817,0cyclictest21593-21kworker/3:110:42:503
7564991817,0cyclictest13512-21kworker/3:211:59:163
7564991816,2cyclictest13751-21kworker/3:209:40:163
7564991816,1cyclictest4459-21kworker/3:110:09:543
7564991816,1cyclictest4459-21kworker/3:110:09:543
7564991816,1cyclictest21593-21kworker/3:110:32:063
7564991716,1cyclictest9889-21kworker/3:211:00:103
7564991716,1cyclictest9889-21kworker/3:210:51:303
7564991716,1cyclictest9889-21kworker/3:210:51:303
7564991716,1cyclictest13512-21kworker/3:211:54:043
7564991716,1cyclictest13512-21kworker/3:211:54:043
7564991716,0cyclictest21593-21kworker/3:110:37:283
7564991715,2cyclictest13751-21kworker/3:209:09:243
7564991715,1cyclictest18598-21kworker/3:109:03:423
7564991715,1cyclictest18598-21kworker/3:109:03:423
7564991715,1cyclictest18598-21kworker/3:108:15:103
7564991715,1cyclictest18598-21kworker/3:108:15:103
7564991715,1cyclictest15709-21kworker/3:309:48:563
7564991715,1cyclictest13751-21kworker/3:209:17:543
7564991715,1cyclictest13751-21kworker/3:209:17:543
7564991713,0cyclictest18598-21kworker/3:108:06:403
7564991713,0cyclictest18598-21kworker/3:108:06:403
7564991615,1cyclictest7264-21kworker/3:207:08:043
7564991615,1cyclictest7264-21kworker/3:207:08:043
7564991615,1cyclictest18598-21kworker/3:108:46:423
7564991615,1cyclictest18598-21kworker/3:108:46:423
7564991615,1cyclictest13751-21kworker/3:209:59:203
7564991615,1cyclictest13751-21kworker/3:209:59:203
7564991615,1cyclictest13751-21kworker/3:209:12:223
7564991615,1cyclictest13751-21kworker/3:209:12:223
7564991615,1cyclictest11032-21kworker/3:212:32:023
7564991615,0cyclictest4459-21kworker/3:110:06:063
7564991615,0cyclictest4459-21kworker/3:110:06:063
7564991615,0cyclictest21593-21kworker/3:110:25:303
7564991615,0cyclictest18598-21kworker/3:107:51:243
7564991615,0cyclictest18598-21kworker/3:107:38:463
7564991614,2cyclictest7316-21kworker/3:211:26:203
7564991614,2cyclictest7316-21kworker/3:211:26:203
7564991614,1cyclictest9217-21kworker/3:112:26:403
7564991614,1cyclictest9217-21kworker/3:112:26:403
7564991614,1cyclictest7316-21kworker/3:211:32:463
7564991614,1cyclictest27941-21kworker/3:309:27:583
7564991614,1cyclictest17846-21kworker/3:312:07:363
7564991614,1cyclictest17846-21kworker/3:312:07:363
7564991614,1cyclictest13751-21kworker/3:209:35:143
7564991613,2cyclictest7264-21kworker/3:207:23:303
7564991514,1cyclictest18598-21kworker/3:108:50:103
7564991514,0cyclictest7316-21kworker/3:211:40:123
7564991514,0cyclictest7316-21kworker/3:211:40:123
7564991514,0cyclictest18598-21kworker/3:108:55:223
7564991514,0cyclictest18598-21kworker/3:108:55:223
7564991513,2cyclictest18598-21kworker/3:108:32:403
7564991513,2cyclictest18598-21kworker/3:107:27:083
7564991513,1cyclictest18598-21kworker/3:107:47:463
7564991513,1cyclictest18598-21kworker/3:107:30:063
7564991513,1cyclictest18598-21kworker/3:107:30:063
7564991511,1cyclictest7264-21kworker/3:207:09:383
67512150,0sleep40-21swapper/410:05:204
67512150,0sleep40-21swapper/410:05:204
756899140,13cyclictest0-21swapper/712:24:437
756899140,13cyclictest0-21swapper/712:24:437
7564991413,1cyclictest2146-21kworker/3:111:10:443
7564991413,1cyclictest18598-21kworker/3:108:34:443
7564991413,1cyclictest18598-21kworker/3:108:22:363
7564991413,1cyclictest18598-21kworker/3:108:22:363
7564991413,1cyclictest18598-21kworker/3:107:42:143
7564991413,0cyclictest18598-21kworker/3:108:43:143
7564991413,0cyclictest18598-21kworker/3:108:43:143
7564991413,0cyclictest18598-21kworker/3:108:25:343
7564991413,0cyclictest18598-21kworker/3:108:25:343
7564991413,0cyclictest18598-21kworker/3:108:09:583
7564991412,2cyclictest12520-21kworker/3:310:20:283
7564991412,1cyclictest7264-21kworker/3:207:16:143
7564991412,1cyclictest7264-21kworker/3:207:16:143
20362140,0sleep40-21swapper/409:29:334
756899120,2cyclictest9828-21grepconf.sh09:35:577
7567991210,1cyclictest8191-21cstates07:04:456
7567991210,1cyclictest8191-21cstates07:04:456
756899114,6cyclictest5363-21grepconf.sh12:17:447
756899114,6cyclictest5363-21grepconf.sh12:17:447
756899114,6cyclictest4668-21bash09:30:217
756899114,6cyclictest4668-21bash09:30:217
756899114,6cyclictest2916-21httpd11:39:537
756899114,6cyclictest2916-21httpd11:39:537
756899114,6cyclictest23748-21grep12:04:107
756899114,6cyclictest12649-21grepconf.sh09:06:047
756899113,2cyclictest737-21dbus-daemon10:35:457
756899113,2cyclictest30042-21latency10:29:477
756899113,2cyclictest25352-21grepconf.sh09:52:207
756899112,2cyclictest1041-21grep12:13:527
756899112,2cyclictest1041-21grep12:13:527
756899111,2cyclictest965-21gdbus10:04:447
756899111,2cyclictest965-21gdbus10:04:447
756599119,1cyclictest49-21ksoftirqd/407:10:154
756599118,2cyclictest20629-21sshd11:26:274
756599118,2cyclictest20629-21sshd11:26:274
756299111,3cyclictest0-21swapper/109:28:381
756199113,4cyclictest28076-21perf08:39:320
756899107,3cyclictest0-21swapper/709:21:417
756899107,3cyclictest0-21swapper/709:21:417
756899104,5cyclictest13385-21switchtime08:09:527
756899103,2cyclictest10262-21grep11:15:177
756899103,1cyclictest8963-21grepconf.sh12:20:057
756899103,1cyclictest8963-21grepconf.sh12:20:057
756899103,1cyclictest794-21snmpd11:19:357
756899103,1cyclictest794-21snmpd11:19:357
756899103,1cyclictest712-21cstates07:49:437
756899103,1cyclictest2922-21httpd09:54:457
756899103,1cyclictest2922-21httpd09:54:457
756899102,2cyclictest965-21gdbus09:09:497
756899102,2cyclictest965-21gdbus09:09:497
756899102,2cyclictest952-21polkitd10:50:167
756899102,2cyclictest952-21polkitd10:50:167
756899102,2cyclictest737-21dbus-daemon10:57:277
756899102,2cyclictest29739-21grep08:39:487
756899102,2cyclictest29739-21grep08:39:487
756899102,2cyclictest23496-21grep08:29:447
756899102,2cyclictest0-21swapper/710:43:067
756899102,1cyclictest0-21swapper/710:44:517
756899102,1cyclictest0-21swapper/710:44:517
756899102,1cyclictest0-21swapper/709:40:007
756899102,1cyclictest0-21swapper/709:26:087
756899101,2cyclictest965-21gdbus09:47:547
756899101,2cyclictest759-21systemd-logind11:28:507
756899101,2cyclictest759-21systemd-logind11:28:507
756899101,2cyclictest737-21dbus-daemon11:32:067
756899101,2cyclictest32693-21nscd07:44:457
756899101,2cyclictest1-21systemd12:29:517
756899101,2cyclictest0-21swapper/708:14:447
756899101,2cyclictest0-21swapper/708:14:447
756899100,2cyclictest952-21polkitd10:27:077
756899100,2cyclictest23106-21gdbus10:02:037
756899100,2cyclictest23106-21gdbus10:02:037
756899100,2cyclictest1-21systemd10:14:127
756899100,2cyclictest1-21systemd10:14:127
756699106,3cyclictest57-21ksoftirqd/511:20:055
756699106,3cyclictest57-21ksoftirqd/511:20:055
756599100,2cyclictest0-21swapper/407:17:024
756599100,2cyclictest0-21swapper/407:17:024
756299109,1cyclictest25-21ksoftirqd/111:59:221
756299109,1cyclictest0-21swapper/108:59:491
756299109,1cyclictest0-21swapper/108:59:491
75689993,1cyclictest3308-21switchtime08:49:527
75689993,1cyclictest15412-21bash11:54:477
75689992,2cyclictest19067-21cpuspeed_turbos07:24:457
75689992,1cyclictest10681-21cpu07:09:427
75689992,1cyclictest10397-21http_loadtime11:49:477
75689992,1cyclictest10397-21http_loadtime11:49:477
75689992,1cyclictest0-21swapper/708:24:467
75689992,1cyclictest0-21swapper/708:24:467
75689991,3cyclictest0-21swapper/707:15:177
75689991,3cyclictest0-21swapper/707:15:177
75689991,2cyclictest737-21dbus-daemon09:17:057
75689991,2cyclictest737-21dbus-daemon09:17:057
75689991,2cyclictest5194-21sshd11:09:567
75689991,2cyclictest10224-21nfsd408:04:497
75689991,2cyclictest10224-21nfsd408:04:497
75689991,2cyclictest0-21swapper/707:29:477
75689991,2cyclictest0-21swapper/707:29:477
75689990,8cyclictest0-21swapper/707:59:427
75689990,3cyclictest0-21swapper/708:37:497
75689990,2cyclictest8421-21cut07:04:477
75689990,2cyclictest8421-21cut07:04:477
75689990,2cyclictest7320-21sshd11:45:377
75689990,2cyclictest5774-21dump-pmu-power07:59:327
75689990,2cyclictest32536-21unixbench-2d11:04:557
75689990,2cyclictest31729-21fschecks_count08:44:457
75689990,2cyclictest31729-21fschecks_count08:44:457
75689990,2cyclictest29815-21sshd11:35:157
75689990,2cyclictest19439-21cut10:19:437
75689990,2cyclictest16426-21cpuspeed_turbos07:19:437
75689990,2cyclictest0-21swapper/708:55:257
75689990,2cyclictest0-21swapper/708:55:257
75689990,0cyclictest0-21swapper/710:16:257
75689990,0cyclictest0-21swapper/709:01:407
75689990,0cyclictest0-21swapper/709:01:407
75689990,0cyclictest0-21swapper/708:22:147
75689990,0cyclictest0-21swapper/708:22:147
75689990,0cyclictest0-21swapper/707:36:157
75679999,0cyclictest0-21swapper/611:35:076
75679999,0cyclictest0-21swapper/610:32:136
75679997,1cyclictest65-21ksoftirqd/610:08:426
75679997,1cyclictest65-21ksoftirqd/610:08:426
75679990,9cyclictest0-21swapper/609:59:546
75679990,9cyclictest0-21swapper/609:59:546
75679990,9cyclictest0-21swapper/609:09:556
75679990,9cyclictest0-21swapper/609:09:556
75679990,9cyclictest0-21swapper/608:29:536
75679990,1cyclictest0-21swapper/612:11:476
75679990,1cyclictest0-21swapper/612:11:476
75679990,0cyclictest0-21swapper/612:01:086
75679990,0cyclictest0-21swapper/609:43:306
75669999,0cyclictest0-21swapper/509:28:165
75669990,9cyclictest0-21swapper/511:28:055
75669990,9cyclictest0-21swapper/511:28:055
75669990,9cyclictest0-21swapper/511:10:255
75669990,9cyclictest0-21swapper/509:34:065
75669990,9cyclictest0-21swapper/509:34:065
75669990,0cyclictest0-21swapper/509:42:185
75659997,1cyclictest49-21ksoftirqd/412:01:314
75659993,2cyclictest23247-21bash10:23:434
75659990,9cyclictest0-21swapper/409:14:484
75659990,9cyclictest0-21swapper/409:14:484
75639999,0cyclictest0-21swapper/212:08:002
75639999,0cyclictest0-21swapper/212:08:002
75639999,0cyclictest0-21swapper/211:54:452
75639999,0cyclictest0-21swapper/211:45:182
75639999,0cyclictest0-21swapper/211:05:442
75639999,0cyclictest0-21swapper/209:26:062
75639999,0cyclictest0-21swapper/209:11:542
75639999,0cyclictest0-21swapper/209:11:542
75639998,1cyclictest0-21swapper/212:09:532
75639998,1cyclictest0-21swapper/212:09:532
75639990,9cyclictest0-21swapper/212:32:362
75639990,9cyclictest0-21swapper/212:15:342
75639990,9cyclictest0-21swapper/212:15:342
75639990,9cyclictest0-21swapper/211:43:242
75639990,9cyclictest0-21swapper/211:43:242
75639990,9cyclictest0-21swapper/210:36:432
75639990,9cyclictest0-21swapper/210:03:272
75639990,9cyclictest0-21swapper/210:03:272
75639990,0cyclictest0-21swapper/211:15:402
75639990,0cyclictest0-21swapper/209:50:512
75639990,0cyclictest0-21swapper/209:42:202
75629998,1cyclictest25-21ksoftirqd/111:38:581
75629997,1cyclictest25-21ksoftirqd/110:39:051
75629997,1cyclictest25-21ksoftirqd/109:22:291
75629997,1cyclictest25-21ksoftirqd/109:22:291
75629997,1cyclictest25-21ksoftirqd/109:06:581
75629990,9cyclictest0-21swapper/112:31:041
75629990,9cyclictest0-21swapper/112:12:191
75629990,9cyclictest0-21swapper/112:12:191
75629990,9cyclictest0-21swapper/111:51:391
75629990,9cyclictest0-21swapper/111:51:391
75629990,9cyclictest0-21swapper/110:56:381
75629990,9cyclictest0-21swapper/110:34:061
75629990,9cyclictest0-21swapper/109:29:561
75629990,9cyclictest0-21swapper/109:29:561
75619999,0cyclictest0-21swapper/012:01:570
75619997,1cyclictest9-21ksoftirqd/012:19:110
75619997,1cyclictest9-21ksoftirqd/012:19:110
75619997,1cyclictest0-21swapper/010:24:500
75619990,9cyclictest0-21swapper/010:38:520
75619990,9cyclictest0-21swapper/010:14:350
75619990,9cyclictest0-21swapper/009:58:310
75619990,9cyclictest0-21swapper/009:58:310
75619990,8cyclictest0-21swapper/011:15:170
25024290,0sleep60-21swapper/607:34:446
75689980,2cyclictest28449-21needreboot07:39:487
75679988,0cyclictest0-21swapper/611:44:356
75679988,0cyclictest0-21swapper/611:44:356
75679988,0cyclictest0-21swapper/610:52:146
75679988,0cyclictest0-21swapper/610:52:146
75679988,0cyclictest0-21swapper/607:39:456
75679986,1cyclictest65-21ksoftirqd/610:20:516
75669988,0cyclictest0-21swapper/510:15:245
75669988,0cyclictest0-21swapper/509:54:545
75669988,0cyclictest0-21swapper/509:54:545
75669988,0cyclictest0-21swapper/509:46:335
75669986,1cyclictest57-21ksoftirqd/512:12:595
75669986,1cyclictest57-21ksoftirqd/512:12:595
75669986,1cyclictest57-21ksoftirqd/511:35:115
75669980,8cyclictest0-21swapper/512:22:475
75669980,8cyclictest0-21swapper/512:22:475
75669980,8cyclictest0-21swapper/510:42:035
75669980,8cyclictest0-21swapper/510:05:095
75669980,8cyclictest0-21swapper/510:05:095
75669980,8cyclictest0-21swapper/507:22:055
75669980,8cyclictest0-21swapper/507:04:585
75669980,8cyclictest0-21swapper/507:04:585
75669980,0cyclictest0-21swapper/512:00:245
75669980,0cyclictest0-21swapper/509:38:025
75659986,1cyclictest49-21ksoftirqd/411:16:304
75659986,1cyclictest49-21ksoftirqd/410:51:134
75659986,1cyclictest49-21ksoftirqd/410:51:134
75659986,1cyclictest49-21ksoftirqd/410:44:444
75659986,1cyclictest49-21ksoftirqd/410:44:444
75659986,1cyclictest49-21ksoftirqd/410:31:334
75659986,1cyclictest49-21ksoftirqd/409:20:584
75659986,1cyclictest49-21ksoftirqd/409:20:584
75659986,1cyclictest0-21swapper/409:29:564
75659986,1cyclictest0-21swapper/409:29:564
75659980,8cyclictest0-21swapper/411:06:534
75659980,8cyclictest0-21swapper/407:34:054
75659980,8cyclictest0-21swapper/407:34:054
75659980,0cyclictest0-21swapper/412:24:464
75659980,0cyclictest0-21swapper/412:24:464
75659980,0cyclictest0-21swapper/410:39:564
75639988,0cyclictest0-21swapper/212:26:552
75639988,0cyclictest0-21swapper/212:26:552
75639988,0cyclictest0-21swapper/211:50:582
75639988,0cyclictest0-21swapper/211:50:582
75639987,1cyclictest33-21ksoftirqd/210:22:002
75639986,1cyclictest33-21ksoftirqd/209:21:582
75639986,1cyclictest33-21ksoftirqd/209:21:582
75639986,1cyclictest33-21ksoftirqd/209:16:112
75639986,1cyclictest33-21ksoftirqd/209:16:112
75639980,8cyclictest0-21swapper/211:11:432
75629988,0cyclictest0-21swapper/112:16:061
75629988,0cyclictest0-21swapper/112:16:061
75629988,0cyclictest0-21swapper/110:54:351
75629988,0cyclictest0-21swapper/110:54:351
75629988,0cyclictest0-21swapper/110:27:571
75629988,0cyclictest0-21swapper/107:19:521
75629987,1cyclictest0-21swapper/111:14:461
75629986,1cyclictest25-21ksoftirqd/112:28:311
75629986,1cyclictest25-21ksoftirqd/112:28:311
75629986,1cyclictest25-21ksoftirqd/112:09:061
75629986,1cyclictest25-21ksoftirqd/112:09:061
75629986,1cyclictest25-21ksoftirqd/110:48:201
75629986,1cyclictest25-21ksoftirqd/110:48:201
75629980,8cyclictest0-21swapper/110:17:321
75629980,8cyclictest0-21swapper/110:12:421
75629980,8cyclictest0-21swapper/110:12:421
75629980,8cyclictest0-21swapper/109:40:011
75629980,0cyclictest0-21swapper/111:04:501
75629980,0cyclictest0-21swapper/111:02:471
75629980,0cyclictest0-21swapper/110:40:141
75629980,0cyclictest0-21swapper/109:48:321
75619988,0cyclictest0-21swapper/012:11:060
75619988,0cyclictest0-21swapper/012:11:060
75619988,0cyclictest0-21swapper/011:41:460
75619988,0cyclictest0-21swapper/011:41:460
75619988,0cyclictest0-21swapper/011:34:030
75619988,0cyclictest0-21swapper/007:39:560
75619987,1cyclictest9-21ksoftirqd/012:05:540
75619987,1cyclictest9-21ksoftirqd/012:05:540
75619986,1cyclictest9-21ksoftirqd/011:38:220
75619986,1cyclictest9-21ksoftirqd/011:26:340
75619986,1cyclictest9-21ksoftirqd/011:26:340
75619986,1cyclictest9-21ksoftirqd/009:14:040
75619986,1cyclictest9-21ksoftirqd/009:14:040
75619980,8cyclictest0-21swapper/012:29:320
75619980,8cyclictest0-21swapper/012:29:320
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional