You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-01 - 21:54

x86 Intel Xeon E3-1220 @3100 MHz, Linux 4.19.8-rt6 (Profile)

Latency plot of system in rack #1, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the alll highest latencies:
System rack1slot0.osadl.org (updated Fri Jul 01, 2022 12:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
775221160,0sleep10-21swapper/112:33:141
293321140,2sleep1713399cyclictest10:16:291
169942890,2sleep3714899cyclictest10:10:103
226982870,2sleep3714899cyclictest10:20:263
289832850,2sleep0712799cyclictest11:03:030
145472670,1sleep214543-21lspci10:32:152
87602640,0sleep38763-21bash09:24:583
25032630,0sleep10-21swapper/111:35:451
25592620,1sleep32561-21id11:43:123
80492600,0sleep20-21swapper/211:01:252
90242590,0sleep00-21swapper/011:53:390
293732590,1sleep129376-21lspci11:55:121
160672590,1sleep216064-21sshd12:28:552
77122580,0sleep20-21swapper/212:05:512
695125733,7sleep00-21swapper/007:09:470
659425734,7sleep10-21swapper/107:05:521
60712570,0sleep30-21swapper/310:31:363
176212570,0sleep3391rcuc/309:35:273
81872560,0sleep00-21swapper/011:11:240
36222560,0sleep10-21swapper/111:06:031
139392560,0sleep20-21swapper/209:15:122
139392560,0sleep20-21swapper/209:15:112
132662560,0sleep013268-21bash10:39:390
252322550,1sleep325229-21lspci09:43:293
231732550,0sleep2321ktimersoftd/210:08:082
183582550,0sleep2311rcuc/210:02:472
179572550,2sleep2714099cyclictest09:55:182
132172550,0sleep00-21swapper/009:30:160
693425442,7sleep30-21swapper/307:09:353
675325342,7sleep20-21swapper/207:07:412
261122530,1sleep2714099cyclictest09:21:132
33712500,0sleep30-21swapper/310:16:313
43342470,2sleep2714099cyclictest11:11:062
193572470,0sleep30-21swapper/311:04:503
81002430,0sleep08104-21grep10:06:580
119662420,0sleep20-21swapper/210:44:302
305512210,1sleep1820-21dbus-daemon12:15:071
305512210,1sleep1820-21dbus-daemon12:15:071
214122200,1sleep121414-21lspci09:48:121
123892200,1sleep112390-21systemd-cgroups09:30:131
81122190,1sleep18105-21sshd09:19:521
81122190,1sleep18105-21sshd09:19:511
289162190,2sleep2714099cyclictest10:35:492
95652180,0sleep30-21swapper/309:27:303
45112180,0sleep10-21swapper/109:22:041
289892180,0sleep028984-21lspci11:55:110
259772180,0sleep3391rcuc/309:18:413
259772180,0sleep3391rcuc/309:18:413
9542170,0sleep3391rcuc/311:33:113
47912170,1sleep24783-21sshd12:13:082
47912170,1sleep24783-21sshd12:13:082
289812170,1sleep328985-21bash10:45:433
113352170,0sleep0111rcuc/009:37:290
96112160,0sleep30-21swapper/310:44:193
248102160,0sleep30-21swapper/312:07:123
121002160,1sleep012099-21grep11:39:010
714099155,1cyclictest33-21ksoftirqd/210:11:272
52672150,0sleep10-21swapper/110:04:191
172102150,0sleep00-21swapper/010:10:100
161622150,0sleep10-21swapper/110:39:521
92482140,0sleep10-21swapper/112:01:011
714099145,1cyclictest33-21ksoftirqd/210:55:122
714099145,1cyclictest33-21ksoftirqd/210:46:542
714099144,1cyclictest33-21ksoftirqd/209:45:492
714099144,1cyclictest33-21ksoftirqd/209:28:312
714099134,3cyclictest33-21ksoftirqd/211:19:072
714099134,2cyclictest33-21ksoftirqd/209:37:292
714099133,2cyclictest33-21ksoftirqd/211:32:432
714099132,2cyclictest3601-21sshd12:35:212
714099132,2cyclictest33-21ksoftirqd/209:54:242
714099131,3cyclictest33-21ksoftirqd/212:30:002
714099131,2cyclictest33-21ksoftirqd/210:25:292
7140991311,1cyclictest820-21dbus-daemon11:25:422
7127991312,0cyclictest25876-21lspci12:02:190
712799131,1cyclictest1-21systemd09:40:240
7127991311,1cyclictest28648-21lspci10:18:300
712799130,1cyclictest20515-21systemd-cgroups09:57:590
712799130,12cyclictest18329-21bash09:45:240
45522130,0sleep00-21swapper/010:46:190
324242130,1sleep232423-21bash09:31:432
7148991210,1cyclictest820-21dbus-daemon11:25:493
714899120,11cyclictest29612-21id11:13:053
714899120,11cyclictest18994-21sshd11:19:413
714099124,2cyclictest33-21ksoftirqd/212:17:492
714099124,2cyclictest33-21ksoftirqd/212:17:492
714099124,1cyclictest33-21ksoftirqd/212:00:062
714099124,1cyclictest33-21ksoftirqd/211:58:542
714099124,1cyclictest33-21ksoftirqd/211:38:252
714099124,1cyclictest33-21ksoftirqd/211:06:122
714099124,1cyclictest33-21ksoftirqd/210:15:292
714099123,3cyclictest33-21ksoftirqd/209:11:362
714099123,3cyclictest33-21ksoftirqd/209:11:362
714099123,1cyclictest33-21ksoftirqd/211:43:022
714099122,4cyclictest1-21systemd10:20:342
714099122,1cyclictest33-21ksoftirqd/211:45:152
714099121,3cyclictest33-21ksoftirqd/211:50:012
714099121,3cyclictest33-21ksoftirqd/210:50:032
714099121,1cyclictest33-21ksoftirqd/212:21:072
714099121,1cyclictest33-21ksoftirqd/212:21:072
714099120,4cyclictest24969-21lspci11:20:092
713399120,1cyclictest27003-21sshd11:10:211
713399120,1cyclictest0-21swapper/111:52:101
7127991211,0cyclictest25791-21kworker/0:212:31:070
7127991211,0cyclictest18688-21lspci11:32:030
7127991211,0cyclictest17186-21kworker/0:207:55:120
7127991211,0cyclictest17186-21kworker/0:207:55:110
7127991210,1cyclictest6406-21sshd10:34:090
7127991210,1cyclictest10516-21sshd12:08:370
7127991210,0cyclictest5279-21kworker/0:110:50:350
712799120,1cyclictest1165-21bash10:03:570
226432120,0sleep30-21swapper/312:16:583
226432120,0sleep30-21swapper/312:16:583
714899119,1cyclictest26099-21sshd11:47:283
7148991110,0cyclictest0-21swapper/310:56:273
714899110,10cyclictest4646-21sshd10:01:433
714899110,10cyclictest0-21swapper/311:05:583
714099113,1cyclictest33-21ksoftirqd/209:40:222
7133991110,0cyclictest27818-21sshd10:28:221
713399110,10cyclictest0-21swapper/111:23:351
713399110,10cyclictest0-21swapper/109:50:161
713399110,10cyclictest0-21swapper/109:27:191
712799119,1cyclictest19516-21kworker/0:412:29:230
712799110,1cyclictest3573-21sshd10:43:510
712799110,1cyclictest0-21swapper/011:22:420
712799110,1cyclictest0-21swapper/011:10:000
712799110,10cyclictest0-21swapper/009:53:270
712799110,10cyclictest0-21swapper/009:10:220
712799110,10cyclictest0-21swapper/009:10:210
38552110,2sleep1713399cyclictest11:43:181
73162100,1sleep37317-21sshd11:23:453
714899109,0cyclictest0-21swapper/310:36:453
714899108,1cyclictest18184-21bash09:12:573
714899108,1cyclictest18184-21bash09:12:563
713399109,0cyclictest0-21swapper/112:39:281
713399109,0cyclictest0-21swapper/111:34:531
713399100,0cyclictest0-21swapper/112:08:571
713399100,0cyclictest0-21swapper/110:50:101
713399100,0cyclictest0-21swapper/110:34:301
712799109,0cyclictest984-21kworker/0:211:25:050
712799109,0cyclictest0-21swapper/010:28:010
712799109,0cyclictest0-21swapper/007:20:150
712799108,1cyclictest9-21ksoftirqd/009:20:130
712799108,1cyclictest8302-21kworker/0:311:17:590
712799108,1cyclictest17896-21kworker/0:109:29:080
712799104,1cyclictest9-21ksoftirqd/010:21:150
712799103,2cyclictest9-21ksoftirqd/011:40:110
712799101,1cyclictest131rcu_sched12:13:280
712799101,1cyclictest131rcu_sched12:13:280
712799100,9cyclictest0-21swapper/008:13:500
712799100,9cyclictest0-21swapper/007:15:040
712799100,0cyclictest0-21swapper/008:39:590
269272100,0sleep30-21swapper/312:32:123
172702100,0sleep30-21swapper/312:38:533
148442100,1sleep0909-21NetworkManager10:59:290
71489998,0cyclictest0-21swapper/312:27:013
71489998,0cyclictest0-21swapper/309:49:063
71489998,0cyclictest0-21swapper/309:32:203
71489998,0cyclictest0-21swapper/308:11:553
71489997,1cyclictest26059-21lspci09:50:593
71489997,1cyclictest0-21swapper/311:55:423
71489990,8cyclictest32760-21sshd12:02:513
71409992,1cyclictest33-21ksoftirqd/208:00:012
71409992,1cyclictest33-21ksoftirqd/208:00:002
71409990,8cyclictest0-21swapper/207:43:322
71339998,0cyclictest0-21swapper/110:42:441
71339998,0cyclictest0-21swapper/109:57:511
71339997,1cyclictest1232-21lspci09:39:131
71339990,0cyclictest0-21swapper/111:49:541
71339990,0cyclictest0-21swapper/107:58:321
71339990,0cyclictest0-21swapper/107:58:311
71279998,0cyclictest22882-21kworker/0:409:16:300
71279998,0cyclictest22882-21kworker/0:409:16:290
71279998,0cyclictest0-21swapper/008:15:180
71279997,1cyclictest9-21ksoftirqd/012:39:240
71279997,1cyclictest9-21ksoftirqd/012:16:370
71279997,1cyclictest9-21ksoftirqd/012:16:360
71279997,1cyclictest9-21ksoftirqd/011:45:350
71279994,2cyclictest9-21ksoftirqd/012:22:380
71279994,2cyclictest9-21ksoftirqd/012:22:380
71489987,0cyclictest3650-21lspci11:35:503
71489987,0cyclictest0-21swapper/312:12:593
71489987,0cyclictest0-21swapper/312:12:593
71489987,0cyclictest0-21swapper/309:56:123
71489987,0cyclictest0-21swapper/307:54:483
71489980,7cyclictest24818-21sshd12:24:363
71489980,7cyclictest24818-21sshd12:24:363
71489980,7cyclictest22277-21sshd10:08:043
71489980,7cyclictest19765-21sshd11:51:583
71489980,7cyclictest11763-21sshd08:41:313
71339987,0cyclictest0-21swapper/111:17:561
71339987,0cyclictest0-21swapper/110:49:411
71339987,0cyclictest0-21swapper/107:32:321
71339986,1cyclictest9864-21lspci10:22:001
71339986,1cyclictest32764-21lspci09:14:091
71339986,1cyclictest32764-21lspci09:14:081
71339986,1cyclictest13965-21lspci09:42:381
71339980,7cyclictest4774-21sshd12:28:021
71339980,7cyclictest29220-21sshd11:25:241
71339980,7cyclictest23079-21sshd12:12:011
71339980,7cyclictest23079-21sshd12:12:011
71339980,7cyclictest14390-21sshd11:01:551
71279987,0cyclictest0-21swapper/008:26:360
71279987,0cyclictest0-21swapper/008:08:020
71279980,0cyclictest0-21swapper/008:49:420
71489970,6cyclictest24395-21sshd10:50:183
71489970,6cyclictest16963-21sshd10:25:043
71339976,0cyclictest0-21swapper/112:22:121
71339976,0cyclictest0-21swapper/112:22:121
71339975,1cyclictest15096-21lspci10:05:041
71339973,1cyclictest0-21swapper/110:56:141
71339970,6cyclictest28273-21sshd10:10:571
71279976,0cyclictest0-21swapper/008:56:180
71489963,2cyclictest1398-21snmpd08:03:313
71489963,2cyclictest1398-21snmpd07:31:243
71489963,2cyclictest1398-21snmpd07:22:203
71489963,2cyclictest1397-21snmpd09:05:193
71489963,2cyclictest1397-21snmpd08:46:453
71489963,2cyclictest1397-21snmpd07:43:293
71489963,2cyclictest1397-21snmpd07:39:103
71489963,2cyclictest1397-21snmpd07:13:363
71489960,5cyclictest0-21swapper/309:00:013
71489960,5cyclictest0-21swapper/308:31:563
71489960,5cyclictest0-21swapper/308:25:003
71409964,1cyclictest33-21ksoftirqd/209:04:592
71409963,2cyclictest1398-21snmpd08:37:502
71409963,2cyclictest1398-21snmpd07:32:062
71409963,2cyclictest1397-21snmpd07:47:192
71409961,4cyclictest19143-21crond07:40:002
71409961,3cyclictest33-21ksoftirqd/208:20:002
71409961,2cyclictest33-21ksoftirqd/208:25:142
71409961,2cyclictest27808-21nfsd408:00:162
71409961,1cyclictest33-21ksoftirqd/207:15:162
71409961,1cyclictest33-21ksoftirqd/207:10:202
71409960,5cyclictest0-21swapper/208:07:052
71409960,3cyclictest0-21swapper/209:07:152
71409960,1cyclictest33-21ksoftirqd/208:15:012
71409960,1cyclictest0-21swapper/207:25:002
71409960,0cyclictest0-21swapper/207:25:012
71339965,0cyclictest0-21swapper/108:14:311
71339963,2cyclictest1397-21snmpd08:52:411
71339963,2cyclictest1397-21snmpd08:37:511
71339960,5cyclictest6223-21sshd08:29:471
71339960,5cyclictest17837-21sshd08:59:211
71339960,5cyclictest0-21swapper/107:35:451
71279965,0cyclictest0-21swapper/008:21:160
71279964,1cyclictest1398-21snmpd08:01:190
71279963,2cyclictest1398-21snmpd08:54:570
71279960,0cyclictest0-21swapper/007:25:430
71489953,1cyclictest9588-21hddtemp_smartct07:15:003
71489953,1cyclictest5862sleep308:15:103
71489953,1cyclictest1398-21snmpd08:55:363
71489953,1cyclictest1398-21snmpd08:50:143
71489953,1cyclictest1398-21snmpd08:35:053
71489953,1cyclictest1398-21snmpd08:25:313
71489953,1cyclictest1398-21snmpd08:05:033
71489953,1cyclictest1398-21snmpd07:46:143
71489953,1cyclictest13976-21cat07:25:133
71489952,2cyclictest1397-21snmpd07:55:073
71489952,2cyclictest1397-21snmpd07:55:073
71409953,1cyclictest7287-21awk08:30:152
71409953,1cyclictest1398-21snmpd08:40:102
71409953,1cyclictest1397-21snmpd08:50:052
71409951,2cyclictest121rcu_preempt07:50:112
71409950,3cyclictest0-21swapper/208:55:102
71409950,2cyclictest0-21swapper/208:45:102
71409950,2cyclictest0-21swapper/208:20:022
71339953,1cyclictest7665-21if_eno207:10:161
71339953,1cyclictest29586-21latency08:05:131
71339953,1cyclictest21370-21uptime09:05:231
71339953,1cyclictest18572-21cat09:00:131
71339953,1cyclictest1398-21snmpd08:30:371
71339953,1cyclictest1398-21snmpd07:46:301
71339953,1cyclictest1398-21snmpd07:40:021
71339953,1cyclictest1397-21snmpd08:20:141
71339953,1cyclictest1397-21snmpd08:02:111
71339953,1cyclictest1397-21snmpd07:50:131
71339953,1cyclictest1397-21snmpd07:18:301
71339953,1cyclictest13956-21cat07:25:121
71339953,1cyclictest11167-21ntp_kernel_pll_08:40:161
71339951,3cyclictest12493-21tr08:45:101
71339950,1cyclictest587-21/usr/sbin/munin08:15:111
71339950,1cyclictest11615-21/usr/sbin/munin07:20:121
71279953,1cyclictest6914-21cat08:30:130
71279953,1cyclictest24053-21ntp_kernel_pll_07:50:160
71279953,1cyclictest16259-21ntpdc07:30:210
71279953,1cyclictest1398-21snmpd09:05:170
71279953,1cyclictest1398-21snmpd07:40:200
71279953,1cyclictest1398-21snmpd07:10:020
71279953,1cyclictest1397-21snmpd08:35:330
71279953,1cyclictest1397-21snmpd07:46:320
71279952,2cyclictest1397-21snmpd07:35:500
71279951,1cyclictest18709-21sort09:00:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional