You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-22 - 03:40

x86 Intel Xeon-E31220L @2200 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #5, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the 376 highest latencies:
System rack5slot0.osadl.org (updated Wed Mar 22, 2023 00:44:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1939121410,3sleep21685899cyclictest21:22:512
1209421380,4sleep11877-21nfsd00:20:201
2987721240,3sleep01685699cyclictest23:04:280
1352721190,2sleep213522-21bash22:54:002
16613210633,69sleep10-21swapper/119:09:131
16647210333,40sleep30-21swapper/319:09:393
214862870,4sleep10-21swapper/123:06:381
300542850,4sleep20-21swapper/223:53:362
242712850,3sleep124270-21bash22:10:271
118042840,0sleep011694-21sshd23:23:400
226572830,0sleep022618-21sshd23:37:060
266432810,1sleep31877-21nfsd22:25:313
168202810,2sleep10-21swapper/121:46:051
324862790,3sleep30-21swapper/322:55:383
262292790,2sleep01685699cyclictest00:31:060
87722770,3sleep01685699cyclictest22:53:340
181442770,3sleep30-21swapper/322:12:533
298562760,4sleep31685999cyclictest00:09:283
322482750,1sleep21685899cyclictest22:58:412
102372750,4sleep11685799cyclictest22:21:001
101032750,0sleep31685999cyclictest23:42:113
28052740,1sleep22798-21sshd23:32:042
269012740,2sleep10-21swapper/123:40:361
135692710,2sleep30-21swapper/300:17:173
193492700,2sleep119334-21sshd22:36:441
1635627033,10sleep20-21swapper/219:06:002
73682680,3sleep21685899cyclictest00:16:402
311382680,3sleep21685899cyclictest22:05:152
302482670,3sleep21685899cyclictest00:22:012
123912620,2sleep11877-21nfsd00:26:371
265212590,57sleep228750irq/26-ahci[00000:37:292
1641225733,13sleep00-21swapper/019:06:390
77722500,0sleep07774-21sshd22:00:080
205022490,1sleep220504-21bash22:30:512
271702430,1sleep10-21swapper/122:55:161
294952380,3sleep30-21swapper/300:37:453
104552350,1sleep00-21swapper/000:20:110
1685999334,19cyclictest7572-21sshd23:38:503
1685999332,29cyclictest1229-21sshd21:15:193
300142320,3sleep330009-21sshd22:43:393
89102310,2sleep20-21swapper/222:23:552
267452300,28sleep1138550irq/27-eth021:38:141
1685999304,23cyclictest4905-21sshd22:35:233
1685999304,23cyclictest30197-21sshd23:22:203
1685999282,8cyclictest22607-21sshd23:34:003
16856992822,5cyclictest26188-21sshd00:09:050
149372281,25sleep3138550irq/27-eth022:03:453
60282270,25sleep2138550irq/27-eth023:29:202
283422270,2sleep00-21swapper/020:45:210
1685999274,8cyclictest28436-21sshd00:34:313
146102270,3sleep30-21swapper/300:20:303
49552260,5sleep31685999cyclictest23:14:053
296052260,0sleep00-21swapper/023:10:220
29482260,2sleep20-21swapper/223:44:422
1685999260,24cyclictest20090-21sshd21:37:373
6762250,1sleep20-21swapper/223:47:352
1685999256,16cyclictest21226-21perf19:19:593
1685999254,5cyclictest0-21swapper/323:55:023
96092240,3sleep11685799cyclictest23:14:311
78322240,2sleep20-21swapper/221:13:012
65682240,2sleep11877-21nfsd23:20:121
1685999245,7cyclictest1089-21sshd22:46:533
1685999244,18cyclictest912-21bash00:28:423
1685999243,6cyclictest138550irq/27-eth022:54:063
1685999240,3cyclictest12206-21sshd22:09:273
132282240,1sleep041ktimersoftd/022:12:260
86142230,5sleep10-21swapper/121:24:501
76412230,3sleep21685899cyclictest23:11:202
293192230,2sleep129303-21sshd21:17:531
254622230,21sleep2138550irq/27-eth021:55:452
1685999237,7cyclictest19376-21sshd21:58:123
1685999236,11cyclictest13291-21sshd23:26:533
1685999235,6cyclictest2909-21rm23:47:473
1685999234,6cyclictest138550irq/27-eth021:24:473
1685999233,9cyclictest7768-21sshd21:51:113
16859992319,1cyclictest44-21ksoftirqd/321:27:423
1685999228,12cyclictest13616-21sshd22:30:193
1685999225,9cyclictest13380-21sshd00:10:543
1685999223,8cyclictest19299-21sshd21:34:383
1685999223,8cyclictest19299-21sshd21:34:373
1685999223,7cyclictest4039-21id21:42:013
16859992216,4cyclictest6775-21bash21:10:033
16859992211,9cyclictest23319-21bash21:46:443
1685699223,15cyclictest16855-21cyclictest22:21:570
315912210,3sleep21685899cyclictest21:41:342
253432210,1sleep20-21swapper/200:27:542
1685999219,10cyclictest11131-21sshd00:01:153
1685999216,7cyclictest5926-21bash22:17:423
1685999215,9cyclictest23588-21consoletype23:06:503
1685999212,8cyclictest138550irq/27-eth023:51:533
1685999212,8cyclictest11902-21sshd22:21:103
1685999211,13cyclictest147450irq/28-eth1-rx-23:01:583
310432200,2sleep131038-21sshd23:31:381
22772200,2sleep20-21swapper/200:03:342
1685999205,7cyclictest32048-21cpuspeed_turbos20:55:143
1685999205,7cyclictest26283-21sshd23:15:593
16859992012,6cyclictest2087-21runrttasks19:47:133
16859992012,6cyclictest2087-21runrttasks19:47:133
16859992011,7cyclictest32406-21runrttasks19:44:323
1685899204,3cyclictest9132-21kworker/u8:121:46:522
1685899203,14cyclictest0-21swapper/219:49:482
1685899203,14cyclictest0-21swapper/219:49:472
1685699200,19cyclictest7319-21sshd21:18:500
139252200,2sleep10-21swapper/121:54:481
306072190,1sleep20-21swapper/222:25:512
26522190,0sleep00-21swapper/000:28:530
264912190,2sleep126483-21sshd23:50:061
1685999193,14cyclictest18969-21munin-node20:25:213
16859991912,5cyclictest25940-21munin-node19:30:193
1685699199,4cyclictest0-21swapper/022:57:350
1685699198,4cyclictest0-21swapper/022:16:040
16856991912,6cyclictest0-21swapper/022:45:210
1685699190,10cyclictest0-21swapper/023:33:430
167492190,2sleep116748-21bash23:39:441
178392180,0sleep00-21swapper/000:39:510
1685999183,7cyclictest5374-21fschecks_time19:55:143
1685999183,3cyclictest0-21swapper/320:40:153
1685999182,14cyclictest2952-21munin-node21:00:213
16859991811,6cyclictest138550irq/27-eth020:35:213
1685999181,13cyclictest138550irq/27-eth020:55:013
16859991810,6cyclictest17864-21runrttasks20:22:173
1685899183,9cyclictest30203-21sa119:40:002
1685899183,10cyclictest23549-21sshd22:10:242
1685899183,10cyclictest22552-21sshd22:45:492
1685899182,8cyclictest36-21rcuop/223:06:252
1685899182,7cyclictest30757-21bash22:37:482
1685699188,4cyclictest0-21swapper/023:40:510
1685699187,4cyclictest5335-21bash21:21:330
1685699181,11cyclictest31851-21sshd22:31:570
1685699180,6cyclictest0-21swapper/023:27:070
1685999175,10cyclictest21690-21taskset20:30:203
1685999173,5cyclictest4346-21runrttasks19:52:113
16859991713,2cyclictest29159-21munin-node19:35:193
16859991712,3cyclictest0-21swapper/320:00:243
1685899175,11cyclictest138550irq/27-eth021:37:242
1685899173,7cyclictest23044-21bash23:15:402
1685899172,3cyclictest23338-21bash21:35:002
1685899172,3cyclictest23338-21bash21:35:002
16858991713,3cyclictest138550irq/27-eth023:04:252
16858991711,5cyclictest138550irq/27-eth020:13:552
1685899171,13cyclictest147450irq/28-eth1-rx-21:53:202
1685899171,10cyclictest15197-21memory21:25:212
1685899170,14cyclictest36-21rcuop/223:56:182
1685899170,14cyclictest138550irq/27-eth019:41:062
1685799171,8cyclictest22-21ksoftirqd/123:18:211
1685799170,16cyclictest22-21ksoftirqd/123:28:481
1685799170,15cyclictest2665-21sshd22:29:191
1685799170,15cyclictest14924-21sshd22:33:241
1685799170,12cyclictest21814-21sshd21:11:181
1685699178,8cyclictest31111-21bash21:29:470
1685699178,3cyclictest550-21bash00:16:000
1685699178,2cyclictest0-21swapper/020:40:160
1685699177,4cyclictest27360-21sshd23:07:120
1685699177,2cyclictest0-21swapper/019:55:230
1685699174,5cyclictest1987-21sshd22:29:520
16856991712,4cyclictest0-21swapper/020:30:160
1685699170,1cyclictest0-21swapper/022:44:070
1685699170,16cyclictest0-21swapper/023:16:250
1685699170,16cyclictest0-21swapper/021:53:010
1685699170,11cyclictest8953-21bash22:35:450
1685699170,11cyclictest21053-21bash21:46:310
1685699170,11cyclictest1874-21nfsd22:09:150
1685699170,11cyclictest14924-21sshd21:37:080
1685699170,11cyclictest0-21swapper/021:57:190
1685699170,10cyclictest1987-21sshd21:42:140
113852170,4sleep11685799cyclictest23:58:121
102412172,13sleep2138550irq/27-eth021:16:072
1685999169,6cyclictest1877-21nfsd19:12:033
1685999169,6cyclictest0-21swapper/319:20:343
1685999162,5cyclictest1877-21nfsd20:05:563
16859991614,1cyclictest0-21swapper/320:15:183
16859991612,3cyclictest5689-21runrttasks21:06:463
16859991610,5cyclictest1877-21nfsd20:10:533
1685899168,3cyclictest20875-21sshd23:21:272
1685899163,8cyclictest138550irq/27-eth020:05:102
1685899163,8cyclictest138550irq/27-eth000:12:322
1685899163,8cyclictest11319-21cp22:03:232
1685899163,5cyclictest0-21swapper/220:44:072
1685899162,10cyclictest0-21swapper/200:30:092
1685899161,9cyclictest19973-21bash00:05:202
1685899160,15cyclictest147450irq/28-eth1-rx-22:42:122
1685799166,8cyclictest3574-21sshd21:59:451
1685799164,11cyclictest36-21rcuop/222:17:471
1685799160,15cyclictest0-21swapper/123:03:171
1685799160,14cyclictest10118-21sshd22:06:131
1685799160,11cyclictest17627-21sshd22:54:231
1685699167,6cyclictest0-21swapper/020:36:370
1685699167,3cyclictest2999-21sshd00:00:270
1685699167,3cyclictest0-21swapper/021:30:330
1685699167,3cyclictest0-21swapper/021:30:330
1685699164,5cyclictest13758-21sshd23:45:390
16856991613,1cyclictest0-21swapper/000:14:050
1685699160,9cyclictest0-21swapper/019:35:160
1685699160,13cyclictest0-21swapper/021:12:420
1685699160,12cyclictest5077-21consoletype23:51:040
1685699160,11cyclictest0-21swapper/019:25:210
112662161,13sleep2138550irq/27-eth023:35:572
93802150,13sleep2138550irq/27-eth022:15:052
1685999156,7cyclictest24773-21nfsd419:25:203
16859991511,3cyclictest27309-21munin-node20:45:193
1685899151,13cyclictest138550irq/27-eth019:21:462
1685899151,10cyclictest36-21rcuop/220:30:212
1685799157,3cyclictest9352-21sshd23:48:261
1685799150,6cyclictest2373-21sshd22:41:041
1685799150,6cyclictest1877-21nfsd19:35:201
1685799150,3cyclictest22746-21sshd21:31:561
1685799150,3cyclictest22746-21sshd21:31:551
1685799150,13cyclictest25390-21id21:26:171
1685799150,13cyclictest23306-21sshd00:15:091
1685799150,13cyclictest0-21swapper/100:05:301
1685699150,1cyclictest15198-21sshd23:58:350
1685699150,12cyclictest19418-21munin-node20:25:280
1685699150,12cyclictest17605-21hddtemp_smartct19:10:150
1685899148,5cyclictest1877-21nfsd19:15:142
1685899148,3cyclictest138550irq/27-eth021:06:002
1685899144,9cyclictest147450irq/28-eth1-rx-20:29:002
1685899142,9cyclictest147450irq/28-eth1-rx-20:55:042
1685899142,7cyclictest23711-21cat20:35:182
1685899142,7cyclictest1877-21nfsd20:51:062
1685899142,1cyclictest0-21swapper/221:04:122
1685899140,11cyclictest0-21swapper/219:25:482
1685799144,9cyclictest26576-21bash22:01:481
16857991410,3cyclictest4683-21iostat21:05:211
1685799140,5cyclictest0-21swapper/120:23:141
1685799140,3cyclictest0-21swapper/121:40:031
1685799140,13cyclictest16736-21sshd00:01:481
1685799140,13cyclictest138550irq/27-eth000:36:411
1685799140,13cyclictest13814-21sshd00:10:571
1685799140,11cyclictest20225-21sshd22:45:371
1685699140,8cyclictest17606-21unixbench_multi20:20:290
1685899137,5cyclictest0-21swapper/219:33:502
1685899133,9cyclictest25-21rcuop/120:49:082
1685899133,5cyclictest138550irq/27-eth019:12:102
1685899133,5cyclictest0-21swapper/220:23:072
1685899131,2cyclictest0-21swapper/219:54:442
1685899130,5cyclictest0-21swapper/219:55:472
1685899130,12cyclictest0-21swapper/220:00:132
1685799138,4cyclictest0-21swapper/119:21:051
1685799132,1cyclictest0-21swapper/120:30:041
16857991311,1cyclictest0-21swapper/121:00:271
16857991311,1cyclictest0-21swapper/120:12:171
1685799130,8cyclictest22-21ksoftirqd/119:14:591
1685799130,5cyclictest0-21swapper/119:55:251
1685799130,4cyclictest10062-21ipmi_fans20:05:181
1685799130,2cyclictest138550irq/27-eth020:15:411
1685799130,12cyclictest1877-21nfsd20:40:241
1685799130,12cyclictest0-21swapper/120:58:321
1685799130,12cyclictest0-21swapper/120:53:231
1685799130,12cyclictest0-21swapper/120:46:551
1685799130,12cyclictest0-21swapper/120:37:381
1685799130,12cyclictest0-21swapper/120:25:371
1685799130,12cyclictest0-21swapper/120:02:321
1685799130,12cyclictest0-21swapper/119:32:321
1685799130,11cyclictest15201-21sshd00:30:061
1685699131,1cyclictest0-21swapper/019:42:120
16856991311,1cyclictest0-21swapper/020:55:090
1685899120,11cyclictest13799-21taskset20:15:002
1685799120,3cyclictest24262-21cut19:25:141
1685799120,3cyclictest1361-21ntp_states19:45:251
1685799120,3cyclictest1361-21ntp_states19:45:241
1685799120,11cyclictest2559-21munin-node19:50:111
1685799120,11cyclictest0-21swapper/119:40:271
1685699128,1cyclictest0-21swapper/020:05:330
1685699121,8cyclictest0-21swapper/019:53:040
1685699120,2cyclictest0-21swapper/021:05:000
1685699120,2cyclictest0-21swapper/020:15:170
1685699120,2cyclictest0-21swapper/019:15:380
1685699120,11cyclictest0-21swapper/021:00:430
1685699120,11cyclictest0-21swapper/019:21:550
1685799110,10cyclictest19233-21atenall_r5power19:15:151
1685699119,1cyclictest0-21swapper/020:10:290
1685699117,2cyclictest0-21swapper/019:31:480
1685699112,1cyclictest0-21swapper/019:45:360
1685699112,1cyclictest0-21swapper/019:45:360
1685699110,1cyclictest0-21swapper/020:51:020
1685699110,1cyclictest0-21swapper/020:01:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional