You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-28 - 07:27

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.39-rt50+ (Profile)

Latency plot of system in rack #7, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rack7slot0.osadl.org (updated Sat Jan 28, 2023 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
110050720,0irq/25-eth00-21swapper/319:06:233
117950710,0irq/26-eth1-rx-0-21swapper/119:06:231
117950610,0irq/26-eth1-rx-0-21swapper/019:06:100
110050560,0irq/25-eth00-21swapper/219:06:162
117950290,0irq/26-eth1-rx-0-21swapper/119:10:001
110050240,0irq/25-eth00-21swapper/219:10:002
952991918,0cyclictest0-21swapper/300:05:193
951991816,0cyclictest32058-21latency_hist23:29:592
351180,0ktimersoftd/319744-21id22:24:543
950991716,0cyclictest30618-21sshd22:18:121
95299150,0cyclictest0-21swapper/322:47:423
95199150,0cyclictest0-21swapper/222:32:382
949991512,0cyclictest28449-21strings00:35:190
110050150,0irq/25-eth026791-21sshd21:57:240
952991413,0cyclictest0-21swapper/323:50:063
952991413,0cyclictest0-21swapper/322:44:343
951991413,0cyclictest0-21swapper/222:17:042
95199141,0cyclictest25186-21bash21:45:072
94999140,0cyclictest0-21swapper/022:15:140
94999140,0cyclictest0-21swapper/022:09:180
41140,0ktimersoftd/011864-21sshd22:03:440
110050140,0irq/25-eth024348-21sshd21:49:010
110050140,0irq/25-eth00-21swapper/022:25:310
952991312,0cyclictest0-21swapper/322:19:363
952991311,0cyclictest0-21swapper/321:23:023
952991310,0cyclictest23296-21sshd23:20:493
95299130,0cyclictest0-21swapper/323:14:533
951991312,0cyclictest32514-21sshd22:42:172
951991312,0cyclictest30763-21sshd22:10:122
951991312,0cyclictest0-21swapper/223:36:252
951991312,0cyclictest0-21swapper/223:19:192
951991312,0cyclictest0-21swapper/222:23:552
951991312,0cyclictest0-21swapper/221:39:322
951991312,0cyclictest0-21swapper/200:16:002
951991311,0cyclictest0-21swapper/200:04:162
95199130,0cyclictest0-21swapper/222:07:262
95199130,0cyclictest0-21swapper/221:14:482
950991312,0cyclictest32266-21sshd22:46:131
950991312,0cyclictest15372-21sshd22:52:071
950991312,0cyclictest14475-21sshd21:39:451
950991311,0cyclictest0-21swapper/123:22:191
949991312,0cyclictest30382-21bash00:01:380
949991312,0cyclictest0-21swapper/023:02:160
949991312,0cyclictest0-21swapper/021:11:420
949991310,0cyclictest0-21swapper/022:23:010
94999130,0cyclictest0-21swapper/023:15:040
41130,0ktimersoftd/00-21swapper/022:11:130
117950130,0irq/26-eth1-rx-24221-21id22:53:160
110050130,0irq/25-eth032295-21date21:50:010
952991212,0cyclictest0-21swapper/300:04:233
952991211,0cyclictest26883-21sshd22:33:393
952991211,0cyclictest0-21swapper/300:33:333
95299120,0cyclictest25157-21diskmemload21:44:573
95299120,0cyclictest110050irq/25-eth022:55:503
95299120,0cyclictest0-21swapper/322:51:473
95299120,0cyclictest0-21swapper/300:36:193
951991211,0cyclictest9831-21bash00:28:402
951991211,0cyclictest30244-21sshd22:50:032
951991210,0cyclictest0-21swapper/200:36:422
95199120,0cyclictest0-21swapper/221:43:162
95199120,0cyclictest0-21swapper/220:20:122
95199120,0cyclictest0-21swapper/219:55:202
95099124,0cyclictest5960-21sshd00:15:221
950991212,0cyclictest0-21swapper/100:01:431
950991211,0cyclictest5541-21bash22:58:591
950991211,0cyclictest25157-21diskmemload22:26:551
950991211,0cyclictest0-21swapper/100:22:231
950991211,0cyclictest0-21swapper/100:11:361
950991210,0cyclictest0-21swapper/123:45:541
95099120,0cyclictest0-21swapper/123:44:581
95099120,0cyclictest0-21swapper/123:12:421
95099120,0cyclictest0-21swapper/122:33:471
95099120,0cyclictest0-21swapper/121:14:051
949991210,0cyclictest0-21swapper/022:36:440
94999120,0cyclictest16057-21sshd00:20:480
94999120,0cyclictest0-21swapper/023:08:130
41120,0ktimersoftd/07622-21bash22:55:140
41120,0ktimersoftd/00-21swapper/000:16:540
351120,0ktimersoftd/30-21swapper/321:37:483
271120,0ktimersoftd/20-21swapper/221:22:282
110050120,0irq/25-eth02564-21sshd21:18:491
952991110,0cyclictest30705-21sshd21:25:493
952991110,0cyclictest26778-21sshd00:18:093
952991110,0cyclictest16826-21sshd23:40:073
952991110,0cyclictest13975-21sshd21:59:583
952991110,0cyclictest0-21swapper/323:35:273
952991110,0cyclictest0-21swapper/300:29:503
95299110,0cyclictest0-21swapper/323:56:023
951991110,0cyclictest21773-21sshd22:56:552
951991110,0cyclictest12877-21bash23:03:452
95099119,0cyclictest0-21swapper/119:15:081
950991111,0cyclictest0-21swapper/121:40:411
950991111,0cyclictest0-21swapper/120:04:301
950991110,0cyclictest0-21swapper/122:11:221
950991110,0cyclictest0-21swapper/120:55:301
950991110,0cyclictest0-21swapper/119:53:501
95099110,0cyclictest0-21swapper/120:40:181
95099110,0cyclictest0-21swapper/119:41:511
94999119,0cyclictest0-21swapper/020:38:070
949991111,0cyclictest0-21swapper/023:22:500
949991111,0cyclictest0-21swapper/019:10:250
949991110,0cyclictest21109-21hddtemp_smartct19:50:150
949991110,0cyclictest15475-21bash23:59:470
949991110,0cyclictest0-21swapper/020:46:510
41110,0ktimersoftd/025157-21diskmemload23:11:090
41110,0ktimersoftd/00-21swapper/023:25:160
351110,0ktimersoftd/311054-21sshd23:47:123
110050110,0irq/25-eth08-21rcu_preempt22:14:513
110050110,0irq/25-eth027293-21sshd21:21:370
110050110,0irq/25-eth025204-21sshd22:45:200
110050110,0irq/25-eth017522-21sshd21:52:111
110050110,0irq/25-eth00-21swapper/123:59:021
110050110,0irq/25-eth00-21swapper/023:33:110
110050110,0irq/25-eth00-21swapper/022:40:340
110050110,0irq/25-eth00-21swapper/022:30:120
95299109,0cyclictest0-21swapper/322:29:363
95299109,0cyclictest0-21swapper/321:50:193
952991010,0cyclictest0-21swapper/323:15:503
95199109,0cyclictest7923-21sshd21:59:112
95199109,0cyclictest0-21swapper/220:01:022
951991010,0cyclictest0-21swapper/223:11:332
95199100,0cyclictest0-21swapper/223:53:152
95199100,0cyclictest0-21swapper/220:42:232
95099109,0cyclictest5682-21sshd23:06:451
95099109,0cyclictest0-21swapper/122:35:391
95099109,0cyclictest0-21swapper/121:34:121
950991010,0cyclictest25157-21diskmemload00:33:111
950991010,0cyclictest0-21swapper/120:24:081
950991010,0cyclictest0-21swapper/120:08:341
94999109,0cyclictest0-21swapper/023:51:260
94999109,0cyclictest0-21swapper/020:55:320
94999109,0cyclictest0-21swapper/019:33:110
94999105,0cyclictest0-21swapper/000:12:160
949991010,0cyclictest0-21swapper/020:28:290
94999100,0cyclictest0-21swapper/020:54:370
94999100,0cyclictest0-21swapper/020:14:460
94999100,0cyclictest0-21swapper/019:45:510
41100,0ktimersoftd/07116-21sshd23:42:480
41100,0ktimersoftd/06136-21sshd00:28:100
41100,0ktimersoftd/017106-21cp23:36:070
41100,0ktimersoftd/00-21swapper/021:41:590
191100,0ktimersoftd/116777-21sshd21:24:201
1499100,0migration/07794-21sshd21:15:270
110050100,0irq/25-eth016498-21sshd23:04:133
110050100,0irq/25-eth00-21swapper/322:03:133
9529999,0cyclictest0-21swapper/319:56:293
9529998,0cyclictest8757-21sshd00:24:103
9529990,0cyclictest0-21swapper/322:35:213
9529990,0cyclictest0-21swapper/321:46:193
9529990,0cyclictest0-21swapper/321:33:363
9519998,0cyclictest16910-21sshd23:40:072
9519998,0cyclictest0-21swapper/223:20:162
9519998,0cyclictest0-21swapper/219:43:362
9519990,0cyclictest0-21swapper/221:26:272
9519990,0cyclictest0-21swapper/221:16:572
9519990,0cyclictest0-21swapper/221:07:222
9519990,0cyclictest0-21swapper/221:01:272
9509999,0cyclictest0-21swapper/123:25:081
9509999,0cyclictest0-21swapper/120:25:041
9509999,0cyclictest0-21swapper/120:17:181
9509999,0cyclictest0-21swapper/119:30:281
9509998,0cyclictest1461-21sshd21:26:131
9509997,0cyclictest0-21swapper/119:29:121
9509990,0cyclictest0-21swapper/121:49:571
9499999,0cyclictest0-21swapper/021:07:020
9499999,0cyclictest0-21swapper/020:01:180
9499999,0cyclictest0-21swapper/019:56:530
9499999,0cyclictest0-21swapper/019:20:240
9499998,0cyclictest0-21swapper/000:34:240
9499992,0cyclictest0-21swapper/020:05:170
9499991,0cyclictest0-21swapper/020:30:160
9499990,0cyclictest0-21swapper/021:25:250
9499990,0cyclictest0-21swapper/019:40:150
4190,0ktimersoftd/026357-21sshd21:37:070
11795090,0irq/26-eth1-rx-31693-21sshd21:58:031
11795090,0irq/26-eth1-rx-28-21ksoftirqd/222:01:422
11795090,0irq/26-eth1-rx-20836-21bash22:28:552
11795090,0irq/26-eth1-rx-0-21swapper/122:01:581
11005090,0irq/25-eth09814-21bash21:31:030
11005090,0irq/25-eth08153-21sshd00:28:261
11005090,0irq/25-eth04944-21sshd23:06:402
11005090,0irq/25-eth04298-21sshd21:54:412
11005090,0irq/25-eth028-21ksoftirqd/200:09:572
11005090,0irq/25-eth026495-21sshd00:05:250
11005090,0irq/25-eth020-21ksoftirqd/123:04:141
11005090,0irq/25-eth019380-21sshd22:20:431
11005090,0irq/25-eth012000-21sshd22:07:451
9529988,0cyclictest0-21swapper/323:05:513
9529987,0cyclictest22238-21sshd22:05:023
9519988,0cyclictest0-21swapper/220:56:262
9519988,0cyclictest0-21swapper/220:10:482
9519987,0cyclictest917-21sshd21:33:582
9519987,0cyclictest4694-21expr20:25:122
9519987,0cyclictest0-21swapper/200:14:102
9509988,0cyclictest0-21swapper/119:59:091
9509985,0cyclictest0-21swapper/122:43:021
9499988,0cyclictest0-21swapper/020:21:370
9499988,0cyclictest0-21swapper/020:15:420
9499987,0cyclictest3-21ksoftirqd/023:45:150
11005080,0irq/25-eth04088-21sshd23:18:331
11005080,0irq/25-eth029021-21sshd23:53:261
11005080,0irq/25-eth025157-21diskmemload00:14:043
11005080,0irq/25-eth024248-21sshd00:05:131
11005080,0irq/25-eth016096-21sshd23:31:583
11005080,0irq/25-eth00-21swapper/323:27:573
11005080,0irq/25-eth00-21swapper/321:10:593
11005080,0irq/25-eth00-21swapper/222:46:402
9529977,0cyclictest0-21swapper/319:25:143
9529976,0cyclictest7163-21kworker/3:220:40:203
9529972,0cyclictest0-21swapper/319:45:173
9529970,0cyclictest0-21swapper/319:15:023
9519977,0cyclictest0-21swapper/220:51:272
9519977,0cyclictest0-21swapper/220:17:042
9519977,0cyclictest0-21swapper/219:35:012
9519976,0cyclictest32612-21sshd00:31:332
9519970,0cyclictest0-21swapper/200:24:152
9509976,0cyclictest0-21swapper/120:10:251
9509970,0cyclictest0-21swapper/123:35:181
9509970,0cyclictest0-21swapper/119:35:161
9499970,0cyclictest0-21swapper/020:40:120
169970,0watchdog/19047-21wget20:35:111
159970,0watchdog/00-21swapper/019:35:150
11795070,0irq/26-eth1-rx-9376-21sshd22:35:232
11795070,0irq/26-eth1-rx-0-21swapper/223:32:352
11005070,0irq/25-eth036-21ksoftirqd/321:15:003
11005070,0irq/25-eth028-21ksoftirqd/223:46:312
11005070,0irq/25-eth025157-21diskmemload00:35:031
11005070,0irq/25-eth020-21ksoftirqd/123:30:101
11005070,0irq/25-eth014895-21sshd23:55:322
9529965,0cyclictest7163-21kworker/3:221:09:583
9529965,0cyclictest7163-21kworker/3:219:35:403
9529962,0cyclictest0-21swapper/321:00:183
9529962,0cyclictest0-21swapper/320:55:253
9529962,0cyclictest0-21swapper/320:50:233
9529962,0cyclictest0-21swapper/320:45:163
9529962,0cyclictest0-21swapper/320:35:163
9529962,0cyclictest0-21swapper/320:25:153
9529962,0cyclictest0-21swapper/320:20:163
9529962,0cyclictest0-21swapper/320:15:163
9529962,0cyclictest0-21swapper/320:10:163
9529962,0cyclictest0-21swapper/320:05:183
9529962,0cyclictest0-21swapper/319:40:173
9529962,0cyclictest0-21swapper/319:30:163
9529962,0cyclictest0-21swapper/319:20:163
9529961,0cyclictest0-21swapper/319:10:153
9519966,0cyclictest0-21swapper/220:34:152
9519966,0cyclictest0-21swapper/219:27:212
9519966,0cyclictest0-21swapper/219:21:542
9519962,0cyclictest0-21swapper/220:35:162
9519962,0cyclictest0-21swapper/219:50:172
9519962,0cyclictest0-21swapper/219:15:002
9519960,0cyclictest0-21swapper/220:05:162
9509965,0cyclictest24332-21proc_pri21:05:271
9509962,0cyclictest0-21swapper/121:00:181
9509962,0cyclictest0-21swapper/120:50:221
9509962,0cyclictest0-21swapper/120:30:151
9509962,0cyclictest0-21swapper/119:45:161
9509962,0cyclictest0-21swapper/119:20:161
9509961,0cyclictest0-21swapper/120:45:121
9499965,0cyclictest8615-21sed19:25:000
339960,0migration/37204-21fschecks_count20:30:133
339960,0migration/325106-21munin-node20:00:123
149960,0migration/03659-21cpuspeed_turbos19:15:020
11795060,0irq/26-eth1-rx-0-21swapper/319:50:153
11005060,0irq/25-eth00-21swapper/021:00:240
9519954,0cyclictest11987-21munin-node19:30:172
9519951,0cyclictest0-21swapper/219:45:172
9519950,0cyclictest0-21swapper/220:45:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional