You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-20 - 23:16

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot2.osadl.org (updated Mon Mar 20, 2023 12:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
357723728,0sleep30-21swapper/306:16:423
364423525,0sleep10-21swapper/106:17:241
386723425,0sleep20-21swapper/206:20:022
357222824,0sleep00-21swapper/006:16:390
40339985,0cyclictest19702-21if_eth1.306:31:131
40339983,0cyclictest16560-21awk09:31:131
40339973,0cyclictest31338-21sadc11:40:541
40339973,0cyclictest25812-21awk09:36:121
40339973,0cyclictest2133-21awk09:06:121
40339972,0cyclictest3978-21awk10:51:111
40339970,0cyclictest31763-21munin-node10:11:291
40339970,0cyclictest14368-21diskmemload11:48:571
40339970,0cyclictest0-21swapper/111:06:301
40339970,0cyclictest0-21swapper/108:40:591
28005070,0irq/28-eth0-rx-15013-21kthreadcore10:56:213
8315060,0irq/26-0000:00:20392-21latency_hist09:50:552
40339964,0cyclictest0-21swapper/110:28:391
40339962,0cyclictest9412-21kthreadcore08:51:201
40339962,0cyclictest3346-21snmpd07:33:561
40339962,0cyclictest32721-21awk11:41:151
40339962,0cyclictest19212-21awk08:01:101
40339962,0cyclictest17743-21grep09:46:241
40339962,0cyclictest16617-21cut11:31:161
40339962,0cyclictest14023-21lldpd07:05:361
40339962,0cyclictest11588-21awk09:11:171
40339961,0cyclictest5901-21sh09:59:521
40339961,0cyclictest4683-21awk11:26:131
40339961,0cyclictest3527-21awk10:16:161
40339961,0cyclictest31684-21munin-node06:36:171
40339961,0cyclictest31180-21sh08:28:001
40339961,0cyclictest23261-21kthreadcore11:01:181
40339960,0cyclictest7067-21cpuspeed_turbos10:01:131
40339960,0cyclictest6363-21runrttasks06:41:131
40339960,0cyclictest5269-21irqrtprio07:51:131
40339960,0cyclictest4447-21cut08:11:151
40339960,0cyclictest391-21aten2_r1power_v07:26:081
40339960,0cyclictest32512-21cut10:31:141
40339960,0cyclictest30999-21ssh09:01:461
40339960,0cyclictest28485-21grep10:46:141
40339960,0cyclictest27982-21sh09:52:561
40339960,0cyclictest25714-21kthreadcore06:56:151
40339960,0cyclictest25607-21cat07:20:551
40339960,0cyclictest22190-21cut11:16:221
40339960,0cyclictest21988-21latency10:06:181
40339960,0cyclictest21242-21grep08:21:231
40339960,0cyclictest19818-21ssh09:16:091
40339960,0cyclictest15371-21ssh11:12:411
40339960,0cyclictest14796-21taskset08:56:041
40339960,0cyclictest14756-21ssh10:37:441
40339960,0cyclictest14734-21ssh09:30:071
40339960,0cyclictest13697-21cut06:26:181
40339960,0cyclictest10394-21kthreadcore08:16:191
40339960,0cyclictest0-21swapper/111:24:001
40339960,0cyclictest0-21swapper/110:41:341
339960,0migration/39080-21munin-node10:51:173
339960,0migration/330012-21sh10:30:533
339960,0migration/329715-21sh11:37:433
339960,0migration/317063-21sh08:37:223
339960,0migration/315053-21sh11:46:293
339960,0migration/310603-21kthreadcore08:51:213
329960,0watchdog/37890-21sh11:45:063
28005060,0irq/28-eth0-rx-31268-21df_abs11:06:093
28005060,0irq/28-eth0-rx-30421-21who08:26:293
28005060,0irq/28-eth0-rx-29742sleep311:25:223
28005060,0irq/28-eth0-rx-17748-21memory10:21:233
259960,0migration/21609-21latency_hist09:40:552
149960,0migration/04495-21gltestperf11:26:110
40409955,0cyclictest31994-21ssh08:29:402
40409955,0cyclictest0-21swapper/210:13:142
40409955,0cyclictest0-21swapper/208:58:122
40409955,0cyclictest0-21swapper/208:40:042
40409955,0cyclictest0-21swapper/207:21:242
40409950,0cyclictest0-21swapper/211:50:402
40409950,0cyclictest0-21swapper/208:41:102
40339951,0cyclictest9792-21sh08:35:581
40339951,0cyclictest7727-21cut06:45:561
40339951,0cyclictest0-21swapper/107:06:111
40339950,0cyclictest29158-21aten2_r1power_p09:21:081
40339950,0cyclictest29005-21ls07:45:551
40339950,0cyclictest28836-21df08:46:121
40339950,0cyclictest25511-21aten2_r1power_v08:06:071
40339950,0cyclictest22632-21cut07:41:091
40339950,0cyclictest18514-21aten2_r1power_e07:16:071
40339950,0cyclictest1674-21sh09:41:001
40339950,0cyclictest14996-21wget06:51:071
40339950,0cyclictest14783-21cat07:35:551
40339950,0cyclictest14012-21lldpd08:31:091
40339950,0cyclictest12275-21ssh10:56:031
40339950,0cyclictest11762-21ssh10:21:111
40339950,0cyclictest11494-21tr07:11:181
40339950,0cyclictest11024-21cut07:55:551
40339950,0cyclictest0-21swapper/106:20:561
35150,0ktimersoftd/30-21swapper/307:56:243
339950,0migration/36136-21munin-node08:11:163
339950,0migration/33515-21taskset07:02:303
339950,0migration/330527-21kthreadcore07:21:143
339950,0migration/327255-21kthreadcore09:36:143
339950,0migration/321473-21taskset06:51:253
339950,0migration/319755-21irqcore06:31:143
339950,0migration/318645-21kthreadcore09:31:163
339950,0migration/317091-21idleruntime10:06:143
339950,0migration/316427-21kthreadcore08:21:153
339950,0migration/313756-21munin-node06:46:193
339950,0migration/312376-21irqcore09:46:153
339950,0migration/311669-21cpuspeed_turbos06:26:133
339950,0migration/311382-21diskmemload11:28:233
329950,0watchdog/3838-21sort09:05:553
329950,0watchdog/38182-21perl10:36:103
329950,0watchdog/37425-21taskset08:15:553
329950,0watchdog/35213-21iostat_ios07:06:153
329950,0watchdog/3493-21sh10:13:063
329950,0watchdog/331740-21ssh09:03:153
329950,0watchdog/318577-21kthreadcore07:36:153
329950,0watchdog/314218-21sh11:11:223
329950,0watchdog/30-21swapper/311:01:143
329950,0watchdog/30-21swapper/308:31:193
28015050,0irq/29-eth0-tx-9308-21sensors10:51:192
28015050,0irq/29-eth0-tx-6442-21df_abs09:26:102
28015050,0irq/29-eth0-tx-32665-21wget08:11:080
28015050,0irq/29-eth0-tx-29840-21munin-node09:56:262
28015050,0irq/29-eth0-tx-27580-21ssh09:36:142
28015050,0irq/29-eth0-tx-26763-21ssh09:16:572
28015050,0irq/29-eth0-tx-25642-21munin-node07:21:150
28015050,0irq/29-eth0-tx-22100-21munin-node06:56:100
28015050,0irq/29-eth0-tx-21986-21df_abs11:01:132
28015050,0irq/29-eth0-tx-19087-21kthreadcore08:21:162
28015050,0irq/29-eth0-tx-18460-21munin-node07:16:160
28015050,0irq/29-eth0-tx-16583-21open_inodes09:11:232
28015050,0irq/29-eth0-tx-14187-21port_sshpriv11:11:222
28015050,0irq/29-eth0-tx-0-21swapper/211:45:132
28015050,0irq/29-eth0-tx-0-21swapper/210:57:032
28015050,0irq/29-eth0-tx-0-21swapper/210:41:262
28015050,0irq/29-eth0-tx-0-21swapper/210:08:462
28015050,0irq/29-eth0-tx-0-21swapper/209:48:052
28015050,0irq/29-eth0-tx-0-21swapper/209:31:002
28015050,0irq/29-eth0-tx-0-21swapper/208:50:492
28015050,0irq/29-eth0-tx-0-21swapper/007:28:550
28015050,0irq/29-eth0-tx-0-21swapper/006:51:220
28005050,0irq/28-eth0-rx-8531-21awk10:16:213
28005050,0irq/28-eth0-rx-32050-21cut09:21:163
28005050,0irq/28-eth0-rx-2691-21gltestperf09:41:133
28005050,0irq/28-eth0-rx-26153-21cpu10:46:083
28005050,0irq/28-eth0-rx-20619-21grep09:16:113
28005050,0irq/28-eth0-rx-10806-21diskstats09:11:143
28005050,0irq/28-eth0-rx-0-21swapper/311:31:463
28005050,0irq/28-eth0-rx-0-21swapper/311:15:583
28005050,0irq/28-eth0-rx-0-21swapper/310:40:593
28005050,0irq/28-eth0-rx-0-21swapper/310:31:193
28005050,0irq/28-eth0-rx-0-21swapper/310:03:273
28005050,0irq/28-eth0-rx-0-21swapper/309:56:053
28005050,0irq/28-eth0-rx-0-21swapper/309:53:443
28005050,0irq/28-eth0-rx-0-21swapper/309:26:133
28005050,0irq/28-eth0-rx-0-21swapper/308:56:163
28005050,0irq/28-eth0-rx-0-21swapper/308:47:383
27150,0ktimersoftd/20-21swapper/209:04:112
259950,0migration/228946-21kthreadcore06:36:152
259950,0migration/22840-21sh11:25:072
259950,0migration/227460-21irqrtprio10:46:132
259950,0migration/223400-21irqcore06:56:142
259950,0migration/222027-21kthreadcore07:16:132
259950,0migration/213179-21cpuspeed_turbos11:31:102
249950,0watchdog/230349-21ssh10:31:062
149950,0migration/09822-21kthreadcore06:46:150
149950,0migration/09075-21sh10:16:260
149950,0migration/08901-21sh09:08:380
149950,0migration/06699-21kthreadcore07:51:130
149950,0migration/032400-21fschecks_time11:41:140
149950,0migration/030476-21sh11:05:380
149950,0migration/029567-21kthreadcore06:36:150
149950,0migration/029129-21sh09:55:150
149950,0migration/022618-21munin-node10:26:150
149950,0migration/021200-21kthreadcore06:31:150
149950,0migration/018249-21aten2_r1power_c08:01:060
149950,0migration/017540-21kthreadcore07:36:150
149950,0migration/017437-21sh09:11:580
149950,0migration/017199-21fschecks_count10:41:170
149950,0migration/016610-21kthreadcore07:56:200
149950,0migration/016027-21fschecks_count09:31:110
149950,0migration/015652-21idleruntime08:21:120
149950,0migration/014285-21kthreadcore08:36:150
149950,0migration/012992-21kthreadcore11:46:220
149950,0migration/012958-21sh09:26:350
8315040,0irq/26-0000:00:0-21swapper/207:10:552
40409943,0cyclictest8527-21awk08:16:172
40409943,0cyclictest3070-21irqbalance08:11:522
40409943,0cyclictest3070-21irqbalance08:07:122
40409943,0cyclictest3070-21irqbalance07:51:222
40409943,0cyclictest3070-21irqbalance07:49:422
40409943,0cyclictest3070-21irqbalance07:41:322
40409943,0cyclictest3070-21irqbalance06:26:122
40409942,0cyclictest7052-21kworker/2:110:01:142
40409940,0cyclictest0-21swapper/207:30:512
40279944,0cyclictest0-21swapper/009:47:240
40279944,0cyclictest0-21swapper/009:41:060
40279943,0cyclictest5242-21awk06:21:120
40279943,0cyclictest4051-21awk10:51:110
40279943,0cyclictest3346-21snmpd07:33:050
40279943,0cyclictest30309-21awk07:01:110
40279943,0cyclictest29946-21sed08:46:170
40279943,0cyclictest25321-21awk09:01:230
40279943,0cyclictest14023-21lldpd11:07:130
40279943,0cyclictest14023-21lldpd10:07:110
40279943,0cyclictest13506-21awk10:56:190
40279943,0cyclictest12828-21awk10:21:180
40279943,0cyclictest12557-21awk06:26:170
40279940,0cyclictest0-21swapper/008:59:190
339940,0migration/34642-21kthreadcore06:41:153
339940,0migration/326433-21sh08:42:253
339940,0migration/325586-21cpuspeed_turbos08:06:083
339940,0migration/318768-21fschecks_count08:01:083
329940,0watchdog/330141-21munin-node07:46:113
329940,0watchdog/30-21swapper/307:17:313
329940,0watchdog/30-21swapper/306:21:163
28015040,0irq/29-eth0-tx-5442-21kthreadcore11:26:152
28015040,0irq/29-eth0-tx-5373-21ssh08:50:562
28015040,0irq/29-eth0-tx-30932-21ssh11:06:062
28015040,0irq/29-eth0-tx-18370-21if_err_eth011:16:172
28015040,0irq/29-eth0-tx-0-21swapper/211:35:592
28015040,0irq/29-eth0-tx-0-21swapper/210:25:582
28015040,0irq/29-eth0-tx-0-21swapper/210:21:062
28015040,0irq/29-eth0-tx-0-21swapper/210:16:042
28015040,0irq/29-eth0-tx-0-21swapper/209:21:102
28015040,0irq/29-eth0-tx-0-21swapper/209:05:592
28015040,0irq/29-eth0-tx-0-21swapper/007:46:210
28015040,0irq/29-eth0-tx-0-21swapper/007:41:070
28005040,0irq/28-eth0-rx-25460-21kthreadcore06:56:153
259940,0migration/28529-21kthreadcore07:06:172
259940,0migration/27299-21runrttasks10:35:582
259940,0migration/216595-21kthreadcore06:51:132
259940,0migration/21373-21fschecks_time08:31:112
259940,0migration/211639-21kthreadcore06:46:162
259940,0migration/210995-21latency_hist07:55:552
249940,0watchdog/224426-21chrt08:01:192
249940,0watchdog/214810-21aten2_r1power_c07:36:072
249940,0watchdog/214227-21seq07:32:072
249940,0watchdog/21417-21idleruntime06:41:112
249940,0watchdog/20-21swapper/206:31:162
249940,0watchdog/20-21swapper/206:21:162
149940,0migration/06723-21taskset10:01:000
149940,0migration/06570-21mdstat08:31:170
149940,0migration/027884-21sh08:45:180
149940,0migration/027505-21kthreadcore11:36:230
149940,0migration/020388-21runrttasks11:32:430
149940,0migration/013256-21sh08:53:290
149940,0migration/010612-21kthreadcore08:16:190
40409933,0cyclictest7052-21kworker/2:107:00:572
40279933,0cyclictest14023-21lldpd10:31:110
40279933,0cyclictest0-21swapper/011:18:500
40279933,0cyclictest0-21swapper/009:56:040
40279932,0cyclictest31085-21grep09:21:160
40279932,0cyclictest25869-21awk09:36:120
40279932,0cyclictest24813-21awk08:26:180
40279932,0cyclictest23415-21cut09:16:160
40279931,0cyclictest7976-21fschecks_count10:36:100
40279931,0cyclictest1111-21tune2fs06:41:100
40279931,0cyclictest0-21swapper/011:11:120
40279930,0cyclictest5551-21irqstats07:06:160
40279930,0cyclictest26651-21perl10:46:100
40279930,0cyclictest26143-21perl08:06:090
40279930,0cyclictest12040-21perl07:11:200
40279930,0cyclictest0-21swapper/010:11:280
329930,0watchdog/30-21swapper/307:50:553
329930,0watchdog/30-21swapper/307:42:353
329930,0watchdog/30-21swapper/307:31:193
329930,0watchdog/30-21swapper/307:26:113
28005030,0irq/28-eth0-rx-0-21swapper/307:11:283
28005030,0irq/28-eth0-rx-0-21swapper/306:36:083
149930,0migration/01028-21sh11:21:360
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional