You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-05-20 - 07:08

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of system in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2.osadl.org (updated Fri May 20, 2022 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2589024028,0sleep20-21swapper/218:55:572
2570423627,0sleep30-21swapper/318:53:463
2584523425,0sleep10-21swapper/118:55:241
2578222824,0sleep00-21swapper/018:54:390
249970,0watchdog/20-21swapper/223:12:592
339960,0migration/330258-21sh00:07:153
339960,0migration/32923-21sh23:51:413
339960,0migration/322143-21runrttasks23:27:233
339960,0migration/312615-21sh22:11:583
339960,0migration/310715-21munin-node20:13:243
329960,0watchdog/322697-21sed22:52:593
28005060,0irq/28-eth0-rx-31170-21df00:08:103
28005060,0irq/28-eth0-rx-18575-21diskstats23:08:113
28005060,0irq/28-eth0-rx-0-21swapper/322:42:433
261459960,0cyclictest0-21swapper/223:28:162
261379960,0cyclictest0-21swapper/123:16:531
261369965,0cyclictest3861-21crond22:07:590
261369960,0cyclictest0-21swapper/021:02:060
149960,0migration/03628-21sh23:17:170
149960,0migration/032298-21fschecks_count21:48:120
149960,0migration/012475-21fschecks_time20:18:130
8315050,0irq/26-0000:00:7763-21idleruntime-cro20:38:002
8315050,0irq/26-0000:00:7763-21idleruntime-cro20:38:002
339950,0migration/39042-21aten2_r1power_p23:03:083
339950,0migration/36275-21kthreadcore23:53:163
339950,0migration/35655-21idleruntime19:28:143
339950,0migration/34688-21kthreadcore20:33:173
339950,0migration/34688-21kthreadcore20:33:173
339950,0migration/332285-21cpuspeed_turbos22:58:093
339950,0migration/329872-21kthreadcore18:58:173
339950,0migration/329306-21kthreadcore22:03:163
339950,0migration/328488-21munin-node19:18:193
339950,0migration/327778-21kthreadcore20:28:163
339950,0migration/324788-21munin-node20:23:203
339950,0migration/323272-21fschecks_time21:08:123
339950,0migration/322187-21latency_hist20:48:003
339950,0migration/318590-21fschecks_count23:43:113
339950,0migration/310023-21irqrtprio21:18:163
329950,0watchdog/38319-21tune2fs00:13:183
329950,0watchdog/329723-21ssh22:53:433
329950,0watchdog/325661-21grep00:23:103
329950,0watchdog/317467-21kthreadcore19:33:203
329950,0watchdog/315966-21kthreadcore23:58:143
329950,0watchdog/30-21swapper/323:38:113
329950,0watchdog/30-21swapper/320:03:163
28015050,0irq/29-eth0-tx-9824-21iostat_ios21:18:162
28015050,0irq/29-eth0-tx-7582-40yum-updatesd-he20:36:230
28015050,0irq/29-eth0-tx-7582-40yum-updatesd-he20:36:230
28015050,0irq/29-eth0-tx-27153-21munin-node21:28:242
28015050,0irq/29-eth0-tx-22575-21munin-node21:43:142
28015050,0irq/29-eth0-tx-20902-21ssh22:14:582
28015050,0irq/29-eth0-tx-17335-21ssh23:06:512
28015050,0irq/29-eth0-tx-15528-21munin-node19:13:180
28015050,0irq/29-eth0-tx-1092-21munin-node19:03:130
28015050,0irq/29-eth0-tx-0-21swapper/223:23:162
28015050,0irq/29-eth0-tx-0-21swapper/222:53:102
28015050,0irq/29-eth0-tx-0-21swapper/222:43:382
28015050,0irq/29-eth0-tx-0-21swapper/222:38:182
28015050,0irq/29-eth0-tx-0-21swapper/221:23:302
28015050,0irq/29-eth0-tx-0-21swapper/019:23:190
28005050,0irq/28-eth0-rx-6300-21ssh21:13:513
28005050,0irq/28-eth0-rx-6287-21kthreadcore20:58:173
28005050,0irq/28-eth0-rx-5891-21kthreadcore22:43:163
28005050,0irq/28-eth0-rx-448-21hwlatdetect22:23:123
28005050,0irq/28-eth0-rx-22926-21df_abs21:43:113
28005050,0irq/28-eth0-rx-20807-21kthreadcore21:23:173
28005050,0irq/28-eth0-rx-15908-21kthreadcore21:03:163
28005050,0irq/28-eth0-rx-15680-21ssh22:28:293
28005050,0irq/28-eth0-rx-13813-21diskstats22:13:133
28005050,0irq/28-eth0-rx-0-21swapper/323:18:103
28005050,0irq/28-eth0-rx-0-21swapper/323:13:143
28005050,0irq/28-eth0-rx-0-21swapper/322:18:313
28005050,0irq/28-eth0-rx-0-21swapper/321:58:143
28005050,0irq/28-eth0-rx-0-21swapper/321:51:553
28005050,0irq/28-eth0-rx-0-21swapper/321:30:453
28005050,0irq/28-eth0-rx-0-21swapper/300:18:113
27150,0ktimersoftd/20-21swapper/222:30:522
261459955,0cyclictest0-21swapper/222:37:392
261459955,0cyclictest0-21swapper/222:05:202
261459954,0cyclictest6227-21sed19:28:162
261459950,0cyclictest0-21swapper/222:24:532
261459950,0cyclictest0-21swapper/221:08:352
261379955,0cyclictest4122-21diskmemload22:48:021
261379955,0cyclictest0-21swapper/123:53:371
261379955,0cyclictest0-21swapper/122:43:281
261379955,0cyclictest0-21swapper/122:00:071
261379955,0cyclictest0-21swapper/121:25:071
261379955,0cyclictest0-21swapper/119:33:201
261379955,0cyclictest0-21swapper/100:11:261
261379950,0cyclictest0-21swapper/123:59:461
261379950,0cyclictest0-21swapper/123:27:271
261379950,0cyclictest0-21swapper/123:09:191
261379950,0cyclictest0-21swapper/121:56:381
261379950,0cyclictest0-21swapper/121:02:341
261379950,0cyclictest0-21swapper/119:05:161
259950,0migration/26324-21sh21:48:422
259950,0migration/231749-21taskset00:08:112
259950,0migration/222030-21sh21:42:292
259950,0migration/221785-21timerandwakeup19:13:252
259950,0migration/218802-21kthreadcore00:18:162
259950,0migration/211475-21sh23:55:252
249950,0watchdog/28320-21grep00:13:182
249950,0watchdog/2442-21irqcore21:13:152
249950,0watchdog/225664-21cut00:23:102
179950,0migration/19770-21kthreadcore19:53:191
179950,0migration/128429-21munin-node23:48:141
179950,0migration/128308-21switchtime00:03:231
179950,0migration/126444-21aten2_r1power_v18:58:111
179950,0migration/126395-21runrttasks22:36:291
179950,0migration/122585-21latency_hist22:17:591
179950,0migration/122537-21kthreadcore23:43:161
179950,0migration/12104-21kthreadcore23:33:161
179950,0migration/116703-21sh21:21:161
169950,0watchdog/132602-21perl22:23:111
169950,0watchdog/10-21swapper/120:03:161
149950,0migration/09060-21aten2_r1power_p23:03:080
149950,0migration/08563-21fschecks_count20:38:130
149950,0migration/08279-21aten2_r1power_v19:53:110
149950,0migration/07682-21taskset19:49:510
149950,0migration/06848-21taskset23:34:290
149950,0migration/0624-21missed_timers21:28:270
149950,0migration/05589-21runrttasks23:18:140
149950,0migration/0427-21kthreadcore20:53:170
149950,0migration/031761-21kthreadcore20:08:160
149950,0migration/030553-21sh22:20:440
149950,0migration/02864-21kthreadcore21:13:160
149950,0migration/028062-21fschecks_time22:38:110
149950,0migration/026050-21latency_hist19:43:000
149950,0migration/024775-21kthreadcore20:03:160
149950,0migration/020516-21taskset21:04:190
149950,0migration/017396-21kthreadcore20:43:160
149950,0migration/015319-21timerandwakeup23:38:240
149950,0migration/015259-21kthreadcore23:23:150
149950,0migration/014055-21fschecks_count22:48:120
149950,0migration/010754-21kthreadcore19:08:180
8315040,0irq/26-0000:00:0-21swapper/219:38:162
339940,0migration/331653-21kthreadcore20:08:163
339940,0migration/329988-21kthreadcore19:43:183
339940,0migration/32884-21latency20:53:183
329940,0watchdog/323704-21ps20:48:153
329940,0watchdog/30-21swapper/319:53:203
28015040,0irq/29-eth0-tx-13608-21grep23:58:102
28015040,0irq/29-eth0-tx-0-21swapper/223:43:092
28015040,0irq/29-eth0-tx-0-21swapper/223:38:492
28015040,0irq/29-eth0-tx-0-21swapper/223:33:022
28015040,0irq/29-eth0-tx-0-21swapper/223:18:152
28015040,0irq/29-eth0-tx-0-21swapper/223:13:012
28015040,0irq/29-eth0-tx-0-21swapper/222:58:002
28015040,0irq/29-eth0-tx-0-21swapper/222:19:092
28015040,0irq/29-eth0-tx-0-21swapper/222:08:022
28015040,0irq/29-eth0-tx-0-21swapper/221:58:112
28015040,0irq/29-eth0-tx-0-21swapper/221:33:502
28015040,0irq/29-eth0-tx-0-21swapper/221:04:032
28015040,0irq/29-eth0-tx-0-21swapper/220:58:182
28015040,0irq/29-eth0-tx-0-21swapper/200:03:112
28015040,0irq/29-eth0-tx-0-21swapper/020:28:200
28005040,0irq/28-eth0-rx-5287-21kthreadcore23:33:173
28005040,0irq/28-eth0-rx-0-21swapper/323:29:543
28005040,0irq/28-eth0-rx-0-21swapper/322:33:383
28005040,0irq/28-eth0-rx-0-21swapper/321:53:123
28005040,0irq/28-eth0-rx-0-21swapper/321:39:123
28005040,0irq/28-eth0-rx-0-21swapper/321:33:083
261459943,0cyclictest9777-21awk21:53:132
261459943,0cyclictest5511-21awk20:13:152
261459943,0cyclictest3346-21snmpd23:48:112
261459943,0cyclictest3070-21irqbalance20:48:362
261459943,0cyclictest3070-21irqbalance20:40:062
261459943,0cyclictest3070-21irqbalance20:23:162
261459943,0cyclictest3070-21irqbalance20:18:362
261459943,0cyclictest3070-21irqbalance20:08:162
261459943,0cyclictest3070-21irqbalance20:02:362
261459943,0cyclictest27189-21awk19:43:152
261379944,0cyclictest4122-21diskmemload22:03:351
261379944,0cyclictest2431-21nvme21:28:271
261379943,0cyclictest3346-21snmpd22:59:401
261379943,0cyclictest3346-21snmpd21:46:401
261379943,0cyclictest3346-21snmpd19:48:101
261379943,0cyclictest3346-21snmpd19:43:101
261379943,0cyclictest3070-21irqbalance23:29:251
261379943,0cyclictest3070-21irqbalance19:14:061
261379943,0cyclictest3070-21irqbalance19:08:261
261379943,0cyclictest20052-21awk19:38:151
261379943,0cyclictest12649-21awk20:18:151
261379943,0cyclictest10104-21awk23:03:121
261369943,0cyclictest803-21awk22:58:130
261369943,0cyclictest5466-21awk22:43:140
261369943,0cyclictest32030-21awk00:08:150
261369943,0cyclictest3070-21irqbalance23:28:250
261369943,0cyclictest3070-21irqbalance22:53:350
261369943,0cyclictest3070-21irqbalance21:53:060
261369943,0cyclictest3070-21irqbalance21:46:160
261369943,0cyclictest3070-21irqbalance21:33:160
261369943,0cyclictest3070-21irqbalance21:23:560
261369943,0cyclictest3070-21irqbalance21:08:060
261369943,0cyclictest18785-21sadc19:38:000
261369943,0cyclictest18784-21uniq22:33:130
261369943,0cyclictest17497-21awk00:18:140
261369943,0cyclictest13885-21awk23:58:120
261369943,0cyclictest12546-21sadc23:57:590
261369940,0cyclictest18787-21munin-node20:23:250
261369940,0cyclictest15355-21munin-node19:58:120
259940,0migration/26350-21kthreadcore19:03:192
259940,0migration/23909-21unixbench-2d19:23:242
259940,0migration/230788-21kthreadcore18:58:172
259940,0migration/230570-21kthreadcore20:28:172
259940,0migration/228837-21sendmail_mailtr19:18:222
259940,0migration/217332-21kthreadcore20:43:162
259940,0migration/210969-21kthreadcore19:53:192
259940,0migration/21066-21kthreadcore20:53:172
249940,0watchdog/20-21swapper/222:47:592
179940,0migration/17782-21cpuspeed_turbos00:13:171
179940,0migration/16357-21kthreadcore19:28:161
179940,0migration/15826-21kernelversion23:18:141
179940,0migration/14452-21fschecks_count21:33:121
179940,0migration/13939-21kthreadcore21:13:171
179940,0migration/132414-21kthreadcore20:08:161
179940,0migration/129866-21df_abs20:53:121
179940,0migration/128376-21munin-node22:38:131
179940,0migration/124489-21kthreadcore20:23:181
179940,0migration/123733-21irqrtprio20:48:151
179940,0migration/120113-21kthreadcore19:58:171
179940,0migration/115632-21kthreadcore21:03:161
179940,0migration/114848-21taskset20:41:481
169940,0watchdog/19991-21if_eth1.223:38:151
169940,0watchdog/131326-21irqstats19:23:161
169940,0watchdog/127103-21iostat_ios20:28:151
169940,0watchdog/125662-21tr00:23:101
169940,0watchdog/121985-21ssh22:17:111
169940,0watchdog/116946-21ssh22:31:071
169940,0watchdog/113682-21diskstats21:38:111
169940,0watchdog/10-21swapper/122:54:111
169940,0watchdog/10-21swapper/121:10:471
149940,0migration/028709-21kthreadcore00:23:170
149940,0migration/02064-21kthreadcore22:23:150
149940,0migration/018049-21aten2_r1power_e21:58:100
149940,0migration/014730-21irqrtprio21:38:150
149940,0migration/012818-21kthreadcore21:18:180
149940,0migration/012620-21sh22:11:580
339930,0migration/322294-21runrttasks19:16:213
329930,0watchdog/30-21swapper/320:38:113
329930,0watchdog/30-21swapper/319:08:043
329930,0watchdog/30-21swapper/319:03:203
28005030,0irq/28-eth0-rx-7188-21python19:48:213
28005030,0irq/28-eth0-rx-0-21swapper/320:18:113
28005030,0irq/28-eth0-rx-0-21swapper/319:27:503
261459933,0cyclictest4669-21kworker/2:120:03:012
261459933,0cyclictest4669-21kworker/2:119:48:012
261459933,0cyclictest4669-21kworker/2:119:33:012
261459933,0cyclictest4669-21kworker/2:119:08:092
261379932,0cyclictest376-21awk21:48:141
261369932,0cyclictest8604-21awk00:13:200
261369932,0cyclictest26346-21munin-node18:58:130
261369932,0cyclictest25071-21kthreadcore19:18:170
261369932,0cyclictest22940-21awk00:03:130
261369932,0cyclictest19952-21cat22:13:250
261369932,0cyclictest19070-21awk23:43:130
261369932,0cyclictest18199-21aten2_r1power_c23:08:090
261369932,0cyclictest0-21swapper/020:13:110
261369932,0cyclictest0-21swapper/019:43:060
261369930,0cyclictest9309-21diskmemload22:28:120
261369930,0cyclictest28034-21perl23:48:110
261369930,0cyclictest22349-21munin-node20:48:110
19130,0ktimersoftd/10-21swapper/120:47:161
179930,0migration/16347-21kthreadcore20:33:181
179930,0migration/16347-21kthreadcore20:33:181
179930,0migration/116804-21sh00:18:111
169930,0watchdog/10-21swapper/122:08:231
169930,0watchdog/10-21swapper/119:18:161
329920,0watchdog/30-21swapper/319:38:043
28005020,0irq/28-eth0-rx-0-21swapper/319:58:093
261369921,0cyclictest0-21swapper/019:28:040
169920,0watchdog/10-21swapper/120:13:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional