You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-31 - 04:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of shadow in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Tue Jan 31, 2023 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
926828682,0sleep20-21swapper/218:57:072
949024738,0sleep10-21swapper/118:59:381
952623723,0sleep00-21swapper/019:00:050
929923329,0sleep30-21swapper/318:57:243
97359988,0cyclictest5158-21sensors20:32:111
97359976,0cyclictest0-21swapper/123:54:561
97349960,0cyclictest0-21swapper/023:08:510
339960,0migration/35401-21sh22:19:473
339960,0migration/330432-21sh22:56:423
339960,0migration/329357-21sh21:13:163
28845060,0irq/26-eth0-rx-6185-21sh23:21:393
28845060,0irq/26-eth0-rx-30865-21cpu23:36:553
28845060,0irq/26-eth0-rx-29245-21ntpdc19:37:043
28845060,0irq/26-eth0-rx-29244-21sh23:33:343
28845060,0irq/26-eth0-rx-28975-21crond21:52:393
28845060,0irq/26-eth0-rx-28367-21ntp_states23:52:093
28845060,0irq/26-eth0-rx-20619-21sed23:27:143
28845060,0irq/26-eth0-rx-15776-21cut23:07:003
28845060,0irq/26-eth0-rx-12509-21grep21:22:083
28845060,0irq/26-eth0-rx-12095-21pmu-power23:22:083
97379955,0cyclictest6634-21diskmemload21:23:092
97379955,0cyclictest0-21swapper/223:16:122
97379955,0cyclictest0-21swapper/221:57:222
97379954,0cyclictest3427-21snmpd20:51:122
97379954,0cyclictest3427-21snmpd19:13:122
97379950,0cyclictest0-21swapper/223:44:062
97379950,0cyclictest0-21swapper/222:02:152
97359955,0cyclictest0-21swapper/123:58:331
97359955,0cyclictest0-21swapper/123:29:231
97359955,0cyclictest0-21swapper/122:51:231
97359955,0cyclictest0-21swapper/122:29:191
97359955,0cyclictest0-21swapper/121:58:351
97359955,0cyclictest0-21swapper/100:27:241
97359950,0cyclictest0-21swapper/121:52:161
97359950,0cyclictest0-21swapper/100:11:481
97349950,0cyclictest0-21swapper/022:44:250
97349950,0cyclictest0-21swapper/000:30:120
8225050,0irq/24-0000:00:12561-21idleruntime-cro20:41:392
339950,0migration/38251-21kthreadcore22:02:043
339950,0migration/37744-21kthreadcore00:02:063
339950,0migration/35693-21taskset22:00:323
339950,0migration/34547-21timerandwakeup20:07:073
339950,0migration/34313-21taskset22:37:083
339950,0migration/34087-21munin-node19:21:553
339950,0migration/3357-21if_eth1.320:32:043
339950,0migration/331713-21munin-node21:17:013
339950,0migration/330857-21fschecks_count19:41:563
339950,0migration/329964-21cpuspeed_turbos19:16:553
339950,0migration/323508-21ksmtuned19:35:073
339950,0migration/322006-21sh21:50:413
339950,0migration/320034-21taskset00:07:113
339950,0migration/315821-21munin-node20:42:023
339950,0migration/314478-21aten2_r1power_p22:46:563
339950,0migration/313313-21kthreadcore19:01:583
339950,0migration/312660-21kthreadcore19:52:003
339950,0migration/312361-21taskset20:39:323
28855050,0irq/27-eth0-tx-4711-21munin-node19:47:022
28855050,0irq/27-eth0-tx-0-21swapper/223:27:112
28855050,0irq/27-eth0-tx-0-21swapper/200:26:022
28855050,0irq/27-eth0-tx-0-21swapper/200:17:362
28855050,0irq/27-eth0-tx-0-21swapper/200:01:562
28845050,0irq/26-eth0-rx-8243-21cut23:02:053
28845050,0irq/26-eth0-rx-817-21cat21:37:023
28845050,0irq/26-eth0-rx-26380-21kthreadcore21:32:123
28845050,0irq/26-eth0-rx-23225-21perl23:11:563
28845050,0irq/26-eth0-rx-21486-21ssh21:08:323
28845050,0irq/26-eth0-rx-21070-21ssh22:28:253
28845050,0irq/26-eth0-rx-15346-21tune2fs21:26:583
28845050,0irq/26-eth0-rx-0-21swapper/322:51:473
28845050,0irq/26-eth0-rx-0-21swapper/322:44:413
28845050,0irq/26-eth0-rx-0-21swapper/322:33:233
28845050,0irq/26-eth0-rx-0-21swapper/300:27:073
28845050,0irq/26-eth0-rx-0-21swapper/300:22:353
27150,0ktimersoftd/20-21swapper/223:53:242
249950,0watchdog/231797-21fschecks_time20:06:582
249950,0watchdog/20-21swapper/223:02:052
249950,0watchdog/20-21swapper/219:56:582
179950,0migration/17897-21kthreadcore20:37:021
179950,0migration/14375-21timerwakeupswit23:37:141
179950,0migration/13288-21unixbench_multi19:17:071
179950,0migration/128763-21sh22:52:461
179950,0migration/124064-21munin-node20:22:031
179950,0migration/120205-21timerwakeupswit00:07:121
179950,0migration/118246-21kthreadcore21:27:021
179950,0migration/115908-21kthreadcore22:07:041
179950,0migration/111743-21latency00:02:071
169950,0watchdog/128320-21rt-features22:32:131
149950,0migration/09866-21kthreadcore00:02:060
149950,0migration/09314-21kthreadcore22:22:030
149950,0migration/08434-21kthreadcore21:02:040
149950,0migration/07875-21kernelversion21:41:590
149950,0migration/06461-21aten2_r1power_c22:01:570
149950,0migration/0573-21kthreadcore21:57:000
149950,0migration/05513-21munin-node19:46:580
149950,0migration/04553-21latency_hist19:46:390
149950,0migration/04024-21munin-node22:17:070
149950,0migration/03606-21taskset19:19:270
149950,0migration/032530-21latency_hist20:56:390
149950,0migration/032418-21kernelversion20:07:010
149950,0migration/032023-21kthreadcore23:16:590
149950,0migration/031217-21fschecks_time22:36:570
149950,0migration/031110-21sh00:16:580
149950,0migration/027565-21kthreadcore21:52:020
149950,0migration/025492-21kthreadcore21:32:120
149950,0migration/024178-21taskset20:00:100
149950,0migration/023297-21gltestperf22:11:580
149950,0migration/022867-21df_abs23:11:550
149950,0migration/020004-21sh22:27:020
149950,0migration/018372-21kthreadcore21:47:000
149950,0migration/018175-21ntp_kernel_pll_20:42:040
149950,0migration/018100-21kthreadcore21:27:020
149950,0migration/017759-21kthreadcore23:47:030
149950,0migration/015991-21irqcore21:07:000
149950,0migration/013396-21sh00:26:380
149950,0migration/013329-21taskset23:24:110
149950,0migration/012832-21taskset23:42:240
149950,0migration/010157-21kthreadcore20:37:030
97379943,0cyclictest3469-21lldpd21:31:502
97379943,0cyclictest3469-21lldpd20:57:052
97379943,0cyclictest3469-21lldpd19:17:052
97379943,0cyclictest3427-21snmpd20:26:122
97379943,0cyclictest3427-21snmpd20:03:122
97379943,0cyclictest31601-21awk23:37:022
97379943,0cyclictest31453-21awk22:57:002
97379943,0cyclictest31247-21awk19:41:582
97379943,0cyclictest23800-21awk21:12:022
97379943,0cyclictest23422-21awk22:32:062
97379943,0cyclictest17403-21awk19:06:582
97379943,0cyclictest13621-21awk20:42:002
97359944,0cyclictest6145-21kworker/1:100:17:061
97359944,0cyclictest0-21swapper/123:45:191
97359944,0cyclictest0-21swapper/122:24:261
97359943,0cyclictest7130-21awk22:41:581
97359943,0cyclictest6586-21awk00:22:011
97359943,0cyclictest3469-21lldpd23:48:011
97359943,0cyclictest3469-21lldpd23:22:541
97359943,0cyclictest3469-21lldpd23:16:541
97359943,0cyclictest3427-21snmpd23:32:281
97359943,0cyclictest3427-21snmpd20:58:271
97359943,0cyclictest3427-21snmpd19:21:421
97359943,0cyclictest31679-21awk21:37:001
97359943,0cyclictest3154-21irqbalance23:01:481
97359943,0cyclictest24074-21awk19:11:591
97359943,0cyclictest18125-21awk19:32:001
97349944,0cyclictest0-21swapper/023:03:580
97349944,0cyclictest0-21swapper/022:08:560
97349943,0cyclictest3427-21snmpd22:48:280
97349943,0cyclictest32685-21awk20:32:030
97349943,0cyclictest3154-21irqbalance23:58:180
97349943,0cyclictest3154-21irqbalance23:52:180
97349943,0cyclictest3154-21irqbalance22:57:580
97349943,0cyclictest3154-21irqbalance21:22:580
97349943,0cyclictest3154-21irqbalance20:26:490
97349943,0cyclictest3154-21irqbalance20:22:090
97349943,0cyclictest3154-21irqbalance20:17:190
97349943,0cyclictest3154-21irqbalance20:12:290
97349943,0cyclictest3154-21irqbalance19:51:490
97349943,0cyclictest3154-21irqbalance19:37:190
97349943,0cyclictest3154-21irqbalance19:31:490
97349943,0cyclictest3154-21irqbalance19:27:290
97349943,0cyclictest3154-21irqbalance19:22:490
97349943,0cyclictest23174-21awk00:12:030
97349943,0cyclictest20290-21awk20:46:590
3760240,0sleep13898-21processes22:37:051
339940,0migration/327757-21kthreadcore20:27:013
339940,0migration/326888-21kthreadcore19:12:023
339940,0migration/320854-21kthreadcore19:07:013
339940,0migration/313677-21kthreadcore19:27:013
339940,0migration/310547-21kthreadcore21:42:003
28855040,0irq/27-eth0-tx-4794-21ssh22:38:212
28855040,0irq/27-eth0-tx-3594-21latency23:17:002
28855040,0irq/27-eth0-tx-25291-21munin-node20:27:062
28855040,0irq/27-eth0-tx-2311-21kthreadcore21:37:032
28855040,0irq/27-eth0-tx-23023-21df_abs21:51:572
28855040,0irq/27-eth0-tx-12901-21ssh21:42:422
28855040,0irq/27-eth0-tx-12003-21awk23:22:062
28855040,0irq/27-eth0-tx-0-21swapper/223:57:022
28855040,0irq/27-eth0-tx-0-21swapper/223:47:482
28855040,0irq/27-eth0-tx-0-21swapper/223:31:482
28855040,0irq/27-eth0-tx-0-21swapper/223:06:582
28855040,0irq/27-eth0-tx-0-21swapper/222:52:002
28855040,0irq/27-eth0-tx-0-21swapper/222:26:452
28855040,0irq/27-eth0-tx-0-21swapper/222:22:482
28855040,0irq/27-eth0-tx-0-21swapper/222:17:092
28855040,0irq/27-eth0-tx-0-21swapper/222:11:402
28855040,0irq/27-eth0-tx-0-21swapper/222:06:512
28855040,0irq/27-eth0-tx-0-21swapper/221:48:162
28855040,0irq/27-eth0-tx-0-21swapper/221:28:212
28855040,0irq/27-eth0-tx-0-21swapper/221:07:412
28855040,0irq/27-eth0-tx-0-21swapper/220:52:052
28855040,0irq/27-eth0-tx-0-21swapper/200:08:272
28845040,0irq/26-eth0-rx-0-21swapper/323:56:543
28845040,0irq/26-eth0-rx-0-21swapper/323:46:423
28845040,0irq/26-eth0-rx-0-21swapper/323:41:383
28845040,0irq/26-eth0-rx-0-21swapper/322:22:263
28845040,0irq/26-eth0-rx-0-21swapper/322:12:373
28845040,0irq/26-eth0-rx-0-21swapper/322:07:043
28845040,0irq/26-eth0-rx-0-21swapper/321:03:433
28845040,0irq/26-eth0-rx-0-21swapper/300:16:483
28845040,0irq/26-eth0-rx-0-21swapper/300:12:083
27140,0ktimersoftd/20-21swapper/200:27:562
259940,0migration/28987-21kthreadcore22:42:002
259940,0migration/22561-21kthreadcore21:17:042
259940,0migration/216873-21kthreadcore22:47:032
259940,0migration/213693-21kthreadcore20:16:592
249940,0watchdog/27014-21ssh21:01:582
249940,0watchdog/232482-21perl20:32:022
249940,0watchdog/20-21swapper/220:12:102
249940,0watchdog/20-21swapper/219:26:582
249940,0watchdog/20-21swapper/219:01:582
249940,0watchdog/20-21swapper/200:11:412
179940,0migration/130743-21cpuspeed_turbos22:56:561
179940,0migration/129838-21taskset19:37:181
179940,0migration/129608-21kthreadcore20:52:011
179940,0migration/126710-21kthreadcore20:02:041
179940,0migration/125847-21latency_hist20:51:391
179940,0migration/125838-21kthreadcore23:12:021
179940,0migration/117066-21latency_hist19:31:391
179940,0migration/115477-21fschecks_time21:06:571
179940,0migration/110577-21kthreadcore21:02:051
169940,0watchdog/19948-21munin-node19:01:541
169940,0watchdog/13427-21snmpd21:47:091
169940,0watchdog/129597-21ssh21:33:581
169940,0watchdog/129440-21grep20:27:021
169940,0watchdog/118710-21idleruntime19:56:581
169940,0watchdog/114470-21munin-node23:07:051
169940,0watchdog/112718-21ssh21:42:101
169940,0watchdog/110508-21timerwakeupswit19:47:061
169940,0watchdog/10-21swapper/122:02:051
169940,0watchdog/10-21swapper/121:22:021
169940,0watchdog/10-21swapper/121:19:341
149940,0migration/032555-21kthreadcore21:17:030
149940,0migration/031932-21irqrtprio23:37:030
149940,0migration/028758-21sendmail_mailst19:12:060
149940,0migration/02639-21kthreadcore21:37:030
149940,0migration/026057-21kthreadcore20:02:040
149940,0migration/025046-21kthreadcore23:32:080
149940,0migration/020906-21taskset23:27:450
97379932,0cyclictest23888-21munin-node19:36:592
97379932,0cyclictest11333-21munin-node19:51:572
97379932,0cyclictest0-21swapper/219:32:432
97359933,0cyclictest6145-21kworker/1:122:16:551
97359933,0cyclictest6145-21kworker/1:121:11:551
97359933,0cyclictest6145-21kworker/1:120:41:451
97359933,0cyclictest6145-21kworker/1:120:16:451
97359932,0cyclictest3469-21lldpd20:12:041
97359932,0cyclictest3469-21lldpd19:42:041
97359932,0cyclictest12149-21awk19:51:581
97349932,0cyclictest3427-21snmpd21:12:270
97349932,0cyclictest30668-21grep20:52:020
97349932,0cyclictest23480-21awk22:32:060
97349932,0cyclictest15108-21awk00:07:000
4130,0ktimersoftd/029010-21ssh22:53:280
28845030,0irq/26-eth0-rx-0-21swapper/321:00:063
28845030,0irq/26-eth0-rx-0-21swapper/320:52:073
28845030,0irq/26-eth0-rx-0-21swapper/320:46:573
28845030,0irq/26-eth0-rx-0-21swapper/320:16:443
28845030,0irq/26-eth0-rx-0-21swapper/320:16:303
28845030,0irq/26-eth0-rx-0-21swapper/320:01:493
28845030,0irq/26-eth0-rx-0-21swapper/319:56:483
249930,0watchdog/20-21swapper/219:22:062
179930,0migration/122229-21switchtime19:07:061
169930,0watchdog/131798-21fschecks_time20:06:581
169930,0watchdog/10-21swapper/122:11:411
97349921,0cyclictest16498-21latency_hist19:06:390
97349920,0cyclictest10249-21df_abs19:01:530
339920,0migration/35056-21diskstats19:46:563
28845020,0irq/26-eth0-rx-0-21swapper/320:21:493
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional