You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-06 - 03:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of shadow in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Wed Jul 06, 2022 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
294450540,0irq/25-eth10-21swapper/318:55:163
1947423824,0sleep00-21swapper/018:58:320
1925423725,0sleep10-21swapper/118:56:011
1920423324,0sleep20-21swapper/218:55:222
8215060,0irq/24-0000:00:10421-21crond23:33:352
339960,0migration/37927-21sh22:29:553
339960,0migration/326805-21aten2_r1power_p23:23:473
339960,0migration/323350-21sh21:00:373
339960,0migration/322905-21sh21:19:143
339960,0migration/317104-21sh23:15:343
339960,0migration/316509-21sh22:55:153
28725060,0irq/27-eth0-tx-17185-21munin-run22:18:352
28715060,0irq/26-eth0-rx-676-21sh22:08:353
28715060,0irq/26-eth0-rx-28057-21cut23:03:523
28715060,0irq/26-eth0-rx-11721-21cut23:33:503
195509960,0cyclictest0-21swapper/021:11:090
195509960,0cyclictest0-21swapper/000:00:100
149960,0migration/03842-21munin-node20:23:520
149960,0migration/025652-21sh23:40:370
149960,0migration/022970-21munin-node22:38:540
149960,0migration/022505-21munin-node20:59:040
149960,0migration/019720-21kthreadcore22:58:530
149960,0migration/015936-21sh21:17:220
149960,0migration/015246-21sh21:34:490
149960,0migration/015093-21latency_hist20:08:360
8215050,0irq/24-0000:00:16449-21port_sshpriv00:13:552
4150,0ktimersoftd/00-21swapper/022:48:160
339950,0migration/39974-21kthreadcore19:13:533
339950,0migration/3720-21sh00:04:193
339950,0migration/36919-21kthreadcore20:48:533
339950,0migration/327762-21kthreadcore21:43:523
339950,0migration/326112-21kernelversion21:23:523
339950,0migration/325884-21taskset19:00:333
339950,0migration/321379-21cpuspeed_turbos19:48:463
339950,0migration/320995-21kthreadcore21:58:533
339950,0migration/32030-21munin-node19:58:473
339950,0migration/320056-21taskset19:21:143
339950,0migration/319842-21munin-node23:18:523
339950,0migration/318945-21kthreadcore20:08:533
339950,0migration/31644-21munin-node20:18:563
339950,0migration/314807-21taskset20:05:483
339950,0migration/311294-21kthreadcore21:13:533
339950,0migration/310260-21cpuspeed_turbos20:53:483
329950,0watchdog/39129-21tr21:53:473
329950,0watchdog/38867-21ssh23:10:313
329950,0watchdog/3287150irq/26-eth0-rx-23:55:153
329950,0watchdog/325273-21ssh23:59:433
329950,0watchdog/321466-21ps19:23:523
329950,0watchdog/31894-21diskstats22:48:473
329950,0watchdog/316623-21perl20:33:483
329950,0watchdog/310609-21kthreadcore20:28:523
329950,0watchdog/30-21swapper/322:17:113
28725050,0irq/27-eth0-tx-9876-21munin-node22:53:552
28725050,0irq/27-eth0-tx-8430-21ssh22:50:372
28725050,0irq/27-eth0-tx-527-21munin-node21:28:582
28725050,0irq/27-eth0-tx-4178-21kthreadcore22:08:532
28725050,0irq/27-eth0-tx-29987-21grep21:43:532
28725050,0irq/27-eth0-tx-25210-21ssh21:23:482
28725050,0irq/27-eth0-tx-21180-21munin-node19:48:580
28725050,0irq/27-eth0-tx-0-21swapper/223:53:072
28725050,0irq/27-eth0-tx-0-21swapper/223:34:322
28725050,0irq/27-eth0-tx-0-21swapper/223:21:382
28725050,0irq/27-eth0-tx-0-21swapper/223:04:502
28725050,0irq/27-eth0-tx-0-21swapper/222:38:542
28725050,0irq/27-eth0-tx-0-21swapper/222:35:352
28725050,0irq/27-eth0-tx-0-21swapper/222:04:112
28725050,0irq/27-eth0-tx-0-21swapper/221:38:512
28725050,0irq/27-eth0-tx-0-21swapper/221:37:142
28725050,0irq/27-eth0-tx-0-21swapper/221:07:582
28715050,0irq/26-eth0-rx-6370-21runrttasks22:18:523
28715050,0irq/26-eth0-rx-3736-21if_eth023:48:523
28715050,0irq/26-eth0-rx-27330-21iostat19:03:523
28715050,0irq/26-eth0-rx-26738-21grep22:43:503
28715050,0irq/26-eth0-rx-24045-21taskset22:39:593
28715050,0irq/26-eth0-rx-0-21swapper/323:40:053
28715050,0irq/26-eth0-rx-0-21swapper/323:30:073
28715050,0irq/26-eth0-rx-0-21swapper/322:23:523
28715050,0irq/26-eth0-rx-0-21swapper/322:08:533
28715050,0irq/26-eth0-rx-0-21swapper/321:49:363
28715050,0irq/26-eth0-rx-0-21swapper/321:33:513
28715050,0irq/26-eth0-rx-0-21swapper/321:04:463
28715050,0irq/26-eth0-rx-0-21swapper/319:53:533
28715050,0irq/26-eth0-rx-0-21swapper/319:38:513
28715050,0irq/26-eth0-rx-0-21swapper/300:25:093
28715050,0irq/26-eth0-rx-0-21swapper/300:20:213
28715050,0irq/26-eth0-rx-0-21swapper/300:08:463
259950,0migration/26703-21kthreadcore19:58:532
259950,0migration/230326-21kthreadcore20:18:522
259950,0migration/226359-21taskset00:22:142
259950,0migration/220656-21kthreadcore23:38:552
259950,0migration/219555-21kthreadcore21:58:532
259950,0migration/213182-21kthreadcore21:53:532
259950,0migration/213086-21kthreadcore20:53:532
259950,0migration/211231-21munin-node23:13:502
249950,0watchdog/2437-21ls23:43:592
249950,0watchdog/217461-21basename21:18:522
249950,0watchdog/20-21swapper/219:23:522
195529954,0cyclictest3415-21snmpd23:11:582
195519955,0cyclictest0-21swapper/123:49:211
195519955,0cyclictest0-21swapper/121:50:561
195519955,0cyclictest0-21swapper/121:29:581
195519955,0cyclictest0-21swapper/121:14:031
195519955,0cyclictest0-21swapper/120:18:521
195519955,0cyclictest0-21swapper/100:26:421
195519955,0cyclictest0-21swapper/100:04:191
195519954,0cyclictest3141-21irqbalance20:36:561
195509955,0cyclictest0-21swapper/023:48:390
195509954,0cyclictest8742-21idleruntime-cro21:53:350
195509950,0cyclictest0-21swapper/023:55:070
195509950,0cyclictest0-21swapper/023:26:060
195509950,0cyclictest0-21swapper/023:12:330
195509950,0cyclictest0-21swapper/022:36:270
195509950,0cyclictest0-21swapper/021:21:050
19150,0ktimersoftd/10-21swapper/123:54:421
19150,0ktimersoftd/10-21swapper/122:48:481
19150,0ktimersoftd/10-21swapper/121:43:571
179950,0migration/19198-21taskset23:30:241
179950,0migration/1438-21sh23:24:151
179950,0migration/131135-21timerandwakeup22:03:591
179950,0migration/123514-21sh21:01:051
179950,0migration/121278-21kthreadcore00:18:531
179950,0migration/113635-21kthreadcore23:33:521
169950,0watchdog/129025-21tr19:28:521
169950,0watchdog/117260-21latency_hist22:18:351
149950,0migration/09101-21kthreadcore19:13:530
149950,0migration/08120-21taskset22:30:240
149950,0migration/04703-21kthreadcore22:48:520
149950,0migration/03730-21kthreadcore22:08:530
149950,0migration/03563-21irqcore23:28:510
149950,0migration/031142-21kthreadcore19:28:530
149950,0migration/03004-21kthreadcore19:33:590
149950,0migration/028885-21kthreadcore23:03:530
149950,0migration/028364-21irqrtprio00:23:520
149950,0migration/026725-21kthreadcore21:43:520
149950,0migration/023865-21runrttasks22:00:320
149950,0migration/020515-21munin-node18:58:500
149950,0migration/018301-21kthreadcore20:33:510
149950,0migration/016510-21latency_hist20:58:350
149950,0migration/015989-21proc_pri23:13:560
149950,0migration/014141-21kthreadcore00:13:520
339940,0migration/331677-21kthreadcore20:43:523
339940,0migration/327882-21runrttasks20:14:313
339940,0migration/324814-21kthreadcore20:38:523
339940,0migration/316099-21kthreadcore19:43:523
329940,0watchdog/3438-21sort23:43:593
329940,0watchdog/315091-21smartctl22:34:033
329940,0watchdog/30-21swapper/319:34:003
329940,0watchdog/30-21swapper/319:28:523
28725040,0irq/27-eth0-tx-812-21ssh00:04:292
28725040,0irq/27-eth0-tx-32445-21ssh23:23:592
28725040,0irq/27-eth0-tx-0-21swapper/222:25:012
28725040,0irq/27-eth0-tx-0-21swapper/222:18:412
28725040,0irq/27-eth0-tx-0-21swapper/221:49:342
28725040,0irq/27-eth0-tx-0-21swapper/221:14:382
28725040,0irq/27-eth0-tx-0-21swapper/221:08:422
28725040,0irq/27-eth0-tx-0-21swapper/220:59:222
28725040,0irq/27-eth0-tx-0-21swapper/200:23:422
28725040,0irq/27-eth0-tx-0-21swapper/200:08:452
28725040,0irq/27-eth0-tx-0-21swapper/020:40:120
28725040,0irq/27-eth0-tx-0-21swapper/020:08:500
28715040,0irq/26-eth0-rx-0-21swapper/322:59:083
28715040,0irq/26-eth0-rx-0-21swapper/321:38:453
28715040,0irq/26-eth0-rx-0-21swapper/321:31:123
28715040,0irq/26-eth0-rx-0-21swapper/321:08:573
28715040,0irq/26-eth0-rx-0-21swapper/319:08:543
28715040,0irq/26-eth0-rx-0-21swapper/300:13:373
259940,0migration/27794-21kthreadcore20:23:542
259940,0migration/223292-21kthreadcore20:13:522
259940,0migration/221166-21kthreadcore18:58:522
259940,0migration/217266-21runrttasks23:54:362
259940,0migration/211314-21kthreadcore19:38:522
249940,0watchdog/2906-21df_inode19:08:482
249940,0watchdog/230381-21irqstats20:43:522
249940,0watchdog/229022-21kthreadcore19:28:522
249940,0watchdog/224213-21kthreadcore19:48:522
249940,0watchdog/216142-21kthreadcore19:43:522
249940,0watchdog/20-21swapper/223:58:472
249940,0watchdog/20-21swapper/222:43:512
249940,0watchdog/20-21swapper/220:28:522
249940,0watchdog/20-21swapper/219:53:562
195529943,0cyclictest8153-21awk19:13:512
195529943,0cyclictest3415-21snmpd20:51:562
195529943,0cyclictest3415-21snmpd20:40:262
195529943,0cyclictest3415-21snmpd20:36:262
195529943,0cyclictest3415-21snmpd20:09:262
195529943,0cyclictest3415-21snmpd20:06:562
195529943,0cyclictest2210-21awk22:28:492
195529943,0cyclictest2193-21awk19:33:572
195529943,0cyclictest19062-21awk22:58:522
195519944,0cyclictest0-21swapper/121:55:311
195519944,0cyclictest0-21swapper/119:48:451
195519943,0cyclictest3415-21snmpd23:48:131
195519943,0cyclictest3415-21snmpd23:21:131
195519943,0cyclictest3415-21snmpd23:09:581
195519943,0cyclictest3415-21snmpd23:00:131
195519943,0cyclictest3415-21snmpd22:44:131
195519943,0cyclictest3415-21snmpd22:35:121
195519943,0cyclictest3415-21snmpd21:08:271
195519943,0cyclictest3141-21irqbalance20:30:561
195519943,0cyclictest30016-21awk20:43:501
195519943,0cyclictest27277-21awk23:03:511
195519943,0cyclictest2177-21awk22:28:491
195519943,0cyclictest2169-21awk19:33:571
195519943,0cyclictest18410-21awk22:38:501
195519943,0cyclictest11230-21awk23:13:501
195509944,0cyclictest0-21swapper/000:08:210
195509943,0cyclictest3457-21lldpd22:05:540
195509943,0cyclictest3415-21snmpd23:21:580
195509943,0cyclictest29400-21awk20:18:500
195509943,0cyclictest1404-21awk19:08:500
195509943,0cyclictest1389-21awk21:28:490
195509943,0cyclictest10286-21awk20:28:500
179940,0migration/17679-21kthreadcore20:23:541
179940,0migration/17287-21kthreadcore00:08:531
179940,0migration/130794-21kthreadcore19:03:541
179940,0migration/128917-21kthreadcore22:23:521
179940,0migration/126565-21kthreadcore20:13:541
179940,0migration/123444-21kthreadcore23:38:561
179940,0migration/121665-21kthreadcore23:58:531
179940,0migration/119463-21kthreadcore22:19:011
179940,0migration/118839-21kthreadcore20:08:531
179940,0migration/117739-21idleruntime21:38:501
179940,0migration/113024-21kthreadcore00:13:511
179940,0migration/111044-21irqcore20:53:521
179940,0migration/110908-21kthreadcore21:33:521
169940,0watchdog/14634-21irqstats20:48:521
169940,0watchdog/13242-21kernelversion19:58:521
169940,0watchdog/131959-21ssh21:26:391
169940,0watchdog/123356-21ssh21:59:111
169940,0watchdog/116141-21kthreadcore19:43:521
169940,0watchdog/10-21swapper/122:53:591
169940,0watchdog/10-21swapper/122:08:351
169940,0watchdog/10-21swapper/119:23:521
149940,0migration/06400-21kthreadcore19:58:530
149940,0migration/030563-21sensors_volt21:23:570
149940,0migration/023225-21kthreadcore19:23:530
149940,0migration/016852-21kthreadcore19:18:530
149940,0migration/012854-21kthreadcore19:38:520
149940,0migration/012789-21kthreadcore22:53:530
329930,0watchdog/30-21swapper/320:23:483
195529932,0cyclictest19654-21timerwakeupswit19:18:592
195529930,0cyclictest0-21swapper/219:04:492
195519932,0cyclictest8880-21awk19:38:501
195519932,0cyclictest28366-21tune2fs19:53:491
195519932,0cyclictest14387-21wc20:03:581
195519932,0cyclictest1270-21grep19:08:491
195519931,0cyclictest0-21swapper/121:08:541
195519930,0cyclictest23147-21perl20:38:481
195519930,0cyclictest14239-21diskstats19:18:481
195509933,0cyclictest6628-21kworker/0:123:43:410
195509933,0cyclictest6628-21kworker/0:123:33:510
195509933,0cyclictest6628-21kworker/0:122:23:430
195509933,0cyclictest6628-21kworker/0:122:18:350
195509933,0cyclictest6628-21kworker/0:122:13:480
195509933,0cyclictest6628-21kworker/0:121:53:380
195509933,0cyclictest26798-21kworker/0:000:18:410
195509933,0cyclictest26798-21kworker/0:000:08:410
195509933,0cyclictest26084-21kworker/0:021:38:380
195509933,0cyclictest26084-21kworker/0:020:43:420
195509933,0cyclictest26084-21kworker/0:020:13:520
195509933,0cyclictest26084-21kworker/0:019:53:400
195509933,0cyclictest26084-21kworker/0:019:43:380
195509932,0cyclictest26386-21munin-node19:03:530
195509932,0cyclictest24841-21basename21:03:480
195509930,0cyclictest0-21swapper/020:48:490
19130,0ktimersoftd/10-21swapper/118:58:401
169930,0watchdog/10-21swapper/121:18:521
169930,0watchdog/10-21swapper/119:16:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional