You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-02-04 - 04:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of shadow in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack1slot2s.osadl.org (updated Sat Feb 04, 2023 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1573524837,0sleep00-21swapper/018:58:100
1569523727,0sleep30-21swapper/318:57:453
1589223624,0sleep10-21swapper/118:59:591
1040323526,0sleep20-21swapper/218:56:282
16068991312,0cyclictest0-21swapper/220:55:342
16068991312,0cyclictest0-21swapper/220:55:342
16068991312,0cyclictest0-21swapper/219:45:242
16068991312,0cyclictest0-21swapper/219:32:532
1606899130,0cyclictest0-21swapper/220:45:332
16068991211,0cyclictest0-21swapper/219:59:252
16068991211,0cyclictest0-21swapper/219:53:362
16068991211,0cyclictest0-21swapper/219:18:342
16068991110,0cyclictest0-21swapper/219:05:342
339960,0migration/34052-21sh21:33:443
339960,0migration/331376-21sh23:09:523
339960,0migration/319372-21sh21:21:573
339960,0migration/316493-21sh00:00:083
339960,0migration/312298-21sh21:57:443
28845060,0irq/26-eth0-rx-7669-21irqstats20:56:293
28845060,0irq/26-eth0-rx-32681-21fschecks_time23:11:263
28845060,0irq/26-eth0-rx-22468-21basename21:46:303
28845060,0irq/26-eth0-rx-10665-21grep23:16:313
160679966,0cyclictest0-21swapper/121:02:281
339950,0migration/36269-21munin-node00:11:333
339950,0migration/35365-21taskset20:29:323
339950,0migration/3514-21cpuspeed_turbos23:31:263
339950,0migration/330740-21timerwakeupswit00:26:363
339950,0migration/330683-21idleruntime21:51:283
339950,0migration/330433-21taskset19:59:063
339950,0migration/326670-21missed_timers21:26:343
339950,0migration/325682-21kthreadcore21:06:313
339950,0migration/323902-21taskset19:54:413
339950,0migration/323458-21latency_hist19:31:143
339950,0migration/322514-21processes00:21:343
339950,0migration/32219-21munin-node22:51:313
339950,0migration/31677-21kthreadcore21:11:323
339950,0migration/313627-21munin-node21:41:263
339950,0migration/312649-21sh21:20:453
339950,0migration/311334-21timerwakeupswit21:36:403
339950,0migration/311327-21taskset20:09:273
339950,0migration/310149-21kthreadcore23:36:303
28855050,0irq/27-eth0-tx-6175-21iceccd22:38:502
28855050,0irq/27-eth0-tx-4669-21grep23:51:332
28855050,0irq/27-eth0-tx-32181-21munin-node23:11:332
28855050,0irq/27-eth0-tx-30857-21munin-node20:01:272
28855050,0irq/27-eth0-tx-21225-21ssh21:06:242
28855050,0irq/27-eth0-tx-14178-21ssh22:58:042
28855050,0irq/27-eth0-tx-13953-21munin-node22:01:282
28855050,0irq/27-eth0-tx-13104-21munin-node21:01:382
28855050,0irq/27-eth0-tx-12832-21diskmemload00:10:292
28855050,0irq/27-eth0-tx-0-21swapper/223:24:572
28855050,0irq/27-eth0-tx-0-21swapper/222:48:482
28855050,0irq/27-eth0-tx-0-21swapper/221:33:552
28855050,0irq/27-eth0-tx-0-21swapper/221:23:042
28845050,0irq/26-eth0-rx-3886-21kthreadcore20:51:333
28845050,0irq/26-eth0-rx-3886-21kthreadcore20:51:333
28845050,0irq/26-eth0-rx-21602-21ssh22:24:003
28845050,0irq/26-eth0-rx-15078-21chrt00:16:473
28845050,0irq/26-eth0-rx-14310-21diskstats22:01:253
28845050,0irq/26-eth0-rx-0-21swapper/323:44:093
28845050,0irq/26-eth0-rx-0-21swapper/323:29:233
28845050,0irq/26-eth0-rx-0-21swapper/323:22:163
28845050,0irq/26-eth0-rx-0-21swapper/323:01:203
28845050,0irq/26-eth0-rx-0-21swapper/322:56:293
28845050,0irq/26-eth0-rx-0-21swapper/322:47:573
28845050,0irq/26-eth0-rx-0-21swapper/322:43:433
28845050,0irq/26-eth0-rx-0-21swapper/322:36:353
28845050,0irq/26-eth0-rx-0-21swapper/322:35:493
28845050,0irq/26-eth0-rx-0-21swapper/322:16:553
28845050,0irq/26-eth0-rx-0-21swapper/322:07:323
28845050,0irq/26-eth0-rx-0-21swapper/321:04:343
28845050,0irq/26-eth0-rx-0-21swapper/320:11:343
28845050,0irq/26-eth0-rx-0-21swapper/300:06:223
259950,0migration/26228-21fschecks_count21:56:262
259950,0migration/23097-21kthreadcore23:31:322
259950,0migration/219812-21kthreadcore22:41:312
259950,0migration/211421-21irqcore19:21:302
249950,0watchdog/230109-21port_sshpriv23:26:352
249950,0watchdog/215825-21sh23:39:112
19150,0ktimersoftd/10-21swapper/123:13:391
160689950,0cyclictest0-21swapper/222:08:152
160679955,0cyclictest12832-21diskmemload23:54:481
160679955,0cyclictest0-21swapper/121:55:081
160679955,0cyclictest0-21swapper/121:51:021
160679955,0cyclictest0-21swapper/121:30:511
160679955,0cyclictest0-21swapper/100:17:391
160679950,0cyclictest0-21swapper/123:03:241
160679950,0cyclictest0-21swapper/122:44:001
160679950,0cyclictest0-21swapper/122:20:401
160679950,0cyclictest0-21swapper/122:08:411
160669954,0cyclictest7239-21awk20:56:270
160669950,0cyclictest0-21swapper/023:07:210
149950,0migration/08066-21sh23:16:220
149950,0migration/07830-21kthreadcore20:06:310
149950,0migration/06621-21kernelversion21:36:310
149950,0migration/05788-21sh22:52:470
149950,0migration/05731-21timerwakeupswit23:11:360
149950,0migration/04558-21kthreadcore23:51:330
149950,0migration/04457-21missed_timers20:26:330
149950,0migration/031310-21latency_hist22:51:120
149950,0migration/02973-21kthreadcore00:11:310
149950,0migration/028462-21runrttasks22:06:570
149950,0migration/027695-21kthreadcore00:06:300
149950,0migration/027631-21kthreadcore00:26:290
149950,0migration/026444-21munin-node23:46:300
149950,0migration/023214-21sh23:23:400
149950,0migration/020194-21sh21:43:050
149950,0migration/020098-21ntp_offset22:21:350
149950,0migration/018295-21kthreadcore19:26:310
149950,0migration/015175-21munin-node23:56:380
149950,0migration/01424-21kthreadcore19:36:310
149950,0migration/013485-21kthreadcore00:16:300
149950,0migration/013181-21taskset22:37:150
149950,0migration/013115-21kthreadcore19:21:310
149950,0migration/011881-21taskset21:18:460
339940,0migration/3653-21kthreadcore20:01:343
339940,0migration/3504-21kthreadcore19:11:323
339940,0migration/34082-21fschecks_count19:16:263
339940,0migration/331802-21taskset20:49:583
339940,0migration/325190-21cpuspeed_turbos20:21:263
28855040,0irq/27-eth0-tx-6368-21munin-node22:16:332
28855040,0irq/27-eth0-tx-31388-21cat22:51:122
28855040,0irq/27-eth0-tx-27016-21cut23:06:322
28855040,0irq/27-eth0-tx-22153-21ssh21:26:312
28855040,0irq/27-eth0-tx-17142-21ssh00:01:172
28855040,0irq/27-eth0-tx-16074-21ssh23:01:252
28855040,0irq/27-eth0-tx-0-21swapper/223:56:262
28855040,0irq/27-eth0-tx-0-21swapper/223:47:222
28855040,0irq/27-eth0-tx-0-21swapper/223:16:222
28855040,0irq/27-eth0-tx-0-21swapper/222:31:292
28855040,0irq/27-eth0-tx-0-21swapper/222:27:082
28855040,0irq/27-eth0-tx-0-21swapper/222:21:192
28855040,0irq/27-eth0-tx-0-21swapper/222:13:132
28855040,0irq/27-eth0-tx-0-21swapper/221:51:252
28855040,0irq/27-eth0-tx-0-21swapper/221:41:182
28855040,0irq/27-eth0-tx-0-21swapper/221:17:542
28855040,0irq/27-eth0-tx-0-21swapper/220:31:092
28855040,0irq/27-eth0-tx-0-21swapper/219:06:372
28855040,0irq/27-eth0-tx-0-21swapper/200:21:262
28855040,0irq/27-eth0-tx-0-21swapper/200:16:152
28845040,0irq/26-eth0-rx-0-21swapper/323:51:193
28845040,0irq/26-eth0-rx-0-21swapper/323:46:393
28845040,0irq/26-eth0-rx-0-21swapper/322:26:183
28845040,0irq/26-eth0-rx-0-21swapper/322:12:153
28845040,0irq/26-eth0-rx-0-21swapper/319:41:303
28845040,0irq/26-eth0-rx-0-21swapper/300:01:393
259940,0migration/25354-21aten2_r1power_c21:36:252
259940,0migration/231233-21kthreadcore21:11:302
259940,0migration/22874-21unixbench-2d19:11:372
259940,0migration/211702-21fschecks_time19:46:272
249940,0watchdog/221330-21kthreadcore19:26:322
249940,0watchdog/212832-21diskmemload23:41:232
249940,0watchdog/20-21swapper/220:06:162
179940,0migration/132001-21fschecks_time20:26:271
179940,0migration/131814-21kthreadcore21:31:361
179940,0migration/131202-21munin-node20:01:261
179940,0migration/120023-21kthreadcore19:01:321
160689943,0cyclictest3469-21lldpd20:47:072
160689943,0cyclictest3469-21lldpd20:22:062
160689943,0cyclictest25633-21awk00:26:272
160689943,0cyclictest22630-21awk21:46:312
160689943,0cyclictest19143-21awk20:16:282
160689943,0cyclictest1665-21awk00:11:272
160679944,0cyclictest0-21swapper/123:34:181
160679944,0cyclictest0-21swapper/122:25:531
160679944,0cyclictest0-21swapper/121:59:141
160679944,0cyclictest0-21swapper/121:41:251
160679943,0cyclictest9994-21awk23:56:271
160679943,0cyclictest3469-21lldpd23:31:091
160679943,0cyclictest3469-21lldpd22:05:381
160679943,0cyclictest3469-21lldpd21:25:381
160679943,0cyclictest3469-21lldpd19:21:361
160679943,0cyclictest3427-21snmpd23:23:061
160679943,0cyclictest3427-21snmpd21:14:501
160679943,0cyclictest3427-21snmpd20:25:491
160679943,0cyclictest3427-21snmpd20:08:341
160679943,0cyclictest3427-21snmpd19:50:491
160679943,0cyclictest3427-21snmpd19:17:481
160679943,0cyclictest3427-21snmpd19:11:181
160679943,0cyclictest3427-21snmpd19:07:031
160679943,0cyclictest31803-21awk22:31:281
160679943,0cyclictest3154-21irqbalance21:16:321
160679943,0cyclictest31021-21awk19:36:281
160679943,0cyclictest28869-21sadc21:11:131
160679943,0cyclictest26184-21awk00:06:281
160679943,0cyclictest24952-21awk23:06:311
160679943,0cyclictest23687-21awk22:26:271
160679943,0cyclictest17752-21awk23:41:271
160669944,0cyclictest12832-21diskmemload22:28:520
160669944,0cyclictest0-21swapper/022:16:250
160669943,0cyclictest9545-21awk23:36:280
160669943,0cyclictest3469-21lldpd23:02:380
160669943,0cyclictest3469-21lldpd21:54:380
160669943,0cyclictest3427-21snmpd22:03:500
160669943,0cyclictest25381-21awk23:26:280
160669943,0cyclictest18149-21awk00:01:310
160669943,0cyclictest12537-21awk20:11:280
160669943,0cyclictest1208-21awk23:31:290
160669940,0cyclictest4659-21awk21:16:130
169940,0watchdog/17811-21df_abs22:56:271
169940,0watchdog/13708-21grep22:51:311
169940,0watchdog/121612-21kthreadcore19:51:321
169940,0watchdog/121332-21kthreadcore19:26:321
169940,0watchdog/117617-21memory20:36:321
169940,0watchdog/112575-21if_eth020:11:281
169940,0watchdog/111719-21kthreadcore22:36:311
169940,0watchdog/10-21swapper/123:46:191
169940,0watchdog/10-21swapper/122:13:311
169940,0watchdog/10-21swapper/121:36:321
169940,0watchdog/10-21swapper/100:23:551
149940,0migration/07639-21kthreadcore21:56:330
149940,0migration/04541-21sh22:12:250
149940,0migration/031927-21kthreadcore19:11:310
149940,0migration/028975-21kthreadcore20:21:340
149940,0migration/025862-21kthreadcore21:46:350
149940,0migration/023329-21sh00:22:350
149940,0migration/022057-21sh21:06:280
149940,0migration/020711-21kthreadcore19:51:320
149940,0migration/019053-21aten2_r1power_p20:41:250
149940,0migration/017815-21kthreadcore21:01:320
339930,0migration/323276-21munin-node19:06:263
28845030,0irq/26-eth0-rx-0-21swapper/320:41:313
28845030,0irq/26-eth0-rx-0-21swapper/320:38:083
28845030,0irq/26-eth0-rx-0-21swapper/320:19:073
28845030,0irq/26-eth0-rx-0-21swapper/319:38:223
28845030,0irq/26-eth0-rx-0-21swapper/319:22:463
28845030,0irq/26-eth0-rx-0-21swapper/319:01:253
259930,0migration/28521-21kthreadcore20:56:302
249930,0watchdog/20-21swapper/220:31:362
179930,0migration/127206-21kthreadcore19:56:311
160689932,0cyclictest30974-21awk19:36:282
160689932,0cyclictest17336-21sensors20:11:362
160689932,0cyclictest16367-21cut20:36:312
160679933,0cyclictest3469-21lldpd20:53:371
160679933,0cyclictest3469-21lldpd20:53:371
160679933,0cyclictest0-21swapper/119:31:211
160679932,0cyclictest3427-21snmpd20:56:201
160679932,0cyclictest3427-21snmpd20:46:491
160679932,0cyclictest3427-21snmpd20:41:191
160679932,0cyclictest3154-21irqbalance23:17:121
160679932,0cyclictest1692-21awk00:11:281
160679932,0cyclictest11389-21sensors20:31:361
160679930,0cyclictest19274-21iostat20:16:291
160669933,0cyclictest13189-21awk20:36:280
160669932,0cyclictest9681-21awk19:16:370
160669932,0cyclictest7009-21irqstats20:31:290
160669932,0cyclictest6209-21munin-run20:56:130
160669932,0cyclictest6209-21munin-run20:56:130
160669932,0cyclictest3427-21snmpd22:31:210
160669932,0cyclictest3427-21snmpd19:46:000
160669932,0cyclictest3427-21snmpd19:35:580
160669932,0cyclictest31689-21awk20:01:290
160669932,0cyclictest31151-21sensors20:46:380
160669932,0cyclictest30176-21grep21:31:310
160669932,0cyclictest25203-21awk19:56:290
160669932,0cyclictest22152-21awk21:26:300
160669932,0cyclictest20257-21kthreadcore22:41:320
160669932,0cyclictest15435-21sadc23:01:120
160669932,0cyclictest14036-21awk21:21:280
160669932,0cyclictest11877-21awk19:46:280
160669930,0cyclictest0-21swapper/019:01:270
169930,0watchdog/10-21swapper/123:36:271
169930,0watchdog/10-21swapper/122:46:511
169930,0watchdog/10-21swapper/119:41:321
169930,0watchdog/10-21swapper/100:26:271
169930,0watchdog/10-21swapper/100:01:271
35120,0ktimersoftd/35746-21runrttasks20:31:173
28845020,0irq/26-eth0-rx-0-21swapper/319:46:323
28845020,0irq/26-eth0-rx-0-21swapper/319:31:233
160669921,0cyclictest0-21swapper/023:41:120
160669921,0cyclictest0-21swapper/019:06:140
160669920,0cyclictest0-21swapper/020:16:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional