You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-08 - 07:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of shadow in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Thu Jun 08, 2023 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
295750410,0irq/25-eth10-21swapper/318:44:273
2599120,0migration/29696-21kthreadcore18:43:072
41100,0ktimersoftd/00-21swapper/018:43:020
19190,0ktimersoftd/10-21swapper/118:43:001
4160,0ktimersoftd/00-21swapper/021:52:300
35160,0ktimersoftd/30-21swapper/321:16:173
339960,0migration/33192-21sh23:00:043
339960,0migration/328014-21sh23:14:493
339960,0migration/323896-21munin-node21:53:153
339960,0migration/321936-21sh00:12:583
339960,0migration/318313-21sh22:30:193
28855060,0irq/27-eth0-tx-5969-21munin-node19:08:130
149960,0migration/029550-21aten2_r1power_c23:18:030
149960,0migration/012189-21sh23:24:480
116709965,0cyclictest23493-21ssh20:54:570
116709960,0cyclictest6039-21avahi-daemon22:37:160
116709960,0cyclictest0-21swapper/020:48:570
8225050,0irq/24-0000:00:5236-21cat23:22:462
4150,0ktimersoftd/00-21swapper/020:58:060
339950,0migration/37071-21timerwakeupswit21:23:193
339950,0migration/34285-21sh23:20:473
339950,0migration/33261-21irqrtprio20:43:093
339950,0migration/332365-21munin-node21:58:103
339950,0migration/331843-21switchtime21:38:173
339950,0migration/330693-21timerandwakeup20:58:183
339950,0migration/325187-21munin-node19:43:133
339950,0migration/319900-21timerwakeupswit00:08:153
339950,0migration/317171-21munin-node22:08:193
28855050,0irq/27-eth0-tx-8985-21df_abs21:08:062
28855050,0irq/27-eth0-tx-6677-21munin-node19:33:210
28855050,0irq/27-eth0-tx-6331-21runrttasks23:47:412
28855050,0irq/27-eth0-tx-3455-21lldpd21:31:392
28855050,0irq/27-eth0-tx-31874-21cat23:58:102
28855050,0irq/27-eth0-tx-30411-21cpu23:38:052
28855050,0irq/27-eth0-tx-27922-21munin-node22:38:062
28855050,0irq/27-eth0-tx-27850-21munin-node20:38:110
28855050,0irq/27-eth0-tx-25599-21ssh21:18:042
28855050,0irq/27-eth0-tx-20806-21cut21:53:132
28855050,0irq/27-eth0-tx-19599-21awk23:48:182
28855050,0irq/27-eth0-tx-14643-21if_eth123:28:072
28855050,0irq/27-eth0-tx-0-21swapper/223:53:462
28855050,0irq/27-eth0-tx-0-21swapper/223:34:312
28855050,0irq/27-eth0-tx-0-21swapper/223:16:272
28855050,0irq/27-eth0-tx-0-21swapper/223:08:582
28855050,0irq/27-eth0-tx-0-21swapper/223:04:462
28855050,0irq/27-eth0-tx-0-21swapper/222:58:182
28855050,0irq/27-eth0-tx-0-21swapper/222:51:392
28855050,0irq/27-eth0-tx-0-21swapper/222:34:112
28855050,0irq/27-eth0-tx-0-21swapper/222:18:052
28855050,0irq/27-eth0-tx-0-21swapper/222:16:452
28855050,0irq/27-eth0-tx-0-21swapper/222:11:112
28855050,0irq/27-eth0-tx-0-21swapper/221:59:332
28855050,0irq/27-eth0-tx-0-21swapper/221:22:542
28855050,0irq/27-eth0-tx-0-21swapper/221:03:282
28855050,0irq/27-eth0-tx-0-21swapper/220:53:002
28855050,0irq/27-eth0-tx-0-21swapper/200:16:312
28855050,0irq/27-eth0-tx-0-21swapper/019:43:310
28855050,0irq/27-eth0-tx-0-21swapper/019:38:170
28845050,0irq/26-eth0-rx-6039-21avahi-daemon21:03:143
28845050,0irq/26-eth0-rx-4723-21cut21:43:133
28845050,0irq/26-eth0-rx-22519-21kthreadcore22:53:113
28845050,0irq/26-eth0-rx-21520-21grep21:33:173
28845050,0irq/26-eth0-rx-20524-21kthreadcore22:13:093
28845050,0irq/26-eth0-rx-185052sleep322:48:473
28845050,0irq/26-eth0-rx-18260-21grep20:53:113
28845050,0irq/26-eth0-rx-14812-21ssh21:08:213
28845050,0irq/26-eth0-rx-11631-21ssh22:47:123
28845050,0irq/26-eth0-rx-0-21swapper/323:58:533
28845050,0irq/26-eth0-rx-0-21swapper/323:52:523
28845050,0irq/26-eth0-rx-0-21swapper/323:51:393
28845050,0irq/26-eth0-rx-0-21swapper/323:46:383
28845050,0irq/26-eth0-rx-0-21swapper/323:42:183
28845050,0irq/26-eth0-rx-0-21swapper/323:35:313
28845050,0irq/26-eth0-rx-0-21swapper/323:23:133
28845050,0irq/26-eth0-rx-0-21swapper/323:08:133
28845050,0irq/26-eth0-rx-0-21swapper/322:39:143
28845050,0irq/26-eth0-rx-0-21swapper/322:23:423
28845050,0irq/26-eth0-rx-0-21swapper/322:17:473
28845050,0irq/26-eth0-rx-0-21swapper/322:07:463
28845050,0irq/26-eth0-rx-0-21swapper/321:50:343
28845050,0irq/26-eth0-rx-0-21swapper/321:31:003
28845050,0irq/26-eth0-rx-0-21swapper/321:17:483
28845050,0irq/26-eth0-rx-0-21swapper/320:48:553
27150,0ktimersoftd/26039-21avahi-daemon20:58:062
259950,0migration/2600-21latency20:38:112
259950,0migration/216882-21kthreadcore18:48:062
259950,0migration/213433-21sh00:07:012
179950,0migration/132335-21sh21:02:101
179950,0migration/128869-21sh23:55:071
179950,0migration/126785-21ntp_kernel_pll_23:13:131
179950,0migration/120736-21cpuspeed_turbos22:53:041
179950,0migration/114891-21munin-node23:48:091
179950,0migration/110032-21sh22:25:061
149950,0migration/08208-21cpu20:23:040
149950,0migration/04185-21kthreadcore21:23:120
149950,0migration/03655-21switchtime23:58:160
149950,0migration/032070-21df_abs19:03:070
149950,0migration/030879-21fschecks_time23:38:070
149950,0migration/028952-21kthreadcore21:38:110
149950,0migration/026436-21taskset19:47:540
149950,0migration/024927-21sh21:53:510
149950,0migration/02287-21munin-node21:43:070
149950,0migration/020524-21sh21:13:140
149950,0migration/017566-21sh22:09:170
149950,0migration/016358-21kthreadcore22:28:090
149950,0migration/015056-21kthreadcore22:48:130
149950,0migration/013355-21idleruntime19:13:080
116729955,0cyclictest0-21swapper/222:54:132
116729955,0cyclictest0-21swapper/221:44:322
116729953,0cyclictest3427-21snmpd19:09:562
116729950,0cyclictest0-21swapper/221:16:292
116719955,0cyclictest0-21swapper/123:01:021
116719955,0cyclictest0-21swapper/122:11:411
116719955,0cyclictest0-21swapper/121:20:081
116719955,0cyclictest0-21swapper/100:13:241
116719950,0cyclictest6039-21avahi-daemon23:39:401
116719950,0cyclictest6039-21avahi-daemon22:47:101
116719950,0cyclictest0-21swapper/121:49:471
116709955,0cyclictest8497-21diskmemload23:55:470
116709955,0cyclictest6039-21avahi-daemon19:53:090
116709955,0cyclictest6039-21avahi-daemon19:23:210
116709955,0cyclictest0-21swapper/023:34:200
116709950,0cyclictest0-21swapper/023:09:350
339940,0migration/37608-21if_eth1.319:33:123
339940,0migration/332762-21fschecks_count19:28:073
339940,0migration/326865-21runrttasks20:12:043
339940,0migration/31237-21fschecks_time19:53:103
28855040,0irq/27-eth0-tx-0-21swapper/222:28:152
28855040,0irq/27-eth0-tx-0-21swapper/222:22:502
28855040,0irq/27-eth0-tx-0-21swapper/221:37:492
28855040,0irq/27-eth0-tx-0-21swapper/221:32:582
28855040,0irq/27-eth0-tx-0-21swapper/220:48:272
28855040,0irq/27-eth0-tx-0-21swapper/200:07:482
28855040,0irq/27-eth0-tx-0-21swapper/020:33:280
28855040,0irq/27-eth0-tx-0-21swapper/020:03:210
28855040,0irq/27-eth0-tx-0-21swapper/018:58:100
28845040,0irq/26-eth0-rx-6039-21avahi-daemon00:02:493
28845040,0irq/26-eth0-rx-0-21swapper/323:28:063
28845040,0irq/26-eth0-rx-0-21swapper/323:04:413
28845040,0irq/26-eth0-rx-0-21swapper/322:33:043
28845040,0irq/26-eth0-rx-0-21swapper/320:33:063
28845040,0irq/26-eth0-rx-0-21swapper/319:58:113
259940,0migration/228052-21munin-node20:13:042
259940,0migration/22637-21latency_hist22:02:462
259940,0migration/220731-21cpuspeed_turbos20:08:032
249940,0watchdog/218558-21taskset19:13:262
249940,0watchdog/216586-21ssh21:48:372
249940,0watchdog/20-21swapper/220:03:052
249940,0watchdog/20-21swapper/219:43:092
179940,0migration/132681-21interrupts19:03:101
179940,0migration/130846-21gltestperf23:58:061
179940,0migration/12467-21fschecks_time21:43:081
179940,0migration/121369-21kthreadcore19:18:121
179940,0migration/11799-21kthreadcore23:18:101
179940,0migration/113910-21unixbench-2d20:23:171
179940,0migration/11356-21runrttasks20:17:591
169940,0watchdog/19053-21ssh22:03:291
169940,0watchdog/18442-21kthreadcore23:03:171
169940,0watchdog/125796-21users19:43:171
169940,0watchdog/116445-21cut00:08:081
169940,0watchdog/115230-21kthreadcore20:03:131
169940,0watchdog/114953-21grep18:48:061
169940,0watchdog/114541-21kthreadcore22:28:091
169940,0watchdog/113717-21ssh23:28:001
169940,0watchdog/10-21swapper/121:56:251
169940,0watchdog/10-21swapper/120:28:091
149940,0migration/029426-21kthreadcore22:18:090
149940,0migration/02123-21sensors_volt22:58:210
149940,0migration/015940-21kthreadcore20:28:090
116729944,0cyclictest1900-21kthreadcore19:53:142
116729943,0cyclictest4688-21awk22:43:062
116729943,0cyclictest3427-21snmpd23:22:582
116729943,0cyclictest3427-21snmpd20:42:562
116729943,0cyclictest3427-21snmpd20:35:412
116729943,0cyclictest3427-21snmpd20:29:112
116729943,0cyclictest3427-21snmpd19:47:562
116729943,0cyclictest3427-21snmpd19:38:262
116729943,0cyclictest3427-21snmpd19:02:552
116729943,0cyclictest3154-21irqbalance20:24:162
116729943,0cyclictest3154-21irqbalance20:20:362
116729943,0cyclictest3154-21irqbalance20:01:362
116729943,0cyclictest3154-21irqbalance19:27:562
116729943,0cyclictest3154-21irqbalance19:22:562
116729943,0cyclictest3154-21irqbalance19:19:062
116729943,0cyclictest3154-21irqbalance18:52:562
116729943,0cyclictest25948-21awk18:58:082
116719944,0cyclictest6039-21avahi-daemon23:45:491
116719944,0cyclictest0-21swapper/122:41:011
116719943,0cyclictest9562-21awk21:08:091
116719943,0cyclictest8517-21kworker/1:120:53:341
116719943,0cyclictest7534-21awk19:33:121
116719943,0cyclictest3469-21lldpd19:55:011
116719943,0cyclictest3427-21snmpd21:28:571
116719943,0cyclictest3427-21snmpd20:15:261
116719943,0cyclictest3427-21snmpd19:12:561
116719943,0cyclictest3427-21snmpd18:59:551
116719943,0cyclictest3154-21irqbalance23:32:551
116719943,0cyclictest3154-21irqbalance23:23:051
116719943,0cyclictest3154-21irqbalance22:35:451
116719943,0cyclictest3154-21irqbalance22:18:551
116719943,0cyclictest3154-21irqbalance22:14:561
116719943,0cyclictest3154-21irqbalance21:58:061
116719943,0cyclictest3154-21irqbalance21:23:561
116719943,0cyclictest3154-21irqbalance21:03:061
116719943,0cyclictest3154-21irqbalance20:48:061
116719943,0cyclictest3154-21irqbalance20:43:261
116719943,0cyclictest3154-21irqbalance20:38:061
116719943,0cyclictest3154-21irqbalance19:59:261
116719943,0cyclictest26858-21awk21:38:091
116719943,0cyclictest17848-21awk21:13:111
116719943,0cyclictest17440-21kworker/1:220:33:161
116719943,0cyclictest17440-21kworker/1:220:07:491
116719943,0cyclictest17440-21kworker/1:219:48:051
116719943,0cyclictest17440-21kworker/1:219:37:491
116719943,0cyclictest17440-21kworker/1:219:08:031
116719943,0cyclictest14487-21kworker/1:223:08:501
116719943,0cyclictest14487-21kworker/1:221:34:481
116719942,0cyclictest3427-21snmpd19:27:111
116709943,0cyclictest6839-21awk23:43:080
116709943,0cyclictest3469-21lldpd21:21:080
116709943,0cyclictest3427-21snmpd23:14:430
116709943,0cyclictest3427-21snmpd20:45:410
116709943,0cyclictest3427-21snmpd19:27:560
116709943,0cyclictest3427-21snmpd19:17:560
116709943,0cyclictest3427-21snmpd18:49:550
116709943,0cyclictest3154-21irqbalance23:48:450
116709943,0cyclictest3154-21irqbalance23:28:550
116709943,0cyclictest3154-21irqbalance23:04:450
116709943,0cyclictest3154-21irqbalance22:43:350
116709943,0cyclictest3154-21irqbalance22:24:550
116709943,0cyclictest3154-21irqbalance22:14:260
116709943,0cyclictest3154-21irqbalance22:03:160
116709943,0cyclictest3154-21irqbalance22:00:160
116709943,0cyclictest3154-21irqbalance21:10:060
116709943,0cyclictest3154-21irqbalance21:06:160
116709943,0cyclictest3154-21irqbalance20:17:560
116709943,0cyclictest3154-21irqbalance20:12:560
116709943,0cyclictest3154-21irqbalance20:08:260
116709943,0cyclictest3154-21irqbalance00:14:150
116709943,0cyclictest3154-21irqbalance00:08:050
116709943,0cyclictest3154-21irqbalance00:02:550
116709943,0cyclictest28757-21awk22:38:050
116709943,0cyclictest18651-21awk21:33:130
339930,0migration/332042-21sendmail_mailtr19:48:123
339930,0migration/317768-21latency19:13:123
339930,0migration/311851-21latency_hist18:47:473
28855030,0irq/27-eth0-tx-0-21swapper/018:53:100
28845030,0irq/26-eth0-rx-6039-21avahi-daemon19:27:043
116729932,0cyclictest3427-21snmpd19:35:112
116719933,0cyclictest17440-21kworker/1:219:27:571
116719933,0cyclictest17440-21kworker/1:218:52:491
116719933,0cyclictest14487-21kworker/1:200:02:471
116719932,0cyclictest3154-21irqbalance22:47:551
116709932,0cyclictest3427-21snmpd21:28:120
116709932,0cyclictest3427-21snmpd20:01:110
116709932,0cyclictest21346-21awk22:53:080
339920,0migration/331453-21kthreadcore20:13:073
339920,0migration/315017-21diskstats20:28:053
329920,0watchdog/30-21swapper/320:03:203
329920,0watchdog/30-21swapper/319:18:123
28845020,0irq/26-eth0-rx-6039-21avahi-daemon20:39:273
28845020,0irq/26-eth0-rx-6039-21avahi-daemon19:38:173
28845020,0irq/26-eth0-rx-6039-21avahi-daemon19:05:453
28845020,0irq/26-eth0-rx-3484-21kthreadcore20:18:083
28845020,0irq/26-eth0-rx-0-21swapper/320:23:123
28845020,0irq/26-eth0-rx-0-21swapper/319:08:113
28845020,0irq/26-eth0-rx-0-21swapper/318:58:083
28845020,0irq/26-eth0-rx-0-21swapper/318:53:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional