You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-09 - 07:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.4.229-rt200 (Profile)

Latency plot of shadow in rack #1, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -n -a -t3 -p99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot2s.osadl.org (updated Tue Aug 09, 2022 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2977528382,0sleep00-21swapper/018:51:130
3063923624,0sleep10-21swapper/118:52:061
3063623525,0sleep30-21swapper/318:52:043
3089323324,0sleep20-21swapper/218:55:032
339960,0migration/3953-21sh23:14:303
339960,0migration/33867-21sh21:19:173
339960,0migration/32735-21sh23:55:153
339960,0migration/32426-21sh20:57:493
339960,0migration/323635-21kthreadcore19:36:143
339960,0migration/32081-21sh23:35:143
339960,0migration/319517-21sh21:26:513
339960,0migration/314161-21munin-node22:41:173
310389960,0cyclictest27952-21diskmemload21:39:201
310389960,0cyclictest0-21swapper/122:37:221
310379966,0cyclictest10965-21kworker/0:122:05:330
310379966,0cyclictest10965-21kworker/0:121:33:210
28715060,0irq/26-eth0-rx-458-21cut22:36:093
28715060,0irq/26-eth0-rx-30699-21sh22:32:213
28715060,0irq/26-eth0-rx-1734-21munin-node22:56:103
28715060,0irq/26-eth0-rx-14074-21ssh22:04:133
28715060,0irq/26-eth0-rx-10182-21grep20:16:133
339950,0migration/37493-21fschecks_time19:51:143
339950,0migration/329827-21aten2_r1power_c21:36:053
339950,0migration/329079-21kthreadcore20:06:123
339950,0migration/321148-21sh21:11:093
339950,0migration/320963-21aten2_r1power_v20:26:093
339950,0migration/319640-21kthreadcore22:06:133
339950,0migration/319640-21kthreadcore22:06:133
339950,0migration/318023-21kernelversion22:46:143
339950,0migration/316807-21if_eth122:26:123
339950,0migration/313748-21ntp_offset20:41:153
339950,0migration/313234-21kthreadcore23:21:133
339950,0migration/31273-21kthreadcore21:56:153
339950,0migration/310792-21taskset00:00:103
329950,0watchdog/36592-21grep20:36:133
329950,0watchdog/35281-21switchtime19:21:183
329950,0watchdog/34571-21grep18:56:183
329950,0watchdog/324348-21sensors_volt19:11:183
329950,0watchdog/320143-21basename23:46:133
329950,0watchdog/318027-21sh23:43:253
329950,0watchdog/313000-21memory22:21:173
329950,0watchdog/312734-21cat00:01:133
329950,0watchdog/30-21swapper/300:05:533
310399955,0cyclictest0-21swapper/223:36:242
310399955,0cyclictest0-21swapper/222:43:542
310399955,0cyclictest0-21swapper/221:58:492
310399955,0cyclictest0-21swapper/221:11:312
310399955,0cyclictest0-21swapper/221:09:472
310399955,0cyclictest0-21swapper/219:56:192
310399950,0cyclictest0-21swapper/223:52:382
310399950,0cyclictest0-21swapper/223:16:512
310399950,0cyclictest0-21swapper/222:56:592
310399950,0cyclictest0-21swapper/222:10:192
310399950,0cyclictest0-21swapper/222:10:192
310399950,0cyclictest0-21swapper/221:49:592
310399950,0cyclictest0-21swapper/219:34:432
310389955,0cyclictest0-21swapper/122:43:491
310389955,0cyclictest0-21swapper/121:22:001
310389955,0cyclictest0-21swapper/100:03:451
310389950,0cyclictest0-21swapper/122:10:151
310389950,0cyclictest0-21swapper/122:10:151
310389950,0cyclictest0-21swapper/121:49:541
310389950,0cyclictest0-21swapper/120:59:181
310389950,0cyclictest0-21swapper/100:11:291
310379955,0cyclictest10965-21kworker/0:123:56:150
310379955,0cyclictest10965-21kworker/0:123:50:590
310379955,0cyclictest10965-21kworker/0:123:46:010
310379955,0cyclictest10965-21kworker/0:123:41:090
310379955,0cyclictest10965-21kworker/0:123:35:530
310379955,0cyclictest10965-21kworker/0:123:30:530
310379955,0cyclictest10965-21kworker/0:123:26:070
310379955,0cyclictest10965-21kworker/0:123:20:530
310379955,0cyclictest10965-21kworker/0:123:15:590
310379955,0cyclictest10965-21kworker/0:123:10:590
310379955,0cyclictest10965-21kworker/0:123:05:530
310379955,0cyclictest10965-21kworker/0:123:00:550
310379955,0cyclictest10965-21kworker/0:122:56:010
310379955,0cyclictest10965-21kworker/0:122:51:090
310379955,0cyclictest10965-21kworker/0:122:46:090
310379955,0cyclictest10965-21kworker/0:122:35:590
310379955,0cyclictest10965-21kworker/0:122:30:590
310379955,0cyclictest10965-21kworker/0:122:27:070
310379955,0cyclictest10965-21kworker/0:122:20:550
310379955,0cyclictest10965-21kworker/0:122:15:590
310379955,0cyclictest10965-21kworker/0:122:11:050
310379955,0cyclictest10965-21kworker/0:122:05:550
310379955,0cyclictest10965-21kworker/0:122:05:550
310379955,0cyclictest10965-21kworker/0:121:55:570
310379955,0cyclictest10965-21kworker/0:121:50:570
310379955,0cyclictest10965-21kworker/0:121:45:550
310379955,0cyclictest10965-21kworker/0:121:41:010
310379955,0cyclictest10965-21kworker/0:121:35:550
310379955,0cyclictest10965-21kworker/0:121:26:050
310379955,0cyclictest10965-21kworker/0:121:21:010
310379955,0cyclictest10965-21kworker/0:121:16:070
310379955,0cyclictest10965-21kworker/0:121:10:550
310379955,0cyclictest10965-21kworker/0:121:06:010
310379955,0cyclictest10965-21kworker/0:121:00:550
310379955,0cyclictest10965-21kworker/0:120:56:130
310379955,0cyclictest10965-21kworker/0:120:50:550
310379955,0cyclictest10965-21kworker/0:120:46:030
310379955,0cyclictest10965-21kworker/0:120:40:570
310379955,0cyclictest10965-21kworker/0:120:35:570
310379955,0cyclictest10965-21kworker/0:120:30:570
310379955,0cyclictest10965-21kworker/0:120:25:590
310379955,0cyclictest10965-21kworker/0:120:20:590
310379955,0cyclictest10965-21kworker/0:120:16:000
310379955,0cyclictest10965-21kworker/0:120:10:560
310379955,0cyclictest10965-21kworker/0:120:05:560
310379955,0cyclictest10965-21kworker/0:120:00:580
310379955,0cyclictest10965-21kworker/0:119:55:580
310379955,0cyclictest10965-21kworker/0:119:50:580
310379955,0cyclictest10965-21kworker/0:119:45:560
310379955,0cyclictest10965-21kworker/0:119:36:020
310379955,0cyclictest10965-21kworker/0:119:30:580
310379955,0cyclictest10965-21kworker/0:119:26:000
310379955,0cyclictest10965-21kworker/0:119:20:580
310379955,0cyclictest10965-21kworker/0:119:16:000
310379955,0cyclictest10965-21kworker/0:119:11:000
310379955,0cyclictest10965-21kworker/0:119:06:140
310379955,0cyclictest10965-21kworker/0:119:01:000
310379955,0cyclictest10965-21kworker/0:118:56:060
310379955,0cyclictest10965-21kworker/0:100:21:090
310379955,0cyclictest10965-21kworker/0:100:15:590
310379955,0cyclictest10965-21kworker/0:100:10:530
310379955,0cyclictest10965-21kworker/0:100:05:550
28725050,0irq/27-eth0-tx-11089-21munin-node23:41:172
28725050,0irq/27-eth0-tx-0-21swapper/223:56:112
28715050,0irq/26-eth0-rx-9313-21ssh23:00:593
28715050,0irq/26-eth0-rx-8209-21cut21:21:173
28715050,0irq/26-eth0-rx-6301-21grep23:16:143
28715050,0irq/26-eth0-rx-26333-21ssh22:51:143
28715050,0irq/26-eth0-rx-21718-21cpu20:51:093
28715050,0irq/26-eth0-rx-21500-21ssh21:31:063
28715050,0irq/26-eth0-rx-19649-21irqrtprio23:26:123
28715050,0irq/26-eth0-rx-19077-21kthreadcore23:06:113
28715050,0irq/26-eth0-rx-0-21swapper/323:37:523
28715050,0irq/26-eth0-rx-0-21swapper/322:16:133
28715050,0irq/26-eth0-rx-0-21swapper/322:12:353
28715050,0irq/26-eth0-rx-0-21swapper/321:46:043
28715050,0irq/26-eth0-rx-0-21swapper/321:41:063
28715050,0irq/26-eth0-rx-0-21swapper/321:06:073
28715050,0irq/26-eth0-rx-0-21swapper/300:23:443
28715050,0irq/26-eth0-rx-0-21swapper/300:16:143
28715050,0irq/26-eth0-rx-0-21swapper/300:12:343
249950,0watchdog/219252-21latency_hist00:05:532
249950,0watchdog/20-21swapper/219:36:132
249950,0watchdog/20-21swapper/219:06:142
179950,0migration/1831-21kthreadcore19:21:111
179950,0migration/122525-21kthreadcore20:01:121
179950,0migration/114346-21idleruntime19:56:121
179950,0migration/111000-21taskset21:03:181
179950,0migration/110570-21df_abs23:21:081
179950,0migration/110145-21kthreadcore19:01:151
169950,0watchdog/129854-21kthreadcore23:11:131
169950,0watchdog/116887-21iostat_ios22:26:131
169950,0watchdog/11474-21fschecks_time22:56:091
149950,0migration/026032-21latency_hist19:40:540
149950,0migration/012975-21kthreadcore00:01:130
149950,0migration/010815-21kthreadcore22:41:120
8215040,0irq/24-0000:00:12186-21iostat_ios00:21:182
339940,0migration/39761-21kthreadcore19:26:113
339940,0migration/39072-21kthreadcore21:01:153
339940,0migration/318368-21kthreadcore20:21:163
329940,0watchdog/31464-21irqrtprio19:46:133
329940,0watchdog/30-21swapper/320:48:093
329940,0watchdog/30-21swapper/320:01:133
329940,0watchdog/30-21swapper/319:16:143
329940,0watchdog/30-21swapper/319:06:143
310399944,0cyclictest0-21swapper/221:36:162
310399943,0cyclictest6406-21awk19:01:122
310399943,0cyclictest3415-21snmpd21:51:332
310399943,0cyclictest3415-21snmpd20:52:482
310399943,0cyclictest3415-21snmpd20:41:482
310399943,0cyclictest3415-21snmpd20:18:482
310399943,0cyclictest3415-21snmpd19:26:022
310399943,0cyclictest3415-21snmpd19:24:172
310399943,0cyclictest3415-21snmpd19:00:472
310399943,0cyclictest27647-21awk20:06:092
310399943,0cyclictest26271-21awk19:16:122
310399943,0cyclictest2476-21awk20:36:102
310399943,0cyclictest1812-21awk20:11:102
310399943,0cyclictest11080-21awk23:21:112
310389944,0cyclictest0-21swapper/123:36:191
310389944,0cyclictest0-21swapper/123:16:461
310389944,0cyclictest0-21swapper/122:16:051
310389943,0cyclictest3457-21lldpd20:53:501
310389943,0cyclictest3457-21lldpd20:13:491
310389943,0cyclictest3457-21lldpd19:31:501
310389943,0cyclictest27695-21awk20:06:101
310389943,0cyclictest27668-21awk23:31:101
310389943,0cyclictest26247-21awk19:16:111
310389943,0cyclictest25888-21awk22:51:121
310389943,0cyclictest23348-21awk21:51:151
310389943,0cyclictest19415-21awk23:26:111
310389943,0cyclictest12083-21awk00:21:171
310389943,0cyclictest10235-21awk23:01:131
310389940,0cyclictest0-21swapper/119:45:501
28725040,0irq/27-eth0-tx-4845-21ssh21:21:082
28725040,0irq/27-eth0-tx-28237-21munin-node20:56:172
28725040,0irq/27-eth0-tx-0-21swapper/223:49:092
28725040,0irq/27-eth0-tx-0-21swapper/223:33:252
28725040,0irq/27-eth0-tx-0-21swapper/223:26:032
28725040,0irq/27-eth0-tx-0-21swapper/223:00:572
28725040,0irq/27-eth0-tx-0-21swapper/222:46:112
28725040,0irq/27-eth0-tx-0-21swapper/222:36:402
28725040,0irq/27-eth0-tx-0-21swapper/222:27:252
28725040,0irq/27-eth0-tx-0-21swapper/222:16:002
28725040,0irq/27-eth0-tx-0-21swapper/222:15:192
28725040,0irq/27-eth0-tx-0-21swapper/222:03:002
28725040,0irq/27-eth0-tx-0-21swapper/221:41:182
28725040,0irq/27-eth0-tx-0-21swapper/220:31:192
28725040,0irq/27-eth0-tx-0-21swapper/200:12:452
28715040,0irq/26-eth0-rx-6370-21runrttasks21:53:423
28715040,0irq/26-eth0-rx-0-21swapper/320:11:123
259940,0migration/25679-21kthreadcore21:01:132
259940,0migration/228541-21kthreadcore19:41:112
259940,0migration/222014-21kthreadcore20:01:122
259940,0migration/215031-21aten2_r1power_p20:46:102
249940,0watchdog/28861-21kthreadcore19:51:172
249940,0watchdog/23461-21diskstats00:16:092
249940,0watchdog/225936-21cat22:31:132
249940,0watchdog/224886-21kthreadcore21:31:132
249940,0watchdog/22391-21timerwakeupswit21:16:212
249940,0watchdog/21463-21irqrtprio19:46:132
249940,0watchdog/20-21swapper/223:11:132
249940,0watchdog/20-21swapper/222:50:532
249940,0watchdog/20-21swapper/222:21:212
249940,0watchdog/20-21swapper/221:26:092
249940,0watchdog/20-21swapper/200:01:132
179940,0migration/19525-21kthreadcore20:16:131
179940,0migration/16830-21kthreadcore00:16:141
179940,0migration/1620-21kthreadcore18:56:121
179940,0migration/130840-21aten2_r1power_e21:56:081
179940,0migration/130801-21kthreadcore23:51:191
179940,0migration/126826-21ksmtuned21:11:431
179940,0migration/122790-21kthreadcore00:06:131
179940,0migration/122478-21iostat21:31:111
179940,0migration/121964-21kthreadcore19:11:131
179940,0migration/121435-21latency_hist20:50:541
179940,0migration/118183-21switchtime21:06:191
179940,0migration/115668-21kthreadcore20:21:151
179940,0migration/110507-21kthreadcore20:41:131
169940,0watchdog/18250-21kthreadcore22:01:131
169940,0watchdog/17695-21cut23:56:171
169940,0watchdog/16683-21if_err_eth1.321:41:131
169940,0watchdog/130180-21ssh22:13:091
169940,0watchdog/121976-21kthreadcore19:36:131
169940,0watchdog/116934-21munin-node22:46:091
169940,0watchdog/113554-21diskstats21:26:091
169940,0watchdog/10-21swapper/123:46:131
169940,0watchdog/10-21swapper/122:31:131
169940,0watchdog/10-21swapper/121:16:211
329930,0watchdog/30-21swapper/319:56:133
310399930,0cyclictest16804-21cut20:21:162
310389932,0cyclictest8476-21awk22:21:131
310389932,0cyclictest8166-21kthreadcore19:26:111
310389932,0cyclictest3415-21snmpd20:31:061
310389932,0cyclictest23807-21kthreadcore20:26:151
310389932,0cyclictest2168-21grep20:36:091
310389932,0cyclictest16802-21sensors23:41:161
28725030,0irq/27-eth0-tx-0-21swapper/223:06:022
28715030,0irq/26-eth0-rx-0-21swapper/320:33:283
28715030,0irq/26-eth0-rx-0-21swapper/319:43:063
28715030,0irq/26-eth0-rx-0-21swapper/319:01:123
27130,0ktimersoftd/219452-21hddtemp_smartct19:11:092
249930,0watchdog/20-21swapper/220:26:132
169930,0watchdog/18862-21kthreadcore19:51:171
169930,0watchdog/127952-21diskmemload23:06:051
169930,0watchdog/115176-21kthreadcore19:06:141
169930,0watchdog/10-21swapper/119:46:171
28715020,0irq/26-eth0-rx-0-21swapper/319:30:543
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional