You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-04 - 10:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot2s.osadl.org (updated Mon Dec 04, 2023 00:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
291350660,0irq/25-eth10-21swapper/319:05:253
2297724329,0sleep00-21swapper/019:05:270
2300923525,0sleep10-21swapper/119:05:521
2285823228,0sleep20-21swapper/219:04:092
230859970,0cyclictest41ktimersoftd/022:17:260
339960,0migration/328396-21sendmail_mailtr22:51:273
339960,0migration/321118-21latency_hist22:25:563
339960,0migration/320318-21munin-node21:26:273
28415060,0irq/27-eth0-tx-0-21swapper/223:56:072
28415060,0irq/27-eth0-tx-0-21swapper/223:38:072
28415060,0irq/27-eth0-tx-0-21swapper/223:38:072
28415060,0irq/27-eth0-tx-0-21swapper/223:26:492
28415060,0irq/27-eth0-tx-0-21swapper/223:03:592
28415060,0irq/27-eth0-tx-0-21swapper/222:53:292
28415060,0irq/27-eth0-tx-0-21swapper/221:13:382
28415060,0irq/27-eth0-tx-0-21swapper/200:24:372
28415060,0irq/27-eth0-tx-0-21swapper/200:12:422
28405060,0irq/26-eth0-rx-18981-21awk19:51:163
249960,0watchdog/25108-21sh23:41:352
249960,0watchdog/20-21swapper/223:22:512
249960,0watchdog/20-21swapper/223:11:232
249960,0watchdog/20-21swapper/222:40:592
249960,0watchdog/20-21swapper/222:30:312
249960,0watchdog/20-21swapper/222:11:192
249960,0watchdog/20-21swapper/221:40:432
249960,0watchdog/20-21swapper/221:29:472
249960,0watchdog/20-21swapper/221:07:152
230859964,0cyclictest0-21swapper/023:35:520
230859961,0cyclictest0-21swapper/020:51:160
230859960,0cyclictest32420-21ssh00:06:220
230859960,0cyclictest0-21swapper/022:27:500
8225050,0irq/24-0000:00:14715-21dump-pmu-power21:00:562
339950,0migration/38514-21kthreadcore19:16:213
339950,0migration/35423-21kthreadcore20:26:273
339950,0migration/35018-21sh21:38:133
339950,0migration/33820-21taskset19:13:193
339950,0migration/331482-21taskset19:58:023
339950,0migration/330777-21kthreadcore22:06:173
339950,0migration/3301-21taskset00:07:313
339950,0migration/323604-21unixbench_multi19:26:293
339950,0migration/323292-21taskset19:06:023
339950,0migration/32054-21kthreadcore20:01:213
339950,0migration/316529-21kthreadcore23:26:323
339950,0migration/315057-21kthreadcore19:46:243
339950,0migration/314561-21latency_hist23:50:563
339950,0migration/313935-21taskset20:36:003
339950,0migration/313763-21munin-node21:46:193
339950,0migration/313757-21taskset22:44:103
28415050,0irq/27-eth0-tx-26291-21ssh22:01:572
28415050,0irq/27-eth0-tx-2221-21load22:31:282
28415050,0irq/27-eth0-tx-19401-21munin-node00:26:332
28415050,0irq/27-eth0-tx-0-21swapper/223:31:452
28415050,0irq/27-eth0-tx-0-21swapper/223:00:042
28415050,0irq/27-eth0-tx-0-21swapper/222:46:312
28415050,0irq/27-eth0-tx-0-21swapper/222:39:072
28415050,0irq/27-eth0-tx-0-21swapper/222:22:232
28415050,0irq/27-eth0-tx-0-21swapper/222:10:162
28415050,0irq/27-eth0-tx-0-21swapper/221:51:052
28415050,0irq/27-eth0-tx-0-21swapper/221:41:492
28415050,0irq/27-eth0-tx-0-21swapper/221:31:352
28415050,0irq/27-eth0-tx-0-21swapper/221:16:182
28415050,0irq/27-eth0-tx-0-21swapper/200:06:402
28415050,0irq/27-eth0-tx-0-21swapper/200:02:062
28405050,0irq/26-eth0-rx-9472-21irqrtprio20:56:233
28405050,0irq/26-eth0-rx-5224-21entropy22:36:223
28405050,0irq/26-eth0-rx-3613-21mdstat22:56:223
28405050,0irq/26-eth0-rx-3346-21taskset21:12:073
28405050,0irq/26-eth0-rx-18059-21basename21:56:103
28405050,0irq/26-eth0-rx-1659-21ssh00:10:593
28405050,0irq/26-eth0-rx-15945-21iostat21:01:183
28405050,0irq/26-eth0-rx-12901-40yum-updatesd-he22:20:333
28405050,0irq/26-eth0-rx-12495-21ps20:08:163
28405050,0irq/26-eth0-rx-0-21swapper/323:52:093
28405050,0irq/26-eth0-rx-0-21swapper/323:31:133
28405050,0irq/26-eth0-rx-0-21swapper/323:05:513
28405050,0irq/26-eth0-rx-0-21swapper/322:46:143
28405050,0irq/26-eth0-rx-0-21swapper/322:31:363
28405050,0irq/26-eth0-rx-0-21swapper/322:21:203
28405050,0irq/26-eth0-rx-0-21swapper/322:01:543
28405050,0irq/26-eth0-rx-0-21swapper/321:31:253
28405050,0irq/26-eth0-rx-0-21swapper/321:22:193
28405050,0irq/26-eth0-rx-0-21swapper/300:33:123
28405050,0irq/26-eth0-rx-0-21swapper/300:21:193
28405050,0irq/26-eth0-rx-0-21swapper/300:17:323
259950,0migration/218338-21cpuspeed_turbos21:56:112
249950,0watchdog/228060-21grep19:31:232
249950,0watchdog/214270-21unixbench_singl21:46:232
249950,0watchdog/212822-21unixbench_multi21:21:232
249950,0watchdog/211645-21ssh22:16:392
249950,0watchdog/20-21swapper/223:17:232
249950,0watchdog/20-21swapper/223:06:472
249950,0watchdog/20-21swapper/220:45:592
249950,0watchdog/20-21swapper/220:41:152
249950,0watchdog/20-21swapper/220:40:072
249950,0watchdog/20-21swapper/220:21:072
249950,0watchdog/20-21swapper/220:15:552
249950,0watchdog/20-21swapper/219:47:312
249950,0watchdog/20-21swapper/219:41:192
249950,0watchdog/20-21swapper/219:28:352
249950,0watchdog/20-21swapper/219:21:072
249950,0watchdog/20-21swapper/219:17:192
249950,0watchdog/20-21swapper/219:12:512
249950,0watchdog/20-21swapper/200:32:392
249950,0watchdog/20-21swapper/200:17:552
230889955,0cyclictest0-21swapper/223:51:182
230889955,0cyclictest0-21swapper/223:47:032
230889950,0cyclictest0-21swapper/219:06:352
230869954,0cyclictest692-21cut20:01:201
230869950,0cyclictest0-21swapper/123:54:191
230869950,0cyclictest0-21swapper/121:24:231
230859955,0cyclictest0-21swapper/022:38:330
230859955,0cyclictest0-21swapper/000:26:050
230859954,0cyclictest21519-21awk20:41:190
230859954,0cyclictest0-21swapper/020:21:160
230859952,0cyclictest7673-21sh23:46:140
230859952,0cyclictest7261-21grep21:21:130
230859952,0cyclictest3461-21find23:16:350
230859952,0cyclictest32015-21grep21:36:220
230859952,0cyclictest29919-21grep22:31:220
230859952,0cyclictest26327-21ps19:56:210
230859952,0cyclictest22825-21aten_r1power_po21:31:190
230859952,0cyclictest19098-21munin-node22:21:250
230859952,0cyclictest15325-21fschecks_count23:51:260
230859952,0cyclictest0-21swapper/023:06:150
230859952,0cyclictest0-21swapper/019:46:270
230859951,0cyclictest29980-21munin-node19:11:270
230859951,0cyclictest27262-21munin-node20:46:190
230859951,0cyclictest1579-21irqcore20:26:250
230859951,0cyclictest14326-21df20:36:150
230859951,0cyclictest0-21swapper/021:46:120
230859951,0cyclictest0-21swapper/021:06:200
230859951,0cyclictest0-21swapper/021:01:130
230859951,0cyclictest0-21swapper/020:06:180
230859951,0cyclictest0-21swapper/020:01:160
230859950,0cyclictest0-21swapper/023:37:120
230859950,0cyclictest0-21swapper/023:37:120
230859950,0cyclictest0-21swapper/023:21:260
230859950,0cyclictest0-21swapper/022:51:220
230859950,0cyclictest0-21swapper/020:56:310
230859950,0cyclictest0-21swapper/019:36:150
179950,0migration/16410-21munin-node22:16:151
179950,0migration/13478-21sendmail_mailst23:16:351
179950,0migration/128513-21taskset23:34:061
179950,0migration/123876-21kthreadcore23:11:301
179950,0migration/122545-21irqrtprio22:26:221
179950,0migration/122489-21munin-node21:06:201
179950,0migration/113323-21taskset23:04:271
169950,0watchdog/131076-21aten_r1power_vo19:36:151
169950,0watchdog/114347-21diskstats20:36:151
339940,0migration/329521-21taskset23:15:213
339940,0migration/325978-21munin-node20:41:233
339940,0migration/316816-21kthreadcore20:11:213
28415040,0irq/27-eth0-tx-0-21swapper/019:16:130
28405040,0irq/26-eth0-rx-16844-21grep23:06:223
28405040,0irq/26-eth0-rx-0-21swapper/323:58:263
28405040,0irq/26-eth0-rx-0-21swapper/323:42:333
28405040,0irq/26-eth0-rx-0-21swapper/323:40:293
28405040,0irq/26-eth0-rx-0-21swapper/323:40:293
28405040,0irq/26-eth0-rx-0-21swapper/323:23:013
28405040,0irq/26-eth0-rx-0-21swapper/323:16:123
28405040,0irq/26-eth0-rx-0-21swapper/322:11:483
28405040,0irq/26-eth0-rx-0-21swapper/321:52:113
28405040,0irq/26-eth0-rx-0-21swapper/321:40:583
28405040,0irq/26-eth0-rx-0-21swapper/321:16:333
28405040,0irq/26-eth0-rx-0-21swapper/321:06:173
28405040,0irq/26-eth0-rx-0-21swapper/319:21:153
28405040,0irq/26-eth0-rx-0-21swapper/300:26:133
28405040,0irq/26-eth0-rx-0-21swapper/300:01:183
259940,0migration/222379-21kthreadcore20:16:212
249940,0watchdog/24955-21missed_timers20:01:232
249940,0watchdog/20-21swapper/220:51:232
249940,0watchdog/20-21swapper/220:06:032
249940,0watchdog/20-21swapper/219:56:312
249940,0watchdog/20-21swapper/219:36:152
230889943,0cyclictest8313-21awk20:31:202
230889943,0cyclictest3383-21snmpd19:51:022
230889943,0cyclictest15580-21grep21:01:162
230869944,0cyclictest12718-21kworker/1:023:47:281
230869944,0cyclictest0-21swapper/123:42:481
230869944,0cyclictest0-21swapper/122:21:271
230869944,0cyclictest0-21swapper/122:07:251
230869943,0cyclictest7182-21awk20:06:191
230869943,0cyclictest3425-21lldpd22:37:211
230869943,0cyclictest3425-21lldpd00:31:231
230869943,0cyclictest3425-21lldpd00:26:231
230869943,0cyclictest3425-21lldpd00:16:531
230869943,0cyclictest3383-21snmpd19:11:321
230869943,0cyclictest27291-21awk20:21:171
230869943,0cyclictest24200-21awk19:06:201
230869943,0cyclictest23661-21awk21:31:241
230869943,0cyclictest13276-21sadc23:25:561
230869943,0cyclictest12718-21kworker/1:023:06:321
230869943,0cyclictest12718-21kworker/1:022:57:281
230869943,0cyclictest12718-21kworker/1:022:11:021
230869943,0cyclictest12718-21kworker/1:021:45:381
230869943,0cyclictest12718-21kworker/1:021:20:081
230869943,0cyclictest12718-21kworker/1:000:22:061
230869943,0cyclictest12718-21kworker/1:000:15:161
230869943,0cyclictest1270-21kworker/1:219:26:181
230859942,0cyclictest6158-21rm21:40:570
230859942,0cyclictest24527-21ssh00:01:070
230859942,0cyclictest22443-21cut23:11:200
230859942,0cyclictest2112-21grep00:11:140
230859942,0cyclictest14624-21sh22:46:080
230859941,0cyclictest97612chrt00:16:280
230859941,0cyclictest6547-21runrttasks19:21:070
230859941,0cyclictest52742sleep019:41:010
230859941,0cyclictest4612-21latency_hist21:15:560
230859941,0cyclictest41ktimersoftd/022:01:280
230859941,0cyclictest3488-21ssh22:11:110
230859941,0cyclictest23278-21latency_hist19:05:560
230859941,0cyclictest19652-21aten2.4-expect20:16:130
230859941,0cyclictest186482sleep019:51:130
230859941,0cyclictest17920-21latency_hist21:55:560
230859941,0cyclictest17671-21cut19:26:130
230859941,0cyclictest16620-21rm21:52:380
230859941,0cyclictest14716-21rm21:26:070
230859941,0cyclictest129522sleep020:10:590
230859941,0cyclictest12897-21sh22:41:310
230859941,0cyclictest11495-21aten_r1power_cu00:21:090
230859941,0cyclictest0-21swapper/023:55:570
230859941,0cyclictest0-21swapper/023:01:170
230859941,0cyclictest0-21swapper/022:06:120
230859941,0cyclictest0-21swapper/000:32:410
230859940,0cyclictest0-21swapper/023:40:570
230859940,0cyclictest0-21swapper/023:26:170
230859940,0cyclictest0-21swapper/022:55:560
230859940,0cyclictest0-21swapper/021:10:560
230859940,0cyclictest0-21swapper/020:30:560
230859940,0cyclictest0-21swapper/019:31:160
179940,0migration/17157-21kthreadcore19:16:201
179940,0migration/13390-21kthreadcore21:36:231
179940,0migration/126892-21sh22:03:451
179940,0migration/125280-21kthreadcore22:51:221
179940,0migration/124201-21munin-node21:56:231
179940,0migration/117438-21kthreadcore21:26:231
179940,0migration/116220-21runrttasks21:51:241
179940,0migration/115120-21kthreadcore19:21:231
179940,0migration/114996-21kthreadcore20:11:211
169940,0watchdog/132696-21nfs_client20:46:231
169940,0watchdog/125221-21perl00:01:151
169940,0watchdog/121766-21ssh22:49:191
169940,0watchdog/119601-21if_eth1.219:51:191
169940,0watchdog/11501-21munin-node20:51:271
169940,0watchdog/10-21swapper/123:56:391
169940,0watchdog/10-21swapper/121:46:111
169940,0watchdog/10-21swapper/121:14:031
339930,0migration/325689-21runrttasks20:16:373
339930,0migration/311429-21runrttasks19:42:433
28405030,0irq/26-eth0-rx-0-21swapper/320:50:493
249930,0watchdog/20-21swapper/220:25:592
230869933,0cyclictest8239-21awk20:31:201
230869933,0cyclictest12718-21kworker/1:022:41:201
230869933,0cyclictest12718-21kworker/1:022:30:581
230869933,0cyclictest12718-21kworker/1:020:41:241
230869933,0cyclictest12718-21kworker/1:020:26:181
230869933,0cyclictest12718-21kworker/1:000:06:141
230869933,0cyclictest1270-21kworker/1:219:56:201
230869933,0cyclictest1270-21kworker/1:219:46:181
230869933,0cyclictest1270-21kworker/1:219:41:141
230869933,0cyclictest1270-21kworker/1:219:30:561
230869932,0cyclictest9221-21awk20:56:221
230869932,0cyclictest31823-21kthreadcore23:36:261
230869932,0cyclictest31823-21kthreadcore23:36:261
230869932,0cyclictest19649-21aten2.4-expect20:16:131
230869932,0cyclictest14966-21aten2.4-expect21:01:131
230869932,0cyclictest12718-21kworker/1:023:26:061
35120,0ktimersoftd/34167-21kthreadcore20:51:213
35120,0ktimersoftd/325083-21hddtemp_smartct19:31:193
329920,0watchdog/329476-21cut20:21:193
28405020,0irq/26-eth0-rx-0-21swapper/320:31:153
28405020,0irq/26-eth0-rx-0-21swapper/319:37:363
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional