You are here: Home / Projects / QA Farm Realtime / Latency plots / 
2021-06-16 - 22:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

Intel(R) Core(TM) i3-6100U CPU @ 2.30GHz, Linux 4.9.47-rt37 (Profile)

Latency plot of shadow in rack #7, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack7slot7s.osadl.org (updated Wed Jun 16, 2021 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19873217661,109sleep30-21swapper/307:06:133
19968217362,105sleep10-21swapper/107:07:281
1144321050,1sleep02064399cyclictest10:52:020
19892210261,35sleep20-21swapper/207:06:282
1989528762,7sleep00-21swapper/007:06:300
56512800,2sleep20-21swapper/211:58:202
206932550,1sleep10-21swapper/111:21:591
114882290,27sleep356150irq/125-enp2s0-10:05:363
131162250,2sleep20-21swapper/211:40:022
90072220,3sleep10-21swapper/109:39:341
90072220,3sleep10-21swapper/109:39:341
119762210,3sleep12064499cyclictest12:26:191
2064499190,2cyclictest0-21swapper/112:18:481
164292180,2sleep30-21swapper/310:55:343
165362170,1sleep00-21swapper/010:55:390
112672170,2sleep30-21swapper/309:16:293
86562140,1sleep20-21swapper/211:13:272
2064699140,2cyclictest0-21swapper/212:03:292
2064399140,7cyclictest17511-21perf10:10:000
2064799139,3cyclictest56350irq/127-enp2s0-11:28:143
2064799136,6cyclictest47-21ksoftirqd/310:17:263
2064699130,12cyclictest0-21swapper/210:32:222
310062120,1sleep00-21swapper/010:19:420
2064799126,5cyclictest0-21swapper/309:42:563
2064799125,6cyclictest47-21ksoftirqd/311:19:103
2064799123,8cyclictest47-21ksoftirqd/311:06:283
2064799123,8cyclictest0-21swapper/312:27:143
2064799122,9cyclictest47-21ksoftirqd/310:36:033
20647991210,1cyclictest0-21swapper/312:22:023
2064399123,5cyclictest3-21ksoftirqd/010:33:460
2064399120,6cyclictest29224-21perf08:20:000
251832111,9sleep356250irq/126-enp2s0-11:49:053
2064799117,3cyclictest21071-21latency_hist07:15:003
2064799117,3cyclictest16550irq/16-i801_smb09:12:083
2064799116,4cyclictest16550irq/16-i801_smb07:21:473
2064799116,4cyclictest0-21swapper/309:52:493
2064799115,5cyclictest47-21ksoftirqd/311:53:553
2064799114,6cyclictest47-21ksoftirqd/310:02:263
2064799114,6cyclictest47-21ksoftirqd/309:49:063
2064799114,6cyclictest0-21swapper/312:10:313
2064799114,6cyclictest0-21swapper/311:22:193
2064799114,6cyclictest0-21swapper/310:43:003
2064799114,6cyclictest0-21swapper/310:27:543
2064799113,7cyclictest47-21ksoftirqd/312:05:013
2064799113,7cyclictest47-21ksoftirqd/310:21:303
2064799113,7cyclictest461ktimersoftd/312:36:093
2064799111,9cyclictest47-21ksoftirqd/311:39:463
2064799111,9cyclictest47-21ksoftirqd/309:23:373
2064499114,6cyclictest2608-21perf09:05:001
2064399119,1cyclictest0-21swapper/011:41:480
2064399114,6cyclictest23756-21munin-run12:35:010
2064399110,5cyclictest0-21swapper/011:38:120
192952110,10sleep156350irq/127-enp2s0-10:57:411
319242100,0sleep00-21swapper/011:06:390
271772100,0sleep027201-21ssh09:53:360
2064799107,1cyclictest461ktimersoftd/309:33:203
2064799106,3cyclictest0-21swapper/312:30:333
2064799106,3cyclictest0-21swapper/310:10:063
2064799106,3cyclictest0-21swapper/309:39:563
2064799106,3cyclictest0-21swapper/309:39:563
2064799105,4cyclictest16550irq/16-i801_smb08:01:553
2064799105,4cyclictest0-21swapper/311:40:043
2064799104,5cyclictest0-21swapper/312:15:303
2064799104,5cyclictest0-21swapper/309:28:023
2064799103,6cyclictest29206-21perf08:20:003
2064799102,7cyclictest0-21swapper/311:57:473
2064799102,7cyclictest0-21swapper/311:00:183
2064799102,7cyclictest0-21swapper/310:45:143
2064799102,7cyclictest0-21swapper/309:58:253
2064799102,6cyclictest25432-21idleruntime-cro07:50:013
2064799101,8cyclictest47-21ksoftirqd/310:32:143
2064799101,8cyclictest0-21swapper/311:33:063
2064699103,6cyclictest24431-21perf10:15:012
2064699103,1cyclictest7082-21ssh10:48:582
2064499105,4cyclictest7806-21sendmail12:00:001
2064499100,2cyclictest0-21swapper/110:16:571
2064399108,1cyclictest81rcu_preempt09:45:000
2064399108,1cyclictest3-21ksoftirqd/011:50:000
2064399108,1cyclictest3-21ksoftirqd/008:50:000
2064399108,1cyclictest3-21ksoftirqd/007:50:010
2064399106,1cyclictest3-21ksoftirqd/009:25:000
2064399101,8cyclictest0-21swapper/010:02:390
2064399101,6cyclictest27113-21kworker/0:208:02:300
2064399101,2cyclictest0-21swapper/012:23:550
2064399100,9cyclictest0-21swapper/011:14:420
2064399100,3cyclictest0-21swapper/010:41:460
2064399100,1cyclictest0-21swapper/012:07:340
7627290,1sleep10-21swapper/112:23:121
25665290,1sleep20-21swapper/212:36:182
23660290,1sleep10-21swapper/112:11:221
23346290,8sleep023371-21ssh10:14:140
206479996,2cyclictest0-21swapper/310:50:063
206479995,3cyclictest1965-21perf09:00:003
206479995,3cyclictest0-21swapper/312:05:433
206479995,3cyclictest0-21swapper/311:10:053
206479994,4cyclictest16550irq/16-i801_smb07:36:493
206479991,6cyclictest229-21systemd-journal08:44:593
206469995,3cyclictest50-21rcuop/312:16:172
206469995,3cyclictest32666-21ssh09:32:372
206469994,4cyclictest56150irq/125-enp2s0-09:20:182
206469993,3cyclictest56350irq/127-enp2s0-12:22:202
206469993,3cyclictest56350irq/127-enp2s0-11:17:242
206469993,3cyclictest56350irq/127-enp2s0-10:55:032
206469993,3cyclictest56350irq/127-enp2s0-10:04:402
206469992,2cyclictest56450irq/128-enp2s0-11:38:012
206469991,7cyclictest56350irq/127-enp2s0-09:35:382
206469991,7cyclictest56350irq/127-enp2s0-09:35:372
206469991,3cyclictest56450irq/128-enp2s0-10:50:512
206469990,8cyclictest36-21ksoftirqd/209:48:532
206469990,8cyclictest36-21ksoftirqd/209:14:052
206449997,1cyclictest0-21swapper/110:31:451
206449997,1cyclictest0-21swapper/108:21:581
206449997,1cyclictest0-21swapper/107:26:481
206449993,0cyclictest0-21swapper/111:19:421
206449993,0cyclictest0-21swapper/110:21:101
206449992,0cyclictest0-21swapper/110:45:541
206449990,6cyclictest0-21swapper/109:34:581
206449990,5cyclictest0-21swapper/112:03:111
206449990,5cyclictest0-21swapper/111:50:001
206449990,5cyclictest0-21swapper/111:43:061
206449990,5cyclictest0-21swapper/111:27:031
206449990,5cyclictest0-21swapper/110:10:041
206449990,4cyclictest39-21rcuop/211:10:021
206449990,4cyclictest39-21rcuop/209:55:371
206449990,4cyclictest39-21rcuop/209:23:041
206449990,2cyclictest0-21swapper/109:42:541
206449990,2cyclictest0-21swapper/109:15:571
206449990,2cyclictest0-21swapper/109:10:231
206439997,1cyclictest3-21ksoftirqd/012:30:000
206439997,1cyclictest3-21ksoftirqd/012:11:320
206439997,1cyclictest3-21ksoftirqd/012:05:000
206439997,1cyclictest3-21ksoftirqd/010:21:100
206439997,1cyclictest3-21ksoftirqd/009:50:010
206439997,1cyclictest3-21ksoftirqd/008:35:000
206439997,1cyclictest3-21ksoftirqd/008:15:000
206439997,1cyclictest3-21ksoftirqd/007:20:000
206439997,1cyclictest3-21ksoftirqd/007:20:000
206439995,3cyclictest3-21ksoftirqd/010:48:340
206439992,0cyclictest0-21swapper/011:57:430
206439991,6cyclictest27113-21kworker/0:208:50:500
206439990,9cyclictest0-21swapper/009:59:200
206439990,9cyclictest0-21swapper/009:37:360
206439990,9cyclictest0-21swapper/009:37:360
206439990,8cyclictest10889-21ssh10:28:310
206439990,8cyclictest0-21swapper/011:27:020
206439990,3cyclictest0-21swapper/012:35:250
206439990,3cyclictest0-21swapper/009:26:230
206439990,1cyclictest26992-21ssh11:50:110
206439990,1cyclictest20117-21ssh10:35:010
206439990,1cyclictest0-21swapper/011:23:030
206439990,1cyclictest0-21swapper/009:15:050
12404290,1sleep10-21swapper/111:39:501
23430280,1sleep20-21swapper/207:33:362
206479986,1cyclictest0-21swapper/308:50:003
206479986,1cyclictest0-21swapper/308:45:313
206479986,1cyclictest0-21swapper/308:08:203
206479984,3cyclictest28572-21munin-run08:15:003
206479984,3cyclictest16550irq/16-i801_smb08:21:583
206479983,4cyclictest56150irq/125-enp2s0-08:30:363
206479983,4cyclictest56150irq/125-enp2s0-07:56:163
206479981,6cyclictest0-21swapper/307:52:463
206479981,5cyclictest513-21rs:main3
206479980,6cyclictest3224-21latency_hist09:10:003
206479980,6cyclictest31648-21cron08:40:003
206479980,6cyclictest229-21systemd-journal07:16:463
206479980,6cyclictest229-21systemd-journal07:16:463
206479980,6cyclictest22653-21sshd07:26:323
206469987,1cyclictest0-21swapper/209:42:122
206469985,2cyclictest0-21swapper/211:30:532
206469985,2cyclictest0-21swapper/210:36:222
206469985,1cyclictest0-21swapper/207:46:322
206469984,3cyclictest32306-21munin-run08:45:002
206469984,3cyclictest29832-21munin-run08:25:002
206469983,4cyclictest56350irq/127-enp2s0-11:00:062
206469983,4cyclictest56350irq/127-enp2s0-09:15:312
206469983,4cyclictest56250irq/126-enp2s0-11:05:032
206469983,3cyclictest56350irq/127-enp2s0-12:11:242
206469983,3cyclictest56350irq/127-enp2s0-11:20:322
206469983,3cyclictest56350irq/127-enp2s0-10:21:132
206469983,2cyclictest0-21swapper/210:28:142
206469982,5cyclictest8242-21kworker/2:110:16:002
206469982,1cyclictest3226-21diskmemload12:34:022
206469981,6cyclictest0-21swapper/212:10:012
206469981,6cyclictest0-21swapper/210:40:082
206469981,6cyclictest0-21swapper/208:52:052
206469981,5cyclictest0-21swapper/212:28:272
206469980,7cyclictest56450irq/128-enp2s0-09:26:122
206469980,7cyclictest229-21systemd-journal08:29:592
206469980,6cyclictest229-21systemd-journal08:31:592
206469980,5cyclictest0-21swapper/211:51:462
206469980,5cyclictest0-21swapper/211:47:392
206469980,5cyclictest0-21swapper/211:25:552
206469980,5cyclictest0-21swapper/210:05:032
206469980,5cyclictest0-21swapper/209:55:152
206469980,5cyclictest0-21swapper/209:50:032
206449986,1cyclictest0-21swapper/107:56:541
206449984,3cyclictest5715-21cron11:34:591
206449984,3cyclictest56150irq/125-enp2s0-10:05:231
206449983,4cyclictest56150irq/125-enp2s0-07:45:321
206449983,4cyclictest229-21systemd-journal08:45:001
206449983,4cyclictest20641-21cyclictest08:50:001
206449983,4cyclictest11450irq/122-ahci[0009:10:001
206449982,2cyclictest56450irq/128-enp2s0-11:50:041
206449981,2cyclictest56350irq/127-enp2s0-11:05:221
206449981,0cyclictest0-21swapper/112:38:291
206449980,7cyclictest1337-21runrttasks07:37:181
206449980,7cyclictest0-21swapper/111:02:101
206449980,7cyclictest0-21swapper/110:40:051
206449980,7cyclictest0-21swapper/110:25:221
206449980,7cyclictest0-21swapper/107:14:461
206449980,6cyclictest23553-21latency_hist07:35:011
206449980,5cyclictest0-21swapper/110:35:181
206449980,4cyclictest0-21swapper/112:33:441
206449980,4cyclictest0-21swapper/110:02:311
206449980,4cyclictest0-21swapper/109:27:521
206449980,3cyclictest56350irq/127-enp2s0-09:45:561
206449980,2cyclictest22574-21ssh09:50:101
206449980,2cyclictest13602-21ssh10:53:371
206439986,1cyclictest3-21ksoftirqd/009:10:100
206439986,1cyclictest3-21ksoftirqd/009:05:000
206439986,1cyclictest3-21ksoftirqd/008:40:000
206439986,1cyclictest3-21ksoftirqd/008:30:000
206439986,1cyclictest3-21ksoftirqd/008:25:000
206439986,1cyclictest3-21ksoftirqd/008:10:000
206439986,1cyclictest3-21ksoftirqd/007:59:590
206439986,1cyclictest3-21ksoftirqd/007:55:000
206439984,1cyclictest0-21swapper/007:45:010
206439982,3cyclictest0-21swapper/012:16:320
206439981,4cyclictest3296-21apt-key09:09:590
206439980,8cyclictest0-21swapper/011:16:220
206439980,7cyclictest5710-21ssh11:35:000
206439980,5cyclictest0-21swapper/009:31:020
21026270,0sleep20-21swapper/207:14:422
206479975,1cyclictest0-21swapper/307:40:003
206479975,1cyclictest0-21swapper/307:30:013
206479972,4cyclictest229-21systemd-journal08:25:323
206469977,0cyclictest0-21swapper/208:47:042
206469974,2cyclictest0-21swapper/207:44:592
206469973,3cyclictest27338-21apt08:05:002
206469972,1cyclictest542-21snmpd09:07:072
206469970,6cyclictest542-21snmpd07:36:502
206469970,6cyclictest25821-21cups-browsed08:00:002
206469970,3cyclictest229-21systemd-journal09:04:322
206469970,3cyclictest229-21systemd-journal07:21:462
206449975,1cyclictest0-21swapper/107:44:121
206449975,1cyclictest0-21swapper/107:21:191
206449973,3cyclictest31377-21sshd08:36:331
206449973,3cyclictest229-21systemd-journal08:52:051
206449973,3cyclictest229-21systemd-journal08:16:571
206449973,3cyclictest229-21systemd-journal07:51:531
206449972,4cyclictest56150irq/125-enp2s0-08:03:581
206449972,2cyclictest56350irq/127-enp2s0-12:05:051
206449970,6cyclictest0-21swapper/108:05:351
206449970,6cyclictest0-21swapper/107:15:301
206449970,6cyclictest0-21swapper/107:15:301
206449970,2cyclictest0-21swapper/108:26:321
206439976,1cyclictest3-21ksoftirqd/009:00:000
206439975,1cyclictest3-21ksoftirqd/011:00:020
206439975,1cyclictest3-21ksoftirqd/007:35:010
206439975,1cyclictest3-21ksoftirqd/007:25:010
206439975,1cyclictest3-21ksoftirqd/007:10:020
206439970,4cyclictest0-21swapper/007:26:320
206469963,2cyclictest287852chrt08:15:042
206469963,2cyclictest28300-21chrt08:11:442
206469962,3cyclictest56250irq/126-enp2s0-08:55:322
206469961,4cyclictest25644-21taskset07:50:132
206469960,5cyclictest7295-1kworker/2:2H08:37:012
206469960,5cyclictest542-21snmpd07:16:462
206469960,5cyclictest542-21snmpd07:16:462
206469960,5cyclictest1337-21runrttasks08:05:062
206469960,5cyclictest0-21swapper/207:27:202
206449960,5cyclictest1337-21runrttasks08:31:041
206449960,5cyclictest0-21swapper/108:11:551
206449960,2cyclictest56150irq/125-enp2s0-08:56:351
206439961,3cyclictest0-21swapper/008:41:070
206439960,6cyclictest0-21swapper/007:36:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional