You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-06 - 18:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

ARM Broadcom BCM2711 @1500 MHz, Linux 5.10.52-rt47-v7l (Profile)

Latency plot of shadow in rack #7, slot #2
Data to construct the above plot have been generated using the RT test utility cyclictest.
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot2s.osadl.org (updated Wed Jul 06, 2022 12:43:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
525921960,2sleep10-21swapper/107:06:461
2154421550,2sleep30-21swapper/309:46:473
1105021160,3sleep20-21swapper/209:11:502
213772910,9sleep121-21ksoftirqd/109:46:451
638899900,56cyclictest6801-21apt-get12:31:423
6388998521,62cyclictest33-21ksoftirqd/307:56:413
638899781,2cyclictest0-21swapper/307:21:413
638899781,27cyclictest0-21swapper/307:51:443
177162770,3chrt17105-21/usr/sbin/munin07:46:490
638899761,2cyclictest0-21swapper/308:13:353
638899751,71cyclictest7255-21/usr/sbin/munin10:46:443
638899751,71cyclictest6395-21latency_hist07:11:303
638899751,71cyclictest0-21swapper/312:08:083
638899741,70cyclictest3533-21ntp_states08:46:503
638899741,70cyclictest26736-21taskset11:51:113
638899741,70cyclictest26736-21taskset11:51:113
638099741,9cyclictest17339-21needreboot11:18:302
638899731,68cyclictest18207-21cpuspeed09:36:423
638899730,69cyclictest16655-21chrt07:43:433
638899721,68cyclictest12730-21fschecks_count07:31:463
638899721,68cyclictest12730-21fschecks_count07:31:463
638899721,67cyclictest18545-21latency_hist07:51:303
638899711,67cyclictest1220-21ntp_kernel_err12:11:493
638899711,67cyclictest0-21swapper/308:53:243
638899711,67cyclictest0-21swapper/308:53:243
638899711,67cyclictest0-21swapper/308:26:353
638899711,66cyclictest20964-21cat09:46:303
638899711,2cyclictest0-21swapper/310:14:533
638899710,41cyclictest31410-21needreboot10:18:363
638899701,66cyclictest687-21runrttasks10:58:173
638899701,66cyclictest28001-21ntp_states10:07:203
638899701,66cyclictest0-21swapper/311:32:003
638899701,66cyclictest0-21swapper/311:16:533
638899701,66cyclictest0-21swapper/311:12:193
638899701,66cyclictest0-21swapper/310:23:173
638899701,66cyclictest0-21swapper/309:00:593
638899701,66cyclictest0-21swapper/308:36:433
638899701,66cyclictest0-21swapper/308:31:433
638899701,65cyclictest25134-21sshd11:45:063
638899701,65cyclictest25134-21sshd11:45:063
638899701,65cyclictest0-21swapper/308:21:453
638899701,65cyclictest0-21swapper/308:21:453
638899701,2cyclictest0-21swapper/311:01:433
638899701,2cyclictest0-21swapper/310:51:503
638899701,2cyclictest0-21swapper/307:11:443
179512700,2sleep117827-21/usr/sbin/munin11:21:451
638899691,65cyclictest7579-21apt-get09:01:403
638899691,65cyclictest17536-21vmstat09:31:533
638899691,65cyclictest0-21swapper/311:56:113
638899691,65cyclictest0-21swapper/308:44:113
638899691,2cyclictest0-21swapper/312:20:293
638899691,2cyclictest0-21swapper/311:21:503
638899691,2cyclictest0-21swapper/309:21:593
638899690,66cyclictest20256-21taskset11:27:003
638899690,66cyclictest1646-21awk10:26:493
638899690,66cyclictest10591-21apt-get09:11:423
636399691,33cyclictest0-21swapper/008:11:410
110899690,22rtkit-daemon0-21swapper/212:35:380
638899681,64cyclictest0-21swapper/310:06:303
638899681,64cyclictest0-21swapper/308:16:403
638899681,63cyclictest0-21swapper/307:28:093
638899681,2cyclictest0-21swapper/312:25:143
638899681,2cyclictest0-21swapper/309:18:073
638899681,2cyclictest0-21swapper/308:02:323
638899680,2cyclictest0-21swapper/311:09:493
638099681,3cyclictest0-21swapper/212:06:492
638899671,63cyclictest0-21swapper/307:37:593
638899671,2cyclictest0-21swapper/309:51:453
638899671,2cyclictest0-21swapper/309:06:433
638899671,2cyclictest0-21swapper/308:06:413
638899670,3cyclictest30025-21apt-get12:01:433
638099671,30cyclictest29082-21ntpq11:56:492
638099671,30cyclictest29082-21ntpq11:56:492
638099671,18cyclictest9292-21/usr/sbin/munin10:51:512
638099671,13cyclictest0-21swapper/210:18:162
638099670,6cyclictest7907-21wc07:16:302
636399671,20cyclictest20712-21sed07:56:500
638899661,2cyclictest0-21swapper/312:39:503
638899661,2cyclictest0-21swapper/307:17:083
638099663,59cyclictest20057-21cut09:41:482
638099661,17cyclictest0-21swapper/210:06:592
638099660,6cyclictest22768-21taskset08:04:472
636399661,44cyclictest2854-21fschecks_time10:31:470
638899651,61cyclictest0-21swapper/311:37:153
638899651,2cyclictest0-21swapper/310:36:523
638899650,4cyclictest5637-21apt-get10:41:413
638099651,3cyclictest0-21swapper/212:01:492
638899641,60cyclictest0-21swapper/312:00:333
638899641,60cyclictest0-21swapper/312:00:333
638099640,26cyclictest687-21runrttasks11:43:552
638099640,26cyclictest687-21runrttasks11:43:552
636399641,23cyclictest22224-21awk08:01:490
638899630,2cyclictest5284-21apt-get12:26:423
638099630,9cyclictest9631-21awk09:06:492
638899621,2cyclictest0-21swapper/310:31:523
638899621,2cyclictest0-21swapper/309:26:503
638099621,11cyclictest0-21swapper/209:01:462
637199621,5cyclictest22345-21kworker/1:110:36:311
636399622,10cyclictest20407-21kworker/u8:208:31:430
59152626,29sleep00-21swapper/007:07:180
638099611,15cyclictest0-21swapper/207:56:412
638899600,2cyclictest0-21swapper/310:00:223
638899600,2cyclictest0-21swapper/310:00:223
638099600,21cyclictest15536-21grep09:26:482
637199601,10cyclictest24788-21fschecks_count08:11:461
6363996023,32cyclictest3779-21apt-get12:21:410
197952600,2sleep219900-21/usr/sbin/munin11:26:492
116072590,4sleep00-21swapper/011:01:300
638099581,54cyclictest11981-21users07:26:522
638099581,4cyclictest27694-21cat08:21:392
638099581,4cyclictest27694-21cat08:21:392
6371995853,3cyclictest22527-21uptime08:01:521
6363995847,3cyclictest60650irq/70-eth011:33:040
638099572,52cyclictest0-21swapper/210:01:482
638099571,53cyclictest27047-21uptime08:16:512
637199571,52cyclictest22241-21apt-config11:36:311
6371995713,9cyclictest131rcu_preempt11:26:411
6363995752,3cyclictest21150-21/usr/sbin/munin09:46:450
6363995717,28cyclictest60550irq/69-eth008:41:410
636399571,53cyclictest12966-21irqstats07:31:480
636399571,53cyclictest12966-21irqstats07:31:480
636399570,54cyclictest16749-21df09:31:440
638099565,7cyclictest131rcu_preempt08:26:422
638099565,16cyclictest131rcu_preempt09:51:432
638099560,53cyclictest4378-21ntp_states12:21:502
638099560,53cyclictest22845-21if_eth011:37:222
638099560,53cyclictest21055-21fschecks_time11:31:452
6371995645,8cyclictest7152-21apt-get10:46:421
637199560,53cyclictest1001-21/usr/sbin/munin12:11:471
636399562,50cyclictest6348-21cyclictest09:41:410
636399561,52cyclictest14778-21/usr/sbin/munin11:12:030
636399561,3cyclictest131rcu_preempt12:16:470
636399561,28cyclictest11094-21apt07:26:400
94362550,2sleep10-21swapper/109:06:471
638099551,51cyclictest16208-21ntp_states07:41:492
638099551,39cyclictest0-21swapper/208:31:432
637199551,52cyclictest28781-21perl11:56:461
637199551,52cyclictest28781-21perl11:56:461
637199551,52cyclictest16500-21users07:41:521
637199551,51cyclictest26761-21ntp_states08:16:481
637199551,51cyclictest26635-21users10:01:551
637199551,40cyclictest2828-21ntp_states12:16:501
637199550,52cyclictest22892-21irqstats11:37:231
637199550,3cyclictest68912sleep110:46:201
636399551,51cyclictest23250-21df08:06:430
636399551,51cyclictest1990-21cron12:16:300
636399551,50cyclictest16413-21idleruntime-cro09:31:300
636399551,45cyclictest30475-21cron08:31:290
636399551,2cyclictest0-21swapper/011:42:220
636399551,2cyclictest0-21swapper/011:42:220
636399550,52cyclictest7682-21df09:01:430
636399550,52cyclictest1420-21lxpanel12:40:060
638099541,5cyclictest17845-21apt-get11:21:432
638099541,51cyclictest11151-21taskset10:58:002
638099541,50cyclictest32495-21fw_packets08:36:462
638099541,50cyclictest13869-21/usr/sbin/munin11:06:512
638099541,12cyclictest27013-21runrttasks11:51:412
638099540,51cyclictest18781-21ps09:36:492
638099540,51cyclictest15420-21ntp_states11:12:072
638099540,13cyclictest630-21apt-get12:11:402
6371995444,6cyclictest0-21swapper/112:26:431
637199541,8cyclictest494-21munin-node08:06:421
637199541,50cyclictest7812-21grep09:01:441
637199541,50cyclictest4542-21df_inode08:51:431
637199541,50cyclictest4542-21df_inode08:51:431
637199541,50cyclictest31119-21grep08:31:491
637199541,50cyclictest14701-21apt-config11:11:311
637199540,50cyclictest2018-21ntp_states08:41:491
6363995433,3cyclictest0-21swapper/009:22:090
636399542,47cyclictest0-21swapper/010:16:420
636399542,32cyclictest0-21swapper/010:01:450
636399541,7cyclictest29638-21timerandwakeup10:11:510
636399541,50cyclictest7335-21wc07:11:530
636399541,50cyclictest4134-21chrt10:36:430
636399541,50cyclictest12941-21awk11:06:300
636399541,50cyclictest12098-21chrt09:16:420
636399541,44cyclictest9381-21sed07:21:300
636399541,34cyclictest32390-21fschecks_count08:36:450
638099532,6cyclictest25414-21find08:11:522
638099531,9cyclictest0-21swapper/212:16:432
638099531,6cyclictest0-21swapper/207:36:432
638099531,49cyclictest32518-21ntp_states10:21:512
638099531,40cyclictest687-21runrttasks12:38:042
638099531,37cyclictest0-21swapper/209:16:412
6371995344,7cyclictest12605-21apt-get07:31:431
6371995344,7cyclictest12605-21apt-get07:31:431
637199532,5cyclictest2930-21apt-get08:46:421
637199531,50cyclictest14208-21df07:36:431
637199531,49cyclictest11530-21taskset09:12:271
637199531,44cyclictest21407-21idleruntime-cro08:01:301
637199531,40cyclictest5449-21latency_hist10:41:301
637199531,34cyclictest14451-21apt-get11:11:311
637199531,13cyclictest8822-21grep10:51:451
637199530,51cyclictest30591-21ntp_kernel_pll_12:01:501
637199530,50cyclictest7348-21ls09:01:301
637199530,44cyclictest23324-21ntp_states09:51:501
636399532,48cyclictest9754-21runrttasks10:54:250
636399531,49cyclictest32284-21awk12:06:510
636399531,49cyclictest23784-21cat11:41:300
636399531,49cyclictest15617-21apt-get07:41:400
636399531,49cyclictest14272-21fschecks_count07:36:460
636399531,48cyclictest700-21snmpd11:06:430
636399531,30cyclictest0-21swapper/009:36:420
636399530,50cyclictest3045-21diskstats08:46:440
636399530,49cyclictest19332-21/usr/sbin/munin11:26:510
110899530,5rtkit-daemon1107-21rtkit-daemon12:26:271
6380995247,3cyclictest1783-21awk08:41:472
6380995233,17cyclictest687-21runrttasks08:11:292
6380995221,28cyclictest0-21swapper/207:31:442
6380995221,28cyclictest0-21swapper/207:31:442
638099521,29cyclictest0-21swapper/210:31:422
638099521,17cyclictest6736-21latency_hist12:31:312
638099521,17cyclictest0-21swapper/209:47:002
638099520,49cyclictest6001-21irqstats10:41:472
638099520,49cyclictest19312-21/usr/sbin/munin07:51:522
638099520,34cyclictest14216-21awk09:22:042
6371995246,4cyclictest406-21open_files08:36:501
637199521,7cyclictest7206-21wc12:31:481
637199521,6cyclictest12389-21latency_hist07:31:301
637199521,49cyclictest28389-21/usr/sbin/munin08:21:511
637199521,49cyclictest28389-21/usr/sbin/munin08:21:511
637199521,42cyclictest25497-21apt-get11:46:431
637199521,42cyclictest25497-21apt-get11:46:431
637199521,33cyclictest8437-21uname07:17:011
637199521,30cyclictest0-21swapper/110:56:411
637199521,27cyclictest17237-21/usr/sbin/munin07:46:441
637199520,49cyclictest18149-21apt-get09:36:401
637199520,48cyclictest27767-21/usr/sbin/munin10:07:151
636399521,48cyclictest9065-21/usr/sbin/munin09:06:460
636399521,48cyclictest700-21snmpd08:52:010
636399521,48cyclictest700-21snmpd08:52:010
636399521,48cyclictest60550irq/69-eth009:56:480
636399521,48cyclictest60550irq/69-eth009:56:480
636399521,47cyclictest1420-21lxpanel11:49:220
636399521,47cyclictest1420-21lxpanel11:49:220
638099511,8cyclictest11909-21fschecks_time11:01:472
638099511,47cyclictest24245-21taskset09:56:462
638099511,47cyclictest24245-21taskset09:56:462
638099511,31cyclictest6494-21latency_hist07:11:302
638099511,25cyclictest18500-21sed07:51:302
638099511,15cyclictest9129-21runrttasks07:18:492
638099510,49cyclictest26170-21ps11:46:522
638099510,49cyclictest26170-21ps11:46:522
638099510,48cyclictest7749-21ntp_states10:46:502
638099510,47cyclictest4436-21apt-get08:51:412
638099510,47cyclictest4436-21apt-get08:51:412
637199511,47cyclictest30946-21/usr/sbin/munin10:16:491
637199511,47cyclictest12093-21apt-get09:16:421
637199511,42cyclictest0-21swapper/110:11:431
637199511,41cyclictest6645-21df07:11:441
637199511,36cyclictest19659-21apt-get09:41:401
637199511,36cyclictest17798-21cat11:21:311
637199511,30cyclictest11674-21apt-get11:01:421
637199511,27cyclictest687-21runrttasks09:27:281
637199511,26cyclictest25517-21sort10:01:301
637199511,26cyclictest25517-21sort10:01:301
637199511,16cyclictest816-21cat10:26:301
637199511,15cyclictest0-21swapper/107:56:311
637199510,37cyclictest31926-21/usr/sbin/munin12:06:471
636399512,19cyclictest407-21avahi-daemon08:21:340
636399512,19cyclictest407-21avahi-daemon08:21:340
636399511,47cyclictest18692-21sort07:51:440
636399511,43cyclictest700-21snmpd11:56:430
636399511,43cyclictest700-21snmpd11:56:430
636399510,39cyclictest7598-21cat10:46:490
638099501,30cyclictest0-21swapper/208:46:422
638099500,48cyclictest6039-21/usr/sbin/munin08:57:242
637199501,47cyclictest17246-21ntp_states09:31:501
636399501,45cyclictest16307-21chrt11:16:410
636399501,45cyclictest0-21swapper/010:41:420
636399501,37cyclictest700-21snmpd09:14:560
636399501,36cyclictest9722-21fschecks_count07:21:440
636399501,36cyclictest407-21avahi-daemon09:55:180
636399501,35cyclictest27117-21taskset08:17:320
636399501,15cyclictest758-21Xorg12:02:290
638099491,15cyclictest0-21swapper/209:31:422
638099490,46cyclictest4247-21entropy10:36:452
637199491,39cyclictest0-21swapper/109:21:541
637199491,35cyclictest0-21swapper/111:51:431
637199490,47cyclictest1433-21kernelversion10:26:471
636399491,46cyclictest17936-21expr11:21:440
636399491,44cyclictest700-21snmpd10:07:160
636399491,29cyclictest6662-21awk08:57:240
636399491,22cyclictest0-21swapper/011:51:420
636399490,45cyclictest499-21seq10:23:130
638099481,29cyclictest6773-21taskset12:31:332
637199481,45cyclictest0-21swapper/112:36:421
636399480,45cyclictest6460-21seq12:30:180
638099474,40cyclictest1988-21timerwakeupswit10:26:522
638099471,29cyclictest10437-21/usr/sbin/munin07:21:532
637199471,6cyclictest25436-21latency_hist11:46:331
637199471,6cyclictest25436-21latency_hist11:46:331
637199471,23cyclictest29217-21apt-get08:26:431
637199471,11cyclictest10351-21timerandwakeup07:21:521
638099460,40cyclictest29907-21taskset10:13:402
22692461,5sleep00-21swapper/010:30:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional