You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-05 - 01:30

ARM Broadcom BCM2711 @1500 MHz, Linux 5.10.52-rt47-v7l (Profile)

Latency plot of system in rack #4, slot #1
Data to construct the above plot have been generated using the RT test utility cyclictest.
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack4slot1.osadl.org (updated Sun Dec 04, 2022 12:44:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3082321270,2sleep1201rcuc/111:34:131
3082321270,2sleep1201rcuc/111:34:131
2434021110,2chrt0-21swapper/312:35:163
2434021110,2chrt0-21swapper/312:35:163
846721020,3chrt0-21swapper/309:28:383
846721020,3chrt0-21swapper/309:28:383
314512960,4sleep30-21swapper/311:01:073
314512960,4sleep30-21swapper/311:01:073
314512960,4sleep30-21swapper/311:01:073
208462960,4sleep30-21swapper/311:56:383
208462960,4sleep30-21swapper/311:56:383
86372910,3chrt0-21swapper/110:02:331
86372910,3chrt0-21swapper/110:02:331
160312890,3sleep216023-21ssh09:36:132
117372890,2chrt0-21swapper/307:11:023
81632880,3sleep10-21swapper/109:28:121
81632880,3sleep10-21swapper/109:28:121
110612860,3sleep20-21swapper/211:13:082
110612860,3sleep20-21swapper/211:13:082
299912780,3sleep30-21swapper/309:51:093
299912780,3sleep30-21swapper/309:51:093
243902750,3chrt0-21swapper/111:26:411
243902750,3chrt0-21swapper/111:26:411
189612740,3sleep30-21swapper/309:40:003
1145899673,56cyclictest23671-21ssh09:10:592
1145899673,56cyclictest23671-21ssh09:10:592
11457996710,22cyclictest21-21ksoftirqd/112:05:591
11457996710,22cyclictest21-21ksoftirqd/112:05:591
294272660,4chrt0-21swapper/110:25:031
294272660,4chrt0-21swapper/110:25:031
242692640,3sleep20-21swapper/212:35:112
242692640,3sleep20-21swapper/212:35:112
1145699638,41cyclictest25035-21ssh12:36:010
1145699638,41cyclictest25035-21ssh12:36:010
11456996310,42cyclictest1687-21ssh09:56:020
11456996310,42cyclictest1687-21ssh09:56:020
47282600,3sleep10-21swapper/111:06:151
47282600,3sleep10-21swapper/111:06:151
1145899596,40cyclictest1678-21ssh09:56:012
1145899596,40cyclictest1678-21ssh09:56:012
91242580,2sleep29213-21bingrep08:31:072
91242580,2sleep29213-21bingrep08:31:072
1145799581,44cyclictest13257-21ssh11:16:041
1145799581,44cyclictest13257-21ssh11:16:041
250522570,3sleep10-21swapper/110:54:131
250522570,3sleep10-21swapper/110:54:131
228792550,3sleep30-21swapper/309:43:463
228792550,3sleep30-21swapper/309:43:463
1145799552,45cyclictest7374-21ssh10:35:591
1145799552,45cyclictest7374-21ssh10:35:591
288682530,5sleep333-21ksoftirqd/312:06:023
288682530,5sleep333-21ksoftirqd/312:06:023
187572530,2sleep00-21swapper/010:13:250
77162520,3sleep10-21swapper/111:43:331
77162520,3sleep10-21swapper/111:43:331
1145799522,42cyclictest15406-21ssh12:26:011
1145799522,42cyclictest15406-21ssh12:26:011
1145699525,9cyclictest131rcu_preempt12:01:010
1145699525,9cyclictest131rcu_preempt12:01:010
99332510,7sleep333-21ksoftirqd/311:46:043
99332510,7sleep333-21ksoftirqd/311:46:043
16652510,3sleep11280-21/usr/sbin/munin08:11:041
16652510,3sleep11280-21/usr/sbin/munin08:11:041
1145799513,43cyclictest10299-21ssh09:31:031
11457995123,21cyclictest21-21ksoftirqd/108:36:011
11457995123,21cyclictest21-21ksoftirqd/108:36:011
39552500,2sleep00-21swapper/009:24:010
39552500,2sleep00-21swapper/009:24:010
1145699502,14cyclictest14394-21ssh11:51:050
1145799492,40cyclictest1073-21ssh12:10:581
1145799492,40cyclictest1073-21ssh12:10:581
279702480,3sleep30-21swapper/312:05:103
279702480,3sleep30-21swapper/312:05:103
126632480,2sleep20-21swapper/210:06:322
126632480,2sleep20-21swapper/210:06:322
67202470,2sleep20-21swapper/210:35:212
67202470,2sleep20-21swapper/210:35:212
222312460,3chrt0-21swapper/111:25:321
222312460,3chrt0-21swapper/111:25:321
216752460,2sleep00-21swapper/012:31:310
216752460,2sleep00-21swapper/012:31:310
1145699461,33cyclictest3754-21ssh11:06:010
1145699461,33cyclictest3754-21ssh11:06:010
317442450,4sleep00-21swapper/009:52:550
317442450,4sleep00-21swapper/009:52:550
206392450,8chrt27-21ksoftirqd/207:35:482
206392450,8chrt27-21ksoftirqd/207:35:482
206392450,8chrt27-21ksoftirqd/207:35:482
166942450,3sleep20-21swapper/210:45:502
166942450,3sleep20-21swapper/210:45:502
282452440,3sleep10-21swapper/107:56:011
282452440,3sleep10-21swapper/107:56:011
262292440,2sleep00-21swapper/010:55:490
262292440,2sleep00-21swapper/010:55:490
177452440,1sleep2261rcuc/211:54:112
140012440,3sleep10-21swapper/108:45:481
140012440,3sleep10-21swapper/108:45:481
11456994319,17cyclictest12-21ksoftirqd/012:06:010
11456994319,17cyclictest12-21ksoftirqd/012:06:010
65982420,3sleep20-21swapper/212:16:142
11458994215,18cyclictest27-21ksoftirqd/210:26:022
11458994215,18cyclictest27-21ksoftirqd/210:26:022
316912410,3chrt0-21swapper/310:26:263
316912410,3chrt0-21swapper/310:26:263
150672410,4sleep215046-21latency_hist12:25:472
150672410,4sleep215046-21latency_hist12:25:472
150672410,4sleep215046-21latency_hist12:25:472
98062400,2sleep20-21swapper/211:46:032
98062400,2sleep20-21swapper/211:46:032
11458994015,20cyclictest27-21ksoftirqd/209:05:572
11458994015,20cyclictest27-21ksoftirqd/209:05:572
11458994011,19cyclictest27-21ksoftirqd/207:15:592
11458994011,19cyclictest27-21ksoftirqd/207:15:592
1145799406,27cyclictest3635-21apt-get11:05:571
1145799406,27cyclictest3635-21apt-get11:05:571
1145799406,27cyclictest3635-21apt-get11:05:571
11456994023,7cyclictest12-21ksoftirqd/010:01:050
11456994023,7cyclictest12-21ksoftirqd/010:01:050
11456994019,10cyclictest37650irq/56-eth012:21:010
11456994019,10cyclictest37650irq/56-eth012:21:010
11456994019,10cyclictest37650irq/56-eth012:21:010
229812390,2sleep223061-21irqstats07:41:042
229812390,2sleep223061-21irqstats07:41:042
150192390,2sleep30-21swapper/308:46:103
150192390,2sleep30-21swapper/308:46:103
150192390,2sleep30-21swapper/308:46:103
149952390,3chrt0-21swapper/009:35:590
148322390,4sleep30-21swapper/311:51:113
11458993916,17cyclictest27-21ksoftirqd/209:31:022
1145699398,18cyclictest37650irq/56-eth011:46:010
1145699398,18cyclictest37650irq/56-eth011:46:010
1145699394,20cyclictest37650irq/56-eth011:21:020
1145699394,20cyclictest37650irq/56-eth011:21:020
294012380,2sleep129415-21ssh12:40:461
294012380,2sleep129415-21ssh12:40:461
196852380,3chrt0-21swapper/211:21:442
196852380,3chrt0-21swapper/211:21:442
155062380,3sleep30-21swapper/310:10:393
155062380,3sleep30-21swapper/310:10:393
136042380,3sleep10-21swapper/111:50:121
136042380,3sleep10-21swapper/111:50:121
1145899382,27cyclictest29335-21ssh09:51:012
1145899382,27cyclictest29335-21ssh09:51:012
103742380,3sleep20-21swapper/210:38:432
103742380,3sleep20-21swapper/210:38:432
320492370,3chrt0-21swapper/109:19:401
197532370,3chrt0-21swapper/009:00:500
197532370,3chrt0-21swapper/009:00:500
174932370,3sleep00-21swapper/011:20:330
174932370,3sleep00-21swapper/011:20:330
127722370,3sleep30-21swapper/312:22:303
127722370,3sleep30-21swapper/312:22:303
127722370,3sleep30-21swapper/312:22:303
1145699373,22cyclictest37650irq/56-eth011:11:010
1145699373,22cyclictest37650irq/56-eth011:11:010
155882360,2sleep10-21swapper/107:21:061
155882360,2sleep10-21swapper/107:21:061
143932360,4chrt0-21swapper/111:51:051
11456993620,10cyclictest37650irq/56-eth011:25:590
11456993620,10cyclictest37650irq/56-eth011:25:590
94502355,23sleep30-21swapper/307:05:583
243732350,3sleep10-21swapper/109:11:081
243732350,3sleep10-21swapper/109:11:081
216182350,3sleep00-21swapper/011:57:470
216182350,3sleep00-21swapper/011:57:470
1145899359,19cyclictest27-21ksoftirqd/207:35:592
1145899359,19cyclictest27-21ksoftirqd/207:35:592
1145899352,28cyclictest0-21swapper/211:01:042
1145899352,28cyclictest0-21swapper/211:01:042
1145899352,28cyclictest0-21swapper/211:01:042
1145799358,18cyclictest21-21ksoftirqd/109:51:001
1145799358,18cyclictest21-21ksoftirqd/109:51:001
11457993523,7cyclictest13348-21latency_hist07:15:551
1145699354,26cyclictest26646-21ssh10:56:080
1145699354,26cyclictest26646-21ssh10:56:080
1145699351,26cyclictest16123-21sed08:50:550
1145699351,26cyclictest16123-21sed08:50:550
1145699351,26cyclictest16123-21sed08:50:550
30872340,3sleep30-21swapper/310:31:123
30872340,3sleep30-21swapper/310:31:123
1145699346,22cyclictest28750irq/49-vc407:33:040
1145699346,22cyclictest28750irq/49-vc407:33:040
1145699342,25cyclictest0-21swapper/008:11:000
1145699342,25cyclictest0-21swapper/008:11:000
110192346,19sleep10-21swapper/107:10:411
96932339,14sleep00-21swapper/007:06:040
63762330,2sleep10-21swapper/110:34:521
63762330,2sleep10-21swapper/110:34:521
237732330,3chrt0-21swapper/112:00:501
237732330,3chrt0-21swapper/112:00:501
140742330,4sleep00-21swapper/010:08:350
140742330,4sleep00-21swapper/010:08:350
1145899336,16cyclictest131rcu_preempt10:11:062
1145899333,19cyclictest21733-21ssh10:51:052
1145899333,19cyclictest21733-21ssh10:51:052
1145899332,25cyclictest13272-21ssh11:16:062
1145899332,25cyclictest13272-21ssh11:16:062
1145699332,19cyclictest37650irq/56-eth009:15:590
72792320,2sleep00-21swapper/009:26:550
72792320,2sleep00-21swapper/009:26:550
12132320,3sleep00-21swapper/012:11:010
12132320,3sleep00-21swapper/012:11:010
1145899321,21cyclictest0-21swapper/210:16:002
1145699322,4cyclictest0-21swapper/007:26:000
1145699322,4cyclictest0-21swapper/007:26:000
1145699321,3cyclictest0-21swapper/008:31:020
1145699321,3cyclictest0-21swapper/008:31:020
106672325,20sleep20-21swapper/207:06:262
289552310,2sleep20-21swapper/209:50:442
289552310,2sleep20-21swapper/209:50:442
1145899312,22cyclictest23309-21diskmemload12:26:012
1145899312,22cyclictest23309-21diskmemload12:26:012
1145899311,22cyclictest4706-21date11:40:562
1145899311,22cyclictest4706-21date11:40:562
11457993115,4cyclictest131rcu_preempt07:40:591
11457993115,4cyclictest131rcu_preempt07:40:591
1145699316,21cyclictest37650irq/56-eth010:35:580
1145699316,21cyclictest37650irq/56-eth010:35:580
1145699315,17cyclictest37650irq/56-eth009:41:000
1145699315,17cyclictest37650irq/56-eth009:41:000
11456993120,4cyclictest0-21swapper/010:16:000
1145699311,3cyclictest0-21swapper/009:46:000
1145699311,3cyclictest0-21swapper/009:46:000
1145699311,25cyclictest27539-21ssh11:31:040
1145699311,25cyclictest27539-21ssh11:31:040
40072300,3sleep10-21swapper/109:24:061
40072300,3sleep10-21swapper/109:24:061
11457992912,4cyclictest131rcu_preempt10:06:021
11457992912,4cyclictest131rcu_preempt10:06:021
1145699294,15cyclictest37650irq/56-eth010:26:020
1145699294,15cyclictest37650irq/56-eth010:26:020
1145799282,23cyclictest19867-21ssh09:40:591
1145799282,23cyclictest19867-21ssh09:40:591
11457992815,10cyclictest25374-21/usr/sbin/munin09:46:111
11457992815,10cyclictest25374-21/usr/sbin/munin09:46:111
11457992813,11cyclictest14996-21ssh09:35:591
247412270,3sleep30-21swapper/310:20:113
215632270,4sleep2261rcuc/209:05:512
215632270,4sleep2261rcuc/209:05:512
1145899273,17cyclictest1092-21sh12:11:002
1145899273,17cyclictest1092-21sh12:11:002
1145899271,21cyclictest26633-21sh10:56:072
1145899271,21cyclictest26633-21sh10:56:072
1145799277,3cyclictest131rcu_preempt10:56:071
1145799277,3cyclictest131rcu_preempt10:56:071
1145799273,21cyclictest8497-21ssh11:11:001
1145799273,21cyclictest8497-21ssh11:11:001
1145699272,17cyclictest2701-21ssh10:31:080
1145699272,17cyclictest2701-21ssh10:31:080
11456992717,5cyclictest0-21swapper/007:55:590
11456992717,5cyclictest0-21swapper/007:55:590
1146099261,19cyclictest0-21swapper/310:55:563
1146099261,19cyclictest0-21swapper/310:55:563
1145899261,19cyclictest4821-21ssh11:41:012
1145899261,19cyclictest4821-21ssh11:41:012
1145899261,19cyclictest3762-21ssh11:06:022
1145899261,19cyclictest3762-21ssh11:06:022
1145799265,13cyclictest0-21swapper/109:56:011
1145799265,13cyclictest0-21swapper/109:56:011
1145799263,19cyclictest16025-21ssh10:11:081
1145799263,19cyclictest10629-21ssh12:21:011
1145799263,19cyclictest10629-21ssh12:21:011
1145799263,19cyclictest10629-21ssh12:21:011
1146099251,16cyclictest0-21swapper/311:21:033
1146099251,16cyclictest0-21swapper/311:21:033
1145799253,17cyclictest16935-21ssh10:46:011
11457992515,6cyclictest0-21swapper/108:50:531
11457992515,6cyclictest0-21swapper/108:50:531
11457992515,6cyclictest0-21swapper/108:50:531
1145699257,10cyclictest12-21ksoftirqd/007:45:580
1145699257,10cyclictest12-21ksoftirqd/007:45:580
1145699256,11cyclictest151rcuc/011:41:000
1145699256,11cyclictest151rcuc/011:41:000
1145699252,10cyclictest12-21ksoftirqd/010:46:010
11456992517,4cyclictest0-21swapper/008:26:000
11456992517,4cyclictest0-21swapper/008:26:000
11458992411,6cyclictest0-21swapper/210:47:422
1145799243,17cyclictest24033-21ssh12:01:011
1145799243,17cyclictest24033-21ssh12:01:011
11456992415,6cyclictest12-21ksoftirqd/011:05:560
11456992415,6cyclictest12-21ksoftirqd/011:05:560
11456992415,6cyclictest12-21ksoftirqd/011:05:560
11456992413,9cyclictest11738-21/usr/sbin/munin07:11:020
11456992413,7cyclictest0-21swapper/007:41:000
11456992413,7cyclictest0-21swapper/007:41:000
11456992413,4cyclictest0-21swapper/010:21:000
11456992413,4cyclictest0-21swapper/010:21:000
1145699241,20cyclictest12608-21cut08:41:030
1145699241,20cyclictest12608-21cut08:41:030
324612230,3sleep30-21swapper/311:36:043
324612230,3sleep30-21swapper/311:36:043
1145899238,11cyclictest27206-21cat07:51:142
1145899238,11cyclictest27206-21cat07:51:142
1145899231,8cyclictest6586-21timerandwakeup09:26:182
1145899231,8cyclictest6586-21timerandwakeup09:26:182
11458992316,5cyclictest18679-21ntp_states08:56:082
11458992316,5cyclictest18679-21ntp_states08:56:082
1145799238,4cyclictest21-21ksoftirqd/111:36:121
1145799238,4cyclictest21-21ksoftirqd/111:36:121
1145799232,17cyclictest20277-21ssh12:31:061
1145799232,17cyclictest20277-21ssh12:31:061
11457992314,5cyclictest21-21ksoftirqd/107:15:581
11457992314,5cyclictest21-21ksoftirqd/107:15:581
1145799231,18cyclictest8683-21chrt08:31:011
1145799231,18cyclictest8683-21chrt08:31:011
1145699236,11cyclictest0-21swapper/008:01:010
1145699236,11cyclictest0-21swapper/008:01:010
193662220,3sleep30-21swapper/307:31:073
193662220,3sleep30-21swapper/307:31:073
193662220,3sleep30-21swapper/307:31:073
1146099222,15cyclictest0-21swapper/311:35:563
1146099222,15cyclictest0-21swapper/311:35:563
1145799228,10cyclictest21958-21grep07:36:121
1145799228,10cyclictest21958-21grep07:36:121
11457992213,7cyclictest30292-21seq10:26:031
11457992213,7cyclictest30292-21seq10:26:031
11457992213,5cyclictest0-21swapper/108:21:031
11457992213,5cyclictest0-21swapper/108:21:031
1145799221,18cyclictest3415-21ls08:16:031
1145799221,18cyclictest3415-21ls08:16:031
11456992215,5cyclictest23437-21ssh09:10:510
11456992215,5cyclictest23437-21ssh09:10:510
1145699221,17cyclictest15599-21ls07:21:050
1145699221,17cyclictest15599-21ls07:21:050
1145699221,16cyclictest638-21snmpd07:51:030
1145699221,16cyclictest638-21snmpd07:51:030
58042217,3sleep012-21ksoftirqd/012:16:010
1146099212,11cyclictest0-21swapper/310:46:013
11458992115,3cyclictest27-21ksoftirqd/210:22:562
11458992115,3cyclictest27-21ksoftirqd/210:22:562
11458992113,5cyclictest27-21ksoftirqd/211:56:582
11458992113,5cyclictest27-21ksoftirqd/211:56:582
11458992112,6cyclictest0-21swapper/209:41:072
11458992112,6cyclictest0-21swapper/209:41:072
11458992111,7cyclictest13311-21sort07:15:532
11458992110,5cyclictest3720-21rm09:23:412
11458992110,5cyclictest3720-21rm09:23:412
1145799214,3cyclictest131rcu_preempt12:20:491
11457992116,3cyclictest21-21ksoftirqd/110:43:011
11457992116,3cyclictest21-21ksoftirqd/110:43:011
11457992115,3cyclictest0-21swapper/108:26:131
11457992115,3cyclictest0-21swapper/108:26:131
1145799211,17cyclictest19007-21sshd07:31:031
1145799211,17cyclictest19007-21sshd07:31:031
1145799211,17cyclictest19007-21sshd07:31:031
1145799211,17cyclictest16330-21cut08:51:031
1145799211,17cyclictest16330-21cut08:51:031
1145699212,17cyclictest15700-21fschecks_time12:26:050
1145699212,17cyclictest15700-21fschecks_time12:26:050
1145699211,14cyclictest13119-21ssh10:41:120
1145699211,14cyclictest13119-21ssh10:41:120
1145699211,10cyclictest131-21systemd-journal08:20:470
1145699211,10cyclictest131-21systemd-journal08:20:470
1146099203,11cyclictest0-21swapper/311:26:003
1146099203,11cyclictest0-21swapper/311:26:003
1146099203,11cyclictest0-21swapper/308:31:023
1146099203,11cyclictest0-21swapper/308:31:023
1146099202,9cyclictest0-21swapper/311:11:013
1146099202,9cyclictest0-21swapper/311:11:013
1146099202,11cyclictest0-21swapper/310:36:003
1146099202,11cyclictest0-21swapper/310:36:003
1146099202,11cyclictest0-21swapper/308:11:013
1146099202,11cyclictest0-21swapper/308:11:013
1146099201,11cyclictest0-21swapper/309:15:593
1145899209,9cyclictest6489-21cpu10:01:072
1145899209,9cyclictest6489-21cpu10:01:072
1145899207,10cyclictest27-21ksoftirqd/211:25:592
1145899207,10cyclictest27-21ksoftirqd/211:25:592
1145899204,8cyclictest0-21swapper/208:45:592
1145899204,8cyclictest0-21swapper/208:45:592
1145899204,8cyclictest0-21swapper/208:45:592
11458992011,6cyclictest0-21swapper/211:31:052
11458992011,6cyclictest0-21swapper/211:31:052
1145899201,14cyclictest25045-21sh12:36:022
1145899201,14cyclictest25045-21sh12:36:022
1145899201,12cyclictest28753-21ssh12:06:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional