You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-06-04 - 05:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Sun Jun 04, 2023 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
242942178104,27sleep20-21swapper/219:08:562
240872168108,29sleep30-21swapper/319:06:173
242182161104,28sleep00-21swapper/019:07:580
243082160103,29sleep10-21swapper/119:09:071
324462620,1sleep00-21swapper/020:40:270
31572620,3sleep0131rcuc/023:00:310
116742590,10sleep334-21ksoftirqd/319:55:153
52352570,2sleep20-21swapper/221:33:182
319362550,2sleep00-21swapper/022:57:210
138462550,2sleep00-21swapper/022:10:330
48822530,1sleep10-21swapper/100:35:111
117102510,2sleep20-21swapper/200:08:552
39302500,2chrt0-21swapper/322:31:113
49842470,1sleep20-21swapper/222:32:382
141712460,3sleep122-21ksoftirqd/123:40:241
286452430,1sleep30-21swapper/321:55:163
198282410,2chrt0-21swapper/223:45:232
101912400,1sleep20-21swapper/221:37:402
283902380,2sleep20-21swapper/222:25:012
191192340,2sleep20-21swapper/222:45:312
109812330,2chrt0-21swapper/300:07:493
2465099310,7cyclictest22-21ksoftirqd/122:10:001
2465299281,25cyclictest0-21swapper/319:31:113
2465199287,3cyclictest141rcu_preempt21:30:002
2465199287,3cyclictest141rcu_preempt21:30:002
2465099282,13cyclictest22-21ksoftirqd/122:16:021
2465099281,25cyclictest0-21swapper/119:36:141
2465099281,25cyclictest0-21swapper/119:22:001
2464999281,25cyclictest0-21swapper/019:23:490
2465299271,24cyclictest0-21swapper/319:27:083
2465299271,24cyclictest0-21swapper/319:23:363
2465199271,2cyclictest0-21swapper/219:31:112
2465199270,25cyclictest24647-21cyclictest19:29:342
2465099278,4cyclictest22-21ksoftirqd/123:52:271
2465099278,3cyclictest141rcu_preempt21:15:261
2465099276,9cyclictest22-21ksoftirqd/122:27:191
2465099271,24cyclictest0-21swapper/119:57:521
2465099271,24cyclictest0-21swapper/119:30:541
2464999271,2cyclictest0-21swapper/019:26:170
171442270,2sleep00-21swapper/000:14:230
130682270,1sleep20-21swapper/223:10:042
130682270,1sleep20-21swapper/223:10:032
31682260,2sleep23174-21cpuspeed_turbos19:35:202
24651992623,2cyclictest0-21swapper/219:22:422
2465099267,10cyclictest22-21ksoftirqd/123:20:261
2465099263,16cyclictest23516-21ssh22:50:221
24650992620,4cyclictest22-21ksoftirqd/123:20:011
2465099261,1cyclictest0-21swapper/119:26:531
24650992611,4cyclictest22-21ksoftirqd/121:55:311
24650992610,4cyclictest22-21ksoftirqd/121:14:261
319512250,1sleep20-21swapper/223:26:212
24651992522,2cyclictest0-21swapper/219:57:432
2465099259,5cyclictest22-21ksoftirqd/122:03:471
2465099251,7cyclictest22-21ksoftirqd/123:35:211
24650992513,4cyclictest22-21ksoftirqd/123:55:291
24650992513,4cyclictest22-21ksoftirqd/123:55:291
24650992511,3cyclictest22-21ksoftirqd/122:43:411
2465099250,7cyclictest22-21ksoftirqd/121:40:381
24649992523,1cyclictest0-21swapper/019:37:480
24649992523,1cyclictest0-21swapper/019:30:450
133522250,6sleep0141rcu_preempt23:40:080
309612240,1sleep10-21swapper/123:25:321
272372240,2sleep10-21swapper/100:24:191
256882240,9sleep334-21ksoftirqd/322:51:183
24652992421,2cyclictest0-21swapper/319:38:043
24652992413,9cyclictest641-21runrttasks21:43:283
2465099249,4cyclictest22-21ksoftirqd/123:06:061
2465099249,4cyclictest22-21ksoftirqd/121:39:211
2465099249,4cyclictest22-21ksoftirqd/100:06:311
2465099248,4cyclictest22-21ksoftirqd/123:46:161
2465099248,4cyclictest22-21ksoftirqd/122:31:061
2465099248,4cyclictest22-21ksoftirqd/100:34:221
24650992411,4cyclictest22-21ksoftirqd/120:35:151
2465099241,10cyclictest22-21ksoftirqd/100:00:011
2464999241,10cyclictest89042grep22:35:390
121442240,5sleep00-21swapper/021:10:170
85242231,17sleep08584-21/usr/sbin/munin23:05:290
39752230,1sleep20-21swapper/223:30:342
24652992316,5cyclictest30497-21awk22:25:353
2465099239,8cyclictest22-21ksoftirqd/122:20:311
2465099237,4cyclictest22-21ksoftirqd/123:30:501
2465099237,4cyclictest22-21ksoftirqd/121:45:571
2465099237,4cyclictest22-21ksoftirqd/121:30:411
2465099237,2cyclictest141rcu_preempt21:23:251
2465099234,8cyclictest22-21ksoftirqd/119:18:201
2465099234,2cyclictest141rcu_preempt22:45:351
2465099234,10cyclictest22-21ksoftirqd/100:19:011
2465099233,6cyclictest22-21ksoftirqd/122:56:541
2465099232,15cyclictest2071-21ssh23:00:181
2465099231,9cyclictest22-21ksoftirqd/121:25:341
2465099231,9cyclictest22-21ksoftirqd/121:25:331
24650992315,5cyclictest11696-21grep21:09:591
24650992310,2cyclictest141rcu_preempt00:10:311
2464999239,11cyclictest0-21swapper/022:25:320
2464999231,7cyclictest364-21ssh21:58:500
2465299221,9cyclictest321952rm23:56:263
2465299221,9cyclictest321952rm23:56:253
24652992214,6cyclictest19332-21python300:15:343
2465299221,19cyclictest11689-21diskmemload23:02:303
2465199221,5cyclictest28-21ksoftirqd/200:11:372
2465099226,8cyclictest22-21ksoftirqd/119:49:501
2465099225,8cyclictest22-21ksoftirqd/121:50:021
2465099224,8cyclictest22-21ksoftirqd/123:12:081
2465099224,8cyclictest22-21ksoftirqd/123:12:081
2465099224,6cyclictest22-21ksoftirqd/120:25:241
2465099223,9cyclictest22-21ksoftirqd/122:37:121
2465099222,9cyclictest22-21ksoftirqd/119:51:151
2465099221,12cyclictest641-21runrttasks21:01:221
24650992210,4cyclictest22-21ksoftirqd/120:02:101
2465099220,10cyclictest22-21ksoftirqd/119:40:191
2464999227,6cyclictest12-21ksoftirqd/023:40:000
2464999227,11cyclictest7589-21fschecks_count22:05:220
24649992213,7cyclictest23499-21apt-get22:50:210
24649992213,7cyclictest22560-21basename21:20:000
24649992213,6cyclictest2106-21df_inode22:00:230
24649992212,7cyclictest1504-21fw_packets20:45:210
24652992113,6cyclictest27751-21ntpq19:15:263
24652992113,6cyclictest17551-21apt21:15:163
24652992110,7cyclictest24693-21apt-get19:10:143
24651992114,4cyclictest19153-21sh21:45:282
2465099217,5cyclictest22-21ksoftirqd/120:24:271
2465099213,4cyclictest22-21ksoftirqd/119:15:001
24650992115,4cyclictest22-21ksoftirqd/120:19:591
24650992110,3cyclictest22-21ksoftirqd/120:09:411
24649992115,4cyclictest5750-21hwlatdetect19:40:200
24649992113,6cyclictest25129-21perl19:10:170
24649992112,7cyclictest24614-21ls21:50:300
24649992112,7cyclictest23022-21apt-get21:20:170
24649992112,7cyclictest19108-21ntp_states22:45:310
24649992112,6cyclictest3344-21cat20:50:150
24649992112,6cyclictest30040-21df_inode23:25:220
24649992111,6cyclictest55150irq/140-enp3s0-23:30:200
24649992110,7cyclictest55150irq/140-enp3s0-23:50:210
233182210,4sleep334-21ksoftirqd/321:20:193
196362210,4sleep00-21swapper/023:45:210
276982200,1sleep20-21swapper/219:15:252
2465299209,9cyclictest25645-21ssh22:21:153
2465299209,8cyclictest1875-21tr20:45:253
2465299207,11cyclictest14173-21basename23:40:243
24652992014,4cyclictest14701-21sh23:10:333
24652992014,4cyclictest14701-21sh23:10:323
24652992013,5cyclictest2046-21systemd-journal21:40:003
24652992013,4cyclictest5658-21sh20:55:203
24652992012,6cyclictest6166-21awk19:40:243
24652992012,6cyclictest29778-21missed_timers22:55:243
24652992012,6cyclictest18646-21cat20:10:183
24652992011,7cyclictest17787-21wc22:45:013
24652992011,6cyclictest8781-21cat21:00:293
2465199208,9cyclictest29653-21ssh22:25:262
2465199204,14cyclictest19313-21wc22:15:312
2465199203,3cyclictest141rcu_preempt00:19:372
2465199201,8cyclictest5392-21df00:35:262
24651992015,3cyclictest8948-21grep22:35:402
2465199201,3cyclictest141rcu_preempt23:20:322
24651992013,4cyclictest11689-21diskmemload23:39:072
24651992012,6cyclictest19978-21ssh21:16:342
2465099208,2cyclictest141rcu_preempt00:26:361
2465099205,3cyclictest141rcu_preempt20:46:171
2465099205,2cyclictest141rcu_preempt20:55:001
2465099202,9cyclictest22-21ksoftirqd/120:10:051
2465099202,13cyclictest10039-21snmpd20:30:131
24650992013,5cyclictest14089-21sed22:10:351
24650992011,6cyclictest366-21users20:40:301
2464999202,3cyclictest141rcu_preempt21:25:340
2464999202,3cyclictest141rcu_preempt21:25:340
2464999201,5cyclictest0-21swapper/021:39:240
24649992013,5cyclictest5325-21apt-get20:55:150
24649992012,6cyclictest23976-21fw_packets22:20:220
24649992012,6cyclictest18040-21apt20:10:120
24649992012,6cyclictest11175-21sed21:05:310
24649992011,6cyclictest17832-21seq22:15:060
2465299198,9cyclictest23431-21cstates21:50:183
2465299194,3cyclictest11689-21diskmemload22:10:143
2465299194,3cyclictest11689-21diskmemload00:39:443
2465299191,4cyclictest141rcu_preempt23:20:193
24652991914,3cyclictest29238-21kernelversion21:25:273
24652991914,3cyclictest29238-21kernelversion21:25:273
24652991914,3cyclictest26774-21sh00:23:383
24652991912,5cyclictest7555-21apt-get23:05:183
24652991912,5cyclictest28214-21cat00:25:243
24652991912,5cyclictest24854-21cat20:25:183
24652991912,5cyclictest20030-21seq21:46:183
24652991912,5cyclictest18459-21cut22:15:223
24652991912,5cyclictest12381-21ls21:10:183
24652991911,6cyclictest4282-21wc00:00:323
24652991911,6cyclictest30128-21sensors_temp20:35:313
24652991911,6cyclictest2205-21sshd00:30:533
24652991911,6cyclictest18235-21grep22:45:213
24652991910,6cyclictest2046-21systemd-journal20:40:013
2465199198,2cyclictest28-21ksoftirqd/221:54:402
2465199192,12cyclictest23499-21apt-get22:50:222
2465199191,8cyclictest17370-21latency_hist21:45:002
2465199191,5cyclictest0-21swapper/222:10:182
24651991912,5cyclictest7609-21grep21:00:182
24651991912,5cyclictest20763-21sed20:15:212
24651991912,5cyclictest2046-21systemd-journal21:00:002
24651991912,4cyclictest11689-21diskmemload00:25:402
24651991911,6cyclictest28818-21grep20:35:002
24651991911,6cyclictest2046-21systemd-journal21:15:002
24651991911,6cyclictest12733-21awk22:40:182
24651991910,6cyclictest814-21fschecks_time00:30:232
2464999191,8cyclictest25217-21ntp_states23:20:270
24649991914,3cyclictest3003-21unixbench-2d21:30:300
24649991914,3cyclictest27740-21ntp_states20:30:220
24649991913,4cyclictest10911-21sh00:07:430
24649991911,7cyclictest29985-21ssh00:25:570
24649991911,6cyclictest27643-21cron00:25:000
24649991911,6cyclictest16261-21awk20:05:200
24649991911,6cyclictest15059-21cat20:00:280
24649991911,5cyclictest26932-21grep19:15:180
24649991911,5cyclictest11689-21diskmemload23:15:230
24649991910,7cyclictest23444-21ntpq20:20:260
2464999190,4cyclictest141rcu_preempt20:00:000
19112190,8sleep228-21ksoftirqd/223:00:082
2465299189,6cyclictest29838-21tr23:25:213
2465299183,2cyclictest11689-21diskmemload22:04:013
24652991811,5cyclictest9921-21fw_packets19:50:183
24652991811,5cyclictest8348-21expr23:35:233
24652991811,5cyclictest3010-21expr23:30:213
24652991811,5cyclictest28155-21cat20:30:263
24652991811,5cyclictest22661-21grep20:20:173
24652991811,5cyclictest15382-21seq20:01:473
24652991810,6cyclictest20587-21tune2fs20:15:183
24652991810,5cyclictest55350irq/142-enp3s0-00:10:193
24652991810,5cyclictest18957-21acpi23:45:153
2465299180,9cyclictest203682rm23:15:323
2465299180,5cyclictest9692-21find21:05:013
2465299180,3cyclictest141rcu_preempt22:35:393
2465199183,10cyclictest10039-21snmpd20:20:162
2465199182,2cyclictest141rcu_preempt22:01:412
2465199182,2cyclictest141rcu_preempt21:22:462
2465199181,5cyclictest0-21swapper/221:58:302
24651991813,3cyclictest29837-21sh23:55:162
24651991813,3cyclictest29837-21sh23:55:162
24651991813,3cyclictest22883-21sh00:20:042
24651991812,4cyclictest16277-21sh23:42:002
24651991811,5cyclictest4354-21wc20:50:252
24651991811,5cyclictest2110-21cut20:45:272
24651991811,5cyclictest18801-21expr23:15:172
24651991811,5cyclictest15965-21wc20:05:192
24651991811,5cyclictest10687-21sed19:50:252
24651991811,4cyclictest11689-21diskmemload23:05:182
24651991810,6cyclictest28943-21awk22:55:172
24651991810,6cyclictest26055-21grep23:50:332
24651991810,5cyclictest6621-21tail19:40:282
2465099180,4cyclictest0-21swapper/120:55:201
2464999189,7cyclictest13554-21cut21:40:320
2464999189,6cyclictest30262-21df_inode23:55:240
2464999189,6cyclictest30262-21df_inode23:55:230
24649991813,3cyclictest4995-21sh22:32:390
24649991813,3cyclictest3668-21kernelversion00:00:250
24649991811,5cyclictest7469-21find21:00:150
24649991811,5cyclictest5361-21cat00:35:260
24649991811,5cyclictest30968-21sed20:40:010
24649991811,5cyclictest13261-21apt23:10:170
24649991811,5cyclictest13261-21apt23:10:170
24649991810,6cyclictest8816-21sed19:45:290
24649991810,6cyclictest18300-21cat00:15:220
24649991810,6cyclictest12906-21cat22:40:190
2465299173,2cyclictest141rcu_preempt23:50:063
2465299172,10cyclictest7145-21apt-get22:05:193
2465299171,3cyclictest141rcu_preempt21:30:103
2465299171,11cyclictest3176-21apt-get20:50:143
24652991710,5cyclictest8251-21awk19:45:253
2465199179,6cyclictest7593-21tune2fs22:05:222
2465199171,7cyclictest11690-21cron21:09:592
24651991713,2cyclictest18017-21/usr/sbin/munin20:10:272
24651991712,3cyclictest5963-21sh00:02:422
24651991711,4cyclictest30199-21sed20:35:322
24651991710,5cyclictest31833-21sed20:40:202
24651991710,5cyclictest25971-21grep19:10:272
2464999179,6cyclictest21590-21grep20:15:280
2464999179,6cyclictest10721-21sed19:50:260
2464999179,5cyclictest533-21cat00:30:200
24649991712,3cyclictest25446-21missed_timers20:25:240
24649991710,5cyclictest17967-21ls21:45:180
24652991611,3cyclictest16262-21fschecks_count20:05:203
2465199169,5cyclictest13913-21cut20:00:152
24651991611,3cyclictest24660-21/usr/sbin/munin20:25:252
24651991610,4cyclictest7299-21sort19:45:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional