You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-08-08 - 17:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Mon Aug 08, 2022 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
299132136101,24sleep10-21swapper/107:09:501
299132136101,24sleep10-21swapper/107:09:491
29883213498,25sleep30-21swapper/307:09:283
29883213498,25sleep30-21swapper/307:09:273
29771213496,27sleep00-21swapper/007:08:010
29771213496,27sleep00-21swapper/007:08:000
29738210885,16sleep20-21swapper/207:07:332
29738210885,16sleep20-21swapper/207:07:332
35892660,3sleep23592-21grep11:20:332
35892660,3sleep23592-21grep11:20:322
199922620,3sleep319982-21ssh09:40:203
211902600,7sleep334-21ksoftirqd/312:05:303
211902600,7sleep334-21ksoftirqd/312:05:293
4682570,2sleep00-21swapper/010:20:270
291342550,2sleep20-21swapper/209:20:002
164702550,2sleep10-21swapper/111:32:231
323962530,3chrt32391-21apt07:15:173
285742520,2sleep20-21swapper/210:45:252
158902520,2sleep00-21swapper/011:31:380
229002500,1sleep30-21swapper/310:40:253
17402430,2sleep00-21swapper/011:48:370
77512400,1sleep20-21swapper/210:55:292
3020399388,18cyclictest12870-21sh11:58:323
3020399385,18cyclictest0-21swapper/312:31:083
3020399376,12cyclictest3620-21sed11:20:343
3020399376,12cyclictest3620-21sed11:20:333
3020399365,9cyclictest0-21swapper/312:12:123
3020399356,12cyclictest13173-21chrt07:45:153
3020399355,13cyclictest25476-21/usr/sbin/munin09:15:343
3020399347,11cyclictest28022-21ssh09:47:053
3020099341,12cyclictest15662-21chrt09:05:170
3020399339,9cyclictest15615-21taskset09:05:033
3020399338,9cyclictest27146-21ssh11:41:463
3020399337,9cyclictest19313-21runrttasks10:37:123
3020399337,17cyclictest5617-21sh10:54:443
3020399337,10cyclictest26242-21latency_hist12:39:593
3020399337,10cyclictest15052-21ssh12:30:003
3020399337,10cyclictest10053-21sh10:58:133
3020399336,8cyclictest10486-21cat10:30:003
3020399335,8cyclictest2461-21cmp09:25:003
3020399335,14cyclictest0-21swapper/309:53:423
3020399335,14cyclictest0-21swapper/309:11:183
3020399335,11cyclictest4512-21mailstats11:50:273
3020399335,10cyclictest10992-21wc07:40:163
3020399332,10cyclictest0-21swapper/311:15:193
3532320,2chrt0-21swapper/309:22:433
3020399327,9cyclictest18057-21cut11:05:203
3020399325,16cyclictest6184-21ssh09:56:193
3020399325,15cyclictest0-21swapper/311:26:443
3020399319,8cyclictest11522-21ssh10:00:553
3020399314,9cyclictest0-21swapper/311:49:323
216272310,9sleep228-21ksoftirqd/211:36:502
3020399304,8cyclictest11647-21taskset10:30:303
30203993011,8cyclictest23932-21sh11:39:523
3020399297,8cyclictest12248-21gunzip08:55:233
3020399295,9cyclictest9824-21timerandwakeup09:30:313
3020399295,9cyclictest32296-21cat07:15:003
3020399295,6cyclictest15667-21ssh10:05:253
3020399293,9cyclictest0-21swapper/310:45:153
3020399292,17cyclictest0-21swapper/310:16:253
3020399292,16cyclictest0-21swapper/310:21:553
3020399292,16cyclictest0-21swapper/310:13:483
3020399291,9cyclictest0-21swapper/311:10:313
3020399284,10cyclictest31307-21cpuspeed_turbos12:15:163
3020399283,8cyclictest17126-21ssh11:33:153
3020399274,8cyclictest0-21swapper/312:21:143
3020399273,9cyclictest17539-21diskmemload09:37:503
3020399273,9cyclictest17539-21diskmemload09:37:493
3020399271,16cyclictest0-21swapper/312:03:223
25342270,8sleep012-21ksoftirqd/012:18:140
41032260,2chrt0-21swapper/007:25:000
308002260,2sleep00-21swapper/010:47:310
3020399264,8cyclictest29536-21sensors08:20:243
3020399264,8cyclictest13998-21rm11:01:023
305592250,2sleep10-21swapper/108:25:131
30201992511,6cyclictest14104-21cut09:35:201
30201992511,6cyclictest14104-21cut09:35:191
3020399244,6cyclictest0-21swapper/308:16:193
30200992422,1cyclictest0-21swapper/007:10:010
3020099241,10cyclictest56092sh09:27:000
207792240,1sleep00-21swapper/008:00:290
56512231,2sleep20-21swapper/209:55:372
3020399238,12cyclictest595-21fschecks_count08:30:183
30202992313,7cyclictest11731-21ntp_kernel_pll_07:40:302
3020199238,4cyclictest22-21ksoftirqd/110:43:011
30201992312,9cyclictest19558-21hwlatdetect11:35:241
30201992312,6cyclictest55250irq/141-enp3s0-12:10:191
3020099231,10cyclictest211842sh12:05:300
3020099231,10cyclictest211842sh12:05:290
30200992310,11cyclictest13467-21seq12:28:170
23642230,19sleep217539-21diskmemload12:18:022
3020399229,11cyclictest17172-21cat07:54:543
3020399222,4cyclictest0-21swapper/307:55:283
3020399222,4cyclictest0-21swapper/307:55:273
30203992211,8cyclictest9995-21sed07:35:453
3020399220,8cyclictest0-21swapper/307:20:163
30202992214,6cyclictest28841-21fw_packets08:20:152
30202992212,7cyclictest24764-21unixbench_singl11:10:302
30201992213,7cyclictest5602-21users09:55:341
30201992213,6cyclictest9351-21systemd-journal09:50:011
30201992211,9cyclictest22-21ksoftirqd/109:19:591
30201992210,3cyclictest22-21ksoftirqd/111:45:231
30200992214,5cyclictest641-21runrttasks10:59:100
30200992210,3cyclictest141rcu_preempt11:55:000
3020399219,9cyclictest20346-21sed08:00:233
3020399214,5cyclictest0-21swapper/307:30:173
3020399213,6cyclictest31455-21ntpq08:25:233
30203992111,7cyclictest13766-21cut09:00:193
3020299212,14cyclictest23590-21apt-get09:15:052
3020299212,13cyclictest14495-21sh12:00:182
30202992113,6cyclictest8468-21ls09:30:172
30202992113,6cyclictest5753-21smartctl10:25:212
30202992113,6cyclictest4057-21cat07:25:012
30202992113,6cyclictest14083-21ls09:00:212
3020199217,4cyclictest18641-21ls11:05:271
3020199212,13cyclictest9634-21apt-get12:25:131
30201992114,5cyclictest30177-21sh10:18:181
30201992113,6cyclictest15523-21ls07:50:151
30201992111,7cyclictest3300-21sensors_temp07:20:311
3020099219,7cyclictest55150irq/140-enp3s0-10:05:200
3020099211,6cyclictest27669-21sh12:10:550
120562210,2sleep10-21swapper/111:29:111
45802200,3sleep10-21swapper/110:24:531
35542201,1sleep20-21swapper/208:35:312
3020399209,8cyclictest24455-21fschecks_count08:10:203
3020399202,6cyclictest34-21ksoftirqd/308:53:503
30203992010,7cyclictest23651-21basename08:10:003
3020399200,7cyclictest0-21swapper/308:35:303
30202992018,1cyclictest0-21swapper/207:10:022
30202992013,5cyclictest8320-21dump-pmu-power07:35:012
30202992012,6cyclictest9184-21cat11:55:182
30202992012,6cyclictest29247-21fw_packets11:15:202
30202992012,6cyclictest26001-21ls09:45:242
30202992012,6cyclictest22223-21perl08:05:232
30202992011,7cyclictest8074-21cat08:45:312
3020199206,3cyclictest790250lldpd12:01:551
3020199206,12cyclictest22-21ksoftirqd/112:34:231
3020199206,12cyclictest141rcu_preempt07:13:581
3020199202,12cyclictest8808-21/usr/sbin/munin11:55:321
3020199201,5cyclictest0-21swapper/111:01:291
30201992015,3cyclictest22-21ksoftirqd/111:50:301
30201992014,4cyclictest14450irq/133-ahci[0009:55:001
30201992013,5cyclictest4894-21diskstats08:40:181
30201992012,6cyclictest1828-21open_files10:50:311
30201992011,7cyclictest9173-21ssh10:57:051
3020199201,17cyclictest28298-21sh11:14:591
30201992011,7cyclictest1928-21cat11:20:161
30201992011,7cyclictest1928-21cat11:20:151
30201992011,7cyclictest11800-21ntpq10:30:301
3020199201,10cyclictest3021latency_hist10:50:011
30201992010,8cyclictest28995-21rm11:44:121
3020199200,3cyclictest141rcu_preempt09:20:011
3020199200,18cyclictest935-21seq11:19:191
30200992013,5cyclictest5445-21sensors_temp07:25:320
30200992013,5cyclictest26072-21sed09:45:240
30200992013,5cyclictest16055-21idleruntime12:30:210
30200992012,6cyclictest13561-21cut07:45:190
30200992011,7cyclictest27723-21seq11:42:300
3020099201,10cyclictest5837-21grep12:20:300
30200992010,7cyclictest32560-21cat08:30:000
117112200,4chrt0-21swapper/210:01:112
3020399199,8cyclictest4492-21ls07:25:153
30203991911,6cyclictest5222-21cut08:40:223
30203991911,5cyclictest8126-21grep08:45:313
3020299199,8cyclictest13701-21diskstats11:30:202
3020299199,6cyclictest24499-21ssh11:40:162
30202991914,3cyclictest20398-21sh11:07:032
30202991913,4cyclictest7411-21sh11:53:452
30202991913,4cyclictest5482-21ntp_states08:40:242
30202991913,4cyclictest31930-21sh11:45:422
30202991913,4cyclictest30884-21sh10:19:132
30202991913,4cyclictest2189-21sh10:21:462
30202991913,4cyclictest14909-21sh10:33:452
30202991912,5cyclictest15493-21cat12:30:162
30202991911,6cyclictest8680-21processes11:25:242
30202991911,6cyclictest3641-21memory09:25:242
30202991911,6cyclictest10652-21cut07:39:592
30202991910,7cyclictest21710-21sed10:10:282
3020199198,2cyclictest22-21ksoftirqd/107:45:151
30201991913,4cyclictest21734-21sh09:13:121
30201991913,4cyclictest18068-21sh10:07:071
30201991912,5cyclictest24392-21cat08:10:171
30201991912,4cyclictest9948-21sh10:29:351
30201991911,6cyclictest7670-21date07:30:351
30201991911,6cyclictest545-21sed07:15:241
30201991911,6cyclictest13764-21tune2fs09:00:191
3020199191,16cyclictest10572-21ntp_states10:00:281
3020199191,11cyclictest0-21swapper/112:16:211
30201991910,7cyclictest26711-21grep08:15:161
30201991910,7cyclictest25432-21sed12:10:011
30201991910,7cyclictest25432-21sed12:10:001
30201991910,6cyclictest11773-21awk07:40:311
3020199190,5cyclictest0-21swapper/109:32:581
3020099199,7cyclictest23135-21apt-get11:10:160
3020099192,15cyclictest14598-21sed09:35:230
3020099192,15cyclictest14598-21sed09:35:220
30200991914,3cyclictest13046-21sh10:31:210
30200991913,4cyclictest23663-21sh11:39:320
3020099191,2cyclictest0-21swapper/009:22:590
30200991911,6cyclictest4159-21cat09:55:190
30200991911,6cyclictest29159-21cut11:15:190
30200991911,6cyclictest29070-21cut09:20:000
30200991911,6cyclictest20885-21ls10:10:200
30200991911,6cyclictest15142-21basename12:00:220
30200991910,6cyclictest8545-21grep11:25:230
3020099190,17cyclictest1981-21rt-features10:50:330
3020299182,11cyclictest4278-21apt-get07:25:132
3020299181,8cyclictest3149-21ssh10:51:312
3020299181,8cyclictest30704-21ssh09:50:172
3020299181,8cyclictest21026-21ssh12:35:182
30202991813,3cyclictest22152-21sh09:41:432
30202991812,4cyclictest25585-21sh10:43:022
30202991812,4cyclictest17534-21sh10:06:252
30202991812,4cyclictest12479-21grep11:00:192
30202991811,5cyclictest21116-21ntp_kernel_pll_12:05:282
30202991811,5cyclictest21116-21ntp_kernel_pll_12:05:272
30202991811,5cyclictest17207-21basename07:55:002
30202991811,5cyclictest10348-21interrupts12:25:192
30202991811,4cyclictest16473-21cpuspeed_turbos10:35:202
30202991811,4cyclictest15645-21cat09:05:162
30202991811,4cyclictest14577-21sed09:35:232
30202991811,4cyclictest14577-21sed09:35:222
30202991810,6cyclictest25408-21sed08:10:282
30202991810,6cyclictest18176-21cat09:10:182
3020199189,7cyclictest8812-21expr07:35:171
3020199182,11cyclictest11160-21apt-get08:55:131
3020199181,4cyclictest0-21swapper/110:11:451
30201991813,3cyclictest4905-21hwlatdetect12:20:211
30201991812,4cyclictest32640-21/usr/sbin/munin08:30:241
30201991812,4cyclictest19335-21sendmail08:00:001
30201991812,4cyclictest19335-21sendmail08:00:001
30201991811,5cyclictest5083-21missed_timers07:25:251
30201991811,5cyclictest4422-21cat08:40:011
30201991811,5cyclictest21141-21expr12:35:181
30201991811,4cyclictest19337-21sh09:40:011
30201991811,4cyclictest0-21swapper/110:35:201
3020199180,2cyclictest0-21swapper/109:29:511
3020099189,7cyclictest7277-21sed08:45:220
3020099189,6cyclictest23128-21cron09:15:000
3020099189,6cyclictest21948-21cat10:40:160
3020099188,6cyclictest55150irq/140-enp3s0-08:55:030
3020099182,12cyclictest8836-21apt-get08:50:150
3020099181,8cyclictest5541-21fschecks_count10:25:200
30200991813,3cyclictest24740-21kernelversion08:10:220
30200991813,3cyclictest14260-21sh10:04:280
30200991813,3cyclictest12112-21sh11:57:330
30200991812,4cyclictest641-21runrttasks11:05:160
30200991811,5cyclictest8496-21grep09:30:170
30200991811,5cyclictest3559-21sensors08:35:300
30200991811,5cyclictest21755-21cat08:05:180
30200991810,6cyclictest9306-21cat07:35:300
30200991810,6cyclictest5672-21tr08:40:270
30200991810,6cyclictest27020-21basename10:15:220
30200991810,6cyclictest22474-21cat12:35:310
30200991810,5cyclictest9351-21systemd-journal08:00:010
30200991810,5cyclictest9351-21systemd-journal08:00:000
3020299179,6cyclictest26420-21cut12:10:232
3020299179,6cyclictest17561-21grep07:55:152
3020299179,6cyclictest17561-21grep07:55:142
3020299171,7cyclictest4632-21ssh12:20:172
30202991713,2cyclictest30500-21/usr/sbin/munin08:25:172
30202991712,3cyclictest27493-21timerandwakeup08:15:262
30202991712,3cyclictest11923-21missed_timers08:55:202
30202991710,5cyclictest20896-21wc08:00:292
30202991710,5cyclictest10029-21cat08:50:252
3020199178,6cyclictest8991-21cstates08:50:161
3020199172,10cyclictest19610-21apt-get08:00:171
30201991710,5cyclictest8789-21cat08:50:001
30201991710,5cyclictest29500-21grep08:20:221
3020099178,5cyclictest12290-21entropy11:00:190
3020099171,7cyclictest19380-21ssh09:40:050
30200991712,3cyclictest3682-21sh11:20:350
30200991712,3cyclictest3682-21sh11:20:340
30200991712,3cyclictest30951-21cstates09:50:210
30200991712,3cyclictest16316-21/usr/sbin/munin10:35:260
30200991712,3cyclictest14249-21ntp_states09:00:240
3020099171,14cyclictest379-21fschecks_time07:15:230
3020299169,5cyclictest1330-21rt-features08:30:262
3020299168,6cyclictest13612-21grep07:45:182
3020299163,7cyclictest32392-21apt-get07:15:152
3020199168,6cyclictest23131-21cat08:05:321
3020199161,10cyclictest15217-21apt-get09:05:031
3020099169,5cyclictest648-21grep08:30:180
3020099168,6cyclictest29045-21sed08:20:190
3020099166,5cyclictest55150irq/140-enp3s0-07:40:140
3020099161,7cyclictest17045-21sshd07:53:180
3020099161,12cyclictest0-21swapper/007:30:140
30200991610,4cyclictest26379-21cat08:15:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional