You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-07-06 - 17:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8s.osadl.org (updated Wed Jul 06, 2022 12:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1604421770,12sleep20-21swapper/207:05:222
16588214896,25sleep30-21swapper/307:06:033
166052139102,25sleep00-21swapper/007:06:160
165662139103,25sleep10-21swapper/107:05:451
43352840,2sleep10-21swapper/112:05:361
131112600,2sleep20-21swapper/212:15:002
171482590,2sleep30-21swapper/310:20:193
321862580,2sleep20-21swapper/211:04:132
45512550,2sleep30-21swapper/310:38:563
66432530,1sleep20-21swapper/212:37:552
246482520,8sleep228-21ksoftirqd/210:56:252
324842510,2sleep032452-21ssh10:35:000
273432500,1sleep10-21swapper/108:45:241
227732460,1sleep00-21swapper/007:20:240
40422430,2sleep30-21swapper/312:35:193
25582420,2sleep10-21swapper/111:05:361
21572420,1sleep00-21swapper/010:35:440
34352390,2sleep10-21swapper/112:35:141
24942390,1sleep30-21swapper/310:06:453
270312380,1sleep20-21swapper/212:26:352
42252330,3chrt0-21swapper/209:39:272
189102280,2sleep10-21swapper/111:20:241
1716899285,13cyclictest34-21ksoftirqd/309:25:223
288052270,11sleep228-21ksoftirqd/211:30:132
288052270,11sleep228-21ksoftirqd/211:30:132
17168992723,3cyclictest331rcuc/311:47:173
17168992710,3cyclictest141rcu_preempt12:15:003
301572260,2sleep10-21swapper/111:01:261
1716899268,8cyclictest34-21ksoftirqd/307:32:513
1716899267,5cyclictest790250lldpd09:05:133
1716899267,10cyclictest34-21ksoftirqd/312:18:133
1716899266,10cyclictest34-21ksoftirqd/309:48:393
1716899264,13cyclictest34-21ksoftirqd/309:40:253
1716899263,6cyclictest34-21ksoftirqd/312:27:283
17168992621,3cyclictest331rcuc/310:56:523
1716899261,7cyclictest34-21ksoftirqd/309:55:183
17168992610,8cyclictest34-21ksoftirqd/311:44:513
1716799261,23cyclictest0-21swapper/207:10:262
1716599261,24cyclictest0-21swapper/007:10:340
1716899258,4cyclictest34-21ksoftirqd/312:00:083
1716899257,9cyclictest34-21ksoftirqd/309:22:313
1716899257,6cyclictest34-21ksoftirqd/307:25:223
1716899256,6cyclictest34-21ksoftirqd/311:25:223
1716899253,7cyclictest34-21ksoftirqd/311:50:133
17168992521,3cyclictest331rcuc/310:29:253
1716899251,9cyclictest6576-21chrt09:11:113
1716899251,23cyclictest0-21swapper/307:10:193
17167992515,7cyclictest26313-21df07:30:152
17166992523,1cyclictest0-21swapper/107:10:251
299392240,3sleep129924-21munin-run08:55:011
1716899247,8cyclictest34-21ksoftirqd/309:38:413
1716899247,4cyclictest34-21ksoftirqd/310:02:013
1716899247,10cyclictest34-21ksoftirqd/312:31:343
1716899244,7cyclictest34-21ksoftirqd/311:15:153
1716899243,8cyclictest34-21ksoftirqd/309:00:103
1716899243,7cyclictest34-21ksoftirqd/310:15:043
1716899242,7cyclictest34-21ksoftirqd/311:05:493
1716899242,6cyclictest34-21ksoftirqd/309:15:423
17168992419,3cyclictest331rcuc/312:22:363
17168992419,3cyclictest331rcuc/312:22:353
17168992419,3cyclictest331rcuc/310:11:123
1716899240,6cyclictest28128-21apt-get07:35:123
17167992412,10cyclictest27558-21sed10:00:192
17166992413,8cyclictest22592-21df10:55:141
1716699241,11cyclictest106152sh10:44:431
1716899237,9cyclictest34-21ksoftirqd/312:06:123
1716899233,7cyclictest34-21ksoftirqd/310:43:233
1716899233,12cyclictest34-21ksoftirqd/309:50:193
1716899233,12cyclictest34-21ksoftirqd/309:50:183
1716899231,7cyclictest34-21ksoftirqd/311:02:133
1716899230,7cyclictest34-21ksoftirqd/311:30:043
1716899230,7cyclictest34-21ksoftirqd/311:30:033
17167992314,7cyclictest415-21df_inode07:45:142
17166992315,6cyclictest23838-21ssh09:27:091
1716599234,14cyclictest6685-21snmpd11:18:340
1716599233,14cyclictest16325-21apt-get10:20:120
17165992317,4cyclictest12-21ksoftirqd/012:20:000
17165992314,7cyclictest7957-21/usr/sbin/munin12:10:200
17165992314,6cyclictest22789-21python309:55:280
1716899228,3cyclictest34-21ksoftirqd/311:12:503
1716899226,9cyclictest34-21ksoftirqd/310:48:323
1716899226,8cyclictest34-21ksoftirqd/307:15:453
1716899225,6cyclictest34-21ksoftirqd/309:30:183
1716899223,10cyclictest34-21ksoftirqd/311:20:383
1716899222,7cyclictest34-21ksoftirqd/310:30:193
1716899222,4cyclictest34-21ksoftirqd/308:40:123
1716899221,6cyclictest34-21ksoftirqd/311:35:023
1716899221,6cyclictest34-21ksoftirqd/310:51:163
1716899220,5cyclictest0-21swapper/307:55:123
1716799221,9cyclictest111502sh11:43:202
17167992217,3cyclictest32042-21sh12:30:572
17167992214,6cyclictest5902-21ls09:10:292
17167992214,6cyclictest20265-21ls08:30:192
17167992214,6cyclictest17037-21cron08:25:002
17167992212,6cyclictest55350irq/142-enp3s0-11:35:112
1716599228,10cyclictest3955-21diskmemload11:20:210
1716599222,15cyclictest16031-21kworker/u8:110:05:110
17165992212,7cyclictest498-21dbus-daemon11:45:230
1716899217,11cyclictest21902-21cut07:20:153
1716899213,7cyclictest34-21ksoftirqd/308:50:183
1716899212,7cyclictest34-21ksoftirqd/308:45:173
1716899211,5cyclictest0-21swapper/311:55:203
17168992113,6cyclictest9044-21entropy08:05:143
17168992112,6cyclictest22517-21ntpq08:35:203
1716899210,7cyclictest34-21ksoftirqd/308:13:313
17167992114,5cyclictest5354-21ssh11:09:242
17167992113,5cyclictest55350irq/142-enp3s0-11:45:232
17167992112,7cyclictest30564-21cat07:40:132
17166992113,6cyclictest24439-21perl11:55:171
17166992112,6cyclictest31984-21seq10:34:371
17166992111,7cyclictest31546-21sensors_temp07:40:221
17166992110,3cyclictest289602needreboot07:35:191
1716599213,15cyclictest3611-21grep12:35:160
1716899201,7cyclictest34-21ksoftirqd/308:19:363
1716899201,7cyclictest34-21ksoftirqd/307:49:113
1716899201,6cyclictest34-21ksoftirqd/307:50:523
17168992015,3cyclictest331rcuc/308:30:493
17168992012,6cyclictest7938-21date08:00:253
1716899200,8cyclictest34-21ksoftirqd/307:40:173
1716799203,12cyclictest23842-21apt-get07:25:122
1716799202,12cyclictest2264-21apt-get07:50:152
17167992014,4cyclictest16510-21latency_hist10:50:002
17167992012,6cyclictest8877-21cat08:05:132
17167992012,6cyclictest18828-21diskstats11:50:152
17167992012,6cyclictest15680-21cat09:20:262
17167992012,6cyclictest14962-21date08:20:002
17167992012,6cyclictest10358-21tune2fs09:15:162
17167992010,2cyclictest28-21ksoftirqd/210:30:202
1716699202,15cyclictest10604-21basename08:10:011
17166992013,5cyclictest8682-21ls11:40:241
17166992013,5cyclictest12164-21ls08:10:251
17166992012,6cyclictest4516-21expr09:10:141
17166992012,6cyclictest32510-21expr10:05:141
17166992012,6cyclictest16368-21cat08:20:231
17166992011,7cyclictest11937-21cat09:45:221
17166992011,6cyclictest55250irq/141-enp3s0-10:25:111
1716599202,6cyclictest21767-21apt-get10:25:110
1716599202,12cyclictest29485-21apt-get12:00:140
1716599202,12cyclictest15154-21apt-get08:20:120
1716599201,8cyclictest3950-21munin-run09:10:010
17165992015,3cyclictest0-21swapper/008:30:250
17165992013,4cyclictest24476-21sh12:25:090
17165992012,6cyclictest534-21cat11:05:010
17165992012,6cyclictest20192-21gzip12:20:220
17165992012,6cyclictest20192-21gzip12:20:220
17165992012,5cyclictest3955-21diskmemload11:30:120
17165992012,5cyclictest3955-21diskmemload11:30:110
1716599200,9cyclictest172402ssh09:50:230
1716599200,9cyclictest172402ssh09:50:230
226232190,7sleep122-21ksoftirqd/108:35:211
1716899191,6cyclictest34-21ksoftirqd/308:55:403
1716899191,6cyclictest34-21ksoftirqd/308:25:163
17168991911,6cyclictest15575-21tune2fs08:20:163
1716799192,11cyclictest23030-21apt-get11:25:112
1716799191,7cyclictest8445-21cron08:05:002
17167991914,3cyclictest22205-21latency07:20:172
17167991913,4cyclictest8399-21sh11:11:082
17167991913,4cyclictest7434-21sh09:41:282
17167991913,4cyclictest7155-21sh12:09:222
17167991913,4cyclictest3852-21sh10:38:012
17167991913,4cyclictest23791-21sh09:27:042
17167991913,4cyclictest21762-21sh10:25:112
17167991913,4cyclictest16660-21sh10:20:142
17167991913,4cyclictest16319-21sh12:17:012
17167991913,4cyclictest15202-21sh10:19:042
17167991912,5cyclictest32385-21ls09:00:122
17167991912,5cyclictest22061-21grep09:55:212
17167991912,5cyclictest17500-21timerwakeupswit09:50:262
17167991912,5cyclictest17500-21timerwakeupswit09:50:252
17167991911,6cyclictest5570-21cat10:10:132
17167991911,6cyclictest30087-21sh12:00:182
17167991911,6cyclictest24596-21basename11:55:172
17167991911,6cyclictest19627-21idleruntime12:20:182
17167991911,6cyclictest19627-21idleruntime12:20:172
17167991910,6cyclictest24148-21tune2fs08:40:142
17167991910,6cyclictest19371-21cpu07:15:202
1716699199,5cyclictest55250irq/141-enp3s0-12:30:121
1716699198,9cyclictest10663-21gdbus10:15:011
1716699191,9cyclictest9732-21ssh09:15:041
1716699191,5cyclictest3955-21diskmemload12:18:371
17166991913,4cyclictest9682-21sh09:44:311
17166991912,5cyclictest7265-21seq10:10:561
17166991912,5cyclictest24021-21ntp_kernel_pll_11:25:201
17166991912,5cyclictest19370-21awk12:20:161
17166991912,5cyclictest19370-21awk12:20:151
17166991912,5cyclictest17025-21ntpq09:50:201
17166991912,5cyclictest17025-21ntpq09:50:201
17166991911,6cyclictest8516-21seq11:11:161
17166991911,6cyclictest32458-21tune2fs09:35:151
17166991911,6cyclictest22187-21if_err_enp3s009:55:231
17166991911,6cyclictest20684-21wc09:25:011
17166991911,6cyclictest20372-21uname08:30:211
17166991911,6cyclictest1642-21ls10:35:251
17166991911,5cyclictest9462-21cat12:10:231
17166991911,5cyclictest25964-21acpi07:30:131
17166991910,7cyclictest12893-21which11:15:231
17166991910,6cyclictest27404-21missed_timers09:30:191
1716699190,17cyclictest20138-21ssh10:23:181
1716599198,7cyclictest11636-21sed10:15:180
1716599192,8cyclictest6614-21fschecks_count11:10:140
1716599192,6cyclictest1968950kworker/u8:208:10:060
1716599192,12cyclictest5762-21ssh10:40:120
1716599191,6cyclictest24080-21/usr/sbin/munin11:25:200
17165991914,3cyclictest21249-21sh07:20:010
17165991912,4cyclictest3955-21diskmemload10:10:250
17165991912,4cyclictest0-21swapper/010:50:110
17165991911,6cyclictest6101-21awk09:40:220
17165991911,6cyclictest27390-21uptime08:45:250
17165991911,6cyclictest27064-21cat10:00:160
17165991911,6cyclictest14966-21seq09:49:050
1716599191,15cyclictest7628-21cron12:09:590
17165991910,6cyclictest28167-1kworker/u9:012:30:110
1716799186,8cyclictest2060-21apt-get09:05:112
1716799182,11cyclictest11669-21apt-get11:15:122
17167991813,3cyclictest5711-21/usr/sbin/munin10:40:212
17167991813,3cyclictest25958-21apt-get09:30:012
17167991812,4cyclictest4169-21sh10:09:032
17167991812,4cyclictest19868-21sh11:21:162
17167991811,5cyclictest28243-21expr08:50:132
17167991811,5cyclictest21900-21expr08:35:142
17167991811,5cyclictest10686-21cat09:45:132
17166991813,3cyclictest23836-21cut08:40:131
17166991813,3cyclictest1026-21sh12:03:291
17166991812,4cyclictest28900-21sh12:29:071
17166991812,4cyclictest28728-21sh10:00:461
17166991812,4cyclictest22792-21sh11:53:581
17166991811,5cyclictest700-21basename07:45:161
17166991811,5cyclictest6748-21expr08:00:131
17166991811,5cyclictest4642-21cat07:55:131
17166991811,5cyclictest30413-21cat08:55:131
17166991811,5cyclictest21574-21awk07:20:131
17166991811,5cyclictest21124-21seq07:18:351
17166991811,4cyclictest3955-21diskmemload11:32:541
17166991811,4cyclictest3955-21diskmemload11:32:541
17166991810,6cyclictest24107-21cat07:25:121
17166991810,6cyclictest2120-21expr11:35:131
17166991810,6cyclictest13064-21cat08:15:121
17166991810,6cyclictest12746-21cat10:45:251
17166991810,5cyclictest9351-21systemd-journal11:45:021
17166991810,5cyclictest18211-21ntp_offset08:25:201
1716599189,7cyclictest30723-21df07:40:130
1716599189,6cyclictest6291-21cron07:59:590
1716599188,7cyclictest24672-21ntp_states08:40:200
17165991813,3cyclictest4703-21sh11:08:310
17165991813,3cyclictest10480-21hwlatdetect09:15:170
17165991812,4cyclictest32062-21idleruntime-cro09:00:010
17165991812,4cyclictest21780-21sh11:52:380
17165991812,4cyclictest18321-21sh09:22:070
17165991811,5cyclictest7520-21ls08:00:220
17165991811,5cyclictest28551-21tune2fs07:35:160
17165991811,5cyclictest2769-21expr07:50:170
17165991811,5cyclictest24262-21cut07:25:150
17165991811,4cyclictest3955-21diskmemload09:10:310
17165991810,6cyclictest7380-21cat11:40:120
17165991810,6cyclictest2611-21ls11:35:180
17165991810,6cyclictest24669-21irqstats11:55:180
17167991712,3cyclictest5065-21kernelversion07:55:172
17167991712,3cyclictest19142-21sh10:51:242
17167991711,4cyclictest30446-21expr08:55:132
17167991711,4cyclictest28298-21cstates07:35:122
17167991710,5cyclictest17775-21expr08:25:162
17167991710,5cyclictest11155-21ls08:10:142
17166991712,3cyclictest388-21gltestperf09:00:151
17166991712,3cyclictest19137-21sh10:51:241
17166991710,5cyclictest2550-21grep09:05:151
1716599179,6cyclictest28185-21expr08:50:130
1716599179,6cyclictest10607-21grep08:10:000
17165991712,3cyclictest27547-21sh09:30:220
17165991712,3cyclictest26833-21sh10:59:220
17165991712,3cyclictest25682-21sh09:29:390
1716599171,11cyclictest12992-21apt-get08:15:110
17165991710,5cyclictest0-21swapper/008:25:150
17165991710,4cyclictest3955-21diskmemload10:46:080
17165991710,4cyclictest3955-21diskmemload09:35:590
17167991610,4cyclictest26105-21cat08:45:142
1716699168,6cyclictest2239-21/usr/sbin/munin07:50:181
1716599169,5cyclictest626-21grep09:00:170
1716599169,5cyclictest22026-21perl08:35:150
17165991612,2cyclictest32509-21/usr/sbin/munin07:45:160
17165991610,4cyclictest26821-21needreboot07:30:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional