You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-20 - 18:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rackaslot8s.osadl.org (updated Mon Mar 20, 2023 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
233522790,49sleep10-21swapper/107:05:241
26452179107,28sleep00-21swapper/007:07:060
25332165104,28sleep20-21swapper/207:05:382
26382164109,27sleep30-21swapper/307:07:003
44062730,1sleep10-21swapper/109:49:341
281322570,1sleep10-21swapper/110:36:391
98772540,2sleep20-21swapper/209:25:282
246932520,2sleep10-21swapper/109:10:451
74182510,2sleep00-21swapper/011:42:440
74182510,2sleep00-21swapper/011:42:430
167222500,1sleep10-21swapper/112:18:161
238742490,5chrt141rcu_preempt10:05:241
206582490,1sleep30-21swapper/309:05:183
82622480,1sleep10-21swapper/110:47:471
314799461,33cyclictest16636-21chrt09:31:212
314799460,29cyclictest25838-21/usr/sbin/munin11:30:322
225062460,1sleep10-21swapper/109:36:261
314799451,26cyclictest12823-21ssh11:20:012
314799441,25cyclictest23801-21sh12:24:492
3147994411,23cyclictest14891-21sh11:49:482
10432440,1sleep00-21swapper/009:19:120
10432440,1sleep00-21swapper/009:19:110
3147994313,14cyclictest4273-21sh09:20:572
314799431,27cyclictest0-21swapper/209:54:552
314799431,25cyclictest15020-21ssh10:53:582
314799431,24cyclictest12891-21taskset09:55:292
314799430,27cyclictest30343-21ssh11:35:052
314799430,25cyclictest20939-21ssh11:54:502
314799424,21cyclictest4053-21/usr/sbin/munin12:35:222
314799421,31cyclictest20278-21sh10:58:162
314799421,25cyclictest1891-21ssh09:46:282
314799411,30cyclictest18918-21sh10:29:512
314799411,26cyclictest24685-21sh11:01:312
314799411,24cyclictest9026-21ssh11:44:432
314799411,24cyclictest9026-21ssh11:44:422
3147994110,15cyclictest29036-21sh09:42:082
314799410,26cyclictest7070-21ssh10:46:162
314799410,24cyclictest25332-21ssh09:39:582
302702410,2chrt0-21swapper/109:43:401
95182400,1sleep20-21swapper/208:40:002
314799408,15cyclictest3311-21sh11:10:452
314799401,25cyclictest24635-21ssh10:34:302
314799401,23cyclictest31813-21ssh12:03:502
314799401,22cyclictest23136-21sh11:28:392
314799401,22cyclictest21669-21sh10:04:072
3147994010,21cyclictest10495-21missed_timers08:40:192
314799400,26cyclictest1149-21ssh12:32:322
314799400,26cyclictest1149-21ssh12:32:322
314799400,25cyclictest13567-21chrt08:47:392
314799400,23cyclictest984-21ssh10:14:152
307032400,1sleep20-21swapper/209:15:492
307032400,1sleep20-21swapper/209:15:482
314899392,10cyclictest844-21chrt12:05:183
314799391,22cyclictest24279-21ssh11:56:482
314799391,22cyclictest22348-21diskmemload12:16:092
314799391,22cyclictest13650-21chrt07:34:512
314799391,21cyclictest4804-21chrt07:11:432
314799391,20cyclictest28576-21taskset10:37:102
314799390,30cyclictest25386-21ssh09:11:382
314799390,25cyclictest11535-21sh10:22:542
314799390,23cyclictest26474-21runrttasks08:02:422
314799390,23cyclictest11169-21chrt07:25:322
314799390,22cyclictest26117-21sh10:07:262
314799390,22cyclictest12152-21sh11:19:292
48812380,1sleep30-21swapper/311:40:223
48812380,1sleep30-21swapper/311:40:213
314799381,28cyclictest22348-21diskmemload12:29:512
314799381,27cyclictest22348-21diskmemload10:18:162
314799381,23cyclictest914-21ssh10:40:522
314799381,23cyclictest8535-21ssh12:10:422
314799380,28cyclictest31206-21sh11:07:212
314799380,25cyclictest6994-21taskset07:17:142
314799380,21cyclictest24185-21taskset07:56:032
314799380,21cyclictest20453-21chrt07:50:082
314799380,21cyclictest20187-21taskset07:49:452
258232380,2sleep012-21ksoftirqd/011:30:310
314899372,10cyclictest5458-21ssh09:50:213
314799371,20cyclictest14952-21taskset08:50:202
314799370,22cyclictest3258-21ssh12:06:252
314799370,21cyclictest5018-21chrt08:28:122
314799370,21cyclictest15781-21chrt07:38:282
314799361,19cyclictest18072-21taskset07:44:592
314799360,23cyclictest523-21chrt08:16:042
314799360,21cyclictest72732sleep208:34:382
314799350,22cyclictest2978-21chrt08:24:352
314799350,22cyclictest28748-21taskset08:09:272
314899342,7cyclictest25831-21sh09:40:053
314899341,16cyclictest11695-21gltestperf09:55:173
314799341,20cyclictest19940-21chrt09:01:512
314899332,8cyclictest6694-21ssh12:10:183
3148993313,3cyclictest29070-21open_files11:05:263
314799330,20cyclictest9166-21chrt07:22:232
314799330,20cyclictest30245-21sed08:10:242
314899321,28cyclictest17558-21fschecks_time10:00:203
314799321,20cyclictest17719-21taskset08:56:012
3148993121,7cyclictest22348-21diskmemload12:25:193
3148993018,1cyclictest0-21swapper/311:10:213
314699301,25cyclictest18359-21python310:00:291
134482300,2sleep10-21swapper/110:51:591
45722290,3chrt0-21swapper/107:10:271
314699291,27cyclictest0-21swapper/109:25:031
314599287,4cyclictest790250lldpd10:40:160
3145992825,2cyclictest0-21swapper/009:25:030
314599281,25cyclictest0-21swapper/010:35:270
216092280,1sleep10-21swapper/109:05:271
3148992722,3cyclictest8807-21hddtemp_smartct09:25:173
314799271,23cyclictest21227-21/usr/sbin/munin09:05:232
314599271,24cyclictest0-21swapper/010:25:290
204802260,2chrt0-21swapper/012:20:450
3146992521,2cyclictest7654-21strings12:10:271
314699251,23cyclictest0-21swapper/111:10:341
314599255,3cyclictest22348-21diskmemload11:08:340
3145992523,1cyclictest0-21swapper/009:25:010
3145992510,12cyclictest32173-21diskstats09:45:160
314599250,1cyclictest0-21swapper/012:10:280
3148992422,1cyclictest0-21swapper/311:00:203
314899242,15cyclictest14052-21ssh09:30:043
3148992421,2cyclictest0-21swapper/310:35:193
3146992420,2cyclictest7719-21perl11:15:181
3146992416,6cyclictest16946-21memory07:40:191
314699240,1cyclictest0-21swapper/111:05:191
3145992422,1cyclictest0-21swapper/009:50:040
3145992414,7cyclictest9351-21systemd-journal07:50:010
3145992413,8cyclictest7892-21apt-get10:20:160
314599240,1cyclictest0-21swapper/011:11:110
3148992321,1cyclictest0-21swapper/310:40:213
3146992321,1cyclictest0-21swapper/110:25:301
3146992316,5cyclictest7920-21apt-config09:25:001
3146992316,5cyclictest28275-21ssh12:28:121
323342220,2sleep10-21swapper/110:40:231
3148992213,5cyclictest15179-21apt-key11:50:003
314699220,11cyclictest77052ssh12:38:251
314599224,4cyclictest157962sh10:54:550
3145992220,1cyclictest0-21swapper/010:15:260
314899212,15cyclictest27496-21ssh12:00:193
314699218,3cyclictest790250lldpd10:24:161
314699211,7cyclictest19376-21processes09:00:231
3146992117,2cyclictest28467-21apt-config09:15:001
3146992117,2cyclictest28467-21apt-config09:15:001
3146992112,7cyclictest8328-21memory08:35:181
3146992112,7cyclictest498-21dbus-daemon11:39:591
3146992112,7cyclictest13143-21cat07:30:261
3146992112,6cyclictest6495-21python307:15:261
3146992111,8cyclictest10763-21munin-run09:54:591
314599214,3cyclictest22348-21diskmemload10:56:110
314599214,3cyclictest22348-21diskmemload09:56:440
314599212,13cyclictest24447-21apt-get12:25:180
314599211,9cyclictest790250lldpd09:30:160
314599211,5cyclictest0-21swapper/012:03:050
138402210,1sleep30-21swapper/308:50:013
314899202,12cyclictest16156-21apt-get08:55:143
3148992012,6cyclictest498-21dbus-daemon08:00:013
3148992012,6cyclictest11580-21ntp_states10:50:253
314699207,10cyclictest0-21swapper/111:50:001
314699201,3cyclictest141rcu_preempt10:33:181
3146992010,7cyclictest23779-21python307:55:251
314599204,3cyclictest141rcu_preempt11:50:180
314599203,12cyclictest19693-21ssh10:30:160
3145992013,4cyclictest8749-21sh11:15:280
3145992012,6cyclictest2472-21sed08:20:300
3145992012,6cyclictest21230-21wc09:05:240
3145992011,7cyclictest8490-21ntp_states08:35:220
3145992011,7cyclictest23873-21ntpq10:05:230
3145992010,8cyclictest1631-21basename10:15:010
314599200,12cyclictest257772rm09:40:020
314899192,14cyclictest0-21swapper/308:30:123
314899192,12cyclictest2175-21apt-get09:20:163
3148991913,3cyclictest2544-21sh11:38:553
3148991912,5cyclictest30221-21cat09:15:313
3148991912,5cyclictest30221-21cat09:15:303
3148991911,6cyclictest862-21wc09:45:273
3148991911,6cyclictest5182-21cut10:45:213
3148991911,6cyclictest3648-21fw_packets07:10:163
314899191,13cyclictest29059-21apt-get08:10:123
3148991910,7cyclictest1232-21expr08:20:153
3148991910,5cyclictest22348-21diskmemload09:35:173
3148991910,5cyclictest20724-21ntp_offset10:30:253
3146991913,4cyclictest26471-21sh11:03:411
3146991913,4cyclictest25805-21sh11:58:391
3146991912,4cyclictest6180-21sh12:09:581
3146991911,6cyclictest7827-21df_inode07:20:151
3146991911,6cyclictest24273-21cut11:29:591
3146991911,6cyclictest15893-21cron08:55:011
3146991911,5cyclictest15689-21cat09:30:301
3146991910,6cyclictest15157-21cat07:35:251
314599193,13cyclictest9172-21ssh12:40:010
314599193,12cyclictest12758-21diskmemload12:15:140
3145991914,3cyclictest13760-21sh08:50:010
3145991911,6cyclictest11853-21cat07:30:130
314899182,11cyclictest19859-21ntp_states11:25:243
3148991813,3cyclictest7855-21/usr/sbin/munin10:20:283
3148991813,3cyclictest13668-21cron07:35:003
3148991812,4cyclictest7007-21sh10:19:303
3148991812,4cyclictest28486-21sh10:10:033
3148991812,3cyclictest0-21swapper/312:20:163
3148991811,5cyclictest7454-21cut07:20:013
3148991811,5cyclictest25306-21grep11:30:253
3148991811,5cyclictest23189-21awk11:55:283
3148991811,4cyclictest22348-21diskmemload11:20:263
314699189,6cyclictest10879-21mailstats08:40:241
3146991813,3cyclictest32249-21sh12:04:221
3146991813,3cyclictest31436-21cpuspeed_turbos08:15:131
3146991813,3cyclictest15530-21sh11:21:291
3146991812,4cyclictest14947-21sh09:58:011
3146991812,3cyclictest19421-21sh10:57:141
3146991811,5cyclictest7499-21ls08:35:001
3146991811,5cyclictest18629-21cat07:45:141
3146991811,5cyclictest16559-21grep08:55:161
3146991811,5cyclictest16338-21smartctl11:50:211
3146991811,5cyclictest11855-21find08:45:151
3146991811,4cyclictest27156-21sh11:31:271
3146991810,6cyclictest30117-21date10:10:271
3146991810,5cyclictest19344-21latency12:20:211
314699180,3cyclictest141rcu_preempt12:31:431
314699180,3cyclictest141rcu_preempt12:31:421
314599189,6cyclictest20446-21df_inode09:35:190
314599183,2cyclictest141rcu_preempt11:20:170
3145991811,5cyclictest9351-21systemd-journal07:24:590
3145991811,5cyclictest32717-21sed08:15:260
3145991811,5cyclictest23190-21sed07:55:200
3145991811,5cyclictest17831-21sed10:00:240
314599181,12cyclictest9784-21taskset11:45:190
3145991810,6cyclictest19242-21grep11:25:180
3145991810,5cyclictest55150irq/140-enp3s0-11:35:130
314899178,6cyclictest18709-21cut09:00:163
314899171,6cyclictest9351-21systemd-journal08:30:013
3148991712,3cyclictest14439-21sh12:15:293
3148991711,4cyclictest8949-21unixbench-2d08:35:263
3148991711,4cyclictest22348-21diskmemload10:56:483
3148991711,4cyclictest19059-21sh10:30:003
3148991711,4cyclictest11532-21dump-pmu-power07:30:013
314899171,13cyclictest4818-21pmu-power12:35:333
3148991711,3cyclictest31918-21sed08:15:163
314899171,13cyclictest28210-21apt-config09:15:003
3148991711,3cyclictest22348-21diskmemload11:17:423
3148991710,5cyclictest30871-21cat12:30:233
3148991710,5cyclictest30871-21cat12:30:223
3148991710,5cyclictest28143-21cat08:05:253
3148991710,5cyclictest25284-21grep08:00:243
3148991710,5cyclictest24423-21awk10:05:283
3148991710,5cyclictest20827-21forks07:50:153
3148991710,5cyclictest15975-21cut07:40:013
314699179,6cyclictest20846-21cut07:50:151
314699172,10cyclictest24787-21apt-get08:00:191
314699171,5cyclictest22348-21diskmemload11:40:201
314699171,5cyclictest22348-21diskmemload11:40:201
3146991712,3cyclictest26896-21/usr/sbin/munin08:05:151
314699171,12cyclictest2030-21ssh10:15:141
314699171,10cyclictest9641-21apt-get07:25:131
3146991710,5cyclictest4234-21awk08:25:231
3146991710,5cyclictest22-21ksoftirqd/108:10:161
314599179,5cyclictest498-21dbus-daemon07:25:240
314599178,6cyclictest22215-21apt-get11:00:190
3145991712,3cyclictest32458-21sh12:31:110
3145991712,3cyclictest32458-21sh12:31:100
3145991712,3cyclictest20457-21/usr/sbin/munin07:50:250
3145991712,3cyclictest18290-21/usr/sbin/munin09:00:160
3145991711,4cyclictest26017-21sed08:00:320
3145991711,4cyclictest23212-21fschecks_time09:10:170
314599171,12cyclictest3186-21apt-get07:10:120
314599171,11cyclictest21495-21apt-get11:55:140
3145991710,5cyclictest5317-21grep10:45:220
3145991710,5cyclictest30480-21grep08:10:260
3145991710,5cyclictest1775-21sed12:05:270
314899169,5cyclictest17723-21seq07:40:393
314899169,5cyclictest16093-21cut08:55:013
314899163,7cyclictest15084-21apt-get11:50:023
314899162,8cyclictest9732-21apt-get08:40:123
3148991611,3cyclictest19510-21switchtime07:45:263
3148991610,4cyclictest7693-21expr07:20:143
314699169,5cyclictest1427-21cat08:20:171
314599168,6cyclictest15940-21sed08:55:000
314599162,10cyclictest26923-21apt-get08:05:130
3145991611,3cyclictest641-21runrttasks08:33:100
3145991611,3cyclictest641-21runrttasks07:18:100
3145991611,3cyclictest3257-21/usr/sbin/munin08:25:220
3145991610,4cyclictest641-21runrttasks08:42:530
3145991610,4cyclictest18088-21latency_hist07:45:000
3145991610,3cyclictest16130-21acpi08:55:120
314599156,6cyclictest13990-21cpuspeed_turbos07:35:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional