You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-10 - 17:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Sun Dec 10, 2023 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
158592182159,15sleep20-21swapper/207:07:002
158932180105,26sleep10-21swapper/107:07:271
159392176100,29sleep00-21swapper/007:08:020
157932163103,29sleep30-21swapper/307:06:093
259512760,2sleep00-21swapper/011:45:230
22772680,2sleep012-21ksoftirqd/010:20:320
181732620,4sleep218172-21ssh12:35:302
251222560,6sleep0141rcu_preempt08:45:010
281992550,3sleep30-21swapper/307:35:333
88952540,2sleep00-21swapper/012:27:210
171102520,3sleep012-21ksoftirqd/007:10:260
206392490,2sleep00-21swapper/010:05:230
206392490,2sleep00-21swapper/010:05:230
42512480,1sleep00-21swapper/011:55:190
301272480,3sleep00-21swapper/011:20:310
213062460,1sleep30-21swapper/310:05:303
213062460,1sleep30-21swapper/310:05:303
166582460,1sleep10-21swapper/108:25:231
255532450,1sleep00-21swapper/008:45:200
94822430,3chrt0-21swapper/112:28:061
1636799437,3cyclictest3340-21diskmemload09:14:581
61262410,2chrt0-21swapper/011:28:180
1636999417,11cyclictest2622-21/usr/sbin/munin09:45:273
47392400,1sleep20-21swapper/209:47:512
1636999399,11cyclictest16562-21/usr/sbin/munin12:05:303
1636799383,8cyclictest29036-21chrt10:15:191
293072370,1sleep20-21swapper/210:15:202
1636999378,11cyclictest20087-21/usr/sbin/munin11:10:363
1636999377,9cyclictest3340-21diskmemload10:50:323
1636999358,9cyclictest24482-21rm10:44:133
1636999358,10cyclictest10577-21ls12:00:283
1636999357,10cyclictest3340-21diskmemload12:32:483
90242340,3sleep00-21swapper/011:30:270
1636999349,8cyclictest19177-21chrt10:38:533
1636999348,11cyclictest5576-21chrt12:25:033
1636999348,10cyclictest17783-21sed07:10:343
1636999346,11cyclictest25411-21cpuspeed_turbos08:45:203
16369993410,10cyclictest13170-21ssh11:34:453
1636999339,10cyclictest14419-21sh09:58:443
1636999338,9cyclictest1120-21sh10:20:213
1636999338,8cyclictest4719-21sh10:55:433
1636999338,8cyclictest4719-21sh10:55:433
1636999338,16cyclictest1429-21sh09:44:473
1636999338,12cyclictest1855-21ssh11:52:483
1636999337,9cyclictest11727-21grep10:30:293
1636999337,17cyclictest26030-21ssh11:16:463
1636999336,11cyclictest9172-21cut09:15:303
16369993310,8cyclictest22254-21sh12:39:583
16369993310,8cyclictest1370-21/usr/sbin/munin12:20:303
16367993311,19cyclictest4360-21expr10:55:341
16367993311,19cyclictest4360-21expr10:55:341
1636999329,8cyclictest3615-21python311:25:313
1636999329,8cyclictest28050-21ssh11:47:023
1636999327,8cyclictest12540-21rm11:04:533
1636999326,9cyclictest2676-21sed07:50:333
1636999326,17cyclictest19898-21sh09:28:143
1636999326,11cyclictest28748-21sh09:37:363
16367993213,16cyclictest10293-21sh11:01:301
1636999318,9cyclictest8032-21ssh10:26:423
1636999315,9cyclictest6672-21ls09:50:223
1636799318,5cyclictest25067-21ls08:45:011
1636799318,4cyclictest2501-21expr11:25:191
1636799317,6cyclictest25160-21cat10:45:001
1636799316,7cyclictest13712-21tune2fs11:05:191
1636799312,7cyclictest5776-21apt-get10:25:171
16367993111,17cyclictest19794-21sh09:28:011
16367993110,6cyclictest7556-21seq08:04:001
1636999308,9cyclictest30747-21sh10:15:453
1636999308,9cyclictest21802-21/usr/sbin/munin07:20:313
1636999308,8cyclictest22903-21chrt11:42:413
1636999306,9cyclictest23669-21ssh09:31:213
1636999306,9cyclictest14266-21ssh09:21:063
1636999305,8cyclictest6807-21ssh09:13:363
1636799309,5cyclictest661-21gdbus09:15:311
1636799307,5cyclictest22761-21grep09:30:311
16367993012,15cyclictest13209-21sh12:02:521
16367993010,5cyclictest21181-21grep11:40:311
16367993010,4cyclictest1320-21latency_hist07:50:011
1636999297,8cyclictest8489-21ssh11:59:043
1636999297,8cyclictest25974-21perl10:45:253
1636999294,8cyclictest0-21swapper/308:10:263
1636799298,5cyclictest2389-21forks09:45:251
1636799298,5cyclictest16444-21seq11:36:411
1636799296,5cyclictest20328-21expr10:05:201
1636799296,5cyclictest20328-21expr10:05:201
16367992911,15cyclictest28349-21sh10:47:251
246152280,2chrt0-21swapper/209:33:102
1636999287,8cyclictest31733-21apt-get09:00:183
1636999287,8cyclictest17773-21ssh11:38:223
1636999286,8cyclictest534-21ssh11:23:253
1636999286,14cyclictest16466-21sh11:07:463
1636999281,17cyclictest0-21swapper/310:01:273
1636799289,4cyclictest23012-21sh12:10:421
1636799289,3cyclictest7812-21sh11:58:111
1636799289,3cyclictest13163-21head12:30:371
1636799288,4cyclictest18849-1kworker/u9:012:36:131
1636799288,3cyclictest29169-1kworker/u9:009:39:361
1636799287,4cyclictest13015-21uname09:20:251
1636799285,7cyclictest601-21dbus-daemon07:20:011
1636799285,5cyclictest21588-21sed08:35:261
1636799281,9cyclictest30844-21rm11:50:181
1636799281,9cyclictest1109-21apt-get10:20:191
1636799281,8cyclictest31900-21grep09:40:351
1636799281,8cyclictest26245-21sed10:10:311
16367992813,13cyclictest32644-21ntp_offset09:00:281
16367992812,14cyclictest8623-21sh09:51:301
16367992810,5cyclictest14831-21seq10:34:561
16367992810,3cyclictest27370-21sh11:46:111
153922280,1sleep20-21swapper/212:33:242
107972280,2chrt0-21swapper/009:55:030
1636999276,9cyclictest25654-21ssh10:10:233
1636999274,11cyclictest26231-21diskmemload12:14:483
1636799279,3cyclictest20400-21sh11:11:011
1636799279,3cyclictest17397-21pmu-power07:10:291
1636799278,3cyclictest2547-21latency_hist10:55:001
1636799277,5cyclictest28823-21seq11:20:021
1636799277,3cyclictest3340-21diskmemload10:35:181
1636799275,5cyclictest8791-21head11:30:241
1636799275,4cyclictest30114-21grep07:40:251
1636799273,5cyclictest285-21systemd-journal08:20:001
1636799272,7cyclictest25299-21apt-get08:45:191
1636799271,8cyclictest1653-21cstates07:50:201
1636799271,7cyclictest19705-21dump-pmu-power10:05:001
16367992710,5cyclictest22-21ksoftirqd/112:21:471
1636799270,8cyclictest24743-21cron07:30:001
16366992715,7cyclictest51650irq/140-enp3s0-11:05:150
130662270,1sleep20-21swapper/211:34:382
1636999266,9cyclictest28230-21awk12:15:333
1636799268,4cyclictest23679-21tr11:15:201
1636799268,3cyclictest8916-21tr08:05:341
1636799265,5cyclictest27593-21cat07:35:261
1636799262,8cyclictest25234-21apt-get07:30:261
1636799261,8cyclictest29622-21/usr/sbin/munin08:55:201
1636799261,8cyclictest28508-21sed12:15:351
1636799261,8cyclictest15886-21grep12:05:231
1636799261,8cyclictest11141-21/usr/sbin/munin09:55:191
1636799261,7cyclictest3488-21sed09:10:001
16367992612,12cyclictest14325-21/usr/sbin/munin08:20:351
1636899251,13cyclictest20725-21awk11:40:282
1636799257,4cyclictest11051-21proc_pri08:10:311
1636799255,4cyclictest21974-21timerandwakeup07:20:321
1636799252,4cyclictest36232sleep107:55:221
1636699253,15cyclictest20403-21ssh10:40:180
1636699251,8cyclictest16168-21ls10:35:270
140242250,2sleep30-21swapper/308:19:403
1636999249,12cyclictest24705-21chrt08:40:543
1636999245,7cyclictest649-21snmpd07:47:123
16369992410,11cyclictest27769-21/usr/sbin/munin08:50:193
1636799241,8cyclictest20142-21/usr/sbin/munin08:30:351
1636799241,8cyclictest20142-21/usr/sbin/munin08:30:351
1636699249,4cyclictest1755350lldpd09:54:400
1636699247,4cyclictest141rcu_preempt08:15:000
16366992414,7cyclictest1782-21df_inode09:05:250
27602230,1sleep00-21swapper/009:45:280
16369992312,9cyclictest30821-21sed07:40:353
16369992311,9cyclictest15444-21/usr/sbin/munin08:20:313
1636899232,9cyclictest10967-21ssh09:55:192
1636899231,12cyclictest15357-21apt-get12:05:192
1636799230,8cyclictest27618-21expr08:50:181
1636999228,10cyclictest0-21swapper/308:30:203
1636999228,10cyclictest0-21swapper/308:30:193
16369992212,8cyclictest4379-21/usr/sbin/munin07:55:303
16369992212,8cyclictest22868-21/usr/sbin/munin07:25:223
16369992210,9cyclictest6326-21cut08:00:263
16369992210,9cyclictest29581-21grep08:55:193
16369992210,9cyclictest21204-21tune2fs08:35:213
1636899223,17cyclictest32104-21awk12:20:002
16368992213,6cyclictest329-21cat12:20:212
16368992213,6cyclictest27288-21sensors_temp09:35:332
16368992212,7cyclictest21053-21df07:20:222
1636699221,7cyclictest1859-21ssh12:20:340
16366992216,4cyclictest12-21ksoftirqd/011:40:000
16366992213,7cyclictest18981-21cut07:15:270
16369992111,8cyclictest25584-21/usr/sbin/munin07:30:303
16369992111,8cyclictest1887-21tune2fs09:05:273
16369992111,8cyclictest18743-21cstates07:15:253
1636899212,16cyclictest2835-21grep09:05:372
16368992114,5cyclictest21302-21ls10:40:262
16368992113,6cyclictest8009-21expr08:05:232
16368992112,7cyclictest14433-21sort08:20:222
1636899211,10cyclictest0-21swapper/212:12:262
1636699217,2cyclictest12-21ksoftirqd/010:48:230
1636699214,14cyclictest7640-21cron08:05:010
16366992114,5cyclictest22762-21seq11:42:310
16366992114,4cyclictest29344-21sh12:16:320
16366992114,4cyclictest21429-21sh11:12:340
16366992113,6cyclictest27366-21cat09:35:340
16366992113,6cyclictest21697-21ntp_states08:35:280
16366992113,6cyclictest20716-21find07:20:190
16366992112,6cyclictest2873-21grep07:50:350
16366992111,8cyclictest15979-21seq08:22:010
16369992012,6cyclictest16500-21apt-get08:25:213
16369992010,8cyclictest8988-21ps08:05:343
1636899208,2cyclictest28-21ksoftirqd/210:10:252
16368992015,3cyclictest51950irq/143-enp3s0-09:13:332
16368992013,5cyclictest6697-21ntpq08:00:322
16368992013,5cyclictest18850irq/138-ahci[0007:35:382
16368992013,5cyclictest13073-21ntpq08:15:282
16368992012,6cyclictest4194-21cat07:55:252
16368992012,6cyclictest19156-21sed11:10:272
16368992012,6cyclictest18821-21expr08:30:212
16368992012,6cyclictest18821-21expr08:30:212
16368992012,6cyclictest17423-21ntp_states08:25:322
16368992012,6cyclictest16014-21tune2fs10:00:222
16368992012,6cyclictest15179-21python311:35:332
16368992011,7cyclictest601-21dbus-daemon12:00:002
1636899201,16cyclictest6277-21ssh12:25:212
16368992011,6cyclictest51950irq/143-enp3s0-11:50:192
1636899201,13cyclictest25178-21apt-get07:30:032
16368992010,7cyclictest285-21systemd-journal10:35:012
1636699209,9cyclictest51650irq/140-enp3s0-09:44:180
1636699209,2cyclictest12-21ksoftirqd/009:33:410
1636699208,2cyclictest12-21ksoftirqd/010:04:340
1636699205,3cyclictest141rcu_preempt09:15:340
1636699202,15cyclictest32767-21pmu-power07:45:360
16366992014,4cyclictest15753-21sh12:33:540
16366992014,4cyclictest15369-21sh09:23:220
16366992012,6cyclictest8106-21cat11:00:190
16366992012,6cyclictest30147-21sed08:55:230
16366992012,6cyclictest23566-21apt11:15:180
16366992012,6cyclictest11761-21awk10:30:310
16366992012,5cyclictest9211-21cat08:05:370
1636699201,13cyclictest27434-21apt-get08:50:160
1636899191,5cyclictest0-21swapper/210:21:252
16368991914,3cyclictest29309-21sh08:55:012
16368991913,3cyclictest23841-21sh10:09:122
16368991913,3cyclictest23841-21sh10:09:112
16368991912,5cyclictest5786-21sh10:57:222
16368991912,5cyclictest5786-21sh10:57:222
16368991912,5cyclictest2363-21ls11:25:182
16368991911,6cyclictest32698-21sh07:45:342
16368991911,6cyclictest25772-21expr11:45:212
16368991911,6cyclictest23806-21sh07:25:342
16368991910,7cyclictest18549-21basename07:15:222
1636699195,5cyclictest12-21ksoftirqd/010:25:510
16366991914,3cyclictest28380-21sh10:14:420
16366991914,3cyclictest17675-21switchtime08:25:350
16366991914,3cyclictest1611-21sh10:53:390
16366991913,4cyclictest16751-21sh09:25:130
16366991912,5cyclictest9981-21ls12:00:220
16366991912,5cyclictest29384-21diskstats10:15:210
16366991912,5cyclictest26650-21grep07:30:420
16366991912,5cyclictest19467-21needreboot08:30:270
16366991912,5cyclictest19467-21needreboot08:30:270
16366991912,4cyclictest5466-21sh10:56:510
16366991912,4cyclictest5466-21sh10:56:500
16366991911,6cyclictest15909-21grep12:05:230
16366991910,6cyclictest4627-21sed09:10:270
16366991910,6cyclictest29975-21forks07:40:250
16366991910,6cyclictest21873-21hwlatdetect12:10:250
1636899183,3cyclictest59562cstates10:25:182
1636899183,3cyclictest141rcu_preempt10:45:272
1636899183,12cyclictest8586-21fschecks_time09:15:232
1636899181,4cyclictest0-21swapper/211:20:222
1636899181,4cyclictest0-21swapper/211:06:122
16368991813,3cyclictest11932-21sh11:03:572
16368991813,3cyclictest11164-21ls12:00:342
16368991812,4cyclictest28-21ksoftirqd/209:00:012
16368991811,5cyclictest7256-21date09:50:272
16368991811,5cyclictest16111-21ls10:35:262
16368991811,5cyclictest10268-21cat08:10:212
16368991811,4cyclictest20583-21sh09:29:242
16368991810,6cyclictest30878-21wc07:40:352
16368991810,6cyclictest30619-21cat09:40:212
16368991810,6cyclictest30534-21expr10:50:192
16368991810,6cyclictest16816-21tune2fs07:10:242
16368991810,6cyclictest12573-21ls09:20:202
1636699187,4cyclictest51650irq/140-enp3s0-12:37:360
1636699182,14cyclictest31566-21uname11:50:250
16366991811,5cyclictest27759-21cut07:35:280
16366991810,6cyclictest32615-21ntp_kernel_err09:00:280
16366991810,6cyclictest12774-21basename08:15:260
1636899179,6cyclictest22548-21seq08:38:472
16368991712,3cyclictest1987-21gltestperf07:50:252
16368991711,4cyclictest23457-21sh11:15:122
16368991710,5cyclictest25860-21which08:45:242
16368991710,5cyclictest23214-21ls08:40:212
1636699179,5cyclictest4828-21switchtime07:55:360
1636699171,13cyclictest23421-21tail07:25:260
16368991611,3cyclictest32110-21fschecks_count09:00:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional