You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-04 - 15:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Mon Dec 04, 2023 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
101392184101,76sleep00-21swapper/007:06:250
103122178101,27sleep20-21swapper/207:08:412
101872176103,30sleep10-21swapper/107:07:031
102712174102,30sleep30-21swapper/307:08:093
1515521110,1sleep00-21swapper/009:30:220
114162810,4sleep10-21swapper/110:41:311
94632730,2sleep10-21swapper/112:25:211
43962700,8sleep228-21ksoftirqd/211:49:232
78882670,2chrt0-21swapper/212:23:572
22512590,2sleep20-21swapper/210:30:292
144182540,1sleep30-21swapper/309:29:573
232092520,4sleep00-21swapper/008:50:330
198542520,2sleep10-21swapper/108:45:251
145142520,1sleep00-21swapper/012:00:150
5692490,1sleep10-21swapper/109:51:221
141572470,1sleep10-21swapper/110:46:001
293502460,2sleep00-21swapper/009:45:470
293502460,2sleep00-21swapper/009:45:470
151022440,2chrt0-21swapper/307:20:243
151022440,2chrt0-21swapper/307:20:243
121132440,1sleep20-21swapper/212:27:192
36002340,1chrt0-21swapper/211:13:222
177512340,2sleep20-21swapper/207:25:242
10690993321,6cyclictest19931-21sh12:05:042
10691993113,9cyclictest34-21ksoftirqd/310:11:103
10691993113,8cyclictest34-21ksoftirqd/311:49:293
10691993111,9cyclictest34-21ksoftirqd/309:30:403
70182300,1sleep20-21swapper/208:15:282
10691993013,8cyclictest34-21ksoftirqd/310:07:463
10691993012,9cyclictest34-21ksoftirqd/311:35:443
10691993011,9cyclictest34-21ksoftirqd/310:21:293
10691993010,11cyclictest34-21ksoftirqd/309:51:353
1069199293,7cyclictest34-21ksoftirqd/309:55:153
10691992911,9cyclictest34-21ksoftirqd/312:12:403
10691992911,9cyclictest34-21ksoftirqd/310:17:233
10691992910,9cyclictest34-21ksoftirqd/312:00:453
1069199288,11cyclictest34-21ksoftirqd/309:00:083
10691992812,8cyclictest34-21ksoftirqd/311:51:233
10691992811,9cyclictest34-21ksoftirqd/311:19:593
10691992810,8cyclictest34-21ksoftirqd/309:46:523
10691992810,8cyclictest34-21ksoftirqd/309:46:513
10690992814,9cyclictest631-21ntpd11:15:232
22442270,2sleep20-21swapper/209:54:042
1069199278,9cyclictest34-21ksoftirqd/311:57:393
1069199278,8cyclictest34-21ksoftirqd/312:08:573
10691992711,9cyclictest34-21ksoftirqd/308:41:223
10691992711,9cyclictest34-21ksoftirqd/308:01:083
10691992711,8cyclictest34-21ksoftirqd/311:26:423
10691992711,8cyclictest34-21ksoftirqd/310:27:083
10691992711,8cyclictest34-21ksoftirqd/309:44:413
10691992711,10cyclictest34-21ksoftirqd/309:19:583
170662260,2chrt0-21swapper/212:01:422
1069199269,10cyclictest34-21ksoftirqd/311:00:543
1069199267,10cyclictest34-21ksoftirqd/310:00:103
1069199264,8cyclictest34-21ksoftirqd/310:30:243
1069199262,5cyclictest34-21ksoftirqd/311:33:313
1069199261,8cyclictest34-21ksoftirqd/310:42:393
10691992611,8cyclictest34-21ksoftirqd/312:36:383
1069199257,9cyclictest34-21ksoftirqd/311:20:143
1069199256,10cyclictest34-21ksoftirqd/310:56:363
1069199251,7cyclictest34-21ksoftirqd/312:26:303
1069199251,6cyclictest34-21ksoftirqd/312:19:003
1069199251,17cyclictest29672-21ssh11:05:143
1069199250,9cyclictest14246-21chrt10:46:103
1069199250,5cyclictest0-21swapper/310:53:373
244022240,2sleep20-21swapper/212:38:192
1069199249,3cyclictest141rcu_preempt10:36:503
1069199249,3cyclictest141rcu_preempt10:36:503
1069199248,8cyclictest34-21ksoftirqd/309:36:333
1069199241,8cyclictest34-21ksoftirqd/312:34:243
1069199241,7cyclictest34-21ksoftirqd/311:13:593
10691992417,5cyclictest34-21ksoftirqd/307:20:003
10691992413,9cyclictest601-21dbus-daemon12:25:003
1069099243,17cyclictest21150-21tr10:15:212
10689992415,5cyclictest25042sleep109:15:191
10689992414,7cyclictest285-21systemd-journal12:00:001
10688992414,7cyclictest26214-21df_inode07:45:250
228582230,2sleep00-21swapper/012:36:270
1069199234,9cyclictest34-21ksoftirqd/311:41:483
10691992319,3cyclictest331rcuc/309:11:313
10691992316,5cyclictest34-21ksoftirqd/308:20:003
10690992315,5cyclictest28-21ksoftirqd/207:45:002
1068999232,16cyclictest20171-21sh12:35:031
10688992313,7cyclictest20645-21fw_packets12:05:240
10688992312,9cyclictest13636-21sh10:06:370
34412220,10sleep122-21ksoftirqd/110:30:551
215442220,2sleep20-21swapper/209:38:472
205182220,2sleep00-21swapper/010:14:530
1069199226,8cyclictest0-21swapper/308:54:303
1069199226,8cyclictest0-21swapper/307:25:043
1069199224,6cyclictest34-21ksoftirqd/307:40:263
10691992217,3cyclictest331rcuc/308:10:213
1069199221,5cyclictest34-21ksoftirqd/307:10:193
1069099227,13cyclictest0-21swapper/209:25:002
1069099222,14cyclictest29133-21ssh10:25:142
10690992212,7cyclictest28349-21df_inode07:50:222
10690992210,8cyclictest8681-21apt-get08:20:192
1068999222,15cyclictest3815-21ssh12:20:211
10689992213,5cyclictest30146-21diskmemload11:43:291
1068999221,15cyclictest21604-21ssh10:55:191
10688992214,6cyclictest30021-21diskstats11:05:160
10688992214,6cyclictest16352-21seq10:49:440
10688992213,7cyclictest3838-21seq08:06:310
10688992213,6cyclictest25838-21df12:10:190
10688992212,6cyclictest51650irq/140-enp3s0-12:30:040
18982210,2sleep122-21ksoftirqd/111:45:521
1069199217,7cyclictest0-21swapper/308:33:063
1069199216,7cyclictest0-21swapper/309:06:493
1069199216,7cyclictest0-21swapper/308:24:193
1069199216,7cyclictest0-21swapper/307:30:383
1069199213,5cyclictest34-21ksoftirqd/309:20:283
1069199211,7cyclictest34-21ksoftirqd/307:35:293
10691992113,6cyclictest16553-21python308:35:413
10691992112,7cyclictest11277-21cut08:25:213
10691992112,6cyclictest27127-21cat07:45:463
1069199210,7cyclictest34-21ksoftirqd/308:55:283
10690992114,4cyclictest21518-21sh10:55:152
10690992113,6cyclictest31468-21ls07:55:332
10690992113,6cyclictest2709-21fw_packets08:05:232
1068999212,13cyclictest19932-21rm12:05:031
10689992111,7cyclictest285-21systemd-journal08:20:001
10689992111,7cyclictest23909-21date08:55:001
10689992110,7cyclictest17491-21cat10:10:261
1068899211,9cyclictest39292diskmemload10:31:500
10688992114,4cyclictest10205-21sh09:25:150
10688992113,6cyclictest6246-21seq10:35:360
10688992113,6cyclictest6246-21seq10:35:360
10688992113,6cyclictest21388-21diskstats10:15:230
10688992113,6cyclictest12844-21seq07:15:070
10688992112,6cyclictest23939-21needreboot11:35:230
10691992015,3cyclictest34-21ksoftirqd/307:55:003
10691992013,5cyclictest34-21ksoftirqd/308:05:013
10691992012,6cyclictest19979-21cat08:45:253
1069099205,3cyclictest9797-21if_err_enp3s011:55:202
1069099202,12cyclictest15387-21apt-get08:35:262
10690992013,5cyclictest17876-21open_files10:50:242
10690992013,4cyclictest26392-21sh11:00:052
10690992012,6cyclictest9204-21open_inodes10:00:322
10690992012,6cyclictest31055-21cat11:05:272
1068999202,13cyclictest8229-21apt-get10:00:211
1068999202,12cyclictest10235-21apt-get09:25:161
10689992013,4cyclictest31007-21sh10:25:491
10689992012,5cyclictest23483-21awk07:40:011
10689992011,6cyclictest601-21dbus-daemon09:34:591
10689992011,6cyclictest14465-21seq10:07:551
1068899201,8cyclictest1130-21cron10:30:000
1068899201,4cyclictest0-21swapper/010:53:100
10688992013,5cyclictest4806-21ntpq12:20:290
10688992013,5cyclictest19291-21seq08:43:320
10688992013,5cyclictest11723-21ntp_kernel_pll_08:25:260
10688992012,6cyclictest31991-21basename11:44:590
10688992012,6cyclictest25712-21seq08:58:320
10688992012,6cyclictest25025-21cat07:40:340
10688992012,6cyclictest20464-21seq09:36:480
10688992011,7cyclictest20997-21date08:45:380
10688992011,7cyclictest1283-21ls08:00:410
10688992011,6cyclictest9657-21fw_packets10:40:180
259912190,3chrt0-21swapper/212:10:212
1069099199,6cyclictest9188-21ssh10:40:142
1069099192,2cyclictest141rcu_preempt11:24:522
1069099191,6cyclictest3308-21sh12:20:012
10690991914,4cyclictest28-21ksoftirqd/211:45:002
10690991913,5cyclictest28-21ksoftirqd/209:20:002
10690991913,4cyclictest28-21ksoftirqd/212:35:002
10690991913,4cyclictest24687-21latency08:55:322
10690991913,4cyclictest16325-21sh10:49:432
10690991912,5cyclictest27969-21sh10:23:422
10690991912,5cyclictest13228-21expr08:30:252
10690991912,5cyclictest10618-21forks09:25:182
10690991911,6cyclictest24557-21grep11:35:302
10690991910,7cyclictest15320-21tune2fs09:30:242
1069099190,6cyclictest1287-21ssh09:13:202
1068999196,10cyclictest15127-21gltestperf12:00:241
10689991912,4cyclictest639-21runrttasks11:52:061
10689991912,4cyclictest28747-21grep09:05:251
10689991912,4cyclictest17730-21sed07:25:251
10689991911,6cyclictest21227-21basename07:35:001
10689991911,6cyclictest15511-21perl07:20:291
10689991911,6cyclictest15511-21perl07:20:281
10689991911,6cyclictest13675-21cat07:15:341
10689991911,5cyclictest30146-21diskmemload09:40:181
10689991911,4cyclictest30146-21diskmemload11:07:031
10689991911,4cyclictest30146-21diskmemload10:36:031
10689991911,4cyclictest30146-21diskmemload10:36:031
1068999191,14cyclictest22734-21cat11:35:001
10689991911,4cyclictest0-21swapper/109:35:031
1068999191,12cyclictest32316-21apt-get08:00:211
1068999191,12cyclictest14544-21ssh12:30:051
10689991910,6cyclictest18478-21grep10:50:281
10689991910,6cyclictest17511-21seq08:40:141
10689991910,6cyclictest12756-21wc07:15:011
1068899199,3cyclictest141rcu_preempt11:02:480
1068899198,3cyclictest96692latency_hist11:20:000
1068899198,3cyclictest141rcu_preempt11:45:200
1068899192,12cyclictest21581-21apt-get10:55:190
10688991914,3cyclictest9766-21sendmail_mailqu08:20:320
10688991913,4cyclictest9156-21ntp_offset10:00:290
10688991913,4cyclictest32651-21sh09:50:440
10688991913,4cyclictest11430-21sh11:22:040
10688991912,5cyclictest7356-21seq09:20:390
10688991912,5cyclictest21647-21expr07:35:210
10688991911,6cyclictest7655-21seq09:59:520
10688991911,6cyclictest3254-21cat09:15:260
10688991911,6cyclictest23460-21cat09:40:190
10688991911,6cyclictest13231-21expr08:30:260
1068899191,15cyclictest9086-21/usr/sbin/munin11:55:300
10688991910,7cyclictest6347-21cut08:15:000
10688991910,7cyclictest15676-21ls08:35:280
1069199182,5cyclictest0-21swapper/307:55:253
1069099188,7cyclictest11669-21sh07:10:312
10690991813,3cyclictest24209-21tail09:40:272
10690991812,4cyclictest15697-21sh11:25:372
10690991812,4cyclictest1413-21users08:00:422
10690991811,5cyclictest18238-21ls08:40:252
10690991811,5cyclictest14721-21sh10:08:212
10690991811,5cyclictest11110-21ls08:25:192
1069099181,12cyclictest576-21irqbalance08:50:202
1069099181,10cyclictest5791-21taskset11:51:022
10690991810,6cyclictest4521-21grep08:10:212
10690991810,6cyclictest29773-21vmstat09:05:362
10690991810,6cyclictest20911-21sed08:45:382
1068999189,6cyclictest31240-21kernelversion09:10:271
10689991812,3cyclictest8962-21sh11:19:031
10689991812,3cyclictest27044-21ntp_offset09:00:351
1068999181,15cyclictest32229-21awk12:15:331
10689991811,5cyclictest14145-21awk08:30:371
10689991811,5cyclictest13727-21sed11:25:001
10689991811,4cyclictest30146-21diskmemload11:00:401
10689991810,6cyclictest30898-21sed07:55:261
10689991810,6cyclictest26128-21basename12:10:241
10689991810,6cyclictest16083-21grep08:35:331
1068899187,9cyclictest0-21swapper/011:51:570
1068899182,12cyclictest6525-21apt-get08:15:240
1068899182,11cyclictest30156-21apt-get07:55:190
1068899181,8cyclictest19593-21rm11:30:500
10688991813,3cyclictest31223-21sh09:10:270
10688991813,3cyclictest2157-21sh11:10:510
10688991813,3cyclictest18504-21timerwakeupswit07:25:350
10688991813,3cyclictest10444-21switchtime12:25:320
10688991811,5cyclictest28565-21expr09:05:220
10688991811,5cyclictest15708-21uname07:20:300
10688991811,5cyclictest15708-21uname07:20:300
10688991810,6cyclictest31343-21perl12:15:240
10688991810,6cyclictest29116-21which07:50:310
10688991810,6cyclictest11074-21diskstats07:10:210
239142170,7sleep122-21ksoftirqd/107:40:201
1069099179,6cyclictest285-21systemd-journal07:36:292
1069099179,6cyclictest26721-21grep07:45:312
1069099179,6cyclictest14872-21sed07:20:002
1069099172,10cyclictest3203-21apt-get09:55:152
1069099172,10cyclictest26130-21apt-get09:00:262
1069099171,7cyclictest7763-21ssh10:38:112
1069099171,7cyclictest7763-21ssh10:38:102
10690991712,3cyclictest19357-21sh11:30:292
10690991712,3cyclictest17800-21sh10:10:302
10690991711,4cyclictest27399-21sh09:45:082
10690991711,4cyclictest27399-21sh09:45:082
10690991710,5cyclictest19585-21expr07:30:212
1068999179,6cyclictest4810-21cut08:10:241
1068999179,6cyclictest24539-21sed08:55:291
1068999172,12cyclictest75-21kswapd011:35:151
10689991713,2cyclictest1-21systemd09:23:441
10689991711,4cyclictest25467-21sendmail10:19:591
10689991710,5cyclictest14712-21sed11:25:201
10689991710,4cyclictest0-21swapper/110:00:001
1068899179,6cyclictest26221-21grep09:00:260
1068899179,6cyclictest15464-21sed11:25:280
1068899171,11cyclictest19326-21apt-get07:30:200
10690991610,4cyclictest16735-21seq07:22:462
10690991610,4cyclictest16735-21seq07:22:462
1068999169,5cyclictest4037-21seq08:09:091
1068999164,9cyclictest28161-21tr07:50:211
1068999162,8cyclictest10823-21apt-get08:25:171
1068999162,11cyclictest29079-21ssh09:45:381
1068999162,11cyclictest29079-21ssh09:45:381
10689991611,3cyclictest2418-21sh11:11:151
1068999161,11cyclictest25889-21taskset07:45:231
10689991610,4cyclictest8852-21cstates08:20:211
10689991610,4cyclictest30146-21diskmemload10:20:111
10688991512,2cyclictest25806-21ssh10:20:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional