You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-03 - 01:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Dec 02, 2023 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29512242167,25sleep30-21swapper/307:06:293
30582232162,57sleep20-21swapper/207:07:532
29372203164,26sleep10-21swapper/107:06:201
29162201164,24sleep00-21swapper/007:06:020
2024821830,3sleep3333399cyclictest11:35:003
119632600,2sleep00-21swapper/008:40:240
113772530,3sleep230-21rcuc/208:40:012
113772530,3sleep230-21rcuc/208:40:012
9572380,23sleep0961-21latency_hist09:30:020
3333993827,3cyclictest0-21swapper/308:43:073
3333993826,3cyclictest0-21swapper/308:51:073
3333993725,3cyclictest0-21swapper/309:04:113
3333993713,9cyclictest0-21swapper/307:18:283
3333993421,3cyclictest0-21swapper/307:53:533
3333993414,9cyclictest0-21swapper/307:59:313
3333993414,8cyclictest0-21swapper/308:58:573
3333993325,3cyclictest0-21swapper/308:16:073
3333993321,3cyclictest0-21swapper/309:07:153
333399331,12cyclictest5585-21cpuspeed_turbos09:40:153
333399329,10cyclictest0-21swapper/309:25:313
3333993219,8cyclictest0-21swapper/308:24:253
333399321,13cyclictest19885-21gltestperf10:15:233
333399318,3cyclictest0-21swapper/311:29:433
333399311,15cyclictest29605-21ls10:40:233
331799314,13cyclictest25951-21munin-run08:00:010
3333993017,8cyclictest0-21swapper/308:48:113
3333992918,3cyclictest0-21swapper/310:28:333
3333992916,8cyclictest0-21swapper/307:32:113
3333992915,9cyclictest0-21swapper/308:30:073
333399291,13cyclictest0-21swapper/308:10:313
332499291,12cyclictest0-21swapper/109:43:171
333399281,11cyclictest0-21swapper/311:52:293
333399281,11cyclictest0-21swapper/311:52:293
333399281,10cyclictest0-21swapper/312:31:333
332499289,8cyclictest16312-21kworker/u8:008:25:241
331799282,14cyclictest32555-21munin-run08:15:020
333399279,8cyclictest0-21swapper/308:37:533
333399279,8cyclictest0-21swapper/308:37:533
333399278,9cyclictest0-21swapper/312:24:353
3333992717,3cyclictest0-21swapper/310:35:233
3333992714,8cyclictest0-21swapper/308:02:293
3333992714,8cyclictest0-21swapper/307:39:293
3333992713,3cyclictest0-21swapper/312:13:033
3333992712,10cyclictest0-21swapper/307:20:173
333399271,18cyclictest7972-21ntpq12:25:273
3333992710,13cyclictest3490-21cstates09:35:233
331799272,13cyclictest21559-21munin-run07:50:010
331799270,12cyclictest5200-21idleruntime-cro09:40:020
333399266,8cyclictest0-21swapper/307:27:053
333399262,9cyclictest0-21swapper/312:15:513
3333992615,3cyclictest0-21swapper/307:42:173
3333992611,3cyclictest12903-21unixbench_singl12:35:413
333399261,10cyclictest0-21swapper/310:46:533
3333992610,4cyclictest0-21swapper/308:07:273
332499261,5cyclictest0-21swapper/109:46:531
332499261,17cyclictest0-21swapper/107:55:221
332499260,12cyclictest0-21swapper/109:18:291
331799261,12cyclictest19372-21awk07:45:010
331799261,12cyclictest18045-21munin-run11:30:010
333399258,8cyclictest0-21swapper/308:25:533
333399251,13cyclictest0-21swapper/309:45:293
333399251,12cyclictest0-21swapper/309:57:213
333399251,10cyclictest0-21swapper/307:10:053
3333992510,3cyclictest24372-21diskmemload11:56:133
3328992521,3cyclictest0-21swapper/211:38:472
332499250,12cyclictest0-21swapper/112:22:531
331799257,15cyclictest25622-21basename10:30:010
331799251,12cyclictest8901-21munin-run12:30:010
333399247,2cyclictest0-21swapper/310:55:393
333399241,14cyclictest0-21swapper/311:45:213
333399240,11cyclictest0-21swapper/311:05:553
333399240,11cyclictest0-21swapper/310:10:213
333399240,11cyclictest0-21swapper/309:54:573
3324992413,5cyclictest0-21swapper/111:30:081
332499241,19cyclictest823-21snmpd12:35:121
332499241,13cyclictest0-21swapper/111:50:331
332499241,13cyclictest0-21swapper/111:50:331
3324992411,11cyclictest23360-21kworker/u8:110:40:211
331799243,8cyclictest27928-40dnf11:50:320
331799243,8cyclictest27928-40dnf11:50:320
333399235,3cyclictest0-21swapper/311:00:393
333399231,14cyclictest0-21swapper/309:33:433
333399231,12cyclictest0-21swapper/310:53:493
333399231,12cyclictest0-21swapper/309:18:533
333399231,11cyclictest0-21swapper/311:11:413
3333992310,3cyclictest0-21swapper/311:18:473
333399230,9cyclictest0-21swapper/311:43:233
333399230,9cyclictest0-21swapper/309:20:213
332499231,3cyclictest27387-21tail10:30:421
332499231,17cyclictest0-21swapper/108:09:321
332499231,16cyclictest0-21swapper/110:55:071
332499231,16cyclictest0-21swapper/108:21:431
332499231,15cyclictest86850irq/129-enp1s0-09:13:201
331799234,16cyclictest24380-21munin-run09:10:020
214222230,4sleep021424-21missed_timers11:35:300
333399229,8cyclictest0-21swapper/307:48:473
333399228,3cyclictest0-21swapper/309:11:173
333399221,13cyclictest24372-21diskmemload10:05:593
333399220,8cyclictest0-21swapper/312:03:153
333399220,14cyclictest0-21swapper/310:04:333
332899221,9cyclictest0-21swapper/211:55:362
332499222,10cyclictest4510-21munin-run12:20:011
3324992217,2cyclictest8813-21chrt08:33:021
332499221,5cyclictest0-21swapper/111:48:281
3324992215,2cyclictest0-21swapper/108:02:231
3324992212,3cyclictest0-21swapper/109:35:301
332499221,15cyclictest0-21swapper/109:25:031
332499221,15cyclictest0-21swapper/108:57:391
332499221,15cyclictest0-21swapper/107:11:211
332499221,12cyclictest0-21swapper/111:57:231
332499220,4cyclictest0-21swapper/109:53:201
332499220,17cyclictest86750irq/128-enp1s0-12:28:031
332499220,16cyclictest0-21swapper/107:15:211
331799223,10cyclictest28648-21/usr/sbin/munin10:40:180
333399217,3cyclictest0-21swapper/310:33:253
333399216,3cyclictest715-21ntp_states12:05:313
333399215,6cyclictest0-21swapper/311:22:183
333399214,3cyclictest0-21swapper/311:36:333
333399212,2cyclictest86850irq/129-enp1s0-10:22:473
332899211,8cyclictest0-21swapper/207:19:052
332499218,4cyclictest0-21swapper/112:00:251
332499212,11cyclictest0-21swapper/112:10:191
3324992116,3cyclictest0-21swapper/110:10:321
3324992113,3cyclictest0-21swapper/107:37:351
332499211,2cyclictest0-21swapper/108:50:151
332499211,18cyclictest0-21swapper/110:00:071
332499211,15cyclictest0-21swapper/108:39:221
332499211,15cyclictest0-21swapper/108:39:221
332499211,15cyclictest0-21swapper/108:15:141
332499211,15cyclictest0-21swapper/108:14:561
332499211,15cyclictest0-21swapper/107:34:041
332499211,13cyclictest0-21swapper/110:37:161
332499211,11cyclictest8952-21df_inode07:20:221
332499211,10cyclictest0-21swapper/109:20:311
331799215,6cyclictest780-21rngd09:50:370
331799213,16cyclictest28159-21munin-run08:05:010
332899204,3cyclictest28755-21kworker/u8:112:33:262
332899202,5cyclictest0-21swapper/209:30:382
332899201,7cyclictest0-21swapper/209:18:212
332899201,10cyclictest0-21swapper/211:32:092
332499208,3cyclictest0-21swapper/110:51:001
332499206,7cyclictest780-21rngd07:41:151
332499202,10cyclictest0-21swapper/109:59:251
3324992014,1cyclictest0-21swapper/107:52:191
3324992013,2cyclictest0-21swapper/111:43:101
3324992012,3cyclictest0-21swapper/107:47:351
332499201,17cyclictest0-21swapper/112:05:241
332499201,17cyclictest0-21swapper/109:01:301
332499201,16cyclictest0-21swapper/111:06:141
332499201,11cyclictest0-21swapper/111:27:441
332499201,11cyclictest0-21swapper/109:30:361
332499201,10cyclictest0-21swapper/112:30:311
332499201,10cyclictest0-21swapper/110:21:451
331799201,9cyclictest0-21swapper/010:04:560
3317992013,4cyclictest28763-21dump-pmu-power09:20:020
331799201,16cyclictest15589-21awk08:50:020
288262200,4sleep2332899cyclictest10:40:162
118672201,3sleep211868-21ntp_states07:25:282
332899192,15cyclictest3414-21tr09:35:232
332899192,14cyclictest27330-21sed10:30:422
3328991914,3cyclictest18638-21fgrep07:40:322
3328991913,3cyclictest14683-21ntp_states08:45:302
332899191,12cyclictest0-21swapper/209:13:152
332899191,12cyclictest0-21swapper/208:15:382
332499192,11cyclictest8906-21kworker/u8:211:20:341
332499191,9cyclictest10742-21find11:10:391
3324991915,3cyclictest0-21swapper/110:05:111
332499191,4cyclictest12736-21ntp_kernel_pll_11:15:301
332499191,4cyclictest11150-21diskstats07:25:211
3324991912,2cyclictest0-21swapper/109:09:531
332499191,16cyclictest0-21swapper/110:25:071
332499191,16cyclictest0-21swapper/108:45:101
332499191,10cyclictest19454-21cat10:15:191
332499191,10cyclictest0-21swapper/111:04:301
332499190,6cyclictest0-21swapper/111:39:561
332499190,15cyclictest0-21swapper/108:42:501
331799197,3cyclictest22486-1kworker/u9:211:10:380
331799193,3cyclictest14967-21awk07:35:010
331799193,12cyclictest1011-21grep08:15:270
331799192,15cyclictest2649-21munin-run10:55:010
331799191,9cyclictest0-21swapper/011:20:250
331799191,7cyclictest7981-21df_inode09:45:210
3317991913,3cyclictest4638-21seq10:57:280
331799191,10cyclictest2623-21/usr/sbin/munin08:20:260
331799191,10cyclictest18277-21/usr/sbin/munin11:30:350
331799191,10cyclictest0-21swapper/010:35:170
236172191,2sleep00-21swapper/007:53:280
91542181,2sleep00-21swapper/011:08:530
332899186,3cyclictest0-21swapper/208:20:152
332899181,5cyclictest10912-21ntp_states09:50:352
332899181,5cyclictest0-21swapper/211:08:312
332899181,5cyclictest0-21swapper/209:59:182
332899181,2cyclictest86750irq/128-enp1s0-08:50:302
3328991812,3cyclictest6065-21find08:25:332
3328991812,3cyclictest5298-21cut12:20:182
332899181,15cyclictest13388-21sshd10:00:202
3328991811,4cyclictest31405-21tail12:00:282
3328991811,4cyclictest29592-21fw_packets09:20:242
3328991811,4cyclictest24518-21/usr/sbin/munin07:55:242
332899181,11cyclictest0-21swapper/209:48:522
332899181,10cyclictest0-21swapper/210:51:102
332499181,16cyclictest0-21swapper/110:46:521
331799181,7cyclictest0-21swapper/011:02:280
331799181,7cyclictest0-21swapper/009:56:100
3317991813,3cyclictest32560-21cut12:05:220
3317991812,3cyclictest16881-21ntpq08:50:290
3317991811,4cyclictest22103-21gltestperf10:20:270
218172181,2sleep00-21swapper/009:00:440
211842181,2sleep00-21swapper/010:19:570
108142181,2sleep00-21swapper/012:31:300
53862171,2sleep00-21swapper/007:13:580
332899173,6cyclictest19924-21taskset08:59:402
332899171,7cyclictest24721-21ntp_states10:25:372
332899171,7cyclictest0-21swapper/211:48:552
332899171,7cyclictest0-21swapper/211:11:482
332899171,7cyclictest0-21swapper/210:38:122
332899171,7cyclictest0-21swapper/210:10:372
332899171,6cyclictest0-21swapper/210:05:212
332899171,5cyclictest0-21swapper/211:29:272
332899171,4cyclictest0-21swapper/211:40:452
332899171,4cyclictest0-21swapper/211:00:182
3328991713,3cyclictest0-21swapper/211:17:352
3328991713,3cyclictest0-21swapper/207:24:302
332899171,2cyclictest6495-21ntpq09:40:252
3328991712,3cyclictest1775-21sed12:10:022
3328991712,3cyclictest13236-21cat07:30:212
3328991712,3cyclictest12365-21missed_timers08:40:272
332899171,14cyclictest0-21swapper/210:45:022
3328991711,3cyclictest3530-21users12:10:402
331799176,3cyclictest86550irq/126-enp1s0-12:10:300
331799176,2cyclictest24610-21/usr/sbin/munin09:10:320
331799172,6cyclictest0-21swapper/009:22:090
331799171,9cyclictest0-21swapper/010:10:220
331799171,7cyclictest0-21swapper/009:30:380
331799171,6cyclictest0-21swapper/011:40:110
331799171,6cyclictest0-21swapper/011:15:540
331799171,6cyclictest0-21swapper/008:05:220
331799171,4cyclictest0-21swapper/009:43:100
331799171,3cyclictest0-21swapper/012:21:300
331799171,3cyclictest0-21swapper/008:35:230
331799171,3cyclictest0-21swapper/008:35:230
3317991712,4cyclictest16312-21kworker/u8:008:27:310
3317991712,3cyclictest30847-21/usr/sbin/munin10:45:360
331799171,15cyclictest0-21swapper/007:16:530
3317991711,3cyclictest29250-21seq11:55:080
3317991711,3cyclictest16437-21uptime10:05:360
331799171,10cyclictest0-21swapper/010:32:560
331799171,10cyclictest0-21swapper/008:30:220
317122171,2sleep00-21swapper/012:03:240
332899164,3cyclictest26920-21if_enp1s008:00:232
332899163,10cyclictest23448-21ntpq09:05:292
332899161,6cyclictest0-21swapper/212:25:262
332899161,6cyclictest0-21swapper/211:20:312
332899161,6cyclictest0-21swapper/209:28:572
3328991612,3cyclictest0-21swapper/212:16:542
3328991612,3cyclictest0-21swapper/210:21:052
3328991612,3cyclictest0-21swapper/207:10:102
3328991611,3cyclictest7954-21sed08:30:262
3328991611,3cyclictest3530-21tail10:55:272
3328991611,3cyclictest29880-21unixbench_multi08:05:342
3328991611,3cyclictest27308-21expr11:50:242
3328991611,3cyclictest27308-21expr11:50:242
3328991611,3cyclictest21180-21sed07:45:402
3328991611,3cyclictest20526-21expr09:00:222
3328991611,3cyclictest20025-21if_err_enp1s010:15:252
3328991610,3cyclictest31066-21fschecks_time08:10:262
332899160,15cyclictest0-21swapper/212:37:542
331799161,9cyclictest0-21swapper/012:35:350
331799161,6cyclictest0-21swapper/011:45:080
3317991612,3cyclictest0-21swapper/008:57:370
331799161,12cyclictest8609-21/usr/sbin/munin07:20:380
3317991611,2cyclictest814-21sshd12:17:200
125762160,2sleep00-21swapper/007:27:480
332899155,3cyclictest86850irq/129-enp1s0-07:35:212
332899152,6cyclictest0-21swapper/207:50:132
331799150,9cyclictest0-21swapper/007:35:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional