You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-10 - 19:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Sun Dec 10, 2023 12:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1013499271,18cyclictest0-21swapper/209:58:252
10125992719,4cyclictest29010-21rm10:41:190
1012599270,22cyclictest0-21swapper/007:55:000
1013499260,21cyclictest0-21swapper/209:00:002
10125992622,4cyclictest3534-21diskmemload11:35:370
10125992620,6cyclictest0-21swapper/009:35:000
10125992620,5cyclictest62150irq/126-eno111:30:010
10125992620,5cyclictest10156-21apt-get10:23:270
1012599260,22cyclictest19616-21ssh11:04:070
10134992523,1cyclictest7395-21cron11:25:002
10134992518,3cyclictest0-21swapper/211:34:472
1013499250,24cyclictest3534-21diskmemload09:44:112
1013499250,24cyclictest0-21swapper/212:24:542
1013499250,1cyclictest16927-21sh10:28:412
1013499250,0cyclictest0-21swapper/210:53:442
1012999250,24cyclictest9775-21idleruntime-cro12:00:001
1012999250,24cyclictest29798-21sh11:14:411
1012999250,24cyclictest0-21swapper/111:00:111
10125992520,4cyclictest27151-21chrt11:13:090
10125992520,4cyclictest2450-21ssh10:14:100
10125992519,5cyclictest12686-21sh11:30:140
1012599250,24cyclictest0-21swapper/010:01:320
1012599250,0cyclictest3534-21diskmemload10:55:580
1012599250,0cyclictest0-21swapper/009:13:430
1013899240,24cyclictest3534-21diskmemload09:53:183
1013899240,24cyclictest0-21swapper/312:29:073
1013899240,24cyclictest0-21swapper/312:07:073
1013899240,24cyclictest0-21swapper/310:15:193
1013899240,23cyclictest3534-21diskmemload12:34:103
1013899240,23cyclictest3534-21diskmemload11:30:343
1013899240,23cyclictest3534-21diskmemload10:35:513
1013899240,23cyclictest28292-21switchtime12:18:343
10134992423,1cyclictest3534-21diskmemload12:33:482
10134992422,1cyclictest0-21swapper/209:48:192
10134992421,1cyclictest19766-21perf07:30:012
10134992419,4cyclictest5836-21ssh09:13:262
1013499240,23cyclictest0-21swapper/210:09:532
1013499240,21cyclictest0-21swapper/211:17:052
1013499240,20cyclictest0-21swapper/210:35:362
1013499240,1cyclictest13056-21apt-get09:53:242
1013499240,1cyclictest0-21swapper/209:26:272
1013499240,0cyclictest0-21swapper/210:41:582
10129992423,1cyclictest0-21swapper/109:44:141
10129992422,1cyclictest0-21swapper/112:25:581
10129992421,2cyclictest32301-21taskset11:49:551
1012999240,23cyclictest3534-21diskmemload12:21:551
1012999240,23cyclictest20596-21cron10:00:001
10125992423,1cyclictest28083-21expr10:08:270
10125992421,2cyclictest0-21swapper/009:52:310
10125992419,4cyclictest3534-21diskmemload12:15:170
10125992418,5cyclictest3534-21diskmemload11:22:230
1012599240,24cyclictest0-21swapper/012:13:530
1012599240,24cyclictest0-21swapper/009:57:380
1012599240,23cyclictest1130-21awk09:05:010
1012599240,21cyclictest0-21swapper/012:09:580
1012599240,20cyclictest4789-21rm12:28:270
1012599240,1cyclictest17719-21rm09:23:550
1012599240,0cyclictest3534-21diskmemload10:31:420
1012599240,0cyclictest0-21swapper/012:01:150
1012599240,0cyclictest0-21swapper/011:50:050
10138992323,0cyclictest3534-21diskmemload10:13:363
10138992323,0cyclictest0-21swapper/312:37:483
10138992323,0cyclictest0-21swapper/312:20:113
10138992323,0cyclictest0-21swapper/312:00:413
10138992323,0cyclictest0-21swapper/309:22:083
10138992321,1cyclictest27987-21ssh11:45:323
10138992320,2cyclictest19144-21perf08:35:003
10138992320,2cyclictest14565-21diskmemload10:58:573
1013899230,2cyclictest31870-21cron07:55:003
1013899230,23cyclictest0-21swapper/310:42:483
1013899230,23cyclictest0-21swapper/310:08:193
1013899230,23cyclictest0-21swapper/309:26:033
1013899230,22cyclictest24970-21rm09:31:533
1013899230,22cyclictest0-21swapper/308:13:273
1013899230,22cyclictest0-21swapper/307:13:263
1013899230,22cyclictest0-21swapper/307:13:263
1013899230,1cyclictest10759-21apt-get09:18:253
1013899230,0cyclictest3534-21diskmemload12:10:203
1013899230,0cyclictest0-21swapper/311:55:103
10134992323,0cyclictest0-21swapper/211:45:452
10134992322,1cyclictest3534-21diskmemload10:55:442
10134992321,1cyclictest23950-21turbostat08:45:002
10134992320,3cyclictest0-21swapper/210:33:002
10134992320,2cyclictest0-21swapper/211:52:202
10134992320,2cyclictest0-21swapper/209:35:312
10134992319,3cyclictest0-21swapper/210:15:012
1013499230,23cyclictest27333-21apt-get11:13:252
1013499230,22cyclictest8204-21sh11:25:572
1013499230,22cyclictest3534-21diskmemload11:08:002
1013499230,22cyclictest3534-21diskmemload10:23:562
1013499230,22cyclictest3534-21diskmemload09:21:452
1013499230,22cyclictest19155-21munin-run08:35:002
1013499230,22cyclictest0-21swapper/212:04:322
1013499230,22cyclictest0-21swapper/210:47:382
1013499230,1cyclictest3534-21diskmemload12:11:022
1013499230,1cyclictest0-21swapper/212:08:382
1013499230,0cyclictest0-21swapper/211:57:082
1013499230,0cyclictest0-21swapper/211:38:042
1013499230,0cyclictest0-21swapper/211:04:052
10129992323,0cyclictest0-21swapper/112:12:011
10129992323,0cyclictest0-21swapper/112:04:581
10129992323,0cyclictest0-21swapper/111:51:371
10129992323,0cyclictest0-21swapper/109:54:571
10129992323,0cyclictest0-21swapper/109:39:221
10129992321,1cyclictest32162-21ssh10:12:321
10129992321,1cyclictest0-21swapper/111:25:011
10129992321,1cyclictest0-21swapper/110:55:001
10129992321,1cyclictest0-21swapper/108:18:301
10129992320,2cyclictest9081-21dump-pmu-power08:15:011
10129992320,2cyclictest28856-21ssh10:41:031
1012999230,2cyclictest13205-21sh10:25:161
1012999230,23cyclictest0-21swapper/112:33:271
1012999230,23cyclictest0-21swapper/110:56:591
1012999230,23cyclictest0-21swapper/109:15:551
1012999230,22cyclictest8167-21apt-get09:48:271
1012999230,22cyclictest7706-21ssh09:13:421
1012999230,22cyclictest3534-21diskmemload12:37:421
1012999230,22cyclictest3534-21diskmemload10:33:331
1012999230,22cyclictest3534-21diskmemload10:20:421
1012999230,22cyclictest21120-21sh11:06:151
1012999230,22cyclictest13937-21ssh09:20:261
1012999230,22cyclictest0-21swapper/111:18:281
1012999230,22cyclictest0-21swapper/107:58:291
1012999230,0cyclictest3534-21diskmemload09:31:061
10125992323,0cyclictest0-21swapper/010:50:480
10125992320,3cyclictest0-21swapper/011:57:050
10125992320,3cyclictest0-21swapper/010:36:530
10125992319,3cyclictest0-21swapper/008:38:290
1012599230,23cyclictest0-21swapper/010:28:040
1012599230,23cyclictest0-21swapper/010:15:540
1012599230,23cyclictest0-21swapper/009:45:450
1012599230,22cyclictest9378-21ssh09:16:070
1012599230,22cyclictest2698-21sh10:48:240
1012599230,22cyclictest24476-21apt-get08:48:270
1012599230,0cyclictest3534-21diskmemload09:28:240
1012599230,0cyclictest0-21swapper/012:37:010
1012599230,0cyclictest0-21swapper/011:47:480
1012599230,0cyclictest0-21swapper/011:40:470
1012599230,0cyclictest0-21swapper/009:44:150
10138992222,0cyclictest3534-21diskmemload10:00:213
10138992222,0cyclictest0-21swapper/311:40:093
10138992222,0cyclictest0-21swapper/311:19:013
10138992222,0cyclictest0-21swapper/310:31:293
10138992222,0cyclictest0-21swapper/310:25:523
10138992222,0cyclictest0-21swapper/310:21:463
10138992222,0cyclictest0-21swapper/309:56:063
10138992221,1cyclictest0-21swapper/310:48:243
10138992220,1cyclictest9603-21apt-get08:18:303
10138992220,1cyclictest6227-21ssh10:51:033
10138992220,1cyclictest3878-21diskmemload11:21:573
10138992220,1cyclictest12022-21apt-get08:23:253
10138992220,1cyclictest0-21swapper/311:25:173
1013899220,21cyclictest1138-21awk09:05:013
1013899220,21cyclictest0-21swapper/311:13:303
1013899220,1cyclictest895-21ssh11:51:153
1013899220,1cyclictest22684-21apt-get07:38:263
1013899220,1cyclictest19357-21ssh11:03:383
1013899220,1cyclictest14903-21apt07:20:003
1013899220,1cyclictest0-21swapper/311:36:233
1013899220,1cyclictest0-21swapper/311:08:163
1013899220,0cyclictest0-21swapper/309:38:323
10134992222,0cyclictest3534-21diskmemload10:10:432
10134992222,0cyclictest0-21swapper/208:40:002
10134992220,2cyclictest0-21swapper/209:17:002
10134992219,2cyclictest0-21swapper/209:30:292
10134992217,4cyclictest0-21swapper/211:43:342
1013499220,21cyclictest0-21swapper/212:35:062
1013499220,21cyclictest0-21swapper/210:00:482
1013499220,21cyclictest0-21swapper/208:53:272
10129992222,0cyclictest3534-21diskmemload11:20:521
10129992222,0cyclictest3534-21diskmemload10:16:111
10129992222,0cyclictest0-21swapper/112:19:051
10129992220,1cyclictest25841-21chrt10:05:071
10129992220,1cyclictest0-21swapper/112:07:051
1012999220,2cyclictest19771-21sh09:26:541
1012999220,22cyclictest0-21swapper/110:45:101
1012999220,21cyclictest0-21swapper/111:36:321
1012999220,21cyclictest0-21swapper/109:08:241
1012999220,21cyclictest0-21swapper/107:38:261
1012999220,1cyclictest21553-21perf08:40:001
1012999220,1cyclictest0-21swapper/110:00:361
1012999220,0cyclictest0-21swapper/111:34:591
1012999220,0cyclictest0-21swapper/107:23:201
10125992222,0cyclictest3534-21diskmemload09:35:110
10125992221,1cyclictest12929-21kworker/u8:212:21:240
10125992219,2cyclictest0-21swapper/012:31:080
1012599220,21cyclictest0-21swapper/007:25:010
1012599220,0cyclictest0-21swapper/011:15:200
1012599220,0cyclictest0-21swapper/011:06:230
10138992121,0cyclictest0-21swapper/309:45:483
10138992119,2cyclictest1622-21sshd09:40:413
10134992120,1cyclictest9090-21munin-run08:15:012
10134992120,1cyclictest0-21swapper/207:58:282
10134992119,1cyclictest3557-21sleep09:10:002
1013499210,20cyclictest0-21swapper/207:18:282
1013499210,0cyclictest0-21swapper/212:15:472
1013499210,0cyclictest0-21swapper/207:33:262
10129992121,0cyclictest0-21swapper/111:41:061
10129992121,0cyclictest0-21swapper/108:23:261
10129992121,0cyclictest0-21swapper/107:18:281
1012999210,0cyclictest0-21swapper/110:35:491
10125992117,4cyclictest0-21swapper/007:13:260
10125992117,4cyclictest0-21swapper/007:13:260
1012599210,21cyclictest0-21swapper/007:28:250
1012599210,0cyclictest29277-21apt-get08:58:280
10138992018,1cyclictest28770-21munin-plugin-st08:55:003
10138992018,1cyclictest21550-21latency_hist08:40:003
1013899200,20cyclictest0-21swapper/308:08:313
1013899200,0cyclictest0-21swapper/308:28:273
10134992020,0cyclictest0-21swapper/209:03:262
1013499200,0cyclictest0-21swapper/208:28:272
10129992020,0cyclictest22078-21apt-get08:43:261
10129992020,0cyclictest0-21swapper/108:48:271
10129992020,0cyclictest0-21swapper/108:28:271
10129992020,0cyclictest0-21swapper/107:33:271
1012999200,0cyclictest0-21swapper/108:08:301
10125992020,0cyclictest16279-21apt-get08:30:010
10125992020,0cyclictest0-21swapper/008:18:300
10125992020,0cyclictest0-21swapper/008:13:270
10138991918,1cyclictest2415-21apt-get08:03:323
10134991919,0cyclictest24476-21apt-get08:48:272
10134991916,3cyclictest0-21swapper/208:03:322
10129991918,0cyclictest17832-21apt-get07:28:251
10129991918,0cyclictest0-21swapper/108:58:281
1012999190,2cyclictest27067-21idleruntime-cro07:45:001
1012599193,15cyclictest0-21swapper/008:43:370
10125991919,0cyclictest14418-21apt-get08:28:270
10125991919,0cyclictest0-21swapper/008:03:320
10125991919,0cyclictest0-21swapper/007:48:290
10125991917,1cyclictest13904-21munin-plugin-st08:25:000
1012599190,4cyclictest0-21swapper/008:53:330
10138991818,0cyclictest0-21swapper/307:23:203
10134991817,1cyclictest29481-21perf07:50:002
10129991818,0cyclictest0-21swapper/108:30:011
10129991818,0cyclictest0-21swapper/107:13:261
10129991818,0cyclictest0-21swapper/107:13:261
10129991817,0cyclictest27605-21apt-get07:48:291
10129991816,1cyclictest0-21swapper/109:03:251
10125991818,0cyclictest4809-21apt-get08:08:300
10125991818,0cyclictest0-21swapper/007:58:280
10138991717,0cyclictest0-21swapper/308:43:263
10134991717,0cyclictest0-21swapper/208:08:312
1013499170,16cyclictest0-21swapper/207:23:202
10125991717,0cyclictest0-21swapper/009:08:240
10125991717,0cyclictest0-21swapper/007:45:000
10138991616,0cyclictest0-21swapper/308:58:283
10138991616,0cyclictest0-21swapper/307:43:253
10138991615,1cyclictest17832-21apt-get07:28:253
10134991616,0cyclictest0-21swapper/208:23:252
10134991615,1cyclictest0-21swapper/207:43:252
10134991615,0cyclictest0-21swapper/207:13:262
10134991615,0cyclictest0-21swapper/207:13:262
1013499160,15cyclictest24691-21munin-run07:40:002
10129991616,0cyclictest2415-21apt-get08:03:321
1012999160,15cyclictest27296-21df_inode08:53:281
10125991616,0cyclictest0-21swapper/007:38:260
1012599160,15cyclictest12736-21taskset07:15:100
1012599160,0cyclictest0-21swapper/007:32:380
10138991515,0cyclictest20285-21apt-get07:33:273
10134991512,0cyclictest0-21swapper/207:50:222
1013499150,0cyclictest0-21swapper/208:17:112
1013899140,14cyclictest0-21swapper/308:45:403
1013899140,14cyclictest0-21swapper/307:55:183
1013899140,0cyclictest0-21swapper/307:45:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional