You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-02 - 19:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Sat Dec 02, 2023 12:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19346992523,1cyclictest24523-21sed12:07:512
19346992523,0cyclictest0-21swapper/210:01:332
1934699250,24cyclictest22462-21rm10:58:482
1934199250,24cyclictest0-21swapper/111:31:181
1934199250,24cyclictest0-21swapper/110:05:571
1934199250,24cyclictest0-21swapper/109:15:121
1934199250,1cyclictest13185-21diskmemload10:56:291
19340992523,1cyclictest13185-21diskmemload10:43:110
1934099250,24cyclictest0-21swapper/009:18:380
19347992423,0cyclictest0-21swapper/311:37:293
1934799240,2cyclictest12194-21rm12:30:223
1934799240,23cyclictest29277-21cron07:30:003
1934799240,23cyclictest0-21swapper/312:10:173
19346992422,1cyclictest13185-21diskmemload09:37:082
19346992419,1cyclictest10120-21rm11:20:482
1934699240,23cyclictest3504-21ssh10:39:572
1934699240,23cyclictest13185-21diskmemload11:33:182
1934699240,23cyclictest13185-21diskmemload10:05:462
19341992423,1cyclictest2084-21ssh09:31:551
1934199240,24cyclictest0-21swapper/112:10:091
1934199240,23cyclictest8122-21sh10:44:231
1934199240,23cyclictest0-21swapper/111:53:471
1934199240,23cyclictest0-21swapper/111:05:051
1934199240,23cyclictest0-21swapper/109:36:241
1934199240,1cyclictest0-21swapper/109:13:241
1934199240,0cyclictest13185-21diskmemload12:22:351
19340992423,1cyclictest0-21swapper/011:14:370
19340992423,0cyclictest0-21swapper/011:40:540
19340992422,1cyclictest9119-21munin-run07:55:000
19340992422,1cyclictest13438-21ssh12:32:290
1934099240,24cyclictest0-21swapper/011:31:180
1934099240,23cyclictest13185-21diskmemload10:19:000
1934099240,23cyclictest0-21swapper/012:22:350
1934099240,23cyclictest0-21swapper/012:15:110
1934099240,23cyclictest0-21swapper/009:12:350
1934099240,1cyclictest0-21swapper/010:01:260
1934099240,0cyclictest0-21swapper/009:40:280
19347992323,0cyclictest0-21swapper/312:01:073
19347992323,0cyclictest0-21swapper/311:03:473
19347992323,0cyclictest0-21swapper/310:47:323
19347992323,0cyclictest0-21swapper/310:01:333
19347992322,1cyclictest0-21swapper/309:47:343
1934799230,23cyclictest13185-21diskmemload10:40:233
1934799230,23cyclictest13185-21diskmemload10:21:303
1934799230,23cyclictest0-21swapper/311:41:543
1934799230,23cyclictest0-21swapper/311:21:563
1934799230,23cyclictest0-21swapper/311:09:243
1934799230,23cyclictest0-21swapper/310:37:253
1934799230,23cyclictest0-21swapper/310:25:153
1934799230,23cyclictest0-21swapper/309:13:213
1934799230,22cyclictest6011-21munin-run08:55:013
1934799230,22cyclictest32256-21sh12:17:083
1934799230,22cyclictest26971-21ssh10:30:593
1934799230,22cyclictest13185-21diskmemload09:20:333
1934799230,0cyclictest13185-21diskmemload09:52:253
1934799230,0cyclictest13185-21diskmemload09:33:023
1934799230,0cyclictest0-21swapper/312:07:043
1934799230,0cyclictest0-21swapper/310:58:483
1934799230,0cyclictest0-21swapper/310:18:563
1934799230,0cyclictest0-21swapper/310:05:463
1934799230,0cyclictest0-21swapper/309:38:243
19346992323,0cyclictest13185-21diskmemload11:37:292
19346992323,0cyclictest0-21swapper/212:01:072
19346992323,0cyclictest0-21swapper/211:25:312
19346992323,0cyclictest0-21swapper/210:47:322
19346992322,1cyclictest9602-21ssh10:11:462
19346992321,1cyclictest0-21swapper/211:17:152
1934699230,23cyclictest0-21swapper/212:10:172
1934699230,23cyclictest0-21swapper/211:51:472
1934699230,23cyclictest0-21swapper/210:18:012
1934699230,22cyclictest32221-21ssh11:10:402
1934699230,22cyclictest31282-21ssh12:15:272
1934699230,22cyclictest29283-21cron07:30:002
1934699230,22cyclictest17324-21sh12:35:592
1934699230,22cyclictest15525-21sh11:59:012
1934699230,1cyclictest18630-21ssh09:47:412
1934699230,0cyclictest27353-21ssh11:03:472
1934699230,0cyclictest0-21swapper/212:32:262
1934699230,0cyclictest0-21swapper/210:25:152
19341992323,0cyclictest13185-21diskmemload12:01:331
19341992323,0cyclictest0-21swapper/110:34:391
19341992323,0cyclictest0-21swapper/110:24:241
19341992323,0cyclictest0-21swapper/110:13:261
19341992322,1cyclictest13185-21diskmemload11:35:061
19341992322,1cyclictest0-21swapper/112:05:031
19341992321,1cyclictest0-21swapper/112:17:361
1934199230,23cyclictest13185-21diskmemload11:40:541
1934199230,23cyclictest0-21swapper/111:22:331
1934199230,23cyclictest0-21swapper/111:00:211
1934199230,23cyclictest0-21swapper/109:53:331
1934199230,22cyclictest75-21kswapd012:38:321
1934199230,22cyclictest17748-21sh10:19:231
1934199230,22cyclictest13185-21diskmemload10:47:051
1934199230,22cyclictest13185-21diskmemload09:20:261
1934199230,1cyclictest774-21runrttasks10:25:421
1934199230,1cyclictest0-21swapper/112:31:121
1934199230,1cyclictest0-21swapper/111:57:461
1934199230,1cyclictest0-21swapper/109:27:041
1934199230,1cyclictest0-21swapper/107:25:011
1934199230,0cyclictest0-21swapper/111:46:581
1934199230,0cyclictest0-21swapper/111:18:091
1934199230,0cyclictest0-21swapper/111:14:471
1934199230,0cyclictest0-21swapper/109:59:121
1934199230,0cyclictest0-21swapper/109:45:051
19340992323,0cyclictest13185-21diskmemload09:53:330
19340992323,0cyclictest0-21swapper/012:10:090
19340992323,0cyclictest0-21swapper/011:53:470
19340992323,0cyclictest0-21swapper/011:18:090
19340992323,0cyclictest0-21swapper/011:09:220
19340992323,0cyclictest0-21swapper/010:47:420
19340992323,0cyclictest0-21swapper/010:05:250
19340992323,0cyclictest0-21swapper/009:59:120
19340992323,0cyclictest0-21swapper/009:49:000
19340992323,0cyclictest0-21swapper/009:36:240
19340992321,1cyclictest0-21swapper/011:24:520
19340992321,1cyclictest0-21swapper/009:31:430
1934099230,2cyclictest29916-21ssh10:33:020
1934099230,23cyclictest0-21swapper/012:37:330
1934099230,22cyclictest23667-21runrttasks10:25:420
1934099230,22cyclictest12996-21/usr/sbin/munin11:57:460
1934099230,22cyclictest0-21swapper/011:25:340
1934099230,22cyclictest0-21swapper/010:50:230
1934099230,22cyclictest0-21swapper/009:22:360
1934099230,21cyclictest23949-21munin-run08:25:000
1934099230,21cyclictest0-21swapper/011:48:030
1934099230,0cyclictest13185-21diskmemload12:26:280
1934099230,0cyclictest0-21swapper/010:23:200
19347992222,0cyclictest0-21swapper/311:46:253
19347992222,0cyclictest0-21swapper/309:40:043
19347992222,0cyclictest0-21swapper/309:16:183
19347992220,1cyclictest19514-21ssh11:30:113
1934799220,22cyclictest13185-21diskmemload09:25:383
1934799220,22cyclictest0-21swapper/311:25:063
1934799220,22cyclictest0-21swapper/311:19:063
1934799220,22cyclictest0-21swapper/311:11:353
1934799220,22cyclictest0-21swapper/309:59:373
1934799220,21cyclictest18640-21systemd-journal10:15:003
1934799220,21cyclictest13185-21diskmemload12:35:253
1934799220,21cyclictest0-21swapper/311:55:213
1934799220,21cyclictest0-21swapper/310:50:133
19346992222,0cyclictest0-21swapper/211:46:252
19346992222,0cyclictest0-21swapper/209:40:042
19346992220,1cyclictest18389-21rm09:15:062
19346992220,1cyclictest0-21swapper/209:30:382
19346992220,0cyclictest0-21swapper/212:25:092
1934699220,22cyclictest6002-21perf08:55:012
1934699220,22cyclictest13185-21diskmemload10:41:052
1934699220,22cyclictest0-21swapper/211:05:372
1934699220,21cyclictest21093-21ssh09:50:222
1934699220,21cyclictest0-21swapper/210:51:432
1934699220,21cyclictest0-21swapper/209:20:032
1934699220,0cyclictest0-21swapper/210:31:192
1934699220,0cyclictest0-21swapper/210:21:302
1934699220,0cyclictest0-21swapper/209:25:382
19341992222,0cyclictest0-21swapper/112:26:091
19341992222,0cyclictest0-21swapper/111:27:461
19341992220,1cyclictest0-21swapper/110:50:191
19341992220,1cyclictest0-21swapper/109:40:121
1934199220,22cyclictest0-21swapper/110:00:521
1934199220,21cyclictest3555-21perf08:50:011
1934199220,21cyclictest31176-21munin-plugin-st08:40:001
1934199220,21cyclictest13933-21idleruntime-cro08:05:011
1934199220,21cyclictest0-21swapper/108:17:391
1934199220,1cyclictest23948-21apt08:25:001
1934199220,1cyclictest0-21swapper/108:12:351
1934199220,0cyclictest0-21swapper/110:39:211
1934199220,0cyclictest0-21swapper/107:55:001
19340992222,0cyclictest13185-21diskmemload10:59:380
19340992222,0cyclictest0-21swapper/011:36:510
19340992222,0cyclictest0-21swapper/011:00:210
19340992222,0cyclictest0-21swapper/009:02:340
1934099220,21cyclictest13185-21diskmemload10:35:440
1934099220,21cyclictest0-21swapper/012:06:300
1934099220,21cyclictest0-21swapper/012:00:290
1934099220,21cyclictest0-21swapper/009:25:100
1934099220,0cyclictest0-21swapper/010:10:530
1934099220,0cyclictest0-21swapper/007:25:010
19347992119,1cyclictest3540-21dump-pmu-power08:50:013
19347992119,1cyclictest19585-21apt-get08:17:393
1934799210,21cyclictest0-21swapper/312:25:173
1934799210,21cyclictest0-21swapper/312:20:363
1934799210,21cyclictest0-21swapper/311:50:243
19346992120,0cyclictest32209-21apt-get07:37:342
1934699210,21cyclictest0-21swapper/209:56:062
1934699210,20cyclictest21532-21idleruntime-cro08:20:002
1934199210,21cyclictest0-21swapper/109:07:361
1934199210,20cyclictest0-21swapper/107:22:361
19340992121,0cyclictest11265-21apt-get09:07:360
1934099210,20cyclictest0-21swapper/008:15:010
1934099210,20cyclictest0-21swapper/008:12:350
1934099210,20cyclictest0-21swapper/008:05:010
1934099210,0cyclictest29200-21apt-get08:37:350
1934099210,0cyclictest0-21swapper/008:50:010
1934099210,0cyclictest0-21swapper/007:37:340
1934099210,0cyclictest0-21swapper/007:22:360
19347992020,0cyclictest7169-21apt-get07:52:373
19347992020,0cyclictest0-21swapper/308:32:343
19347992020,0cyclictest0-21swapper/307:37:343
19347992020,0cyclictest0-21swapper/307:22:363
19347992020,0cyclictest0-21swapper/307:17:323
1934799200,0cyclictest16775-21apt-get08:12:353
19346992020,0cyclictest26792-21apt-get08:32:342
19346992020,0cyclictest24565-21apt-get07:22:362
19346992020,0cyclictest0-21swapper/207:52:372
19346992020,0cyclictest0-21swapper/207:17:322
19346992018,1cyclictest3536-21awk08:50:012
1934699200,0cyclictest31619-21apt-get08:42:382
1934699200,0cyclictest0-21swapper/208:12:352
19341992020,0cyclictest0-21swapper/108:27:361
19341992018,1cyclictest24091-21perf07:20:001
19340992019,1cyclictest24395-21apt-get08:27:360
19340992017,2cyclictest24102-21awk07:20:000
1934099200,0cyclictest0-21swapper/008:07:360
19347991919,0cyclictest0-21swapper/309:02:343
19347991919,0cyclictest0-21swapper/308:37:353
19347991919,0cyclictest0-21swapper/307:42:383
19347991919,0cyclictest0-21swapper/307:12:363
19347991918,0cyclictest0-21swapper/308:02:363
19346991919,0cyclictest8867-21apt-get09:02:342
19346991919,0cyclictest2324-21apt-get07:42:382
19346991919,0cyclictest19747-21apt-get07:12:362
19346991919,0cyclictest0-21swapper/208:37:352
19341991918,0cyclictest0-21swapper/107:36:451
1934199190,0cyclictest0-21swapper/108:42:381
19347991818,0cyclictest0-21swapper/308:27:363
19347991818,0cyclictest0-21swapper/308:07:363
19347991818,0cyclictest0-21swapper/307:57:333
19346991818,0cyclictest9560-21apt-get07:57:332
19346991818,0cyclictest0-21swapper/208:27:362
19346991818,0cyclictest0-21swapper/208:02:362
19346991817,0cyclictest14379-21apt-get08:07:362
19341991818,0cyclictest0-21swapper/108:32:341
19341991818,0cyclictest0-21swapper/107:42:381
19340991817,0cyclictest0-21swapper/007:42:380
19340991816,1cyclictest19747-21apt-get07:12:350
19341991717,0cyclictest0-21swapper/109:02:341
19341991717,0cyclictest0-21swapper/108:07:361
19341991717,0cyclictest0-21swapper/107:12:361
19341991715,1cyclictest6718-21awk07:50:001
19341991714,1cyclictest11535-21grep08:00:011
19340991717,0cyclictest0-21swapper/008:32:340
19346991616,0cyclictest6449-21apt-get08:57:352
19341991616,0cyclictest0-21swapper/107:32:371
19341991614,1cyclictest3998-21apt-get08:52:341
19340991616,0cyclictest0-21swapper/008:42:380
19340991614,1cyclictest6725-21sleep07:50:000
19347991515,0cyclictest0-21swapper/308:57:353
19347991513,1cyclictest6728-21grep07:50:003
1934699150,1cyclictest6724-21perf07:50:002
19341991515,0cyclictest0-21swapper/108:57:351
19340991515,0cyclictest29765-21apt-get07:32:370
19340991513,1cyclictest11524-21sendmail-msp08:00:010
19347991414,0cyclictest0-21swapper/308:40:183
1934799140,14cyclictest0-21swapper/309:07:033
1934799140,14cyclictest0-21swapper/308:20:013
1934799140,14cyclictest0-21swapper/307:30:403
19346991414,0cyclictest0-21swapper/209:05:502
1934699140,14cyclictest0-21swapper/208:20:162
1934699140,14cyclictest0-21swapper/207:30:192
19340991414,0cyclictest0-21swapper/008:50:450
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional