You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-20 - 16:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Apr 20, 2024 12:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7467210389,9sleep10-21swapper/107:07:191
7552210162,34sleep30-21swapper/307:08:263
751128566,8sleep00-21swapper/007:07:540
755627757,16sleep20-21swapper/207:08:292
7777992719,5cyclictest39-21ksoftirqd/312:05:003
7777992621,3cyclictest39-21ksoftirqd/311:34:313
777399250,24cyclictest0-21swapper/211:45:312
777399240,23cyclictest0-21swapper/211:55:492
776899234,6cyclictest13-21ksoftirqd/011:47:100
777799224,3cyclictest39-21ksoftirqd/312:09:363
777799220,10cyclictest0-21swapper/307:30:133
777799220,10cyclictest0-21swapper/307:30:133
7773992219,2cyclictest25655-21head07:45:172
7768992220,1cyclictest13-21ksoftirqd/011:00:310
777799213,4cyclictest39-21ksoftirqd/308:20:163
777399210,20cyclictest0-21swapper/208:30:152
776899210,18cyclictest0-21swapper/011:20:120
777799209,4cyclictest39-21ksoftirqd/310:50:173
777799202,5cyclictest39-21ksoftirqd/310:30:243
777399200,0cyclictest0-21swapper/210:54:162
7771992020,0cyclictest0-21swapper/112:06:511
776899207,1cyclictest141rcu_preempt10:39:050
776899202,14cyclictest141rcu_preempt10:10:180
777799193,6cyclictest31232-21diskmemload10:15:453
777799191,2cyclictest141rcu_preempt12:36:293
7777991912,2cyclictest39-21ksoftirqd/312:31:403
777799190,17cyclictest0-21swapper/309:19:053
7773991914,4cyclictest29107-21apache_volume09:05:102
776899192,1cyclictest141rcu_preempt12:18:200
7768991918,1cyclictest1284-21kworker/u8:309:14:140
7768991917,1cyclictest13-21ksoftirqd/011:41:180
776899191,2cyclictest141rcu_preempt08:25:270
777799189,3cyclictest39-21ksoftirqd/311:11:223
777799188,3cyclictest39-21ksoftirqd/312:10:313
7777991814,3cyclictest0-21swapper/309:30:123
7777991814,1cyclictest39-21ksoftirqd/310:07:053
777799181,2cyclictest381rcuc/309:25:033
777799181,2cyclictest381rcuc/309:25:023
777799181,14cyclictest39-21ksoftirqd/312:18:193
777799180,4cyclictest0-21swapper/310:40:253
777799180,3cyclictest0-21swapper/311:49:353
777799180,0cyclictest0-21swapper/310:14:173
777399183,8cyclictest32040-21munin-run12:05:002
777399181,1cyclictest0-21swapper/209:56:402
777199180,18cyclictest0-21swapper/111:19:081
777199180,18cyclictest0-21swapper/111:19:081
777199180,18cyclictest0-21swapper/110:13:381
776899180,18cyclictest0-21swapper/010:42:550
777799179,1cyclictest381rcuc/309:40:233
777799175,2cyclictest141rcu_preempt11:52:503
777799173,7cyclictest30363-21/usr/sbin/munin12:25:293
7777991716,1cyclictest39-21ksoftirqd/311:55:423
7777991715,2cyclictest39-21ksoftirqd/308:50:283
7777991714,3cyclictest39-21ksoftirqd/311:25:303
777199175,7cyclictest3641-21munin-run09:40:001
7771991714,1cyclictest0-21swapper/110:02:571
777199170,1cyclictest0-21swapper/112:21:481
777199170,16cyclictest0-21swapper/109:32:061
776899175,6cyclictest24091-21perf08:55:000
7768991715,1cyclictest13-21ksoftirqd/009:42:260
776899170,3cyclictest141rcu_preempt10:50:550
777799164,6cyclictest39-21ksoftirqd/309:22:343
777799162,2cyclictest39-21ksoftirqd/312:22:523
7777991614,1cyclictest0-21swapper/308:40:133
777799160,2cyclictest0-21swapper/310:25:593
777799160,1cyclictest0-21swapper/311:23:503
7773991616,0cyclictest0-21swapper/210:13:072
777399161,13cyclictest0-21swapper/211:40:132
777399160,0cyclictest0-21swapper/209:31:082
7771991616,0cyclictest0-21swapper/109:47:081
7771991616,0cyclictest0-21swapper/109:23:081
776899161,7cyclictest27064-21idleruntime-cro07:50:010
7768991613,2cyclictest13-21ksoftirqd/007:45:000
7768991613,1cyclictest13-21ksoftirqd/011:54:290
7768991611,1cyclictest17148-21kworker/u8:011:25:480
7777991511,3cyclictest31232-21diskmemload09:37:263
7777991510,3cyclictest39-21ksoftirqd/309:57:133
7777991510,1cyclictest39-21ksoftirqd/311:44:533
7773991514,1cyclictest37850irq/131-snd_hda09:46:112
7773991514,1cyclictest0-21swapper/210:45:062
7773991511,3cyclictest20712-21latency_hist11:10:012
777199151,13cyclictest24936-21apt-get07:45:131
7771991510,4cyclictest0-21swapper/111:52:421
777199150,1cyclictest0-21swapper/111:04:391
776899157,1cyclictest141rcu_preempt11:37:380
776899153,6cyclictest22042-21sendmail-msp10:20:000
776899152,2cyclictest141rcu_preempt09:24:430
7768991513,2cyclictest13-21ksoftirqd/009:45:260
777799146,8cyclictest381rcuc/311:36:053
777799144,10cyclictest39-21ksoftirqd/309:11:433
7777991412,2cyclictest39-21ksoftirqd/309:52:243
7777991411,2cyclictest29938-21tr10:00:143
777799140,3cyclictest0-21swapper/310:20:023
777799140,2cyclictest0-21swapper/310:38:343
777799140,13cyclictest9136-21tr07:10:213
777399146,2cyclictest1291-21runrttasks12:21:222
7773991412,1cyclictest3910-21taskset12:30:192
7773991410,3cyclictest17394-21cat11:30:202
777399140,1cyclictest0-21swapper/211:27:412
7771991411,2cyclictest17572-21head09:50:181
7771991410,3cyclictest3645-21ls08:10:251
7771991410,3cyclictest22390-21wc11:10:251
7771991410,3cyclictest17934-21grep08:40:171
777199140,2cyclictest0-21swapper/110:09:281
777199140,0cyclictest0-21swapper/110:47:421
776899149,1cyclictest13-21ksoftirqd/010:45:010
776899146,3cyclictest224-21systemd-journal07:20:540
776899144,9cyclictest13-21ksoftirqd/012:01:570
7768991412,1cyclictest141rcu_preempt11:11:390
7768991412,1cyclictest13-21ksoftirqd/010:21:400
776899141,1cyclictest141rcu_preempt12:25:260
7768991411,2cyclictest10121-21munin-run07:15:000
776899140,13cyclictest1196-21wc12:05:220
777799139,3cyclictest39-21ksoftirqd/311:20:003
777799139,3cyclictest39-21ksoftirqd/311:20:003
777799139,3cyclictest16148-21cut07:25:193
777799133,8cyclictest14519-21cron11:05:003
7777991310,2cyclictest20382-21tr07:35:133
777799130,12cyclictest31240-21basename09:10:003
777399139,3cyclictest31039-21grep08:00:192
777399139,3cyclictest23788-21sort11:35:192
777399139,3cyclictest11618-21ls12:35:232
777399137,5cyclictest5266-21cat10:05:302
7773991311,1cyclictest32549-21sh09:36:112
7773991310,2cyclictest25061-21head08:55:192
7773991310,1cyclictest0-21swapper/211:02:522
777399130,13cyclictest16542-21kworker/2:012:18:182
777399130,12cyclictest0-21swapper/211:53:142
777399130,0cyclictest0-21swapper/212:09:082
777199139,3cyclictest5961-21rm11:45:241
777199139,2cyclictest25065-21rm11:35:321
7771991310,2cyclictest4565-21grep11:20:311
7771991310,2cyclictest30038-21sh08:00:001
7771991310,2cyclictest27954-21ls09:00:251
7771991310,2cyclictest22004-21sh10:20:001
7771991310,2cyclictest1890-21dump-pmu-power10:55:001
7771991310,2cyclictest16412-21tail08:35:261
776899132,10cyclictest28161-21apache_processe11:15:130
776899132,10cyclictest28161-21apache_processe11:15:130
7768991312,1cyclictest181irq_work/009:15:400
7768991310,2cyclictest4480-21sed10:05:200
7768991310,2cyclictest27492-21sed09:00:200
7768991310,2cyclictest16358-21wc11:05:240
777799129,2cyclictest2610-21tr10:55:173
777799129,2cyclictest25609-21cut07:45:173
777799129,2cyclictest23226-21grep07:40:233
777799129,2cyclictest16869-21sh08:40:013
777799129,2cyclictest14217-21tr07:20:233
777799128,3cyclictest22907-21grep10:45:213
777799128,3cyclictest12216-21sed07:15:283
777799128,2cyclictest4743-21cron08:14:593
7777991210,1cyclictest31232-21diskmemload11:05:163
777399129,3cyclictest37850irq/131-snd_hda10:40:322
777399129,2cyclictest9101-21sed08:20:292
777399129,2cyclictest8286-21grep07:10:142
777399129,2cyclictest28976-21cat07:55:162
777399129,2cyclictest24073-21cat10:20:302
777399129,2cyclictest16871-21sh08:40:002
777399126,5cyclictest18539-21ssh09:25:172
777399126,5cyclictest18539-21ssh09:25:172
777399124,4cyclictest224-21systemd-journal07:30:002
777399121,8cyclictest516-21gdbus11:20:262
7773991210,1cyclictest1984-21grep08:05:272
777399120,12cyclictest0-21swapper/210:25:202
777399120,11cyclictest7997-21grep12:10:202
777399120,11cyclictest17552-21grep09:50:182
777199129,2cyclictest29047-21tr12:25:161
777199129,2cyclictest23621-21head09:55:181
777199129,2cyclictest20673-21tr08:45:231
777199129,2cyclictest20366-21cut07:35:131
777199129,2cyclictest19092-21cut09:25:211
777199129,2cyclictest19092-21cut09:25:211
777199129,2cyclictest16878-21sshd12:40:001
777199129,2cyclictest15611-21tr07:25:161
777199129,2cyclictest10877-21grep07:15:141
777199127,4cyclictest15366-21cut10:40:171
777199124,4cyclictest224-21systemd-journal08:24:591
777199120,11cyclictest31208-21cron09:10:001
776899129,2cyclictest914-21sort08:05:160
776899129,2cyclictest29886-21cut09:05:170
776899129,2cyclictest22342-21cat07:40:000
776899129,2cyclictest18597-21chrt09:25:170
776899129,2cyclictest18597-21chrt09:25:170
776899129,2cyclictest17784-21sort08:40:160
776899129,2cyclictest17541-21tail09:50:180
776899128,3cyclictest25047-21grep08:55:190
7768991210,2cyclictest13-21ksoftirqd/012:13:380
7768991210,1cyclictest161rcuc/007:15:200
16442120,0sleep00-21swapper/009:37:440
777799118,2cyclictest32729-21wget08:05:123
777799118,2cyclictest30806-21df08:00:173
777799118,2cyclictest28335-21date09:00:313
777799118,2cyclictest11947-21ssh09:45:263
777799118,2cyclictest11437-21grep08:25:323
777399119,2cyclictest0-21swapper/210:55:112
777399119,1cyclictest29474-21ntp_states11:15:242
777399119,1cyclictest29474-21ntp_states11:15:242
777399118,2cyclictest916-21ssh10:02:292
777399118,2cyclictest6155-21tr08:15:222
777399118,2cyclictest4563-21expr09:40:162
777399118,2cyclictest28537-21ssh12:25:042
777399118,2cyclictest24346-21sh11:11:502
777399118,2cyclictest22186-21grep08:50:142
777399118,2cyclictest21197-21tr07:35:192
777399118,2cyclictest20159-21uniq08:45:182
777399118,2cyclictest18123-21expr07:30:152
777399118,2cyclictest18123-21expr07:30:152
777399118,2cyclictest18095-21uname08:40:202
777399118,2cyclictest12286-21head07:15:282
777399118,2cyclictest11803-21sh10:36:312
777399118,2cyclictest11508-21basename09:20:142
777399118,2cyclictest10747-21taskset09:19:072
777399111,8cyclictest27223-21df09:00:162
7773991111,0cyclictest0-21swapper/210:17:572
777199118,2cyclictest8322-21expr07:10:141
777199118,2cyclictest8217-21ssh09:42:451
777199118,2cyclictest8085-21sh09:16:081
777199118,2cyclictest6702-21ssh12:31:561
777199118,2cyclictest6677-21tail08:15:281
777199118,2cyclictest5638-21ssh10:57:011
777199118,2cyclictest3325-21ssh11:44:171
777199118,2cyclictest29989-21ssh12:03:021
777199118,2cyclictest28622-21ssh10:25:071
777199118,2cyclictest23228-21sh10:20:211
777199118,2cyclictest19738-21chrt07:33:521
777199118,2cyclictest19738-21chrt07:33:511
777199118,2cyclictest17713-21grep11:30:251
777199118,2cyclictest15808-21head11:05:191
777199118,2cyclictest14381-21ssh12:15:131
777199118,2cyclictest13820-21ssh11:27:541
777199118,2cyclictest10467-21ssh10:35:261
777199118,2cyclictest10299-21ssh12:11:371
777199118,1cyclictest31232-21diskmemload11:55:081
777199110,10cyclictest479-21ntpq09:10:211
777199110,10cyclictest2651-21rm10:30:101
776899118,2cyclictest5797-21cut08:15:170
776899118,2cyclictest4779-21grep12:30:280
776899118,2cyclictest31076-21cut08:00:190
776899118,2cyclictest3059-21tr10:30:170
776899118,2cyclictest2913-21tr08:10:180
776899118,2cyclictest26074-21sh09:56:440
776899118,2cyclictest24791-21grep09:30:170
776899118,2cyclictest24334-21ssh12:21:120
776899118,2cyclictest20638-21head08:45:220
776899118,2cyclictest19134-21sort11:55:170
776899118,2cyclictest16748-21ssh11:30:150
776899118,2cyclictest133402sleep008:30:240
776899111,9cyclictest8049-21fschecks_count08:20:160
776899111,1cyclictest141rcu_preempt10:01:210
777799108,1cyclictest27615-21taskset07:54:353
777799108,1cyclictest19938-21expr08:45:163
777799107,2cyclictest5768-21sed08:15:173
777799107,2cyclictest29701-21chrt07:55:543
777799107,2cyclictest13505-21grep08:30:263
777799100,9cyclictest26194-21taskset08:56:543
777399108,1cyclictest3101-21expr08:10:192
777399108,1cyclictest22894-21expr07:40:202
777399108,1cyclictest15221-21date07:25:012
777399108,1cyclictest10945-21ntp_kernel_pll_08:25:252
777399107,2cyclictest31917-21tr09:10:132
777399106,3cyclictest3533-21grep10:30:202
777199108,1cyclictest22763-21cut07:40:201
777199108,1cyclictest148102sleep107:21:301
777199107,2cyclictest30575-21grep08:00:161
777199107,2cyclictest27513-21chrt07:53:141
777199107,2cyclictest24804-21expr08:55:161
777199107,2cyclictest1501-21grep08:05:221
776899108,1cyclictest13-21ksoftirqd/010:25:100
776899107,2cyclictest4142-21sed10:55:320
776899107,2cyclictest27507-21chrt07:53:090
776899107,2cyclictest16337-21sed07:25:220
776899107,2cyclictest15486-21head08:35:160
776899107,2cyclictest10052-21ssh12:35:010
776899101,2cyclictest0-21swapper/007:30:140
776899101,2cyclictest0-21swapper/007:30:130
77739997,1cyclictest27644-21taskset07:54:572
77719997,1cyclictest22780-21cat08:50:221
77719997,1cyclictest14059-21seq08:32:121
77689997,1cyclictest13-21ksoftirqd/007:55:150
77719986,1cyclictest10137-21expr08:25:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional