You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-16 - 14:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue Jul 16, 2024 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2472099702,53cyclictest13025-21ssh21:30:130
2472099613,38cyclictest2545-21ssh23:20:150
2473299604,48cyclictest3521-21ssh23:40:121
2472099492,11cyclictest131rcu_preempt21:35:110
2474899465,26cyclictest25866irq/58-eth022:10:163
2472099463,4cyclictest131rcu_preempt22:35:110
2474099454,10cyclictest131rcu_preempt00:10:102
24740994512,26cyclictest19639-21cat22:10:422
2473299454,29cyclictest28627-21ssh23:55:121
24732994538,5cyclictest11608-21diskmemload00:25:111
2472099453,35cyclictest25242-21ssh22:55:110
2473299434,32cyclictest629-21tcf-agent20:45:151
2474099408,22cyclictest27-21ksoftirqd/220:55:142
2473299404,28cyclictest11932-21ssh23:45:151
2472099401,29cyclictest4139-21ssh00:20:110
2474099382,28cyclictest24961-21ssh00:30:432
2474099373,26cyclictest261rcuc/200:00:122
2474099373,26cyclictest261rcuc/200:00:122
2474099373,17cyclictest7690-21ssh22:25:112
2474099372,18cyclictest5709-21ssh22:22:352
2472099371,22cyclictest11608-21diskmemload21:25:140
2472099371,22cyclictest11608-21diskmemload21:25:140
2474899364,20cyclictest25866irq/58-eth021:40:113
2474099364,22cyclictest27-21ksoftirqd/221:40:122
24732993619,8cyclictest9824-21apt-get23:05:161
2472099363,26cyclictest11608-21diskmemload00:35:150
24740993515,13cyclictest27-21ksoftirqd/221:15:152
24732993525,6cyclictest0-21swapper/120:25:131
24732993524,6cyclictest11608-21diskmemload22:50:131
2474899343,17cyclictest0-21swapper/323:40:123
2474899342,18cyclictest25866irq/58-eth021:20:133
2474099341,24cyclictest25254-21ssh22:55:122
2474899335,16cyclictest25866irq/58-eth022:40:143
2474899334,21cyclictest25866irq/58-eth000:20:123
2474099333,19cyclictest22146irq/40-dwc2_hso20:20:122
2474099332,25cyclictest0-21swapper/222:02:282
24748993214,12cyclictest25866irq/58-eth000:30:143
2474099321,15cyclictest32624-21ssh22:40:142
24748993116,10cyclictest25866irq/58-eth022:45:133
2474099312,25cyclictest28616-21ssh23:55:122
2474099311,23cyclictest22728-21ssh21:55:142
2472099315,19cyclictest20338-21ssh21:15:170
2474099305,13cyclictest22146irq/40-dwc2_hso19:55:152
2474099303,18cyclictest22146irq/40-dwc2_hso22:15:152
24740993021,4cyclictest0-21swapper/223:50:142
2473299301,23cyclictest22706-21apt-get21:55:131
2474899292,16cyclictest25866irq/58-eth023:55:133
2474099292,21cyclictest16306-21cat23:45:452
2474099292,20cyclictest0-21swapper/220:30:142
2474099292,11cyclictest12424-21ssh21:29:432
2474099292,11cyclictest12424-21ssh21:29:432
2474899285,12cyclictest25966irq/59-eth023:50:143
2474099283,20cyclictest0-21swapper/200:39:442
2474099283,14cyclictest22146irq/40-dwc2_hso21:00:122
2474099282,21cyclictest0-21swapper/200:15:152
2474099282,19cyclictest0-21swapper/219:15:152
2472099282,21cyclictest12011-21ssh21:10:150
2472099282,20cyclictest4854-21ssh22:02:280
24720992817,6cyclictest24715-21cyclictest22:15:170
2474099273,8cyclictest0-21swapper/200:25:102
2474099272,20cyclictest0-21swapper/223:39:442
2473299273,19cyclictest11484-21ssh22:25:341
2473299271,19cyclictest1334-21ssh23:00:131
2473299270,6cyclictest21-21ksoftirqd/121:53:471
2472099273,3cyclictest131rcu_preempt20:40:150
2472099272,19cyclictest27333-21sh23:53:430
2474899262,20cyclictest25866irq/58-eth022:50:153
24748992613,7cyclictest0-21swapper/300:10:103
2474899261,14cyclictest25866irq/58-eth023:15:123
24748992610,9cyclictest25866irq/58-eth021:35:123
2474099263,16cyclictest22146irq/40-dwc2_hso20:25:122
2474099263,16cyclictest22146irq/40-dwc2_hso19:50:122
2474099263,16cyclictest22146irq/40-dwc2_hso19:50:122
2474099263,15cyclictest22146irq/40-dwc2_hso20:45:152
2473299264,12cyclictest21-21ksoftirqd/121:17:521
2473299262,18cyclictest32635-21ssh22:40:151
24732992612,9cyclictest201rcuc/121:31:441
2473299261,18cyclictest1220-21ssh21:20:401
2473299261,16cyclictest26557-21sh23:15:121
24720992619,3cyclictest0-21swapper/022:25:100
24748992517,5cyclictest0-21swapper/321:10:143
2474099254,13cyclictest22146irq/40-dwc2_hso21:50:122
2474099253,18cyclictest22146irq/40-dwc2_hso22:30:172
2474099253,16cyclictest22146irq/40-dwc2_hso20:50:162
2474099253,13cyclictest22146irq/40-dwc2_hso23:30:122
2474099252,18cyclictest13952-21ssh22:08:432
2474099252,16cyclictest22146irq/40-dwc2_hso23:20:142
2473299254,4cyclictest21-21ksoftirqd/123:13:181
2473299253,5cyclictest21-21ksoftirqd/100:31:411
2473299252,5cyclictest21-21ksoftirqd/123:40:011
24732992520,3cyclictest201rcuc/121:14:101
2472099253,15cyclictest27614-21ssh23:35:150
24720992515,5cyclictest0-21swapper/019:55:150
2474099241,16cyclictest22146irq/40-dwc2_hso23:05:172
2473299247,8cyclictest21-21ksoftirqd/121:38:531
2473299246,9cyclictest21-21ksoftirqd/100:05:201
2473299246,10cyclictest21-21ksoftirqd/122:30:511
24732992419,3cyclictest201rcuc/123:50:231
2472099243,15cyclictest0-21swapper/022:10:430
2472099241,17cyclictest0-21swapper/000:30:120
24720992410,9cyclictest7133-21ssh22:24:430
24748992314,6cyclictest25866irq/58-eth022:20:143
24748992312,8cyclictest0-21swapper/323:35:243
2474899231,11cyclictest0-21swapper/300:05:143
2474099239,11cyclictest26655-21df23:15:152
2474099235,15cyclictest7266-21kthreadcore00:20:262
2474099232,6cyclictest0-21swapper/221:48:432
2474099232,18cyclictest17719-21ssh22:50:262
2474099232,16cyclictest0-21swapper/221:32:232
2474099231,7cyclictest24188-21ssh22:35:122
2473299239,4cyclictest21-21ksoftirqd/121:40:441
2473299236,9cyclictest21-21ksoftirqd/121:47:401
2473299236,8cyclictest21-21ksoftirqd/122:37:401
2473299235,3cyclictest131rcu_preempt00:38:331
2473299233,5cyclictest21-21ksoftirqd/122:15:161
2473299233,5cyclictest21-21ksoftirqd/121:10:021
2473299232,4cyclictest21-21ksoftirqd/100:15:261
24732992317,4cyclictest201rcuc/122:04:241
24732992316,3cyclictest7023-21apt-get19:25:141
2473299231,4cyclictest0-21swapper/123:25:141
2473299231,19cyclictest8080-21ls00:00:271
2473299231,19cyclictest8080-21ls00:00:261
2473299231,18cyclictest20467-21ssh00:10:111
2472099236,8cyclictest18386-21sort20:10:180
2472099235,5cyclictest20001-21kthreadcore23:30:220
2472099234,13cyclictest12-21ksoftirqd/022:45:130
2472099232,15cyclictest14181-21ssh22:09:010
2472099230,19cyclictest0-21swapper/022:50:150
2474899222,9cyclictest25866irq/58-eth020:45:153
2474899222,14cyclictest0-21swapper/323:00:133
24748992214,5cyclictest0-21swapper/322:34:383
24748992212,4cyclictest0-21swapper/323:13:333
2474099227,11cyclictest12092-21df21:10:162
2474099222,6cyclictest0-21swapper/221:35:432
2474099222,14cyclictest22146irq/40-dwc2_hso21:20:132
2474099221,3cyclictest0-21swapper/223:40:122
24740992211,8cyclictest27-21ksoftirqd/221:10:012
2473299227,7cyclictest21-21ksoftirqd/123:23:531
2473299226,8cyclictest21-21ksoftirqd/121:26:471
2473299226,8cyclictest21-21ksoftirqd/121:26:471
2473299224,5cyclictest21-21ksoftirqd/100:20:381
2473299224,4cyclictest21-21ksoftirqd/122:10:161
2473299223,4cyclictest21-21ksoftirqd/122:20:421
2473299222,7cyclictest21-21ksoftirqd/119:30:191
2473299222,2cyclictest131rcu_preempt19:20:161
24732992218,3cyclictest201rcuc/122:45:281
24732992211,8cyclictest23389-21sed23:30:381
2473299221,15cyclictest16887-21cut19:35:191
2472099227,3cyclictest131rcu_preempt22:30:370
2472099221,18cyclictest18146-21find23:10:120
2472099221,18cyclictest10808-21find23:25:110
24748992113,5cyclictest0-21swapper/322:18:443
24748992112,5cyclictest0-21swapper/322:35:123
2474099219,10cyclictest27739-21sendmail20:20:012
2474099218,6cyclictest20795-21tr23:10:242
2474099212,9cyclictest11077-21fschecks_time23:25:172
2474099212,3cyclictest131rcu_preempt20:05:242
2474099211,17cyclictest11396-21kthreadcore20:00:322
24740992111,3cyclictest27-21ksoftirqd/219:25:192
24740992110,9cyclictest0-21swapper/200:05:322
2473299214,6cyclictest21-21ksoftirqd/120:10:341
2473299212,16cyclictest21598-21acpi19:40:111
2473299211,7cyclictest9174-21kthreadcore22:05:261
24732992114,5cyclictest201rcuc/122:55:081
2473299211,15cyclictest2297-21cut21:00:161
24732992110,7cyclictest3736-21apt-get19:55:161
2473299210,6cyclictest29912-21/usr/sbin/munin19:45:401
2472099218,3cyclictest131rcu_preempt22:40:040
2472099212,14cyclictest26816-21ssh00:12:530
2472099211,16cyclictest5076-21ssh23:00:320
2474899209,7cyclictest0-21swapper/323:29:103
2474899203,12cyclictest0-21swapper/322:09:013
24748992014,4cyclictest0-21swapper/323:45:353
24748992014,4cyclictest0-21swapper/323:05:383
24748992014,4cyclictest0-21swapper/322:01:583
24748992014,4cyclictest0-21swapper/321:45:243
24748992014,4cyclictest0-21swapper/321:32:113
24748992014,4cyclictest0-21swapper/321:25:183
24748992014,4cyclictest0-21swapper/321:25:183
24748992014,4cyclictest0-21swapper/300:25:283
24748992013,5cyclictest0-21swapper/323:20:143
24748992013,5cyclictest0-21swapper/321:16:353
24748992013,5cyclictest0-21swapper/321:00:183
24748992013,4cyclictest0-21swapper/322:25:123
24748992011,6cyclictest0-21swapper/321:55:143
2474099206,11cyclictest0-21swapper/220:40:012
2474099205,11cyclictest0-21swapper/222:50:022
2474099201,9cyclictest24373-21kthreadcore19:40:252
2474099201,9cyclictest15254-21tune2fs20:40:232
2474099201,16cyclictest18361-21fschecks_count20:10:162
2474099201,10cyclictest131rcu_preempt19:10:392
2474099200,9cyclictest131rcu_preempt19:30:182
2474099200,8cyclictest4350-21cut23:00:262
2473299207,7cyclictest21-21ksoftirqd/119:10:261
2473299205,9cyclictest21-21ksoftirqd/120:05:371
2473299203,11cyclictest0-21swapper/120:55:151
2473299203,11cyclictest0-21swapper/119:15:141
2473299202,6cyclictest21-21ksoftirqd/120:50:171
2473299201,7cyclictest24761-21latency_hist19:10:021
2473299201,4cyclictest21-21ksoftirqd/120:35:011
2472099202,14cyclictest7258-21ssh23:40:310
2472099202,12cyclictest18999-21ssh23:48:430
2472099201,9cyclictest27254-21kthreadcore20:50:290
24720992011,6cyclictest24503-21cron20:50:000
2472099201,16cyclictest10578-21wc19:25:370
2472099200,16cyclictest12127-21idleruntime19:30:190
24748991913,4cyclictest0-21swapper/323:30:283
24748991913,4cyclictest0-21swapper/322:56:303
24748991913,4cyclictest0-21swapper/321:50:123
24748991913,4cyclictest0-21swapper/320:10:253
24748991913,4cyclictest0-21swapper/320:10:023
24748991913,4cyclictest0-21swapper/319:50:393
24748991913,4cyclictest0-21swapper/319:50:393
24748991913,4cyclictest0-21swapper/319:25:323
24748991913,4cyclictest0-21swapper/319:20:163
24748991913,4cyclictest0-21swapper/300:35:413
24748991913,4cyclictest0-21swapper/300:15:183
24748991913,4cyclictest0-21swapper/300:00:273
24748991913,4cyclictest0-21swapper/300:00:273
24748991912,4cyclictest0-21swapper/319:10:313
24748991911,6cyclictest0-21swapper/320:25:193
2474099192,9cyclictest24791-21latency_hist19:10:022
2474099192,12cyclictest22146irq/40-dwc2_hso19:45:142
2474099191,9cyclictest18875-21kthreadcore19:35:262
2474099190,15cyclictest2374-21fschecks_time19:20:162
2473299193,5cyclictest0-21swapper/120:00:351
24732991911,6cyclictest3499-21cron19:55:011
24732991911,6cyclictest3499-21cron19:55:011
2473299191,12cyclictest23074-21find20:15:121
2472099193,11cyclictest28537-21apt-get00:15:130
2472099192,14cyclictest14425-21diskstats21:50:150
2472099191,8cyclictest9795-21/usr/sbin/munin23:05:350
2472099191,5cyclictest0-21swapper/020:30:250
24720991912,4cyclictest0-21swapper/020:20:150
2472099191,16cyclictest3524-21latency_hist19:55:010
2472099191,16cyclictest3524-21latency_hist19:55:000
24720991911,5cyclictest0-21swapper/000:25:110
2472099191,10cyclictest4678-21diskstats00:00:180
2472099191,10cyclictest4678-21diskstats00:00:180
2472099191,10cyclictest29593-21cut23:55:230
2472099191,10cyclictest14080-21latency_hist21:50:020
24720991910,7cyclictest29837-21apt-get21:40:120
2472099190,14cyclictest21794-21fschecks_time19:40:170
2472099190,14cyclictest12246-21ls20:00:400
2474899182,9cyclictest0-21swapper/320:55:143
2474899182,10cyclictest0-21swapper/319:40:133
2474899182,10cyclictest0-21swapper/319:35:153
24748991813,3cyclictest0-21swapper/320:30:383
24748991812,4cyclictest0-21swapper/321:05:293
24748991812,4cyclictest0-21swapper/320:20:313
24748991812,4cyclictest0-21swapper/320:15:203
24748991812,4cyclictest0-21swapper/319:45:283
24748991812,4cyclictest0-21swapper/319:15:273
24748991811,4cyclictest0-21swapper/320:50:283
2474899181,12cyclictest0-21swapper/319:30:343
2473299187,3cyclictest16766systemd-network20:40:061
2473299183,8cyclictest21-21ksoftirqd/120:35:451
2472099185,3cyclictest131rcu_preempt20:05:400
2472099182,11cyclictest0-21swapper/021:05:120
2472099181,9cyclictest3577-21kthreadcore20:25:260
2472099181,7cyclictest11382-21tr20:35:250
2472099181,4cyclictest3160-21kthreadcore21:00:210
2472099181,14cyclictest27121-21uname19:45:270
24720991810,6cyclictest17568-21cat00:05:480
2472099180,9cyclictest3016-21cut19:20:220
2472099180,15cyclictest28900-21grep21:20:180
2474899174,8cyclictest321rcuc/320:00:153
24748991711,4cyclictest0-21swapper/320:35:323
24732991710,4cyclictest0-21swapper/120:20:211
2472099179,5cyclictest30859-21kthreadcore19:15:230
2472099178,6cyclictest23222-21rm21:55:210
2472099174,10cyclictest28266-21kthreadcore23:15:240
2472099173,4cyclictest131rcu_preempt19:35:420
2472099171,12cyclictest28373-21ntp_states19:10:350
24720991710,5cyclictest875-21mailstats20:55:390
24748991610,4cyclictest0-21swapper/319:55:253
2472099161,12cyclictest23089-21apt-get20:15:130
24748991510,3cyclictest0-21swapper/319:10:013
2472099151,11cyclictest24617-21latency_hist19:10:020
2474899131,10cyclictest0-21swapper/320:40:313
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional