You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 03:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Tue May 13, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2272999868,7cyclictest131rcu_preempt19:50:150
2272999717,10cyclictest131rcu_preempt20:05:140
2272999712,5cyclictest131rcu_preempt23:40:160
22742996146,8cyclictest27-21ksoftirqd/220:45:162
2274299601,4cyclictest0-21swapper/222:50:152
2274299581,3cyclictest0-21swapper/223:55:152
2274299551,31cyclictest0-21swapper/200:35:152
2272999542,42cyclictest17687-21ssh21:15:150
2274999518,26cyclictest25866irq/58-eth021:45:153
2274299511,47cyclictest0-21swapper/219:50:152
2272999505,33cyclictest28539-21sendmail-msp23:20:140
22749994919,17cyclictest25866irq/58-eth023:55:143
2273699491,40cyclictest31649-21sendmail-msp21:00:131
2274299482,42cyclictest25426-21sendmail-msp21:20:142
2274299471,25cyclictest21187-21ssh23:35:162
2274299454,4cyclictest0-21swapper/221:35:152
2272999452,4cyclictest131rcu_preempt22:00:130
22742994312,9cyclictest0-21swapper/220:05:142
2272999433,28cyclictest27147-21ssh22:20:140
22742994217,21cyclictest25569-21sendmail-msp21:40:142
2274299420,9cyclictest0-21swapper/222:55:132
2273699422,31cyclictest29491-21sendmail-msp00:00:151
2273699422,31cyclictest29491-21sendmail-msp00:00:141
22742994110,20cyclictest27-21ksoftirqd/219:10:142
22729994114,12cyclictest12-21ksoftirqd/022:05:150
2274299401,21cyclictest19390-21ssh22:35:152
2274299391,25cyclictest3294-21ssh22:45:152
2274299391,15cyclictest20283-21ssh00:15:142
2273699391,32cyclictest27638-21sendmail-msp23:00:171
22749993830,5cyclictest25866irq/58-eth022:10:163
2274299381,8cyclictest4396-21ssh23:25:012
22729993810,21cyclictest12-21ksoftirqd/023:50:150
2274299374,20cyclictest27-21ksoftirqd/221:55:162
2274299373,8cyclictest7340-21kthreadcore20:00:232
2274299371,6cyclictest9288-21sort00:30:022
2274299371,26cyclictest12339-21sendmail-msp20:40:142
22742993712,20cyclictest27-21ksoftirqd/220:10:152
2273699372,25cyclictest27040-21sendmail-msp00:20:171
2272999372,27cyclictest0-21swapper/019:45:150
2272999371,18cyclictest845-21munin-node23:25:150
2274299362,21cyclictest26791-21sendmail-msp22:20:152
2274299352,17cyclictest29059-21sendmail-msp23:41:412
2274299351,9cyclictest27507-21cut22:00:232
2274299351,9cyclictest10618-21kthreadcore21:10:212
2274299351,7cyclictest28686-21irqrtprio23:00:252
2274299351,7cyclictest1157-21ntpq00:00:332
2274299351,7cyclictest1157-21ntpq00:00:322
2274299351,5cyclictest18870-21/usr/sbin/munin22:15:302
22736993516,13cyclictest21-21ksoftirqd/120:55:111
22749993419,11cyclictest25866irq/58-eth023:30:153
2274299347,6cyclictest27211-21sendmail-msp22:40:382
2274299346,21cyclictest25481-1kworker/2:0H21:00:392
2274299345,6cyclictest20728-21tr21:15:252
2274299344,19cyclictest22146irq/40-dwc2_hso21:25:132
2274299342,2cyclictest131rcu_preempt21:30:132
2274299341,9cyclictest7557-21cut19:25:222
2274299341,9cyclictest2367-21cut19:55:202
2274299341,9cyclictest13322-21cut23:30:182
2274299341,8cyclictest6072-21kthreadcore00:05:222
2274299341,7cyclictest20516-21/usr/sbin/munin23:15:242
2274299341,24cyclictest4738-21ssh23:25:162
2274299341,12cyclictest23172-21sed19:40:352
2274299341,11cyclictest16272-21kthreadcore23:50:222
22736993415,13cyclictest26468-21ssh22:00:121
22729993415,13cyclictest25569-21sendmail-msp21:40:160
2272999341,27cyclictest19150-21sendmail-msp19:40:130
2274999333,16cyclictest25866irq/58-eth021:55:153
2274299331,8cyclictest8955-21kthreadcore20:35:222
2274299331,8cyclictest1847-21cpu21:45:162
2274299331,8cyclictest13176-21fschecks_time00:10:192
2274299331,13cyclictest20143-21munin-plugin-st23:15:022
2274299331,12cyclictest31184-21irqrtprio20:25:202
2274299330,8cyclictest3056-21grep20:30:192
2274299330,8cyclictest25679-21/usr/sbin/munin20:20:392
2274299330,8cyclictest25679-21/usr/sbin/munin20:20:382
2272999332,27cyclictest25482-21sendmail-msp20:20:160
2272999332,27cyclictest25482-21sendmail-msp20:20:160
2274299328,3cyclictest8896-21diskmemload21:53:172
2274299321,8cyclictest4254-21tr22:25:232
2274299321,8cyclictest27021-21/usr/sbin/munin20:55:272
2274299321,8cyclictest14545-21/usr/sbin/munin19:35:302
2274299321,10cyclictest7360-21kthreadcore23:45:232
2274299320,9cyclictest12839-21kthreadcore19:30:242
2274299320,8cyclictest30898-21grep00:20:312
2272999327,12cyclictest12181-21ssh23:10:160
22729993215,13cyclictest32281-21sendmail-msp19:20:180
2274999313,11cyclictest25866irq/58-eth020:25:153
22749993114,10cyclictest25866irq/58-eth000:05:173
2274299311,9cyclictest0-21swapper/222:10:202
2274299311,8cyclictest4209-21usage_Anybus19:20:402
2274299311,11cyclictest27770-21forks19:15:182
2273699313,5cyclictest21-21ksoftirqd/119:40:001
2272999314,23cyclictest27638-21sendmail-msp23:00:180
2274999304,12cyclictest25866irq/58-eth000:25:153
2274299301,8cyclictest4564-21tail23:05:222
2274299301,5cyclictest23195-21kthreadcore20:50:222
2273699301,21cyclictest25426-21sendmail-msp21:21:381
2274999292,13cyclictest0-21swapper/321:15:163
2274299292,8cyclictest14945-21switchtime22:30:302
2274299291,8cyclictest21014-21uniq20:15:152
2274299291,8cyclictest10735-21kthreadcore00:30:232
2274299291,10cyclictest0-21swapper/221:05:172
2274299291,10cyclictest0-21swapper/221:05:172
22736992918,8cyclictest27548-21ssh22:20:211
22736992915,11cyclictest21-21ksoftirqd/120:00:151
2274299286,5cyclictest25525-21kthreadcore19:45:212
2274299280,11cyclictest601-21rngd22:07:022
2272999282,21cyclictest16470-21ssh21:33:390
2272999282,19cyclictest1986-21usage_Anybus00:00:410
2272999282,19cyclictest1986-21usage_Anybus00:00:410
2272999281,21cyclictest10091-21ssh21:50:130
22729992811,11cyclictest12-21ksoftirqd/021:07:390
22729992811,11cyclictest12-21ksoftirqd/021:07:380
2274999273,17cyclictest0-21swapper/322:35:153
2274999273,16cyclictest0-21swapper/323:53:403
2274999272,16cyclictest0-21swapper/323:00:193
22749992714,8cyclictest25866irq/58-eth023:45:143
22749992712,11cyclictest0-21swapper/323:10:173
22736992717,8cyclictest25569-21sendmail-msp21:40:331
2272999272,19cyclictest31649-21sendmail-msp21:01:380
2272999272,19cyclictest10323-21usage_Anybus20:00:400
2272999272,18cyclictest27211-21sendmail-msp22:40:120
22729992712,9cyclictest20554-21rm23:15:180
2274999264,11cyclictest25866irq/58-eth022:25:163
2274299261,7cyclictest22770-21sort19:10:002
2272999261,18cyclictest5349-21ssh23:45:140
2272999260,8cyclictest9749-21apt-get19:30:130
2274999254,11cyclictest0-21swapper/321:50:133
2273699256,3cyclictest131rcu_preempt23:55:161
2273699255,4cyclictest201rcuc/122:41:381
22736992511,4cyclictest0-21swapper/100:17:081
2272999251,18cyclictest25426-21sendmail-msp21:20:350
2274999249,12cyclictest0-21swapper/319:30:143
2273699248,4cyclictest21-21ksoftirqd/121:34:231
2273699243,11cyclictest21-21ksoftirqd/122:09:511
2273699242,16cyclictest22221-21apt-get20:50:151
22736992413,8cyclictest19150-21sendmail-msp19:40:311
2274999236,12cyclictest0-21swapper/300:30:163
2274999233,13cyclictest0-21swapper/300:00:413
2274999233,13cyclictest0-21swapper/300:00:413
2274999233,12cyclictest0-21swapper/323:35:153
22749992313,6cyclictest0-21swapper/300:10:163
22749992312,7cyclictest25866irq/58-eth019:45:133
2273699235,3cyclictest131rcu_preempt22:15:301
2273699235,3cyclictest131rcu_preempt19:25:231
2273699232,16cyclictest32281-21sendmail-msp19:20:351
2273699232,16cyclictest28539-21sendmail-msp23:20:171
2272999231,16cyclictest3848-21ssh23:05:170
22749992212,5cyclictest0-21swapper/321:25:133
2273699225,3cyclictest131rcu_preempt21:12:391
2273699225,3cyclictest131rcu_preempt00:30:271
2273699224,2cyclictest131rcu_preempt21:55:171
2273699223,4cyclictest21-21ksoftirqd/121:45:181
2273699222,6cyclictest131rcu_preempt21:25:251
22736992212,7cyclictest23559-21kthreadcore23:35:251
2272999229,10cyclictest0-21swapper/020:35:160
2272999222,14cyclictest24625-21ssh21:38:380
2272999221,17cyclictest4741-21missed_timers19:55:240
2272999221,17cyclictest12714-21cut20:40:190
2272999221,16cyclictest27369-21ssh00:20:150
2272999221,14cyclictest12905-21ssh23:30:140
2274999219,7cyclictest0-21swapper/322:45:143
2274999219,6cyclictest0-21swapper/322:50:133
2274999213,8cyclictest0-21swapper/300:35:163
2274999213,16cyclictest0-21swapper/322:20:143
2274999213,12cyclictest0-21swapper/323:05:413
2274999213,12cyclictest0-21swapper/322:40:133
2274999213,12cyclictest0-21swapper/320:00:403
22749992113,3cyclictest0-21swapper/322:05:143
2273699219,4cyclictest21-21ksoftirqd/122:37:461
2273699216,4cyclictest21-21ksoftirqd/121:50:581
2273699216,4cyclictest201rcuc/122:50:151
2273699214,3cyclictest131rcu_preempt20:15:141
2273699214,14cyclictest29405-21sh23:40:141
2273699213,7cyclictest131rcu_preempt23:11:161
2273699213,3cyclictest131rcu_preempt19:55:251
2273699212,4cyclictest21-21ksoftirqd/121:35:261
2273699212,3cyclictest131rcu_preempt20:35:271
2273699211,3cyclictest131rcu_preempt20:20:391
2273699211,3cyclictest131rcu_preempt20:20:381
2273699211,16cyclictest32756-21ls19:50:311
2273699211,14cyclictest845-21munin-node19:30:151
2272999211,17cyclictest14953-21ls22:30:290
2274999209,7cyclictest0-21swapper/323:20:593
2274999203,11cyclictest0-21swapper/321:37:393
2274999201,12cyclictest0-21swapper/323:25:153
2273699205,3cyclictest131rcu_preempt23:50:281
2273699205,3cyclictest131rcu_preempt22:25:291
2273699204,3cyclictest131rcu_preempt20:45:261
2273699203,4cyclictest131rcu_preempt23:06:571
2273699203,3cyclictest131rcu_preempt23:30:201
2273699202,8cyclictest21-21ksoftirqd/121:17:501
2273699202,3cyclictest131rcu_preempt23:19:421
2273699202,3cyclictest131rcu_preempt00:12:171
2273699202,3cyclictest131rcu_preempt00:05:071
2273699201,5cyclictest21-21ksoftirqd/122:31:361
2273699201,4cyclictest21-21ksoftirqd/100:25:211
2273699200,18cyclictest14101-21latency22:10:261
2272999205,9cyclictest601-21rngd19:35:140
2272999201,8cyclictest9153-21munin-run00:30:020
22729992013,5cyclictest0-21swapper/019:10:270
22729992012,5cyclictest0-21swapper/021:10:170
2272999201,15cyclictest2949-21apt-get20:30:150
22729992010,7cyclictest6373-21memory22:25:270
22729992010,6cyclictest13110-21cut22:50:210
2272999200,16cyclictest8600-21ls19:25:280
2274999195,8cyclictest0-21swapper/300:15:143
2274999193,11cyclictest0-21swapper/321:12:403
2274999193,11cyclictest0-21swapper/319:13:383
2274999192,11cyclictest0-21swapper/322:00:153
2274999192,11cyclictest0-21swapper/321:03:383
2274999192,11cyclictest0-21swapper/320:50:393
2274999192,11cyclictest0-21swapper/319:22:403
2274999192,10cyclictest0-21swapper/319:15:173
22749991913,2cyclictest0-21swapper/322:55:133
22749991912,4cyclictest0-21swapper/321:40:363
22749991912,4cyclictest0-21swapper/300:21:203
2274999191,12cyclictest0-21swapper/320:55:223
2274999191,12cyclictest0-21swapper/320:45:313
2274999191,11cyclictest0-21swapper/321:05:163
2274999191,11cyclictest0-21swapper/321:05:153
2274999191,11cyclictest0-21swapper/319:55:153
22749991910,6cyclictest0-21swapper/323:18:313
2273699196,2cyclictest131rcu_preempt19:45:301
2273699195,3cyclictest131rcu_preempt20:10:011
2273699194,3cyclictest131rcu_preempt23:25:271
2273699194,3cyclictest131rcu_preempt20:30:211
2273699192,4cyclictest52246usb-storage20:41:401
2273699192,3cyclictest131rcu_preempt22:57:161
2273699192,3cyclictest131rcu_preempt21:05:171
2273699192,3cyclictest131rcu_preempt21:05:161
2273699192,2cyclictest131rcu_preempt00:35:241
2273699191,6cyclictest21-21ksoftirqd/123:45:241
2273699191,3cyclictest601-21rngd22:48:191
2272999191,7cyclictest20964-21irqstats00:15:210
22729991912,5cyclictest24647-21ssh22:16:540
2272999191,13cyclictest601-21rngd20:50:310
2274999189,6cyclictest25866irq/58-eth021:30:143
2274999188,4cyclictest0-21swapper/320:35:173
2274999183,10cyclictest0-21swapper/322:31:413
2274999183,10cyclictest0-21swapper/319:40:393
2274999182,11cyclictest0-21swapper/321:23:393
2274999182,10cyclictest25866irq/58-eth022:15:153
22749991812,4cyclictest0-21swapper/323:40:143
2273699182,3cyclictest131rcu_preempt19:15:181
2272999187,8cyclictest17415-21apt-get20:45:160
22729991811,5cyclictest0-21swapper/023:55:240
2272999181,10cyclictest19544-21cut22:35:190
2274999177,6cyclictest0-21swapper/319:50:153
2274999173,7cyclictest0-21swapper/320:40:143
2274999172,9cyclictest0-21swapper/320:10:053
22749991712,2cyclictest0-21swapper/320:05:143
2274999171,12cyclictest0-21swapper/320:30:253
2273699179,3cyclictest21-21ksoftirqd/120:10:041
2273699174,3cyclictest131rcu_preempt20:25:301
2273699174,3cyclictest131rcu_preempt19:10:151
2272999179,5cyclictest7708-21sshd21:27:390
2272999178,6cyclictest601-21rngd22:45:250
2272999173,11cyclictest10046-21ps00:30:210
2272999171,5cyclictest10747-21df_inode22:10:180
2272999171,13cyclictest5455-21ssh00:05:180
2272999171,12cyclictest1988-21/usr/sbin/munin21:45:170
2272999171,11cyclictest601-21rngd23:35:320
2272999171,11cyclictest25849-21ssh21:59:380
2272999168,6cyclictest27670-21apt-get19:15:160
2272999166,7cyclictest0-21swapper/020:15:120
2272999163,10cyclictest15607-21ssh00:10:250
2272999162,10cyclictest601-21rngd20:55:400
2272999161,12cyclictest30584-21apt-get20:25:150
2272999161,12cyclictest19949-21kthreadcore00:35:250
2272999161,11cyclictest20628-21cron20:15:020
2272999161,11cyclictest19904-21uniq22:55:170
2272999160,5cyclictest22739-21latency_hist19:10:010
2274999153,7cyclictest0-21swapper/320:20:163
2274999153,7cyclictest0-21swapper/320:20:163
2274999149,3cyclictest321rcuc/320:15:303
2274999144,7cyclictest0-21swapper/319:35:133
2273699141,10cyclictest22806-21wc19:10:011
2274999138,3cyclictest0-21swapper/319:25:043
2274999101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional