You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-21 - 20:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Jul 21, 2024 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1227899633,48cyclictest20948-21ssh09:25:110
1227899534,35cyclictest28209-21ssh10:50:100
1230599514,32cyclictest25866irq/58-eth012:00:123
1229799502,26cyclictest14664-21sh12:00:122
1227899501,41cyclictest3902-21ssh09:35:120
1227899492,37cyclictest2782-21ssh09:55:170
1228999469,17cyclictest21-21ksoftirqd/109:20:121
1229799433,32cyclictest15303-21ssh12:20:122
1229799391,15cyclictest31206-21awk12:10:142
1229799391,12cyclictest22146irq/40-dwc2_hso10:45:132
1229799391,12cyclictest22146irq/40-dwc2_hso10:45:132
1228999375,27cyclictest31998-21ssh12:30:151
1229799362,6cyclictest0-21swapper/210:05:332
1229799362,18cyclictest16611-21sh09:40:322
1227899360,22cyclictest0-21swapper/008:35:180
1230599356,13cyclictest25866irq/58-eth010:35:103
1229799353,12cyclictest0-21swapper/212:15:122
12289993520,7cyclictest0-21swapper/112:05:141
12278993530,3cyclictest0-21swapper/011:35:190
12305993418,14cyclictest321rcuc/307:45:123
1227899341,26cyclictest19881-21sh10:45:140
1227899341,26cyclictest19881-21sh10:45:130
1229799333,20cyclictest20807-21ssh11:05:122
1229799331,16cyclictest11138-21ssh09:39:332
1228999333,23cyclictest7872-21ssh12:35:131
1227899332,7cyclictest131rcu_preempt09:15:120
1227899331,24cyclictest11580-21ssh10:40:130
12305993216,10cyclictest25866irq/58-eth009:45:113
12305993215,12cyclictest25866irq/58-eth011:40:133
12305993215,12cyclictest25866irq/58-eth011:40:123
1228999325,21cyclictest20826-21diskmemload11:05:131
1229799314,18cyclictest22146irq/40-dwc2_hso07:30:132
1229799314,18cyclictest22146irq/40-dwc2_hso07:30:122
1229799313,15cyclictest22146irq/40-dwc2_hso11:00:172
1229799311,17cyclictest2381-21ssh10:15:142
1229799305,15cyclictest22146irq/40-dwc2_hso07:25:092
1229799303,17cyclictest22146irq/40-dwc2_hso07:15:152
1228999302,21cyclictest3252-21ssh09:34:331
1227899306,12cyclictest632-21Async17:44:200
1227899303,20cyclictest151rcuc/011:05:130
12278993018,5cyclictest12-21ksoftirqd/007:40:110
1227899301,23cyclictest23646-21ssh12:25:170
1229799293,17cyclictest22146irq/40-dwc2_hso11:15:122
1229799292,17cyclictest22146irq/40-dwc2_hso12:30:152
1229799292,17cyclictest22146irq/40-dwc2_hso11:30:102
1229799292,10cyclictest0-21swapper/209:20:132
1228999293,19cyclictest201rcuc/110:25:121
1228999291,17cyclictest13325-21ssh11:20:141
1227899295,15cyclictest30425-21ssh09:10:110
1227899295,15cyclictest13916-21ssh11:40:120
1227899295,15cyclictest13916-21ssh11:40:120
1227899293,17cyclictest5040-21ssh11:15:130
1227899292,16cyclictest12-21ksoftirqd/011:30:110
1230599284,15cyclictest0-21swapper/309:25:123
1230599283,18cyclictest0-21swapper/309:38:333
1229799283,20cyclictest0-21swapper/212:35:342
1229799282,19cyclictest0-21swapper/208:20:132
1229799281,17cyclictest22146irq/40-dwc2_hso12:05:112
1228999282,18cyclictest201rcuc/111:30:131
1227899283,20cyclictest21625-21ssh11:25:150
1227899283,12cyclictest12-21ksoftirqd/007:15:170
1227899282,18cyclictest6957-21ssh12:15:120
12305992719,5cyclictest0-21swapper/310:50:103
1229799273,15cyclictest22146irq/40-dwc2_hso11:50:122
1228999272,19cyclictest19369-21ssh09:45:111
1228999271,21cyclictest31072-21ssh12:10:111
1230599267,16cyclictest33-21ksoftirqd/311:14:093
1230599263,15cyclictest0-21swapper/312:08:333
1230599263,11cyclictest0-21swapper/310:40:133
1230599262,18cyclictest0-21swapper/309:53:333
1229799266,13cyclictest22146irq/40-dwc2_hso10:30:122
1229799262,17cyclictest22146irq/40-dwc2_hso07:10:112
1229799261,9cyclictest29177-21ssh11:10:122
1229799261,21cyclictest856-21ssh09:52:322
1229799261,16cyclictest22146irq/40-dwc2_hso12:25:172
1228999262,19cyclictest3206-21ssh10:35:121
12289992613,9cyclictest30977-21ssh09:50:281
12289992613,10cyclictest22008-21ssh10:45:221
12289992613,10cyclictest22008-21ssh10:45:211
1227899262,17cyclictest22212-21ssh11:45:140
1230599253,18cyclictest25866irq/58-eth008:50:123
1230599253,14cyclictest0-21swapper/309:40:333
1230599253,11cyclictest0-21swapper/312:32:333
1230599252,15cyclictest25866irq/58-eth012:25:163
1230599251,13cyclictest25866irq/58-eth012:10:113
12305992511,11cyclictest25866irq/58-eth011:25:143
1229799252,6cyclictest0-21swapper/208:50:122
1229799251,4cyclictest1849266ssh11:40:332
1229799251,4cyclictest1849266ssh11:40:322
12297992512,6cyclictest15846-21ssh10:02:122
1228999258,5cyclictest21-21ksoftirqd/110:15:131
1228999252,18cyclictest16607-21sh09:40:331
1228999252,14cyclictest21-21ksoftirqd/109:00:101
12289992512,9cyclictest26903-21ssh12:25:281
12289992512,10cyclictest21777-21cron07:20:021
1228999251,19cyclictest26530-21ssh10:10:111
1227899253,14cyclictest0-21swapper/012:20:120
1227899252,17cyclictest18081-21ssh10:05:150
1230599244,14cyclictest0-21swapper/309:15:123
1230599243,11cyclictest25866irq/58-eth009:20:123
1230599242,14cyclictest33-21ksoftirqd/311:50:123
12305992412,9cyclictest0-21swapper/311:05:203
12305992411,9cyclictest0-21swapper/312:15:123
1229799243,14cyclictest22146irq/40-dwc2_hso09:10:132
1229799241,16cyclictest22146irq/40-dwc2_hso09:45:122
1227899241,15cyclictest19002-21ssh10:25:130
1230599239,8cyclictest25866irq/58-eth011:45:143
1230599233,13cyclictest0-21swapper/309:34:333
1230599232,16cyclictest25866irq/58-eth010:30:103
1229799233,10cyclictest9424-21kthreadcore07:40:172
1229799232,18cyclictest0-21swapper/209:05:122
1229799232,16cyclictest0-21swapper/207:35:332
1229799232,15cyclictest0-21swapper/210:25:132
1229799231,9cyclictest3191-21ssh10:35:112
12297992311,9cyclictest28960-21ssh09:30:152
1229799230,18cyclictest17010-21ssh11:20:272
1228999232,18cyclictest14797-21apt-get08:20:121
1228999231,15cyclictest1579-21find08:40:121
1228999231,15cyclictest1579-21find08:40:111
12289992311,3cyclictest0-21swapper/111:25:141
1227899231,18cyclictest18258-21ssh12:00:250
1227899231,16cyclictest845-21munin-node07:55:120
1227899231,16cyclictest7956-21df12:35:130
12278992310,8cyclictest30549-21rm11:50:130
1230599222,12cyclictest0-21swapper/311:00:183
1229799229,5cyclictest27849-21sendmail08:00:012
1229799227,12cyclictest0-21swapper/208:15:182
1229799226,8cyclictest10271-21tr10:00:022
1229799223,11cyclictest18644-21apt10:25:012
1229799222,4cyclictest0-21swapper/210:10:102
1229799222,16cyclictest514-21fschecks_count08:05:162
1229799222,15cyclictest0-21swapper/207:20:342
1229799221,18cyclictest23769-21kthreadcore11:45:202
12297992210,9cyclictest10538-21ssh09:15:572
12297992210,5cyclictest0-21swapper/210:50:322
1228999229,10cyclictest24111-21cron08:30:001
1228999229,10cyclictest22424-21ssh10:05:321
1228999223,13cyclictest0-21swapper/110:40:141
1228999222,16cyclictest30067-21diskmemload11:35:181
1228999222,14cyclictest15289-21ssh12:20:121
12289992213,6cyclictest4996-21/usr/sbin/munin11:15:291
1228999221,18cyclictest12629-21uniq07:10:151
1228999221,17cyclictest10534-21diskmemload09:38:321
1227899227,8cyclictest12-21ksoftirqd/008:45:130
1227899224,7cyclictest23058-21fw_packets12:05:160
1227899222,15cyclictest1539-21ssh09:53:330
1227899222,14cyclictest24812-21ssh09:46:330
12278992212,7cyclictest11235-21sendmail-msp10:40:000
1227899221,18cyclictest8502-21latency_hist07:40:020
1227899221,15cyclictest25668-21cut09:05:160
1227899221,14cyclictest6286-21ls08:45:020
1227899221,14cyclictest6286-21ls08:45:020
1230599213,13cyclictest0-21swapper/311:30:133
1230599212,11cyclictest0-21swapper/310:10:113
1229799211,7cyclictest22503-21kthreadcore11:25:222
1229799211,17cyclictest0-21swapper/208:40:222
1229799211,17cyclictest0-21swapper/208:40:212
1229799211,10cyclictest15698-21unixbench_singl10:40:312
1228999213,3cyclictest131rcu_preempt07:40:011
1228999211,17cyclictest28371-21grep10:50:141
1228999211,17cyclictest12590-21kthreadcore10:00:211
1227899218,4cyclictest131rcu_preempt10:15:160
1227899213,3cyclictest131rcu_preempt09:40:100
1227899213,15cyclictest28769-21ssh09:30:120
1227899211,15cyclictest30826-21ssh10:30:230
12278992110,8cyclictest32155-21tune2fs12:30:180
12278992110,7cyclictest121ksoftirqd/011:00:180
1230599204,12cyclictest25866irq/58-eth009:10:103
1230599202,11cyclictest25866irq/58-eth010:20:133
12305992012,5cyclictest25866irq/58-eth010:15:123
12305992010,7cyclictest25866irq/58-eth010:25:123
1229799203,6cyclictest20954-21ssh09:25:122
1229799202,15cyclictest21520-21kthreadcore07:50:212
1229799201,9cyclictest31804-21timerwakeupswit08:00:312
1229799201,9cyclictest31804-21timerwakeupswit08:00:302
1229799201,8cyclictest6762-21ssh11:55:192
1229799201,8cyclictest20563-21grep09:00:012
1229799201,13cyclictest22146irq/40-dwc2_hso08:35:192
1229799201,10cyclictest27953-21grep08:30:272
1229799200,10cyclictest5945-21grep11:35:222
1228999204,12cyclictest14961-21sendmail-msp12:20:011
1228999203,12cyclictest0-21swapper/108:45:131
1228999202,15cyclictest18806-21irqcore07:50:171
1228999202,10cyclictest2791-21ssh09:55:171
1228999201,9cyclictest131rcu_preempt11:50:171
1228999201,5cyclictest131rcu_preempt08:10:131
12289992013,4cyclictest13886-21/usr/sbin/munin11:40:181
12289992013,4cyclictest13886-21/usr/sbin/munin11:40:181
12289992012,5cyclictest17864-21ssh12:00:211
1228999200,13cyclictest5706-21ls09:15:151
1227899203,9cyclictest28528-21kthreadcore10:10:200
1227899201,10cyclictest13488-21fschecks_count11:20:170
1230599199,6cyclictest25866irq/58-eth012:20:283
1230599193,10cyclictest0-21swapper/308:35:173
12305991911,6cyclictest0-21swapper/311:37:203
1230599191,13cyclictest0-21swapper/307:15:203
1230599191,12cyclictest0-21swapper/310:55:163
1230599191,12cyclictest0-21swapper/309:55:173
1230599191,12cyclictest0-21swapper/307:35:143
12305991910,6cyclictest25866irq/58-eth011:20:143
12305991910,6cyclictest0-21swapper/311:55:583
12305991910,6cyclictest0-21swapper/311:18:513
1229799198,4cyclictest4067-21/usr/sbin/munin10:55:112
1229799192,8cyclictest22098-21kthreadcore09:00:182
1229799191,9cyclictest5336-21fschecks_count08:10:162
1229799191,8cyclictest13949-21sed07:45:182
1229799191,10cyclictest23174-21switchtime08:25:292
1229799190,12cyclictest6366-21find08:45:102
1228999197,10cyclictest0-21swapper/111:10:271
1228999196,10cyclictest5226-21diskmemload09:15:011
1228999194,3cyclictest131rcu_preempt10:35:011
1228999194,11cyclictest28899-21munin-plugin-st08:35:011
1228999192,13cyclictest10712-21ssh10:20:151
1228999191,9cyclictest10273-21switchtime11:55:301
1228999191,16cyclictest19203-21ls08:55:211
1228999191,14cyclictest11158-21apt-get08:50:111
1228999191,10cyclictest11300-21kthreadcore07:40:201
12289991910,6cyclictest12689-21ssh11:00:221
1227899199,7cyclictest27682-21cat08:30:230
1227899197,7cyclictest632-21Async17:44:200
1227899197,3cyclictest131rcu_preempt09:20:320
1227899194,10cyclictest0-21swapper/011:55:130
12278991911,6cyclictest3562-21sshd12:10:320
1227899191,13cyclictest11225-21irqcore10:20:200
1230599189,6cyclictest33-21ksoftirqd/307:25:123
1230599189,5cyclictest25866irq/58-eth010:05:123
1230599182,10cyclictest0-21swapper/308:40:133
1230599182,10cyclictest0-21swapper/308:40:133
1230599182,10cyclictest0-21swapper/308:15:143
1230599182,10cyclictest0-21swapper/308:05:153
1230599182,10cyclictest0-21swapper/308:00:143
1230599182,10cyclictest0-21swapper/308:00:133
1230599182,10cyclictest0-21swapper/307:40:133
12305991811,5cyclictest25866irq/58-eth010:00:123
12305991811,4cyclictest0-21swapper/310:45:283
12305991811,4cyclictest0-21swapper/310:45:283
1230599181,12cyclictest0-21swapper/308:30:143
1230599181,12cyclictest0-21swapper/308:20:293
1230599181,11cyclictest0-21swapper/312:35:283
1230599181,11cyclictest0-21swapper/309:05:113
1228999182,4cyclictest131rcu_preempt08:05:171
1228999182,11cyclictest0-21swapper/108:00:131
1228999182,11cyclictest0-21swapper/108:00:131
1228999181,2cyclictest131rcu_preempt11:45:141
1228999181,14cyclictest25604-21kthreadcore07:55:191
12289991811,4cyclictest21-21ksoftirqd/108:40:011
12289991811,4cyclictest0-21swapper/108:15:241
12289991810,5cyclictest8154-21unixbench_multi10:55:281
12289991810,4cyclictest22897-21kthreadcore09:25:211
1227899183,11cyclictest0-21swapper/007:25:100
12278991812,4cyclictest0-21swapper/011:10:110
12278991811,5cyclictest0-21swapper/007:50:280
1227899181,14cyclictest1322-21kthreadcore08:05:220
1227899181,14cyclictest10162-21cut08:15:160
1230599172,9cyclictest0-21swapper/309:00:113
1230599172,9cyclictest0-21swapper/308:45:133
1230599172,9cyclictest0-21swapper/307:55:113
1230599172,9cyclictest0-21swapper/307:20:133
1230599171,10cyclictest0-21swapper/308:25:293
1230599171,10cyclictest0-21swapper/307:30:143
1230599171,10cyclictest0-21swapper/307:30:143
1228999178,5cyclictest22451-21irqcore07:20:191
1228999174,10cyclictest15217-21kthreadcore07:45:211
1228999172,11cyclictest26893-21diskstats07:25:131
1228999171,13cyclictest27909-21cut09:05:221
1227899175,9cyclictest29055-21kthreadcore08:00:210
1227899175,9cyclictest29055-21kthreadcore08:00:210
1227899175,5cyclictest12-21ksoftirqd/008:55:110
1227899172,11cyclictest0-21swapper/008:10:120
1227899171,12cyclictest7764-21ntp_states10:55:260
1230599162,9cyclictest0-21swapper/307:10:113
1230599162,8cyclictest0-21swapper/307:50:043
1228999161,12cyclictest2214-21kthreadcore07:30:221
1228999161,12cyclictest2214-21kthreadcore07:30:211
1227899168,5cyclictest12-21ksoftirqd/007:10:110
1227899162,9cyclictest0-21swapper/007:45:120
1227899162,11cyclictest31715-21fschecks_count07:30:160
1227899162,11cyclictest31715-21fschecks_count07:30:160
1227899162,11cyclictest22523-21kthreadcore09:00:190
1227899162,10cyclictest11632-21irqcore08:50:180
1227899161,12cyclictest21009-21kthreadcore08:25:210
1227899161,12cyclictest18529-21wc08:20:290
1230599152,9cyclictest0-21swapper/308:10:123
1228999153,9cyclictest12210-21latency_hist07:10:021
1227899151,11cyclictest12315-21latency_hist07:10:020
1229799131,9cyclictest12356-21latency_hist07:10:012
1230599121,9cyclictest0-21swapper/308:55:193
1230599111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional