You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-10-05 - 14:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Sat Oct 05, 2024 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1203699623,36cyclictest16776-21ssh23:40:122
1203699583,35cyclictest8470-21sendmail-msp19:40:112
12044995741,10cyclictest25866irq/58-eth022:45:123
1202599552,44cyclictest14389-21sendmail-msp21:20:121
12025995433,12cyclictest211ksoftirqd/123:25:131
1203699532,38cyclictest900-21snmpd21:50:122
12044995228,13cyclictest25866irq/58-eth023:15:113
1201599522,46cyclictest8462-21ssh23:15:110
12025995015,22cyclictest21-21ksoftirqd/119:30:141
1201599505,42cyclictest22940-21ssh21:25:130
1202599494,3cyclictest131rcu_preempt23:50:121
1202599481,4cyclictest131rcu_preempt22:20:141
12015994718,22cyclictest12-21ksoftirqd/022:05:120
1202599468,24cyclictest21-21ksoftirqd/100:00:151
12044994327,9cyclictest25866irq/58-eth022:40:123
12036994319,16cyclictest0-21swapper/221:15:122
1201599433,8cyclictest52246usb-storage00:20:140
1202599421,8cyclictest131rcu_preempt19:40:111
12036994120,13cyclictest0-21swapper/221:20:122
1204499404,20cyclictest25866irq/58-eth022:55:133
12015993812,19cyclictest12-21ksoftirqd/000:25:150
1202599376,21cyclictest24539-21ssh23:05:111
1201599372,20cyclictest4314-21ssh21:31:250
1201599372,20cyclictest4314-21ssh21:31:240
12015993717,12cyclictest12-21ksoftirqd/023:50:130
1201599365,24cyclictest12-21ksoftirqd/020:10:140
1204499351,20cyclictest25866irq/58-eth021:40:123
1202599352,26cyclictest30629-21ssh21:10:121
1203699343,20cyclictest22146irq/40-dwc2_hso20:20:112
1203699343,20cyclictest22146irq/40-dwc2_hso20:20:102
1204499332,21cyclictest0-21swapper/323:45:113
1203699332,15cyclictest11507-21rm21:56:182
1201599334,21cyclictest1451-21sendmail-msp20:40:120
1203699322,21cyclictest32083-21ssh22:50:152
1203699322,11cyclictest30644-21ssh21:10:132
1202599321,24cyclictest22833-21ssh21:45:131
1202599321,22cyclictest27619-21ssh00:30:131
1201599323,19cyclictest12-21ksoftirqd/019:10:150
1201599322,24cyclictest31946-21ssh22:30:120
1204499312,18cyclictest25866irq/58-eth023:10:123
1203699314,19cyclictest22146irq/40-dwc2_hso00:35:122
1202599313,20cyclictest2294-21ssh00:35:121
1202599312,23cyclictest14487-21sendmail-msp20:20:121
1202599312,23cyclictest14487-21sendmail-msp20:20:121
12025993111,15cyclictest16363-21sendmail-msp23:40:141
1201599312,22cyclictest6255-21ssh21:55:120
12015993112,10cyclictest12-21ksoftirqd/020:30:090
12044993025,3cyclictest25866irq/58-eth022:25:123
1204499302,17cyclictest0-21swapper/322:15:123
1203699303,9cyclictest14156-21sendmail22:00:122
1203699303,18cyclictest22146irq/40-dwc2_hso20:05:132
1203699303,16cyclictest15080-21ssh22:20:122
1203699302,11cyclictest16893-21ntp_192.168.11419:45:262
1201599303,22cyclictest14636-21sendmail-msp21:40:120
1201599303,21cyclictest7923-21ssh22:55:140
1201599303,21cyclictest3961-21kworker/u8:122:15:130
12015993023,2cyclictest0-21swapper/022:10:130
1203699293,21cyclictest0-21swapper/222:44:252
1201599292,22cyclictest23858-21ssh22:45:130
1204499283,16cyclictest0-21swapper/322:08:243
1203699283,20cyclictest0-21swapper/221:26:222
12036992822,2cyclictest23906-21ssh00:05:132
12036992822,2cyclictest23906-21ssh00:05:122
1203699282,21cyclictest4198-21ssh21:31:152
1203699282,21cyclictest4198-21ssh21:31:142
1203699282,19cyclictest0-21swapper/221:05:122
1203699281,21cyclictest30265-21ssh23:26:332
12036992811,11cyclictest27-21ksoftirqd/222:35:132
1201599282,18cyclictest24807-21ssh23:45:120
12015992812,11cyclictest30266-21diskmemload23:35:120
1204499273,17cyclictest0-21swapper/321:31:153
1204499273,17cyclictest0-21swapper/321:31:153
1202599273,19cyclictest7113-21ssh21:35:141
1202599273,19cyclictest7113-21ssh21:35:131
1201599276,12cyclictest0-21swapper/023:10:120
1201599272,19cyclictest19782-21ntp_192.168.11400:00:250
1201599272,19cyclictest12339-21ssh21:36:150
1201599272,19cyclictest12339-21ssh21:36:140
1201599272,17cyclictest845-21munin-node20:25:140
1204499262,16cyclictest25866irq/58-eth000:05:133
1204499262,16cyclictest25866irq/58-eth000:05:133
1203699262,19cyclictest32009-21ssh23:09:242
1203699262,18cyclictest0-21swapper/220:40:142
1203699261,21cyclictest27796-21cut00:30:162
12015992618,5cyclictest12-21ksoftirqd/000:05:120
12015992618,5cyclictest12-21ksoftirqd/000:05:110
1204499257,15cyclictest33-21ksoftirqd/300:19:363
1203699252,9cyclictest0-21swapper/219:30:152
1203699252,19cyclictest0-21swapper/222:05:142
1203699252,17cyclictest0-21swapper/223:55:142
1203699251,8cyclictest7171-21ls21:35:142
1203699251,8cyclictest7171-21ls21:35:142
12036992513,9cyclictest0-21swapper/220:25:222
1202599252,7cyclictest131rcu_preempt21:54:251
1202599252,17cyclictest8324-21ssh23:55:141
1202599251,19cyclictest6480-21ssh21:15:141
1204499247,10cyclictest25866irq/58-eth021:55:123
1204499243,14cyclictest0-21swapper/321:50:253
12044992415,6cyclictest25866irq/58-eth022:20:133
1203699247,13cyclictest900-21snmpd22:10:212
12036992414,7cyclictest20608-21sh23:20:292
12036992413,5cyclictest20670-21ssh21:41:562
1202599247,6cyclictest131rcu_preempt23:10:111
1201599243,16cyclictest481-21ssh23:30:140
1201599241,18cyclictest5878-21ssh00:35:250
12015992410,11cyclictest21615-21cron19:20:000
1201599240,17cyclictest12-21ksoftirqd/019:20:120
1204499235,13cyclictest0-21swapper/322:10:143
1204499231,17cyclictest25866irq/58-eth000:25:133
1203699232,6cyclictest0-21swapper/219:35:252
1203699232,16cyclictest0-21swapper/223:18:252
12036992312,9cyclictest0-21swapper/220:15:212
12036992311,9cyclictest21408-21ssh23:01:002
12036992310,6cyclictest17685-21cat00:20:322
1202599233,5cyclictest131rcu_preempt21:40:121
1202599232,14cyclictest0-21swapper/121:05:101
1201599234,16cyclictest151rcuc/023:00:130
1204499228,8cyclictest0-21swapper/322:50:143
1204499225,11cyclictest0-21swapper/300:02:253
1204499223,13cyclictest0-21swapper/321:25:253
12044992215,5cyclictest25866irq/58-eth022:30:133
12044992213,6cyclictest0-21swapper/323:40:293
1203699222,6cyclictest0-21swapper/219:10:262
12036992214,5cyclictest0-21swapper/221:45:172
12036992211,9cyclictest7026-21ssh23:13:232
1203699221,11cyclictest7979-21cpuspeed22:55:142
12036992210,10cyclictest0-21swapper/219:25:292
1202599222,15cyclictest27825-21ssh22:06:191
12025992210,8cyclictest24172-21ntpq00:25:281
1201599223,4cyclictest12-21ksoftirqd/000:31:460
1201599222,15cyclictest12-21ksoftirqd/020:15:110
12015992214,4cyclictest12-21ksoftirqd/019:35:270
1201599221,17cyclictest73-21kswapd022:50:140
12015992210,3cyclictest131rcu_preempt21:45:220
12044992113,5cyclictest0-21swapper/323:55:243
12044992112,6cyclictest0-21swapper/323:30:273
1203699217,12cyclictest23700-21latency_hist00:05:012
1203699217,11cyclictest10773-21timerwakeupswit22:15:282
1203699212,16cyclictest29305-21kthreadcore20:00:192
1203699211,9cyclictest30266-21diskmemload23:50:212
12036992111,7cyclictest0-21swapper/220:10:222
1203699210,13cyclictest24131-21ls22:45:172
12025992113,5cyclictest31195-21/usr/sbin/munin21:30:271
12025992113,5cyclictest31195-21/usr/sbin/munin21:30:261
12025992112,6cyclictest0-21swapper/120:40:261
1202599211,16cyclictest11261-21apt-get20:50:121
1202599211,15cyclictest8473-21ssh23:15:131
1202599211,15cyclictest201rcuc/122:50:141
1201599217,11cyclictest32600-21tail21:10:190
1201599217,10cyclictest12-21ksoftirqd/019:55:120
1201599215,4cyclictest131rcu_preempt22:25:140
1201599212,7cyclictest22588-21latency_hist21:25:020
1201599211,7cyclictest28486-21grep23:25:260
1201599211,4cyclictest131rcu_preempt20:00:290
12015992113,6cyclictest0-21swapper/020:05:280
1201599211,15cyclictest26682-21apt-get19:25:120
1203699209,5cyclictest10817-21cat00:15:322
1203699208,10cyclictest0-21swapper/221:00:122
1203699206,5cyclictest23333-21cut00:25:222
1203699206,5cyclictest12880-21kthreadcore20:50:192
1203699202,9cyclictest9355-21irqstats23:35:182
1203699202,5cyclictest32698-21sed22:30:182
1203699201,16cyclictest19332-21kthreadcore19:50:212
12036992011,5cyclictest0-21swapper/223:30:132
1203699201,11cyclictest25561-21cat19:20:292
1202599208,5cyclictest131rcu_preempt00:10:111
1202599207,10cyclictest14487-21sendmail-msp20:20:021
1202599202,5cyclictest52246usb-storage22:20:011
1202599201,16cyclictest25447-21cron21:05:011
1202599201,14cyclictest20190-21ssh23:20:241
1201599209,2cyclictest12-21ksoftirqd/021:20:020
1201599208,8cyclictest12004-21cyclictest22:20:260
1201599205,9cyclictest12004-21cyclictest23:20:120
1201599202,2cyclictest131rcu_preempt21:52:120
1201599202,16cyclictest17774-21kthreadcore20:55:220
12015992014,5cyclictest12-21ksoftirqd/023:55:110
12015992014,3cyclictest15286-21sendmail-msp22:40:110
12015992014,3cyclictest12-21ksoftirqd/020:50:250
1201599201,2cyclictest131rcu_preempt23:09:400
1204499199,6cyclictest0-21swapper/300:23:163
1204499193,11cyclictest0-21swapper/322:01:183
1204499193,11cyclictest0-21swapper/321:36:153
1204499193,11cyclictest0-21swapper/321:36:143
12044991911,5cyclictest0-21swapper/323:20:293
1204499191,14cyclictest0-21swapper/320:55:183
1204499191,13cyclictest0-21swapper/319:10:243
1204499191,12cyclictest0-21swapper/319:35:223
12044991910,6cyclictest25866irq/58-eth023:50:143
12044991910,6cyclictest25866irq/58-eth021:20:113
12044991910,6cyclictest0-21swapper/323:04:333
12044991910,5cyclictest25866irq/58-eth020:40:123
1203699196,4cyclictest17684-21irqrtprio19:15:182
1203699194,12cyclictest0-21swapper/220:55:212
1203699191,9cyclictest26806-21kthreadcore23:45:202
1203699191,9cyclictest10346-21sed20:45:282
1203699191,9cyclictest10346-21sed20:45:272
1203699191,8cyclictest32589-21wc20:35:222
12036991912,5cyclictest23521-21ssh22:25:112
1203699191,15cyclictest5994-21latency_hist00:15:012
1202599199,7cyclictest15906-21sendmail-msp23:00:021
1202599196,3cyclictest131rcu_preempt23:35:241
1202599192,15cyclictest486-21ssh23:30:151
1202599191,3cyclictest52246usb-storage22:00:021
12025991911,5cyclictest17332-21usage_Anybus00:20:311
12025991911,5cyclictest0-21swapper/122:00:191
12025991910,6cyclictest30158-21sshd23:46:251
1201599199,7cyclictest16363-21sendmail-msp23:40:040
1201599198,8cyclictest13488-21apt-get19:45:140
1201599196,10cyclictest6577-21if_err_eth100:15:160
1201599194,3cyclictest131rcu_preempt22:35:290
1201599193,10cyclictest0-21swapper/020:35:140
1201599191,13cyclictest845-21munin-node21:00:130
1204499183,11cyclictest0-21swapper/321:19:263
1204499182,10cyclictest0-21swapper/320:35:143
12044991812,4cyclictest0-21swapper/323:28:283
12044991812,4cyclictest0-21swapper/320:30:023
12044991811,4cyclictest0-21swapper/322:40:013
1204499181,12cyclictest0-21swapper/321:05:203
1203699186,5cyclictest23220-21cut19:55:172
1203699181,14cyclictest12051-21apt-config19:10:012
1203699180,8cyclictest24611-21tune2fs20:30:132
1202599187,8cyclictest19219-21nfs_client22:40:241
1202599182,14cyclictest23578-21ssh22:25:131
1202599182,12cyclictest31957-21ssh22:30:131
1202599182,12cyclictest11017-21memory22:35:221
1202599181,9cyclictest11024-21cron20:50:011
1202599181,9cyclictest11024-21cron20:50:001
1202599181,7cyclictest428-21switchtime20:35:281
1202599181,4cyclictest52246usb-storage20:00:091
1201599184,4cyclictest131rcu_preempt20:45:110
1201599184,4cyclictest131rcu_preempt20:45:110
1201599181,3cyclictest131rcu_preempt20:20:110
1201599181,3cyclictest131rcu_preempt20:20:100
12015991812,4cyclictest14860-21kthreadcore22:00:190
1201599181,15cyclictest25810-21fschecks_count21:05:140
1201599181,11cyclictest31704-21uniq00:10:150
1201599180,5cyclictest52246usb-storage19:40:100
1204499179,6cyclictest0-21swapper/323:36:443
1204499179,6cyclictest0-21swapper/321:12:463
1204499179,6cyclictest0-21swapper/300:13:423
1204499172,10cyclictest0-21swapper/320:30:043
1204499171,12cyclictest0-21swapper/319:15:233
1204499171,11cyclictest0-21swapper/323:05:143
12044991710,4cyclictest25866irq/58-eth019:25:123
1202599179,5cyclictest25807-21kthreadcore22:45:211
1202599177,7cyclictest25915-21kthreadcore00:05:201
1202599177,7cyclictest25915-21kthreadcore00:05:191
1202599176,8cyclictest8431-21cron19:40:011
1202599175,9cyclictest21773-21latency_hist19:20:021
1202599173,11cyclictest27294-21sed19:25:191
1202599172,11cyclictest12047-21rm19:10:011
1202599171,8cyclictest9025-21grep22:55:201
1202599171,8cyclictest19508-21ntp_kernel_pll_20:55:251
12025991711,4cyclictest2283-21sh22:10:291
12025991711,4cyclictest0-21swapper/119:50:231
12025991711,4cyclictest0-21swapper/119:20:231
12025991711,4cyclictest0-21swapper/119:10:191
1202599171,12cyclictest12920-21basename00:20:021
12025991710,4cyclictest24915-21kthreadcore21:25:221
12025991710,4cyclictest0-21swapper/120:30:181
1201599177,7cyclictest32528-21kthreadcore19:30:200
1201599172,10cyclictest0-21swapper/019:50:160
12015991711,3cyclictest12-21ksoftirqd/019:10:020
1204499162,8cyclictest0-21swapper/300:35:253
1204499162,7cyclictest0-21swapper/320:50:113
1204499161,8cyclictest0-21swapper/319:45:023
1204499161,12cyclictest0-21swapper/321:45:183
1204499160,13cyclictest25866irq/58-eth000:30:123
1202599168,5cyclictest15751-21cut19:45:231
1202599168,5cyclictest0-21swapper/120:10:241
1202599164,9cyclictest440-21cut20:05:161
1202599161,11cyclictest27836-21latency_hist20:00:021
1202599161,11cyclictest23059-21ntp_states20:25:261
1204499144,6cyclictest0-21swapper/321:00:133
1204499142,9cyclictest0-21swapper/319:10:023
1204499142,8cyclictest0-21swapper/320:20:023
1204499141,7cyclictest0-21swapper/320:20:143
1204499141,7cyclictest0-21swapper/320:20:133
1204499141,7cyclictest0-21swapper/319:45:283
1204499141,10cyclictest0-21swapper/320:10:143
1204499138,3cyclictest0-21swapper/319:55:283
1204499138,3cyclictest0-21swapper/319:20:293
1204499135,5cyclictest25866irq/58-eth019:50:153
1204499131,8cyclictest25866irq/58-eth020:45:133
1204499131,8cyclictest25866irq/58-eth020:45:133
1204499131,10cyclictest0-21swapper/320:05:193
1204499121,8cyclictest0-21swapper/319:30:223
1204499121,7cyclictest0-21swapper/320:05:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional