You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-03 - 00:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri May 02, 2025 12:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9467996415,6cyclictest131rcu_preempt10:40:171
9486995938,11cyclictest25866irq/58-eth011:50:143
946799572,39cyclictest21203-21ssh11:15:141
947699561,35cyclictest24940-21sendmail-msp08:00:132
945599543,45cyclictest11898-21ssh10:05:140
9467995319,27cyclictest2126-21sendmail-msp12:20:131
945599534,40cyclictest11666-21sendmail-msp08:20:170
945599534,39cyclictest16543-21ssh12:10:120
947699521,38cyclictest11601-21ssh12:25:142
946799512,35cyclictest12728-21ssh11:30:161
9455995116,26cyclictest12-21ksoftirqd/010:10:140
945599501,42cyclictest4297-21sendmail-msp10:00:150
946799496,5cyclictest131rcu_preempt10:15:151
946799496,5cyclictest131rcu_preempt10:15:141
945599491,38cyclictest30583-21sendmail-msp12:00:150
947699483,41cyclictest0-21swapper/207:50:112
947699461,40cyclictest6665-21sendmail-msp09:40:142
945599463,30cyclictest2300-21sendmail-msp10:20:140
945599462,7cyclictest131rcu_preempt12:25:140
947699442,34cyclictest22146irq/40-dwc2_hso09:45:182
945599423,35cyclictest20685-21ssh12:30:130
945599423,28cyclictest17712-21sendmail-msp09:00:160
9455994224,11cyclictest10118-21sendmail-msp09:20:130
947699416,12cyclictest27-21ksoftirqd/208:55:152
9467994136,3cyclictest28701-21sendmail-msp11:20:121
947699401,24cyclictest18914-21sendmail-msp07:20:142
946799403,2cyclictest0-21swapper/111:45:121
945599401,6cyclictest131rcu_preempt08:10:150
945599401,26cyclictest28647-21sendmail-msp11:40:160
948699393,26cyclictest25866irq/58-eth008:30:143
9476993913,17cyclictest0-21swapper/208:10:152
946799393,2cyclictest601-21rngd09:25:141
945599391,32cyclictest5745-21sendmail-msp07:40:130
945599391,28cyclictest28701-21sendmail-msp11:20:100
948699382,20cyclictest25866irq/58-eth010:25:153
9476993828,8cyclictest18659-21ssh11:51:392
946799383,3cyclictest25807-21ssh12:15:131
945599382,5cyclictest131rcu_preempt09:35:160
945599381,31cyclictest27313-21diskmemload10:35:150
947699372,27cyclictest601-21rngd11:03:392
946799371,7cyclictest30641-21sendmail-msp11:00:241
946799371,5cyclictest0-21swapper/109:55:171
946799371,3cyclictest601-21rngd09:05:141
946799363,3cyclictest0-21swapper/108:45:131
946799362,8cyclictest30862-21sendmail-msp08:40:161
946799361,5cyclictest52246usb-storage08:20:321
946799357,19cyclictest17675-21ssh10:30:161
9486993421,8cyclictest25866irq/58-eth009:45:183
947699342,24cyclictest7409-21ssh12:05:132
9476993410,16cyclictest601-21rngd09:55:182
946799347,24cyclictest2300-21sendmail-msp10:21:341
946799342,28cyclictest15787-21grep12:25:331
946799341,9cyclictest3822-21grep09:35:321
946799341,8cyclictest4297-21sendmail-msp10:00:241
946799341,4cyclictest0-21swapper/107:45:171
947699332,25cyclictest16545-21ssh10:28:392
947699332,21cyclictest22146irq/40-dwc2_hso08:40:172
946799338,21cyclictest667-21cat11:40:381
946799331,9cyclictest24012-21ps10:55:201
946799331,9cyclictest24012-21ps10:55:191
946799331,6cyclictest25156-21apt-get08:00:121
946799331,6cyclictest19532-21ssh10:50:261
946799331,6cyclictest19532-21ssh10:50:261
945599334,14cyclictest131rcu_preempt11:55:140
946799321,8cyclictest13555-21latency_hist11:10:021
946799321,6cyclictest11067-21timerwakeupswit09:40:361
946799321,10cyclictest5760-21sort07:40:011
947699314,21cyclictest601-21rngd12:10:132
947699313,16cyclictest22146irq/40-dwc2_hso08:05:122
947699312,22cyclictest17490-1kworker/2:0H12:00:402
947699312,17cyclictest6209-21ssh11:05:152
9476993110,8cyclictest27-21ksoftirqd/211:55:142
946799311,9cyclictest26569-21usage_EthcTask11:55:351
946799311,8cyclictest26876-21kthreadcore07:25:241
946799311,8cyclictest2243-21cat12:00:311
946799311,8cyclictest21596-21sh09:50:021
946799311,8cyclictest19362-21dpkg10:10:021
946799311,8cyclictest17712-21sendmail-msp09:01:091
946799311,7cyclictest52246usb-storage07:21:491
946799311,6cyclictest6089-21cat08:10:301
946799311,6cyclictest27130-21wc10:15:031
946799311,6cyclictest17377-21latency_hist09:25:021
946799311,5cyclictest16035-21cut08:55:251
946799310,8cyclictest8373-21kthreadcore07:40:231
946799310,7cyclictest17311-21sed08:25:201
9486993014,9cyclictest25866irq/58-eth012:20:143
947699303,17cyclictest22146irq/40-dwc2_hso08:15:142
947699303,17cyclictest22146irq/40-dwc2_hso08:15:142
9476993012,7cyclictest22146irq/40-dwc2_hso08:20:182
9476993011,7cyclictest22146irq/40-dwc2_hso10:40:162
9476993011,10cyclictest22146irq/40-dwc2_hso12:20:142
946799305,3cyclictest0-21swapper/107:10:201
946799304,24cyclictest601-21rngd08:35:161
946799303,3cyclictest10001-21apt-get10:25:141
946799301,9cyclictest29527-21sed10:35:301
946799301,9cyclictest22502-21ssh09:50:201
946799301,8cyclictest24940-21sendmail-msp08:00:021
946799301,8cyclictest17236-21awk11:10:261
946799301,7cyclictest19009-21date07:20:021
946799300,6cyclictest29406-21cat09:30:381
9455993015,8cyclictest0-21swapper/011:25:160
946799297,18cyclictest32023-21getconf09:10:321
946799296,20cyclictest21218-21irqcore11:35:201
9467992920,3cyclictest601-21rngd12:05:121
946799291,8cyclictest29165-21cut07:30:191
946799291,8cyclictest18723-21kthreadcore12:10:201
946799291,8cyclictest16171-21kthreadcore11:50:221
946799291,6cyclictest22517-21kthreadcore08:30:201
946799291,5cyclictest601-21rngd09:20:011
946799291,4cyclictest601-21rngd12:35:331
945599294,16cyclictest131rcu_preempt09:50:150
945599292,20cyclictest27343-21ssh10:15:170
945599292,20cyclictest27343-21ssh10:15:170
945599291,19cyclictest10013-21ssh10:25:130
947699283,14cyclictest22146irq/40-dwc2_hso10:55:152
947699283,14cyclictest22146irq/40-dwc2_hso10:55:152
947699282,10cyclictest601-21rngd11:46:392
947699281,8cyclictest27-21ksoftirqd/210:35:302
946799281,8cyclictest31383-21kthreadcore08:05:211
946799281,7cyclictest5344-21tr11:25:221
946799281,7cyclictest12035-21xargs08:50:281
946799281,4cyclictest11889-21ntp_states10:45:271
946799280,9cyclictest23517-21kthreadcore12:30:221
946799280,6cyclictest7604-21sed08:15:201
946799280,6cyclictest7604-21sed08:15:201
9486992715,8cyclictest25866irq/58-eth012:25:153
947699273,4cyclictest601-21rngd11:31:392
947699272,4cyclictest27-21ksoftirqd/211:40:362
947699272,10cyclictest601-21rngd11:39:402
9476992717,8cyclictest261rcuc/208:45:122
947699271,22cyclictest0-21swapper/209:25:142
947699271,16cyclictest22146irq/40-dwc2_hso12:15:142
946799271,10cyclictest19036-21ntp_states07:50:241
945599272,21cyclictest24806-21ssh09:30:190
948699263,14cyclictest0-21swapper/311:20:113
948699263,12cyclictest25866irq/58-eth010:45:133
948699261,20cyclictest0-21swapper/310:55:163
948699261,20cyclictest0-21swapper/310:55:153
948699261,13cyclictest25866irq/58-eth011:15:153
947699265,18cyclictest32059-21latency07:30:232
947699263,3cyclictest601-21rngd10:13:402
947699262,20cyclictest601-21rngd10:04:392
9476992612,9cyclictest2805-21ssh09:15:122
947699261,17cyclictest27-21ksoftirqd/210:50:012
947699261,17cyclictest0-21swapper/212:35:172
947699260,17cyclictest601-21rngd11:13:192
945599261,16cyclictest4470-21ssh11:45:120
948699253,15cyclictest0-21swapper/311:03:403
948699252,17cyclictest0-21swapper/311:07:383
947699258,12cyclictest0-21swapper/208:50:132
947699252,3cyclictest601-21rngd10:33:232
947699252,13cyclictest22146irq/40-dwc2_hso07:55:172
9476992510,5cyclictest0-21swapper/207:15:132
945599252,16cyclictest5120-21ssh11:03:400
948699244,15cyclictest0-21swapper/312:10:113
948699243,14cyclictest0-21swapper/312:05:133
948699242,17cyclictest0-21swapper/309:59:393
9486992415,6cyclictest25866irq/58-eth011:55:163
9486992415,4cyclictest0-21swapper/312:15:113
947699247,8cyclictest6398-21grep10:20:352
947699244,12cyclictest27-21ksoftirqd/211:27:322
947699242,7cyclictest27-21ksoftirqd/208:35:212
947699242,4cyclictest271ksoftirqd/209:00:292
9476992410,9cyclictest601-21rngd09:50:142
9455992415,6cyclictest24791-21ssh11:15:280
9455992410,8cyclictest12-21ksoftirqd/009:49:390
948699235,13cyclictest0-21swapper/310:40:173
948699233,11cyclictest25866irq/58-eth010:35:143
948699231,15cyclictest25866irq/58-eth009:10:133
947699238,9cyclictest24785-21apt-get09:30:172
947699236,12cyclictest261rcuc/209:14:462
947699235,5cyclictest27-21ksoftirqd/209:35:202
947699233,4cyclictest271ksoftirqd/212:30:352
947699232,4cyclictest27-21ksoftirqd/207:25:282
947699232,13cyclictest27-21ksoftirqd/207:40:232
9476992316,2cyclictest601-21rngd11:15:142
9476992315,3cyclictest601-21rngd10:15:152
9476992315,3cyclictest601-21rngd10:15:152
947699231,13cyclictest345-21grep11:20:312
9476992311,10cyclictest26036-21sshd08:31:382
947699230,9cyclictest15424-21missed_timers10:05:252
945599230,19cyclictest8540-21cut08:50:160
948699228,9cyclictest25866irq/58-eth010:05:153
948699222,14cyclictest0-21swapper/310:10:153
947699223,7cyclictest10825-21acpi07:45:162
947699223,4cyclictest271ksoftirqd/209:20:142
947699221,5cyclictest271ksoftirqd/209:05:202
947699220,14cyclictest15995-21find10:50:142
947699220,14cyclictest15995-21find10:50:132
945599228,11cyclictest2126-21sendmail-msp12:20:070
945599222,17cyclictest26510-21idleruntime08:35:200
945599221,7cyclictest522-21usb-storage08:40:290
945599221,17cyclictest19831-21ls10:50:300
945599221,17cyclictest19831-21ls10:50:290
945599221,15cyclictest9612-21ssh11:05:250
945599221,13cyclictest12-21ksoftirqd/008:00:130
945599221,11cyclictest12-21ksoftirqd/007:20:140
948699216,10cyclictest0-21swapper/310:15:173
948699216,10cyclictest0-21swapper/310:15:173
948699211,15cyclictest0-21swapper/312:35:163
947699215,6cyclictest27-21ksoftirqd/208:25:162
947699213,5cyclictest27-21ksoftirqd/207:35:302
945599219,9cyclictest25621-21tr10:55:240
945599219,9cyclictest25621-21tr10:55:240
9455992115,3cyclictest25967-21rm12:15:170
945599211,4cyclictest131rcu_preempt11:33:100
945599211,15cyclictest21029-21uniq11:35:170
945599211,15cyclictest10211-21ls09:20:020
945599211,14cyclictest3697-21cut10:40:300
945599211,13cyclictest13307-21cut08:55:190
948699202,12cyclictest321rcuc/308:10:143
9486992014,4cyclictest0-21swapper/311:40:323
9486992013,5cyclictest0-21swapper/312:30:213
9486992012,4cyclictest0-21swapper/309:20:403
9486992011,6cyclictest0-21swapper/309:50:253
948699201,13cyclictest0-21swapper/307:20:023
9486992010,7cyclictest0-21swapper/312:00:593
947699201,10cyclictest10663-21kthreadcore07:10:222
945599209,3cyclictest131rcu_preempt07:20:020
945599207,3cyclictest131rcu_preempt09:10:300
945599201,3cyclictest601-21rngd08:25:210
9455992011,6cyclictest601-21rngd09:40:150
945599200,3cyclictest131rcu_preempt09:57:400
945599200,17cyclictest15730-21ls10:50:010
948699198,6cyclictest0-21swapper/310:30:163
9486991913,4cyclictest0-21swapper/309:35:303
9486991912,5cyclictest0-21swapper/311:35:143
9486991912,5cyclictest0-21swapper/309:30:533
948699191,13cyclictest0-21swapper/308:20:173
948699191,12cyclictest0-21swapper/308:05:023
9486991910,5cyclictest25866irq/58-eth011:45:223
945599192,4cyclictest131rcu_preempt11:10:170
9455991911,6cyclictest942-21cron07:35:000
945599191,13cyclictest21069-21ssh09:25:270
9486991812,4cyclictest0-21swapper/311:30:193
9486991812,4cyclictest0-21swapper/311:25:323
9486991812,4cyclictest0-21swapper/310:00:133
9486991812,4cyclictest0-21swapper/309:40:293
9486991812,4cyclictest0-21swapper/307:20:283
9486991811,5cyclictest0-21swapper/310:20:273
9486991811,4cyclictest0-21swapper/308:40:173
948699181,13cyclictest0-21swapper/307:10:173
948699181,12cyclictest0-21swapper/308:50:133
9486991810,6cyclictest0-21swapper/311:14:293
9486991810,6cyclictest0-21swapper/309:19:373
945599181,6cyclictest601-21rngd09:05:220
945599181,3cyclictest131rcu_preempt07:10:200
945599181,2cyclictest131rcu_preempt12:39:380
9455991811,5cyclictest862-21sshd11:52:390
948699179,6cyclictest0-21swapper/310:52:313
948699179,6cyclictest0-21swapper/310:52:313
948699179,3cyclictest0-21swapper/309:26:023
948699172,10cyclictest0-21swapper/308:15:153
948699172,10cyclictest0-21swapper/308:15:153
948699171,12cyclictest0-21swapper/308:55:203
948699171,10cyclictest0-21swapper/307:55:193
945599176,7cyclictest12-21ksoftirqd/007:45:180
945599174,3cyclictest131rcu_preempt08:05:210
945599172,2cyclictest131rcu_preempt10:30:120
945599171,2cyclictest131rcu_preempt12:06:440
945599171,12cyclictest23756-21ntp_192.168.11407:55:280
9455991710,5cyclictest0-21swapper/008:15:260
9455991710,5cyclictest0-21swapper/008:15:250
948699167,6cyclictest33-21ksoftirqd/309:05:193
948699162,3cyclictest0-21swapper/308:25:163
948699162,10cyclictest0-21swapper/308:45:143
948699162,10cyclictest0-21swapper/308:05:143
948699161,6cyclictest0-21swapper/307:50:113
945599168,6cyclictest601-21rngd07:50:240
945599162,3cyclictest131rcu_preempt07:35:200
945599162,10cyclictest24469-21irqcore07:25:200
945599161,3cyclictest131rcu_preempt08:30:150
945599161,12cyclictest12-21ksoftirqd/008:45:300
948699159,4cyclictest0-21swapper/307:45:323
948699153,8cyclictest0-21swapper/307:40:143
948699151,9cyclictest0-21swapper/307:35:173
948699151,9cyclictest0-21swapper/307:25:233
945599151,10cyclictest9514-21latency_hist07:10:020
948699149,3cyclictest0-21swapper/309:00:153
948699147,5cyclictest33-21ksoftirqd/307:30:153
948699140,10cyclictest0-21swapper/308:35:173
947699121,8cyclictest9488-21latency_hist07:10:022
946799112,7cyclictest9482-21latency_hist07:10:011
948699101,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional