You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 21:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Tue Aug 26, 2025 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24375996128,20cyclictest25366irq/58-eth010:55:143
2431299603,44cyclictest13067-21ssh11:40:140
2431299583,42cyclictest25406-21ssh09:20:150
2431299574,9cyclictest121ksoftirqd/007:10:120
2431299544,35cyclictest20492-21ssh12:05:140
2431299541,39cyclictest8362-21ssh09:30:190
2431299511,29cyclictest31145-21ssh09:45:120
24375994722,11cyclictest25366irq/58-eth009:15:133
2431299465,36cyclictest15930-21ssh09:35:120
2431299431,32cyclictest23503-21ssh11:05:120
2435099410,36cyclictest0-21swapper/208:00:142
24350993920,15cyclictest27-21ksoftirqd/209:05:182
2433299397,4cyclictest131rcu_preempt11:30:041
2435099382,27cyclictest21192-21ssh11:45:172
2431299382,29cyclictest20197-21ssh12:25:130
2437599363,19cyclictest25366irq/58-eth009:45:123
24375993618,12cyclictest25366irq/58-eth012:35:143
2433299365,26cyclictest6373-21ssh09:50:131
2433299362,31cyclictest17744-21ssh09:15:151
2437599352,14cyclictest0-21swapper/309:25:133
24312993524,5cyclictest0-21swapper/008:40:140
24312993520,9cyclictest0-21swapper/012:10:170
24312993520,9cyclictest0-21swapper/012:10:160
2431299351,28cyclictest27091-21ssh10:25:130
24332993423,8cyclictest21-21ksoftirqd/110:05:151
2435099333,19cyclictest942-21snmpd08:15:162
2433299338,4cyclictest131rcu_preempt11:45:171
2431299332,23cyclictest12244-21ssh10:15:170
24350993213,9cyclictest27-21ksoftirqd/211:40:132
2435099321,22cyclictest28984-21ssh09:20:262
2435099313,20cyclictest16118-21ssh11:00:172
2435099312,18cyclictest21846irq/40-dwc2_hso07:55:132
24350993110,12cyclictest27-21ksoftirqd/209:48:582
24375993019,5cyclictest25366irq/58-eth011:40:143
2437599301,16cyclictest25366irq/58-eth012:25:123
2435099301,23cyclictest4610-21ssh10:10:152
2435099301,12cyclictest2147666ssh10:00:142
2435099301,12cyclictest2147666ssh10:00:132
2437599291,12cyclictest25366irq/58-eth010:25:123
2435099297,8cyclictest131rcu_preempt11:30:302
2435099295,15cyclictest0-21swapper/209:15:152
2433299293,22cyclictest28444-21ssh12:30:191
2433299293,22cyclictest28444-21ssh12:30:191
2433299292,4cyclictest131rcu_preempt10:00:141
2433299292,4cyclictest131rcu_preempt10:00:131
2433299292,24cyclictest21695-21ssh11:25:141
2433299292,24cyclictest21695-21ssh11:25:131
2433299292,20cyclictest4186-21ssh12:15:151
24312992910,9cyclictest131rcu_preempt11:50:140
24375992819,6cyclictest0-21swapper/311:30:153
2435099289,5cyclictest27-21ksoftirqd/212:10:412
2435099289,5cyclictest27-21ksoftirqd/212:10:412
2435099283,18cyclictest21846irq/40-dwc2_hso09:25:132
2435099282,22cyclictest4180-21sh12:15:142
24350992821,3cyclictest23675-21ssh09:40:142
2433299283,7cyclictest21-21ksoftirqd/112:03:071
24332992813,7cyclictest0-21swapper/109:30:121
2431299282,18cyclictest4774-21ssh11:55:130
2437599272,15cyclictest25366irq/58-eth012:10:173
2437599272,15cyclictest25366irq/58-eth012:10:163
2435099273,17cyclictest21846irq/40-dwc2_hso07:45:122
2435099272,15cyclictest21846irq/40-dwc2_hso11:25:132
2435099272,15cyclictest21846irq/40-dwc2_hso11:25:122
2433299276,15cyclictest131rcu_preempt08:00:141
24332992713,3cyclictest0-21swapper/111:16:301
2433299271,18cyclictest9784-21diskmemload09:55:171
24332992711,7cyclictest0-21swapper/111:03:381
2431299272,19cyclictest9906-21ssh10:35:180
24312992716,7cyclictest24297-21cyclictest09:25:140
2437599264,14cyclictest25366irq/58-eth010:15:173
2435099269,7cyclictest131rcu_preempt09:30:262
2435099268,14cyclictest15344-21cut10:15:252
2435099267,11cyclictest27-21ksoftirqd/211:59:512
2435099263,14cyclictest21846irq/40-dwc2_hso11:15:152
24350992616,8cyclictest27-21ksoftirqd/210:20:322
2435099260,9cyclictest27-21ksoftirqd/211:35:272
2433299266,3cyclictest131rcu_preempt10:35:191
2433299264,4cyclictest131rcu_preempt10:10:291
2431299262,20cyclictest25272-21ssh10:45:130
2437599252,13cyclictest0-21swapper/309:20:153
24375992516,6cyclictest25366irq/58-eth011:25:143
24375992516,6cyclictest25366irq/58-eth011:25:143
2435099256,12cyclictest131rcu_preempt08:50:192
2435099251,6cyclictest26024-21kthreadcore10:45:212
2435099251,5cyclictest271ksoftirqd/207:35:212
24350992513,9cyclictest11242-21ssh09:50:402
24350992513,8cyclictest28029-21apt-get08:55:142
2435099251,21cyclictest21007-21timerwakeupswit08:10:302
2435099251,21cyclictest21007-21timerwakeupswit08:10:302
2435099251,10cyclictest455366ssh11:55:022
2433299259,12cyclictest131rcu_preempt12:20:141
2433299255,3cyclictest131rcu_preempt12:05:141
2433299253,13cyclictest23674-21ssh09:40:141
24332992512,7cyclictest0-21swapper/108:35:131
24332992510,7cyclictest131rcu_preempt11:12:051
24332992510,3cyclictest9784-21diskmemload10:21:481
2431299253,4cyclictest131rcu_preempt10:00:170
2431299253,4cyclictest131rcu_preempt10:00:170
2431299252,18cyclictest14234-21ssh11:20:140
24375992410,9cyclictest25366irq/58-eth010:20:153
2435099249,7cyclictest131rcu_preempt07:55:012
2435099245,4cyclictest27-21ksoftirqd/211:10:512
2435099243,7cyclictest0-21swapper/211:21:112
2435099243,16cyclictest29342-21ssh10:05:152
2435099243,12cyclictest0-21swapper/210:26:542
2435099242,5cyclictest27-21ksoftirqd/207:25:162
2435099242,11cyclictest21846irq/40-dwc2_hso08:35:132
2435099242,10cyclictest27-21ksoftirqd/208:40:142
2435099241,12cyclictest0-21swapper/209:12:282
24350992410,8cyclictest27-21ksoftirqd/209:38:292
2433299249,8cyclictest21-21ksoftirqd/110:40:081
2433299245,10cyclictest21-21ksoftirqd/109:28:451
2433299244,13cyclictest0-21swapper/107:45:151
2433299241,17cyclictest23715-21ssh12:25:221
2433299241,15cyclictest2374-21ssh10:30:191
2431299245,16cyclictest12-21ksoftirqd/012:15:140
2431299245,16cyclictest10177-21ssh09:10:170
24312992410,2cyclictest12-21ksoftirqd/011:35:050
2435099236,7cyclictest131rcu_preempt09:55:272
2435099233,6cyclictest27-21ksoftirqd/212:08:242
2435099233,2cyclictest131rcu_preempt12:00:242
2435099233,12cyclictest27-21ksoftirqd/210:56:502
2435099232,9cyclictest0-21swapper/212:25:132
2435099232,8cyclictest1-21systemd12:30:192
2435099232,8cyclictest1-21systemd12:30:182
24350992313,8cyclictest0-21swapper/208:30:152
24350992312,4cyclictest29038-21idleruntime-cro07:15:012
24350992312,4cyclictest27-21ksoftirqd/207:15:342
24350992312,4cyclictest261rcuc/210:40:142
2435099230,10cyclictest22124-21timerandwakeup08:45:272
2433299237,3cyclictest131rcu_preempt11:40:161
2433299232,4cyclictest21-21ksoftirqd/108:35:011
2433299232,4cyclictest131rcu_preempt08:25:141
2433299232,17cyclictest27074-21apt-get10:25:131
24332992313,4cyclictest0-21swapper/109:35:141
24332992312,3cyclictest0-21swapper/110:18:551
2433299231,19cyclictest22136-21ls08:15:171
24332992310,2cyclictest21-21ksoftirqd/108:50:271
2433299230,17cyclictest8260-21ssh10:55:141
2431299234,11cyclictest0-21swapper/010:30:200
2431299232,3cyclictest131rcu_preempt08:45:130
2431299232,14cyclictest6378-21ssh09:50:140
2431299231,16cyclictest4049-21sh12:35:150
2437599228,9cyclictest25366irq/58-eth009:35:143
24375992211,8cyclictest0-21swapper/311:00:243
2435099226,5cyclictest27-21ksoftirqd/207:40:202
2435099225,7cyclictest131rcu_preempt11:05:132
2435099225,14cyclictest12687-21kthreadcore10:35:272
2435099224,2cyclictest131rcu_preempt12:35:352
2435099223,16cyclictest4205-21kthreadcore07:20:252
2435099222,4cyclictest27-21ksoftirqd/208:25:242
24350992215,4cyclictest27-21ksoftirqd/209:00:152
24350992212,8cyclictest0-21swapper/212:23:112
2435099221,12cyclictest1014-21cut10:50:182
2435099221,10cyclictest7111-21users10:30:392
2433299229,3cyclictest131rcu_preempt12:10:161
2433299229,3cyclictest131rcu_preempt12:10:151
2433299228,3cyclictest131rcu_preempt11:05:191
2433299227,5cyclictest21-21ksoftirqd/108:55:281
2433299226,8cyclictest0-21swapper/110:52:051
2433299226,8cyclictest0-21swapper/109:11:491
2433299225,8cyclictest131rcu_preempt12:35:171
24332992212,5cyclictest0-21swapper/107:35:141
2433299221,17cyclictest26895-21apt-get08:20:121
2433299220,16cyclictest29286-21ls10:45:311
2431299227,2cyclictest131rcu_preempt08:05:230
2431299225,4cyclictest30059-21cut11:30:200
2431299222,13cyclictest31357-21ssh11:10:170
24312992216,3cyclictest121ksoftirqd/010:10:140
2431299221,3cyclictest131rcu_preempt12:21:010
2431299221,18cyclictest6702-21find11:15:140
2437599212,13cyclictest25366irq/58-eth009:40:143
2435099216,8cyclictest131rcu_preempt07:30:212
24350992112,7cyclictest0-21swapper/208:05:312
2433299219,6cyclictest131rcu_preempt09:20:511
2433299214,2cyclictest131rcu_preempt11:50:281
2433299213,4cyclictest131rcu_preempt07:55:191
2433299213,3cyclictest0-21swapper/108:40:281
2433299211,13cyclictest20989-21cut07:40:161
2433299211,13cyclictest11225-21find07:30:111
2431299219,9cyclictest16807-21sed11:00:230
2431299212,3cyclictest131rcu_preempt09:40:150
2431299211,4cyclictest131rcu_preempt07:40:200
2431299211,3cyclictest131rcu_preempt10:05:220
2431299211,3cyclictest131rcu_preempt07:50:140
2431299211,14cyclictest21937-21ls11:25:170
2431299211,14cyclictest21937-21ls11:25:170
24312992110,3cyclictest0-21swapper/010:22:390
2437599206,8cyclictest25366irq/58-eth009:30:193
2437599201,12cyclictest25466irq/59-eth011:50:133
2435099201,9cyclictest29134-21kthreadcore08:20:212
2433299204,3cyclictest131rcu_preempt09:05:271
2433299202,6cyclictest131rcu_preempt11:24:011
2433299201,3cyclictest131rcu_preempt07:10:131
24332992013,4cyclictest21-21ksoftirqd/109:45:231
2433299201,15cyclictest5103-21diskmemload11:35:151
2431299208,6cyclictest131rcu_preempt09:55:220
2431299204,5cyclictest3225-21kthreadcore10:50:220
2431299203,13cyclictest17727-21apt-get09:15:140
2431299201,9cyclictest0-21swapper/011:48:230
2431299201,3cyclictest131rcu_preempt12:03:450
2431299201,2cyclictest131rcu_preempt07:35:170
24312992010,3cyclictest12-21ksoftirqd/010:40:140
2437599197,8cyclictest25366irq/58-eth010:40:143
2437599192,11cyclictest0-21swapper/307:50:143
2437599191,13cyclictest0-21swapper/312:20:313
2437599191,12cyclictest0-21swapper/308:15:143
24375991910,7cyclictest0-21swapper/310:02:443
24375991910,7cyclictest0-21swapper/310:02:443
24375991910,6cyclictest0-21swapper/309:58:383
2435099193,8cyclictest24437-21latency_hist07:10:022
2433299195,5cyclictest131rcu_preempt11:55:191
2433299194,4cyclictest709-21unixbench_singl07:15:321
2433299191,9cyclictest562-21fschecks_time09:00:171
2433299191,4cyclictest21-21ksoftirqd/107:25:291
2433299191,3cyclictest131rcu_preempt08:45:201
2431299199,7cyclictest10515-21kthreadcore08:35:210
2431299198,4cyclictest12-21ksoftirqd/010:55:150
2431299194,3cyclictest131rcu_preempt07:55:300
2431299192,14cyclictest11558-21irqcore07:30:170
2431299191,7cyclictest131rcu_preempt08:00:210
2431299190,15cyclictest27133-21ls08:20:170
2437599189,6cyclictest0-21swapper/311:46:353
2437599189,6cyclictest0-21swapper/311:22:353
2437599188,8cyclictest0-21swapper/312:05:453
2437599182,10cyclictest0-21swapper/312:15:043
2437599182,10cyclictest0-21swapper/311:10:043
2437599182,10cyclictest0-21swapper/310:35:183
2437599182,10cyclictest0-21swapper/309:00:153
2437599182,10cyclictest0-21swapper/308:45:123
2437599182,10cyclictest0-21swapper/308:40:153
2437599182,10cyclictest0-21swapper/308:30:153
2437599182,10cyclictest0-21swapper/308:10:163
2437599182,10cyclictest0-21swapper/308:10:163
2437599182,10cyclictest0-21swapper/307:40:153
2437599182,10cyclictest0-21swapper/307:20:163
24375991812,4cyclictest25466irq/59-eth011:05:133
24375991812,4cyclictest25366irq/58-eth010:50:143
2437599181,14cyclictest0-21swapper/311:35:183
24375991810,6cyclictest25366irq/58-eth010:30:193
2433299183,4cyclictest24360-21latency_hist07:10:011
2433299183,4cyclictest131rcu_preempt07:50:211
24332991812,4cyclictest0-21swapper/107:20:181
24332991810,5cyclictest21-21ksoftirqd/108:10:021
2433299180,3cyclictest131rcu_preempt08:10:181
2433299180,3cyclictest131rcu_preempt08:10:171
2431299184,2cyclictest131rcu_preempt08:55:170
2431299182,7cyclictest29494-21grep07:15:200
2431299181,5cyclictest1119-21kthreadcore09:00:200
2431299181,3cyclictest131rcu_preempt08:10:240
2431299181,3cyclictest131rcu_preempt08:10:230
24312991813,3cyclictest0-21swapper/012:31:390
24312991813,3cyclictest0-21swapper/012:31:390
24312991813,3cyclictest0-21swapper/008:15:270
24312991812,4cyclictest31806-21cut08:25:150
2437599179,6cyclictest0-21swapper/312:01:163
2437599179,6cyclictest0-21swapper/311:55:073
2437599179,6cyclictest0-21swapper/311:15:123
2437599179,6cyclictest0-21swapper/310:46:043
2437599179,6cyclictest0-21swapper/309:50:063
2437599179,6cyclictest0-21swapper/309:12:323
2437599178,6cyclictest0-21swapper/310:12:073
2437599178,6cyclictest0-21swapper/310:05:123
2437599172,10cyclictest0-21swapper/308:50:153
2431299175,3cyclictest12-21ksoftirqd/008:50:260
2431299173,11cyclictest0-21swapper/008:30:140
2431299172,6cyclictest7115-21kthreadcore07:25:200
2431299171,3cyclictest131rcu_preempt09:05:210
2437599169,5cyclictest0-21swapper/307:10:173
24375991611,3cyclictest0-21swapper/312:33:303
24375991611,3cyclictest0-21swapper/312:33:293
2431299163,9cyclictest0-21swapper/007:20:170
2431299161,13cyclictest27376-21kthreadcore07:45:210
2437599159,4cyclictest0-21swapper/307:45:193
2431299151,11cyclictest24227-21latency_hist07:10:020
2437599149,3cyclictest0-21swapper/308:25:163
2437599149,3cyclictest0-21swapper/308:20:153
2437599148,4cyclictest33-21ksoftirqd/307:35:273
2437599145,6cyclictest25366irq/58-eth008:05:193
2437599141,10cyclictest0-21swapper/309:10:013
2437599141,10cyclictest0-21swapper/307:55:283
2437599138,3cyclictest0-21swapper/308:55:243
2437599138,3cyclictest0-21swapper/308:35:163
2437599132,7cyclictest0-21swapper/308:00:143
2437599123,7cyclictest0-21swapper/307:15:313
2437599122,8cyclictest0-21swapper/307:30:143
2437599121,6cyclictest0-21swapper/307:30:023
2437599101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional