You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-23 - 17:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Aug 23, 2026 12:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7791997044,12cyclictest35966irq/58-eth011:35:123
777599696,57cyclictest26543-21ssh09:10:151
776399561,36cyclictest908-21ssh10:25:130
777599554,39cyclictest12463-21ssh11:25:131
778199541,38cyclictest22769-21ssh10:55:132
7763995327,18cyclictest12-21ksoftirqd/007:35:130
776399520,40cyclictest15116-21ssh10:15:140
7763995111,24cyclictest12-21ksoftirqd/008:30:140
777599503,37cyclictest30594-21ssh11:35:131
777599491,40cyclictest717-21ssh12:30:141
777599482,3cyclictest131rcu_preempt11:00:131
776399481,35cyclictest20261-21ssh10:00:120
777599432,23cyclictest30634-21ssh09:30:121
7791994130,5cyclictest35966irq/58-eth011:25:133
7791993923,9cyclictest35966irq/58-eth009:25:133
777599396,23cyclictest21-21ksoftirqd/112:00:131
776399394,27cyclictest0-21swapper/009:50:130
776399394,10cyclictest12-21ksoftirqd/009:15:150
778199383,8cyclictest131rcu_preempt08:00:142
776399383,24cyclictest12-21ksoftirqd/010:50:140
7763993821,9cyclictest630-21Async09:36:310
778199372,28cyclictest10080-21ssh12:35:152
777599375,19cyclictest25309-21ssh11:50:131
779199361,21cyclictest35966irq/58-eth012:35:113
7763993622,9cyclictest24234-21ssh10:20:140
7791993417,11cyclictest35966irq/58-eth010:35:153
778199344,8cyclictest131rcu_preempt07:35:122
778199343,10cyclictest0-21swapper/212:30:202
776399343,21cyclictest29368-21ssh10:05:130
7763993416,13cyclictest151rcuc/011:45:140
778199332,20cyclictest27346irq/40-dwc2_hso10:30:142
778199331,25cyclictest7241-21ssh09:35:122
777599333,17cyclictest21-21ksoftirqd/109:00:131
777599332,28cyclictest2078-21ssh09:50:131
777599331,21cyclictest13706-21ssh10:50:141
779199323,16cyclictest35966irq/58-eth012:20:123
7791993217,11cyclictest35966irq/58-eth012:05:143
776399322,25cyclictest30643-21ssh09:30:120
779199313,18cyclictest35966irq/58-eth007:40:123
778199311,19cyclictest27346irq/40-dwc2_hso09:40:132
778199311,19cyclictest25394-21ssh09:45:132
776399312,20cyclictest8537-21sh11:05:130
7763993114,14cyclictest12-21ksoftirqd/010:45:140
7791993018,9cyclictest35966irq/58-eth011:50:133
778199302,26cyclictest20431-21ssh12:21:152
778199301,23cyclictest24036-21ssh12:25:132
776399302,22cyclictest16144-21kthreadcore12:20:190
7791992911,9cyclictest0-21swapper/310:00:183
776399292,19cyclictest26044-21grep12:25:200
776399292,18cyclictest844-21munin-node07:15:150
7763992916,3cyclictest131rcu_preempt11:30:120
776399291,18cyclictest12504-21ssh09:20:140
779199283,17cyclictest35966irq/58-eth012:25:123
779199282,15cyclictest0-21swapper/311:05:143
779199281,17cyclictest35966irq/58-eth012:30:153
7791992811,11cyclictest35966irq/58-eth009:35:123
778199283,16cyclictest27346irq/40-dwc2_hso07:40:112
778199282,20cyclictest11099-21ssh12:00:142
778199282,20cyclictest0-21swapper/208:45:152
777599282,18cyclictest26671-21ssh11:15:161
779199272,21cyclictest35966irq/58-eth011:00:133
778199274,8cyclictest20133-21ssh12:05:132
778199272,20cyclictest0-21swapper/211:10:142
778199271,10cyclictest8539-21ssh11:05:132
777599272,19cyclictest17832-21ssh12:39:151
777599271,20cyclictest0-21swapper/109:20:131
779199268,14cyclictest35966irq/58-eth010:45:123
779199262,16cyclictest0-21swapper/312:10:133
779199262,14cyclictest0-21swapper/310:15:143
7791992612,8cyclictest0-21swapper/310:05:143
778199263,16cyclictest27346irq/40-dwc2_hso07:45:152
778199263,15cyclictest27346irq/40-dwc2_hso11:35:122
778199262,17cyclictest27346irq/40-dwc2_hso10:25:132
778199261,21cyclictest32114-21ls10:40:292
778199260,11cyclictest26123-21diskmemload11:50:122
776399267,15cyclictest151rcuc/011:00:140
776399262,8cyclictest12-21ksoftirqd/011:20:130
776399262,15cyclictest29868-21sshd08:40:120
776399262,15cyclictest29868-21sshd08:40:120
779199252,13cyclictest35966irq/58-eth007:45:143
779199251,12cyclictest35966irq/58-eth010:30:133
7791992510,9cyclictest35966irq/58-eth009:20:153
778199253,14cyclictest27346irq/40-dwc2_hso08:40:142
778199253,14cyclictest27346irq/40-dwc2_hso08:40:142
778199252,8cyclictest20250-21sshd10:00:122
778199252,18cyclictest27346irq/40-dwc2_hso11:45:162
778199250,21cyclictest2249-21grep09:50:162
778199250,13cyclictest30335-21ls11:35:022
7775992516,5cyclictest0-21swapper/110:10:131
7775992516,3cyclictest0-21swapper/111:30:121
777599251,20cyclictest19048-21ssh10:35:151
7763992512,4cyclictest12-21ksoftirqd/011:50:190
776399251,19cyclictest28091-21ssh10:40:140
779199244,2cyclictest0-21swapper/312:15:133
7791992415,6cyclictest35966irq/58-eth011:30:133
778199243,17cyclictest27346irq/40-dwc2_hso11:20:132
778199243,14cyclictest27346irq/40-dwc2_hso09:00:132
778199242,16cyclictest0-21swapper/212:15:162
778199242,12cyclictest0-21swapper/208:25:112
778199241,12cyclictest12483-21ssh11:25:142
7781992410,11cyclictest27-21ksoftirqd/208:35:132
778199240,11cyclictest15691-21grep10:50:222
776399242,17cyclictest7247-21diskmemload09:35:140
776399242,17cyclictest16735-21apt-get09:00:130
776399242,16cyclictest9464-21ssh12:15:260
7763992416,5cyclictest0-21swapper/012:35:120
779199239,7cyclictest35966irq/58-eth009:50:143
779199239,11cyclictest0-21swapper/309:15:013
779199233,16cyclictest35966irq/58-eth010:25:123
779199231,8cyclictest36066irq/59-eth009:15:153
7791992318,3cyclictest321rcuc/309:41:473
7791992316,5cyclictest321rcuc/309:31:473
7791992316,4cyclictest0-21swapper/310:13:123
778199236,10cyclictest15100-21ssh10:15:132
778199232,15cyclictest0-21swapper/212:10:142
7775992314,5cyclictest999-21cpuspeed10:25:141
777599231,15cyclictest21172-21ssh11:10:251
776399236,8cyclictest12-21ksoftirqd/011:14:220
7763992315,6cyclictest22680-21ssh10:35:260
7763992310,7cyclictest131rcu_preempt12:30:140
7763992310,3cyclictest12-21ksoftirqd/010:55:280
778199223,12cyclictest27346irq/40-dwc2_hso07:25:122
778199222,13cyclictest0-21swapper/209:20:142
7781992213,6cyclictest0-21swapper/208:20:282
7781992213,6cyclictest0-21swapper/208:20:282
7781992211,9cyclictest7648-21ssh11:56:272
7781992211,9cyclictest3096-21ssh09:30:332
7781992211,9cyclictest12911-21ssh11:41:332
778199221,12cyclictest26640-21apt-get11:15:162
777599228,10cyclictest0-21swapper/111:40:131
777599222,5cyclictest53646usb-storage09:25:161
777599222,14cyclictest24155-21df_inode12:25:151
777599222,14cyclictest15032-21df_inode12:20:161
777599221,17cyclictest24865-21ls08:35:021
776399229,4cyclictest12-21ksoftirqd/009:45:460
776399223,6cyclictest12-21ksoftirqd/011:40:140
7763992216,3cyclictest12-21ksoftirqd/012:12:330
7763992215,3cyclictest12-21ksoftirqd/008:00:160
776399221,3cyclictest131rcu_preempt09:05:140
776399221,18cyclictest21564-21sh09:25:140
7763992210,3cyclictest12-21ksoftirqd/012:00:130
7791992116,3cyclictest321rcuc/311:49:023
7791992111,6cyclictest35966irq/58-eth010:20:143
7791992111,4cyclictest0-21swapper/310:43:063
7791992110,8cyclictest35966irq/58-eth010:50:143
778199219,10cyclictest27-21ksoftirqd/210:10:022
778199211,7cyclictest0-21swapper/207:15:152
778199211,16cyclictest4842-21cut10:45:162
778199211,16cyclictest29772-21kthreadcore08:05:212
778199211,10cyclictest3595-21sed11:00:302
778199211,10cyclictest27286-21ssh09:10:202
778199211,10cyclictest24035-21cut09:25:212
778199211,10cyclictest20791-21kthreadcore10:35:212
7781992110,9cyclictest17844-21if_err_eth107:20:182
778199210,14cyclictest9427-21ls08:15:292
778199210,12cyclictest11826-21irqrtprio09:55:192
777599212,15cyclictest1951-21apt-get11:55:121
777599212,11cyclictest19151-21tr09:40:211
777599211,5cyclictest28825-21munin-run12:10:011
777599211,10cyclictest7225-21acpi09:35:121
777599210,16cyclictest11739-21ls07:10:281
776399213,3cyclictest131rcu_preempt07:50:180
7763992113,5cyclictest20338-21ssh12:05:160
776399211,18cyclictest27011-21apt-key07:30:010
7763992111,7cyclictest1976-21ssh11:35:280
779199209,7cyclictest35966irq/58-eth010:55:133
778199204,9cyclictest21248-21kthreadcore08:30:202
778199204,9cyclictest14582-21irqcore07:50:182
778199203,9cyclictest14496-21kthreadcore08:55:222
778199203,14cyclictest20893-21kthreadcore07:55:222
778199203,14cyclictest1649-21kthreadcore08:10:192
778199202,9cyclictest9432-21kthreadcore07:10:202
778199202,10cyclictest3183-21ssh09:15:052
778199201,9cyclictest29376-21kthreadcore07:30:222
778199201,9cyclictest26124-21latency_hist09:10:012
778199201,16cyclictest26344-21kthreadcore10:20:212
778199200,2cyclictest0-21swapper/210:10:112
777599201,8cyclictest20950-21irqrtprio10:00:181
777599201,15cyclictest17353-21awk07:20:011
777599201,13cyclictest4586-21cut07:40:141
777599201,13cyclictest11133-21find09:55:121
777599201,10cyclictest8454-21kthreadcore08:15:231
776399208,2cyclictest131rcu_preempt09:10:290
776399206,5cyclictest12-21ksoftirqd/011:26:400
7763992012,5cyclictest12-21ksoftirqd/007:55:270
776399201,13cyclictest2443-21cut08:45:160
776399200,16cyclictest14299-21timerandwakeup08:20:290
776399200,16cyclictest14299-21timerandwakeup08:20:290
779199193,11cyclictest0-21swapper/312:00:133
779199192,10cyclictest35966irq/58-eth007:35:133
7791991914,3cyclictest321rcuc/311:15:123
7791991914,3cyclictest321rcuc/309:58:193
7791991914,3cyclictest321rcuc/307:20:303
7791991911,6cyclictest0-21swapper/311:40:403
779199191,13cyclictest0-21swapper/307:10:013
779199191,12cyclictest0-21swapper/308:55:233
778199191,9cyclictest7238-21fschecks_count08:50:152
777599198,8cyclictest23890-21df08:00:141
777599195,9cyclictest630-21Async09:36:311
777599193,14cyclictest6798-21kthreadcore10:45:211
777599191,9cyclictest28081-21ssh10:40:141
7775991911,5cyclictest0-21swapper/107:50:151
777599191,14cyclictest18725-21ntp_192.168.11410:15:251
777599191,14cyclictest14847-21ssh10:30:341
777599191,12cyclictest516-21ssh10:05:261
777599191,10cyclictest4036-21irqrtprio09:15:211
776399198,2cyclictest12-21ksoftirqd/011:57:110
776399197,2cyclictest12-21ksoftirqd/010:34:250
776399191,9cyclictest6357-21sed08:15:200
7763991913,4cyclictest0-21swapper/008:55:260
7763991913,4cyclictest0-21swapper/007:20:140
7763991913,3cyclictest12-21ksoftirqd/008:10:180
7763991910,2cyclictest131rcu_preempt10:11:290
779199185,7cyclictest0-21swapper/307:10:123
779199182,10cyclictest0-21swapper/308:45:053
779199182,10cyclictest0-21swapper/308:00:143
7791991813,3cyclictest321rcuc/307:50:143
7791991813,3cyclictest0-21swapper/309:46:203
7791991812,4cyclictest0-21swapper/311:59:503
7791991812,4cyclictest0-21swapper/311:20:293
7791991810,6cyclictest35966irq/58-eth009:10:023
777599189,7cyclictest6242-21cut11:20:221
777599189,6cyclictest0-21swapper/107:25:221
777599186,8cyclictest21-21ksoftirqd/107:55:151
777599186,7cyclictest4179-21latency_hist07:40:011
777599184,9cyclictest0-21swapper/110:55:141
777599183,9cyclictest0-21swapper/111:05:131
777599182,11cyclictest13007-21ssh12:18:151
777599181,8cyclictest26125-21idleruntime-cro09:10:011
777599181,14cyclictest15728-21unixbench_multi08:55:291
777599181,12cyclictest24235-21sh10:20:141
776399189,7cyclictest31801-21sshd07:31:110
776399186,3cyclictest12-21ksoftirqd/008:05:310
776399185,3cyclictest131rcu_preempt08:35:130
776399181,3cyclictest131rcu_preempt07:45:010
7763991813,2cyclictest12-21ksoftirqd/008:30:010
7763991812,4cyclictest0-21swapper/011:15:150
776399181,14cyclictest16347-21ssh09:40:120
7763991811,4cyclictest12-21ksoftirqd/007:10:190
779199172,9cyclictest0-21swapper/307:25:133
7791991712,3cyclictest321rcuc/309:00:243
7791991711,4cyclictest0-21swapper/311:11:043
7791991711,4cyclictest0-21swapper/307:30:253
779199171,13cyclictest0-21swapper/307:15:283
779199171,10cyclictest35966irq/58-eth008:30:143
777599177,7cyclictest29200-21apt-get12:10:121
777599173,2cyclictest131rcu_preempt08:20:131
777599173,2cyclictest131rcu_preempt08:20:131
777599172,10cyclictest536-21usb-storage08:40:041
777599172,10cyclictest536-21usb-storage08:40:041
777599172,10cyclictest0-21swapper/107:30:161
777599171,8cyclictest26141-21kthreadcore09:45:201
777599171,5cyclictest53646usb-storage08:45:051
7775991711,4cyclictest20230-21timerandwakeup11:45:321
7775991710,5cyclictest0-21swapper/108:35:221
776399178,6cyclictest11695-21cron08:55:010
7763991710,3cyclictest12-21ksoftirqd/007:45:270
779199162,9cyclictest0-21swapper/308:35:133
779199162,8cyclictest0-21swapper/308:40:043
779199162,8cyclictest0-21swapper/308:40:043
7791991611,3cyclictest0-21swapper/308:20:163
7791991611,3cyclictest0-21swapper/308:20:163
7791991611,3cyclictest0-21swapper/308:10:153
7791991611,3cyclictest0-21swapper/308:05:263
779199161,12cyclictest0-21swapper/308:50:153
778199161,8cyclictest7786-21latency_hist07:10:012
777599162,10cyclictest17760-21fschecks_time07:20:171
777599161,12cyclictest15407-21/usr/sbin/munin08:25:171
777599161,11cyclictest9468-21fschecks_count07:45:181
777599161,11cyclictest4894-21unixbench_singl08:10:291
777599160,8cyclictest10011-21kthreadcore08:50:211
776399161,11cyclictest7882-21latency_hist07:10:010
779199159,4cyclictest321rcuc/308:25:213
779199159,4cyclictest0-21swapper/307:55:173
7791991510,3cyclictest0-21swapper/308:15:303
777599131,9cyclictest7833-21latency_hist07:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional