You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-20 - 13:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Jun 20, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2727399644,54cyclictest23592-21ssh00:15:141
2727399644,54cyclictest23592-21ssh00:15:131
27273996038,9cyclictest21-21ksoftirqd/122:00:121
2727399577,41cyclictest28617-21ssh22:30:111
2725199564,44cyclictest30708-21ssh23:25:120
2727399556,37cyclictest3652-21ssh23:10:111
2727399553,37cyclictest24825-21ssh22:10:121
2727399553,37cyclictest24825-21ssh22:10:121
2731699542,35cyclictest11914-21ssh21:45:122
2725199531,38cyclictest7227-21ssh23:30:120
2725199523,41cyclictest8967-21ssh22:55:130
2725199503,33cyclictest2748-21ssh21:40:140
2725199503,33cyclictest2748-21ssh21:40:130
27355994926,13cyclictest35966irq/58-eth022:55:143
2725199491,32cyclictest7971-21ssh21:25:110
2731699481,35cyclictest27065-21ssh23:05:122
2727399481,32cyclictest5538-21ssh00:05:151
2735599476,25cyclictest35966irq/58-eth023:20:133
2735599473,25cyclictest35966irq/58-eth000:35:123
2735599472,27cyclictest35966irq/58-eth022:15:113
2725199472,40cyclictest15776-21ssh22:05:120
27251994633,6cyclictest0-21swapper/020:50:110
2735599453,28cyclictest35966irq/58-eth023:25:133
27355994526,12cyclictest35966irq/58-eth022:20:123
2735599433,25cyclictest35966irq/58-eth000:15:123
2735599433,25cyclictest35966irq/58-eth000:15:123
27355994227,11cyclictest35966irq/58-eth022:45:123
2727399425,25cyclictest13271-21ssh21:10:121
2727399408,23cyclictest21-21ksoftirqd/121:40:131
2727399408,23cyclictest21-21ksoftirqd/121:40:121
27273994019,11cyclictest21-21ksoftirqd/120:15:121
2727399397,22cyclictest21-21ksoftirqd/119:55:131
2727399382,27cyclictest28892-21ssh00:00:131
27273993813,15cyclictest32641-21ssh00:20:131
2735599372,21cyclictest35966irq/58-eth021:45:123
2731699368,22cyclictest27-21ksoftirqd/219:20:112
2727399362,24cyclictest31367-21ssh21:20:111
27273993611,17cyclictest21-21ksoftirqd/120:20:141
2735599356,20cyclictest35966irq/58-eth023:50:133
27355993520,8cyclictest35966irq/58-eth021:50:143
2725199353,24cyclictest1396-21ssh22:15:120
2725199351,27cyclictest12861-21diskmemload23:55:130
27316993422,8cyclictest261rcuc/221:20:122
2731699342,26cyclictest0-21swapper/222:00:112
2731699342,19cyclictest19899-21ssh23:55:142
2727399346,17cyclictest12861-21diskmemload21:55:121
2727399334,23cyclictest5226-21ssh22:35:101
2725199332,18cyclictest10520-21ssh22:20:120
2735599324,14cyclictest35966irq/58-eth022:35:113
2731699324,11cyclictest131rcu_preempt19:30:282
2731699322,18cyclictest7242-21ssh23:30:122
2727399321,22cyclictest16219-21sh23:35:131
27251993222,5cyclictest0-21swapper/020:35:140
2731699319,14cyclictest27-21ksoftirqd/222:09:042
2731699311,23cyclictest10536-21ssh22:20:142
2727399314,18cyclictest27256-21ssh00:35:141
2725199313,20cyclictest2471266kworker/u8:122:35:120
27251993118,4cyclictest12861-21diskmemload21:50:130
2735599304,18cyclictest35966irq/58-eth000:10:133
2735599304,18cyclictest35966irq/58-eth000:10:133
27355993016,9cyclictest35966irq/58-eth023:55:143
2731699309,15cyclictest0-21swapper/221:17:002
2731699303,17cyclictest27346irq/40-dwc2_hso21:35:112
2731699303,17cyclictest27346irq/40-dwc2_hso21:35:112
2731699301,19cyclictest27346irq/40-dwc2_hso19:50:112
2727399302,18cyclictest18025-21ssh23:00:131
2725199302,19cyclictest11920-21ssh21:45:120
27355992914,10cyclictest35966irq/58-eth023:30:133
2735599291,15cyclictest35966irq/58-eth000:20:123
2731699297,17cyclictest27-21ksoftirqd/223:25:012
2731699293,9cyclictest27-21ksoftirqd/221:25:192
2731699291,16cyclictest17049-21ssh21:30:132
27273992913,8cyclictest21-21ksoftirqd/100:25:131
27273992912,11cyclictest4805-21kworker/u8:019:10:111
27273992912,11cyclictest4805-21kworker/u8:019:10:101
2725199292,22cyclictest10908-21ssh23:50:120
27316992812,2cyclictest131rcu_preempt00:20:252
27316992810,5cyclictest0-21swapper/223:25:332
27273992817,4cyclictest0-21swapper/120:55:121
27251992821,4cyclictest0-21swapper/021:05:140
2735599273,16cyclictest0-21swapper/321:40:133
2735599273,16cyclictest0-21swapper/321:40:133
27355992710,11cyclictest35966irq/58-eth022:10:133
27355992710,11cyclictest35966irq/58-eth022:10:123
2731699272,19cyclictest25187-21ssh23:40:142
2731699272,16cyclictest27346irq/40-dwc2_hso22:50:112
2727399272,18cyclictest8991-21ssh22:55:141
2725199271,21cyclictest2613366ssh21:35:130
2725199271,21cyclictest2613366ssh21:35:120
27251992710,8cyclictest0-21swapper/000:22:350
2731699265,12cyclictest27-21ksoftirqd/200:19:142
2731699265,12cyclictest27-21ksoftirqd/200:19:132
2731699264,8cyclictest131rcu_preempt23:15:172
2727399269,6cyclictest32345-21ssh22:50:121
2727399262,21cyclictest18214-21ssh00:30:111
2725199265,5cyclictest12-21ksoftirqd/000:27:400
27251992611,5cyclictest0-21swapper/023:23:390
27251992611,10cyclictest12-21ksoftirqd/022:52:410
2735599252,12cyclictest35966irq/58-eth021:15:133
2731699259,9cyclictest0-21swapper/220:00:132
2731699257,4cyclictest27-21ksoftirqd/222:33:202
2731699257,12cyclictest27-21ksoftirqd/200:26:572
2731699256,9cyclictest131rcu_preempt23:35:252
2731699255,7cyclictest87766lldpd23:12:132
2731699254,6cyclictest131rcu_preempt21:42:562
2731699254,6cyclictest131rcu_preempt21:42:552
2731699254,14cyclictest0-21swapper/223:50:122
2731699252,10cyclictest27-21ksoftirqd/222:40:172
2731699251,8cyclictest1962266ssh22:25:132
27316992516,4cyclictest0-21swapper/221:50:132
2731699251,20cyclictest31270-21cut20:55:172
2731699251,17cyclictest27346irq/40-dwc2_hso19:15:112
2725199253,14cyclictest12-21ksoftirqd/000:10:140
2725199253,14cyclictest12-21ksoftirqd/000:10:130
27251992512,4cyclictest12-21ksoftirqd/023:45:180
27251992512,4cyclictest12-21ksoftirqd/023:45:180
27251992512,4cyclictest12-21ksoftirqd/022:26:290
2725199251,19cyclictest27080-21ssh23:05:120
27251992511,5cyclictest0-21swapper/021:12:210
2735599244,15cyclictest35966irq/58-eth000:30:123
2731699249,12cyclictest18292-21/usr/sbin/munin00:30:132
2731699248,8cyclictest27-21ksoftirqd/223:00:272
2731699243,18cyclictest26045-21kthreadcore22:45:222
2731699242,9cyclictest0-21swapper/219:10:042
2731699242,9cyclictest0-21swapper/219:10:042
2731699242,4cyclictest27-21ksoftirqd/220:50:142
27316992413,9cyclictest733-21ssh22:14:432
27316992413,9cyclictest733-21ssh22:14:422
2727399245,12cyclictest21718-21ssh23:20:121
27273992415,6cyclictest0-21swapper/119:40:011
2727399241,17cyclictest23306-21sh22:45:131
2725199249,5cyclictest0-21swapper/021:58:020
2725199249,4cyclictest12-21ksoftirqd/000:01:580
27251992412,9cyclictest12313-21cron23:15:020
2725199241,17cyclictest22326-21apt-get21:15:120
27251992411,5cyclictest0-21swapper/023:15:160
27251992410,8cyclictest0-21swapper/000:16:040
27251992410,8cyclictest0-21swapper/000:16:030
2735599232,11cyclictest35966irq/58-eth021:35:123
2735599232,11cyclictest35966irq/58-eth021:35:113
2735599232,11cyclictest35966irq/58-eth020:20:133
2731699238,3cyclictest27-21ksoftirqd/221:13:102
2731699238,2cyclictest131rcu_preempt22:15:222
2731699237,8cyclictest27-21ksoftirqd/222:56:322
2731699235,10cyclictest27-21ksoftirqd/200:36:352
2731699234,13cyclictest27-21ksoftirqd/223:46:242
2731699234,13cyclictest27-21ksoftirqd/223:46:232
2731699232,8cyclictest27-21ksoftirqd/200:10:122
2731699232,8cyclictest27-21ksoftirqd/200:10:112
2731699231,5cyclictest307166rm00:02:362
27316992312,8cyclictest27-21ksoftirqd/220:30:122
2731699231,12cyclictest0-21swapper/219:45:122
27316992310,10cyclictest261rcuc/220:25:012
2727399234,12cyclictest0-21swapper/120:05:131
2727399231,17cyclictest14304-21sh22:40:131
2725199239,6cyclictest0-21swapper/000:07:120
27251992311,5cyclictest0-21swapper/022:41:420
27251992311,4cyclictest0-21swapper/022:13:360
27251992311,4cyclictest0-21swapper/022:13:350
2735599229,8cyclictest35966irq/58-eth021:25:123
2731699226,7cyclictest27-21ksoftirqd/200:05:402
2731699224,7cyclictest131rcu_preempt19:25:202
2731699223,16cyclictest25297-21fschecks_count20:15:172
2731699223,12cyclictest10532-21munin-plugin-st20:00:012
2731699222,12cyclictest27346irq/40-dwc2_hso20:25:142
27316992214,6cyclictest5409-21ssh22:35:142
2731699221,13cyclictest12866-21apt21:10:002
2727399222,12cyclictest15801-21ssh22:05:141
2727399221,16cyclictest19625-21rm22:25:131
2727399221,15cyclictest3516-21ls19:15:271
2725199229,5cyclictest0-21swapper/023:00:580
2725199229,5cyclictest0-21swapper/000:38:060
2725199229,4cyclictest12-21ksoftirqd/022:33:350
2725199228,8cyclictest0-21swapper/020:20:310
2725199228,2cyclictest131rcu_preempt23:37:540
2725199227,8cyclictest12-21ksoftirqd/021:20:360
27251992213,4cyclictest0-21swapper/020:25:120
2725199221,18cyclictest4649-21find19:20:110
2725199221,18cyclictest3506-21find21:00:110
2725199221,18cyclictest3506-21find21:00:110
27251992210,4cyclictest0-21swapper/022:48:330
2731699218,6cyclictest27-21ksoftirqd/221:55:382
2731699213,15cyclictest27346irq/40-dwc2_hso21:00:132
2731699213,15cyclictest27346irq/40-dwc2_hso21:00:122
2731699213,15cyclictest17560-21kthreadcore20:05:202
2731699211,6cyclictest0-21swapper/219:40:142
2727399212,16cyclictest24605-21uname19:40:191
2727399211,13cyclictest20368-21find20:10:111
27273992110,6cyclictest664-21ntpd20:45:121
2725199219,5cyclictest131rcu_preempt00:32:150
2725199215,4cyclictest131rcu_preempt23:40:260
2725199215,2cyclictest131rcu_preempt21:31:550
2725199214,3cyclictest131rcu_preempt19:55:250
2725199212,16cyclictest22273-21kthreadcore20:10:210
2725199212,16cyclictest16695-21kthreadcore20:05:190
2725199211,15cyclictest10736-21ssh22:00:280
27355992011,4cyclictest0-21swapper/323:10:133
27355992010,7cyclictest0-21swapper/323:00:343
2731699203,11cyclictest24775-21kthreadcore20:45:242
2731699201,16cyclictest19691-21kthreadcore20:40:212
2731699201,11cyclictest20927-21sort19:35:202
2727399209,3cyclictest131rcu_preempt21:25:181
2727399205,4cyclictest131rcu_preempt21:30:161
2727399201,9cyclictest8509-21grep19:20:441
2727399201,2cyclictest131rcu_preempt21:35:121
2727399201,2cyclictest131rcu_preempt21:35:111
2727399201,16cyclictest14261-21apt-get19:30:121
2727399201,16cyclictest12040-21grep20:35:151
2727399201,15cyclictest22343-21ssh21:15:131
2727399201,14cyclictest27205-21entropy23:05:141
2727399201,13cyclictest30081-21ls20:50:271
2727399201,13cyclictest1410-21latency_hist23:45:011
2725199203,5cyclictest12-21ksoftirqd/019:40:170
2725199202,15cyclictest27338-21cp19:10:020
2725199202,10cyclictest0-21swapper/019:15:120
27251992014,4cyclictest0-21swapper/020:00:150
2725199201,16cyclictest23104-21kthreadcore20:45:220
27251992010,7cyclictest0-21swapper/019:50:270
2735599199,8cyclictest0-21swapper/322:33:143
2735599193,12cyclictest35966irq/58-eth021:10:123
2735599192,11cyclictest0-21swapper/320:15:143
2735599191,13cyclictest0-21swapper/323:35:123
2735599191,13cyclictest0-21swapper/321:55:163
2735599191,13cyclictest0-21swapper/319:25:163
2735599191,13cyclictest0-21swapper/300:05:293
27355991910,6cyclictest35966irq/58-eth022:50:113
27355991910,6cyclictest35966irq/58-eth022:05:133
27355991910,6cyclictest0-21swapper/321:20:483
27355991910,6cyclictest0-21swapper/300:02:433
2731699192,9cyclictest12433-21irqrtprio20:35:172
2731699191,10cyclictest20613-21idleruntime20:10:162
2727399198,8cyclictest6848-21cron23:30:021
2727399192,14cyclictest21424-21irqcore21:50:171
2727399191,9cyclictest710-21cron19:30:011
2727399191,6cyclictest1362-21/usr/sbin/munin22:15:251
2727399191,5cyclictest0-21swapper/122:20:241
2727399191,10cyclictest13881-21tr20:00:211
2727399191,10cyclictest12796-21diskstats23:15:141
2725199199,2cyclictest131rcu_preempt20:30:180
2725199196,5cyclictest12-21ksoftirqd/019:45:260
2725199191,15cyclictest20493-21kthreadcore19:35:190
2735599189,6cyclictest35966irq/58-eth020:00:143
2735599183,8cyclictest35966irq/58-eth020:45:143
2735599182,10cyclictest0-21swapper/321:30:123
2735599182,10cyclictest0-21swapper/320:10:143
2735599182,10cyclictest0-21swapper/319:40:143
2735599182,10cyclictest0-21swapper/319:20:133
2735599182,10cyclictest0-21swapper/319:15:133
27355991811,4cyclictest0-21swapper/322:25:143
27355991810,6cyclictest0-21swapper/323:19:593
2731699183,12cyclictest27391-21latency_hist19:10:022
2727399188,8cyclictest19479-21cron23:55:011
2727399182,14cyclictest8462-21kthreadcore23:30:191
27273991811,4cyclictest2202-21irqcore23:45:181
27273991811,4cyclictest2202-21irqcore23:45:171
2727399181,14cyclictest11767-21ls20:35:021
2727399181,10cyclictest11449-21apt-get21:45:021
27251991813,3cyclictest0-21swapper/020:15:150
27251991813,3cyclictest0-21swapper/019:25:140
27251991812,4cyclictest0-21swapper/020:55:200
2725199181,15cyclictest30730-21grep19:10:210
2725199181,15cyclictest30730-21grep19:10:200
2725199181,15cyclictest17070-21irqcore20:40:170
2735599179,6cyclictest0-21swapper/323:40:283
2735599179,6cyclictest0-21swapper/300:25:103
2735599179,5cyclictest35966irq/58-eth022:00:133
2735599178,6cyclictest0-21swapper/322:40:153
2735599172,10cyclictest0-21swapper/323:05:123
2735599172,10cyclictest0-21swapper/321:05:153
2735599172,10cyclictest0-21swapper/320:55:143
2735599171,8cyclictest35966irq/58-eth020:30:123
2735599171,13cyclictest0-21swapper/319:30:153
2735599171,10cyclictest0-21swapper/320:50:183
27355991710,4cyclictest0-21swapper/323:45:133
27355991710,4cyclictest0-21swapper/323:45:133
2727399172,12cyclictest9539-21kthreadcore21:05:201
2727399172,10cyclictest0-21swapper/120:40:141
2727399171,14cyclictest31367-21kthreadcore19:45:201
2727399171,13cyclictest1381-21wc19:50:161
2727399171,12cyclictest14893-21if_err_eth100:10:181
2727399171,12cyclictest14893-21if_err_eth100:10:171
2727399170,4cyclictest19872-21apt-get23:55:131
2735599162,10cyclictest0-21swapper/319:55:133
2735599162,10cyclictest0-21swapper/319:45:133
2727399161,12cyclictest3962-21kthreadcore20:25:191
2727399161,10cyclictest5280-21kthreadcore21:00:211
2727399161,10cyclictest5280-21kthreadcore21:00:201
27251991610,4cyclictest0-21swapper/019:30:120
2735599152,9cyclictest0-21swapper/319:50:133
2735599151,11cyclictest0-21swapper/319:40:003
2735599151,10cyclictest0-21swapper/320:35:183
2727399151,9cyclictest27311-21latency_hist19:10:021
2735599138,3cyclictest0-21swapper/320:40:153
2735599131,10cyclictest0-21swapper/320:25:273
2735599131,10cyclictest0-21swapper/319:10:293
2735599131,10cyclictest0-21swapper/319:10:283
2735599124,5cyclictest35966irq/58-eth021:00:223
2735599124,5cyclictest35966irq/58-eth021:00:213
2735599111,7cyclictest0-21swapper/320:10:013
2735599111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional