You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 14:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sat Dec 20, 2025 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
577998323,8cyclictest131rcu_preempt21:30:131
58599722,43cyclictest6711-21ssh23:05:122
58599643,44cyclictest15377-21ssh00:20:142
56999527,40cyclictest25254-21ssh22:40:140
57799493,34cyclictest2074-21ssh22:45:111
58599473,22cyclictest27-21ksoftirqd/219:45:132
57799462,4cyclictest131rcu_preempt00:20:141
56999441,31cyclictest20212-21sh21:45:120
57799431,4cyclictest131rcu_preempt19:10:101
58599412,26cyclictest20635-21ssh22:55:142
57799411,32cyclictest25171-21ssh22:05:121
57799411,29cyclictest20467-21ssh23:30:131
56999411,32cyclictest11317-21ssh22:50:110
585993927,6cyclictest27-21ksoftirqd/223:25:142
56999391,30cyclictest10783-21rm21:40:110
569993911,9cyclictest131rcu_preempt00:10:130
58599387,8cyclictest131rcu_preempt19:40:132
58599387,8cyclictest131rcu_preempt19:40:132
58599381,20cyclictest15944-21ssh22:35:142
58599381,20cyclictest15944-21ssh22:35:142
58599370,3cyclictest131rcu_preempt20:00:122
57799373,12cyclictest131rcu_preempt22:10:131
59399368,16cyclictest28366irq/58-eth023:00:123
59399361,17cyclictest28366irq/58-eth022:20:103
57799361,24cyclictest10889-21rm00:00:121
56999362,31cyclictest24510-21ssh00:25:130
58599356,12cyclictest27-21ksoftirqd/220:25:042
56999354,23cyclictest18796-21diskmemload00:30:150
577993421,5cyclictest18796-21diskmemload23:25:131
57799340,27cyclictest0-21swapper/119:55:131
56999341,24cyclictest25229-21ssh23:15:160
585993320,5cyclictest0-21swapper/222:25:132
57799334,23cyclictest0-21swapper/120:55:151
56999338,4cyclictest131rcu_preempt21:20:120
569993324,6cyclictest151rcuc/020:00:120
56999331,29cyclictest1426-21ssh21:35:130
56999323,15cyclictest18796-21diskmemload22:00:110
56999321,24cyclictest0-21swapper/020:50:100
593993119,8cyclictest28366irq/58-eth023:25:143
56999301,23cyclictest15022-21ssh21:25:130
58599292,20cyclictest0-21swapper/223:40:142
57799295,22cyclictest20235-21ssh21:45:141
57799291,25cyclictest11271-21ssh22:15:121
56999291,23cyclictest24902-21ssh23:50:120
59399282,20cyclictest0-21swapper/300:35:123
59399282,20cyclictest0-21swapper/300:35:123
58599283,18cyclictest22346irq/40-dwc2_hso23:35:122
58599282,20cyclictest0-21swapper/200:25:142
58599282,19cyclictest0-21swapper/219:30:132
58599282,19cyclictest0-21swapper/219:15:142
58599282,19cyclictest0-21swapper/219:15:142
577992813,11cyclictest21-21ksoftirqd/100:05:131
57799281,20cyclictest15951-21sh22:35:141
57799281,20cyclictest15951-21sh22:35:141
56999283,21cyclictest29948-21ssh23:00:140
56999282,20cyclictest6730-21ssh23:05:130
569992820,6cyclictest0-21swapper/023:20:120
569992815,10cyclictest12-21ksoftirqd/019:25:120
59399275,16cyclictest28366irq/58-eth022:15:123
59399272,15cyclictest28366irq/58-eth022:05:103
58599279,14cyclictest8557-21ssh22:30:212
58599273,17cyclictest22346irq/40-dwc2_hso19:55:132
58599273,16cyclictest22346irq/40-dwc2_hso19:10:112
58599272,20cyclictest0-21swapper/223:30:132
58599272,16cyclictest22346irq/40-dwc2_hso22:20:102
585992713,7cyclictest19562-21ssh21:25:302
58599271,18cyclictest16090-21grep22:00:152
577992715,6cyclictest21-21ksoftirqd/121:35:141
59399263,17cyclictest28366irq/58-eth021:25:133
57799262,4cyclictest131rcu_preempt22:55:151
57799262,19cyclictest10451-21ssh00:35:141
57799262,19cyclictest10451-21ssh00:35:141
57799262,19cyclictest10389-21apt-get19:20:101
577992612,11cyclictest21-21ksoftirqd/123:15:161
56999262,18cyclictest12-21ksoftirqd/023:40:130
593992514,7cyclictest0-21swapper/321:55:123
593992512,10cyclictest28366irq/58-eth022:50:123
593992510,12cyclictest0-21swapper/300:20:143
58599252,16cyclictest22346irq/40-dwc2_hso20:55:142
58599252,12cyclictest2043-21ssh23:20:132
585992510,7cyclictest0-21swapper/200:30:182
585992510,5cyclictest348-21kthreadcore23:00:222
57799257,4cyclictest131rcu_preempt23:50:111
57799252,17cyclictest845-21ssh23:35:241
56999252,19cyclictest18796-21diskmemload23:35:130
569992518,4cyclictest12-21ksoftirqd/021:50:130
569992514,3cyclictest12-21ksoftirqd/021:19:020
59399242,9cyclictest28366irq/58-eth020:05:133
58599242,5cyclictest0-21swapper/220:40:102
58599242,5cyclictest0-21swapper/220:40:102
58599240,17cyclictest17862-21ls20:35:142
57799248,5cyclictest19347-21ssh22:00:211
57799244,4cyclictest131rcu_preempt21:25:201
57799242,15cyclictest21-21ksoftirqd/120:15:141
59399238,3cyclictest28366irq/58-eth023:35:123
59399234,13cyclictest28466irq/59-eth021:45:133
59399233,16cyclictest28366irq/58-eth022:10:133
593992310,10cyclictest0-21swapper/300:15:193
58599239,10cyclictest1-21systemd20:16:092
58599232,9cyclictest28879-21ssh22:40:252
58599232,18cyclictest22346irq/40-dwc2_hso00:00:132
585992312,9cyclictest13450-21ssh21:57:512
585992311,9cyclictest10273-21ssh00:35:032
585992311,9cyclictest10273-21ssh00:35:032
58599231,11cyclictest5587-21ssh21:20:122
58599230,15cyclictest32449-21ls20:50:152
57799239,11cyclictest21-21ksoftirqd/120:40:141
57799239,11cyclictest21-21ksoftirqd/120:40:141
577992313,5cyclictest211ksoftirqd/119:50:121
56999231,16cyclictest11291-21ssh22:15:130
569992310,9cyclictest12-21ksoftirqd/000:35:130
569992310,9cyclictest12-21ksoftirqd/000:35:130
59399226,13cyclictest28466irq/59-eth023:15:163
59399223,5cyclictest0-21swapper/321:40:123
59399223,11cyclictest28366irq/58-eth021:10:133
58599229,10cyclictest31854-21sh21:15:222
58599222,11cyclictest19228-21ssh21:10:132
585992215,4cyclictest3176-21ssh22:10:202
57799223,8cyclictest21-21ksoftirqd/120:50:101
57799222,15cyclictest29574-21ssh21:50:141
577992214,6cyclictest21-21ksoftirqd/121:40:111
57799221,3cyclictest131rcu_preempt21:15:141
57799221,18cyclictest11831-21ls20:25:271
577992210,10cyclictest0-21swapper/119:40:231
577992210,10cyclictest0-21swapper/119:40:231
56999229,3cyclictest12-21ksoftirqd/023:30:030
56999222,15cyclictest24279-21ssh22:55:260
569992216,4cyclictest0-21swapper/000:15:250
569992216,3cyclictest0-21swapper/000:20:180
569992215,3cyclictest12-21ksoftirqd/023:25:260
56999221,17cyclictest2073-21apt-get22:45:120
569992210,4cyclictest12-21ksoftirqd/022:07:580
569992210,4cyclictest12-21ksoftirqd/021:55:310
59399217,9cyclictest28366irq/58-eth022:40:143
59399212,12cyclictest0-21swapper/323:30:123
593992113,6cyclictest0-21swapper/322:29:143
59399211,15cyclictest28366irq/58-eth000:05:143
58599218,10cyclictest26097-21kthreadcore22:05:172
58599217,12cyclictest0-21swapper/219:50:222
58599216,12cyclictest27-21ksoftirqd/221:45:152
58599213,9cyclictest5354-21ntp_offset23:55:232
58599213,9cyclictest12393-21kthreadcore21:00:212
58599212,13cyclictest22346irq/40-dwc2_hso22:45:122
58599212,13cyclictest22346irq/40-dwc2_hso20:05:142
58599212,10cyclictest0-21swapper/220:30:142
585992115,4cyclictest25143-21ssh21:30:192
585992112,6cyclictest0-21swapper/221:50:282
58599211,17cyclictest1365366ssh22:50:192
58599211,11cyclictest5624-21unixbench_singl21:35:302
585992110,9cyclictest16237-21ssh21:41:072
58599210,11cyclictest69366ssh23:18:592
57799211,4cyclictest6261-21cpuspeed00:15:131
577992113,5cyclictest0-21swapper/119:35:191
577992112,7cyclictest13955-21cron21:05:001
57799211,17cyclictest1103-21ntp_kernel_pll_22:25:251
57799211,16cyclictest20550-21rm22:20:111
56999218,10cyclictest19674-21ntp_states22:35:260
56999218,10cyclictest19674-21ntp_states22:35:260
56999212,16cyclictest30005-21ssh22:25:150
56999211,14cyclictest20763-21cut22:20:130
569992110,3cyclictest12-21ksoftirqd/020:10:140
569992110,3cyclictest12-21ksoftirqd/020:10:140
59399209,7cyclictest0-21swapper/300:30:323
59399204,9cyclictest0-21swapper/323:45:133
59399203,8cyclictest28366irq/58-eth023:20:123
593992011,6cyclictest28366irq/58-eth022:45:123
58599206,11cyclictest30914-21wc20:45:262
58599203,9cyclictest608-21latency_hist19:10:012
58599203,9cyclictest29777-21tail20:10:272
58599203,9cyclictest29777-21tail20:10:272
58599201,9cyclictest9339-21kthreadcore00:15:212
58599201,9cyclictest17291-21kthreadcore19:25:202
58599201,16cyclictest11709-21irqcore22:15:172
58599201,15cyclictest16706-21cut23:45:202
58599201,14cyclictest18555-21kthreadcore23:10:212
58599201,11cyclictest25090-21fschecks_time23:50:142
585992010,8cyclictest0-21swapper/200:07:462
57799208,9cyclictest4021-21tr00:30:241
57799202,4cyclictest29318-21apt-get00:10:131
57799202,13cyclictest16527-21kworker/u8:121:55:121
57799201,9cyclictest16738-21cut19:25:191
577992012,6cyclictest12619-21ssh22:31:431
577992012,5cyclictest0-21swapper/100:25:131
57799201,16cyclictest21154-21find20:05:121
57799201,14cyclictest7823-21basename20:25:011
57799201,14cyclictest19439-21diskmemload23:45:271
56999208,6cyclictest12692-21sh20:30:000
56999207,2cyclictest12-21ksoftirqd/022:35:000
56999207,10cyclictest17484-21basename20:35:010
569992014,3cyclictest12-21ksoftirqd/000:00:010
569992013,4cyclictest12-21ksoftirqd/020:40:130
569992013,4cyclictest12-21ksoftirqd/020:40:130
569992012,4cyclictest12-21ksoftirqd/000:05:150
56999201,16cyclictest19365-21cut21:10:150
56999201,14cyclictest19612-21ntp_192.168.11423:10:240
56999201,10cyclictest28061-21awk21:30:240
59399192,11cyclictest0-21swapper/319:50:133
593991911,5cyclictest28366irq/58-eth000:25:143
59399191,13cyclictest0-21swapper/321:15:243
59399191,12cyclictest0-21swapper/323:55:123
593991910,6cyclictest0-21swapper/323:44:003
58599198,9cyclictest0-21swapper/200:11:532
58599198,8cyclictest5426-21kthreadcore20:20:202
58599191,9cyclictest14382-21fschecks_time21:05:162
58599191,8cyclictest10613-21wc19:20:142
58599191,10cyclictest28082-21cat19:35:232
57799194,12cyclictest131rcu_preempt23:00:141
57799192,7cyclictest10144-21munin-run19:20:001
57799192,7cyclictest10144-21munin-run19:20:001
57799192,13cyclictest2014-21apt-get23:20:131
57799192,11cyclictest0-21swapper/123:55:121
57799191,8cyclictest0-21swapper/120:45:121
57799191,5cyclictest131rcu_preempt22:40:141
57799191,4cyclictest9978-21memory23:40:241
57799191,3cyclictest131rcu_preempt23:10:141
57799191,14cyclictest22283-21kthreadcore19:30:201
57799191,14cyclictest17787-21apt-get20:35:121
57799191,14cyclictest17654-21cat20:35:011
577991910,6cyclictest18196-21ssh22:52:451
56999197,2cyclictest131rcu_preempt19:10:100
56999192,13cyclictest151rcuc/023:45:140
569991912,4cyclictest12-21ksoftirqd/020:35:170
569991912,4cyclictest12-21ksoftirqd/019:20:200
59399189,6cyclictest28366irq/58-eth000:00:133
59399189,6cyclictest0-21swapper/323:06:363
59399184,9cyclictest28366irq/58-eth022:55:143
59399182,9cyclictest0-21swapper/320:20:123
59399182,10cyclictest0-21swapper/322:35:143
59399182,10cyclictest0-21swapper/322:35:143
593991812,4cyclictest0-21swapper/321:35:233
59399181,13cyclictest0-21swapper/320:25:273
59399181,11cyclictest0-21swapper/319:10:193
593991810,6cyclictest0-21swapper/321:33:403
593991810,3cyclictest0-21swapper/300:10:083
57799189,6cyclictest28552-21kthreadcore20:10:201
57799189,6cyclictest28552-21kthreadcore20:10:201
57799183,12cyclictest9010-21kthreadcore23:05:201
57799182,11cyclictest0-21swapper/119:45:141
57799181,7cyclictest131rcu_preempt20:00:121
56999188,7cyclictest4670-21df_inode20:55:160
56999182,9cyclictest3265-21/usr/sbin/munin20:20:220
56999182,2cyclictest131rcu_preempt21:10:000
56999182,2cyclictest131rcu_preempt19:15:130
56999182,2cyclictest131rcu_preempt19:15:130
569991812,4cyclictest0-21swapper/000:00:170
59399179,6cyclictest0-21swapper/322:32:363
59399179,6cyclictest0-21swapper/321:50:073
59399179,6cyclictest0-21swapper/321:21:473
59399172,10cyclictest0-21swapper/319:45:143
593991711,4cyclictest0-21swapper/322:00:123
59399171,12cyclictest0-21swapper/320:00:143
59399171,12cyclictest0-21swapper/319:30:273
59399171,11cyclictest0-21swapper/323:10:213
57799172,12cyclictest19776-21irqcore21:10:171
57799172,12cyclictest15530-21kthreadcore21:05:201
57799171,13cyclictest14596-21basename21:25:011
56999173,11cyclictest21253-21kthreadcore19:30:190
56999172,3cyclictest131rcu_preempt19:40:190
56999172,3cyclictest131rcu_preempt19:40:190
56999172,10cyclictest0-21swapper/021:00:140
56999171,4cyclictest131rcu_preempt20:15:150
569991712,3cyclictest11909-21irqcore19:55:180
569991711,4cyclictest0-21swapper/020:05:250
56999171,13cyclictest5248-21tr22:10:220
56999171,13cyclictest10462-21cat19:50:270
59399166,6cyclictest0-21swapper/323:50:343
59399161,9cyclictest0-21swapper/319:55:163
56999162,10cyclictest0-21swapper/019:35:150
56999161,2cyclictest131rcu_preempt20:45:120
56999161,10cyclictest2389-21irqcore19:45:180
59399152,9cyclictest0-21swapper/320:45:133
59399151,12cyclictest0-21swapper/319:45:003
59399151,12cyclictest0-21swapper/319:45:003
593991510,3cyclictest33-21ksoftirqd/320:50:143
57799151,11cyclictest617-21grep19:10:011
56999151,9cyclictest705-21latency_hist19:10:010
59399149,3cyclictest33-21ksoftirqd/319:25:243
59399141,9cyclictest28366irq/58-eth019:15:143
59399141,9cyclictest28366irq/58-eth019:15:143
59399141,10cyclictest0-21swapper/320:15:293
59399139,2cyclictest33-21ksoftirqd/320:35:173
59399133,8cyclictest0-21swapper/320:10:143
59399133,8cyclictest0-21swapper/320:10:143
59399132,9cyclictest0-21swapper/319:35:223
59399131,6cyclictest0-21swapper/320:55:303
59399122,7cyclictest0-21swapper/319:20:103
59399121,9cyclictest0-21swapper/321:00:163
59399121,8cyclictest0-21swapper/320:40:133
59399121,8cyclictest0-21swapper/320:40:133
59399121,7cyclictest0-21swapper/320:30:183
59399111,6cyclictest0-21swapper/321:05:153
59399101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional