You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-08 - 06:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Apr 08, 2026 00:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3263699673,39cyclictest30139-21ssh23:40:142
32636996332,18cyclictest131rcu_preempt21:05:132
3263699613,35cyclictest6783-21ssh23:45:132
3261999589,24cyclictest12-21ksoftirqd/000:35:140
32619995812,9cyclictest131rcu_preempt21:25:120
32636995120,18cyclictest131rcu_preempt21:45:122
3263699441,10cyclictest0-21swapper/223:05:142
3263699431,36cyclictest10394-21ssh00:05:122
3261999416,8cyclictest131rcu_preempt20:25:140
3261999416,8cyclictest131rcu_preempt20:25:130
32636994020,10cyclictest27-21ksoftirqd/219:15:132
3261999408,5cyclictest131rcu_preempt23:05:130
32645993922,11cyclictest28366irq/58-eth023:40:133
32645993922,10cyclictest28366irq/58-eth021:25:133
3263699396,6cyclictest131rcu_preempt21:30:132
32619993727,7cyclictest12-21ksoftirqd/020:35:130
32645993615,12cyclictest28366irq/58-eth022:10:123
32636993618,8cyclictest27-21ksoftirqd/200:10:132
3262899351,24cyclictest1384-21ssh00:00:141
32636993429,3cyclictest2746ksoftirqd/219:40:132
32636993429,3cyclictest2746ksoftirqd/219:40:132
32645993317,12cyclictest0-21swapper/321:45:123
3264599331,17cyclictest28366irq/58-eth021:20:133
3262899331,28cyclictest4949-21ssh00:20:121
32619993314,5cyclictest131rcu_preempt00:05:140
32619993216,13cyclictest12-21ksoftirqd/020:55:150
3263699312,14cyclictest2661366ssh23:20:122
3263699311,8cyclictest22914-21ssh00:30:132
32619993118,8cyclictest0-21swapper/023:10:130
3261999311,22cyclictest17574-21ssh23:15:130
3262899309,6cyclictest131rcu_preempt20:05:121
32628993017,4cyclictest9844-21kthreadcore21:00:191
3261999308,18cyclictest12-21ksoftirqd/019:55:150
3261999308,18cyclictest12-21ksoftirqd/019:55:140
3261999307,6cyclictest131rcu_preempt22:25:150
3263699292,20cyclictest0-21swapper/220:30:032
3263699283,9cyclictest131rcu_preempt22:35:142
3263699282,21cyclictest22284-21ssh00:29:442
3262899284,20cyclictest21186-21ssh23:35:151
3262899281,20cyclictest0-21swapper/122:10:121
3262899281,19cyclictest10182-21ssh22:35:121
3261999282,11cyclictest28279-21ssh22:45:130
3263699272,18cyclictest22346irq/40-dwc2_hso20:40:122
3263699271,9cyclictest24892-21ssh23:55:122
3263699271,9cyclictest24892-21ssh23:55:122
3263699271,19cyclictest18728-21ssh21:10:132
3262899271,22cyclictest13860-21rm22:55:141
3261999272,21cyclictest11385-21ssh22:15:440
3261999272,18cyclictest1174-21rm22:30:140
3261999272,18cyclictest1174-21rm22:30:130
32619992718,7cyclictest12-21ksoftirqd/000:20:100
3264599261,18cyclictest0-21swapper/323:49:443
3263699263,17cyclictest22346irq/40-dwc2_hso23:30:112
3263699263,16cyclictest22346irq/40-dwc2_hso22:10:122
32636992618,5cyclictest26651-21ssh21:50:162
32636992618,5cyclictest26651-21ssh21:50:152
32636992617,4cyclictest19252-21sh22:40:132
3263699261,14cyclictest31730-21fschecks_time21:35:162
3262899261,18cyclictest21-21ksoftirqd/121:30:121
3261999265,17cyclictest18319-21diskmemload21:50:110
3261999265,17cyclictest18319-21diskmemload21:50:100
3264599252,13cyclictest28366irq/58-eth023:20:133
32645992518,5cyclictest28366irq/58-eth000:35:153
3263699253,17cyclictest22346irq/40-dwc2_hso20:10:132
3263699252,18cyclictest22884-21ssh23:00:132
3263699251,9cyclictest13143-21ssh21:40:432
3262899253,19cyclictest201rcuc/100:10:111
3262899253,16cyclictest5021-21forks22:50:151
3262899252,16cyclictest27736-21rm21:15:121
3262899251,20cyclictest22120-21ssh21:45:441
3262899251,16cyclictest29630-21kworker/1:122:31:001
3262899251,16cyclictest29630-21kworker/1:122:31:001
3262899251,11cyclictest0-21swapper/123:23:081
3261999257,5cyclictest131rcu_preempt21:30:130
3261999252,18cyclictest6761-21apt-get23:45:110
32619992512,3cyclictest12-21ksoftirqd/020:05:180
32619992510,4cyclictest131rcu_preempt00:25:210
3263699243,15cyclictest22346irq/40-dwc2_hso22:05:132
3263699242,3cyclictest131rcu_preempt22:55:122
3262899249,7cyclictest131rcu_preempt23:19:341
3262899249,4cyclictest21-21ksoftirqd/121:10:281
3262899243,3cyclictest131rcu_preempt00:15:151
32628992415,6cyclictest12609-21grep23:10:291
3262899241,2cyclictest131rcu_preempt23:45:031
3262899241,14cyclictest131rcu_preempt23:32:081
3264599233,13cyclictest0-21swapper/321:30:443
32645992314,6cyclictest0-21swapper/321:35:113
32645992313,7cyclictest28366irq/58-eth000:30:133
3263699239,10cyclictest7842-21kthreadcore22:15:182
3263699237,3cyclictest91-21pr/ttyS020:40:012
3263699237,14cyclictest131rcu_preempt00:20:042
3263699232,6cyclictest0-21swapper/223:29:462
3263699232,16cyclictest0-21swapper/222:33:452
3263699232,16cyclictest0-21swapper/222:33:442
3263699232,14cyclictest22346irq/40-dwc2_hso19:25:142
3263699232,12cyclictest2661-21ssh21:55:122
3263699231,7cyclictest489466ssh21:20:172
3263699231,7cyclictest0-21swapper/222:25:162
3263699231,12cyclictest8080-21ssh00:40:022
3262899238,9cyclictest30208-21kworker/1:220:15:231
3262899234,3cyclictest131rcu_preempt20:10:281
3262899232,17cyclictest11733-21apt-get22:00:121
3262899232,15cyclictest0-21swapper/120:50:211
3262899231,2cyclictest131rcu_preempt23:25:451
3262899231,2cyclictest131rcu_preempt23:04:081
3262899231,2cyclictest131rcu_preempt22:41:421
3262899231,18cyclictest8167-21ssh21:40:131
3262899231,17cyclictest0-21swapper/100:05:141
3262899231,14cyclictest131rcu_preempt22:20:261
3262899231,13cyclictest131rcu_preempt22:47:521
32628992311,10cyclictest0-21swapper/100:36:441
3261999239,8cyclictest0-21swapper/022:55:120
3261999239,3cyclictest12-21ksoftirqd/000:04:560
3261999232,17cyclictest12-21ksoftirqd/023:40:130
32619992311,3cyclictest12-21ksoftirqd/021:18:000
32619992311,3cyclictest12-21ksoftirqd/020:45:020
32619992310,3cyclictest12-21ksoftirqd/021:35:260
3263699228,11cyclictest29529-21kthreadcore20:45:212
3263699224,14cyclictest22346irq/40-dwc2_hso23:10:132
3263699222,16cyclictest0-21swapper/223:53:452
32636992215,4cyclictest13618-21ssh21:25:172
32636992211,9cyclictest2814-21ssh21:18:412
32636992210,8cyclictest23157-21kthreadcore23:35:222
3262899229,10cyclictest1421-21kthreadcore23:05:211
3262899224,3cyclictest131rcu_preempt23:43:561
3262899222,3cyclictest131rcu_preempt22:05:221
3262899222,17cyclictest12167-21ssh22:16:381
3262899221,3cyclictest131rcu_preempt21:55:351
3262899221,3cyclictest0-21swapper/123:56:571
3262899221,3cyclictest0-21swapper/123:56:571
32628992212,6cyclictest942-21kthreadcore21:35:201
32628992212,5cyclictest0-21swapper/121:25:141
3262899221,18cyclictest26080-21sh21:50:121
3262899221,18cyclictest26080-21sh21:50:111
3262899221,13cyclictest131rcu_preempt21:22:071
32628992211,3cyclictest0-21swapper/123:50:231
3262899220,2cyclictest131rcu_preempt00:26:341
3261999228,3cyclictest12-21ksoftirqd/023:55:150
3261999228,3cyclictest12-21ksoftirqd/023:55:140
3261999223,10cyclictest3177-21ssh23:25:130
32619992212,8cyclictest12-21ksoftirqd/019:30:130
32619992212,5cyclictest12-21ksoftirqd/021:50:010
3261999221,17cyclictest15797-21ssh23:50:140
32619992211,3cyclictest12-21ksoftirqd/020:50:210
32619992210,3cyclictest12-21ksoftirqd/019:50:010
32619992210,3cyclictest12-21ksoftirqd/019:50:000
32619992210,3cyclictest12-21ksoftirqd/019:25:220
32645992112,4cyclictest28366irq/58-eth023:50:133
32645992110,9cyclictest0-21swapper/321:14:033
3263699217,11cyclictest8670-21kthreadcore20:25:212
3263699217,11cyclictest8670-21kthreadcore20:25:202
3263699211,16cyclictest6694-21fschecks_time19:50:162
3263699211,16cyclictest413-21cpuspeed19:10:142
3263699211,11cyclictest6388-21grep20:20:272
3263699211,11cyclictest15646-21fschecks_count22:20:152
3263699211,11cyclictest1526-21kthreadcore20:50:202
3262899211,2cyclictest131rcu_preempt00:30:491
3262899211,16cyclictest10018-21apt-get19:20:131
3262899210,4cyclictest131rcu_preempt20:30:181
3262899210,13cyclictest131rcu_preempt19:50:221
3262899210,10cyclictest19720-21switchtime20:00:271
3261999219,9cyclictest2792-21forks21:55:140
3261999219,3cyclictest12-21ksoftirqd/000:15:220
3261999218,4cyclictest12-21ksoftirqd/021:40:250
3261999218,3cyclictest12-21ksoftirqd/022:05:130
3261999214,13cyclictest4354-21ssh21:20:140
3261999214,13cyclictest22924-21ssh00:30:130
3261999212,14cyclictest21184-21sh23:35:140
3261999211,9cyclictest10880-21irqrtprio22:35:180
3261999211,4cyclictest131rcu_preempt19:10:170
32619992114,3cyclictest12-21ksoftirqd/022:00:120
3261999211,10cyclictest20288-21cut21:10:210
32619992110,3cyclictest12-21ksoftirqd/020:00:210
3264599209,8cyclictest0-21swapper/322:25:353
32645992012,5cyclictest28366irq/58-eth021:55:113
3263699208,9cyclictest20666-21cron20:05:012
3263699203,9cyclictest27510-21kthreadcore19:35:212
3263699202,9cyclictest4538-21cut00:20:012
3263699201,9cyclictest12783-21tail21:00:272
32636992013,5cyclictest0-21swapper/222:50:022
3263699201,16cyclictest21747-21kthreadcore19:30:222
3263699201,15cyclictest11713-21/usr/sbin/munin22:00:182
3263699201,13cyclictest1388-21ssh00:00:152
3263699201,11cyclictest6202-21cron19:50:012
3263699201,11cyclictest6202-21cron19:50:002
3263699201,10cyclictest4897-21ssh22:50:122
3263699201,10cyclictest30572-21cut20:15:152
3263699201,10cyclictest18235-21irqrtprio23:15:192
3262899208,4cyclictest21-21ksoftirqd/122:25:301
3262899202,3cyclictest131rcu_preempt19:10:241
3262899201,3cyclictest131rcu_preempt19:35:191
3262899201,3cyclictest131rcu_preempt19:25:161
3261999209,3cyclictest12-21ksoftirqd/022:22:560
3261999209,3cyclictest12-21ksoftirqd/022:10:110
3261999207,3cyclictest12-21ksoftirqd/019:35:160
3261999205,5cyclictest12-21ksoftirqd/000:10:160
32619992014,3cyclictest12-21ksoftirqd/022:50:120
32619992011,6cyclictest27006-21sort20:45:140
32619992010,2cyclictest12-21ksoftirqd/020:10:180
3264599199,8cyclictest0-21swapper/323:37:113
3264599199,8cyclictest0-21swapper/321:52:213
3264599199,8cyclictest0-21swapper/321:52:213
3264599193,11cyclictest0-21swapper/321:41:353
32645991913,4cyclictest28466irq/59-eth022:17:053
32645991913,4cyclictest0-21swapper/323:55:213
32645991913,4cyclictest0-21swapper/323:55:213
32645991913,4cyclictest0-21swapper/321:15:293
32645991912,4cyclictest0-21swapper/323:30:123
32645991911,4cyclictest0-21swapper/323:05:243
3264599191,11cyclictest0-21swapper/319:20:113
32645991910,6cyclictest0-21swapper/300:12:513
32645991910,5cyclictest0-21swapper/300:25:173
3264599190,14cyclictest28366irq/58-eth022:05:123
3263699197,9cyclictest21538-21uname20:05:182
3263699196,6cyclictest4154-21apt-get20:55:142
3263699193,8cyclictest11420-21kthreadcore19:20:192
3263699191,8cyclictest12968-21kthreadcore19:55:212
3263699191,8cyclictest12968-21kthreadcore19:55:202
3262899196,8cyclictest201rcuc/120:25:141
3262899196,8cyclictest201rcuc/120:25:131
3262899194,11cyclictest21-21ksoftirqd/119:45:121
3262899194,11cyclictest21-21ksoftirqd/119:45:111
3262899192,2cyclictest131rcu_preempt20:40:231
3262899191,3cyclictest0-21swapper/119:30:181
3262899191,2cyclictest131rcu_preempt21:05:161
3262899191,2cyclictest131rcu_preempt20:20:281
3262899191,12cyclictest0-21swapper/120:55:131
3262899190,3cyclictest131rcu_preempt20:45:201
3261999199,2cyclictest12-21ksoftirqd/019:20:190
3261999198,3cyclictest12-21ksoftirqd/021:05:020
3261999191,5cyclictest53146usb-storage19:50:030
32619991913,3cyclictest12-21ksoftirqd/023:30:210
32619991913,3cyclictest12-21ksoftirqd/019:40:200
32619991913,3cyclictest12-21ksoftirqd/019:40:190
3264599189,7cyclictest0-21swapper/322:34:183
3264599189,7cyclictest0-21swapper/322:34:173
3264599189,6cyclictest28366irq/58-eth023:10:233
3264599189,6cyclictest0-21swapper/300:05:193
3264599188,8cyclictest0-21swapper/323:26:493
3264599188,7cyclictest28366irq/58-eth022:00:133
3264599188,6cyclictest28366irq/58-eth022:35:133
3264599183,12cyclictest28366irq/58-eth022:45:123
3264599182,10cyclictest0-21swapper/321:05:133
3264599182,10cyclictest0-21swapper/320:15:143
3264599182,10cyclictest0-21swapper/319:40:143
3264599182,10cyclictest0-21swapper/319:40:133
32645991812,4cyclictest28366irq/58-eth023:15:123
32645991810,6cyclictest0-21swapper/323:02:293
32645991810,6cyclictest0-21swapper/322:55:043
32645991810,6cyclictest0-21swapper/322:43:063
32645991810,6cyclictest0-21swapper/300:15:403
3263699185,10cyclictest27-21ksoftirqd/219:10:022
3262899184,10cyclictest131rcu_preempt19:15:121
3262899181,10cyclictest14772-21ntp_states19:55:271
3262899181,10cyclictest14772-21ntp_states19:55:261
3262899180,3cyclictest131rcu_preempt19:40:141
3262899180,3cyclictest131rcu_preempt19:40:131
3261999189,7cyclictest2898-21apt-get20:20:130
3261999187,8cyclictest12-21ksoftirqd/021:05:120
32619991812,4cyclictest0-21swapper/023:20:120
32619991812,3cyclictest12-21ksoftirqd/023:00:140
32619991811,4cyclictest12-21ksoftirqd/022:40:230
3261999180,8cyclictest17228-21latency_hist20:35:020
3264599179,6cyclictest0-21swapper/322:53:493
3264599177,8cyclictest0-21swapper/322:20:403
3264599172,10cyclictest0-21swapper/320:30:143
3264599172,10cyclictest0-21swapper/320:10:133
3264599172,10cyclictest0-21swapper/300:00:153
3262899172,2cyclictest131rcu_preempt20:35:201
3261999177,2cyclictest12-21ksoftirqd/020:15:160
32619991711,3cyclictest12-21ksoftirqd/019:15:160
32619991710,4cyclictest12-21ksoftirqd/019:10:020
3264599168,5cyclictest28366irq/58-eth019:35:133
3264599164,9cyclictest28366irq/58-eth019:45:133
3264599164,9cyclictest28366irq/58-eth019:45:123
3264599161,13cyclictest28366irq/58-eth000:20:113
3264599161,10cyclictest0-21swapper/319:25:293
3264599151,11cyclictest0-21swapper/320:45:263
3264599149,3cyclictest0-21swapper/321:01:313
3262899141,10cyclictest32656-21latency_hist19:10:011
3264599138,3cyclictest33-21ksoftirqd/320:20:153
3264599138,3cyclictest33-21ksoftirqd/319:51:123
3264599138,3cyclictest0-21swapper/320:03:093
3264599138,3cyclictest0-21swapper/319:55:583
3264599138,3cyclictest0-21swapper/319:55:583
3264599131,7cyclictest0-21swapper/320:25:293
3264599131,7cyclictest0-21swapper/320:25:283
3264599128,2cyclictest33-21ksoftirqd/320:36:093
3264599127,3cyclictest0-21swapper/320:50:273
3264599127,3cyclictest0-21swapper/320:41:353
3264599127,3cyclictest0-21swapper/320:05:233
3264599121,7cyclictest0-21swapper/319:15:143
3264599118,2cyclictest33-21ksoftirqd/319:10:173
3264599112,6cyclictest0-21swapper/319:30:143
3264599111,6cyclictest0-21swapper/320:55:213
3264599101,6cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional