You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-10-10 - 00:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Fri Oct 09, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
415099688,47cyclictest756-21ssh11:40:121
415099573,9cyclictest131rcu_preempt09:50:121
415099554,9cyclictest131rcu_preempt08:25:131
415099533,41cyclictest24588-21ssh10:05:141
415999521,15cyclictest22577-21rm09:10:132
4150995218,22cyclictest21-21ksoftirqd/107:55:111
415999501,9cyclictest0-21swapper/209:15:152
415099502,37cyclictest22285-21ssh12:10:111
415099443,31cyclictest22579-21ssh09:10:131
415099442,6cyclictest131rcu_preempt09:40:131
417099432,27cyclictest33366irq/58-eth011:30:133
415999431,5cyclictest2766-21ssh09:35:102
415999422,18cyclictest15543-21ssh10:00:122
415999421,3cyclictest0-21swapper/211:20:132
415099421,32cyclictest0-21swapper/111:15:121
415999403,33cyclictest29915-21ssh09:50:112
417099394,22cyclictest33366irq/58-eth007:40:123
415999391,12cyclictest27602-21latency_hist11:55:012
415999390,9cyclictest10972-21cut10:50:192
414299393,21cyclictest22176-21diskmemload09:15:140
414299391,32cyclictest19109-21ssh10:20:120
415999382,28cyclictest637-21ssh11:40:052
415999381,9cyclictest3568-21kthreadcore07:40:212
415999381,8cyclictest19341-21ssh12:25:212
415999381,8cyclictest0-21swapper/212:05:122
415999381,26cyclictest11880-21ssh09:40:142
415999381,10cyclictest19836-21kthreadcore07:25:192
415999380,10cyclictest9778-21latency_hist10:15:022
415999372,3cyclictest0-21swapper/209:45:132
415999371,9cyclictest14984-21missed_timers08:25:232
415999371,8cyclictest15547-21kthreadcore07:20:202
415999371,10cyclictest6216-21ssh11:25:132
415999371,10cyclictest6216-21ssh11:25:132
415999370,8cyclictest23939-21irqcore07:30:172
415999370,6cyclictest15084-21latency_hist07:55:012
415999369,3cyclictest7114-21uptime08:50:272
415999361,9cyclictest6283-21threads12:35:272
415999361,9cyclictest31758-21wc10:45:162
415999361,9cyclictest24602-21grep11:35:182
415999361,7cyclictest17738-21irqrtprio10:55:182
415999361,7cyclictest17738-21irqrtprio10:55:172
415999361,14cyclictest31489-21kthreadcore07:35:222
415999361,13cyclictest3039-21latency_hist08:50:022
415999361,10cyclictest13060-21idleruntime09:00:162
415999361,10cyclictest13060-21idleruntime09:00:152
415999360,13cyclictest11321-21cut12:20:232
414299362,21cyclictest12-21ksoftirqd/007:45:140
417099353,22cyclictest33366irq/58-eth010:35:123
417099351,20cyclictest33366irq/58-eth009:15:143
415999359,5cyclictest26479-21timerandwakeup10:40:292
415999351,8cyclictest10200-21ntp_states09:55:242
415999351,8cyclictest10200-21ntp_states09:55:232
415999351,12cyclictest21463-21wc08:35:172
415999351,11cyclictest19869-21latency_hist08:00:012
415999351,10cyclictest22337-21cpu12:10:112
415999350,11cyclictest15227-21sh11:30:112
4150993511,18cyclictest21-21ksoftirqd/111:20:121
415999349,22cyclictest2293-21idleruntime-cro11:05:012
415999346,17cyclictest0-21swapper/209:30:122
415999341,9cyclictest839-21kthreadcore12:15:192
415999341,8cyclictest8432-21irqcore08:55:182
415999341,13cyclictest13898-21timerandwakeup11:45:282
415999341,13cyclictest13898-21timerandwakeup11:45:282
415999341,10cyclictest2306-21sed08:15:172
415999340,10cyclictest28537-21kthreadcore12:30:192
417099332,27cyclictest0-21swapper/307:35:133
417099330,2cyclictest0-21swapper/308:05:113
415999337,3cyclictest0-21swapper/210:15:272
415999336,5cyclictest20195-21ls08:00:142
415999335,17cyclictest22346irq/40-dwc2_hso11:10:122
415999332,9cyclictest9238-21fschecks_time07:15:152
415999331,9cyclictest0-21swapper/209:05:232
415999331,9cyclictest0-21swapper/209:05:232
415999331,8cyclictest6780-21tail11:05:302
415999331,8cyclictest6780-21tail11:05:292
415999331,8cyclictest17776-21kthreadcore08:30:202
415999331,10cyclictest22117-21sendmail-msp10:40:012
415999330,10cyclictest28255-21irqcore11:55:172
414299332,27cyclictest18897-21ssh11:50:110
417099322,27cyclictest0-21swapper/311:10:123
417099322,25cyclictest0-21swapper/308:25:123
4170993217,10cyclictest33366irq/58-eth011:20:133
417099321,25cyclictest0-21swapper/309:25:123
415999325,5cyclictest29607-21cpuspeed08:10:122
415999321,8cyclictest4548-21/usr/sbin/munin10:30:162
415999321,8cyclictest4548-21/usr/sbin/munin10:30:152
415999321,8cyclictest30816-21latency_hist08:45:012
4159993210,18cyclictest4133-21cyclictest10:29:442
415999320,9cyclictest10411-21find08:20:242
414299321,22cyclictest990366ssh11:45:130
414299321,22cyclictest990366ssh11:45:120
417099312,21cyclictest0-21swapper/312:30:123
417099311,21cyclictest0-21swapper/308:00:133
415999311,8cyclictest24537-21/usr/sbin/munin10:05:312
415099311,17cyclictest21-21ksoftirqd/111:10:111
414299312,21cyclictest2282-21sh12:35:090
417099302,23cyclictest0-21swapper/307:55:123
417099302,20cyclictest0-21swapper/312:10:113
417099301,23cyclictest0-21swapper/311:50:123
417099301,22cyclictest0-21swapper/312:00:123
417099301,22cyclictest0-21swapper/311:40:103
417099301,22cyclictest0-21swapper/309:35:113
417099301,22cyclictest0-21swapper/307:20:123
4170993011,12cyclictest33366irq/58-eth011:15:113
415999301,6cyclictest16785-21cron09:25:022
415999300,6cyclictest26781-21rm10:23:542
415999300,5cyclictest28955-21ssh11:19:382
415999300,4cyclictest4426-21fschecks_count07:10:152
415999300,13cyclictest0-21swapper/212:00:072
415099303,23cyclictest7897-21ssh12:20:131
415099301,26cyclictest0-21swapper/109:00:131
415099301,26cyclictest0-21swapper/109:00:131
415099301,18cyclictest20913-21ssh09:45:131
417099291,2cyclictest0-21swapper/308:30:133
417099291,2cyclictest0-21swapper/308:10:123
417099291,25cyclictest0-21swapper/309:10:133
417099291,23cyclictest0-21swapper/312:05:123
417099290,2cyclictest0-21swapper/307:45:133
415999291,9cyclictest18853-21ssh09:25:192
415099292,19cyclictest4290-21sh12:00:031
4150992912,9cyclictest131rcu_preempt10:00:131
415099291,20cyclictest0-21swapper/111:35:121
414299292,21cyclictest25805-21ssh12:30:110
414299292,19cyclictest7749-21ssh12:20:050
414299291,20cyclictest22176-21diskmemload11:05:150
414299291,20cyclictest22176-21diskmemload11:05:150
4142992911,13cyclictest1351-21sh09:33:450
417099289,10cyclictest33366irq/58-eth012:15:113
417099286,13cyclictest33366irq/58-eth011:55:113
417099281,2cyclictest0-21swapper/310:10:123
417099281,2cyclictest0-21swapper/307:10:103
417099281,23cyclictest0-21swapper/310:15:123
417099281,22cyclictest0-21swapper/309:50:123
417099281,22cyclictest0-21swapper/309:40:123
417099281,22cyclictest0-21swapper/308:15:103
417099280,2cyclictest0-21swapper/311:45:113
417099280,2cyclictest0-21swapper/311:45:113
417099280,2cyclictest0-21swapper/309:00:113
417099280,2cyclictest0-21swapper/309:00:103
415099286,13cyclictest4571-21apt-get10:30:121
415099286,13cyclictest4571-21apt-get10:30:111
414299284,18cyclictest2785-21ssh09:35:120
417099273,16cyclictest0-21swapper/312:35:093
417099271,2cyclictest0-21swapper/312:20:143
417099271,21cyclictest0-21swapper/307:50:143
417099271,20cyclictest0-21swapper/307:30:143
417099270,2cyclictest0-21swapper/310:55:033
417099270,2cyclictest0-21swapper/310:55:023
415999271,3cyclictest0-21swapper/207:45:302
415999270,9cyclictest28047-21latency08:05:232
415099272,21cyclictest8122-21ssh10:50:101
4142992714,8cyclictest0-21swapper/011:35:120
417099262,20cyclictest0-21swapper/310:00:123
417099262,18cyclictest0-21swapper/308:45:113
417099261,2cyclictest0-21swapper/309:45:123
417099261,22cyclictest0-21swapper/311:35:153
415099262,17cyclictest21-21ksoftirqd/110:45:121
414299268,8cyclictest12-21ksoftirqd/011:13:190
414299264,11cyclictest12-21ksoftirqd/010:38:590
417099251,2cyclictest0-21swapper/312:25:293
417099251,2cyclictest0-21swapper/310:25:143
417099251,22cyclictest0-21swapper/311:00:103
417099250,2cyclictest0-21swapper/310:50:103
417099250,2cyclictest0-21swapper/310:40:133
415099252,19cyclictest25777-21apt-get12:30:111
415099251,21cyclictest8184-21ssh09:20:111
417099241,2cyclictest0-21swapper/310:30:283
417099241,2cyclictest0-21swapper/310:30:283
417099241,2cyclictest0-21swapper/309:30:103
417099241,2cyclictest0-21swapper/308:50:113
4170992412,7cyclictest33366irq/58-eth011:25:143
4170992412,7cyclictest33366irq/58-eth011:25:143
417099241,20cyclictest0-21swapper/310:49:593
415099242,18cyclictest10142-21ssh10:15:131
414299248,9cyclictest12-21ksoftirqd/010:15:350
414299247,5cyclictest12-21ksoftirqd/012:12:170
414299246,8cyclictest12-21ksoftirqd/011:25:050
414299246,8cyclictest12-21ksoftirqd/011:25:040
4142992418,3cyclictest0-21swapper/010:30:120
4142992418,3cyclictest0-21swapper/010:30:120
414299241,18cyclictest13356-21ssh12:05:120
417099232,18cyclictest0-21swapper/311:05:153
417099232,18cyclictest0-21swapper/311:05:143
417099231,2cyclictest0-21swapper/308:55:113
4170992312,8cyclictest0-21swapper/309:55:123
4170992312,8cyclictest0-21swapper/309:55:123
417099231,19cyclictest0-21swapper/310:05:183
417099231,19cyclictest0-21swapper/307:15:233
417099230,2cyclictest0-21swapper/307:25:123
415099232,17cyclictest14763-21apt-get11:30:031
415099232,14cyclictest9434-21apt-get11:45:051
415099232,14cyclictest9434-21apt-get11:45:051
415099232,14cyclictest27697-21ssh11:55:011
415099231,10cyclictest13475-21diskstats12:05:141
414299237,4cyclictest12-21ksoftirqd/012:03:580
414299236,12cyclictest12-21ksoftirqd/010:47:450
414299235,9cyclictest12-21ksoftirqd/011:44:160
414299232,11cyclictest12-21ksoftirqd/010:00:040
417099221,2cyclictest0-21swapper/309:20:123
417099221,2cyclictest0-21swapper/308:40:123
417099221,2cyclictest0-21swapper/308:20:113
417099221,18cyclictest0-21swapper/309:05:153
417099221,18cyclictest0-21swapper/309:05:143
417099220,2cyclictest0-21swapper/310:20:133
415099228,4cyclictest131rcu_preempt11:55:011
415099222,15cyclictest26159-21ssh09:30:101
4150992212,7cyclictest13648-21cron07:20:011
415099221,14cyclictest2918-21cut11:05:171
415099221,14cyclictest2918-21cut11:05:171
415099221,12cyclictest5682-21kworker/u8:108:20:111
414299229,7cyclictest0-21swapper/012:15:450
414299229,4cyclictest0-21swapper/009:25:550
414299228,7cyclictest12-21ksoftirqd/011:02:070
414299228,5cyclictest12-21ksoftirqd/010:13:460
414299228,4cyclictest16666systemd-network09:13:290
414299228,2cyclictest131rcu_preempt07:55:230
414299225,8cyclictest12-21ksoftirqd/011:24:490
414299225,7cyclictest12-21ksoftirqd/011:00:010
414299225,7cyclictest12-21ksoftirqd/011:00:010
414299225,10cyclictest12-21ksoftirqd/009:44:460
414299224,7cyclictest12-21ksoftirqd/009:55:550
414299224,7cyclictest12-21ksoftirqd/009:55:550
414299224,4cyclictest131rcu_preempt10:53:020
414299221,4cyclictest12-21ksoftirqd/010:09:400
4142992214,5cyclictest0-21swapper/007:30:020
417099211,2cyclictest0-21swapper/308:35:123
415099213,12cyclictest2352-21ssh12:35:121
414299218,7cyclictest12-21ksoftirqd/008:15:280
414299216,6cyclictest12-21ksoftirqd/008:35:170
414299214,9cyclictest12-21ksoftirqd/011:59:440
414299212,4cyclictest12-21ksoftirqd/012:26:050
4142992110,3cyclictest0-21swapper/010:25:280
415999201,10cyclictest4204-21grep07:10:022
415099207,9cyclictest19088-21apt-get10:20:111
415099202,12cyclictest22176-21diskmemload12:15:121
415099201,15cyclictest30848-21apt-get08:45:111
415099201,14cyclictest26173-21ssh10:40:261
414299209,3cyclictest12-21ksoftirqd/010:40:340
414299207,4cyclictest12-21ksoftirqd/009:50:130
414299207,4cyclictest12-21ksoftirqd/008:05:230
414299206,8cyclictest12-21ksoftirqd/011:31:560
414299206,11cyclictest12-21ksoftirqd/011:15:140
414299204,8cyclictest12-21ksoftirqd/007:10:120
414299203,12cyclictest12-21ksoftirqd/008:10:130
4142992014,4cyclictest0-21swapper/007:35:230
4142992011,6cyclictest12577-21cron09:00:010
4142992011,3cyclictest0-21swapper/007:15:300
4142992010,4cyclictest12-21ksoftirqd/009:48:100
415099197,9cyclictest6887-21tail09:35:271
415099195,11cyclictest31828-21kthreadcore08:10:201
415099192,15cyclictest23868-21rm09:27:451
415099192,15cyclictest16762-21ssh12:25:071
415099192,14cyclictest22124-21basename10:40:001
415099192,13cyclictest6206-21ssh11:25:131
415099192,13cyclictest6206-21ssh11:25:121
415099191,7cyclictest10334-21kthreadcore08:55:211
4150991911,6cyclictest29317-21sshd08:09:461
415099191,12cyclictest17819-21cut09:05:171
415099191,12cyclictest17819-21cut09:05:171
415099191,11cyclictest5397-21ntp_states08:15:231
414299197,5cyclictest12-21ksoftirqd/008:25:240
414299197,3cyclictest12-21ksoftirqd/009:24:520
4142991913,4cyclictest0-21swapper/009:05:150
4142991913,4cyclictest0-21swapper/009:05:140
4142991913,4cyclictest0-21swapper/008:30:260
4142991913,4cyclictest0-21swapper/007:40:230
414299191,11cyclictest14955-21kthreadcore07:20:180
4142991910,7cyclictest1-21systemd07:54:100
417099181,14cyclictest0-21swapper/307:10:023
415099189,7cyclictest10106-21sshd07:48:441
415099189,6cyclictest4914-21ssh10:10:261
415099186,9cyclictest21420-21fschecks_time08:35:171
415099186,7cyclictest211ksoftirqd/107:20:131
415099186,6cyclictest14928-21cron07:55:011
415099183,12cyclictest20352-21irqcore08:00:171
415099182,9cyclictest28329-21/usr/sbin/munin07:35:281
415099182,12cyclictest30989-21users11:00:301
415099181,9cyclictest721-21/usr/sbin/munin07:40:291
415099181,12cyclictest481-21users10:25:321
415099181,11cyclictest2775-21ssh09:15:251
4150991810,5cyclictest0-21swapper/110:55:291
4150991810,5cyclictest0-21swapper/110:55:291
414299189,7cyclictest12918-21df_inode09:00:140
414299189,7cyclictest12918-21df_inode09:00:140
414299185,3cyclictest131rcu_preempt08:20:140
415099173,9cyclictest0-21swapper/108:30:151
415099172,11cyclictest6848-21if_eth009:55:151
415099172,11cyclictest6848-21if_eth009:55:141
415099171,8cyclictest27401-21kthreadcore08:40:201
414299172,11cyclictest0-21swapper/008:40:140
4142991711,4cyclictest31486-21kernelversion08:45:170
4142991711,4cyclictest0-21swapper/008:00:260
414299171,12cyclictest6868-21wc08:50:270
415099167,7cyclictest5596-21kthreadcore08:50:201
415099163,10cyclictest6637-21kthreadcore07:10:201
415099161,12cyclictest19338-21sed07:25:191
415099161,11cyclictest23712-21fschecks_count07:30:161
414299162,9cyclictest0-21swapper/007:30:130
415099141,10cyclictest4205-21wc07:10:011
414299141,10cyclictest4242-21cat07:10:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional