You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-28 - 00:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Aug 27, 2026 12:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2856099684,51cyclictest19860-21ssh10:25:121
2855199644,35cyclictest131rcu_preempt11:35:120
2855199644,35cyclictest131rcu_preempt11:35:110
2858299597,34cyclictest35966irq/58-eth012:30:133
2856099566,34cyclictest30394-21ssh09:55:111
2855199562,47cyclictest7708-21ssh11:30:130
28582995335,12cyclictest35966irq/58-eth011:05:133
2856099531,44cyclictest0-21swapper/109:15:131
2856099531,14cyclictest24351-21apt-get12:15:121
28551994913,27cyclictest12-21ksoftirqd/007:25:130
2856099471,7cyclictest0-21swapper/109:00:121
2856099461,41cyclictest0-21swapper/107:10:121
2856099461,36cyclictest28889-21ssh10:30:121
2856099461,27cyclictest29985-21find08:15:291
2855199454,30cyclictest23723-21ssh10:45:140
28582994420,15cyclictest35966irq/58-eth012:15:133
2858299434,24cyclictest35966irq/58-eth009:10:153
2858299434,24cyclictest35966irq/58-eth009:10:153
2856099431,3cyclictest0-21swapper/108:40:131
2856099431,3cyclictest0-21swapper/107:20:141
2855199431,3cyclictest131rcu_preempt10:55:120
28582994212,21cyclictest35966irq/58-eth011:20:143
2856099424,33cyclictest0-21swapper/112:25:101
2856099421,2cyclictest18376-21apt-get11:00:131
2856099421,2cyclictest10665-21apt-get07:25:121
28582994128,9cyclictest35966irq/58-eth010:45:143
2856099411,8cyclictest0-21swapper/107:55:131
2856099411,2cyclictest0-21swapper/110:35:141
2856099411,2cyclictest0-21swapper/110:35:131
2856099411,10cyclictest13375-21apt-get08:35:111
2856099411,10cyclictest13375-21apt-get08:35:101
2856099410,2cyclictest0-21swapper/111:45:131
2856099401,33cyclictest0-21swapper/110:50:131
2856099401,2cyclictest0-21swapper/109:40:121
2856099400,2cyclictest0-21swapper/112:00:141
2855199402,28cyclictest1015-21ssh12:20:140
2856099391,6cyclictest0-21swapper/108:25:131
2856099391,35cyclictest0-21swapper/110:10:131
2855199393,8cyclictest12-21ksoftirqd/008:05:130
28582993819,12cyclictest35966irq/58-eth011:15:133
2856099381,32cyclictest0-21swapper/111:35:141
2856099381,32cyclictest0-21swapper/111:35:131
2855199388,4cyclictest131rcu_preempt10:15:130
2855199388,4cyclictest131rcu_preempt10:15:130
2857099372,6cyclictest27346irq/40-dwc2_hso09:15:122
2856099371,32cyclictest4727-21ntpq12:20:271
2856099371,2cyclictest6354-21apt-get12:05:131
2856099370,2cyclictest0-21swapper/108:05:121
2858299362,21cyclictest35966irq/58-eth010:35:143
2858299362,21cyclictest35966irq/58-eth010:35:133
28570993619,11cyclictest131rcu_preempt12:15:112
2856099361,5cyclictest0-21swapper/108:10:101
2856099361,4cyclictest0-21swapper/110:00:141
2856099361,3cyclictest0-21swapper/107:40:121
2856099361,2cyclictest0-21swapper/111:10:131
2856099361,2cyclictest0-21swapper/110:20:141
2856099361,2cyclictest0-21swapper/109:05:121
2856099360,2cyclictest0-21swapper/111:15:121
2858299351,23cyclictest35966irq/58-eth012:10:103
2856099351,7cyclictest0-21swapper/110:55:111
2856099351,6cyclictest0-21swapper/109:20:121
2856099351,5cyclictest0-21swapper/108:45:131
2856099351,32cyclictest17388-21ssh09:30:141
2856099351,29cyclictest20640-21apt-get11:55:131
2858299346,16cyclictest35966irq/58-eth012:20:133
2858299345,20cyclictest35966irq/58-eth012:35:123
2856099342,26cyclictest11584-21ssh11:50:131
2856099341,3cyclictest0-21swapper/112:10:111
2856099341,30cyclictest31086-21/usr/sbin/munin11:25:131
2856099341,2cyclictest32695-21apt-get08:55:121
2856099340,2cyclictest0-21swapper/112:35:111
2855199342,22cyclictest14491-21diskmemload12:15:120
2856099332,29cyclictest25167-21ps09:50:291
2856099331,9cyclictest22665-21cut11:20:191
2856099331,8cyclictest27521-21diskstats11:05:141
2856099331,11cyclictest22608-21memory12:30:231
2856099331,10cyclictest18969-21awk07:30:261
2856099330,2cyclictest29974-21apt-get07:45:131
2856099330,2cyclictest0-21swapper/110:05:131
2856099330,11cyclictest12654-21cut09:45:191
2856099330,11cyclictest12654-21cut09:45:181
2858299324,20cyclictest35966irq/58-eth008:25:123
2856099323,26cyclictest16984-21kthreadcore09:10:231
2856099323,26cyclictest16984-21kthreadcore09:10:221
2856099322,25cyclictest1685-21ssh10:15:131
2856099322,25cyclictest1685-21ssh10:15:131
2856099321,2cyclictest0-21swapper/111:40:121
2856099321,27cyclictest23697-21apt-get10:45:121
2856099320,9cyclictest15087-21ssh11:33:281
2856099320,8cyclictest29211-21cut08:50:201
2856099320,4cyclictest26485-21apt-get09:35:111
28551993223,5cyclictest14491-21diskmemload11:25:140
2855199322,22cyclictest28178-21ssh12:35:120
28551993210,18cyclictest12-21ksoftirqd/008:35:110
28551993210,18cyclictest12-21ksoftirqd/008:35:100
2856099312,8cyclictest12040-21/usr/sbin/munin08:00:231
2856099311,8cyclictest11887-21/usr/sbin/munin08:30:221
2856099311,2cyclictest0-21swapper/109:25:131
2858299304,15cyclictest35966irq/58-eth011:35:133
2858299304,15cyclictest35966irq/58-eth011:35:123
2857099302,21cyclictest11500-21ssh11:50:092
2857099301,11cyclictest6365-21ssh12:05:112
28570993010,11cyclictest261rcuc/212:25:112
2856099301,3cyclictest0-21swapper/108:20:121
2856099300,7cyclictest14934-21sed10:40:141
2855199302,21cyclictest19774-21ssh10:25:090
2857099298,7cyclictest131rcu_preempt12:00:112
2856099291,7cyclictest5796-21ntpq07:50:251
2856099290,4cyclictest1092-21apt-get07:15:131
2856099290,2cyclictest0-21swapper/107:35:111
2855199292,18cyclictest0-21swapper/009:30:130
2857099285,14cyclictest27346irq/40-dwc2_hso08:10:092
2857099283,21cyclictest30400-21ssh09:55:132
2857099282,23cyclictest16828-21ssh11:35:132
2857099282,23cyclictest16828-21ssh11:35:132
2857099281,23cyclictest15382-21ssh12:10:122
2857099281,18cyclictest27346irq/40-dwc2_hso08:50:112
28570992811,11cyclictest14491-21diskmemload09:10:152
28570992811,11cyclictest14491-21diskmemload09:10:142
2855199282,21cyclictest16221-21ssh09:29:090
2855199281,20cyclictest21390-21ssh09:50:140
2858299273,18cyclictest0-21swapper/309:33:083
2857099273,15cyclictest27346irq/40-dwc2_hso07:40:122
2857099271,14cyclictest7725-21rm11:30:132
2856099271,6cyclictest28559-21latency_hist07:10:011
2855199272,19cyclictest29784-21ssh12:00:140
2855199271,19cyclictest12-21ksoftirqd/012:05:120
2858299261,13cyclictest35966irq/58-eth011:40:133
2858299260,15cyclictest35966irq/58-eth009:50:143
2857099269,14cyclictest27-21ksoftirqd/210:35:122
2857099269,14cyclictest27-21ksoftirqd/210:35:112
2857099267,4cyclictest27-21ksoftirqd/209:22:092
2857099263,13cyclictest23495-21ssh10:25:252
2857099262,21cyclictest26430-21sshd08:15:152
2857099262,10cyclictest0-21swapper/207:15:132
28570992615,4cyclictest27-21ksoftirqd/212:30:122
28570992614,9cyclictest0-21swapper/210:20:172
28570992613,10cyclictest27-21ksoftirqd/211:40:172
2857099261,19cyclictest14683-21ssh10:40:122
2857099261,18cyclictest17287-21ssh09:30:092
2855199261,6cyclictest131rcu_preempt08:50:120
2855199261,21cyclictest23900-21ssh09:15:140
2857099253,19cyclictest261rcuc/211:45:132
2857099252,16cyclictest27346irq/40-dwc2_hso08:30:122
2857099251,9cyclictest20668-21ssh11:55:142
2857099251,18cyclictest14491-21diskmemload11:10:112
2855199258,12cyclictest12-21ksoftirqd/009:48:090
2855199258,12cyclictest12-21ksoftirqd/009:48:090
2855199255,13cyclictest0-21swapper/008:55:130
2855199252,13cyclictest11573-21ssh11:50:140
2858299243,13cyclictest0-21swapper/310:25:093
28582992416,5cyclictest35966irq/58-eth012:25:133
28582992414,8cyclictest0-21swapper/310:50:303
2857099245,4cyclictest27-21ksoftirqd/209:42:522
2857099244,14cyclictest30013-21missed_timers09:35:212
2857099244,11cyclictest27346irq/40-dwc2_hso09:00:102
2857099243,9cyclictest6991-21ssh10:00:122
2857099243,14cyclictest27346irq/40-dwc2_hso10:05:142
2857099242,15cyclictest27346irq/40-dwc2_hso08:05:122
2857099242,15cyclictest0-21swapper/209:50:032
2857099241,17cyclictest27346irq/40-dwc2_hso08:00:122
2857099241,10cyclictest261rcuc/212:20:132
28570992410,10cyclictest15235-21sh07:30:012
2855199241,15cyclictest19151-21rm12:30:140
2857099239,6cyclictest0-21swapper/211:20:522
2857099239,11cyclictest25052-21apt-get10:10:132
2857099236,14cyclictest19987-21cron07:35:002
2857099236,11cyclictest27-21ksoftirqd/211:17:562
28570992313,7cyclictest27-21ksoftirqd/211:04:592
28570992312,8cyclictest27-21ksoftirqd/207:50:202
28570992311,9cyclictest743-21ssh11:06:392
2857099231,16cyclictest27-21ksoftirqd/207:10:122
2855199238,7cyclictest131rcu_preempt10:40:090
2858299225,14cyclictest35966irq/58-eth011:50:133
2858299222,13cyclictest0-21swapper/310:20:133
28582992214,5cyclictest35966irq/58-eth010:30:143
2858299221,14cyclictest35966irq/58-eth010:00:143
2857099229,10cyclictest4446-21munin-plugin-st12:40:012
2857099228,11cyclictest23927-21fschecks_time10:45:172
2857099228,11cyclictest0-21swapper/207:20:252
2857099224,16cyclictest27346irq/40-dwc2_hso10:30:122
2857099223,13cyclictest0-21swapper/208:45:132
2857099222,18cyclictest0-21swapper/209:47:092
2857099222,18cyclictest0-21swapper/209:47:082
2857099222,11cyclictest12860-21memory10:55:232
28570992214,3cyclictest27-21ksoftirqd/211:25:142
28570992213,6cyclictest27-21ksoftirqd/209:27:092
2857099220,12cyclictest2079-21grep10:15:182
2857099220,12cyclictest2079-21grep10:15:182
2855199229,9cyclictest28542-21cyclictest10:38:100
2855199229,9cyclictest28542-21cyclictest10:38:100
2855199229,9cyclictest28542-21cyclictest09:10:150
2855199229,9cyclictest28542-21cyclictest09:10:150
2855199226,3cyclictest12-21ksoftirqd/011:45:140
2855199223,17cyclictest12-21ksoftirqd/011:20:130
28551992211,7cyclictest3312-1kworker/u9:110:10:150
2855199221,17cyclictest10806-21ssh10:20:140
2858299211,2cyclictest0-21swapper/308:05:113
2858299211,15cyclictest0-21swapper/307:10:123
2858299211,12cyclictest35966irq/58-eth009:20:113
28582992110,9cyclictest0-21swapper/310:05:273
2857099219,9cyclictest27-21ksoftirqd/208:20:182
2857099219,4cyclictest27-21ksoftirqd/207:55:202
2857099218,5cyclictest27-21ksoftirqd/207:45:132
2857099218,10cyclictest13573-21fschecks_time08:35:142
2857099218,10cyclictest13573-21fschecks_time08:35:142
2857099212,16cyclictest0-21swapper/208:25:042
28570992110,9cyclictest0-21swapper/210:51:002
2855199212,4cyclictest131rcu_preempt07:15:140
2855199212,15cyclictest32749-21ssh10:50:140
2855199212,14cyclictest13016-21ssh11:15:150
2855199211,3cyclictest131rcu_preempt11:07:300
2855199211,16cyclictest12624-21ls11:15:000
2855199211,14cyclictest1983-21timerwakeupswit09:55:290
2855199210,16cyclictest30376-21switchtime09:35:270
2858299209,8cyclictest0-21swapper/312:02:523
2858299207,10cyclictest0-21swapper/309:15:313
2858299203,11cyclictest0-21swapper/309:25:093
28582992014,4cyclictest0-21swapper/310:15:173
28582992014,4cyclictest0-21swapper/310:15:163
28582992011,6cyclictest0-21swapper/311:10:213
2857099207,3cyclictest131rcu_preempt07:35:282
2857099207,10cyclictest0-21swapper/209:05:142
2857099206,11cyclictest18665-21irqcore08:40:172
2857099202,4cyclictest0-21swapper/208:55:132
2855199204,8cyclictest131rcu_preempt12:10:180
2855199202,3cyclictest1636-21ssh10:30:490
28551992015,3cyclictest0-21swapper/011:40:290
2855199201,2cyclictest131rcu_preempt11:55:130
2855199201,17cyclictest12265-21ls08:30:270
2855199200,13cyclictest8771-21ls09:00:270
2858299199,6cyclictest0-21swapper/310:42:233
2858299198,6cyclictest0-21swapper/309:39:023
2858299193,11cyclictest35966irq/58-eth009:00:113
2858299193,11cyclictest0-21swapper/309:40:373
2858299192,11cyclictest0-21swapper/311:45:093
28582991910,6cyclictest35966irq/58-eth012:05:133
2857099191,9cyclictest28673-21latency_hist07:10:012
2855199198,8cyclictest26511-21df08:15:160
2855199191,9cyclictest131rcu_preempt07:50:180
2855199191,3cyclictest131rcu_preempt07:45:130
2855199191,2cyclictest131rcu_preempt08:05:010
28551991912,5cyclictest0-21swapper/007:35:250
2855199191,16cyclictest32334-21ls07:10:270
2855199191,14cyclictest7117-21ssh10:00:150
28551991911,4cyclictest6728-21missed_timers09:40:240
2855199190,16cyclictest7859-21ls09:25:010
2858299189,6cyclictest0-21swapper/311:56:373
2858299183,10cyclictest0-21swapper/311:25:093
2858299182,10cyclictest35966irq/58-eth008:40:133
2858299182,10cyclictest0-21swapper/309:45:093
2858299182,10cyclictest0-21swapper/309:45:093
2858299182,10cyclictest0-21swapper/308:35:123
2858299182,10cyclictest0-21swapper/308:35:113
2858299182,10cyclictest0-21swapper/307:40:133
2858299182,10cyclictest0-21swapper/307:25:143
28582991812,4cyclictest0-21swapper/309:55:133
28582991811,5cyclictest0-21swapper/310:10:313
2858299181,13cyclictest0-21swapper/308:15:163
2858299181,12cyclictest0-21swapper/308:50:233
28582991810,6cyclictest0-21swapper/310:57:563
2855199183,4cyclictest131rcu_preempt10:05:230
2855199183,3cyclictest0-21swapper/012:25:220
2855199182,14cyclictest5525-21kthreadcore08:25:210
2855199181,2cyclictest131rcu_preempt11:01:510
2855199181,13cyclictest24792-21apt-get07:40:050
2855199181,13cyclictest23070-21apt-get08:45:120
28551991811,3cyclictest12-21ksoftirqd/007:20:120
2855199180,3cyclictest131rcu_preempt08:20:180
2858299178,6cyclictest0-21swapper/311:00:203
2858299172,8cyclictest0-21swapper/308:45:123
2858299172,10cyclictest0-21swapper/311:30:093
2855199179,5cyclictest685-21in:imuxsock09:10:010
2855199175,3cyclictest131rcu_preempt08:40:110
2855199172,12cyclictest6906-21apt-get07:55:050
2855199171,5cyclictest0-21swapper/007:30:220
2858299162,9cyclictest0-21swapper/308:55:143
2858299162,9cyclictest0-21swapper/307:55:053
2858299162,11cyclictest0-21swapper/307:15:143
2858299162,10cyclictest0-21swapper/307:45:133
2858299162,10cyclictest0-21swapper/307:20:133
2855199161,7cyclictest24885-21sort08:10:200
2858299156,6cyclictest0-21swapper/307:50:183
2858299151,8cyclictest0-21swapper/308:25:023
2858299151,11cyclictest0-21swapper/308:00:183
2858299149,3cyclictest0-21swapper/308:30:093
2858299141,7cyclictest0-21swapper/307:35:143
2855199140,11cyclictest28542-21cyclictest07:10:000
2858299132,8cyclictest0-21swapper/307:30:163
2858299131,10cyclictest0-21swapper/308:10:273
2858299122,6cyclictest0-21swapper/309:05:143
2858299111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional