You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-16 - 21:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Aug 16, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2277099753,51cyclictest29135-21ssh12:20:120
2278199735,46cyclictest23085-21ssh11:05:141
2278999583,26cyclictest24291-21ssh10:30:122
2277099582,37cyclictest25623-21ssh09:55:130
2278199502,39cyclictest2278-21ssh10:00:141
22770994914,27cyclictest12-21ksoftirqd/012:15:140
22801994813,19cyclictest35966irq/58-eth011:10:123
2278199484,36cyclictest26871-21ssh11:25:131
22789994726,16cyclictest14780-21apt-get12:30:122
22781994715,20cyclictest21-21ksoftirqd/109:15:141
2278199452,3cyclictest131rcu_preempt10:35:131
2278199452,33cyclictest8898-21sh09:10:151
2278199452,33cyclictest8898-21sh09:10:141
22770994527,12cyclictest12-21ksoftirqd/010:29:070
22789994410,25cyclictest16572-21ssh09:50:142
22789994326,11cyclictest24155-21apt-get07:45:132
22770994323,11cyclictest12699-21sh09:30:150
2278999421,26cyclictest23069-21sh11:05:132
22770994226,7cyclictest0-21swapper/011:15:160
2277099422,27cyclictest26946-21ssh09:20:120
2278999413,29cyclictest0-21swapper/207:20:132
2278199414,33cyclictest14800-21ssh12:30:131
22801994026,10cyclictest35966irq/58-eth010:50:143
22801994010,15cyclictest35966irq/58-eth012:00:153
22789994027,2cyclictest30703-21apt-get11:45:162
2278999402,21cyclictest14932-21ntp_192.168.11410:05:272
22789994019,14cyclictest14032-21apt-get11:00:152
2277099408,19cyclictest12-21ksoftirqd/008:55:150
22789993918,16cyclictest19219-21apt-get10:45:162
2277099392,30cyclictest24264-21ssh10:30:100
2280199381,26cyclictest35966irq/58-eth009:45:143
22789993829,4cyclictest27716-21apt-get07:15:152
2278999382,28cyclictest12422-21/usr/sbin/munin11:15:262
2278999382,18cyclictest18935-21ssh10:25:272
2278999381,26cyclictest21752-21ssh09:35:122
22770993822,8cyclictest0-21swapper/009:40:130
2277099374,25cyclictest21666-21ssh11:40:110
22789993624,8cyclictest27-21ksoftirqd/212:07:292
22789993624,8cyclictest27-21ksoftirqd/212:07:292
2278999361,28cyclictest25470-21ssh12:00:142
2278999350,15cyclictest26535-21cut12:35:212
2278199354,22cyclictest16353-21ssh11:55:141
2277099352,23cyclictest12-21ksoftirqd/008:15:150
2277099351,27cyclictest19222-21ssh10:45:150
22789993416,15cyclictest0-21swapper/207:40:122
22789993411,19cyclictest0-21swapper/209:40:132
2278999337,21cyclictest4720-21sh10:35:272
2278999336,16cyclictest0-21swapper/207:25:152
2278999335,9cyclictest1746-21ssh12:20:412
2278999333,23cyclictest5763-21ssh12:25:122
22789993323,6cyclictest6227-21apt-get08:00:142
2278999332,25cyclictest4971-21ssh10:19:082
22789993317,10cyclictest0-21swapper/209:10:122
22789993317,10cyclictest0-21swapper/209:10:122
2278199331,26cyclictest8484-21diskmemload12:25:131
2277099333,18cyclictest12-21ksoftirqd/007:30:120
22770993316,10cyclictest17786-21ssh11:20:130
2277099331,29cyclictest12560-21ssh11:35:120
22789993210,18cyclictest27346irq/40-dwc2_hso09:00:132
2278999314,16cyclictest27346irq/40-dwc2_hso09:15:142
2278999312,20cyclictest27346irq/40-dwc2_hso07:50:132
2278999312,14cyclictest4269-21cut09:25:192
2278999312,14cyclictest25899-21kthreadcore07:10:232
2278199312,20cyclictest32091-21ssh11:10:141
2277099312,22cyclictest2678-21ssh10:16:260
2280199304,11cyclictest35966irq/58-eth010:40:163
2278999308,17cyclictest0-21swapper/208:20:162
2278999304,12cyclictest15979-21ssh12:10:432
2278999304,12cyclictest15979-21ssh12:10:422
22789993022,3cyclictest10106-21apt-get10:40:142
22789993020,7cyclictest15872-21apt-get08:10:132
22789993014,11cyclictest28260-21ssh10:50:142
2278199309,8cyclictest22759-21cyclictest08:00:131
2278199305,12cyclictest10137-21ssh10:40:151
2278199302,22cyclictest2170-21processes09:40:271
22781993013,10cyclictest21-21ksoftirqd/110:25:121
2277099303,20cyclictest12-21ksoftirqd/007:25:160
22770993015,10cyclictest4250-1kworker/u9:012:10:130
22770993015,10cyclictest4250-1kworker/u9:012:10:130
2277099301,25cyclictest7296-21ssh11:50:140
2280199293,17cyclictest35966irq/58-eth008:05:133
2278999296,9cyclictest0-21swapper/212:15:032
2278999295,13cyclictest0-21swapper/211:42:182
2278999294,7cyclictest0-21swapper/209:33:312
2278999294,18cyclictest27346irq/40-dwc2_hso11:35:132
2278999293,8cyclictest10909-21ntp_192.168.11408:35:252
2278999293,20cyclictest0-21swapper/210:11:262
2278999293,11cyclictest31534-21ssh09:56:482
22789992915,11cyclictest0-21swapper/207:30:132
2278999291,12cyclictest22311-21ssh11:56:482
2278199295,3cyclictest131rcu_preempt10:05:151
2278199294,18cyclictest7582-21ssh09:45:151
2280199284,16cyclictest35966irq/58-eth010:20:143
2280199283,20cyclictest35966irq/58-eth008:35:133
2280199283,18cyclictest0-21swapper/311:54:213
22801992812,12cyclictest35966irq/58-eth011:15:163
2278999283,17cyclictest27346irq/40-dwc2_hso08:15:152
2278999283,13cyclictest3077-21irqcore08:30:182
2278999282,11cyclictest0-21swapper/210:23:262
2278999281,14cyclictest5619-21cat07:55:302
2278199283,21cyclictest25636-21sh09:55:141
2278199282,19cyclictest29526-21ssh10:15:161
2280199273,16cyclictest0-21swapper/310:30:273
2280199273,15cyclictest0-21swapper/310:25:263
2280199273,13cyclictest35966irq/58-eth012:10:133
2280199273,13cyclictest35966irq/58-eth012:10:123
2280199272,18cyclictest35966irq/58-eth012:15:153
2280199272,15cyclictest35966irq/58-eth011:25:133
22801992710,11cyclictest0-21swapper/310:39:093
2278999273,18cyclictest0-21swapper/211:31:272
2278999273,18cyclictest0-21swapper/211:31:272
2278999273,11cyclictest4432-21ps09:05:192
22789992720,4cyclictest12327-21apt-get08:40:142
22789992720,4cyclictest12327-21apt-get08:40:142
22789992713,6cyclictest7567-21ssh09:45:152
2278999271,11cyclictest11221-21ssh10:57:182
2278999271,11cyclictest11221-21ssh10:57:172
2278199272,19cyclictest9143-21ssh11:31:271
2278199272,19cyclictest9143-21ssh11:31:271
2277099272,19cyclictest29577-21ssh09:39:010
2277099272,19cyclictest13947-21rm12:29:280
2280199263,16cyclictest0-21swapper/312:29:293
2280199263,15cyclictest0-21swapper/311:31:283
2280199263,15cyclictest0-21swapper/311:31:273
2280199263,15cyclictest0-21swapper/310:19:073
2280199262,14cyclictest0-21swapper/309:40:123
22801992613,9cyclictest35966irq/58-eth012:35:133
22801992613,9cyclictest35966irq/58-eth009:35:243
22801992610,10cyclictest0-21swapper/309:59:033
2278999263,8cyclictest32059-21apt-get11:10:122
2278999263,17cyclictest0-21swapper/211:24:162
22789992614,7cyclictest2265-21apt-get10:00:142
2278999260,11cyclictest0-21swapper/211:54:172
2278199262,18cyclictest14064-21rm11:00:151
2277099263,15cyclictest2047-21ssh12:05:130
2277099263,15cyclictest2047-21ssh12:05:130
2280199254,13cyclictest35966irq/58-eth011:00:153
2280199251,16cyclictest0-21swapper/310:00:133
2278999250,13cyclictest31860-21sh11:25:392
2278199253,10cyclictest0-21swapper/112:00:151
22770992517,3cyclictest12-21ksoftirqd/010:10:140
2277099251,21cyclictest6150-21ssh10:20:130
2277099251,17cyclictest30488-21ssh11:25:250
2280199243,13cyclictest0-21swapper/310:46:273
2280199242,16cyclictest35966irq/58-eth011:20:133
2280199241,17cyclictest0-21swapper/311:55:143
2278999242,10cyclictest0-21swapper/209:24:112
2278199247,9cyclictest131rcu_preempt11:15:131
22770992413,9cyclictest12-21ksoftirqd/010:40:150
22770992412,3cyclictest12-21ksoftirqd/010:00:490
2280199233,13cyclictest35966irq/58-eth010:10:123
2280199231,17cyclictest35966irq/58-eth011:35:123
2280199231,13cyclictest35966irq/58-eth010:55:143
2280199231,13cyclictest35966irq/58-eth010:55:133
2278999237,12cyclictest17117-21/usr/sbin/munin08:45:232
2278999236,13cyclictest0-21swapper/208:50:132
22789992310,6cyclictest26528-21cron08:55:032
2277099232,11cyclictest14042-21ssh11:00:140
2277099231,18cyclictest11296-21apt-get10:05:140
2277099231,17cyclictest3633-21ssh09:25:150
2280199223,13cyclictest0-21swapper/312:09:233
2280199223,13cyclictest0-21swapper/312:09:233
2278999227,13cyclictest11010-21apt08:05:112
2278199222,15cyclictest0-21swapper/109:36:261
2278199222,12cyclictest630-21Async09:36:311
2278199221,15cyclictest6171-21ssh10:20:151
22770992216,4cyclictest0-21swapper/011:05:250
22770992214,5cyclictest0-21swapper/010:35:130
2280199217,8cyclictest0-21swapper/311:05:133
2280199214,7cyclictest0-21swapper/312:20:133
2278999211,16cyclictest15783-21kthreadcore07:35:202
2278199215,11cyclictest17798-21ssh11:20:141
22781992114,4cyclictest23822-21/usr/sbin/munin12:35:171
2278199211,14cyclictest30486-21cut08:25:161
2278199211,12cyclictest12281-21latency_hist09:30:011
2278199210,17cyclictest20478-21latency_hist08:15:021
2280199209,8cyclictest0-21swapper/310:06:083
2280199203,9cyclictest0-21swapper/307:15:163
2278999203,7cyclictest1756-21head08:25:302
2278199202,12cyclictest19072-21ssh11:37:271
2278199201,16cyclictest744-21timerwakeupswit12:20:281
2278199201,11cyclictest4883-21ssh10:55:141
2278199201,11cyclictest4883-21ssh10:55:141
22781992010,8cyclictest32519-21apt-get07:20:131
2278199200,17cyclictest17743-21ls07:35:221
2277099208,9cyclictest32468-21cron12:40:020
2277099202,15cyclictest30138-21kthreadcore07:15:220
2277099201,8cyclictest0-21swapper/011:31:420
2277099201,8cyclictest0-21swapper/011:31:420
22770992015,3cyclictest0-21swapper/011:10:140
22770992014,4cyclictest0-21swapper/009:45:150
22770992012,6cyclictest0-21swapper/008:05:240
22770992012,5cyclictest0-21swapper/010:57:270
22770992012,5cyclictest0-21swapper/010:57:260
22770992011,7cyclictest16490-21sshd09:13:500
22770992011,7cyclictest16490-21sshd09:13:490
2277099201,15cyclictest2846-21cut08:30:160
2277099201,13cyclictest5984-21latency_hist08:00:010
22770992010,2cyclictest12-21ksoftirqd/012:00:170
2280199194,11cyclictest0-21swapper/312:30:133
2280199192,8cyclictest0-21swapper/311:40:123
2280199192,11cyclictest0-21swapper/308:10:043
22801991912,4cyclictest0-21swapper/309:25:543
2280199191,12cyclictest0-21swapper/309:30:133
2280199191,11cyclictest0-21swapper/307:55:013
22801991910,7cyclictest0-21swapper/311:45:313
2278199192,8cyclictest5706-21ntp_states12:05:261
2278199192,8cyclictest5706-21ntp_states12:05:261
2278199192,7cyclictest19633-21ps10:45:201
2278199191,9cyclictest28390-21sed07:15:201
2278199191,15cyclictest4905-21apt-get07:25:131
2278199191,15cyclictest26764-21apt-get08:55:121
2278199191,15cyclictest22319-21kthreadcore07:40:211
22781991911,4cyclictest32019-21sendmail_mailst10:50:281
2278199191,10cyclictest24273-21sh10:30:121
2278199190,16cyclictest24405-21timerandwakeup08:15:291
2277099199,2cyclictest12-21ksoftirqd/008:05:010
2277099198,3cyclictest12-21ksoftirqd/009:15:250
2277099196,4cyclictest12-21ksoftirqd/007:50:160
2277099193,13cyclictest151rcuc/007:20:120
22770991914,3cyclictest0-21swapper/011:45:290
2277099191,3cyclictest131rcu_preempt10:52:110
22770991910,6cyclictest23920-21cron07:45:010
2280199189,6cyclictest0-21swapper/309:52:333
2280199182,11cyclictest0-21swapper/309:20:143
2280199182,10cyclictest0-21swapper/308:55:143
2280199182,10cyclictest0-21swapper/308:15:153
2280199181,13cyclictest0-21swapper/309:15:243
2280199181,13cyclictest0-21swapper/307:30:023
2280199181,10cyclictest0-21swapper/307:10:123
22801991810,6cyclictest0-21swapper/309:11:563
22801991810,6cyclictest0-21swapper/309:11:553
2278999181,13cyclictest22836-21grep07:10:022
2278199183,11cyclictest12680-21ssh09:30:141
2278199182,11cyclictest30720-21ssh11:45:151
2278199182,10cyclictest7285-21ssh11:50:141
22781991811,4cyclictest19340-21kthreadcore09:50:241
2277099189,6cyclictest16435-21kthreadcore12:30:190
2277099189,2cyclictest12-21ksoftirqd/008:40:250
2277099189,2cyclictest12-21ksoftirqd/008:40:240
2277099187,2cyclictest131rcu_preempt09:50:170
2277099182,11cyclictest0-21swapper/008:20:170
2277099181,6cyclictest20069-21ntp_offset11:55:280
2277099181,11cyclictest20454-21missed_timers08:45:240
2280199172,8cyclictest35966irq/58-eth008:50:133
2280199171,13cyclictest0-21swapper/307:35:173
2280199171,11cyclictest0-21swapper/309:00:173
2280199171,10cyclictest0-21swapper/307:55:303
2278199178,7cyclictest614-21dbus-daemon07:13:261
2278199174,10cyclictest25787-21grep08:50:301
2278199173,11cyclictest3511-21kthreadcore08:30:191
2278199172,12cyclictest22922-21ssh10:10:221
2278199171,6cyclictest26935-21apt-get09:20:131
22781991711,4cyclictest0-21swapper/107:30:271
2278199171,12cyclictest21676-21ssh11:40:121
2278199171,12cyclictest19606-21apt-get12:15:031
2278199171,12cyclictest1111-21apt-get07:55:051
2278199171,12cyclictest0-21swapper/107:45:161
2278199171,11cyclictest2465-21missed_timers09:00:241
2278199170,14cyclictest20764-21ls08:45:281
2277099178,2cyclictest12-21ksoftirqd/008:10:170
2277099172,11cyclictest0-21swapper/008:25:140
2277099172,10cyclictest0-21swapper/008:35:130
2277099171,7cyclictest25967-21cut07:10:220
2277099170,9cyclictest3912-21/usr/sbin/munin09:05:240
2280199161,12cyclictest0-21swapper/308:00:233
2280199161,12cyclictest0-21swapper/307:40:213
2278199165,8cyclictest26199-21tr08:20:231
2278199164,9cyclictest21-21ksoftirqd/108:40:161
2278199164,9cyclictest21-21ksoftirqd/108:40:151
2278199163,10cyclictest11682-21kthreadcore08:05:191
2278199161,12cyclictest29117-21fschecks_time07:50:161
2278199161,12cyclictest11138-21diskstats12:10:161
2278199161,12cyclictest11138-21diskstats12:10:151
2278199161,11cyclictest12111-21basename08:40:011
2277099167,7cyclictest14600-21diskstats07:35:150
2277099161,12cyclictest2529-21ntp_192.168.11409:00:240
2277099161,12cyclictest24542-21irqcore07:45:190
2277099161,11cyclictest25423-21ntp_states08:50:260
2280199152,9cyclictest0-21swapper/308:20:183
2280199151,12cyclictest0-21swapper/308:35:023
2280199146,3cyclictest0-21swapper/308:40:183
2280199146,3cyclictest0-21swapper/308:40:183
2280199141,7cyclictest0-21swapper/309:10:013
2280199141,11cyclictest0-21swapper/307:25:013
2278199144,8cyclictest22785-21latency_hist07:10:021
2280199131,10cyclictest0-21swapper/308:45:223
2277099132,8cyclictest22839-21ls07:10:020
2280199123,6cyclictest0-21swapper/308:25:143
2280199122,6cyclictest0-21swapper/307:30:143
2280199121,4cyclictest0-21swapper/307:45:223
2280199101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional