You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-10 - 10:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Sat Jan 10, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3782995224,23cyclictest25494-21ssh22:05:122
377499511,36cyclictest13565-21ssh23:10:121
376499498,4cyclictest131rcu_preempt00:05:120
376499491,8cyclictest0-21swapper/019:50:130
3791994834,6cyclictest28366irq/58-eth023:05:123
376499488,6cyclictest131rcu_preempt19:45:120
3791994732,7cyclictest28366irq/58-eth023:40:123
377499463,36cyclictest17210-21ssh23:30:131
377499462,34cyclictest5867-21ssh22:30:121
376499452,38cyclictest4964-21apt-get00:35:120
377499440,33cyclictest2117-21ssh22:10:111
376499449,6cyclictest131rcu_preempt21:05:110
376499442,7cyclictest0-21swapper/020:05:120
377499422,29cyclictest24069-21ssh22:40:121
3782994028,8cyclictest22923-21apt-get21:10:132
3782994022,13cyclictest30583-21ssh21:50:122
3782994016,19cyclictest0-21swapper/223:25:122
376499407,6cyclictest131rcu_preempt23:25:130
376499401,4cyclictest8731-21apt-get19:15:120
376499401,4cyclictest0-21swapper/021:50:120
376499401,2cyclictest21455-21apt-get21:45:110
376499401,2cyclictest13096-21apt-get21:00:130
376499401,2cyclictest0-21swapper/021:15:120
376499401,2cyclictest0-21swapper/020:15:120
3782993923,12cyclictest10282-21apt-get00:20:142
3782993912,10cyclictest2259-21ssh23:02:502
376499391,3cyclictest18757-21ssh22:55:120
376499391,34cyclictest0-21swapper/000:20:140
376499391,2cyclictest0-21swapper/021:10:130
377499382,27cyclictest24637-21ssh00:10:121
376499382,5cyclictest0-21swapper/019:25:140
376499382,2cyclictest0-21swapper/019:30:120
376499382,2cyclictest0-21swapper/019:30:120
376499381,8cyclictest26128-21ssh23:35:120
376499381,8cyclictest0-21swapper/020:40:110
376499381,2cyclictest0-21swapper/020:50:120
376499381,2cyclictest0-21swapper/020:10:110
378299379,17cyclictest0-21swapper/222:48:172
3782993728,4cyclictest19933-21apt-get20:00:132
3782993710,8cyclictest22535-21diskmemload21:49:442
376499372,32cyclictest4556-21ssh23:05:130
376499371,3cyclictest0-21swapper/000:30:130
378299369,10cyclictest0-21swapper/221:43:322
3782993624,9cyclictest22346irq/40-dwc2_hso20:50:132
3782993621,12cyclictest2702-21ssh23:40:132
377499362,30cyclictest30578-21ssh21:50:111
376499364,3cyclictest131rcu_preempt22:05:110
376499361,4cyclictest6560-21apt-get00:00:120
376499361,3cyclictest0-21swapper/023:40:130
376499361,3cyclictest0-21swapper/020:45:140
378299357,11cyclictest24395-21ssh22:04:102
3782993510,12cyclictest28461-21sh23:16:442
3782993510,12cyclictest25694-21ssh00:27:152
376499352,4cyclictest0-21swapper/019:35:110
376499351,4cyclictest27870-21ssh23:00:130
376499351,2cyclictest0-21swapper/023:55:110
376499351,2cyclictest0-21swapper/000:10:120
3782993428,2cyclictest20796-21apt-get23:50:132
3782993421,7cyclictest27-21ksoftirqd/219:45:112
3782993415,15cyclictest15778-21fschecks_count00:05:152
3782993410,8cyclictest29193-21ssh22:25:052
3782993410,8cyclictest29193-21ssh22:25:052
3782993410,12cyclictest18488-21ssh23:47:432
376499342,28cyclictest28139-21grep22:40:280
376499341,5cyclictest0-21swapper/019:40:110
376499341,2cyclictest0-21swapper/023:50:130
376499341,12cyclictest15338-21sed22:15:280
378299339,16cyclictest0-21swapper/222:10:112
3782993322,7cyclictest18751-21apt-get22:55:122
3782993317,12cyclictest0-21swapper/220:30:132
3782993317,11cyclictest0-21swapper/219:30:122
3782993317,11cyclictest0-21swapper/219:30:122
3782993315,16cyclictest0-21swapper/220:10:112
3782993313,13cyclictest22535-21diskmemload00:13:472
3782993310,7cyclictest22535-21diskmemload22:41:022
3782993310,10cyclictest19359-21ssh22:19:252
377499332,28cyclictest17720-21ssh21:25:121
376499331,9cyclictest8359-21df_inode20:55:140
376499331,29cyclictest0-21swapper/021:55:110
376499331,29cyclictest0-21swapper/020:00:130
376499331,10cyclictest18317-21/usr/sbin/munin20:30:210
3782993217,8cyclictest8678-21basename21:20:102
3782993215,13cyclictest24967-21ssh21:28:212
3782993210,7cyclictest14010-21ssh22:34:242
376499321,9cyclictest4284-21ntp_192.168.11422:45:250
376499321,9cyclictest11744-21df_inode20:25:140
376499321,28cyclictest18873-21tr23:30:200
376499321,27cyclictest4960-21ntpq00:15:240
376499321,27cyclictest2785-21ntpq23:20:240
376499321,11cyclictest20990-21rm22:20:180
376499321,10cyclictest9353-21sed22:30:220
376499320,3cyclictest12432-21apt-get21:40:120
376499320,10cyclictest17301-21ssh23:10:280
3782993122,6cyclictest28386-21apt-get00:30:132
3782993121,7cyclictest9703-21apt-get22:50:132
3782993117,9cyclictest1267-21ssh00:15:122
3782993115,8cyclictest3322-21sed20:50:022
3782993115,8cyclictest12185-21cut19:50:202
3782993115,13cyclictest8991-21timerandwakeup00:35:282
3782993113,14cyclictest4011-21cut19:10:152
3782993112,14cyclictest0-21swapper/221:35:132
377499311,22cyclictest12444-21ssh21:40:121
376499317,22cyclictest20214-21ssh23:49:450
376499314,24cyclictest13909-21grep22:50:310
376499314,23cyclictest10072-21cut20:20:200
376499313,25cyclictest29290-21ssh22:25:110
376499313,25cyclictest29290-21ssh22:25:110
376499312,2cyclictest15084-21apt-get19:55:130
376499312,25cyclictest23046-21ntp_states00:25:240
376499311,9cyclictest8800-21df21:20:120
376499311,7cyclictest28805-21tr21:30:200
376499311,6cyclictest15625-21tr19:20:180
376499311,26cyclictest4751-21cut22:10:200
376499310,3cyclictest3390-21apt-get21:35:120
378299308,9cyclictest12771-21sh23:09:332
378299303,21cyclictest0-21swapper/223:31:362
3782993020,7cyclictest29861-21apt-get23:55:112
3782993019,8cyclictest7213-21/usr/sbin/munin21:55:132
3782993015,6cyclictest14528-21apt-get22:35:042
3782993015,12cyclictest15430-21kthreadcore23:10:202
377499303,18cyclictest6570-21ssh00:00:141
377499302,22cyclictest18905-21ssh00:24:581
376499302,2cyclictest0-21swapper/023:15:140
376499301,8cyclictest24945-21/usr/sbin/munin20:35:270
376499301,8cyclictest18095-21irqcore21:25:170
376499301,3cyclictest0-21swapper/022:00:120
376499301,12cyclictest18720-21ntp_192.168.11422:35:250
3791992923,3cyclictest0-21swapper/300:15:123
3791992916,6cyclictest0-21swapper/323:25:133
3791992910,7cyclictest28366irq/58-eth022:40:113
378299298,7cyclictest26440-21sh22:22:002
378299298,17cyclictest6200-21ssh23:23:092
378299297,13cyclictest5430-21ssh21:16:462
3782992916,10cyclictest0-21swapper/220:25:112
3782992914,12cyclictest26096-21awk20:40:102
3782992913,8cyclictest31739-21/usr/sbin/munin19:35:262
3782992913,8cyclictest26963-21sh21:30:162
3782992913,8cyclictest18551-21ntp_kernel_pll_19:55:252
3782992913,8cyclictest12390-21sed19:15:262
3782992913,12cyclictest659-21fschecks_count19:40:142
377499292,21cyclictest31951-21ssh23:36:371
377499291,20cyclictest11203-21ssh22:15:131
376499291,7cyclictest3931-21cpu19:10:130
379199289,10cyclictest28366irq/58-eth023:45:113
3791992821,4cyclictest0-21swapper/321:10:133
378299286,8cyclictest22535-21diskmemload23:35:022
3782992817,9cyclictest21271-21apt-get20:35:122
3782992817,7cyclictest0-21swapper/220:15:122
3782992815,10cyclictest0-21swapper/200:00:122
3782992812,12cyclictest18000-21cpuspeed21:05:132
3782992812,12cyclictest13747-21fschecks_time19:20:142
377499281,21cyclictest9659-21ssh23:42:571
3791992720,4cyclictest0-21swapper/323:00:143
3791992718,4cyclictest0-21swapper/321:05:123
3791992717,8cyclictest0-21swapper/322:10:113
3782992716,8cyclictest12104-21unixbench_multi20:55:272
3782992713,11cyclictest13095-21apt21:00:122
377499274,6cyclictest21-21ksoftirqd/123:15:251
377499272,6cyclictest21-21ksoftirqd/123:05:111
3791992620,3cyclictest0-21swapper/321:45:113
3791992618,5cyclictest0-21swapper/323:50:113
3791992617,3cyclictest0-21swapper/321:00:123
3791992614,7cyclictest0-21swapper/320:45:143
3791992613,10cyclictest28366irq/58-eth021:55:123
3782992615,8cyclictest18959-21cut19:25:192
3782992613,10cyclictest7548-21cut20:20:172
377499267,3cyclictest131rcu_preempt23:00:141
377499263,14cyclictest633-21Async12:11:531
377499262,18cyclictest22535-21diskmemload23:27:571
3774992621,3cyclictest201rcuc/123:55:501
376499260,7cyclictest3752-21cyclictest19:10:020
3791992521,2cyclictest0-21swapper/321:50:123
3791992519,2cyclictest0-21swapper/322:30:133
3791992518,5cyclictest0-21swapper/320:00:133
3782992514,8cyclictest3787-21latency_hist19:10:022
3782992513,9cyclictest28637-21grep20:05:282
377499256,3cyclictest131rcu_preempt21:35:171
377499255,6cyclictest21-21ksoftirqd/100:16:111
377499253,6cyclictest21-21ksoftirqd/121:56:121
377499253,4cyclictest21-21ksoftirqd/119:30:011
377499252,18cyclictest19526-21ssh23:48:571
377499251,19cyclictest14962-21kworker/u8:322:35:131
3791992420,2cyclictest0-21swapper/319:40:103
3791992417,4cyclictest0-21swapper/323:15:143
3791992416,5cyclictest0-21swapper/320:10:113
3791992413,6cyclictest0-21swapper/300:10:123
3791992413,4cyclictest0-21swapper/300:30:133
3791992412,9cyclictest0-21swapper/322:09:073
3791992411,10cyclictest0-21swapper/323:10:123
377499246,6cyclictest21-21ksoftirqd/119:45:021
377499246,3cyclictest131rcu_preempt21:32:241
377499245,5cyclictest21-21ksoftirqd/119:50:241
377499245,3cyclictest131rcu_preempt21:45:321
377499245,3cyclictest131rcu_preempt19:45:171
377499244,4cyclictest21-21ksoftirqd/100:33:501
377499242,6cyclictest21-21ksoftirqd/122:27:171
377499242,6cyclictest21-21ksoftirqd/122:27:171
3774992417,4cyclictest0-21swapper/121:00:131
377499241,6cyclictest21-21ksoftirqd/122:01:021
377499241,4cyclictest21-21ksoftirqd/122:50:181
3774992411,7cyclictest0-21swapper/123:51:371
3774992411,7cyclictest0-21swapper/121:17:041
379199233,17cyclictest28366irq/58-eth000:25:123
3791992319,2cyclictest0-21swapper/322:35:123
3791992319,2cyclictest0-21swapper/322:25:113
3791992319,2cyclictest0-21swapper/322:25:113
3791992319,2cyclictest0-21swapper/320:55:123
3791992318,2cyclictest0-21swapper/322:00:123
3791992317,2cyclictest0-21swapper/320:30:143
3791992316,2cyclictest0-21swapper/319:50:123
3791992315,6cyclictest0-21swapper/321:30:133
379199231,15cyclictest28366irq/58-eth021:25:113
377499239,10cyclictest26099-21apt-get20:40:101
377499236,3cyclictest131rcu_preempt21:14:141
377499235,8cyclictest131rcu_preempt22:55:141
377499235,7cyclictest21-21ksoftirqd/120:52:251
377499233,5cyclictest21-21ksoftirqd/121:21:171
377499232,5cyclictest21-21ksoftirqd/121:05:231
3774992310,10cyclictest23990-21ssh22:20:251
379199226,9cyclictest28366irq/58-eth021:20:123
3791992217,2cyclictest0-21swapper/321:40:113
3791992217,2cyclictest0-21swapper/300:05:123
3791992217,2cyclictest0-21swapper/300:00:133
3791992214,6cyclictest0-21swapper/323:20:273
3791992214,3cyclictest0-21swapper/320:50:123
3791992213,7cyclictest0-21swapper/322:15:243
3791992213,6cyclictest0-21swapper/323:57:163
377499225,7cyclictest131rcu_preempt20:20:291
377499225,3cyclictest131rcu_preempt22:45:231
377499224,4cyclictest21-21ksoftirqd/100:25:021
377499222,5cyclictest21-21ksoftirqd/123:23:451
3774992213,5cyclictest0-21swapper/119:55:131
3774992210,9cyclictest20880-21ssh00:05:581
379199212,12cyclictest0-21swapper/319:30:123
379199212,12cyclictest0-21swapper/319:30:123
3791992116,2cyclictest0-21swapper/320:25:113
3791992115,4cyclictest0-21swapper/321:15:133
3791992114,5cyclictest0-21swapper/322:23:403
3791992114,4cyclictest0-21swapper/319:20:123
3791992113,6cyclictest0-21swapper/323:32:393
3791992112,7cyclictest0-21swapper/322:55:153
3791992112,7cyclictest0-21swapper/322:45:153
3791992112,6cyclictest0-21swapper/321:35:303
3791992112,6cyclictest0-21swapper/300:35:133
3791992110,9cyclictest0-21swapper/319:25:253
377499215,7cyclictest21-21ksoftirqd/119:20:131
377499214,4cyclictest21-21ksoftirqd/122:05:491
377499212,4cyclictest21-21ksoftirqd/119:10:171
377499211,15cyclictest8806-21ls19:15:121
3791992016,2cyclictest0-21swapper/300:20:123
3791992015,2cyclictest0-21swapper/322:50:143
3791992011,7cyclictest0-21swapper/319:45:263
3791992011,6cyclictest0-21swapper/323:35:053
3791992010,7cyclictest0-21swapper/320:40:123
3791992010,7cyclictest0-21swapper/320:05:123
3791992010,7cyclictest0-21swapper/319:35:143
377499205,8cyclictest21-21ksoftirqd/120:25:151
377499204,9cyclictest21-21ksoftirqd/120:00:331
377499202,5cyclictest21-21ksoftirqd/120:15:151
379199197,8cyclictest0-21swapper/319:55:123
3791991911,6cyclictest0-21swapper/320:20:133
3791991911,6cyclictest0-21swapper/319:15:103
377499199,8cyclictest30937-21apt-get20:45:131
377499195,4cyclictest21-21ksoftirqd/120:35:161
377499194,8cyclictest21-21ksoftirqd/120:30:311
377499192,5cyclictest21-21ksoftirqd/119:10:021
377499191,4cyclictest26715-21ntp_192.168.11419:30:231
377499191,4cyclictest26715-21ntp_192.168.11419:30:231
3791991814,2cyclictest0-21swapper/320:35:113
3791991810,6cyclictest0-21swapper/319:10:253
3791991810,5cyclictest0-21swapper/320:15:123
377499187,9cyclictest201rcuc/120:05:281
377499182,4cyclictest21-21ksoftirqd/119:35:131
377499174,4cyclictest21-21ksoftirqd/120:10:161
377499171,2cyclictest131rcu_preempt20:55:141
379199139,2cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional