You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-03 - 01:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Thu Jul 02, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
474699721,59cyclictest0-21swapper/209:15:122
472599654,5cyclictest131rcu_preempt07:25:120
472599612,42cyclictest6679-21ssh12:00:130
472599612,42cyclictest6679-21ssh12:00:120
473799602,44cyclictest4452-21ssh11:05:131
472599604,47cyclictest18107-21ssh09:25:150
472599572,10cyclictest131rcu_preempt07:20:120
473799544,40cyclictest2321-21ssh10:10:121
472599542,40cyclictest13614-21ssh11:10:140
473799535,34cyclictest6191-21ssh10:30:121
473799512,37cyclictest12028-21ssh11:45:121
472599503,12cyclictest131rcu_preempt12:25:120
472599491,40cyclictest23317-21ssh09:10:140
474699483,38cyclictest0-21swapper/212:00:122
474699483,38cyclictest0-21swapper/212:00:112
474699481,42cyclictest7558-21apt-get09:55:152
474699480,43cyclictest27106-21ssh09:30:112
473799483,33cyclictest17336-21ssh11:30:111
474699471,40cyclictest0-21swapper/209:25:162
4761994632,9cyclictest35966irq/58-eth012:20:143
474699463,39cyclictest27346irq/40-dwc2_hso07:45:132
474699463,39cyclictest27346irq/40-dwc2_hso07:45:132
474699461,40cyclictest25395-21apt-get08:05:132
474699461,28cyclictest5300-21ssh09:52:492
474699451,39cyclictest0-21swapper/209:10:122
473799452,30cyclictest8319-21ssh11:25:111
4725994519,19cyclictest12-21ksoftirqd/010:40:120
472599451,35cyclictest29576-21ssh10:25:120
474699443,37cyclictest1072-21apt-get12:15:102
476199432,25cyclictest35966irq/58-eth012:10:133
474699432,34cyclictest0-21swapper/211:35:122
474699431,37cyclictest20620-21apt-get10:20:122
474699430,37cyclictest0-21swapper/209:35:132
476199424,22cyclictest35966irq/58-eth012:05:123
474699423,35cyclictest8743-21apt-get08:55:142
474699422,36cyclictest30047-21apt-get11:55:122
474699422,35cyclictest27346irq/40-dwc2_hso11:50:132
474699421,6cyclictest25670-21ssh10:05:122
474699421,36cyclictest15207-21apt-get10:35:132
474699421,27cyclictest0-21swapper/211:00:422
474699420,38cyclictest0-21swapper/210:00:102
474699420,28cyclictest13039-21ssh10:32:542
472599422,31cyclictest4442-21ssh11:05:120
474699412,35cyclictest0-21swapper/211:30:132
474699411,35cyclictest4430-21apt-get11:05:122
474699411,29cyclictest0-21swapper/209:20:102
474699410,32cyclictest26789-21apt-get08:40:132
474699410,23cyclictest0-21swapper/211:23:012
472599412,29cyclictest21023-21ssh11:50:130
474699401,35cyclictest0-21swapper/208:10:102
474699401,32cyclictest27346irq/40-dwc2_hso07:20:132
474699400,22cyclictest0-21swapper/210:29:582
476199393,20cyclictest35966irq/58-eth009:10:133
474699392,34cyclictest0-21swapper/212:25:122
474699392,33cyclictest2933-21apt-get11:40:122
474699391,34cyclictest0-21swapper/210:40:132
474699391,26cyclictest10962-21rm12:37:102
474699391,25cyclictest0-21swapper/210:58:232
474699390,34cyclictest0-21swapper/208:25:122
474699390,28cyclictest9931-21apt-get10:50:142
474699390,27cyclictest1306-21sh12:31:232
474699390,25cyclictest21937-21find08:35:132
474699390,25cyclictest21937-21find08:35:122
472599393,31cyclictest9951-21ssh10:50:150
476199383,23cyclictest35966irq/58-eth010:05:133
476199382,23cyclictest35966irq/58-eth009:05:113
474699382,33cyclictest16292-21kthreadcore09:00:232
474699382,33cyclictest16292-21kthreadcore09:00:222
474699381,31cyclictest10146-21apt-get12:20:132
474699381,28cyclictest16715-21ssh12:05:202
474699380,35cyclictest10922-21kthreadcore07:15:182
474699380,35cyclictest0-21swapper/207:35:102
474699380,30cyclictest73-21kswapd007:55:312
474699380,27cyclictest19557-21sh11:48:442
474699380,25cyclictest930-21snmpd12:12:202
472599384,31cyclictest10168-21ssh12:20:130
474699371,34cyclictest22363-21latency_hist11:15:032
474699371,33cyclictest0-21swapper/210:15:122
474699371,33cyclictest0-21swapper/209:45:142
474699371,32cyclictest4763-21kthreadcore10:10:192
474699371,32cyclictest20993-21grep08:30:302
474699370,29cyclictest0-21swapper/211:19:572
474699370,29cyclictest0-21swapper/211:19:572
474699370,28cyclictest19410-21apt07:25:122
473799372,31cyclictest9007-21ssh09:20:121
473799371,28cyclictest12848-21ssh09:40:121
472599371,29cyclictest21911-21rm09:45:130
474699361,31cyclictest11409-21head08:20:292
474699360,31cyclictest5888-21ssh10:45:392
474699360,21cyclictest0-21swapper/208:15:142
474699351,30cyclictest10184-21ssh11:25:182
474699350,32cyclictest12454-21kthreadcore07:50:192
474699350,27cyclictest27614-21/usr/sbin/munin07:30:242
473799351,26cyclictest22598-21ssh11:15:121
473799351,26cyclictest22598-21ssh11:15:111
474699345,27cyclictest31712-21cut08:45:152
474699341,30cyclictest18570-21sed09:05:162
474699340,31cyclictest21589-21cut08:00:192
474699340,30cyclictest8527-21basename08:55:022
4725993416,14cyclictest15697-21ssh12:05:130
472599341,4cyclictest0-21swapper/008:25:140
476199332,24cyclictest35966irq/58-eth007:10:133
476199331,20cyclictest35966irq/58-eth011:35:143
474699331,29cyclictest4841-21cat07:10:112
474699331,29cyclictest27346irq/40-dwc2_hso09:40:132
474699331,24cyclictest2046-21kernelversion07:40:192
472599332,23cyclictest11434-21ssh10:15:120
476199325,19cyclictest35966irq/58-eth012:15:113
4761993215,11cyclictest35966irq/58-eth010:15:253
476199321,25cyclictest35966irq/58-eth010:20:133
472599324,21cyclictest2964-21ssh11:40:130
472599322,24cyclictest24633-21ssh12:10:140
472599322,22cyclictest2100-21ssh09:50:230
4725993216,10cyclictest9455-21kworker/u8:309:40:140
472599321,24cyclictest30063-21ssh11:55:130
472599313,20cyclictest27838-21ssh11:00:120
472599312,19cyclictest18983-1kworker/0:1H08:10:110
4725993119,7cyclictest0-21swapper/010:35:140
472599311,27cyclictest1104-21ssh12:15:110
472599311,18cyclictest31703-21ssh11:20:120
476199302,18cyclictest35966irq/58-eth010:50:153
476199301,18cyclictest35966irq/58-eth007:50:133
476199294,20cyclictest35966irq/58-eth009:40:143
473799292,19cyclictest3851-21ssh09:35:151
4737992915,9cyclictest9650irq/52-mmc009:26:361
4737992915,7cyclictest0-21swapper/109:15:111
472599291,22cyclictest8338-21ssh11:25:120
473799283,6cyclictest21-21ksoftirqd/109:45:291
473799278,12cyclictest4710-21cyclictest10:45:141
473799276,8cyclictest21-21ksoftirqd/110:07:011
473799272,19cyclictest21845-21sh10:37:361
473799272,19cyclictest1192-21ssh11:01:351
472599271,19cyclictest7590-21ssh09:55:140
476199262,18cyclictest0-21swapper/312:36:103
4761992610,10cyclictest35966irq/58-eth009:50:333
4737992620,4cyclictest201rcuc/110:26:131
476199256,9cyclictest0-21swapper/311:55:133
4761992512,8cyclictest35966irq/58-eth009:35:143
473799257,8cyclictest21-21ksoftirqd/112:21:221
473799257,8cyclictest21-21ksoftirqd/109:14:131
473799257,7cyclictest21-21ksoftirqd/110:42:071
473799256,9cyclictest21-21ksoftirqd/109:54:111
473799255,3cyclictest131rcu_preempt11:23:151
473799253,5cyclictest21-21ksoftirqd/111:14:561
473799253,4cyclictest21-21ksoftirqd/107:45:251
473799253,4cyclictest21-21ksoftirqd/107:45:241
4737992518,4cyclictest19411-21apt-get07:25:131
472599251,18cyclictest16590-21ssh10:00:110
474699240,21cyclictest4710-21cyclictest07:10:022
473799248,5cyclictest21-21ksoftirqd/109:56:561
473799247,8cyclictest21-21ksoftirqd/112:30:081
473799247,8cyclictest21-21ksoftirqd/111:37:561
473799247,7cyclictest21-21ksoftirqd/109:31:171
473799246,9cyclictest21-21ksoftirqd/110:03:001
473799246,7cyclictest21-21ksoftirqd/112:19:191
473799246,7cyclictest21-21ksoftirqd/111:41:411
473799244,7cyclictest24238-1kworker/u9:210:16:011
473799242,14cyclictest18869-21ssh10:55:151
4737992419,3cyclictest201rcuc/112:11:141
4737992410,12cyclictest201rcuc/110:51:011
472599249,3cyclictest131rcu_preempt10:30:200
4725992413,7cyclictest0-21swapper/008:40:140
476199232,17cyclictest35966irq/58-eth011:40:113
4761992315,3cyclictest0-21swapper/311:00:113
4761992314,6cyclictest35966irq/58-eth011:25:103
4761992314,6cyclictest0-21swapper/310:00:133
473799236,8cyclictest21-21ksoftirqd/112:06:041
473799236,8cyclictest21-21ksoftirqd/112:02:101
473799236,8cyclictest21-21ksoftirqd/112:02:091
473799235,8cyclictest21-21ksoftirqd/109:00:241
473799235,8cyclictest21-21ksoftirqd/109:00:231
473799234,5cyclictest21-21ksoftirqd/111:52:031
473799234,4cyclictest21-21ksoftirqd/108:10:021
473799232,5cyclictest21-21ksoftirqd/107:50:141
4737992318,3cyclictest201rcuc/108:50:181
472599231,18cyclictest32398-21ls09:15:140
476199224,15cyclictest0-21swapper/310:40:123
4761992211,8cyclictest35966irq/58-eth011:05:133
473799227,10cyclictest21-21ksoftirqd/108:45:131
473799226,7cyclictest21-21ksoftirqd/111:58:241
473799225,9cyclictest21-21ksoftirqd/107:55:141
473799225,8cyclictest21-21ksoftirqd/112:39:201
473799225,5cyclictest21-21ksoftirqd/112:29:461
473799225,5cyclictest21-21ksoftirqd/108:15:181
473799222,5cyclictest21-21ksoftirqd/110:20:021
473799222,4cyclictest21-21ksoftirqd/108:20:231
4737992216,4cyclictest21-21ksoftirqd/108:00:151
472599229,10cyclictest5985-21basename07:45:000
472599221,16cyclictest25694-21ssh10:05:130
4725992210,9cyclictest29182-21cron10:25:020
476199215,10cyclictest0-21swapper/309:55:333
476199213,12cyclictest0-21swapper/310:36:353
4761992114,4cyclictest0-21swapper/310:30:223
473799216,7cyclictest32464-21ntp_states07:35:251
473799214,9cyclictest21-21ksoftirqd/108:10:231
472599212,15cyclictest21952-21apt-get08:35:140
472599212,15cyclictest21952-21apt-get08:35:130
472599212,13cyclictest8798-21ls08:55:130
472599211,8cyclictest29534-21cut11:35:220
472599211,17cyclictest6289-21ls08:15:270
472599211,17cyclictest1184-21ls07:40:020
4725992110,6cyclictest5504-21kthreadcore12:35:200
472599210,18cyclictest5776-21ls10:10:210
476199202,11cyclictest0-21swapper/310:25:133
473799203,4cyclictest21-21ksoftirqd/108:35:211
473799203,4cyclictest21-21ksoftirqd/108:35:201
473799202,16cyclictest19494-21kthreadcore08:30:201
473799200,15cyclictest23012-21ls09:10:001
4725992012,5cyclictest26243-21ntp_kernel_pll_11:15:250
4725992012,5cyclictest26243-21ntp_kernel_pll_11:15:240
472599201,15cyclictest9144-21fschecks_count09:20:150
472599200,8cyclictest26949-21tr08:05:190
476199196,9cyclictest35966irq/58-eth010:45:133
476199193,12cyclictest0-21swapper/312:25:133
476199192,14cyclictest35966irq/58-eth009:15:113
476199192,12cyclictest0-21swapper/309:48:343
476199192,11cyclictest0-21swapper/309:30:303
476199191,13cyclictest0-21swapper/311:50:133
476199191,12cyclictest0-21swapper/308:45:153
473799198,8cyclictest4813-21ntp_192.168.11407:40:241
473799196,9cyclictest201rcuc/108:40:191
473799193,4cyclictest21-21ksoftirqd/108:25:211
4737991914,3cyclictest201rcuc/107:15:231
473799191,15cyclictest14711-21uniq07:20:151
472599198,2cyclictest131rcu_preempt07:50:200
472599192,12cyclictest31596-21ssh09:30:300
472599191,9cyclictest9415-21latency_hist07:15:010
472599191,9cyclictest18450-21fschecks_count09:05:150
472599191,14cyclictest4485-21cut09:35:180
4725991910,6cyclictest23985-21cron07:30:020
4725991910,6cyclictest22903-21diskmemload11:45:310
4725991910,6cyclictest17509-21grep11:30:150
476199182,10cyclictest0-21swapper/307:25:053
476199182,10cyclictest0-21swapper/307:15:043
4761991812,4cyclictest0-21swapper/311:30:253
476199181,12cyclictest0-21swapper/307:20:133
476199181,11cyclictest0-21swapper/308:20:293
4761991810,5cyclictest0-21swapper/309:25:233
473799183,3cyclictest131rcu_preempt07:10:161
473799181,5cyclictest21-21ksoftirqd/107:30:171
472599188,7cyclictest28143-21apt-get12:30:120
472599186,4cyclictest131rcu_preempt10:45:190
472599183,12cyclictest8805-21tr08:20:200
472599183,12cyclictest8534-21latency_hist08:55:020
472599183,12cyclictest6753-21irqrtprio07:45:180
472599183,12cyclictest6753-21irqrtprio07:45:170
472599183,12cyclictest16888-21grep09:00:240
472599183,12cyclictest16888-21grep09:00:230
472599181,9cyclictest703-21kthreadcore08:45:190
476199179,6cyclictest0-21swapper/309:21:093
476199178,6cyclictest0-21swapper/311:45:223
476199178,6cyclictest0-21swapper/311:17:093
476199178,6cyclictest0-21swapper/311:17:083
476199178,6cyclictest0-21swapper/310:55:503
476199172,9cyclictest0-21swapper/308:55:133
476199172,10cyclictest0-21swapper/312:00:143
476199172,10cyclictest0-21swapper/312:00:143
476199172,10cyclictest0-21swapper/311:10:143
476199171,14cyclictest35966irq/58-eth010:10:203
473799179,5cyclictest710-21cron09:00:011
472599171,13cyclictest20318-21cut08:30:220
4725991711,3cyclictest19604-21kthreadcore10:55:200
472599171,12cyclictest14271-21cron07:20:010
476199168,6cyclictest0-21swapper/311:24:543
476199168,4cyclictest0-21swapper/312:30:153
476199162,10cyclictest0-21swapper/307:40:133
472599162,11cyclictest20635-21apt-get08:00:120
472599162,11cyclictest17134-21kthreadcore07:55:190
472599161,12cyclictest4802-21latency_hist07:10:010
476199151,8cyclictest35966irq/58-eth008:40:153
476199151,11cyclictest0-21swapper/307:35:173
473799153,9cyclictest4605-21latency_hist07:10:021
476199149,3cyclictest0-21swapper/309:00:283
476199149,3cyclictest0-21swapper/309:00:283
476199149,3cyclictest0-21swapper/308:13:363
476199141,7cyclictest0-21swapper/307:55:113
476199141,6cyclictest0-21swapper/308:35:013
476199141,10cyclictest0-21swapper/308:05:163
476199133,8cyclictest0-21swapper/308:50:243
476199131,8cyclictest0-21swapper/308:35:213
476199131,8cyclictest0-21swapper/308:35:203
476199131,8cyclictest0-21swapper/307:35:013
476199131,7cyclictest0-21swapper/308:15:223
476199131,4cyclictest0-21swapper/308:25:303
476199131,4cyclictest0-21swapper/307:45:253
476199131,4cyclictest0-21swapper/307:45:243
476199121,8cyclictest0-21swapper/308:00:213
476199101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional