You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-06 - 21:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Mar 06, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1284699593,44cyclictest2997-21ssh09:30:130
1286499482,27cyclictest17452-21ssh09:20:132
1286499462,37cyclictest15152-21ssh09:55:122
1285699463,36cyclictest29800-21ssh09:45:121
1284699462,33cyclictest3093-21ssh11:55:120
1287799453,12cyclictest28366irq/58-eth010:25:133
1285699451,31cyclictest7398-21ssh10:45:111
12877994425,14cyclictest28366irq/58-eth011:50:133
1286499442,25cyclictest19707-21ssh11:10:132
1287799433,25cyclictest28366irq/58-eth009:15:143
1287799433,25cyclictest28366irq/58-eth009:15:133
1287799431,12cyclictest28366irq/58-eth010:10:103
1284699438,23cyclictest12-21ksoftirqd/007:35:140
1284699431,33cyclictest31987-21ssh09:10:140
1287799416,22cyclictest28366irq/58-eth012:15:133
1286499398,22cyclictest12837-21cyclictest10:15:112
12864993818,5cyclictest0-21swapper/212:35:132
12864993818,5cyclictest0-21swapper/212:35:122
1285699381,6cyclictest131rcu_preempt11:50:121
1284699382,20cyclictest28723-21ssh11:15:150
1285699372,26cyclictest23170-21ssh11:30:121
1287799363,18cyclictest28366irq/58-eth009:20:123
12864993619,8cyclictest0-21swapper/209:50:112
12846993619,7cyclictest12-21ksoftirqd/009:50:130
1287799357,16cyclictest28366irq/58-eth011:25:113
1284699351,24cyclictest8741-21ssh11:40:110
12877993415,11cyclictest28366irq/58-eth009:05:123
1286499332,17cyclictest21997-21ssh10:35:152
1284699332,25cyclictest29807-21sh09:45:140
12877993210,5cyclictest0-21swapper/310:17:423
1286499323,19cyclictest22346irq/40-dwc2_hso08:00:132
1286499323,18cyclictest27-21ksoftirqd/208:20:142
1287799318,13cyclictest28366irq/58-eth009:55:103
1286499313,21cyclictest22346irq/40-dwc2_hso07:10:132
1286499311,16cyclictest0-21swapper/210:55:122
1285699316,20cyclictest17758-21ssh11:45:131
1285699312,6cyclictest31973-21apt-get09:10:131
12846993110,15cyclictest12-21ksoftirqd/007:40:120
1287799302,17cyclictest28366irq/58-eth009:10:143
12877993016,9cyclictest28366irq/58-eth010:40:123
1286499303,21cyclictest0-21swapper/210:08:402
12856993023,5cyclictest8488-21diskmemload09:15:141
12856993023,5cyclictest8488-21diskmemload09:15:131
1284699302,22cyclictest26938-21sh10:19:420
1284699302,19cyclictest24105-21ssh10:00:130
1284699302,19cyclictest24105-21ssh10:00:130
1284699302,16cyclictest660-21ssh10:05:140
1287799295,12cyclictest28466irq/59-eth010:00:123
1287799295,12cyclictest28466irq/59-eth010:00:113
1286499298,14cyclictest27-21ksoftirqd/211:05:142
1286499293,17cyclictest22346irq/40-dwc2_hso08:50:112
1286499292,20cyclictest0-21swapper/209:10:132
12864992913,13cyclictest2208-21ps10:40:272
1286499291,22cyclictest27-21ksoftirqd/211:00:402
1286499291,16cyclictest15648-21ssh12:20:132
1285699292,23cyclictest15143-21ssh09:55:111
1285699292,19cyclictest5240-21ssh11:20:131
1284699296,13cyclictest13008-21ssh10:30:120
1284699291,23cyclictest23062-21ssh09:59:160
1287799282,16cyclictest28366irq/58-eth012:10:113
1287799282,16cyclictest28366irq/58-eth011:55:153
12877992811,11cyclictest28366irq/58-eth010:55:133
1286499287,10cyclictest27-21ksoftirqd/212:11:372
1286499283,19cyclictest0-21swapper/210:54:412
1286499283,17cyclictest22346irq/40-dwc2_hso08:45:142
1286499282,21cyclictest21077-21ssh12:05:122
1286499282,20cyclictest0-21swapper/209:00:142
1286499282,20cyclictest0-21swapper/207:30:152
1285699286,8cyclictest131rcu_preempt08:15:121
1284699285,18cyclictest12-21ksoftirqd/010:50:140
1284699284,16cyclictest12100-21ssh12:00:140
1287799273,16cyclictest0-21swapper/311:24:223
1287799273,16cyclictest0-21swapper/310:54:183
1287799272,11cyclictest0-21swapper/310:09:153
1287799271,17cyclictest0-21swapper/311:30:123
1286499274,13cyclictest22346irq/40-dwc2_hso08:35:142
1286499273,11cyclictest27-21ksoftirqd/212:29:012
12864992720,4cyclictest7807-21ssh10:45:182
1285699274,13cyclictest1278-21ssh12:30:151
1284699278,16cyclictest14326-1kworker/u9:109:20:130
1284699272,19cyclictest13553-21sh11:24:420
1287799264,14cyclictest28366irq/58-eth012:20:133
1287799263,16cyclictest0-21swapper/311:05:133
1286499269,3cyclictest131rcu_preempt09:37:262
1286499269,11cyclictest27-21ksoftirqd/211:56:332
1286499267,9cyclictest27-21ksoftirqd/208:10:242
1286499264,13cyclictest27-21ksoftirqd/209:15:032
1286499264,13cyclictest27-21ksoftirqd/209:15:022
1286499263,17cyclictest0-21swapper/210:00:422
1286499263,17cyclictest0-21swapper/210:00:412
1286499263,16cyclictest22346irq/40-dwc2_hso08:30:142
1286499262,9cyclictest3059-21ssh10:24:172
1286499262,21cyclictest0-21swapper/208:55:132
12864992611,12cyclictest0-21swapper/212:30:112
1284699266,14cyclictest12-21ksoftirqd/008:45:150
1284699261,22cyclictest32112-21ssh11:35:100
1287799253,14cyclictest28366irq/58-eth010:20:123
12877992514,8cyclictest28366irq/58-eth011:40:123
1286499252,10cyclictest28700-21diskmemload11:15:132
12864992513,9cyclictest10609-21ssh09:33:562
1285699256,4cyclictest131rcu_preempt10:59:181
1285699253,14cyclictest24187-1kworker/u9:012:10:121
1285699251,4cyclictest131rcu_preempt12:20:141
1285699251,21cyclictest628-21ssh10:05:131
1285699251,17cyclictest16379-21sh10:50:151
1284699254,11cyclictest633-21Async12:11:530
1284699253,3cyclictest131rcu_preempt10:35:150
1284699252,18cyclictest20873-21ssh09:40:140
1284699251,19cyclictest30852-21ssh10:40:120
1284699251,18cyclictest28438-21ssh12:25:270
1287799243,10cyclictest28366irq/58-eth012:30:143
1286499248,13cyclictest27-21ksoftirqd/210:25:242
1286499247,8cyclictest27-21ksoftirqd/209:40:152
1286499247,11cyclictest27-21ksoftirqd/212:18:142
1286499246,11cyclictest27-21ksoftirqd/210:30:032
1286499243,16cyclictest0-21swapper/210:11:412
1286499242,8cyclictest13259-21ssh11:24:202
1286499242,16cyclictest0-21swapper/211:40:052
12864992411,10cyclictest27-21ksoftirqd/211:30:292
12864992410,11cyclictest0-21swapper/207:45:222
1285699249,12cyclictest21-21ksoftirqd/108:05:231
1285699242,17cyclictest26591-21ssh10:19:171
1285699241,17cyclictest26489-21ssh09:25:151
1284699242,17cyclictest14230-21ssh11:25:130
1284699241,3cyclictest131rcu_preempt10:45:130
1284699241,14cyclictest27624-21ssh10:20:130
1287799233,13cyclictest0-21swapper/309:54:423
1286499239,12cyclictest42-21kauditd09:10:012
1286499236,11cyclictest0-21swapper/207:25:012
1286499232,6cyclictest0-21swapper/211:54:422
1286499232,15cyclictest22346irq/40-dwc2_hso07:20:122
12864992316,4cyclictest32676-21ssh11:35:172
12864992312,6cyclictest17745-21apt-get11:45:122
1286499231,16cyclictest0-21swapper/208:05:122
1286499231,14cyclictest22346irq/40-dwc2_hso07:40:122
1286499231,10cyclictest30531-21timerwakeupswit09:25:292
12856992312,4cyclictest21-21ksoftirqd/110:25:151
1285699231,19cyclictest1774-21ssh11:00:141
1285699231,18cyclictest12253-21grep12:00:161
1285699231,17cyclictest0-21swapper/109:00:151
1284699234,11cyclictest15841-21kworker/u8:012:10:110
1287799222,11cyclictest28466irq/59-eth012:35:133
1287799222,11cyclictest28466irq/59-eth012:35:133
1286499229,6cyclictest22388-21cron07:20:002
1286499228,5cyclictest27-21ksoftirqd/211:29:152
1286499221,12cyclictest1208-21awk09:45:282
1285699222,17cyclictest10728-21ssh11:05:131
12856992211,8cyclictest6401-21entropy09:50:151
1285699221,18cyclictest21076-21sh12:05:131
1284699228,5cyclictest12-21ksoftirqd/008:55:120
1284699221,11cyclictest31758-21munin-run11:35:010
1287799213,12cyclictest0-21swapper/311:00:413
1287799212,14cyclictest0-21swapper/312:05:133
1287799211,12cyclictest28366irq/58-eth008:40:113
1286499216,5cyclictest21472-21kthreadcore08:25:182
1286499212,5cyclictest27-21ksoftirqd/212:00:212
1286499211,17cyclictest7037-21kthreadcore07:35:202
1286499211,10cyclictest4973-21kthreadcore08:40:192
1285699212,16cyclictest13394-21ps10:30:191
1285699211,16cyclictest30454-21fschecks_time08:35:151
1284699219,8cyclictest0-21swapper/007:45:130
1284699214,12cyclictest0-21swapper/009:35:140
1284699213,11cyclictest12837-21cyclictest07:15:120
1284699212,14cyclictest7605-21ssh11:01:400
1284699212,14cyclictest12011-21ssh10:29:170
1284699211,6cyclictest18379-21sed11:45:180
1284699211,10cyclictest15159-21apt-get12:20:010
1287799208,8cyclictest28366irq/58-eth009:45:123
1287799204,12cyclictest28366irq/58-eth008:05:123
1287799202,11cyclictest0-21swapper/308:15:133
12877992011,6cyclictest0-21swapper/311:45:253
1287799201,13cyclictest0-21swapper/307:35:233
12877992010,8cyclictest0-21swapper/311:17:433
1286499201,9cyclictest20582-21kthreadcore07:50:192
1286499201,9cyclictest20582-21kthreadcore07:50:182
1285699208,9cyclictest10883-21cut10:10:191
1285699208,7cyclictest21-21ksoftirqd/112:25:141
1285699202,15cyclictest6649-21sh12:15:131
1285699202,14cyclictest3323-21users10:40:311
12856992013,4cyclictest27217-21sh07:25:001
1285699201,15cyclictest6107-21apt-get08:10:111
1285699201,15cyclictest21820-21ssh10:35:021
1285699201,14cyclictest32130-21rm11:35:131
1285699201,14cyclictest17606-21latency07:45:231
1284699203,12cyclictest9616-21ssh10:10:100
1284699201,9cyclictest15863-21sort12:20:150
12846992012,5cyclictest0-21swapper/007:20:160
12846992010,7cyclictest20935-21cat11:10:200
1284699200,3cyclictest276941cut11:50:200
1284699200,17cyclictest22120-21find09:00:130
1287799193,11cyclictest0-21swapper/311:12:423
1287799193,11cyclictest0-21swapper/310:34:163
1287799192,11cyclictest0-21swapper/309:35:143
1287799192,11cyclictest0-21swapper/308:35:143
1287799192,11cyclictest0-21swapper/308:25:053
1287799192,11cyclictest0-21swapper/308:20:133
1287799192,10cyclictest0-21swapper/308:45:153
1287799192,10cyclictest0-21swapper/308:10:053
12877991912,4cyclictest0-21swapper/312:00:223
12877991911,4cyclictest0-21swapper/307:30:133
1287799191,13cyclictest0-21swapper/307:50:273
1287799191,13cyclictest0-21swapper/307:50:263
12877991910,6cyclictest0-21swapper/312:26:173
1286499199,8cyclictest0-21swapper/207:55:142
1286499191,12cyclictest0-21swapper/208:15:112
1285699198,8cyclictest32382-21tr11:15:251
1285699195,11cyclictest27869-21irqcore07:25:171
1285699194,9cyclictest0-21swapper/111:25:141
1285699194,11cyclictest20996-21/usr/sbin/munin09:40:171
1285699191,6cyclictest131rcu_preempt11:55:131
12856991912,5cyclictest25835-21cut08:30:191
12856991911,5cyclictest5175-21irqrtprio07:35:181
1285699191,15cyclictest27665-21ls10:00:221
1285699191,15cyclictest27665-21ls10:00:211
1285699191,12cyclictest11898-21ssh09:35:131
1284699191,8cyclictest9887-21kthreadcore09:15:220
1284699191,8cyclictest9887-21kthreadcore09:15:220
1284699191,7cyclictest15531-21latency_hist08:20:010
12846991913,4cyclictest0-21swapper/008:25:050
12846991912,5cyclictest0-21swapper/008:10:170
1284699191,14cyclictest15781-21apt-get08:20:120
1284699191,12cyclictest25630-21cut08:30:160
1284699190,16cyclictest19351-21ls11:10:020
1287799189,6cyclictest0-21swapper/310:45:483
1287799189,6cyclictest0-21swapper/309:40:323
1287799183,10cyclictest0-21swapper/311:39:213
1287799182,11cyclictest0-21swapper/308:30:143
1287799182,10cyclictest0-21swapper/308:50:133
1287799182,10cyclictest0-21swapper/307:40:123
1287799182,10cyclictest0-21swapper/307:10:133
12877991812,3cyclictest0-21swapper/310:35:163
12877991811,4cyclictest0-21swapper/309:30:303
1287799181,13cyclictest0-21swapper/309:00:183
1287799181,12cyclictest0-21swapper/307:20:253
1285699186,9cyclictest11656-21latency_hist09:35:021
1285699186,9cyclictest11395-21kthreadcore12:35:191
1285699186,9cyclictest11395-21kthreadcore12:35:181
1285699184,11cyclictest28805-21kthreadcore10:20:201
1285699184,11cyclictest20369-21kthreadcore09:20:221
1285699182,12cyclictest8292-21apt-get11:40:011
1285699182,12cyclictest20371-21basename08:25:011
1285699181,9cyclictest29923-21kthreadcore09:05:221
12856991813,3cyclictest0-21swapper/107:55:231
1285699181,13cyclictest22995-21grep07:50:281
1285699181,13cyclictest22995-21grep07:50:271
1285699181,13cyclictest15727-21missed_timers08:50:221
12856991810,5cyclictest0-21swapper/107:30:181
1285699180,15cyclictest7674-21cut08:45:161
1284699183,10cyclictest0-21swapper/010:59:410
1284699181,8cyclictest957-21cron08:05:020
1284699181,2cyclictest131rcu_preempt09:10:020
12846991810,5cyclictest12934-21kthreadcore12:35:210
12846991810,5cyclictest12934-21kthreadcore12:35:200
1284699180,12cyclictest5270-21timerandwakeup12:30:290
1287799178,5cyclictest28366irq/58-eth009:25:133
1287799172,10cyclictest0-21swapper/307:25:133
1285699174,11cyclictest32086-21latency08:00:221
1285699172,11cyclictest7360-21latency_hist08:45:011
1285699171,6cyclictest131rcu_preempt07:40:171
1285699171,12cyclictest20523-21kthreadcore08:55:211
1285699170,7cyclictest19753-21/usr/sbin/munin11:10:141
1284699179,5cyclictest0-21swapper/007:30:260
1284699178,6cyclictest17552-21cron07:15:020
12846991711,4cyclictest0-21swapper/007:25:150
1284699171,13cyclictest1198-21apt-get08:05:110
1287799162,9cyclictest0-21swapper/308:00:153
1287799161,12cyclictest0-21swapper/308:55:153
1285699163,11cyclictest21178-21grep08:25:181
1285699162,12cyclictest18700-21tr07:15:191
1285699162,11cyclictest13704-21kthreadcore07:10:181
1284699168,6cyclictest12-21ksoftirqd/008:35:140
1284699167,6cyclictest12-21ksoftirqd/007:55:120
1284699167,3cyclictest131rcu_preempt08:40:260
1284699165,3cyclictest131rcu_preempt12:05:120
1284699162,10cyclictest0-21swapper/008:50:120
1287799152,9cyclictest0-21swapper/307:45:153
1286499151,10cyclictest12867-21latency_hist07:10:012
1284699157,6cyclictest19358-21cut07:50:160
1284699157,6cyclictest19358-21cut07:50:160
1285699141,9cyclictest12912-21grep07:10:021
1284699141,9cyclictest12837-21cyclictest07:10:010
1287799121,5cyclictest0-21swapper/307:15:113
1287799111,8cyclictest0-21swapper/307:55:163
1287799111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional