You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-09 - 19:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Mar 09, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26606996238,11cyclictest28366irq/58-eth011:00:133
26599995637,14cyclictest6347-21apt-get08:30:112
2660699503,31cyclictest28366irq/58-eth009:35:133
26599995039,7cyclictest14699-21apt-get08:05:102
2658299501,41cyclictest0-21swapper/010:15:130
26599994828,16cyclictest0-21swapper/209:40:132
2658299471,33cyclictest22437-21ssh10:45:130
2659999461,39cyclictest0-21swapper/211:45:122
26599994533,9cyclictest29106-21apt-get08:20:122
2658299452,39cyclictest6321-21apt-get11:30:140
2658299451,41cyclictest0-21swapper/008:20:120
2658299442,37cyclictest25575-21apt-get09:35:130
2658299440,26cyclictest13621-21/usr/sbin/munin12:10:160
26599994321,16cyclictest15338-21ssh11:35:122
26599994319,16cyclictest0-21swapper/210:50:142
26599994315,9cyclictest10909-21ssh11:14:352
26599994315,10cyclictest15575-21ssh12:28:222
26599994314,23cyclictest0-21swapper/212:30:542
2658299430,27cyclictest2311-21ssh12:39:450
2658299430,27cyclictest2311-21ssh12:39:450
26599994236,2cyclictest11682-21apt-get11:15:152
26599994228,10cyclictest3002-21apt-get09:00:122
26599994226,13cyclictest20376-21apt-get09:50:142
26599994226,13cyclictest20376-21apt-get09:50:132
2658299421,37cyclictest0-21swapper/010:20:130
26599994136,2cyclictest9961-21apt-get11:50:132
26599994127,9cyclictest4622-21apt-get12:05:112
26599994119,17cyclictest0-21swapper/210:45:132
2658299411,24cyclictest0-21swapper/012:28:170
26599994030,8cyclictest6323-21ssh11:30:132
26599994022,15cyclictest29731-21ssh11:25:132
26599994022,15cyclictest0-21swapper/208:15:132
26599994022,15cyclictest0-21swapper/208:15:132
26599994016,21cyclictest0-21swapper/209:05:132
26599994015,20cyclictest0-21swapper/210:20:132
26599994010,25cyclictest19870-21sh12:12:092
2658299401,36cyclictest16485-21apt-get09:30:130
2658299401,34cyclictest0-21swapper/009:05:130
2658299401,25cyclictest14645-21ssh10:57:310
2658299400,24cyclictest21049-21ssh10:06:510
2660699394,27cyclictest28366irq/58-eth008:25:133
2660699393,24cyclictest28366irq/58-eth012:30:123
26599993915,8cyclictest29104-21ssh09:18:262
2659099394,10cyclictest131rcu_preempt07:20:111
2658299391,33cyclictest0-21swapper/007:40:120
2658299390,35cyclictest15332-21apt-get11:35:130
2658299390,34cyclictest28111-21ssh12:00:150
2658299390,26cyclictest961-21ssh11:08:180
2658299390,26cyclictest29842-21ssh11:41:130
2660699384,21cyclictest28366irq/58-eth009:25:123
2660699384,21cyclictest28366irq/58-eth009:25:113
2660699384,19cyclictest28366irq/58-eth012:15:123
2659999388,19cyclictest0-21swapper/212:00:122
26599993829,6cyclictest0-21swapper/210:15:112
26599993823,9cyclictest16496-21ssh09:30:132
2658299381,35cyclictest0-21swapper/009:55:130
2658299381,33cyclictest0-21swapper/007:55:110
2658299381,32cyclictest11702-21ssh11:15:150
2658299381,24cyclictest29694-21ssh12:18:030
2658299381,23cyclictest0-21swapper/011:01:260
2658299380,30cyclictest2538-21nfs4_client07:15:220
2660699373,24cyclictest28366irq/58-eth007:50:113
26599993728,6cyclictest22346irq/40-dwc2_hso07:40:132
26599993722,7cyclictest30708-21cut09:55:202
26599993721,12cyclictest0-21swapper/209:35:122
26599993718,17cyclictest0-21swapper/209:45:122
26599993716,17cyclictest585-21kthreadcore09:20:212
26599993715,17cyclictest9894-21kthreadcore10:55:212
2658299374,18cyclictest16089-21cron09:30:010
2658299374,18cyclictest16089-21cron09:30:000
26582993720,12cyclictest0-21swapper/011:25:140
2658299371,32cyclictest21725-21apt-get09:15:120
2658299371,31cyclictest0-21swapper/011:50:120
2658299371,27cyclictest0-21swapper/012:31:570
2658299371,24cyclictest24188-21wc08:45:220
2658299370,22cyclictest4590-21ssh12:21:010
2660699364,17cyclictest28366irq/58-eth011:55:123
2660699363,25cyclictest28366irq/58-eth010:45:133
26599993620,8cyclictest16086-21apt08:40:122
26599993619,12cyclictest22346irq/40-dwc2_hso07:55:102
26599993613,13cyclictest0-21swapper/212:17:402
26599993611,17cyclictest0-21swapper/211:08:532
2659099361,30cyclictest9992-21ssh11:50:131
2658299362,31cyclictest13362-21apt-get10:40:140
2658299361,31cyclictest31534-21apt-get10:50:130
2658299361,21cyclictest0-21swapper/009:47:180
2658299361,20cyclictest7648-21ssh11:10:390
26599993518,13cyclictest15743-21sshd08:39:082
26599993516,15cyclictest30430-21unixbench_singl12:35:292
26599993516,15cyclictest30430-21unixbench_singl12:35:282
26599993513,9cyclictest7499-21sh12:24:302
2658299351,30cyclictest26751-21apt-get07:10:120
2658299351,21cyclictest26650-21sh09:52:110
2658299351,21cyclictest26650-21sh09:52:110
2658299350,24cyclictest28718-21kthreadcore08:50:220
2658299350,22cyclictest31080-21rm10:13:040
2658299350,21cyclictest12359-21diskmemload12:07:120
26606993413,11cyclictest28366irq/58-eth012:25:133
26599993423,8cyclictest13635-21sed08:00:302
26599993423,7cyclictest24097-21apt-get10:10:152
26599993422,3cyclictest0-21swapper/208:45:112
26599993419,8cyclictest4241-21if_err_eth107:20:172
26599993419,11cyclictest21732-21ntp_192.168.11407:35:232
26599993418,8cyclictest3800-21/usr/sbin/munin07:50:262
26599993418,13cyclictest31339-21/usr/sbin/munin07:45:222
26599993418,10cyclictest0-21swapper/211:40:142
26599993414,7cyclictest25804-21ssh11:20:482
26599993414,7cyclictest25804-21ssh11:20:472
26599993413,7cyclictest0-21swapper/211:00:412
26599993413,16cyclictest0-21swapper/210:40:142
26599993410,12cyclictest0-21swapper/211:59:362
2659099341,6cyclictest131rcu_preempt09:10:151
2658299341,28cyclictest16087-21apt-get08:40:130
2658299341,20cyclictest3278-21ssh10:34:050
2658299340,21cyclictest6253-21ssh11:46:020
2658299340,18cyclictest5043-21ssh09:22:310
2660699332,17cyclictest28366irq/58-eth010:05:123
2659999339,8cyclictest0-21swapper/210:31:222
26599993322,8cyclictest0-21swapper/207:15:112
26599993319,11cyclictest30848-21if_err_eth108:55:172
26599993318,12cyclictest8945-21cut07:25:152
26599993316,8cyclictest30555-21unixbench_multi07:10:282
26599993316,13cyclictest0-21swapper/208:50:132
2658299331,29cyclictest2228-21apt-get09:40:130
2658299330,28cyclictest24291-21apt-get08:15:130
2658299330,28cyclictest24291-21apt-get08:15:120
2658299330,19cyclictest28594-21sh11:24:080
2658299330,19cyclictest28594-21sh11:24:070
2658299330,18cyclictest24946-21ssh11:56:480
2660699323,19cyclictest28366irq/58-eth011:45:133
26606993214,11cyclictest28366irq/58-eth012:10:133
26599993218,11cyclictest13315-21rm10:03:212
26599993213,11cyclictest8473-21cut09:25:192
26599993213,11cyclictest8473-21cut09:25:182
2658299321,28cyclictest26578-21cyclictest07:10:010
2658299320,28cyclictest18338-21apt-get07:35:130
26599993120,9cyclictest22959-21ntp_states08:10:272
26599993118,9cyclictest13902-21cut07:30:172
26599993116,12cyclictest24744-21ssh10:26:422
26599993113,7cyclictest12359-21diskmemload09:12:162
26599993112,8cyclictest22707-21sh10:08:502
2659099313,5cyclictest131rcu_preempt11:20:121
2659099313,5cyclictest131rcu_preempt11:20:111
2659099311,27cyclictest17321-21ssh12:30:131
2658299311,28cyclictest6103-21apt-get10:00:130
2658299311,27cyclictest6442-21diskstats08:30:130
2658299311,27cyclictest0-21swapper/008:05:120
2658299311,26cyclictest1735-21wc08:25:150
2658299311,26cyclictest12740-21/usr/sbin/munin09:10:300
2658299310,27cyclictest4444-21apt-get10:35:130
2660699301,18cyclictest28366irq/58-eth010:55:143
26599993017,10cyclictest1493-21munin-node08:25:112
26599993012,14cyclictest4437-21acpi10:35:102
26590993023,3cyclictest21-21ksoftirqd/109:30:111
2658299301,25cyclictest11509-21grep08:35:170
2660699294,15cyclictest28366irq/58-eth008:55:133
2660699292,14cyclictest0-21swapper/309:55:133
2659099297,19cyclictest2255-21ssh09:40:141
2659099291,25cyclictest9724-21ssh10:20:131
2659099291,13cyclictest13210-21kworker/u8:009:20:141
2658299292,20cyclictest3966-21apt-get07:20:120
26582992915,9cyclictest0-21swapper/009:00:120
2660699288,12cyclictest28366irq/58-eth009:10:143
2660699286,18cyclictest28366irq/58-eth012:35:143
2660699286,18cyclictest28366irq/58-eth012:35:133
2660699284,17cyclictest0-21swapper/311:10:163
2660699283,18cyclictest28366irq/58-eth010:15:113
2659099282,23cyclictest21-21ksoftirqd/112:20:141
2659099281,21cyclictest8349-21ssh12:25:131
2658299281,25cyclictest18713-21ssh10:25:040
2658299281,24cyclictest30746-21fschecks_time08:55:160
2658299281,22cyclictest0-21swapper/008:00:130
2660699272,16cyclictest0-21swapper/310:40:133
2660699271,15cyclictest28366irq/58-eth010:30:123
2658299271,24cyclictest16958-21ntp_kernel_err07:30:230
2658299271,23cyclictest3723-21tr07:50:230
2658299271,23cyclictest28026-21diskstats07:45:150
2658299271,23cyclictest22987-21/usr/sbin/munin08:10:270
2660699266,12cyclictest28366irq/58-eth009:50:133
2660699266,12cyclictest28366irq/58-eth009:50:123
2660699264,15cyclictest0-21swapper/310:00:143
2660699261,17cyclictest0-21swapper/311:35:143
26590992619,5cyclictest12359-21diskmemload09:45:131
2658299262,18cyclictest8752-21sshd07:25:120
2660699253,15cyclictest0-21swapper/309:30:133
2659099255,15cyclictest7538-21ssh09:25:121
2659099255,15cyclictest7538-21ssh09:25:111
2659099251,20cyclictest6130-21ssh10:00:141
2660699242,12cyclictest0-21swapper/311:50:133
2660699241,17cyclictest28366irq/58-eth011:20:133
2660699241,17cyclictest28366irq/58-eth011:20:123
2659099247,15cyclictest4454-21ssh10:35:121
2659099244,15cyclictest8748-21apt-get07:25:121
2659099243,17cyclictest26217-21ssh11:05:111
2659099242,5cyclictest131rcu_preempt10:15:121
2659099242,19cyclictest15344-21ssh11:35:141
2659099242,18cyclictest21751-21ssh09:15:121
26590992416,5cyclictest131rcu_preempt10:55:121
2659099241,19cyclictest73-21kswapd011:15:141
2660699233,11cyclictest0-21swapper/312:20:143
2659099235,15cyclictest20393-21ssh09:50:131
2659099235,15cyclictest20393-21ssh09:50:121
2659099232,14cyclictest26354-21ssh12:35:141
2659099232,14cyclictest26354-21ssh12:35:131
26590992314,6cyclictest7553-21cron09:05:021
26590992314,5cyclictest24426-21ssh11:40:131
2659099231,17cyclictest13389-21ssh10:40:131
2659099231,16cyclictest19493-21find08:10:141
26590992310,10cyclictest21-21ksoftirqd/111:55:121
2660699226,10cyclictest28366irq/58-eth011:40:153
2660699222,11cyclictest0-21swapper/311:15:143
26606992211,8cyclictest0-21swapper/309:20:143
2659099222,16cyclictest31561-21ssh10:50:151
2659099222,14cyclictest6339-21ssh11:30:131
2659099222,13cyclictest0-21swapper/109:35:131
2660699212,13cyclictest0-21swapper/310:35:133
2660699212,13cyclictest0-21swapper/309:15:123
26606992112,6cyclictest28366irq/58-eth012:05:133
2659099216,10cyclictest26578-21cyclictest08:15:141
2659099216,10cyclictest26578-21cyclictest08:15:131
26590992114,4cyclictest22617-21proc_pri10:25:271
2659099210,17cyclictest30734-21cut08:55:151
2659099210,15cyclictest28070-21find12:00:121
2660699203,11cyclictest0-21swapper/312:00:093
2660699202,11cyclictest0-21swapper/308:45:133
2659099209,9cyclictest24012-21kthreadcore10:45:211
2659099208,9cyclictest6403-21ls08:30:111
2659099205,11cyclictest21-21ksoftirqd/111:29:091
26590992011,7cyclictest0-21swapper/112:15:121
2659099201,16cyclictest16410-21grep08:40:171
2659099201,15cyclictest11306-21cut08:35:161
26590992010,8cyclictest1511-21apt-get08:25:131
2659099200,15cyclictest11287-21latency_hist11:15:021
2659099200,14cyclictest15272-21cut10:05:161
2660699192,8cyclictest0-21swapper/309:45:133
2660699192,12cyclictest0-21swapper/310:25:133
2660699192,10cyclictest0-21swapper/308:50:133
2660699192,10cyclictest0-21swapper/307:10:143
2660699191,13cyclictest0-21swapper/308:35:173
2660699191,12cyclictest0-21swapper/307:45:013
2660699191,11cyclictest28366irq/58-eth007:25:133
26606991910,6cyclictest28366irq/58-eth010:20:143
26606991910,6cyclictest28366irq/58-eth009:40:133
26606991910,6cyclictest0-21swapper/310:50:183
2659099197,3cyclictest21-21ksoftirqd/110:35:011
2659099194,13cyclictest21-21ksoftirqd/110:10:131
2659099193,13cyclictest8459-21kthreadcore09:05:201
26590991912,5cyclictest0-21swapper/108:05:131
2659099191,15cyclictest28172-21ls07:45:161
2660699182,11cyclictest0-21swapper/311:30:083
2660699182,10cyclictest0-21swapper/309:05:143
2660699182,10cyclictest0-21swapper/308:15:153
2660699182,10cyclictest0-21swapper/308:15:143
2660699182,10cyclictest0-21swapper/307:35:143
26606991810,5cyclictest28366irq/58-eth011:25:283
2659099189,5cyclictest14412-21kthreadcore12:10:201
2659099187,8cyclictest21-21ksoftirqd/107:10:121
2659099186,9cyclictest17388-21grep11:00:171
2659099186,9cyclictest13484-21cat12:10:021
2659099181,14cyclictest5032-21unixbench_multi11:45:291
2659099181,13cyclictest953-21ssh09:55:281
26590991810,6cyclictest1-21systemd07:35:071
2660699179,6cyclictest0-21swapper/310:12:203
2660699172,9cyclictest0-21swapper/307:15:143
2660699172,10cyclictest0-21swapper/308:40:133
2660699172,10cyclictest0-21swapper/308:00:153
2660699172,10cyclictest0-21swapper/307:45:143
2660699172,10cyclictest0-21swapper/307:30:123
2660699171,13cyclictest0-21swapper/307:55:273
2659099178,7cyclictest21-21ksoftirqd/108:50:141
2659099175,9cyclictest29288-21cut08:20:141
2659099175,10cyclictest5688-21irqrtprio07:55:161
2659099174,3cyclictest131rcu_preempt07:40:191
2659099171,14cyclictest15016-21kthreadcore07:30:191
2660699168,6cyclictest0-21swapper/311:05:103
2660699163,6cyclictest0-21swapper/308:10:133
2660699161,12cyclictest0-21swapper/308:30:253
2659099164,9cyclictest23089-21kthreadcore08:45:201
2659099162,11cyclictest31857-21grep07:15:171
2659099162,11cyclictest2107-21kthreadcore07:50:201
2659099162,11cyclictest13076-21kthreadcore08:00:241
2660699152,6cyclictest0-21swapper/308:20:213
2659099153,9cyclictest26651-21cat07:10:021
2660699146,5cyclictest28366irq/58-eth007:20:143
2660699142,10cyclictest0-21swapper/308:05:123
2660699133,7cyclictest0-21swapper/309:00:243
2659999131,10cyclictest26604-21latency_hist07:10:022
2660699121,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional