You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-21 - 23:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Jun 21, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27802996036,15cyclictest35966irq/58-eth009:20:113
27802996029,15cyclictest35966irq/58-eth009:10:123
2771599591,10cyclictest12-21ksoftirqd/012:25:120
2771599583,40cyclictest2022-21ssh10:50:110
2771599491,41cyclictest29459-21ssh11:05:140
27740994524,18cyclictest13484-21diskmemload11:30:111
27740994436,5cyclictest0-21swapper/111:35:121
27740994435,6cyclictest27253-21apt-get12:15:121
2771599443,6cyclictest12-21ksoftirqd/011:15:130
2771599436,9cyclictest12-21ksoftirqd/011:25:130
2771599435,4cyclictest131rcu_preempt08:30:120
2771599432,33cyclictest26807-21sh09:35:110
2771599431,32cyclictest24323-21ssh11:20:130
27740994228,9cyclictest12707-21apt-get09:45:141
27740994216,8cyclictest2456-21ssh09:55:251
27740994210,16cyclictest14414-21cut08:35:211
27740994135,3cyclictest9122-21apt-get12:05:141
27740994134,3cyclictest5071-21apt-get11:45:111
27740994117,11cyclictest19894-21ntp_kernel_pll_10:40:251
27740994115,11cyclictest5611-21netstat10:50:221
27740994115,10cyclictest30806-21ssh11:58:331
2771599415,6cyclictest131rcu_preempt12:00:130
2771599413,35cyclictest8674-21ssh09:25:130
2771599411,3cyclictest131rcu_preempt11:00:120
2771599411,34cyclictest12725-21ssh09:45:140
2780299401,24cyclictest35966irq/58-eth010:15:133
2780299401,21cyclictest35966irq/58-eth011:50:133
2774099403,29cyclictest22940-21ssh09:15:111
27740994030,6cyclictest18211-21apt-get12:10:121
27740994028,8cyclictest30250-21apt-get10:30:131
27740994025,9cyclictest13028-21apt-get12:25:131
2774099402,32cyclictest7007-21ssh10:35:131
27740994018,17cyclictest0-21swapper/112:35:121
27740994017,9cyclictest31788-21/usr/sbin/munin08:55:151
27740994016,10cyclictest30494-21ssh09:54:501
27740994015,22cyclictest0-21swapper/108:05:131
27740994014,10cyclictest16970-21ssh10:05:141
27740994013,11cyclictest10570-21ssh09:42:591
2771599407,7cyclictest131rcu_preempt09:05:150
2780299392,27cyclictest35966irq/58-eth011:25:133
27740993917,10cyclictest18736-21ssh09:10:341
27740993916,10cyclictest28412-21irqrtprio07:10:181
27740993914,9cyclictest18024-21cut11:15:221
27740993914,10cyclictest16499-21ssh10:04:591
27740993913,9cyclictest10980-21users11:10:311
27740993913,11cyclictest21978-21latency_hist08:45:011
27740993913,10cyclictest9449-21kthreadcore09:25:171
27740993912,12cyclictest13314-21ntp_192.168.11407:25:231
27740993912,11cyclictest28233-21cut08:50:191
2780299386,26cyclictest35966irq/58-eth012:20:133
2780299382,20cyclictest0-21swapper/312:35:113
27740993826,9cyclictest5101-21apt-get07:20:121
27740993817,10cyclictest31077-21ssh10:46:311
27740993815,15cyclictest7878-21/usr/sbin/munin08:30:161
27740993815,14cyclictest7144-21kthreadcore09:00:231
27740993814,8cyclictest28135-21mailstats11:20:271
27740993814,8cyclictest17758-21load11:50:231
27740993814,8cyclictest11570-21rm12:23:461
27740993814,10cyclictest18690-21ssh10:58:361
27740993812,9cyclictest5824-21cat11:25:281
27740993812,10cyclictest1601-21ssh10:14:091
2777499372,28cyclictest18731-21cut10:05:232
27774993711,20cyclictest26955-21ntp_kernel_pll_11:55:232
27740993721,13cyclictest6320-21apt-get07:55:111
27740993714,8cyclictest4852-21latency_hist07:20:011
27740993714,8cyclictest27761-21ntp_192.168.11407:40:231
27740993714,8cyclictest27761-21ntp_192.168.11407:40:221
27740993714,10cyclictest24364-21sh11:00:291
27740993713,8cyclictest23450-21cut12:30:201
27740993713,10cyclictest4445-21cat12:00:301
27740993712,9cyclictest23153-21ssh09:31:101
27740993712,9cyclictest23153-21ssh09:31:091
27740993712,9cyclictest14585-21ntp_192.168.11408:00:251
27740993712,11cyclictest1973-21ssh09:38:401
27740993711,10cyclictest3498-21/usr/sbin/munin08:25:181
2771599371,23cyclictest9120-21ssh12:05:130
27740993616,7cyclictest13484-21diskmemload11:43:151
27740993614,9cyclictest19616-21entropy07:35:151
27740993613,8cyclictest735-21ntp_states11:05:261
27740993612,11cyclictest32150-21cut07:45:201
27740993611,8cyclictest9065-21ssh10:17:071
27740993610,10cyclictest13009-21usage_Anybus09:05:301
2780299356,19cyclictest35966irq/58-eth009:50:133
2780299353,22cyclictest35966irq/58-eth010:05:153
2780299352,22cyclictest35966irq/58-eth012:05:133
27740993523,8cyclictest21069-21apt-get10:25:121
27740993523,8cyclictest21069-21apt-get10:25:121
27740993519,13cyclictest0-21swapper/108:20:131
27740993514,15cyclictest0-21swapper/110:20:131
27740993513,14cyclictest7684-21sh09:24:211
2777499342,16cyclictest11231-21sh10:19:392
27740993413,8cyclictest21441-21kthreadcore08:10:191
2771599347,3cyclictest131rcu_preempt10:15:140
27740993311,7cyclictest18123-21ntp_192.168.11407:30:251
2771599339,3cyclictest131rcu_preempt08:45:110
2771599334,5cyclictest131rcu_preempt08:40:120
2780299324,17cyclictest35966irq/58-eth010:50:123
2780299324,17cyclictest35966irq/58-eth009:45:133
2780299323,17cyclictest35966irq/58-eth010:45:143
2777499321,19cyclictest2486-21ssh12:35:242
2771599322,22cyclictest18698-21ssh10:05:230
2780299314,13cyclictest35966irq/58-eth010:10:133
2777499318,13cyclictest27-21ksoftirqd/210:40:022
2777499315,2cyclictest131rcu_preempt09:35:112
27774993110,13cyclictest27-21ksoftirqd/211:38:112
27740993119,9cyclictest29463-21uptime08:15:291
2771599312,22cyclictest26967-21ntp_kernel_pll_11:55:220
2771599311,3cyclictest131rcu_preempt09:50:130
2780299303,21cyclictest0-21swapper/310:00:143
2777499309,10cyclictest27-21ksoftirqd/211:25:272
2777499303,19cyclictest0-21swapper/212:00:072
2777499301,17cyclictest27346irq/40-dwc2_hso08:50:112
27774993011,7cyclictest27-21ksoftirqd/210:10:142
27740993019,7cyclictest1604-21diskstats07:50:171
27740993017,9cyclictest25735-21switchtime08:45:271
2771599301,24cyclictest26099-21rm10:10:140
27802992913,10cyclictest35966irq/58-eth011:05:143
2777499297,16cyclictest261rcuc/209:55:172
2777499292,20cyclictest0-21swapper/209:40:142
2780299284,16cyclictest35966irq/58-eth011:35:123
2780299283,16cyclictest0-21swapper/309:30:233
2780299283,16cyclictest0-21swapper/309:30:233
2777499289,11cyclictest27-21ksoftirqd/211:49:382
2777499288,11cyclictest27-21ksoftirqd/212:19:162
2777499283,19cyclictest27695-21cyclictest10:04:372
2777499283,18cyclictest0-21swapper/212:20:102
2777499282,21cyclictest0-21swapper/211:35:012
27774992811,11cyclictest27-21ksoftirqd/211:50:232
27740992818,7cyclictest27818-21grep07:10:011
27740992818,7cyclictest27818-21grep07:10:011
2771599283,7cyclictest12-21ksoftirqd/007:30:120
2771599282,19cyclictest3905-21sh12:20:100
2780299273,16cyclictest0-21swapper/311:55:233
2780299272,14cyclictest0-21swapper/311:20:133
2777499279,3cyclictest131rcu_preempt09:15:162
2777499278,8cyclictest27-21ksoftirqd/211:13:242
2777499278,10cyclictest131rcu_preempt08:00:252
2777499275,13cyclictest27-21ksoftirqd/211:42:492
2777499272,20cyclictest0-21swapper/209:05:152
27774992712,4cyclictest27-21ksoftirqd/211:17:262
2777499271,14cyclictest13055-21ssh12:25:142
2771599272,6cyclictest131rcu_preempt11:40:100
2780299269,9cyclictest35966irq/58-eth012:15:133
2780299264,15cyclictest0-21swapper/307:20:123
2780299264,11cyclictest35966irq/58-eth009:55:123
2780299263,16cyclictest0-21swapper/310:35:243
2777499269,11cyclictest27-21ksoftirqd/210:29:502
2777499269,11cyclictest27-21ksoftirqd/210:29:502
2777499266,4cyclictest27-21ksoftirqd/212:30:312
2777499261,16cyclictest32045-21ssh09:20:122
2777499261,14cyclictest24224-21ls11:00:282
2771599263,3cyclictest131rcu_preempt10:25:120
2771599263,3cyclictest131rcu_preempt10:25:120
2771599262,18cyclictest28855-21memory10:45:230
2771599261,19cyclictest13484-21diskmemload10:40:130
27715992610,7cyclictest0-21swapper/011:48:160
2780299252,17cyclictest0-21swapper/310:30:233
2777499258,11cyclictest27-21ksoftirqd/210:42:012
2777499258,10cyclictest131rcu_preempt09:52:512
2777499251,11cyclictest18223-21ssh12:10:122
2777499251,11cyclictest131rcu_preempt09:25:202
27774992510,13cyclictest0-21swapper/210:22:132
2771599256,14cyclictest131rcu_preempt09:40:120
2771599251,19cyclictest14896-21ssh10:55:260
27715992511,4cyclictest12-21ksoftirqd/011:32:530
2780299248,11cyclictest35966irq/58-eth011:45:123
2780299243,14cyclictest0-21swapper/310:59:233
2780299243,13cyclictest35966irq/58-eth008:30:133
2777499245,6cyclictest131rcu_preempt09:33:252
2777499245,6cyclictest131rcu_preempt09:33:252
2777499245,15cyclictest131rcu_preempt10:55:242
2777499244,4cyclictest27-21ksoftirqd/211:05:312
2777499244,3cyclictest131rcu_preempt10:30:222
2777499243,8cyclictest27-21ksoftirqd/209:45:122
2777499242,5cyclictest14625-21grep09:10:182
2777499242,11cyclictest17205-21latency_hist08:40:022
27774992419,2cyclictest0-21swapper/207:10:132
27774992416,6cyclictest0-21swapper/208:05:132
27774992412,5cyclictest24241-21ssh11:20:092
2771599248,7cyclictest12-21ksoftirqd/009:14:010
2771599248,4cyclictest131rcu_preempt08:55:120
2771599247,3cyclictest131rcu_preempt08:15:130
2771599241,8cyclictest0-21swapper/009:55:180
27715992411,4cyclictest12-21ksoftirqd/007:10:130
27715992411,3cyclictest12-21ksoftirqd/012:15:270
27715992410,7cyclictest0-21swapper/010:38:040
2780299233,13cyclictest0-21swapper/312:32:243
27802992311,8cyclictest35966irq/58-eth011:40:113
2780299230,16cyclictest35966irq/58-eth012:00:133
2777499233,4cyclictest271ksoftirqd/207:45:162
2777499231,3cyclictest0-21swapper/210:50:172
27774992311,10cyclictest16303-21ssh12:08:132
2771599239,3cyclictest12-21ksoftirqd/010:01:180
2771599237,10cyclictest0-21swapper/011:54:570
2771599231,3cyclictest131rcu_preempt12:35:120
2771599231,3cyclictest131rcu_preempt09:20:130
27715992312,3cyclictest12-21ksoftirqd/009:30:030
27715992312,3cyclictest12-21ksoftirqd/009:30:030
27715992310,7cyclictest0-21swapper/012:13:050
27715992310,3cyclictest12-21ksoftirqd/012:31:190
2780299223,13cyclictest0-21swapper/311:35:013
2780299222,14cyclictest35966irq/58-eth009:35:113
2780299222,13cyclictest0-21swapper/311:10:133
27802992216,4cyclictest33-21ksoftirqd/309:40:133
2777499228,8cyclictest27-21ksoftirqd/208:40:242
2777499227,2cyclictest131rcu_preempt10:45:072
2777499227,2cyclictest131rcu_preempt07:20:132
2777499224,12cyclictest27-21ksoftirqd/207:30:152
2777499224,10cyclictest131rcu_preempt07:25:232
2777499223,7cyclictest27-21ksoftirqd/208:20:152
2777499222,13cyclictest27-21ksoftirqd/207:55:012
2777499222,11cyclictest131rcu_preempt08:15:292
2771599223,9cyclictest12-21ksoftirqd/010:30:180
2771599222,16cyclictest22929-21apt-get09:15:100
2771599221,15cyclictest15534-21ssh10:20:230
27715992211,3cyclictest12-21ksoftirqd/008:20:200
27715992210,3cyclictest12-21ksoftirqd/011:10:130
2780299211,10cyclictest35966irq/58-eth009:15:113
2777499218,5cyclictest27-21ksoftirqd/207:15:222
2777499214,6cyclictest27-21ksoftirqd/208:25:172
2777499214,3cyclictest131rcu_preempt08:30:242
2777499212,10cyclictest31820-21fschecks_count08:55:152
2777499211,9cyclictest7982-21tail09:00:292
2777499211,11cyclictest27486-21kthreadcore07:40:212
2777499211,11cyclictest27486-21kthreadcore07:40:212
2771599218,4cyclictest12-21ksoftirqd/007:50:210
2771599217,3cyclictest12-21ksoftirqd/011:35:230
27715992110,2cyclictest12-21ksoftirqd/008:00:260
2780299203,12cyclictest0-21swapper/312:25:143
2780299202,14cyclictest0-21swapper/310:20:133
2780299202,12cyclictest0-21swapper/310:25:123
2780299202,12cyclictest0-21swapper/310:25:123
27802992010,3cyclictest0-21swapper/309:27:543
2777499201,9cyclictest23068-21kthreadcore08:45:192
2777499201,15cyclictest22317-21kthreadcore08:10:192
2771599209,3cyclictest12-21ksoftirqd/007:45:200
2771599208,3cyclictest12-21ksoftirqd/008:25:170
2771599207,4cyclictest12-21ksoftirqd/007:40:270
2771599207,4cyclictest12-21ksoftirqd/007:40:270
27715992014,3cyclictest12-21ksoftirqd/007:30:020
2771599200,15cyclictest21387-21grep07:35:200
2780299193,11cyclictest0-21swapper/311:19:233
2780299193,11cyclictest0-21swapper/311:04:523
2780299192,7cyclictest0-21swapper/312:15:013
2780299192,11cyclictest0-21swapper/309:00:153
2780299191,13cyclictest0-21swapper/310:40:133
2780299191,11cyclictest0-21swapper/307:25:133
2777499191,14cyclictest19512-21find07:35:122
2777499191,10cyclictest8181-21kthreadcore07:55:182
2771599199,2cyclictest12-21ksoftirqd/008:35:130
2771599196,9cyclictest12-21ksoftirqd/008:50:130
2771599196,3cyclictest12-21ksoftirqd/008:05:130
2771599195,3cyclictest131rcu_preempt09:00:130
27715991913,3cyclictest12-21ksoftirqd/007:20:200
2780299182,10cyclictest0-21swapper/309:05:153
2780299182,10cyclictest0-21swapper/308:20:063
2780299182,10cyclictest0-21swapper/308:15:143
2780299182,10cyclictest0-21swapper/307:30:143
2780299181,13cyclictest0-21swapper/308:45:273
2780299181,13cyclictest0-21swapper/307:55:263
2777499181,7cyclictest27865-21latency_hist07:10:012
2777499181,7cyclictest27865-21latency_hist07:10:012
27715991811,4cyclictest12-21ksoftirqd/007:10:010
27715991811,4cyclictest12-21ksoftirqd/007:10:010
2771599181,11cyclictest0-21swapper/007:15:160
27715991810,5cyclictest10896-21cron08:00:010
2780299172,10cyclictest0-21swapper/308:10:133
2780299172,10cyclictest0-21swapper/307:40:133
2780299172,10cyclictest0-21swapper/307:40:133
2780299171,13cyclictest0-21swapper/308:05:173
2780299171,10cyclictest0-21swapper/308:55:273
2780299171,10cyclictest0-21swapper/307:10:283
2780299162,10cyclictest0-21swapper/307:45:123
2780299161,8cyclictest0-21swapper/308:45:013
2780299161,8cyclictest0-21swapper/308:05:023
2780299161,10cyclictest0-21swapper/308:50:193
2771599161,8cyclictest12-21ksoftirqd/008:10:150
2780299142,7cyclictest0-21swapper/308:35:143
2780299141,8cyclictest0-21swapper/307:50:143
2780299141,7cyclictest0-21swapper/307:35:283
2780299141,7cyclictest0-21swapper/307:15:293
2780299141,10cyclictest0-21swapper/308:25:223
2780299111,7cyclictest0-21swapper/307:10:013
2780299111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional