You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 19:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Jan 07, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23877997345,20cyclictest28366irq/58-eth009:45:113
2384299695,37cyclictest495-21ssh10:55:150
23865994836,9cyclictest0-21swapper/210:50:142
2384299483,39cyclictest24932-21ssh09:55:140
2384299441,35cyclictest28363-21ssh12:25:130
2386599411,36cyclictest26795-21sh11:10:142
2387799403,23cyclictest28366irq/58-eth011:00:143
2386599404,32cyclictest4707-21ssh12:30:112
2386599402,18cyclictest261rcuc/210:10:132
2384299402,31cyclictest21327-21ssh10:30:130
2386599383,23cyclictest5668-21ssh11:35:102
2385499385,25cyclictest4727-21ssh12:30:131
2385499381,34cyclictest23231-21ssh11:45:141
2385499373,27cyclictest15252-21ssh10:45:141
23854993715,4cyclictest21-21ksoftirqd/110:15:141
2384299372,26cyclictest27407-21ssh10:15:130
2384299371,27cyclictest2123-21ssh12:10:130
2387799361,21cyclictest28366irq/58-eth008:50:133
2385499363,31cyclictest24100-21ssh10:50:141
2386599337,16cyclictest27-21ksoftirqd/211:15:122
2385499331,28cyclictest30845-21sh09:40:131
2384299332,24cyclictest16005-21ssh09:50:120
2384299332,24cyclictest16005-21ssh09:50:120
2387799326,16cyclictest0-21swapper/310:05:163
23877993217,5cyclictest0-21swapper/309:40:123
23865993213,17cyclictest20360-21ssh10:29:172
2387799312,17cyclictest28366irq/58-eth009:35:133
2386599312,20cyclictest22346irq/40-dwc2_hso11:45:122
2384299312,21cyclictest20796-21ssh11:25:140
2386599302,20cyclictest0-21swapper/210:45:142
2384299302,21cyclictest17070-21ssh12:00:140
2384299302,20cyclictest27417-21ssh09:36:100
2387799299,9cyclictest0-21swapper/307:55:133
23877992915,10cyclictest0-21swapper/312:35:133
2386599294,16cyclictest22346irq/40-dwc2_hso12:05:112
23865992914,12cyclictest27-21ksoftirqd/209:40:202
2385499295,21cyclictest7178-21ssh09:45:121
2385499293,21cyclictest201rcuc/111:15:121
2385499292,20cyclictest16569-21apt-get12:00:041
2387799284,12cyclictest28366irq/58-eth010:30:153
2387799281,15cyclictest0-21swapper/312:00:123
2386599283,16cyclictest22346irq/40-dwc2_hso12:15:132
2386599282,19cyclictest0-21swapper/210:40:142
2384299282,20cyclictest7156-21ssh09:45:100
23842992812,12cyclictest0-21swapper/010:05:160
2387799273,16cyclictest0-21swapper/309:34:083
2387799272,18cyclictest0-21swapper/310:23:103
2387799272,14cyclictest28366irq/58-eth011:15:113
2387799271,14cyclictest28366irq/58-eth010:45:143
2386599274,15cyclictest22346irq/40-dwc2_hso08:40:152
2385499271,18cyclictest10874-21ssh12:15:121
2384299272,19cyclictest23010-21sh10:49:100
23842992710,7cyclictest12-21ksoftirqd/010:50:130
2387799268,12cyclictest28366irq/58-eth009:25:153
2387799261,17cyclictest28366irq/58-eth011:10:153
23877992610,10cyclictest28366irq/58-eth011:35:123
2386599261,7cyclictest1093566ssh09:27:352
23865992615,8cyclictest27-21ksoftirqd/212:25:122
2387799253,13cyclictest28366irq/58-eth010:50:133
2387799252,18cyclictest28366irq/58-eth011:50:143
2387799252,18cyclictest28366irq/58-eth011:50:133
2387799252,16cyclictest28366irq/58-eth011:40:103
2387799251,16cyclictest0-21swapper/309:10:143
2386599252,15cyclictest22346irq/40-dwc2_hso12:35:132
2386599251,8cyclictest365566ssh10:36:562
23865992517,5cyclictest4919-21ssh12:10:212
23865992513,8cyclictest27-21ksoftirqd/209:35:132
23865992513,5cyclictest21407-21ssh09:51:082
23865992513,5cyclictest21407-21ssh09:51:082
2386599251,13cyclictest1325-21ssh10:00:132
2386599251,12cyclictest11637-21latency11:55:242
2385499258,4cyclictest131rcu_preempt10:00:231
23854992511,9cyclictest633-21Async12:11:531
2387799244,12cyclictest28366irq/58-eth011:25:123
2386599247,14cyclictest973-21cron07:20:012
2386599241,8cyclictest1958266ssh12:20:112
2386599241,7cyclictest2755566ssh10:32:152
2386599241,7cyclictest2102366ssh11:42:422
23865992413,9cyclictest24802-21ssh11:07:552
2385499241,6cyclictest10075-21munin-run09:10:021
2384299241,18cyclictest19147-21ssh09:15:140
2387799233,10cyclictest0-21swapper/309:15:143
23877992316,5cyclictest28366irq/58-eth009:55:153
2387799231,15cyclictest28366irq/58-eth009:00:113
2386599232,16cyclictest25105-21tune2fs09:55:172
2386599232,11cyclictest7179-21sh09:45:122
2386599231,6cyclictest673366ssh11:53:332
2386599231,6cyclictest673366ssh11:53:332
2386599231,6cyclictest1863466ssh09:31:082
2386599231,6cyclictest1531866ssh10:06:062
23865992313,8cyclictest0-21swapper/209:05:282
2386599231,14cyclictest2576566ssh11:25:422
2385499231,17cyclictest8195-21ssh11:55:141
2385499231,16cyclictest13584-21rm12:35:141
2387799227,10cyclictest28366irq/58-eth010:25:153
2387799223,9cyclictest28366irq/58-eth007:25:123
2386599228,11cyclictest19290-21sh09:15:172
2386599224,10cyclictest15573-21netstat11:20:252
2386599222,11cyclictest5534-21kthreadcore10:20:212
2386599221,5cyclictest575566ssh10:55:582
2386599221,10cyclictest30622-21kthreadcore10:15:222
2385499224,2cyclictest131rcu_preempt12:05:281
2385499221,3cyclictest131rcu_preempt11:10:181
2385499221,13cyclictest1097-21ls10:00:021
2385499221,11cyclictest0-21swapper/112:20:221
23854992210,3cyclictest131rcu_preempt11:40:171
23854992210,3cyclictest131rcu_preempt10:30:291
2384299229,3cyclictest131rcu_preempt10:00:310
2384299222,2cyclictest131rcu_preempt11:08:440
23842992215,4cyclictest16106-21unixbench_singl11:20:310
23842992211,8cyclictest3749-21ssh10:20:130
2384299221,17cyclictest30859-21ssh09:40:140
2384299221,16cyclictest19593-21ssh12:20:130
2384299220,17cyclictest2694-21find07:55:120
2387799219,8cyclictest0-21swapper/311:55:143
2387799219,8cyclictest0-21swapper/311:45:143
2387799218,8cyclictest0-21swapper/309:50:133
2387799218,8cyclictest0-21swapper/309:50:133
2387799215,11cyclictest0-21swapper/310:15:273
2386599219,4cyclictest10008-21/usr/sbin/munin11:00:192
2386599216,12cyclictest42-21kauditd07:40:012
2386599212,16cyclictest6435-21timerandwakeup07:55:292
2386599211,9cyclictest27651-21switchtime07:10:292
2386599211,9cyclictest17898-21kthreadcore12:00:202
2386599211,6cyclictest88166lldpd07:25:022
2386599211,16cyclictest32268-21kthreadcore07:50:212
2386599211,16cyclictest12939-21kthreadcore07:30:212
2386599211,11cyclictest13835-21tr09:10:242
23865992110,9cyclictest27934-21sshd09:20:102
2385499217,3cyclictest131rcu_preempt11:25:021
2385499216,3cyclictest131rcu_preempt09:35:221
2385499212,14cyclictest6428-21ssh10:40:121
23854992110,8cyclictest17835-21cat09:30:301
23854992110,8cyclictest13826-21cat11:00:311
23854992110,8cyclictest0-21swapper/109:00:131
2384299212,14cyclictest4392-21ssh09:25:150
23842992112,6cyclictest1752-21cron12:10:010
2384299211,17cyclictest3282-21ssh11:15:150
2384299211,14cyclictest15639-21find07:35:130
23842992110,8cyclictest4781-21cpuspeed12:30:130
2384299210,18cyclictest10182-21ls09:10:020
2387799208,8cyclictest28366irq/58-eth012:10:133
2387799203,11cyclictest0-21swapper/310:35:103
2387799202,13cyclictest0-21swapper/311:20:253
23877992013,4cyclictest0-21swapper/312:05:153
23877992011,7cyclictest0-21swapper/311:05:303
23877992011,6cyclictest33-21ksoftirqd/310:40:123
2386599202,10cyclictest22245-21fschecks_time08:15:182
2386599202,10cyclictest12246-21grep08:05:022
2386599201,9cyclictest28932-21proc_pri07:45:242
2386599201,8cyclictest27230-21irqcore08:20:182
2386599201,8cyclictest25311-21kthreadcore08:50:202
2386599201,8cyclictest17362-21fschecks_time08:10:162
2386599201,16cyclictest24316-21tail07:40:302
2386599201,10cyclictest8752-21latency_hist08:35:012
2386599201,10cyclictest14539-21kthreadcore08:05:202
2386599201,10cyclictest1213-21find07:20:112
2386599200,17cyclictest11624-21kthreadcore08:35:212
2385499203,14cyclictest23828-21cyclictest11:35:111
2385499202,13cyclictest30004-21apt-get07:50:031
2385499201,3cyclictest131rcu_preempt07:10:141
2385499201,11cyclictest11303-21cut07:30:191
23854992010,3cyclictest10072-21diskmemload12:11:281
2385499200,7cyclictest29117-21kthreadcore12:25:191
2385499200,14cyclictest10652-21ssh10:23:101
2384299208,3cyclictest131rcu_preempt11:35:290
2384299202,15cyclictest26937-21ntp_states11:45:270
2384299201,8cyclictest3157-21ntp_192.168.11411:50:250
2384299201,8cyclictest3157-21ntp_192.168.11411:50:250
23842992011,6cyclictest16545-21kthreadcore09:30:220
2384299201,14cyclictest29484-21sh11:30:130
2387799199,6cyclictest0-21swapper/312:23:573
2387799199,3cyclictest0-21swapper/310:00:373
2387799198,6cyclictest0-21swapper/310:59:213
2387799192,11cyclictest0-21swapper/307:10:153
2387799192,10cyclictest0-21swapper/307:45:123
2387799191,13cyclictest0-21swapper/308:20:013
2387799191,12cyclictest0-21swapper/308:10:013
2387799191,11cyclictest0-21swapper/307:30:243
2386599192,15cyclictest20173-21kthreadcore08:45:192
2386599191,9cyclictest29618-21cut11:30:152
2386599191,14cyclictest31799-21cpuspeed08:25:152
2386599191,10cyclictest30400-21kthreadcore08:55:202
23854991913,4cyclictest14510-21kthreadcore10:25:231
23854991913,4cyclictest0-21swapper/111:25:251
23854991912,4cyclictest460-21/usr/sbin/munin10:55:301
23854991911,6cyclictest23193-21cron08:50:011
23854991911,5cyclictest0-21swapper/108:55:191
2384299198,8cyclictest10611-21uptime10:40:300
2384299197,9cyclictest19856-21sshd11:41:110
2384299193,11cyclictest0-21swapper/011:02:110
2384299192,12cyclictest17900-21ssh10:26:110
23842991912,4cyclictest15359-21usage_EthcTask12:15:310
2384299191,15cyclictest14344-21awk09:10:290
2384299191,13cyclictest8891-21irqstats11:55:190
2384299190,3cyclictest0-21swapper/009:24:020
2387799188,2cyclictest0-21swapper/309:05:133
2387799182,11cyclictest0-21swapper/309:24:063
2387799182,10cyclictest0-21swapper/310:10:143
2387799182,10cyclictest0-21swapper/308:10:143
2387799182,10cyclictest0-21swapper/307:50:043
2387799181,13cyclictest28366irq/58-eth012:30:123
2387799181,13cyclictest0-21swapper/308:35:283
2387799181,11cyclictest28366irq/58-eth011:30:123
2386599181,8cyclictest795-21fschecks_count09:00:162
2385499189,7cyclictest4240-21df08:30:141
2385499183,3cyclictest10072-21diskmemload09:11:251
2385499182,3cyclictest131rcu_preempt10:35:211
2385499181,9cyclictest18921-21kthreadcore08:10:201
2385499181,9cyclictest0-21swapper/109:50:111
2385499181,9cyclictest0-21swapper/109:50:111
2385499181,8cyclictest10235-21kthreadcore08:00:221
2385499181,3cyclictest131rcu_preempt10:05:151
2385499181,3cyclictest131rcu_preempt09:20:301
2385499181,14cyclictest5930-21grep09:25:211
2385499181,12cyclictest18755-21ssh11:05:181
2385499180,10cyclictest0-21swapper/111:51:321
2385499180,10cyclictest0-21swapper/111:51:321
2384299186,4cyclictest27195-21irqcore11:10:190
2384299182,11cyclictest0-21swapper/008:35:140
2384299182,11cyclictest0-21swapper/008:30:130
2384299182,11cyclictest0-21swapper/008:15:160
23842991811,5cyclictest0-21swapper/007:50:230
2384299181,14cyclictest6048-21munin-run10:40:010
2384299181,13cyclictest18776-21diskstats10:10:150
2384299181,10cyclictest25476-21ls07:45:130
2387799179,6cyclictest0-21swapper/312:26:533
2387799179,6cyclictest0-21swapper/312:18:023
2387799173,11cyclictest28366irq/58-eth008:45:113
2387799172,10cyclictest0-21swapper/308:20:153
2386599171,13cyclictest23828-21cyclictest07:10:022
2385499179,6cyclictest25603-21perl07:45:141
2385499171,6cyclictest0-21swapper/109:15:461
23854991711,4cyclictest2923-21hwlatdetect07:55:161
2385499171,13cyclictest31413-21kthreadcore11:30:211
2385499171,13cyclictest27433-21sed08:20:191
2385499171,10cyclictest2624-21latency08:25:231
23854991710,4cyclictest9204-21cut08:35:161
2384299179,5cyclictest0-21swapper/007:40:190
2384299172,11cyclictest0-21swapper/008:10:130
2384299172,10cyclictest0-21swapper/008:45:130
2384299172,10cyclictest0-21swapper/008:20:150
2384299171,13cyclictest7066-21kthreadcore07:25:200
2387799162,9cyclictest0-21swapper/307:15:133
2387799161,10cyclictest0-21swapper/307:35:183
2385499168,6cyclictest25392-21memory08:15:241
2385499167,7cyclictest15748-21diskstats07:35:171
2385499163,10cyclictest26978-21cat08:50:261
2385499162,10cyclictest0-21swapper/108:40:151
2385499161,12cyclictest23969-21ntp_offset07:40:261
2385499161,12cyclictest17076-21grep08:10:011
2385499161,11cyclictest1222-21apt-get07:20:131
2384299168,6cyclictest15415-21cron07:35:010
2384299163,10cyclictest13557-21ssh12:35:110
2384299162,11cyclictest2614-21kthreadcore09:00:200
2384299162,10cyclictest0-21swapper/008:05:130
2384299162,10cyclictest0-21swapper/007:20:130
2384299162,10cyclictest0-21swapper/007:15:140
2384299161,7cyclictest23969-21/usr/sbin/munin07:10:180
2384299161,7cyclictest14642-21kthreadcore08:40:200
2384299161,12cyclictest9791-21kthreadcore08:00:210
23842991610,4cyclictest0-21swapper/008:25:190
2387799152,6cyclictest28366irq/58-eth008:40:133
2387799151,9cyclictest0-21swapper/308:25:173
2385499157,6cyclictest6270-21sort07:25:171
2385499151,11cyclictest28790-21apt-get07:15:131
2384299153,7cyclictest0-21swapper/008:50:140
2384299151,11cyclictest29032-21kthreadcore08:55:200
2387799141,6cyclictest0-21swapper/308:00:183
2387799141,11cyclictest0-21swapper/307:40:223
2385499141,10cyclictest23932-21cat07:10:011
2384299141,10cyclictest23914-21latency_hist07:10:020
2387799130,11cyclictest33-21ksoftirqd/308:55:283
2387799122,6cyclictest0-21swapper/307:20:133
2387799121,8cyclictest0-21swapper/308:30:263
2387799101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional