You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-23 - 01:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Jul 22, 2026 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32654995536,15cyclictest35966irq/58-eth011:05:133
3265499474,29cyclictest35966irq/58-eth009:25:153
32654994614,21cyclictest35966irq/58-eth009:45:133
3263799450,37cyclictest22917-21ssh09:30:131
32646994336,2cyclictest261rcuc/211:20:122
3263799432,36cyclictest15260-21ssh12:25:121
32629994222,13cyclictest131rcu_preempt08:05:130
3265499413,24cyclictest35966irq/58-eth007:30:123
3262999404,33cyclictest26657-21ssh09:50:110
3262999392,32cyclictest6300-21ssh12:20:120
3262999392,30cyclictest12489-21sh10:00:140
3265499383,23cyclictest35966irq/58-eth010:05:133
3265499382,20cyclictest35966irq/58-eth009:55:153
32654993817,10cyclictest35966irq/58-eth011:55:143
3264699383,29cyclictest0-21swapper/212:20:122
3262999383,6cyclictest131rcu_preempt11:20:120
32646993732,3cyclictest27-21ksoftirqd/207:30:122
32646993614,18cyclictest0-21swapper/207:15:132
32646993614,18cyclictest0-21swapper/207:15:132
3263799361,27cyclictest0-21swapper/112:35:131
32654993523,8cyclictest35966irq/58-eth011:45:123
3264699352,24cyclictest27-21ksoftirqd/208:40:122
3265499344,16cyclictest35966irq/58-eth009:50:113
3262999343,25cyclictest31430-21ssh11:40:120
32629993418,13cyclictest12-21ksoftirqd/012:00:150
3265499332,24cyclictest35966irq/58-eth010:35:143
3264699332,23cyclictest19075-21ssh09:10:122
3262999332,22cyclictest30537-21ssh10:10:120
32629993310,16cyclictest12-21ksoftirqd/007:55:170
3265499323,15cyclictest35966irq/58-eth012:35:133
3265499322,19cyclictest35966irq/58-eth010:10:133
32646993226,4cyclictest9329-21ssh10:33:262
32646993226,2cyclictest0-21swapper/212:05:152
3262999323,22cyclictest29724-21ssh12:15:150
32629993223,6cyclictest4825-21ssh09:20:150
32629993218,6cyclictest12-21ksoftirqd/007:30:130
3264699313,18cyclictest27346irq/40-dwc2_hso08:15:132
3263799312,22cyclictest5827-21ssh10:50:141
32629993113,15cyclictest12-21ksoftirqd/010:15:130
32654993015,10cyclictest321rcuc/309:10:113
3264699303,17cyclictest27346irq/40-dwc2_hso10:45:122
3264699302,20cyclictest27346irq/40-dwc2_hso09:50:112
3263799304,15cyclictest20714-21ssh12:10:131
3263799303,20cyclictest17084-21ssh11:50:151
3263799302,22cyclictest12119-21sh09:23:261
32629993022,5cyclictest0-21swapper/010:35:140
3262999302,23cyclictest26238-21ssh11:55:150
32629993016,11cyclictest12-21ksoftirqd/010:45:120
3265499294,17cyclictest35966irq/58-eth009:20:143
3265499292,17cyclictest35966irq/58-eth011:40:133
3264699293,18cyclictest27346irq/40-dwc2_hso11:00:142
3264699292,22cyclictest0-21swapper/208:00:152
3264699292,11cyclictest26223-21ssh11:55:142
3264699291,11cyclictest94166ssh12:15:272
3262999291,23cyclictest25371-21ssh10:25:150
3265499283,16cyclictest0-21swapper/311:53:263
32654992813,9cyclictest35966irq/58-eth011:30:133
3264699282,11cyclictest26645-21ntp_states09:30:262
3263799282,20cyclictest17454-21rm11:14:001
3262999282,20cyclictest16269-21ssh10:20:140
3265499271,15cyclictest35966irq/58-eth010:15:133
3264699273,15cyclictest27346irq/40-dwc2_hso11:30:122
3264699272,18cyclictest12492-21ssh10:00:152
3264699272,11cyclictest0-21swapper/208:05:142
3264699271,9cyclictest1461166ssh10:35:232
3264699271,20cyclictest0-21swapper/207:25:142
3263799272,18cyclictest14923-21ssh10:55:131
3263799272,17cyclictest7175-21ssh10:15:141
32637992710,12cyclictest21-21ksoftirqd/110:43:261
3262999275,17cyclictest13389-21ssh11:30:130
3262999272,22cyclictest4001-21/usr/sbin/munin07:10:260
3262999271,21cyclictest11743-21ssh12:05:150
3262999271,20cyclictest863-21rm12:35:130
3265499267,12cyclictest35966irq/58-eth009:40:133
3265499263,15cyclictest0-21swapper/310:40:263
3264699263,20cyclictest15486-21kthreadcore09:05:212
3264699263,15cyclictest27346irq/40-dwc2_hso08:30:132
3264699262,18cyclictest27346irq/40-dwc2_hso08:55:142
3265499253,16cyclictest35966irq/58-eth012:05:153
3265499252,17cyclictest0-21swapper/310:23:263
3265499251,15cyclictest0-21swapper/310:00:143
3264699252,16cyclictest29522-21unixbench_singl10:25:302
3264699252,15cyclictest27346irq/40-dwc2_hso08:50:112
3264699252,12cyclictest7155-21ssh10:15:132
3264699251,3cyclictest13901-21ssh09:25:162
3264699251,17cyclictest16997-21kthreadcore10:20:192
3264699251,16cyclictest12140-21grep11:45:302
3263799253,14cyclictest18682-21diskmemload09:45:121
3262999253,15cyclictest0-21swapper/009:25:160
3262999252,17cyclictest30477-21rm09:33:410
32629992511,4cyclictest12-21ksoftirqd/008:45:020
3265499247,11cyclictest0-21swapper/310:33:263
3264699244,14cyclictest862-21diskmemload12:35:132
3264699243,18cyclictest3293-21ntp_kernel_pll_09:35:272
3264699243,13cyclictest25894-21/usr/sbin/munin08:10:172
3264699242,7cyclictest0-21swapper/209:40:262
3264699242,13cyclictest6246-21kthreadcore08:20:222
3264699242,13cyclictest4538-21fschecks_count11:25:162
3264699242,13cyclictest29888-21kthreadcore08:45:222
32646992412,9cyclictest0-21swapper/207:55:282
3264699241,13cyclictest30555-21rm10:10:132
3263799241,16cyclictest3336-21ssh09:55:151
3265499233,13cyclictest0-21swapper/307:10:263
32654992317,4cyclictest321rcuc/311:26:493
3264699234,16cyclictest930-21snmpd12:00:162
3264699234,12cyclictest19156-21ssh12:25:282
3264699233,17cyclictest20952-21cut12:10:162
3264699232,16cyclictest0-21swapper/209:22:262
3264699232,16cyclictest0-21swapper/207:10:262
3264699231,7cyclictest0-21swapper/209:15:182
3264699231,7cyclictest0-21swapper/209:15:182
32646992312,9cyclictest25425-21ssh11:54:392
32646992312,9cyclictest23230-21ssh09:46:222
3264699231,19cyclictest25506-21timerandwakeup10:05:292
3263799234,11cyclictest32030-21ssh09:35:141
3263799231,17cyclictest19072-21sh09:10:121
3262999231,18cyclictest24304-21ls11:00:170
3265499223,13cyclictest0-21swapper/311:14:063
3265499223,12cyclictest0-21swapper/310:25:153
3264699229,10cyclictest3743-21kthreadcore11:05:222
3264699228,6cyclictest17472-21ssh10:55:222
3264699223,12cyclictest11325-21cut07:20:192
3264699222,12cyclictest10486-21kthreadcore09:00:202
3264699221,6cyclictest462666rm11:41:242
32646992211,9cyclictest25138-21ssh10:40:382
3264699221,13cyclictest2421266ssh12:30:062
3264699221,13cyclictest2421266ssh12:30:062
3263799223,13cyclictest28119-21ssh09:15:141
3263799223,13cyclictest28119-21ssh09:15:141
32637992213,6cyclictest0-21swapper/110:05:281
3263799221,17cyclictest30875-21cut08:15:171
3263799221,13cyclictest0-21swapper/107:10:131
3262999222,18cyclictest4800-21sed11:05:300
3262999222,17cyclictest1023-21ssh09:35:210
3262999222,16cyclictest27285-21fschecks_count08:45:160
3262999222,14cyclictest630-21Async09:36:310
32629992212,7cyclictest25014-21sshd11:54:090
3262999221,15cyclictest6765-21ls09:55:230
3262999221,15cyclictest24543-21ls07:35:150
3265499213,6cyclictest0-21swapper/311:24:263
3265499212,10cyclictest35966irq/58-eth007:40:143
32654992114,4cyclictest0-21swapper/310:59:183
32654992113,5cyclictest35966irq/58-eth012:30:143
32654992113,5cyclictest35966irq/58-eth012:30:143
32654992113,3cyclictest0-21swapper/311:35:113
32654992112,7cyclictest0-21swapper/309:34:343
3264699217,11cyclictest6315-21cron07:50:012
3264699212,17cyclictest12031-21ls10:00:022
3264699212,16cyclictest26862-21kthreadcore07:35:212
32646992110,9cyclictest24943-21ssh11:17:082
3263799212,13cyclictest23600-21ssh10:23:261
3263799211,8cyclictest12877-21sed10:00:191
32637992112,6cyclictest0-21swapper/109:40:251
3263799211,17cyclictest29677-21find12:15:131
3263799211,16cyclictest18740-21sh11:15:141
3263799210,18cyclictest2150-21cut10:30:161
3262999215,6cyclictest12-21ksoftirqd/007:25:130
3262999213,15cyclictest8036-21ssh11:45:120
32629992112,7cyclictest25273-21awk10:05:270
3262999211,17cyclictest24600-21ls12:10:280
3262999211,17cyclictest12587-21kthreadcore11:10:220
3265499203,11cyclictest0-21swapper/309:35:153
3265499202,12cyclictest35966irq/58-eth012:00:153
32654992014,4cyclictest0-21swapper/312:25:123
32654992014,4cyclictest0-21swapper/312:10:103
32654992014,4cyclictest0-21swapper/310:53:083
32654992013,5cyclictest0-21swapper/312:22:393
32654992013,5cyclictest0-21swapper/310:45:123
3265499201,13cyclictest0-21swapper/312:15:133
3264699209,9cyclictest5850-21grep10:50:142
3264699202,15cyclictest19092-21kthreadcore08:35:212
3264699202,11cyclictest10125-21kthreadcore08:25:222
3264699201,9cyclictest22961-21cut11:35:172
3264699201,15cyclictest9856-21diskstats11:10:152
3264699201,11cyclictest29835-21irqrtprio07:40:192
3264699201,11cyclictest11229-21latency_hist07:55:012
3263799209,8cyclictest11737-21sshd11:45:261
3263799202,15cyclictest22392-21ssh11:35:121
3263799201,6cyclictest29910-21ntpq11:55:281
32637992013,3cyclictest17708-21sendmail_mailst09:25:301
3263799201,13cyclictest30894-21ls08:45:291
32637992010,8cyclictest14608-21load10:35:231
3262999204,14cyclictest9263-21ssh11:25:320
3262999203,14cyclictest11260-21cat08:25:240
3262999202,15cyclictest6713-21kthreadcore10:50:200
3262999202,15cyclictest20097-21kthreadcore08:35:220
3262999202,14cyclictest10199-21kthreadcore09:00:200
3262999201,6cyclictest28841-21munin-run10:45:010
3262999201,17cyclictest3544-21kthreadcore07:45:220
3262999201,16cyclictest17489-21kthreadcore12:25:210
3262999201,15cyclictest29594-21timerandwakeup08:10:280
3265499192,11cyclictest0-21swapper/308:25:133
32654991913,4cyclictest0-21swapper/311:15:383
32654991913,4cyclictest0-21swapper/311:00:023
32654991913,4cyclictest0-21swapper/309:15:143
32654991913,4cyclictest0-21swapper/309:15:143
32654991911,6cyclictest0-21swapper/308:57:103
3265499191,12cyclictest0-21swapper/307:30:023
3263799199,8cyclictest29486-21sed10:45:161
3263799192,12cyclictest2644-21ssh11:23:261
3263799191,9cyclictest0-21swapper/111:01:101
3263799191,9cyclictest0-21swapper/109:51:591
32637991912,4cyclictest0-21swapper/111:30:171
3263799191,15cyclictest4399-21sh11:25:141
3263799191,14cyclictest24342-21ssh12:30:141
3263799191,14cyclictest24342-21ssh12:30:141
3263799190,16cyclictest1727-21ls07:45:011
3262999198,8cyclictest18552-21sshd10:55:260
3262999193,13cyclictest32213-21idleruntime08:50:140
3262999192,15cyclictest22997-21timerandwakeup09:10:270
3262999192,15cyclictest20225-21kthreadcore09:45:220
3262999192,15cyclictest14300-21usage_EthcTask07:20:290
3262999192,15cyclictest13197-21irqcore08:30:190
3262999192,14cyclictest30289-21kthreadcore07:40:200
3262999191,10cyclictest26061-21kthreadcore12:30:200
3262999191,10cyclictest26061-21kthreadcore12:30:200
3265499182,10cyclictest0-21swapper/309:05:153
3265499182,10cyclictest0-21swapper/308:05:143
32654991813,3cyclictest321rcuc/307:55:223
32654991812,4cyclictest321rcuc/309:03:403
32654991812,4cyclictest321rcuc/307:35:153
32654991812,4cyclictest0-21swapper/308:15:153
32654991812,4cyclictest0-21swapper/308:00:213
3265499181,10cyclictest0-21swapper/308:55:013
3264699181,11cyclictest301-21latency_hist07:10:022
3263799189,6cyclictest31948-21cut11:40:171
3263799182,3cyclictest131rcu_preempt12:05:251
3263799181,11cyclictest15205-21tr07:25:181
3262999185,2cyclictest131rcu_preempt09:16:550
3262999185,2cyclictest131rcu_preempt09:16:550
3262999183,13cyclictest22798-21cut11:35:160
3262999182,11cyclictest0-21swapper/008:55:160
3262999182,11cyclictest0-21swapper/007:15:150
3262999182,11cyclictest0-21swapper/007:15:150
32629991811,5cyclictest2168-21unixbench_multi08:15:290
3262999181,14cyclictest14232-21fschecks_time09:05:170
3262999181,14cyclictest10331-21ls07:50:300
32629991810,6cyclictest18688-21/usr/sbin/munin11:15:250
3265499172,9cyclictest0-21swapper/308:45:043
3265499172,10cyclictest0-21swapper/308:40:133
3265499172,10cyclictest0-21swapper/308:30:153
3265499172,10cyclictest0-21swapper/308:20:143
3265499172,10cyclictest0-21swapper/307:45:043
32654991712,3cyclictest0-21swapper/307:15:263
32654991712,3cyclictest0-21swapper/307:15:263
32654991711,4cyclictest0-21swapper/307:20:263
3263799179,6cyclictest4104-21sshd08:53:361
3263799179,5cyclictest4446-21kthreadcore12:00:221
3263799178,7cyclictest4456-21awk11:05:281
3263799177,8cyclictest12807-21df_inode08:30:161
3263799176,9cyclictest31145-21cut10:10:181
3263799175,9cyclictest20099-21kthreadcore08:35:221
3263799175,3cyclictest131rcu_preempt12:25:011
3263799172,8cyclictest7641-21kthreadcore07:50:201
3263799172,11cyclictest0-21swapper/108:25:151
3263799171,8cyclictest11646-21kthreadcore07:20:191
3263799171,7cyclictest20821-21munin-run08:05:011
3263799171,13cyclictest5125-21cut08:20:211
3263799171,12cyclictest9858-21grep07:20:011
3263799171,12cyclictest9858-21grep07:20:011
3263799171,12cyclictest22912-21latency07:30:211
3263799171,12cyclictest18698-21ssh09:10:011
3263799170,11cyclictest28237-21timerwakeupswit07:35:291
3262999178,7cyclictest6093-21cut08:20:220
3262999172,12cyclictest2010-21ssh10:30:130
3262999171,11cyclictest18682-21diskmemload09:40:260
32654991611,3cyclictest0-21swapper/307:50:133
3263799169,4cyclictest27342-21kthreadcore10:25:231
3263799168,5cyclictest2139-21irqcore07:45:181
3263799163,10cyclictest23679-21kthreadcore08:05:211
3263799162,11cyclictest27312-21kthreadcore08:10:201
3263799161,13cyclictest9787-21cut09:00:191
3263799161,12cyclictest9219-21latency_hist09:00:011
3263799161,12cyclictest11360-21apt-get07:55:151
3265499152,9cyclictest0-21swapper/308:10:143
32654991510,3cyclictest0-21swapper/308:35:283
3263799151,11cyclictest22308-21apt-get08:40:131
3262999142,10cyclictest32647-21latency_hist07:10:010
3263799137,4cyclictest32621-21cyclictest07:10:011
3265499111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional