You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-12 - 21:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Tue May 12, 2026 12:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2539996736,17cyclictest35966irq/58-eth011:55:153
250699501,37cyclictest1724-21ssh11:40:140
253999444,26cyclictest35966irq/58-eth009:50:123
253999442,25cyclictest35966irq/58-eth009:55:133
253999435,23cyclictest35966irq/58-eth012:15:133
253999435,23cyclictest35966irq/58-eth012:15:123
251599431,29cyclictest28916-21ssh11:55:161
251599431,12cyclictest9909-21ssh10:15:131
250699400,35cyclictest16227-21ssh09:25:130
2539993915,17cyclictest0-21swapper/311:30:143
2506993917,3cyclictest12-21ksoftirqd/010:55:140
251599371,28cyclictest14591-21sh12:05:151
253999362,20cyclictest35966irq/58-eth012:25:133
2530993527,5cyclictest6042-21ssh09:55:132
253999344,14cyclictest35966irq/58-eth010:20:113
251599345,7cyclictest0-21swapper/108:45:141
253999334,24cyclictest35966irq/58-eth011:05:123
253999332,19cyclictest35966irq/58-eth010:05:143
253999324,19cyclictest35966irq/58-eth010:40:153
253999321,19cyclictest35966irq/58-eth011:35:133
253099321,16cyclictest29363-21ssh09:50:122
251599325,19cyclictest21-21ksoftirqd/110:45:111
253999313,19cyclictest35966irq/58-eth010:00:133
2539993112,11cyclictest35966irq/58-eth009:30:133
253999311,19cyclictest35966irq/58-eth009:25:133
253099314,18cyclictest27346irq/40-dwc2_hso08:10:132
253099312,20cyclictest27346irq/40-dwc2_hso11:35:142
253099311,20cyclictest18976-21ssh10:20:132
253999291,16cyclictest0-21swapper/311:15:123
253099295,17cyclictest261rcuc/210:55:132
253099294,16cyclictest27346irq/40-dwc2_hso07:10:142
2530992911,12cyclictest10776-21ssh11:45:142
251599291,23cyclictest11278-21ssh09:40:131
2506992916,8cyclictest12-21ksoftirqd/012:20:130
253999287,15cyclictest321rcuc/308:00:143
253999282,16cyclictest35966irq/58-eth012:35:143
253099282,21cyclictest0-21swapper/207:45:162
253099282,20cyclictest0-21swapper/208:50:142
253099282,15cyclictest16048-21sh11:30:142
2515992820,5cyclictest20949-21diskmemload09:10:151
251599280,20cyclictest21383-21ssh11:15:131
253999273,11cyclictest35966irq/58-eth009:45:133
253999273,10cyclictest0-21swapper/311:20:143
253999272,13cyclictest35966irq/58-eth010:45:123
253999272,11cyclictest35966irq/58-eth012:30:123
253099273,16cyclictest27346irq/40-dwc2_hso08:05:132
253099273,16cyclictest27346irq/40-dwc2_hso08:05:122
253099272,19cyclictest27346irq/40-dwc2_hso07:15:132
253099271,15cyclictest20285-21ssh09:45:132
250699271,13cyclictest27659-21idleruntime-cro10:25:000
253999264,16cyclictest35966irq/58-eth010:30:133
253099264,16cyclictest27346irq/40-dwc2_hso07:20:142
253099264,16cyclictest27346irq/40-dwc2_hso07:20:132
253099263,16cyclictest27346irq/40-dwc2_hso08:25:132
253099262,12cyclictest93066snmpd11:08:362
2530992612,7cyclictest27379-21ssh12:30:142
251599263,12cyclictest22840-21ssh10:40:141
2506992618,5cyclictest20949-21diskmemload09:45:140
250699261,17cyclictest19784-21ssh11:50:140
253999259,12cyclictest35966irq/58-eth009:40:133
253999252,19cyclictest35966irq/58-eth011:10:133
253099254,14cyclictest27346irq/40-dwc2_hso08:00:132
253099252,20cyclictest15143-21ssh10:00:142
2530992520,3cyclictest0-21swapper/211:55:152
2530992513,10cyclictest6402-21ssh10:11:192
253099251,20cyclictest16405-21grep09:25:162
2530992511,11cyclictest24213-21grep12:10:182
251599251,15cyclictest8501-21ssh10:50:131
2506992514,3cyclictest131rcu_preempt12:00:270
2539992411,10cyclictest35966irq/58-eth011:00:163
253099243,15cyclictest27346irq/40-dwc2_hso09:30:132
253099242,16cyclictest27346irq/40-dwc2_hso07:35:132
253099241,12cyclictest24483-21cut10:05:172
2530992410,4cyclictest27029-21grep11:00:182
2515992413,7cyclictest0-21swapper/109:35:151
250699242,19cyclictest32654-21ssh12:15:130
250699242,19cyclictest32654-21ssh12:15:130
250699242,17cyclictest32534-21ssh11:55:270
2539992312,6cyclictest0-21swapper/312:00:153
253099237,14cyclictest22215-21kthreadcore09:10:212
2530992315,6cyclictest0-21swapper/212:25:122
253099231,20cyclictest930-21snmpd12:00:142
2530992311,4cyclictest4785-21df10:30:142
2530992310,6cyclictest10766-21cron09:40:002
251599238,12cyclictest26764-21diskmemload11:00:161
251599233,16cyclictest3983-21ssh12:35:151
251599232,4cyclictest14721-21munin-run10:00:021
251599232,16cyclictest30494-21ssh09:15:151
251599232,12cyclictest20257-21ssh09:45:131
2515992314,6cyclictest0-21swapper/109:00:271
251599231,19cyclictest5527-21ssh12:00:121
2515992310,6cyclictest630-21Async09:36:311
2506992316,5cyclictest6046-21sh09:55:150
253999223,12cyclictest0-21swapper/309:15:153
2539992213,6cyclictest35966irq/58-eth009:10:163
253999221,15cyclictest0-21swapper/311:50:143
2539992210,10cyclictest321rcuc/311:40:153
253099228,7cyclictest16119-21cron09:05:012
253099224,7cyclictest27-21ksoftirqd/212:20:132
253099222,4cyclictest0-21swapper/210:45:132
253099222,11cyclictest15653-21ls11:30:022
253099222,11cyclictest10658-21kthreadcore10:15:212
253099222,10cyclictest6641-21kthreadcore08:20:182
253099221,5cyclictest28044-21ssh10:25:132
253099221,5cyclictest23246-21ls11:50:232
2530992211,9cyclictest16358-21ssh10:54:032
253099221,17cyclictest12293-21/usr/sbin/munin11:10:152
253099221,16cyclictest30424-21apt-get11:20:132
253099221,13cyclictest4878-21tr11:40:242
2530992210,9cyclictest0-21swapper/207:55:222
2530992210,10cyclictest11292-1kworker/u9:110:35:232
251599224,6cyclictest131rcu_preempt09:05:131
251599221,17cyclictest7135-21ssh09:20:131
2515992210,8cyclictest12328-21ssh11:10:131
250699222,17cyclictest13804-21ssh10:35:140
250699222,16cyclictest11253-21apt-get09:40:110
253999218,8cyclictest33-21ksoftirqd/310:55:133
253999213,14cyclictest33-21ksoftirqd/312:10:143
253999212,11cyclictest35966irq/58-eth010:35:143
253099217,11cyclictest17672-21kthreadcore08:30:202
253099214,9cyclictest32262-21kthreadcore08:45:212
253099214,11cyclictest27346irq/40-dwc2_hso12:15:122
253099214,11cyclictest27346irq/40-dwc2_hso12:15:112
253099212,11cyclictest19899-21fschecks_time08:35:152
253099212,11cyclictest19899-21fschecks_time08:35:142
253099211,11cyclictest6168-21kthreadcore12:35:212
253099211,11cyclictest29247-21cmp08:45:002
253099211,11cyclictest26492-21ntp_192.168.11410:40:252
253099211,11cyclictest14736-21ssh12:05:172
253099211,11cyclictest11506-21fschecks_time09:40:162
253099210,13cyclictest25096-21ssh11:15:282
251599217,7cyclictest2498-21cyclictest08:15:141
2515992113,6cyclictest0-21swapper/110:05:201
251599210,14cyclictest4763-21ls10:30:141
251599210,14cyclictest18133-21ls12:25:021
250699213,15cyclictest8520-21ssh10:50:130
250699211,9cyclictest0-21swapper/011:10:200
2506992115,4cyclictest0-21swapper/011:05:140
250699211,14cyclictest30456-21ssh11:20:150
253999209,8cyclictest0-21swapper/311:49:343
253999208,9cyclictest0-21swapper/309:35:563
253999203,9cyclictest0-21swapper/312:05:163
253099202,8cyclictest527-21fschecks_time08:15:162
253099202,10cyclictest16890-21irqrtprio09:05:182
253099201,11cyclictest30605-21ssh09:15:152
253099201,10cyclictest31685-21cut07:40:172
253099201,10cyclictest12248-21ps07:50:272
253099201,10cyclictest11816-21cat09:20:332
251599208,10cyclictest16617-21grep09:25:181
251599203,2cyclictest0-21swapper/111:40:131
2515992013,5cyclictest0-21swapper/107:35:231
2515992013,4cyclictest0-21swapper/111:05:241
2515992012,6cyclictest7187-21sshd07:12:381
251599201,16cyclictest25118-21apt-get11:35:141
2515992011,6cyclictest0-21swapper/107:55:221
250699203,14cyclictest25450-21ssh09:30:140
250699203,12cyclictest0-21swapper/008:05:050
250699203,12cyclictest0-21swapper/008:05:040
250699202,15cyclictest25133-21ssh11:35:150
250699202,15cyclictest12-21ksoftirqd/012:30:140
250699201,9cyclictest0-21swapper/010:11:050
250699201,16cyclictest22184-21ls12:25:270
250699201,16cyclictest16049-21rm11:30:140
2506992011,6cyclictest0-21swapper/008:50:250
253999198,8cyclictest0-21swapper/312:20:413
253999192,12cyclictest0-21swapper/308:10:113
253999191,13cyclictest0-21swapper/310:25:163
253999191,13cyclictest0-21swapper/307:25:303
253999191,12cyclictest0-21swapper/310:15:153
253999191,11cyclictest0-21swapper/308:30:143
2539991910,6cyclictest0-21swapper/310:12:473
253099199,8cyclictest0-21swapper/208:55:122
253099191,14cyclictest22319-21irqcore07:30:202
253099191,10cyclictest17227-21fschecks_count07:25:172
251599199,8cyclictest1847-21apt-get08:50:141
251599191,8cyclictest993-21ssh11:20:221
251599191,8cyclictest15510-21if_eth110:00:191
2515991912,4cyclictest0-21swapper/112:10:111
2515991911,6cyclictest26447-21sshd07:33:381
2515991911,5cyclictest0-21swapper/108:55:291
251599191,14cyclictest19755-21ssh11:50:111
2515991910,7cyclictest14441-21ntpq11:45:271
2515991910,6cyclictest15289-21kthreadcore10:35:211
250699199,7cyclictest3178-21awk10:45:260
250699197,10cyclictest8683-21cat10:30:280
250699197,10cyclictest16203-21sshd09:05:020
250699197,10cyclictest12-21ksoftirqd/007:45:160
250699193,12cyclictest29368-21ssh09:50:120
250699192,13cyclictest9898-21apt-get10:15:140
250699192,13cyclictest10766-21ssh11:45:120
2506991913,4cyclictest23604-21/usr/sbin/munin12:10:310
2506991912,4cyclictest22120-21kthreadcore11:15:200
250699191,15cyclictest26761-21ssh11:00:140
2506991911,5cyclictest21775-21/usr/sbin/munin09:10:190
250699191,15cyclictest20949-21diskmemload11:25:140
2506991910,7cyclictest19980-21cat10:00:330
250699190,16cyclictest2050-21find09:35:130
253999185,9cyclictest0-21swapper/309:20:133
253999183,10cyclictest0-21swapper/308:20:103
253999182,10cyclictest0-21swapper/309:05:133
253999182,10cyclictest0-21swapper/309:00:153
253999182,10cyclictest0-21swapper/308:55:043
253999182,10cyclictest0-21swapper/308:25:053
253999182,10cyclictest0-21swapper/307:40:143
253999182,10cyclictest0-21swapper/307:35:153
253999181,13cyclictest0-21swapper/310:50:293
253999181,11cyclictest0-21swapper/308:00:023
253999181,11cyclictest0-21swapper/307:20:173
253999181,11cyclictest0-21swapper/307:20:163
253999181,11cyclictest0-21swapper/307:10:153
253999181,10cyclictest0-21swapper/308:45:013
2539991810,6cyclictest0-21swapper/311:28:413
251599189,6cyclictest0-21swapper/108:10:161
251599189,6cyclictest0-21swapper/107:40:331
251599188,7cyclictest5003-21cat12:15:321
251599188,7cyclictest5003-21cat12:15:311
251599186,9cyclictest10377-21kthreadcore11:25:241
251599182,13cyclictest18332-21ssh12:25:121
251599181,9cyclictest23451-21hwlatdetect08:05:161
251599181,9cyclictest23451-21hwlatdetect08:05:151
251599181,8cyclictest25119-21irqrtprio08:40:191
2515991812,4cyclictest0-21swapper/107:25:261
2515991811,5cyclictest0-21swapper/107:15:231
2515991811,4cyclictest29391-21timerandwakeup09:30:281
250699188,8cyclictest9831-21cron08:25:010
250699186,10cyclictest1-21systemd08:44:560
250699183,12cyclictest18480-21cat12:05:300
250699183,11cyclictest0-21swapper/007:10:140
250699182,12cyclictest31243-21apt-get07:40:040
250699181,8cyclictest7838-21irqstats09:20:190
250699181,3cyclictest0-21swapper/009:17:000
2506991812,4cyclictest18099-21sh08:00:010
2506991811,5cyclictest22798-21/usr/sbin/munin10:40:210
2506991810,5cyclictest28769-21kernelversion10:25:190
250699180,15cyclictest12103-21ls07:20:010
253999172,9cyclictest0-21swapper/308:50:153
253999172,10cyclictest0-21swapper/307:30:153
253999171,12cyclictest0-21swapper/308:35:153
253999171,12cyclictest0-21swapper/308:35:143
251599177,7cyclictest21-21ksoftirqd/110:25:131
251599171,8cyclictest19720-21/usr/sbin/munin08:35:161
251599171,8cyclictest19720-21/usr/sbin/munin08:35:151
251599171,7cyclictest17180-21apt-get10:55:041
251599171,10cyclictest31288-21ls12:30:281
251599171,10cyclictest12468-21fschecks_time07:20:171
251599171,10cyclictest12468-21fschecks_time07:20:161
2515991710,4cyclictest20958-21kthreadcore10:20:221
251599170,15cyclictest5668-21ls09:55:011
250699174,8cyclictest131rcu_preempt08:30:110
250699171,8cyclictest135-21systemd-journal07:40:010
2506991711,4cyclictest28060-21sendmail_mailtr10:05:300
250699171,14cyclictest10194-21kthreadcore07:50:210
250699171,13cyclictest135-21systemd-journal12:35:480
253999165,8cyclictest0-21swapper/308:15:213
251599161,2cyclictest0-21swapper/110:10:151
251599161,12cyclictest11459-21cut08:25:201
251599161,11cyclictest15997-21/usr/sbin/munin11:30:271
2515991610,4cyclictest0-21swapper/107:50:261
250699169,5cyclictest21674-21cut08:35:200
250699169,5cyclictest21674-21cut08:35:200
250699168,5cyclictest19908-21cut08:00:210
250699167,7cyclictest17077-21apt-get07:25:150
2506991611,3cyclictest0-21swapper/007:30:180
250699161,12cyclictest12668-21cut07:20:190
250699161,12cyclictest12668-21cut07:20:180
250699161,11cyclictest30137-21cut08:45:180
2506991610,3cyclictest718-21tail08:15:180
253999152,9cyclictest0-21swapper/307:45:163
253999151,9cyclictest0-21swapper/308:05:123
253999151,9cyclictest0-21swapper/308:05:123
253999151,8cyclictest0-21swapper/308:45:043
253099152,10cyclictest2602-21wc07:10:022
251599158,5cyclictest19030-21cat08:30:271
251599151,10cyclictest3969-21apt-get07:45:151
251599151,10cyclictest18498-21tune2fs08:00:181
250699151,11cyclictest6415-21apt-get08:55:020
250699151,11cyclictest28202-21cut08:10:160
250699151,11cyclictest16814-21sed09:05:170
250699151,11cyclictest10242-21diskstats08:25:160
250699141,10cyclictest2557-21latency_hist07:10:010
253999131,7cyclictest0-21swapper/307:50:183
253999131,6cyclictest0-21swapper/307:15:153
251599131,10cyclictest2498-21cyclictest07:10:021
253999111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional