You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-01 - 21:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Fri May 01, 2026 12:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2953399653,48cyclictest15903-21ssh11:35:122
29533995311,35cyclictest27-21ksoftirqd/207:15:132
29533995016,28cyclictest27-21ksoftirqd/212:00:122
2952399471,36cyclictest20320-21sh10:25:131
2952399471,15cyclictest1343766ssh09:45:141
2953399460,35cyclictest15175-21diskmemload11:20:122
2954199445,27cyclictest35966irq/58-eth011:10:123
2954199432,27cyclictest35966irq/58-eth011:00:143
2954199417,23cyclictest35966irq/58-eth009:15:123
2954199417,23cyclictest35966irq/58-eth009:15:123
2954199415,30cyclictest35966irq/58-eth010:45:133
2954199411,25cyclictest35966irq/58-eth008:55:113
2953399394,24cyclictest17871-21ssh12:30:112
2953399382,19cyclictest20180-21ssh11:18:552
29533993813,21cyclictest20321-21sh10:25:132
2954199372,25cyclictest35966irq/58-eth011:15:143
2952399371,31cyclictest8827-21ssh12:25:121
2951599373,26cyclictest1518-21ssh11:45:150
2954199354,21cyclictest35966irq/58-eth011:25:113
2954199353,21cyclictest35966irq/58-eth009:20:113
2954199352,22cyclictest35966irq/58-eth010:20:133
29533993530,3cyclictest261rcuc/208:35:132
29533993530,3cyclictest261rcuc/208:35:132
2952399351,26cyclictest14740-21ssh10:40:121
2954199343,15cyclictest35966irq/58-eth009:45:143
2954199331,20cyclictest35966irq/58-eth012:00:123
2954199331,17cyclictest35966irq/58-eth011:30:103
2951599332,26cyclictest5117-21ssh12:05:140
2954199322,21cyclictest35966irq/58-eth010:30:113
29515993213,17cyclictest18029-21ssh11:00:150
29541993118,8cyclictest35966irq/58-eth009:35:133
2953399313,21cyclictest31423-21ssh09:55:122
2953399311,18cyclictest2429-21ssh10:15:132
2953399311,18cyclictest2429-21ssh10:15:132
29515993118,10cyclictest12-21ksoftirqd/012:35:130
2954199303,17cyclictest35966irq/58-eth012:25:123
2954199301,22cyclictest35966irq/58-eth010:40:123
2953399308,6cyclictest0-21swapper/209:50:122
29515993015,13cyclictest12-21ksoftirqd/010:45:120
2953399294,17cyclictest27346irq/40-dwc2_hso08:30:112
2953399292,20cyclictest0-21swapper/211:55:132
2953399291,14cyclictest32492-21ssh10:50:102
2952399297,6cyclictest131rcu_preempt10:55:011
2954199284,17cyclictest35966irq/58-eth008:45:133
2954199284,14cyclictest35966irq/58-eth010:05:133
2954199283,18cyclictest0-21swapper/311:08:553
2954199282,14cyclictest35966irq/58-eth009:40:123
29541992812,9cyclictest35966irq/58-eth012:30:123
2953399284,15cyclictest27346irq/40-dwc2_hso08:15:132
2953399283,15cyclictest27346irq/40-dwc2_hso08:50:132
29523992810,3cyclictest131rcu_preempt10:30:241
2951599281,23cyclictest24894-21ssh11:40:120
2954199278,14cyclictest321rcuc/307:50:123
2954199273,16cyclictest0-21swapper/311:35:143
2953399272,19cyclictest0-21swapper/207:40:122
2953399272,10cyclictest0-21swapper/210:00:122
2953399271,19cyclictest7180-21ssh11:30:142
29533992710,11cyclictest1643-21sh11:45:172
2952399277,13cyclictest21-21ksoftirqd/107:30:041
2952399272,18cyclictest1641-21df_inode11:45:171
2952399271,5cyclictest131rcu_preempt11:10:111
2952399270,18cyclictest28513-21sh12:00:111
2954199263,17cyclictest0-21swapper/311:45:183
2954199263,15cyclictest0-21swapper/311:21:553
2954199262,16cyclictest0-21swapper/311:40:123
29533992612,10cyclictest14670-21unixbench_singl11:50:282
2953399261,17cyclictest10420-21ls09:05:012
2953399261,17cyclictest10420-21ls09:05:012
2953399261,13cyclictest14107-21sh12:10:132
2952399262,18cyclictest27488-21ssh11:21:551
29523992618,4cyclictest0-21swapper/111:15:151
2952399261,19cyclictest17887-21ssh12:30:121
2951599265,19cyclictest29844-21kworker/u8:010:55:130
2951599265,19cyclictest29844-21kworker/u8:010:55:130
2954199252,11cyclictest35966irq/58-eth009:00:133
2954199252,11cyclictest35966irq/58-eth009:00:133
2954199251,13cyclictest35966irq/58-eth012:20:133
2953399258,11cyclictest4667-1kworker/u9:210:35:112
2953399252,16cyclictest27346irq/40-dwc2_hso10:30:102
29533992512,7cyclictest261rcuc/207:30:042
2952399251,19cyclictest14106-21ssh12:10:131
2951599252,17cyclictest17009-21sh10:05:140
2951599251,20cyclictest29336-21ssh10:30:110
2951599251,20cyclictest15175-21diskmemload09:20:120
2953399243,15cyclictest27346irq/40-dwc2_hso11:10:112
2953399242,6cyclictest0-21swapper/211:40:552
2953399242,19cyclictest0-21swapper/207:10:152
2951599246,7cyclictest131rcu_preempt11:35:150
2951599244,11cyclictest0-21swapper/012:15:110
2951599244,11cyclictest0-21swapper/012:15:110
2951599243,6cyclictest32193-21apt-get12:20:130
2951599242,18cyclictest30435-21ssh11:25:130
2951599242,17cyclictest4498-21ssh09:40:130
29515992418,4cyclictest10028-21ssh09:25:110
29515992411,9cyclictest12-21ksoftirqd/010:20:130
2954199233,15cyclictest0-21swapper/309:30:123
2953399232,6cyclictest0-21swapper/211:28:552
2953399232,16cyclictest0-21swapper/211:02:552
29533992311,9cyclictest29718-21ssh09:15:532
29533992311,9cyclictest29718-21ssh09:15:522
2952399235,10cyclictest3161-21cron12:40:011
2952399232,15cyclictest31418-21ssh09:55:121
2952399231,16cyclictest32222-21ssh12:20:131
2951599235,9cyclictest0-21swapper/007:25:120
2951599235,9cyclictest0-21swapper/007:25:120
2954199222,14cyclictest0-21swapper/310:50:103
29541992216,3cyclictest0-21swapper/312:15:123
29541992216,3cyclictest0-21swapper/312:15:123
2953399228,10cyclictest10438-21sed09:25:172
2953399223,5cyclictest0-21swapper/212:20:132
2953399223,11cyclictest15660-21ssh09:10:142
2953399221,17cyclictest16462-21cut10:40:192
2952399225,6cyclictest21-21ksoftirqd/111:55:121
2952399223,12cyclictest0-21swapper/108:40:131
29523992214,5cyclictest5869-21ssh10:35:121
29523992214,5cyclictest4496-21sh09:40:131
2952399221,18cyclictest1014-21ls08:55:131
2951599227,12cyclictest12-21ksoftirqd/009:15:120
2951599227,12cyclictest12-21ksoftirqd/009:15:120
2951599224,9cyclictest12-21ksoftirqd/007:55:100
29515992211,9cyclictest30039-21cron11:25:010
2951599221,16cyclictest8266-21ls10:00:150
2954199219,6cyclictest35966irq/58-eth009:25:123
29541992110,8cyclictest35966irq/58-eth010:55:133
29541992110,8cyclictest35966irq/58-eth010:55:123
29541992110,6cyclictest0-21swapper/312:37:533
2953399219,5cyclictest30589-21cat10:10:302
2953399218,9cyclictest23231-21ssh12:15:132
2953399218,9cyclictest23231-21ssh12:15:132
2953399214,10cyclictest614-21dbus-daemon09:49:512
2953399211,8cyclictest1086-21diskmemload09:20:102
2953399211,6cyclictest1517566diskmemload10:45:192
2953399211,16cyclictest18149-21kthreadcore10:05:202
2953399211,12cyclictest0-21swapper/209:33:572
2953399211,10cyclictest8174-21ntpq09:40:252
2952399212,16cyclictest16990-21ssh10:05:121
2952399211,16cyclictest14989-21ssh10:20:251
2952399211,14cyclictest22415-21ls08:10:021
2952399211,14cyclictest22415-21ls08:10:011
29523992110,8cyclictest16205-21cron07:30:011
29523992110,8cyclictest16205-21cron07:30:001
2952399210,17cyclictest10616-21find09:05:121
2952399210,17cyclictest10616-21find09:05:121
2951599212,15cyclictest14081-21apt-get12:10:120
2951599211,8cyclictest12622-21/usr/sbin/munin11:15:140
29515992111,7cyclictest32319-21cat09:35:290
29515992111,6cyclictest15532-21ssh09:10:040
2951599211,11cyclictest22835-21grep08:10:170
2951599210,15cyclictest18022-21grep12:30:160
2954199204,9cyclictest0-21swapper/311:55:133
2954199202,14cyclictest35966irq/58-eth010:25:143
2953399208,9cyclictest11300-21ssh10:20:072
2953399207,10cyclictest29847-21kthreadcore11:05:212
2953399206,12cyclictest27-21ksoftirqd/209:05:222
2953399206,12cyclictest27-21ksoftirqd/209:05:212
2953399206,11cyclictest29285-21kthreadcore09:35:202
2953399205,12cyclictest12931-21apt-get08:00:122
2953399202,15cyclictest5214-21df_inode12:05:152
2953399202,11cyclictest0-21swapper/208:40:132
29533992013,4cyclictest10887-21ssh12:25:212
2953399201,15cyclictest27074-21cron08:15:002
2953399201,10cyclictest9066-21ssh10:55:132
2953399201,10cyclictest9066-21ssh10:55:132
2952399204,3cyclictest131rcu_preempt11:00:131
29523992012,4cyclictest1384-21idleruntime09:20:151
2952399201,16cyclictest30394-21find11:25:111
2952399201,11cyclictest22034-21sort07:35:181
2952399200,16cyclictest24909-21ssh11:40:131
2951599207,9cyclictest7558-21cat11:10:260
2951599202,16cyclictest7206-21/usr/sbin/munin11:30:150
2951599202,15cyclictest3864-21ssh10:50:240
2951599201,8cyclictest18216-21grep08:05:190
2951599201,8cyclictest18216-21grep08:05:180
29515992015,3cyclictest536-21usb-storage08:20:110
2951599201,15cyclictest6168-21awk10:15:260
2951599201,15cyclictest6168-21awk10:15:260
2951599201,15cyclictest32387-21kthreadcore07:45:210
2951599201,15cyclictest14723-21apt-get10:40:110
2951599201,11cyclictest7176-21cut10:35:180
2951599200,16cyclictest9496-21timerandwakeup09:00:280
2951599200,16cyclictest9496-21timerandwakeup09:00:280
2954199198,7cyclictest35966irq/58-eth012:10:123
2954199194,8cyclictest35966irq/58-eth010:35:123
29541991911,5cyclictest0-21swapper/309:50:113
2954199191,13cyclictest0-21swapper/308:50:153
2954199191,13cyclictest0-21swapper/307:35:143
2954199191,12cyclictest0-21swapper/307:30:273
2953399198,9cyclictest24994-21switchtime07:35:252
2953399196,10cyclictest23891-21fschecks_count08:45:152
2953399192,10cyclictest6385-21kthreadcore08:25:192
2953399191,9cyclictest9316-21kthreadcore07:20:192
2953399191,9cyclictest30374-21latency12:35:222
2953399191,15cyclictest8112-21/usr/sbin/munin07:55:282
2953399191,14cyclictest11853-21wc07:25:162
2953399191,14cyclictest11853-21wc07:25:162
2953399191,10cyclictest2123-21kthreadcore08:20:202
2953399191,10cyclictest1785-21latency07:45:232
2952399199,7cyclictest19267-21ntpq09:10:241
2952399198,8cyclictest15737-21sshd11:50:551
2952399194,13cyclictest27048-21ssh11:05:121
2952399191,13cyclictest24574-21sh09:15:141
2952399191,13cyclictest24574-21sh09:15:141
2952399190,15cyclictest4177-21ls09:40:011
2951599199,8cyclictest28510-21apt-get12:00:110
2951599198,8cyclictest31743-21if_eth009:55:150
2951599194,12cyclictest13877-21irqcore09:45:190
2951599192,14cyclictest9135-21cron08:30:010
2951599192,12cyclictest15175-21diskmemload11:05:130
2951599191,15cyclictest18969-21apt-get08:40:110
2951599191,12cyclictest31821-21latency08:50:220
2954199189,6cyclictest0-21swapper/312:05:553
2954199188,7cyclictest0-21swapper/309:10:213
2954199186,9cyclictest33-21ksoftirqd/310:10:163
2954199182,10cyclictest0-21swapper/309:05:143
2954199182,10cyclictest0-21swapper/309:05:143
2954199182,10cyclictest0-21swapper/308:35:133
2954199182,10cyclictest0-21swapper/308:35:133
29541991811,4cyclictest0-21swapper/309:55:263
2954199181,12cyclictest0-21swapper/307:20:203
2954199181,11cyclictest0-21swapper/307:15:153
2953399181,9cyclictest1093-21fschecks_count08:55:152
2953399181,8cyclictest17849-21fschecks_count08:05:162
2953399181,8cyclictest17849-21fschecks_count08:05:162
2953399181,10cyclictest3321-21apt-get07:50:142
2952399188,7cyclictest17583-21cron11:00:011
2952399188,7cyclictest17583-21cron11:00:011
2952399183,12cyclictest15884-21ssh11:35:121
2952399182,6cyclictest23847-21/usr/sbin/munin12:15:171
2952399182,6cyclictest23847-21/usr/sbin/munin12:15:171
29523991812,4cyclictest32435-21unixbench_singl08:50:281
29523991811,4cyclictest12018-21kthreadcore09:25:191
29523991811,4cyclictest0-21swapper/107:45:151
2952399181,13cyclictest8599-21memory12:05:241
2952399181,12cyclictest8009-21ssh10:00:121
2952399181,11cyclictest22509-21ssh09:30:231
29523991810,6cyclictest4333-21sshd08:20:521
29523991810,5cyclictest0-21swapper/109:50:041
2951599186,9cyclictest28143-21basename12:00:010
2951599185,4cyclictest131rcu_preempt12:25:150
2951599183,11cyclictest0-21swapper/007:35:120
2951599182,12cyclictest19111-21df09:30:140
2951599181,9cyclictest25867-21kthreadcore08:45:200
2951599181,9cyclictest14585-21cut08:00:190
29515991811,5cyclictest6927-21swap07:50:260
29515991811,5cyclictest15150-21sshd09:05:520
29515991811,5cyclictest15150-21sshd09:05:520
2951599181,12cyclictest11197-21sed11:50:180
2954199179,6cyclictest0-21swapper/310:04:233
2954199173,5cyclictest0-21swapper/308:40:113
2954199172,9cyclictest0-21swapper/307:25:133
2954199172,9cyclictest0-21swapper/307:25:133
2954199171,11cyclictest35966irq/58-eth010:15:143
2954199171,11cyclictest35966irq/58-eth010:15:133
2954199171,10cyclictest0-21swapper/308:25:153
2954199171,10cyclictest0-21swapper/308:20:053
2954199171,10cyclictest0-21swapper/308:05:143
2954199171,10cyclictest0-21swapper/308:05:143
2954199171,10cyclictest0-21swapper/307:45:183
2952399179,5cyclictest26517-21irqcore07:40:161
2952399179,5cyclictest0-21swapper/110:15:131
2952399179,5cyclictest0-21swapper/110:15:131
2952399178,6cyclictest0-21swapper/107:10:161
2952399172,11cyclictest12968-21ssh11:31:551
2952399171,14cyclictest27775-21ssh10:45:291
2951599179,5cyclictest29266-21sshd09:52:520
2951599178,7cyclictest22786-21cut10:25:210
2951599178,7cyclictest12812-21ntpq08:30:230
2951599172,12cyclictest26613-21irqstats10:10:180
2951599172,12cyclictest17610-21ntp_states08:35:240
2951599172,12cyclictest17610-21ntp_states08:35:240
2951599171,9cyclictest31896-21tr07:10:210
2951599171,7cyclictest5799-21unixbench_multi07:15:280
2951599171,13cyclictest5762-21grep09:00:010
2954199168,6cyclictest0-21swapper/311:50:353
2952399169,4cyclictest15333-21kthreadcore08:00:201
2952399163,7cyclictest0-21swapper/108:15:131
2952399163,2cyclictest131rcu_preempt07:50:111
2952399162,9cyclictest0-21swapper/109:00:141
2952399162,9cyclictest0-21swapper/109:00:141
2952399161,12cyclictest6819-21apt-get07:20:111
2952399161,12cyclictest27345-21cat08:45:281
2952399161,12cyclictest22597-21diskstats08:10:151
2952399161,11cyclictest3395-21tr07:15:201
2952399161,11cyclictest25899-21/usr/sbin/munin10:10:111
2951599167,6cyclictest29994-21grep07:40:270
2951599162,9cyclictest0-21swapper/007:30:130
2951599162,8cyclictest12-21ksoftirqd/007:20:110
2951599161,12cyclictest31906-21http08:20:020
2954199151,9cyclictest0-21swapper/307:55:263
2954199151,11cyclictest0-21swapper/307:10:273
2954199150,12cyclictest33-21ksoftirqd/308:00:163
2952399152,9cyclictest0-21swapper/108:35:141
2952399152,9cyclictest0-21swapper/108:35:131
2952399152,9cyclictest0-21swapper/108:30:131
2952399151,11cyclictest8137-21cpu07:55:111
2952399151,11cyclictest4682-21diskstats08:25:141
2951599151,11cyclictest29554-21latency_hist07:10:020
2954199148,4cyclictest0-21swapper/308:10:123
2954199143,7cyclictest0-21swapper/308:30:113
2952399132,9cyclictest29544-21latency_hist07:10:021
2954199122,7cyclictest0-21swapper/308:15:133
2954199121,3cyclictest0-21swapper/307:40:213
2953399121,8cyclictest29585-21latency_hist07:10:022
2954199113,6cyclictest0-21swapper/307:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional