You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-02 - 18:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Aug 02, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2552299823,7cyclictest131rcu_preempt11:30:120
2552299703,47cyclictest11814-21ssh09:10:130
2552799581,45cyclictest14477-21ssh11:35:141
2552299575,41cyclictest8932-21ssh11:50:150
2552299571,48cyclictest16270-21ssh12:30:140
25527995123,19cyclictest21-21ksoftirqd/110:15:131
2552299513,7cyclictest131rcu_preempt11:05:130
2552799482,42cyclictest29949-21ssh09:20:151
2552799463,40cyclictest1237-21ssh09:40:141
2552799463,38cyclictest14009-21ssh10:05:121
2552799456,7cyclictest131rcu_preempt12:10:141
2552799455,28cyclictest11414-21diskmemload09:45:151
2552799455,28cyclictest11414-21diskmemload09:45:151
25522994420,15cyclictest25276-21ssh12:35:110
25542994316,15cyclictest35966irq/58-eth008:10:133
2552299423,30cyclictest26897-21ssh10:30:130
25522994110,25cyclictest12-21ksoftirqd/010:10:130
2554299393,21cyclictest35966irq/58-eth011:30:123
2552299392,30cyclictest17933-21rm11:55:150
2552299392,2cyclictest131rcu_preempt08:25:140
2552799385,24cyclictest21-21ksoftirqd/107:20:151
2554299363,14cyclictest35966irq/58-eth008:45:103
25527993622,9cyclictest24660-21sh09:35:131
2552799356,26cyclictest8771-21ssh10:20:141
2552799346,20cyclictest201rcuc/112:15:141
2552299342,18cyclictest26965-21ssh12:00:130
25542993327,4cyclictest35966irq/58-eth009:15:113
2554299332,15cyclictest35966irq/58-eth008:25:143
2554299331,19cyclictest35966irq/58-eth011:05:123
2553499334,19cyclictest27346irq/40-dwc2_hso09:55:142
2553499334,19cyclictest27346irq/40-dwc2_hso09:00:132
2553499332,19cyclictest27346irq/40-dwc2_hso11:05:132
25527993312,15cyclictest21-21ksoftirqd/111:25:131
2552299331,26cyclictest12544-21ssh12:10:140
2553499321,12cyclictest8951-21sh11:50:162
2554299311,17cyclictest35966irq/58-eth010:35:143
2553499312,24cyclictest9978-21ssh11:14:342
2553499311,20cyclictest27346irq/40-dwc2_hso07:35:142
2552299312,21cyclictest16102-21ssh11:00:140
2554299304,17cyclictest35966irq/58-eth007:25:133
25542993013,12cyclictest35966irq/58-eth009:55:143
2552799303,18cyclictest30488-21ssh12:20:121
25522993013,4cyclictest12-21ksoftirqd/009:40:130
2552299301,21cyclictest73-21kswapd007:10:130
2553499293,21cyclictest0-21swapper/209:31:352
2553499293,18cyclictest27346irq/40-dwc2_hso12:30:132
2553499291,19cyclictest27346irq/40-dwc2_hso08:10:122
2553499291,16cyclictest7105-21ssh10:55:142
25534992910,15cyclictest27-21ksoftirqd/212:05:132
25534992910,15cyclictest27-21ksoftirqd/212:05:132
25527992911,4cyclictest131rcu_preempt11:15:141
2554299283,18cyclictest0-21swapper/311:14:343
2553499283,18cyclictest0-21swapper/209:10:142
2553499283,10cyclictest0-21swapper/210:13:362
2553499282,20cyclictest0-21swapper/208:35:152
2552799282,20cyclictest21201-21ssh12:30:331
25527992815,9cyclictest27285-21irqcore10:30:181
25527992814,8cyclictest21-21ksoftirqd/109:52:351
2552799281,25cyclictest12439-21ssh10:40:141
2554299274,14cyclictest35966irq/58-eth012:05:133
2554299274,14cyclictest35966irq/58-eth012:05:133
25542992717,5cyclictest35966irq/58-eth012:30:143
2554299271,14cyclictest35966irq/58-eth011:35:143
2553499275,13cyclictest27346irq/40-dwc2_hso12:35:112
2553499273,17cyclictest27346irq/40-dwc2_hso08:50:132
2553499273,17cyclictest27346irq/40-dwc2_hso08:05:182
2552799272,19cyclictest911-21ssh11:09:351
2552799272,19cyclictest7114-21ssh12:25:151
2552799272,19cyclictest2576-21ssh12:04:211
25527992711,12cyclictest21-21ksoftirqd/112:05:131
25527992711,12cyclictest21-21ksoftirqd/112:05:121
2552299272,20cyclictest3541-21ssh12:05:130
2552299272,20cyclictest3541-21ssh12:05:130
2554299267,16cyclictest35966irq/58-eth011:20:133
2554299267,16cyclictest35966irq/58-eth011:20:133
2554299264,17cyclictest0-21swapper/311:55:133
2554299261,14cyclictest35966irq/58-eth009:35:123
2553499263,16cyclictest27346irq/40-dwc2_hso07:25:142
2553499263,15cyclictest27346irq/40-dwc2_hso08:30:132
2553499262,20cyclictest17882-21grep10:05:272
2553499261,22cyclictest23822-21timerandwakeup11:20:292
2553499261,22cyclictest23822-21timerandwakeup11:20:292
2553499261,22cyclictest10340-21kthreadcore12:25:242
2553499261,13cyclictest0-21swapper/207:50:152
2552299263,16cyclictest20850-21ssh09:15:130
2553499252,15cyclictest27346irq/40-dwc2_hso10:50:152
2553499251,21cyclictest29243-21mailstats07:10:272
2553499251,16cyclictest31903-21kthreadcore07:15:202
25534992511,11cyclictest7454-21cron07:25:012
25522992516,6cyclictest0-21swapper/009:45:140
25522992516,6cyclictest0-21swapper/009:45:130
2552299251,17cyclictest7104-21sh10:55:140
25522992510,11cyclictest28355-21diskmemload09:55:140
2554299247,14cyclictest0-21swapper/310:20:213
2554299244,17cyclictest35966irq/58-eth007:30:143
2554299244,13cyclictest35966irq/58-eth012:35:113
2554299243,16cyclictest35966irq/58-eth012:25:153
25542992415,7cyclictest35966irq/58-eth010:45:123
2554299241,17cyclictest35966irq/58-eth009:40:143
2553499243,17cyclictest27346irq/40-dwc2_hso11:30:132
2553499242,19cyclictest7008-21kthreadcore10:00:212
2553499242,19cyclictest12837-21irqcore07:30:172
2553499242,15cyclictest14721-21ssh12:10:232
2553499242,14cyclictest13606-21basename08:05:012
2553499241,8cyclictest1019-21kthreadcore10:15:202
2553499241,14cyclictest26267-21ssh09:35:202
25534992410,5cyclictest21671-21ssh12:15:182
2552299249,5cyclictest131rcu_preempt10:45:210
2552299242,18cyclictest32573-21ssh11:25:260
2552299241,9cyclictest30720-21cut09:20:210
2552299241,6cyclictest30603-21df_inode10:50:160
2552299241,5cyclictest5411-21ntp_states11:10:260
2554299236,13cyclictest35966irq/58-eth011:25:133
2554299235,11cyclictest0-21swapper/311:00:153
25542992314,6cyclictest35966irq/58-eth009:30:143
2554299231,11cyclictest35966irq/58-eth009:10:143
2553499238,12cyclictest16598-21kthreadcore08:40:192
2553499233,12cyclictest0-21swapper/208:15:122
2553499232,6cyclictest0-21swapper/212:24:352
2553499232,16cyclictest0-21swapper/210:42:352
2553499231,19cyclictest14597-21diskstats11:35:162
2553499231,16cyclictest27346irq/40-dwc2_hso11:15:132
2553499230,7cyclictest0-21swapper/210:40:002
2552799235,11cyclictest0-21swapper/111:40:141
2552799234,11cyclictest21479-21diskmemload10:45:121
2552799232,15cyclictest25679-21apt-get07:10:131
25527992311,7cyclictest21-21ksoftirqd/107:40:111
2552799231,17cyclictest19250-21ssh09:30:261
25522992316,5cyclictest0-21swapper/010:00:150
2554299224,9cyclictest0-21swapper/311:15:133
2554299222,13cyclictest0-21swapper/310:10:133
2553499224,7cyclictest131rcu_preempt10:20:212
2553499222,12cyclictest27346irq/40-dwc2_hso08:55:152
2553499222,12cyclictest16780-21irqrtprio11:00:192
2553499221,9cyclictest2648366ssh10:30:012
25534992211,9cyclictest14774-21ssh09:29:372
25534992211,8cyclictest23100-21open_inodes09:50:302
2553499221,14cyclictest28834-21/usr/sbin/munin11:25:282
2553499221,12cyclictest7585-21kthreadcore09:05:202
2553499221,11cyclictest24882-21latency10:45:222
25534992210,9cyclictest0-21swapper/207:55:202
25534992210,5cyclictest27311-21ssh12:00:192
2553499220,20cyclictest24047-21kthreadcore09:15:212
2552799224,11cyclictest1712-21ssh11:10:141
2552799222,15cyclictest19830-21ssh11:20:141
2552799222,15cyclictest19830-21ssh11:20:141
2552799222,11cyclictest0-21swapper/111:53:471
25527992211,8cyclictest3364-21cat11:45:231
2552299224,3cyclictest131rcu_preempt09:30:250
2552299221,8cyclictest8738-21/usr/sbin/munin10:20:280
2552299221,8cyclictest1709-21ntp_192.168.11412:20:260
25522992210,3cyclictest12-21ksoftirqd/009:35:280
25542992113,6cyclictest0-21swapper/310:05:273
25542992113,5cyclictest0-21swapper/312:20:143
25542992112,6cyclictest0-21swapper/312:10:313
25542992111,7cyclictest35966irq/58-eth011:45:133
2554299211,13cyclictest35966irq/58-eth010:55:133
25542992110,6cyclictest35966irq/58-eth010:40:133
2553499219,10cyclictest28537-21kthreadcore10:30:202
2553499216,12cyclictest29919-21apt-get09:20:142
2553499211,6cyclictest18791-21kthreadcore11:55:202
2553499211,11cyclictest25257-21kthreadcore07:40:202
25534992110,9cyclictest17691-21ssh09:48:332
25534992110,9cyclictest17691-21ssh09:48:332
2552799212,13cyclictest131rcu_preempt10:26:121
2552799211,3cyclictest131rcu_preempt10:54:291
25527992112,6cyclictest6594-21ssh09:25:151
2552799211,17cyclictest20986-21cut09:15:141
2552799211,17cyclictest18885-21ls08:40:281
25527992111,7cyclictest13817-21ssh10:05:021
2552799211,16cyclictest12699-21ls08:00:321
25527992111,5cyclictest32194-21switchtime09:55:281
2552299212,16cyclictest22014-21latency_hist07:40:020
2552299212,16cyclictest17150-21missed_timers08:05:270
2552299212,15cyclictest3339-21ssh10:15:250
2552299211,8cyclictest10966-21diskstats11:15:160
2552299211,7cyclictest14294-21cat10:05:150
25522992114,5cyclictest19469-21ssh09:50:190
2552299211,13cyclictest12605-21cut07:30:150
2554299201,11cyclictest35966irq/58-eth009:50:153
2553499209,8cyclictest3419-21missed_timers11:45:232
2553499208,4cyclictest2920666ssh11:41:412
2553499201,9cyclictest1760-21kthreadcore08:25:222
2552799203,12cyclictest0-21swapper/107:55:161
2552799201,15cyclictest7690-21apt-get07:25:121
2552799201,15cyclictest23449-21ssh10:10:171
2552799201,14cyclictest25263-21apt12:35:101
2552799200,14cyclictest5771-21ls08:30:171
2552299203,15cyclictest1847-21latency_hist09:00:020
2552299202,16cyclictest22270-21fschecks_time07:40:140
2552299202,16cyclictest11728-21usage_Anybus07:25:290
2552299201,4cyclictest0-21swapper/010:36:330
25522992011,6cyclictest20488-21sort08:10:200
2552299201,10cyclictest11753-21usage_EthcTask12:25:320
2552299200,14cyclictest3069-21ls07:50:300
2554299198,8cyclictest0-21swapper/312:01:213
2554299196,8cyclictest35966irq/58-eth011:50:163
2554299191,9cyclictest0-21swapper/309:00:123
2554299191,12cyclictest0-21swapper/308:00:323
2554299191,12cyclictest0-21swapper/307:45:203
25542991910,6cyclictest35966irq/58-eth009:50:013
25542991910,6cyclictest35966irq/58-eth009:50:013
2553499192,15cyclictest27447-21irqcore07:45:182
2553499192,10cyclictest0-21swapper/208:20:152
2553499191,16cyclictest1197-21sort09:40:122
2553499191,15cyclictest20117-21fschecks_count08:45:142
2552799199,7cyclictest12404-21cut09:10:191
2552799197,9cyclictest6317-21cut11:30:191
2552799192,14cyclictest4328-21timerandwakeup08:25:311
2552799192,13cyclictest26556-21munin-run12:00:011
2552799191,16cyclictest28085-21ls08:50:221
25527991910,7cyclictest2014-21apt-get09:00:131
2552299193,13cyclictest25176-21/usr/sbin/munin12:15:270
2552299193,11cyclictest121ksoftirqd/008:30:130
25522991912,5cyclictest12263-21ssh10:40:020
2554299189,6cyclictest35966irq/58-eth012:15:143
2554299189,4cyclictest0-21swapper/310:28:293
2554299186,8cyclictest0-21swapper/311:40:143
2554299184,10cyclictest0-21swapper/310:15:153
2554299182,10cyclictest0-21swapper/308:30:153
25542991811,4cyclictest0-21swapper/310:31:353
25542991810,6cyclictest0-21swapper/310:00:233
2552799189,7cyclictest28253-21apt-get08:20:141
2552799189,7cyclictest13826-21apt-get08:05:161
2552799187,8cyclictest23438-21apt-get08:15:121
2552799182,12cyclictest6627-21apt-get10:55:021
25527991813,3cyclictest0-21swapper/107:15:191
2552799181,2cyclictest131rcu_preempt10:36:481
25527991812,4cyclictest0-21swapper/108:10:271
2552799181,14cyclictest302-21kthreadcore07:50:211
2552799181,14cyclictest27722-21tr07:45:191
2552299189,6cyclictest23514-21df11:40:150
2552299184,5cyclictest12-21ksoftirqd/007:20:020
2552299183,13cyclictest7842-21kthreadcore09:05:200
2552299183,13cyclictest1584-21tr11:45:200
2552299183,12cyclictest6498-21ssh09:25:130
2552299183,12cyclictest14935-21awk08:40:010
2552299183,12cyclictest14845-21irqcore11:35:190
2552299182,14cyclictest27878-21kthreadcore08:50:220
25522991811,5cyclictest19650-21ssh11:20:030
25522991811,5cyclictest19650-21ssh11:20:020
25522991810,6cyclictest24210-21ssh10:27:200
2554299179,5cyclictest35966irq/58-eth009:20:153
2554299178,6cyclictest35966irq/58-eth010:50:153
2554299176,8cyclictest35966irq/58-eth007:15:133
2554299173,7cyclictest0-21swapper/308:05:183
2554299173,11cyclictest0-21swapper/309:25:353
2554299172,10cyclictest0-21swapper/307:10:143
2552799172,11cyclictest17878-21kthreadcore11:00:221
2552799171,9cyclictest23447-21ntp_offset08:45:231
25527991710,4cyclictest10685-21irqcore08:35:191
2552299175,9cyclictest19943-21latency_hist08:45:020
2552299172,12cyclictest3354-21kthreadcore09:00:200
2552299172,11cyclictest0-21swapper/008:45:120
2552799162,11cyclictest29777-21fschecks_time08:55:181
2552799162,10cyclictest0-21swapper/109:05:151
2552799161,6cyclictest12908-21sed07:30:171
2552799161,12cyclictest19538-21kthreadcore07:35:211
2552299168,6cyclictest9056-21diskstats08:00:170
2552299164,9cyclictest3073-21fschecks_time07:20:170
2552299162,12cyclictest4162-21/usr/sbin/munin07:55:160
2552299162,10cyclictest0-21swapper/008:20:160
2552299162,10cyclictest0-21swapper/008:15:130
2552299161,12cyclictest29266-21cut07:45:210
2554299159,4cyclictest0-21swapper/307:50:153
2554299151,9cyclictest0-21swapper/308:55:263
2554299151,8cyclictest35966irq/58-eth008:40:143
2554299141,9cyclictest0-21swapper/307:35:133
2554299141,11cyclictest0-21swapper/308:20:293
2554299141,10cyclictest0-21swapper/308:35:213
2553499148,4cyclictest27-21ksoftirqd/207:10:012
2552799141,10cyclictest25552-21ls07:10:011
2554299138,3cyclictest0-21swapper/308:15:043
2554299133,5cyclictest0-21swapper/307:55:163
2554299132,9cyclictest0-21swapper/307:20:153
2552299131,9cyclictest25442-21latency_hist07:10:010
2554299122,6cyclictest0-21swapper/308:50:143
2554299122,6cyclictest0-21swapper/307:40:123
2554299111,7cyclictest0-21swapper/309:05:163
2554299111,7cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional