You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-07 - 14:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Sun Jun 07, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3038299773,58cyclictest18150-21ssh10:40:141
3039099538,18cyclictest27-21ksoftirqd/208:55:122
30390995318,26cyclictest27-21ksoftirqd/210:20:132
3038299504,5cyclictest131rcu_preempt12:00:121
3038299504,5cyclictest131rcu_preempt12:00:121
3039099472,31cyclictest11519-21ssh11:30:132
3038299472,4cyclictest131rcu_preempt12:20:121
3037499464,7cyclictest131rcu_preempt10:45:130
3039099454,19cyclictest27-21ksoftirqd/212:15:152
3038299451,11cyclictest7741-21ssh11:10:131
3037499453,37cyclictest25636-21ssh09:15:140
3037499453,37cyclictest25636-21ssh09:15:130
3037499451,37cyclictest31082-21ssh11:05:130
3038299442,36cyclictest11479-21ssh09:25:111
3037499443,38cyclictest25878-21ssh11:20:140
3038299433,7cyclictest131rcu_preempt12:05:141
3038299433,7cyclictest131rcu_preempt12:05:141
3037499433,36cyclictest32374-21rm10:30:110
3037499421,33cyclictest29616-21ssh11:40:110
3039899408,24cyclictest35966irq/58-eth010:10:123
30390994016,14cyclictest0-21swapper/209:20:112
3037499401,36cyclictest19343-21ssh10:05:120
3039099392,20cyclictest261rcuc/210:40:122
30382993928,7cyclictest0-21swapper/111:45:141
3039899383,25cyclictest35966irq/58-eth008:05:133
3038299382,10cyclictest15018-21latency_hist08:35:031
3038299381,9cyclictest6719-21apt-get09:00:111
3037499382,27cyclictest0-21swapper/008:35:120
3039899374,21cyclictest35966irq/58-eth011:50:123
30390993715,9cyclictest0-21swapper/211:20:132
3038299373,9cyclictest18557-21wc08:35:251
3037499373,26cyclictest5067-21ssh10:15:130
3038299362,9cyclictest27116-21kthreadcore11:20:201
3038299362,11cyclictest23302-21kthreadcore11:35:221
3038299361,9cyclictest712-21kthreadcore11:05:231
3038299361,11cyclictest504-21kernelversion12:35:201
3038299361,11cyclictest12984-21switchtime10:35:281
3038299361,10cyclictest32112-21cut08:50:211
3038299361,10cyclictest26092-21kthreadcore12:30:211
3039099353,17cyclictest261rcuc/211:50:142
3038299358,5cyclictest26452-21kthreadcore09:50:201
3038299353,11cyclictest31966-21kthreadcore07:10:201
3038299351,9cyclictest15440-21/usr/sbin/munin09:45:271
3038299351,25cyclictest12971-21ssh10:55:141
3038299351,10cyclictest25852-21fschecks_time09:15:171
3038299351,10cyclictest25852-21fschecks_time09:15:161
3038299351,10cyclictest19216-21cut08:05:191
3037499351,25cyclictest24527-21sh09:50:130
3039899343,19cyclictest35966irq/58-eth012:35:143
3038299346,7cyclictest5470-21cut10:50:201
3038299344,8cyclictest21509-21kthreadcore12:10:221
3038299342,11cyclictest17059-21cut09:10:181
3038299341,9cyclictest571-21kthreadcore07:45:191
3038299341,9cyclictest22911-21kthreadcore11:00:181
3038299341,9cyclictest18973-21kthreadcore11:15:221
3038299341,10cyclictest300-21apt-get08:20:021
3038299341,10cyclictest24314-21kthreadcore08:10:181
30374993418,12cyclictest0-21swapper/007:30:120
3039099331,16cyclictest6344-21ssh09:40:132
3039099331,11cyclictest21985-21ssh11:00:122
3038299332,10cyclictest5904-21apt11:45:011
3038299331,9cyclictest23498-21ssh10:25:151
3038299331,9cyclictest17037-21kthreadcore11:50:201
3038299331,9cyclictest1103-21apt-get09:55:131
3038299331,9cyclictest10033-21latency_hist08:30:011
3038299331,8cyclictest22017-21kthreadcore09:30:201
3037499333,22cyclictest28534-21ssh12:15:150
3039899323,20cyclictest35966irq/58-eth010:25:123
3039099325,20cyclictest0-21swapper/208:10:132
3038299328,3cyclictest16074-21diskmemload10:10:251
3038299327,5cyclictest5822-21sed09:20:221
3038299322,9cyclictest4376-21kthreadcore07:15:211
3038299322,10cyclictest9184-21kthreadcore07:20:191
3038299322,10cyclictest9184-21kthreadcore07:20:181
3038299321,9cyclictest4085-21/usr/sbin/munin07:50:241
3038299321,9cyclictest3691-21ssh10:30:241
3038299321,9cyclictest30196-21cut09:35:181
3038299321,9cyclictest12775-21hwlatdetect07:25:151
3038299321,10cyclictest5032-21/usr/sbin/munin10:15:281
3038299321,10cyclictest29808-21kthreadcore12:15:201
3038299321,10cyclictest14407-21fschecks_count10:20:171
3038299320,10cyclictest16254-21ssh09:10:011
3037499322,11cyclictest0-21swapper/007:25:130
3039099314,15cyclictest27346irq/40-dwc2_hso12:20:132
30390993110,14cyclictest27-21ksoftirqd/212:10:132
3038299311,9cyclictest8315-21cut09:40:201
3038299311,11cyclictest2088-21fschecks_time08:55:151
3038299311,11cyclictest20143-21idleruntime-cro11:35:011
3038299311,10cyclictest9009-21tune2fs07:55:161
3038299311,10cyclictest300-21apt-get08:20:051
3038299311,10cyclictest29567-21kthreadcore10:45:221
3038299311,10cyclictest24545-21latency_hist08:45:011
3039899302,17cyclictest35966irq/58-eth008:25:113
3039099303,14cyclictest19363-21ssh10:05:132
3039099302,21cyclictest0-21swapper/207:35:152
3039099301,15cyclictest16830-21ssh11:15:142
3038299304,9cyclictest12803-21kthreadcore10:00:211
3038299304,9cyclictest12803-21kthreadcore10:00:201
3038299301,9cyclictest26644-21cron07:40:011
3038299301,9cyclictest24867-21grep08:45:161
3038299301,9cyclictest24479-21fschecks_time11:55:161
3038299301,10cyclictest4929-21kthreadcore11:25:221
3038299301,10cyclictest19624-21sed10:05:151
3038299301,10cyclictest14176-21irqcore08:00:171
3037499303,22cyclictest985-21ssh12:00:140
3037499303,22cyclictest985-21ssh12:00:130
3039899299,3cyclictest0-21swapper/310:05:533
3039899293,18cyclictest35966irq/58-eth007:20:113
3039899293,18cyclictest35966irq/58-eth007:20:113
3039899291,15cyclictest35966irq/58-eth007:40:133
3039099295,15cyclictest27346irq/40-dwc2_hso09:45:132
3039099292,20cyclictest10221-21sh10:00:132
3039099292,20cyclictest10221-21sh10:00:132
3038299291,10cyclictest14340-21fschecks_time12:25:161
3038299291,10cyclictest14340-21fschecks_time12:25:151
3038299290,10cyclictest30358-21awk07:40:261
30374992915,10cyclictest2339-21ssh09:20:120
3039899283,15cyclictest35966irq/58-eth010:40:133
30398992810,3cyclictest0-21swapper/311:47:453
30390992819,5cyclictest9080-21ssh10:35:142
3038299281,9cyclictest20473-21kthreadcore07:30:221
3037499283,14cyclictest844-21munin-node07:35:140
30374992813,7cyclictest12-21ksoftirqd/008:00:130
3039899278,3cyclictest0-21swapper/312:15:533
3039899272,15cyclictest35966irq/58-eth012:00:133
3039899272,15cyclictest35966irq/58-eth012:00:123
3039099274,15cyclictest27346irq/40-dwc2_hso07:55:122
3039099272,8cyclictest27308-21ssh10:45:142
3039099272,19cyclictest0-21swapper/208:05:142
3039099271,9cyclictest14134-21ssh12:25:132
3039099271,9cyclictest14134-21ssh12:25:122
3037499272,20cyclictest2510-21ssh11:25:130
3037499272,20cyclictest10055-21ssh12:05:150
3037499272,20cyclictest10055-21ssh12:05:140
3037499272,15cyclictest14242-21ssh10:20:130
3039899267,3cyclictest0-21swapper/311:43:533
3039899263,12cyclictest35966irq/58-eth009:25:123
3039899261,10cyclictest0-21swapper/311:00:193
3039899261,10cyclictest0-21swapper/309:10:163
3039099262,15cyclictest27-21ksoftirqd/212:00:132
3039099262,15cyclictest27-21ksoftirqd/212:00:122
3039099262,11cyclictest0-21swapper/209:50:142
3039099261,16cyclictest27346irq/40-dwc2_hso09:00:122
3037499262,17cyclictest3968-21ssh10:50:150
3039899258,6cyclictest0-21swapper/309:44:113
3039899255,6cyclictest0-21swapper/309:35:213
3039899254,6cyclictest0-21swapper/312:20:213
30398992510,3cyclictest0-21swapper/310:30:263
30398992510,3cyclictest0-21swapper/310:15:073
3039099254,16cyclictest0-21swapper/211:05:132
3039099252,6cyclictest0-21swapper/209:25:122
3039099251,9cyclictest8092-21ssh12:39:382
3039099251,19cyclictest4556-21ssh09:55:222
3039099251,15cyclictest23335-21ssh10:25:142
3037499253,15cyclictest844-21munin-node07:55:140
3037499252,16cyclictest20546-21ssh11:35:130
3037499252,16cyclictest14150-21ssh12:25:140
3037499252,16cyclictest14150-21ssh12:25:130
3037499251,19cyclictest23183-21ssh12:30:120
3039899249,3cyclictest0-21swapper/311:57:033
3039899241,11cyclictest0-21swapper/308:50:163
30398992410,3cyclictest0-21swapper/309:32:013
3039099243,18cyclictest27346irq/40-dwc2_hso08:50:122
3039099243,18cyclictest27017-21fschecks_count07:40:162
3039099243,13cyclictest0-21swapper/208:00:122
3039099242,14cyclictest20568-21apt-get09:30:112
3039099241,9cyclictest13879-21switchtime08:30:262
30390992413,9cyclictest18892-21ssh10:56:472
3039899231,7cyclictest0-21swapper/310:00:133
3039899231,7cyclictest0-21swapper/310:00:133
3039099236,14cyclictest20056-21fschecks_time08:40:152
3039099233,13cyclictest3967-21ssh10:50:142
3039099232,16cyclictest0-21swapper/212:09:382
3039099232,16cyclictest0-21swapper/212:09:382
30390992316,5cyclictest6973-21ssh11:45:192
3039099231,15cyclictest27346irq/40-dwc2_hso07:45:142
3039099231,11cyclictest28038-21ssh12:30:322
3037499237,3cyclictest131rcu_preempt11:15:300
3037499231,19cyclictest32073-21ls10:30:010
3039899229,10cyclictest0-21swapper/310:47:233
3039899224,13cyclictest0-21swapper/311:15:143
3039899222,11cyclictest35966irq/58-eth011:30:123
30398992213,6cyclictest0-21swapper/310:50:173
3039099224,12cyclictest0-21swapper/207:20:132
3039099224,12cyclictest0-21swapper/207:20:122
3039099222,5cyclictest0-21swapper/209:05:132
3039099221,9cyclictest2686-21fschecks_count11:25:152
3039099221,15cyclictest27346irq/40-dwc2_hso08:20:042
3039099221,13cyclictest20940-21ssh11:35:172
3039099221,10cyclictest32337-21acpi10:30:102
3039099221,10cyclictest25437-21kthreadcore11:55:202
3037499222,16cyclictest32248-21ssh12:35:140
30374992211,4cyclictest12-21ksoftirqd/007:16:370
30374992210,5cyclictest12-21ksoftirqd/008:50:130
3039899212,12cyclictest35966irq/58-eth008:40:133
3039099219,9cyclictest28645-21fschecks_time10:10:142
3039099218,10cyclictest7213-21kthreadcore07:50:212
3039099218,10cyclictest5792-21kernelversion10:15:202
3039099217,11cyclictest5579-21diskstats08:25:152
3039099217,11cyclictest10373-21cut11:10:232
3039099212,16cyclictest31656-21kthreadcore09:35:222
30390992113,5cyclictest2707-21cron07:15:012
3039099211,16cyclictest983-21proc_pri11:40:272
3039099211,16cyclictest7510-21grep07:20:012
3039099211,16cyclictest28010-21cat08:45:242
3039099211,15cyclictest27346irq/40-dwc2_hso08:35:122
3039099210,11cyclictest19514-21ssh09:10:222
3037499213,16cyclictest12991-21ssh10:55:140
3037499212,14cyclictest844-21munin-node10:35:140
3037499211,17cyclictest10282-21ls11:45:290
30374992110,8cyclictest9425-21cat12:20:290
3037499210,16cyclictest30639-21cut07:10:170
30398992012,6cyclictest0-21swapper/309:51:423
30398992011,7cyclictest0-21swapper/310:59:513
3039899201,13cyclictest0-21swapper/308:35:213
3039099201,9cyclictest30402-21latency_hist07:10:012
30390992014,4cyclictest27-21ksoftirqd/207:25:172
3039099201,10cyclictest28734-21irqrtprio08:15:202
3039099201,10cyclictest25802-21fschecks_count09:15:162
3039099201,10cyclictest25802-21fschecks_count09:15:162
3039099201,10cyclictest21964-21latency_hist07:35:012
3037499201,7cyclictest18244-21df10:40:140
30374992013,4cyclictest28366-21/usr/sbin/munin10:10:250
30374992012,6cyclictest1-21systemd07:44:260
3037499201,13cyclictest18928-21ssh11:50:260
3037499200,16cyclictest23252-21ls08:10:010
3039899199,7cyclictest0-21swapper/312:33:343
3039899191,12cyclictest0-21swapper/308:50:013
3039899191,12cyclictest0-21swapper/307:55:163
3039899191,12cyclictest0-21swapper/307:45:163
30398991910,6cyclictest35966irq/58-eth011:25:143
30398991910,6cyclictest0-21swapper/310:20:213
3037499199,3cyclictest12-21ksoftirqd/009:38:280
3037499198,8cyclictest7072-21kthreadcore09:40:180
3037499193,6cyclictest14146-21grep08:30:270
3037499193,12cyclictest12-21ksoftirqd/009:55:110
3037499191,9cyclictest15041-21sendmail_mailqu09:05:260
3037499191,14cyclictest7770-21sh11:10:140
3037499191,10cyclictest0-21swapper/011:02:400
3037499190,16cyclictest25293-21ls09:15:020
3037499190,15cyclictest20206-21latency_hist09:30:020
3039899189,7cyclictest0-21swapper/311:23:403
3039899189,6cyclictest35966irq/58-eth009:15:133
3039899189,6cyclictest35966irq/58-eth009:15:133
3039899189,6cyclictest0-21swapper/311:11:443
3039899189,6cyclictest0-21swapper/310:37:103
3039899188,8cyclictest0-21swapper/312:27:503
3039899188,8cyclictest0-21swapper/312:27:493
3039899187,5cyclictest36066irq/59-eth011:35:213
3039899182,10cyclictest0-21swapper/307:50:143
3039899182,10cyclictest0-21swapper/307:15:143
3039899181,12cyclictest0-21swapper/308:15:173
30398991810,6cyclictest35966irq/58-eth009:20:113
30398991810,5cyclictest35966irq/58-eth012:10:133
3038299181,13cyclictest30480-21wc07:10:031
3037499189,6cyclictest17242-21cut09:45:200
3037499185,9cyclictest29407-21cat09:35:020
3037499182,11cyclictest0-21swapper/009:00:140
3037499182,11cyclictest0-21swapper/008:10:140
3037499181,13cyclictest22748-21sh12:10:250
3037499181,13cyclictest12-21ksoftirqd/011:55:130
3037499181,11cyclictest11612-21cpuspeed11:30:140
3039899179,6cyclictest0-21swapper/311:06:363
3039899179,6cyclictest0-21swapper/309:58:533
3039899174,8cyclictest35966irq/58-eth009:45:143
3039899171,12cyclictest0-21swapper/312:05:173
3039899171,12cyclictest0-21swapper/312:05:173
3039899171,12cyclictest0-21swapper/308:20:183
3039899171,10cyclictest0-21swapper/308:30:283
3037499178,7cyclictest20162-21if_eth108:40:180
3037499171,13cyclictest8221-21tr08:25:200
3037499171,13cyclictest6475-21munin-run09:00:010
3037499171,13cyclictest25126-21sed08:45:180
3037499171,12cyclictest10193-21apt-get10:00:130
3037499171,12cyclictest10193-21apt-get10:00:120
3037499171,11cyclictest0-21swapper/008:15:150
3039899161,8cyclictest0-21swapper/308:00:123
3037499169,5cyclictest0-21swapper/007:50:190
3037499162,9cyclictest0-21swapper/007:45:140
3037499161,8cyclictest647-21find08:20:120
3039899152,10cyclictest0-21swapper/309:05:133
3037499155,7cyclictest30439-21cat07:10:020
3037499151,11cyclictest7748-21apt-get07:20:130
3037499151,11cyclictest7748-21apt-get07:20:120
3039899147,5cyclictest35966irq/58-eth007:10:133
3039899138,3cyclictest0-21swapper/307:38:153
3039899131,9cyclictest0-21swapper/308:15:023
3039899131,8cyclictest0-21swapper/307:30:133
3039899131,6cyclictest0-21swapper/309:00:163
3039899127,3cyclictest0-21swapper/307:25:113
3039899111,7cyclictest0-21swapper/308:55:153
3039899111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional