You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-29 - 17:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri May 29, 2026 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16412996345,10cyclictest0-21swapper/109:25:121
1640499572,47cyclictest31479-21ssh11:00:130
1640499552,33cyclictest15919-21ssh11:45:120
1641299513,9cyclictest10677-21sshd12:00:131
1642099431,26cyclictest1614-21ssh11:55:122
16404994324,12cyclictest30288-21apt-get09:30:120
16404994324,12cyclictest30288-21apt-get09:30:120
16428994225,12cyclictest35966irq/58-eth011:10:123
1642099421,37cyclictest0-21swapper/209:45:162
16404994131,5cyclictest32444-21apt-get12:30:140
16404994116,21cyclictest28956-21ssh10:05:130
1642099402,22cyclictest32461-21ssh12:30:142
16420994014,12cyclictest25041-21sh11:50:142
1641299402,8cyclictest3185-21ssh11:20:131
1641299402,4cyclictest131rcu_preempt12:20:111
16404994026,10cyclictest0-21swapper/012:35:110
1640499393,28cyclictest1623-21ssh11:55:120
1642899383,23cyclictest35966irq/58-eth009:35:133
1642899383,17cyclictest35966irq/58-eth010:50:123
1642899374,18cyclictest35966irq/58-eth009:45:163
1642899372,23cyclictest35966irq/58-eth010:10:143
1642099372,29cyclictest13328-21ssh10:50:122
1642099371,28cyclictest0-21swapper/209:00:112
1641299373,10cyclictest30245-21ssh11:35:141
16404993726,7cyclictest21275-21apt-get09:05:120
1641299365,23cyclictest28679-21ssh12:10:111
1641299351,29cyclictest23401-21ssh12:25:121
1642899344,18cyclictest35966irq/58-eth012:05:123
1642899342,19cyclictest35966irq/58-eth012:30:153
1642899342,18cyclictest35966irq/58-eth010:00:113
16428993413,10cyclictest35966irq/58-eth011:00:133
1641299346,4cyclictest0-21swapper/108:25:121
1641299342,22cyclictest12176-21ssh11:25:111
16404993427,5cyclictest26332-21apt-get09:10:120
16404993410,10cyclictest24907-21latency_hist09:45:020
1641299337,6cyclictest5257-21basename10:10:021
1641299335,25cyclictest6873-21ssh11:40:121
1641299332,26cyclictest18548-21rm10:35:131
16404993325,4cyclictest21609-21apt-get07:20:130
1640499332,19cyclictest25125-21ssh09:45:160
1642899321,25cyclictest35966irq/58-eth011:20:123
1642099323,19cyclictest27346irq/40-dwc2_hso10:00:122
1641299324,24cyclictest30291-21ssh09:30:111
1641299324,24cyclictest30291-21ssh09:30:111
1641299322,27cyclictest13318-21ssh10:50:111
16404993215,12cyclictest0-21swapper/007:10:100
16404993211,12cyclictest0-21swapper/010:25:150
1642899311,15cyclictest35966irq/58-eth012:20:123
1641299312,23cyclictest32448-21rm12:30:131
1641299311,26cyclictest9468-21ssh10:30:111
1640499317,9cyclictest19413-21munin-run10:00:010
1640499317,10cyclictest29591-21ssh10:21:390
16404993115,13cyclictest22155-21tr08:15:200
16404993114,12cyclictest0-21swapper/009:50:110
16404993111,15cyclictest0-21swapper/011:20:140
1642899303,12cyclictest35966irq/58-eth011:15:113
1642099303,16cyclictest27346irq/40-dwc2_hso08:00:132
1641299302,10cyclictest523-21grep10:25:171
1641299301,10cyclictest32419-21latency_hist10:25:011
16404993026,2cyclictest28661-21apt-get12:10:110
16404993022,4cyclictest26631-21apt-get07:30:130
1642899292,11cyclictest35966irq/58-eth008:50:123
16428992914,11cyclictest35966irq/58-eth011:35:133
1642099292,24cyclictest0-21swapper/210:20:122
1642099291,16cyclictest28965-21ssh10:05:142
1640499298,8cyclictest29884-21ssh09:29:590
1640499295,11cyclictest6597-21kthreadcore10:10:200
1640499293,21cyclictest25061-21ssh11:50:150
16404992924,2cyclictest6744-21apt-get08:50:120
1640499287,8cyclictest18143-21rm12:03:420
1640499287,7cyclictest25854-21diskmemload10:37:300
1640499286,9cyclictest12636-21sh11:41:340
1640499286,8cyclictest24797-21rm10:00:360
1640499286,8cyclictest21499-21sed07:20:020
1640499286,8cyclictest21499-21sed07:20:010
1640499282,11cyclictest0-21swapper/008:05:280
16404992816,8cyclictest9484-21ssh10:30:120
16404992815,8cyclictest0-21swapper/012:15:130
1642099273,17cyclictest27346irq/40-dwc2_hso09:55:122
1642099272,19cyclictest0-21swapper/208:50:132
1642099272,17cyclictest27346irq/40-dwc2_hso10:15:122
1642099272,11cyclictest0-21swapper/208:15:112
16420992719,3cyclictest26481-21ssh11:15:102
1642099271,21cyclictest16046-21ssh09:40:142
1641299274,13cyclictest12016-21ssh09:20:131
1641299272,20cyclictest17491-21ssh11:10:121
1640499277,8cyclictest20584-21ssh09:24:490
1640499277,7cyclictest25854-21diskmemload11:06:070
1640499276,8cyclictest31735-21ssh11:15:550
1640499274,8cyclictest23303-21ssh10:19:580
16404992713,10cyclictest0-21swapper/011:25:130
1642899262,18cyclictest0-21swapper/311:40:133
1642899261,18cyclictest35966irq/58-eth009:10:133
1642099264,14cyclictest27346irq/40-dwc2_hso09:05:122
1642099262,16cyclictest27346irq/40-dwc2_hso09:35:132
1642099261,18cyclictest31801-21cut07:40:152
1641299263,18cyclictest19644-21ssh12:05:141
1640499266,17cyclictest0-21swapper/010:45:140
1640499265,10cyclictest21385-21ssh10:54:160
1640499262,9cyclictest24245-21/usr/sbin/munin08:25:290
1640499262,9cyclictest13539-21ssh09:37:310
16404992620,3cyclictest19617-21apt-get12:05:130
1640499261,7cyclictest0-21swapper/012:29:070
1642899254,11cyclictest0-21swapper/312:15:113
1642899252,13cyclictest35966irq/58-eth009:50:113
1642899252,11cyclictest35966irq/58-eth008:05:123
1642099251,13cyclictest26263-21ls07:25:012
1642099251,10cyclictest0-21swapper/207:10:112
1642099250,16cyclictest12470-21ls11:25:162
1641299251,17cyclictest4285-21ssh10:45:131
1640499257,11cyclictest0-21swapper/008:00:120
1640499256,3cyclictest0-21swapper/011:30:150
1640499252,8cyclictest3034-21ssh11:35:560
16404992512,9cyclictest26172-21sed11:15:010
16428992414,7cyclictest0-21swapper/309:30:213
16428992414,7cyclictest0-21swapper/309:30:203
1642099241,18cyclictest27346irq/40-dwc2_hso11:45:112
1641299241,11cyclictest27675-21ssh10:40:121
1640499241,8cyclictest25854-21diskmemload10:59:440
16404992417,3cyclictest4268-21apt-get07:50:090
1642899234,11cyclictest0-21swapper/309:55:143
1642099233,11cyclictest20753-21sh09:25:012
1642099232,13cyclictest27346irq/40-dwc2_hso11:35:122
1642099232,12cyclictest23080-21sed10:55:192
1642099230,9cyclictest23553-21grep12:25:162
1641299232,18cyclictest21271-21ssh11:30:151
16412992314,6cyclictest2596-21sed11:00:241
16404992317,3cyclictest16377-21apt-get09:00:120
16404992312,7cyclictest14317-21apt-get12:20:110
16428992213,6cyclictest0-21swapper/312:10:353
1642099228,12cyclictest5860-21fschecks_time10:10:172
1642099228,11cyclictest24263-21apt-get08:25:122
1642099222,16cyclictest26491-21fschecks_count09:10:152
1642099222,14cyclictest27346irq/40-dwc2_hso12:20:112
16420992211,9cyclictest3594-21ssh09:31:322
16420992211,9cyclictest3594-21ssh09:31:312
16420992211,8cyclictest18724-21sh08:10:002
1642099221,11cyclictest19249-21grep12:05:012
16420992210,10cyclictest5933-21usage_Anybus09:50:302
1641299221,16cyclictest14717-21ssh10:15:131
1640499221,7cyclictest8127-21ssh09:15:530
16404992212,7cyclictest5402-21ntp_states08:45:220
16404992210,8cyclictest28405-21kernelversion10:40:190
1642099219,9cyclictest4520-21sh11:20:202
1642099219,4cyclictest28638-21/usr/sbin/munin12:10:132
1642099213,11cyclictest11033-21kthreadcore12:35:202
1642099212,9cyclictest14110-21latency_hist12:20:022
1642099212,11cyclictest10400-21kthreadcore11:05:212
1642099211,18cyclictest17457-21find11:10:102
16420992111,3cyclictest2614-21missed_timers11:00:242
1642099211,12cyclictest4499-21ls07:50:122
1642099211,11cyclictest18541-21apt-get10:35:142
1642099211,10cyclictest24471-21kthreadcore11:30:242
1641299215,2cyclictest131rcu_preempt09:13:341
1641299214,8cyclictest131rcu_preempt09:55:251
16412992112,7cyclictest25114-21apt-get09:45:151
16412992112,6cyclictest13947-21sh12:20:011
1641299211,17cyclictest20071-21ls09:00:261
1641299211,16cyclictest2903-21find09:15:101
1641299210,18cyclictest21441-21ls07:20:011
1641299210,18cyclictest21441-21ls07:20:011
1640499218,9cyclictest19053-21cat08:15:020
1640499214,13cyclictest581-21kthreadcore07:40:200
1640499211,17cyclictest9127-21ls08:00:020
16404992110,8cyclictest29062-21sshd08:34:050
16404992110,8cyclictest29062-21sshd08:34:050
1642899205,10cyclictest0-21swapper/307:50:083
1642899203,12cyclictest0-21swapper/310:05:143
16428992012,5cyclictest35966irq/58-eth009:20:133
16428992010,8cyclictest0-21swapper/310:20:103
1642899200,11cyclictest35966irq/58-eth010:45:153
1642099206,11cyclictest3929-21kthreadcore09:15:192
1642099203,13cyclictest27346irq/40-dwc2_hso11:40:132
1642099202,15cyclictest2048-21fschecks_count08:45:122
1642099202,15cyclictest12700-21kthreadcore08:55:192
1642099201,9cyclictest9542-21cpuspeed10:30:132
1642099201,9cyclictest23077-21kthreadcore09:25:212
1642099201,15cyclictest5365-21kthreadcore10:45:202
1642099201,15cyclictest31439-21sendmail07:40:012
1642099201,11cyclictest833-21sed10:25:202
1642099201,10cyclictest21365-21munin-run07:20:012
1642099201,10cyclictest21365-21munin-run07:20:012
1642099200,14cyclictest27346irq/40-dwc2_hso12:05:122
1642099200,11cyclictest28523-21sh10:40:192
1641299208,9cyclictest25143-21grep09:05:281
1641299206,11cyclictest30418-21grep07:30:271
16412992012,5cyclictest27240-21kthreadcore11:15:181
1641299201,10cyclictest2377-21grep11:55:181
1640499209,8cyclictest4011-21sshd07:46:050
1640499209,8cyclictest12268-21grep08:55:190
1640499207,10cyclictest24147-21cat08:25:010
1640499200,17cyclictest26501-21ls07:30:020
1642899199,8cyclictest0-21swapper/311:26:083
1642899198,8cyclictest0-21swapper/311:47:293
1642899198,7cyclictest0-21swapper/309:40:163
1642899193,7cyclictest0-21swapper/308:45:113
1642899192,10cyclictest35966irq/58-eth010:30:123
1642899192,10cyclictest0-21swapper/309:05:043
1642899191,14cyclictest35966irq/58-eth010:44:463
1642899191,12cyclictest35966irq/58-eth010:35:153
1642899191,12cyclictest0-21swapper/307:40:193
1642899191,11cyclictest0-21swapper/310:25:153
16428991910,6cyclictest0-21swapper/311:09:493
16428991910,6cyclictest0-21swapper/309:19:053
16428991910,5cyclictest35966irq/58-eth011:53:313
1642099192,9cyclictest8846-21latency_hist07:55:022
1642099192,8cyclictest32685-21awk08:35:222
1642099191,9cyclictest27126-21sed07:30:172
1642099191,8cyclictest1700-21munin-plugin-st08:45:012
1641299192,14cyclictest9554-21grep08:00:161
1641299191,14cyclictest6955-21apt-get09:35:131
1641299190,16cyclictest14130-21find08:05:101
16404991910,6cyclictest1679-21sshd08:44:060
1642899189,6cyclictest0-21swapper/312:01:503
1642899189,6cyclictest0-21swapper/310:15:223
1642899188,8cyclictest0-21swapper/309:28:413
1642899182,9cyclictest35966irq/58-eth007:30:133
1642899182,10cyclictest0-21swapper/312:35:123
1642899182,10cyclictest0-21swapper/309:00:123
1642899182,10cyclictest0-21swapper/307:10:123
16428991811,5cyclictest0-21swapper/311:55:163
16428991810,5cyclictest0-21swapper/312:30:023
1642099186,9cyclictest4030-21cron07:50:012
1642099181,9cyclictest24191-21latency_hist08:25:022
1642099181,8cyclictest29138-21cat08:35:022
1642099181,8cyclictest29138-21cat08:35:012
1642099181,8cyclictest18963-21sh08:15:012
1641299188,7cyclictest3022-21kworker/u8:010:00:131
1641299182,11cyclictest8476-21ssh11:05:141
16412991811,4cyclictest3114-21grep07:40:281
1641299181,14cyclictest2545-21irqrtprio08:45:141
1641299181,12cyclictest26042-21ssh10:55:261
1640499187,8cyclictest29557-21basename08:35:150
1640499185,10cyclictest31451-21basename07:40:020
1642899179,6cyclictest0-21swapper/310:56:463
1642899177,8cyclictest0-21swapper/311:31:103
1642899172,9cyclictest0-21swapper/308:35:133
1642899172,9cyclictest0-21swapper/308:25:133
1642899171,13cyclictest0-21swapper/308:55:173
1642099171,13cyclictest26405-21idleruntime-cro07:30:012
1641299174,2cyclictest131rcu_preempt12:35:111
1641299173,9cyclictest25062-21sshd11:50:151
1641299173,11cyclictest5646-21rm10:10:131
1641299172,9cyclictest0-21swapper/107:20:141
1641299172,12cyclictest16470-21irqcore09:40:171
1641299171,12cyclictest6838-21diskstats08:50:141
1641299171,12cyclictest20784-21users11:45:321
1641299168,5cyclictest4026-21sshd07:49:231
1641299168,5cyclictest0-21swapper/108:59:371
1641299164,9cyclictest29079-21ls08:35:021
1641299164,9cyclictest29079-21ls08:35:011
1641299163,10cyclictest18954-21cron08:15:011
1641299162,11cyclictest9183-21latency_hist08:00:011
1641299162,10cyclictest0-21swapper/108:35:121
1641299161,12cyclictest3597-21kthreadcore09:50:201
1641299161,12cyclictest18800-21tr07:10:181
1641299161,11cyclictest5147-21kthreadcore07:50:141
1641299161,10cyclictest23804-21sendmail-msp08:20:011
1640499165,8cyclictest16331-21latency_hist07:10:010
1642899151,12cyclictest0-21swapper/308:15:143
1641299157,6cyclictest24008-21sshd08:21:061
1641299152,4cyclictest21-21ksoftirqd/108:45:011
1641299151,11cyclictest31412-21cron07:40:011
1642899149,3cyclictest0-21swapper/308:00:223
1642899141,9cyclictest0-21swapper/308:45:023
1641299142,9cyclictest26401-21sh07:30:011
1642899138,3cyclictest0-21swapper/307:35:283
1642899132,8cyclictest0-21swapper/307:50:013
1642899131,9cyclictest0-21swapper/307:16:053
1642899131,9cyclictest0-21swapper/307:16:053
1642899131,8cyclictest0-21swapper/307:20:143
1641299130,10cyclictest16435-21latency_hist07:10:011
1642899121,8cyclictest0-21swapper/308:35:013
1642899121,8cyclictest0-21swapper/308:25:013
1642899121,8cyclictest0-21swapper/308:15:023
1642099122,8cyclictest16430-21latency_hist07:10:012
1642899111,7cyclictest0-21swapper/308:35:013
1642899111,7cyclictest0-21swapper/308:00:013
1642899111,7cyclictest0-21swapper/307:30:023
164289991,5cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional