You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-31 - 02:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sun May 31, 2026 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
59921870,6sleep12370399cyclictest00:21:571
59921870,6sleep12370399cyclictest00:21:571
241121100,5sleep02408-21munin-run00:25:060
241121100,5sleep02408-21munin-run00:25:060
237029910510,57cyclictest0-21swapper/023:50:190
209592980,5sleep22370499cyclictest22:50:072
23702999528,55cyclictest29566-21ssh22:20:320
2370299921,79cyclictest1671-21awk23:45:060
2370299871,74cyclictest23186-21grep23:30:340
2370299851,70cyclictest0-21swapper/022:00:070
2370299849,71cyclictest9609-21grep20:00:220
23702998431,38cyclictest31946-21ssh21:45:210
226442810,1sleep00-21swapper/019:05:310
283212800,2sleep10-21swapper/123:37:081
23702998036,27cyclictest27339-21ssh21:40:060
2370299801,62cyclictest26016-21grep22:55:280
23702997830,35cyclictest0-21swapper/019:25:220
2370299781,6cyclictest2789-21ssh00:25:210
23702997810,49cyclictest0-21swapper/023:35:100
2370299779,5cyclictest32097-21unixbench_singl19:30:350
2370299777,58cyclictest0-21swapper/021:27:160
23702997711,51cyclictest17798-21grep20:20:340
2370299767,19cyclictest268572sleep000:15:200
2370299761,60cyclictest18610-21aten_rbpower_en00:05:210
2370299753,5cyclictest9450irq/48-DMA10:20:150
23702997432,31cyclictest0-21swapper/023:25:330
2370299742,6cyclictest0-21swapper/022:35:190
2370299741,62cyclictest6953-21ssh21:54:160
2370299741,61cyclictest14602-21/usr/sbin/munin23:20:330
2370299741,55cyclictest20760-21grep20:30:250
89222730,2sleep30-21swapper/323:14:523
23702997321,38cyclictest27915-21wget20:50:200
2370299731,58cyclictest13649-21ntp_states22:40:300
2370299724,60cyclictest26360-21ssh22:16:340
2370299724,59cyclictest25615-21chrt20:44:280
2370299721,5cyclictest375550ssh22:29:120
23702997214,46cyclictest25034-21latency_hist22:55:070
23702997210,48cyclictest0-21swapper/020:55:340
2370599714,5cyclictest7850irq/56-dwc_otg22:00:133
2370499711,57cyclictest24361-21grep22:15:072
2370299718,54cyclictest0-21swapper/023:05:430
2370299711,59cyclictest12848-21taskset00:36:210
23702997019,35cyclictest0-21swapper/022:32:080
2370299701,8cyclictest0-21swapper/023:01:180
2370299701,8cyclictest0-21swapper/023:01:180
2370299701,6cyclictest2323-21grep23:45:240
23702997012,50cyclictest11043-21ssh23:16:010
23702997012,35cyclictest7950irq/56-dwc_otg_21:40:280
23702997012,35cyclictest7950irq/56-dwc_otg_21:40:280
23702997010,49cyclictest3432-21taskset19:41:500
2370499695,40cyclictest0-21swapper/223:40:572
2370299699,51cyclictest21471-21ssh21:31:570
2370299691,8cyclictest0-21swapper/023:56:380
2370299691,8cyclictest0-21swapper/023:56:380
2370299691,55cyclictest3886-21ssh21:10:350
2370299691,42cyclictest0-21swapper/022:05:270
23702996910,50cyclictest0-21swapper/020:25:180
23704996827,38cyclictest21842-21cron20:35:062
2370299682,7cyclictest0-21swapper/021:00:410
2370299682,7cyclictest0-21swapper/000:13:570
2370299681,6cyclictest0-21swapper/019:20:190
2370299681,49cyclictest11071-21ssh21:20:120
2370299681,49cyclictest11071-21ssh21:20:120
2370499679,36cyclictest0-21swapper/223:25:062
2370299671,8cyclictest0-21swapper/019:39:040
2370299671,52cyclictest14762-21grep20:15:070
2370299671,52cyclictest0-21swapper/022:00:080
23702996711,46cyclictest7717-21chrt19:55:210
2370499664,47cyclictest7850irq/56-dwc_otg22:55:312
2370499662,53cyclictest16983-21taskset21:26:182
2370299662,8cyclictest0-21swapper/000:33:580
2370299661,54cyclictest16748-21ssh00:02:280
23702996610,40cyclictest6226-21switchtime23:10:320
302772650,2sleep30-21swapper/323:40:163
2370499658,46cyclictest12043-21apt-get22:00:132
2370299651,6cyclictest20674-21ssh22:49:550
2370299651,5cyclictest21637-21grep22:10:350
2370299651,50cyclictest5535-21latency_hist19:50:070
2370299651,50cyclictest5535-21latency_hist19:50:070
2370299651,50cyclictest0-21swapper/019:54:500
23702996510,43cyclictest16199-21date20:17:060
2370499643,5cyclictest29556-21rt-features22:20:322
2370499641,51cyclictest29066-21taskset00:17:142
2370299641,45cyclictest2410-21sed21:10:060
2370599631,7cyclictest3487-21ntpq00:25:303
2370499635,46cyclictest8050irq/56-dwc_otg_00:34:252
2370499631,52cyclictest16095-21rm22:44:072
2370599622,5cyclictest8406-21apt22:35:063
2370499622,48cyclictest0-21swapper/223:03:412
2370499622,48cyclictest0-21swapper/223:03:412
2370499621,4cyclictest7558-21basename19:55:192
2370499612,4cyclictest11135-21apt-get21:20:182
2370499612,4cyclictest11135-21apt-get21:20:182
2370499611,5cyclictest18794-21sh00:05:232
2370499611,4cyclictest0-21swapper/219:20:192
23704996112,34cyclictest2395-21ssh21:48:292
2370299611,50cyclictest27211-21runrttasks20:46:520
2370499601,43cyclictest7593-21sh23:50:522
2370499601,40cyclictest8050irq/56-dwc_otg_20:50:062
23703996036,20cyclictest9437-21ssh23:15:131
2370299601,34cyclictest0-21swapper/020:05:170
23702996013,35cyclictest0-21swapper/020:36:140
2370599595,5cyclictest8050irq/56-dwc_otg_19:25:273
2370599592,6cyclictest7850irq/56-dwc_otg23:50:373
23705995922,34cyclictest11373-21ssh22:38:013
2370499591,4cyclictest8050irq/56-dwc_otg_19:15:072
2370499591,47cyclictest24816-21ssh23:33:302
2370399592,14cyclictest0-21swapper/119:55:171
2370499585,4cyclictest7950irq/56-dwc_otg_20:35:132
2370499582,52cyclictest429-21ssh22:25:192
2370499582,45cyclictest8387-21ssh22:35:062
2370499581,4cyclictest8050irq/56-dwc_otg_21:05:212
2370499581,4cyclictest4625-21rm23:47:592
2370499581,4cyclictest0-21swapper/223:35:362
2370499581,46cyclictest27094-21seq19:20:022
23703995816,9cyclictest8050irq/56-dwc_otg_00:30:181
2370599571,5cyclictest7850irq/56-dwc_otg22:55:343
2370599571,5cyclictest20374-21ssh22:10:183
2370499572,9cyclictest0-21swapper/220:00:242
2370499572,4cyclictest0-21swapper/200:02:482
2370499571,5cyclictest8050irq/56-dwc_otg_21:40:312
2370499571,5cyclictest8050irq/56-dwc_otg_21:40:312
2370499571,5cyclictest0-21swapper/220:55:322
2370499571,5cyclictest0-21swapper/219:25:202
2370499571,4cyclictest8050irq/56-dwc_otg_21:17:482
2370499571,4cyclictest0-21swapper/221:50:342
2370499571,46cyclictest18571-21ssh22:07:282
2370499571,45cyclictest17761-21/usr/sbin/munin23:25:342
2370299570,6cyclictest8050irq/56-dwc_otg_19:20:070
2370499567,36cyclictest8050irq/56-dwc_otg_21:55:132
2370499565,40cyclictest8050irq/56-dwc_otg_00:25:492
2370499564,42cyclictest8050irq/56-dwc_otg_00:22:592
2370499564,42cyclictest8050irq/56-dwc_otg_00:22:592
2370499562,43cyclictest7950irq/56-dwc_otg_23:05:532
2370499561,6cyclictest27339-21ssh21:40:062
2370499561,5cyclictest8050irq/56-dwc_otg_23:56:472
2370499561,5cyclictest8050irq/56-dwc_otg_23:56:472
2370499561,5cyclictest14995-21latency_hist00:40:072
2370499561,4cyclictest0-21swapper/222:38:162
2370599556,3cyclictest0-21swapper/323:32:383
2370599556,3cyclictest0-21swapper/323:00:273
2370599556,3cyclictest0-21swapper/323:00:273
2370499552,3cyclictest0-21swapper/223:13:252
2370499551,5cyclictest23428-21ntp_kernel_pll_00:10:292
2370599544,4cyclictest0-21swapper/321:32:373
2370599541,14cyclictest8050irq/56-dwc_otg_20:05:263
2370599541,13cyclictest7950irq/56-dwc_otg_00:30:203
2370499542,37cyclictest0-21swapper/222:20:062
2370499541,5cyclictest9421-21ssh23:15:112
2370499541,4cyclictest8050irq/56-dwc_otg_21:30:072
2370499541,42cyclictest15872sleep219:36:272
23704995412,39cyclictest24425-21sh22:54:132
2370399545,9cyclictest7850irq/56-dwc_otg19:20:171
2370399541,4cyclictest7850irq/56-dwc_otg23:00:191
2370399541,4cyclictest7850irq/56-dwc_otg23:00:191
2370599537,3cyclictest0-21swapper/323:47:053
2370599532,9cyclictest7850irq/56-dwc_otg22:05:153
23705995318,4cyclictest30433-21sh00:19:443
2370499531,5cyclictest0-21swapper/221:11:492
2370499531,42cyclictest326632sleep221:01:082
2370499531,36cyclictest0-21swapper/220:40:192
2370399538,27cyclictest60-21kswapd022:50:161
2370299531,39cyclictest23945-21runrttasks19:10:210
2370599528,4cyclictest0-21swapper/300:11:273
2370599523,5cyclictest32210-21cut21:45:243
2370499523,37cyclictest0-21swapper/220:08:582
23703995211,19cyclictest7950irq/56-dwc_otg_22:00:131
2370599516,4cyclictest7850irq/56-dwc_otg20:25:253
2370599511,6cyclictest31753-21latency_hist21:45:083
2370499511,5cyclictest0-21swapper/220:11:182
2370599502,9cyclictest8050irq/56-dwc_otg_23:55:303
2370599502,9cyclictest8050irq/56-dwc_otg_23:55:303
2370599501,5cyclictest10048-21ntp_states20:00:293
2370499509,37cyclictest32105-21unixbench_singl19:30:352
2370499503,4cyclictest7950irq/56-dwc_otg_19:55:062
2370499501,3cyclictest0-21swapper/220:50:192
2370399502,21cyclictest7950irq/56-dwc_otg_20:35:191
23703995015,9cyclictest0-21swapper/123:40:191
2370599491,11cyclictest8050irq/56-dwc_otg_19:20:073
2370499491,5cyclictest16193-21cron20:17:062
2370499491,3cyclictest8050irq/56-dwc_otg_20:25:222
2370399492,9cyclictest7850irq/56-dwc_otg20:10:171
2370599482,8cyclictest7950irq/56-dwc_otg_22:52:593
2370599482,5cyclictest0-21swapper/322:20:353
2370599482,12cyclictest0-21swapper/320:55:393
2370399487,27cyclictest0-21swapper/120:05:201
2370399487,26cyclictest0-21swapper/120:50:141
2370399477,25cyclictest0-21swapper/122:55:191
2370399477,25cyclictest0-21swapper/121:00:201
2370499462,5cyclictest0-21swapper/219:45:202
2370499462,5cyclictest0-21swapper/219:45:202
2370499462,4cyclictest0-21swapper/220:20:192
2370499461,5cyclictest2351-21/usr/sbin/munin19:40:222
2370399466,28cyclictest7794-21ssh21:55:121
2370399457,25cyclictest0-21swapper/122:40:191
23703994418,15cyclictest0-21swapper/122:45:201
23703994326,6cyclictest23-21ksoftirqd/119:25:191
23703994322,6cyclictest23811-21aten_rbpower_po19:10:191
233852430,6sleep12370399cyclictest22:13:421
111402430,6sleep311135-21ssh00:35:203
2370399423,4cyclictest0-21swapper/119:30:171
2370399415,4cyclictest8050irq/56-dwc_otg_00:25:191
23703994111,4cyclictest0-21swapper/123:25:211
142232410,5sleep121-21rcuc/100:00:071
142232410,5sleep121-21rcuc/100:00:071
2370599407,19cyclictest0-21swapper/320:15:123
2370399408,4cyclictest7850irq/56-dwc_otg00:10:171
23703994012,5cyclictest2726-21ssh21:10:191
2370399393,10cyclictest0-21swapper/123:30:191
2370399392,6cyclictest0-21swapper/122:15:191
2370399391,25cyclictest11829-21rm22:38:501
136412390,2sleep10-21swapper/123:20:171
23703993832,4cyclictest17187-21ls20:20:261
2370599376,5cyclictest7950irq/56-dwc_otg_21:20:193
2370599376,5cyclictest7950irq/56-dwc_otg_21:20:193
2370599371,23cyclictest28310-21latency_hist22:20:073
2370399379,8cyclictest32126-21ssh22:24:321
2370399376,19cyclictest0-21swapper/122:25:131
23703993730,5cyclictest1133-21runrttasks21:20:061
23703993717,4cyclictest11169-21basename21:20:211
23703993717,4cyclictest11169-21basename21:20:211
2370399366,17cyclictest0-21swapper/120:30:161
2370599358,14cyclictest0-21swapper/319:50:203
23705993525,6cyclictest381ktimersoftd/320:35:203
2370399358,15cyclictest0-21swapper/121:45:121
2370399357,23cyclictest0-21swapper/120:40:121
2370399354,25cyclictest8050irq/56-dwc_otg_00:15:181
2370399351,5cyclictest0-21swapper/123:50:151
2370599348,12cyclictest7370-21fschecks_time21:15:273
23705993421,3cyclictest8050irq/56-dwc_otg_21:39:483
23705993410,5cyclictest0-21swapper/323:15:363
2370399346,20cyclictest0-21swapper/121:54:191
2370399343,7cyclictest15595-21fschecks_time21:25:251
23703993419,10cyclictest21-21rcuc/122:30:201
2370399341,6cyclictest8050irq/56-dwc_otg_20:55:191
23703993413,4cyclictest14418-21aten_rbpower_cu00:00:191
2370399341,20cyclictest3051-21timerandwakeup19:40:321
2370599335,3cyclictest0-21swapper/300:07:123
2370599335,15cyclictest7950irq/56-dwc_otg_00:20:233
2370599335,15cyclictest7950irq/56-dwc_otg_00:20:233
2370399333,17cyclictest1952-21latency_hist19:40:071
23703993314,5cyclictest1744-21missed_timers23:05:281
23703993310,5cyclictest0-21swapper/121:43:171
23703993310,5cyclictest0-21swapper/121:43:171
229312334,16sleep30-21swapper/319:05:353
2370599328,4cyclictest0-21swapper/321:28:553
2370599327,16cyclictest0-21swapper/321:50:193
2370599327,14cyclictest0-21swapper/323:35:113
2370599325,7cyclictest7850irq/56-dwc_otg23:30:063
2370599321,4cyclictest0-21swapper/322:43:483
2370599321,20cyclictest0-21swapper/322:45:113
2370399329,11cyclictest7850irq/56-dwc_otg19:15:281
2370399328,5cyclictest8050irq/56-dwc_otg_20:45:291
2370399327,4cyclictest0-21swapper/123:12:341
2370399327,15cyclictest0-21swapper/100:35:201
2370399325,4cyclictest0-21swapper/100:05:181
220492325,16sleep20-21swapper/219:05:222
175412320,4sleep32370599cyclictest23:25:063
2370599316,14cyclictest0-21swapper/319:40:193
2370599314,13cyclictest7850irq/56-dwc_otg20:55:063
2370599313,14cyclictest8050irq/56-dwc_otg_21:10:243
2370399316,16cyclictest0-21swapper/119:53:591
2370399312,5cyclictest2354-21modprobe21:10:061
2370399311,16cyclictest224050ssh23:45:231
2370599308,5cyclictest27066-21grep20:45:363
2370599306,4cyclictest0-21swapper/321:05:183
2370599305,5cyclictest7850irq/56-dwc_otg21:55:253
2370599302,6cyclictest8050irq/56-dwc_otg_22:25:233
2370599302,6cyclictest24245-21/usr/sbin/munin19:10:253
2370599302,5cyclictest7850irq/56-dwc_otg23:05:353
2370599302,15cyclictest8050irq/56-dwc_otg_00:00:163
23705993021,5cyclictest4054-21aten_rbpower_vo19:45:203
23705993021,5cyclictest4054-21aten_rbpower_vo19:45:203
23705993011,5cyclictest32541-21tail21:00:353
2370399309,12cyclictest8050irq/56-dwc_otg_22:05:391
2370399306,15cyclictest0-21swapper/121:35:191
2370399302,7cyclictest4698-21/usr/sbin/munin19:45:301
2370399302,7cyclictest4698-21/usr/sbin/munin19:45:301
2370399302,6cyclictest7950irq/56-dwc_otg_20:15:221
23703993024,4cyclictest1133-21runrttasks21:30:161
2370399301,15cyclictest8050irq/56-dwc_otg_20:00:311
231222307,14sleep10-21swapper/119:06:131
2370599296,6cyclictest8050irq/56-dwc_otg_19:35:263
2370599295,6cyclictest12960-21awk20:10:163
23703992910,3cyclictest7950irq/56-dwc_otg_20:25:331
2370599285,4cyclictest0-21swapper/321:41:113
2370599281,3cyclictest7850irq/56-dwc_otg20:30:173
2370599274,9cyclictest7850irq/56-dwc_otg19:55:283
2370599273,4cyclictest7950irq/56-dwc_otg_19:35:073
2370599272,7cyclictest28928-21munin-run19:25:063
2370599262,7cyclictest7850irq/56-dwc_otg20:40:203
23705992510,4cyclictest182202chrt20:24:503
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional