You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-08 - 10:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sat Aug 08, 2026 00:43:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1107526220,6sleep211076-21/usr/sbin/munin19:05:132
2937521570,8sleep21286099cyclictest22:35:082
12858991534,123cyclictest10832-21grep00:10:190
12858991473,29cyclictest0-21swapper/023:20:240
12858991355,99cyclictest0-21swapper/022:05:300
128609913413,86cyclictest1703-21irqstats22:40:202
12858991313,46cyclictest25397-21tr21:50:240
12858991235,86cyclictest0-21swapper/023:05:180
128589912263,43cyclictest101ktimersoftd/022:40:210
12858991203,34cyclictest9337-21ssh22:10:220
128589911825,73cyclictest31767-21ssh00:35:230
12860991167,23cyclictest21016-21sh21:45:222
12860991152,9cyclictest22440-21ssh23:05:252
128599911513,20cyclictest26665-21ntp_states23:10:241
12859991133,10cyclictest19576-21ssh00:20:271
12858991123,10cyclictest10551-21memory23:30:220
12858991122,44cyclictest8654-21irqstats21:30:200
12858991115,77cyclictest0-21swapper/023:50:180
12860991102,9cyclictest7074-21unixbench_singl23:25:312
12860991063,13cyclictest23679-21ntp_states00:25:242
128589910616,69cyclictest9-21ksoftirqd/000:20:240
128589910613,75cyclictest0-21swapper/023:45:240
12860991053,11cyclictest25276-21memory21:50:222
12860991043,26cyclictest0-21swapper/222:15:222
12860991003,11cyclictest1182-21runrttasks21:20:302
128589910021,62cyclictest23732-21proc_pri00:25:260
12859999711,11cyclictest20967-21ssh21:45:211
1286099953,11cyclictest9343-21memory22:10:222
12859999513,10cyclictest11083-21date23:30:311
12860999414,21cyclictest19505-21ssh00:20:252
1285899943,17cyclictest14424-21/usr/sbin/munin22:55:260
1286199934,61cyclictest140-21systemd-journal23:30:293
1286099933,9cyclictest0-21swapper/200:10:182
1286099933,26cyclictest0-21swapper/223:10:242
1286099909,9cyclictest7750irq/56-dwc_otg23:30:242
12859999012,26cyclictest1824-21memory22:40:231
12858998933,7cyclictest0-21swapper/022:20:190
1286099873,13cyclictest7676-21ssh00:05:302
1285899874,11cyclictest19401-21unixbench_multi23:40:310
12861998621,23cyclictest0-21swapper/322:40:083
1285899858,11cyclictest20896-21fschecks_time21:45:200
1285899853,54cyclictest0-21swapper/023:00:250
1285899853,10cyclictest0-21swapper/022:15:240
12858998511,57cyclictest0-21swapper/019:45:300
1285899833,10cyclictest6596-21ntp_states23:25:240
1285899832,17cyclictest7850irq/56-dwc_otg_22:35:070
12861998213,48cyclictest0-21swapper/323:05:233
12859998220,17cyclictest5842-21grep22:05:301
1285899823,7cyclictest0-21swapper/021:20:290
1286199813,43cyclictest13398-21ls22:15:223
1286099807,10cyclictest14502-21ssh22:55:282
12861997918,26cyclictest26660-21ssh23:10:233
1285999797,52cyclictest0-21swapper/123:50:231
1285999793,46cyclictest0-21swapper/122:20:211
12860997819,27cyclictest17336-21cut22:20:212
1285999783,44cyclictest0-21swapper/100:10:251
12858997826,22cyclictest13-21rcuc/022:00:120
1285899774,50cyclictest0-21swapper/023:15:290
1286199763,21cyclictest5911-21ssh22:05:313
1285999769,8cyclictest0-21swapper/121:50:231
12861997522,26cyclictest7750irq/56-dwc_otg23:00:253
12861997518,27cyclictest1101-21sed21:20:303
12861997513,37cyclictest29000-21apt-get22:35:083
1285999753,10cyclictest0-21swapper/123:40:291
1285899753,14cyclictest0-21swapper/000:05:290
275182740,2sleep30-21swapper/300:30:213
1285999746,15cyclictest22823-21irqstats23:45:241
12859997411,11cyclictest8727-21ssh21:30:221
12859997410,11cyclictest31208-21unixbench-2d23:15:291
1285899743,10cyclictest26669-21ssh23:10:240
116132740,3sleep10-21swapper/122:52:021
15812730,3sleep229-21rcuc/223:20:022
1286099739,35cyclictest2423-21ssh23:20:242
1286099733,12cyclictest8654-21irqstats21:30:202
12860997313,28cyclictest0-21swapper/222:05:302
1285999733,41cyclictest0-21swapper/123:25:231
1286199717,32cyclictest23689-21ssh00:25:253
1286199715,41cyclictest7950irq/56-dwc_otg_23:25:293
12861997012,38cyclictest25277-21ssh21:50:223
1285999703,37cyclictest0-21swapper/122:10:231
12859997021,15cyclictest1157-21grep21:20:301
1286199693,22cyclictest0-21swapper/323:20:223
12858996938,3cyclictest0-21swapper/000:30:130
12861996821,24cyclictest17331-21fschecks_time22:20:213
12860996811,28cyclictest19293-21ssh23:40:292
1285999688,11cyclictest7671-21users00:05:291
12858996824,26cyclictest14386-21fschecks_time23:35:180
12858996824,26cyclictest14386-21fschecks_time23:35:180
1285899681,54cyclictest32074-21head20:00:270
1285899672,46cyclictest0-21swapper/019:30:180
12858996711,41cyclictest23327-21grep21:05:300
1286199666,45cyclictest0-21swapper/322:55:293
1286199664,39cyclictest0-21swapper/323:45:253
1285999669,11cyclictest0-21swapper/123:20:231
12859996615,12cyclictest14377-21ssh22:55:241
1286199652,38cyclictest27054-21ls23:50:233
12860996521,22cyclictest26994-21cut23:50:222
12860996511,28cyclictest7750irq/56-dwc_otg23:45:252
1285999653,36cyclictest23749-21ps00:25:261
12859996512,28cyclictest22438-21ssh23:05:241
1285899657,45cyclictest0-21swapper/021:10:250
1286099644,16cyclictest31739-21if_eth000:35:222
1286099643,41cyclictest0-21swapper/223:00:242
1285899642,6cyclictest2366-21aten_rbpower_po00:00:160
1286199635,30cyclictest7950irq/56-dwc_otg_00:35:203
1286199634,18cyclictest10837-21fschecks_time00:10:193
1285899631,42cyclictest0-21swapper/020:10:120
12861996219,36cyclictest0-21swapper/322:10:233
12861996210,31cyclictest7950irq/56-dwc_otg_00:20:223
1285899621,50cyclictest20235-21grep21:00:010
1286199619,31cyclictest0-21swapper/321:30:213
12861996126,18cyclictest31174-21ls23:15:283
1285899612,35cyclictest0-21swapper/000:15:120
1285899612,35cyclictest0-21swapper/000:15:120
12861996025,8cyclictest7950irq/56-dwc_otg_22:45:143
12858996029,14cyclictest7950irq/56-dwc_otg_20:40:290
1285899601,4cyclictest28982-21ssh22:35:010
1285899597,40cyclictest0-21swapper/019:20:180
1285899595,50cyclictest4060-21aten_rbpower_en21:25:150
1285899595,50cyclictest4060-21aten_rbpower_en21:25:150
1286199583,24cyclictest0-21swapper/323:40:313
1285899589,31cyclictest0-21swapper/019:50:180
1285899589,25cyclictest0-21swapper/020:45:190
1285899588,28cyclictest0-21swapper/023:55:170
12858995816,28cyclictest7518-21latency_hist20:25:020
57162560,3sleep20-21swapper/221:26:122
57162560,3sleep20-21swapper/221:26:122
198972560,4sleep01285899cyclictest19:30:010
1286199567,26cyclictest7850irq/56-dwc_otg_21:47:133
1285899561,5cyclictest17658-21ssh21:40:300
12858995615,31cyclictest22291-21grep22:25:280
12858995615,31cyclictest22291-21grep22:25:280
13302550,2sleep21332-21uname22:00:262
12861995526,22cyclictest7699-21ssh00:05:313
124402553,12sleep00-21swapper/019:08:030
12859995422,8cyclictest7950irq/56-dwc_otg_22:35:071
1285899541,34cyclictest582-21tune2fs20:05:190
1285999533,8cyclictest7950irq/56-dwc_otg_00:35:211
1285999533,10cyclictest0-21swapper/122:00:121
1285899524,21cyclictest0-21swapper/020:30:010
1285899521,25cyclictest29441-21cat21:55:220
12859995113,19cyclictest18466-21ssh23:00:241
1285899511,25cyclictest0-21swapper/020:30:120
12858995112,24cyclictest6142-21open_inodes22:45:250
1286099506,30cyclictest25256-21ssh21:10:282
12859995010,9cyclictest7950irq/56-dwc_otg_22:15:221
12858995027,9cyclictest7850irq/56-dwc_otg_21:15:240
12858995021,9cyclictest7950irq/56-dwc_otg_21:05:010
1285899501,16cyclictest23948-21grep19:40:150
1285899491,28cyclictest0-21swapper/022:50:220
1286199481,29cyclictest0-21swapper/319:55:143
12858994821,4cyclictest7750irq/56-dwc_otg21:35:190
1286199477,26cyclictest0-21swapper/320:05:153
12860994732,11cyclictest0-21swapper/200:15:132
12860994732,11cyclictest0-21swapper/200:15:132
1286199468,23cyclictest0-21swapper/319:30:143
1286199467,24cyclictest0-21swapper/320:00:133
12861994613,13cyclictest7950irq/56-dwc_otg_20:55:283
1286199461,23cyclictest22268-21ls19:35:193
12859994625,5cyclictest1180-21runrttasks20:20:291
1285899461,9cyclictest13484-21cat19:10:210
1286199459,20cyclictest6916-21/usr/sbin/munin20:20:323
1286199457,23cyclictest0-21swapper/320:35:153
1286099452,20cyclictest0-21swapper/219:50:122
1285899451,4cyclictest0-21swapper/020:50:260
12858994513,20cyclictest14683-21/usr/sbin/munin19:15:230
1286199447,24cyclictest0-21swapper/320:25:153
12861994423,8cyclictest7950irq/56-dwc_otg_21:25:353
12861994423,8cyclictest7950irq/56-dwc_otg_21:25:353
12861994419,7cyclictest7850irq/56-dwc_otg_22:50:443
12861994415,7cyclictest7850irq/56-dwc_otg_21:10:113
1286099447,25cyclictest0-21swapper/220:20:152
1285899441,8cyclictest30749-21latency_hist20:00:010
1285899441,4cyclictest4982-21grep20:15:300
12861994323,12cyclictest7750irq/56-dwc_otg22:34:113
1286199431,26cyclictest0-21swapper/319:50:193
12860994226,8cyclictest7950irq/56-dwc_otg_22:29:302
12860994226,8cyclictest7950irq/56-dwc_otg_22:29:302
1285999426,5cyclictest23737-21diskmemload21:35:301
1285999423,6cyclictest14617-21/usr/sbin/munin00:15:191
1285999423,6cyclictest14617-21/usr/sbin/munin00:15:191
1285899421,21cyclictest12163-21sensors20:35:270
241272410,1sleep30-21swapper/322:28:453
241272410,1sleep30-21swapper/322:28:453
12860994118,4cyclictest14521-21if_wlan023:35:202
12860994118,4cyclictest14521-21if_wlan023:35:202
12860994110,15cyclictest29230-21apt-config22:35:022
12859994111,6cyclictest0-21swapper/119:35:181
1285899411,20cyclictest22581-21ntpq19:35:230
49312400,4sleep14934-21sh21:25:261
49312400,4sleep14934-21sh21:25:261
1286199407,20cyclictest0-21swapper/319:10:143
12861994021,7cyclictest23288-21grep21:05:303
12861994013,19cyclictest381ktimersoftd/321:58:243
1286199393,7cyclictest7950irq/56-dwc_otg_00:00:143
12861993918,4cyclictest25814-21fschecks_time19:45:193
1286099393,21cyclictest544-21fschecks_count20:05:192
12860993910,4cyclictest0-21swapper/221:20:012
1285999397,11cyclictest23-21ksoftirqd/122:45:181
1285999391,8cyclictest22292-21wget21:05:161
116042393,27sleep10-21swapper/119:05:211
1286199386,27cyclictest7950irq/56-dwc_otg_23:55:143
12861993817,10cyclictest7950irq/56-dwc_otg_00:19:313
12861993817,10cyclictest7950irq/56-dwc_otg_00:19:313
1286099385,21cyclictest14281-21unixbench_multi20:40:292
1286099381,22cyclictest0-21swapper/220:25:192
12859993825,7cyclictest221ktimersoftd/121:15:241
12859993820,5cyclictest24874-21latency_hist22:30:021
12859993820,5cyclictest24874-21latency_hist22:30:021
1286199379,7cyclictest0-21swapper/320:40:293
1286199378,13cyclictest7950irq/56-dwc_otg_23:35:283
1286199378,13cyclictest7950irq/56-dwc_otg_23:35:283
1286199377,23cyclictest0-21swapper/319:25:143
12861993715,4cyclictest7750irq/56-dwc_otg20:50:183
1286099378,15cyclictest30177-21ls19:55:282
1286099375,15cyclictest7850irq/56-dwc_otg_21:05:302
1285999372,10cyclictest24717-21ssh21:10:221
12859993719,11cyclictest7626-21/usr/sbin/munin20:25:191
1285999371,6cyclictest4126-21diskstats20:15:191
12859993715,15cyclictest17034-21ssh21:40:221
1285999371,20cyclictest2277-21ssh00:00:151
120322375,21sleep312125-21/usr/sbin/munin19:05:273
12861993615,4cyclictest12793-21sh21:35:223
12861993614,4cyclictest7850irq/56-dwc_otg_21:17:413
12860993618,9cyclictest24229-21latency19:40:182
1286099361,5cyclictest1191-21snmpd23:56:112
1286099361,23cyclictest0-21swapper/219:30:252
1285999364,7cyclictest7950irq/56-dwc_otg_20:30:151
1285999362,23cyclictest0-21swapper/120:44:511
1286199352,19cyclictest20635-21grep21:00:193
12861993516,8cyclictest7950irq/56-dwc_otg_22:01:363
1286199351,21cyclictest16580-21ssh21:40:163
1286099358,14cyclictest0-21swapper/220:15:202
12860993520,9cyclictest6902-21ssh22:45:582
1285999359,14cyclictest27266-21aten_rbpower_vo00:30:171
1285999357,10cyclictest0-21swapper/123:38:481
1285999357,10cyclictest0-21swapper/123:38:481
1285999354,4cyclictest7950irq/56-dwc_otg_20:55:281
1285999351,5cyclictest1910-21munin-run00:00:011
1285999351,13cyclictest28810-21ssh22:34:411
1286199343,14cyclictest1468350/usr/sbin/munin19:15:233
12861993411,13cyclictest0-21swapper/320:30:073
1286099347,15cyclictest19361-21processes20:55:272
1286099344,21cyclictest7750irq/56-dwc_otg19:20:202
1286099342,15cyclictest3714-21latency_hist20:15:012
1286099342,15cyclictest0-21swapper/221:36:222
1285999344,4cyclictest0-21swapper/121:55:191
1285999341,5cyclictest20577-21df21:00:171
1286199336,4cyclictest7850irq/56-dwc_otg_20:15:153
1286199336,3cyclictest0-21swapper/319:24:183
1286099339,9cyclictest0-21swapper/200:30:142
1286099339,12cyclictest29303-21kernelversion21:55:212
1286099337,16cyclictest301ktimersoftd/222:50:252
1286099335,7cyclictest3526-21head00:00:312
1286099334,4cyclictest12486-21unixbench_singl20:35:302
1286099332,18cyclictest0-21swapper/221:05:022
1286099331,25cyclictest0-21swapper/220:02:372
1286099331,20cyclictest0-21swapper/221:41:192
1286099331,13cyclictest0-21swapper/219:16:242
1285999333,15cyclictest7850irq/56-dwc_otg_19:10:221
1285999331,28cyclictest17074-21missed_timers19:20:211
1286199323,14cyclictest16298-21runrttasks20:47:183
12861993211,7cyclictest7750irq/56-dwc_otg19:43:043
1286099322,15cyclictest0-21swapper/220:47:452
1286099321,7cyclictest11109-21apt20:35:012
1285999328,13cyclictest12444-21grep20:35:291
1285999327,6cyclictest7850irq/56-dwc_otg_19:40:201
1285999322,10cyclictest0-21swapper/119:50:121
1285999321,6cyclictest18905-21grep19:25:221
12859993213,13cyclictest29943-21/usr/sbin/munin19:55:261
1286099312,21cyclictest0-21swapper/219:47:022
1286099312,14cyclictest0-21swapper/219:25:192
1286099311,6cyclictest13378-21latency19:10:202
1285999319,8cyclictest21839-21date19:35:021
1285999318,10cyclictest0-21swapper/120:10:151
1285999312,10cyclictest0-21swapper/119:16:561
1285999311,7cyclictest32082-21head20:00:271
1286199306,4cyclictest2763-21ntpq20:10:233
1286099305,15cyclictest7750irq/56-dwc_otg20:50:292
1286099301,23cyclictest0-21swapper/219:36:372
1285999301,5cyclictest520-21rs:main1
1285999301,25cyclictest17047-21cut20:50:181
1285999291,5cyclictest487-21diskstats20:05:181
1285999287,7cyclictest26221-21ntp_states19:45:261
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional