You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-08 - 18:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Thu Jan 08, 2026 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7604991001,80cyclictest21286-21ntpq10:50:360
760499971,6cyclictest11519-21wc10:25:160
760499971,46cyclictest0-21swapper/010:00:150
760499951,5cyclictest24251-21proc_pri12:30:380
760499921,6cyclictest25200-21apt-get09:30:260
7604999015,41cyclictest0-21swapper/011:55:270
760499901,4cyclictest1133-21latency_hist08:25:150
7605998614,47cyclictest31742-21idleruntime-cro08:20:151
760499866,65cyclictest0-21swapper/012:16:520
760499852,3cyclictest0-21swapper/011:35:150
760499851,69cyclictest8024-21sed08:40:440
760499842,75cyclictest0-21swapper/007:10:170
760499831,6cyclictest29747-21unixbench_singl11:15:410
760499831,5cyclictest5559-21grep08:35:340
266642830,6sleep1760599cyclictest11:10:161
760499822,62cyclictest23643-21fschecks_count09:25:320
760499821,3cyclictest0-21swapper/008:30:270
760499811,5cyclictest28568-21apt09:40:150
760499801,4cyclictest31756-21basename08:20:150
760499783,4cyclictest4103-21open_inodes11:35:390
760499782,56cyclictest0-21swapper/007:50:150
760499781,57cyclictest12355-21taskset07:20:430
760499781,57cyclictest12355-21taskset07:20:430
760499762,58cyclictest0-21swapper/008:25:170
760499762,58cyclictest0-21swapper/008:25:170
760499752,4cyclictest0-21swapper/011:10:290
760499751,5cyclictest4580-21fschecks_count10:05:320
7604997514,43cyclictest7850irq/56-dwc_otg_10:00:280
760499741,5cyclictest23843-21apt-config07:55:150
760499741,4cyclictest0-21swapper/012:25:280
760499741,43cyclictest18686-21seq10:45:150
205012740,2sleep10-21swapper/110:50:151
7604997328,22cyclictest7850irq/56-dwc_otg_12:00:250
760499732,64cyclictest25692-21munin-run08:00:150
760499732,64cyclictest25692-21munin-run08:00:150
760499732,51cyclictest0-21swapper/010:48:070
760499731,5cyclictest27586-21date08:05:150
760499731,57cyclictest25605-21grep12:35:330
7604997248,5cyclictest7750irq/56-dwc_otg11:10:150
218552720,2sleep121858-21fschecks_time09:20:321
760499716,46cyclictest23156-21awk09:25:150
760499712,57cyclictest0-21swapper/011:01:470
760499711,5cyclictest19897-21grep09:15:310
760499711,4cyclictest11811-21fschecks_count10:25:320
760599696,46cyclictest7850irq/56-dwc_otg_10:50:421
760599692,61cyclictest7608-21taskset07:10:181
760499692,51cyclictest29911-21grep09:40:430
760499691,56cyclictest0-21swapper/011:46:260
760499691,4cyclictest0-21swapper/007:34:270
760499691,45cyclictest0-21swapper/009:51:570
7604996911,48cyclictest28427-21aten_rbpower_po08:10:250
760599671,34cyclictest7978-21sed08:40:431
760499678,46cyclictest21559-21sed12:25:160
760499672,7cyclictest0-21swapper/011:21:370
760499672,7cyclictest0-21swapper/011:21:370
760499671,4cyclictest0-21swapper/007:19:070
760599661,36cyclictest30081-21runrttasks09:41:351
760499661,4cyclictest0-21swapper/011:27:130
197242660,2sleep30-21swapper/310:45:393
760499659,48cyclictest22724-21runrttasks10:59:270
760499651,5cyclictest1125-21runrttasks10:35:300
760499651,4cyclictest117182sleep008:51:040
760499651,4cyclictest0-21swapper/011:41:120
760499644,4cyclictest7950irq/56-dwc_otg_11:55:150
760499641,5cyclictest0-21swapper/009:11:560
760499641,5cyclictest0-21swapper/009:11:560
760499641,5cyclictest0-21swapper/009:04:000
760499641,4cyclictest0-21swapper/010:33:410
760499641,4cyclictest0-21swapper/010:33:410
760599631,43cyclictest5696-21ls08:35:351
760599631,41cyclictest23467-21wget09:25:291
760499632,3cyclictest0-21swapper/008:56:160
760499631,5cyclictest0-21swapper/009:46:430
760499631,51cyclictest17495-21runrttasks12:11:270
760499631,4cyclictest19415-21sensors07:40:400
760499631,4cyclictest0-21swapper/009:06:370
760499621,6cyclictest1125-21runrttasks07:37:020
760499621,3cyclictest146442chrt07:29:210
760499621,3cyclictest0-21swapper/010:16:430
760599611,41cyclictest2401-21timerandwakeup08:25:421
760599611,41cyclictest2401-21timerandwakeup08:25:421
760599611,40cyclictest24253-21processes12:30:381
760599611,39cyclictest2627-21/usr/sbin/munin10:00:251
760599611,39cyclictest13773-21taskset08:59:091
760599611,38cyclictest18070-21taskset07:37:001
760599611,38cyclictest12976-21taskset10:28:221
760499611,6cyclictest0-21swapper/010:11:280
7604996110,44cyclictest0-21swapper/008:47:360
760599601,43cyclictest0-21swapper/112:00:251
760599601,38cyclictest3307-21basename08:30:301
760599601,34cyclictest27333-21/usr/sbin/munin11:10:341
760499608,39cyclictest0-21swapper/012:08:420
760599591,40cyclictest23918-21sed07:55:161
760599591,37cyclictest5508-21users10:05:441
760599581,39cyclictest11753-21seq08:51:241
760599581,36cyclictest3370-21/usr/sbin/munin11:35:361
760599581,35cyclictest17711-21sshd12:14:071
760499581,43cyclictest27719-21taskset08:05:220
760599571,42cyclictest0-21swapper/108:10:241
760599561,41cyclictest16464-21runrttasks07:34:311
7605995610,29cyclictest25482-21fschecks_time09:30:321
760599551,39cyclictest0-21swapper/109:55:001
760599551,33cyclictest21469-21modprobe12:25:151
760599551,31cyclictest1125-21runrttasks09:12:131
760599551,31cyclictest1125-21runrttasks09:12:131
760599551,30cyclictest7950irq/56-dwc_otg_09:10:161
760599541,36cyclictest9892-21seq08:45:531
760599541,31cyclictest0-21swapper/108:02:461
760599532,38cyclictest0-21swapper/108:20:351
760599532,36cyclictest0-21swapper/111:57:591
760599531,41cyclictest0-21swapper/111:44:071
760599531,40cyclictest0-21swapper/110:13:581
760599531,39cyclictest7950irq/56-dwc_otg_07:55:211
760599531,39cyclictest7950irq/56-dwc_otg_07:55:211
760599531,39cyclictest22479-21taskset10:56:161
760599531,39cyclictest0-21swapper/112:38:541
760599523,45cyclictest23131-21apt-get11:00:221
760599521,37cyclictest7950irq/56-dwc_otg_09:49:341
760599521,36cyclictest0-21swapper/110:30:541
760599521,36cyclictest0-21swapper/110:30:541
760599521,33cyclictest7950irq/56-dwc_otg_09:20:161
760599521,32cyclictest7950irq/56-dwc_otg_12:28:031
760599521,31cyclictest16478-21runrttasks10:36:511
760599511,31cyclictest153672chrt09:01:281
760599502,35cyclictest7950irq/56-dwc_otg_08:10:151
760599501,43cyclictest18306-21sshd10:42:071
760599501,30cyclictest12827-21runrttasks07:24:141
760599501,30cyclictest12827-21runrttasks07:24:141
760599492,35cyclictest20103-21runrttasks07:44:481
760599491,36cyclictest0-21swapper/111:52:051
760599491,34cyclictest0-21swapper/107:27:111
760599491,33cyclictest0-21swapper/110:15:281
760599491,27cyclictest0-21swapper/111:20:401
760599491,27cyclictest0-21swapper/111:20:401
760799481,25cyclictest10642-21grep07:15:443
760599482,43cyclictest0-21swapper/111:25:261
760599482,32cyclictest29779-21chrt11:15:471
760599481,40cyclictest11016-21taskset07:19:131
7606994718,6cyclictest4976-21latency_hist11:40:152
760599472,42cyclictest9770-21apt-get10:20:251
760599471,43cyclictest27039-21cut09:35:291
760599471,42cyclictest0-21swapper/111:30:241
760599471,35cyclictest0-21swapper/111:49:181
760799461,5cyclictest7950irq/56-dwc_otg_09:40:273
760599461,42cyclictest18140-21grep12:15:281
760599461,42cyclictest15006-21memory12:05:341
760599461,34cyclictest0-21swapper/107:49:581
760599451,41cyclictest2379-21cat10:00:151
7606994223,4cyclictest1125-21runrttasks10:15:372
760699421,5cyclictest7750irq/56-dwc_otg10:00:232
760699421,4cyclictest16993-21/usr/sbin/munin10:40:422
760699411,5cyclictest19326-21ls10:45:322
701524126,7sleep00-21swapper/007:06:140
760799405,6cyclictest26334-21head09:30:433
7607994030,8cyclictest381ktimersoftd/310:00:233
7607994018,4cyclictest14334-21cat12:05:163
7606994016,5cyclictest21048-21if_wlan010:50:332
760799394,8cyclictest0-21swapper/308:30:283
7606993925,9cyclictest7601-21cyclictest07:10:182
760699391,4cyclictest7950irq/56-dwc_otg_10:35:402
760699391,3cyclictest0-21swapper/208:40:272
233742390,4sleep3760799cyclictest12:30:163
760699382,3cyclictest0-21swapper/208:55:272
760699381,5cyclictest0-21swapper/210:05:272
760699381,19cyclictest7950irq/56-dwc_otg_07:55:402
760699381,19cyclictest7950irq/56-dwc_otg_07:55:402
7607993715,4cyclictest20152-21idleruntime-cro07:45:153
7607993712,6cyclictest7850irq/56-dwc_otg_09:50:153
760799371,21cyclictest10282-21latency10:20:353
7606993716,5cyclictest28150-21latency_hist08:10:152
7606993715,5cyclictest435-21cron12:35:152
111972371,2sleep20-21swapper/208:50:392
760799367,20cyclictest0-21swapper/308:35:273
760799361,28cyclictest1289-21cp08:25:163
760799361,22cyclictest22146-21ls07:50:163
7606993627,7cyclictest14726-21sh07:30:152
7606993617,5cyclictest25960-21wget08:00:312
760699361,5cyclictest0-21swapper/210:25:282
760699361,24cyclictest11257-21ls07:20:162
760799358,6cyclictest7850irq/56-dwc_otg_07:10:433
7607993516,5cyclictest25777-21sensors11:05:413
7607993511,6cyclictest28514-21seq09:39:473
760699355,4cyclictest7750irq/56-dwc_otg11:00:222
71772356,23sleep10-21swapper/107:08:211
7607993416,11cyclictest23583-21seq07:53:083
7607993415,4cyclictest7160-21cut11:45:293
760799341,28cyclictest12451-21switchtime10:25:413
760799341,23cyclictest15102-21ls10:35:163
760799341,23cyclictest15102-21ls10:35:163
7607993411,4cyclictest14965-21cat12:05:343
760699348,6cyclictest7850irq/56-dwc_otg_07:45:292
760699347,7cyclictest7950irq/56-dwc_otg_08:25:322
760699347,7cyclictest7950irq/56-dwc_otg_08:25:322
760699346,11cyclictest7750irq/56-dwc_otg08:35:362
760699343,9cyclictest7950irq/56-dwc_otg_09:50:162
760699342,8cyclictest10710-21/usr/sbin/munin11:55:362
7606993416,5cyclictest16004-21latency_hist12:10:162
760699341,5cyclictest0-21swapper/212:20:272
7606993415,6cyclictest14376-21uniq09:00:322
66132340,3sleep231-21ksoftirqd/207:05:422
760799337,23cyclictest0-21swapper/309:50:223
760799332,5cyclictest0-21swapper/309:15:313
760799331,7cyclictest3583-21/usr/sbin/munin11:35:303
760799331,6cyclictest0-21swapper/311:00:223
7606993318,4cyclictest28713-21idleruntime08:10:302
288452330,4sleep2760699cyclictest09:40:292
760799329,5cyclictest16298-21aten_rbpower_en12:10:283
760799329,5cyclictest15219-21fschecks_count07:30:333
760799329,5cyclictest13244-21/usr/sbin/munin07:25:293
760799327,4cyclictest13295-21timerwakeupswit08:55:433
760799327,22cyclictest0-21swapper/309:20:283
760799327,22cyclictest0-21swapper/309:05:273
760799327,21cyclictest0-21swapper/311:15:273
760799326,4cyclictest7950irq/56-dwc_otg_12:30:253
760799326,23cyclictest0-21swapper/308:50:283
760799326,23cyclictest0-21swapper/307:35:273
760799321,7cyclictest5593-21tune2fs11:40:323
7607993213,4cyclictest381ktimersoftd/308:25:283
7607993213,4cyclictest381ktimersoftd/308:25:283
760799321,22cyclictest7123-21timerwakeupswit10:10:423
760699325,5cyclictest7850irq/56-dwc_otg_12:10:312
760699325,10cyclictest7950irq/56-dwc_otg_09:30:432
760699323,5cyclictest18142-21aten_rbpower_po12:15:282
760699322,3cyclictest7850irq/56-dwc_otg_07:50:272
760699322,16cyclictest0-21swapper/207:35:302
760799317,5cyclictest15577-21seq09:04:123
760799317,4cyclictest7950irq/56-dwc_otg_09:25:423
7607993116,5cyclictest9373-21interrupts11:50:333
7607993114,14cyclictest39-21ksoftirqd/308:15:263
7607993113,5cyclictest12918-21latency_hist07:25:153
7607993113,5cyclictest12918-21latency_hist07:25:153
760799311,10cyclictest18113-21grep12:15:283
760699317,14cyclictest0-21swapper/208:15:572
760699315,5cyclictest7850irq/56-dwc_otg_11:30:302
760699313,12cyclictest7850irq/56-dwc_otg_08:30:392
760699311,7cyclictest9349-21grep08:45:362
760699311,5cyclictest7950irq/56-dwc_otg_07:20:292
760699311,5cyclictest7950irq/56-dwc_otg_07:20:292
7606993115,8cyclictest27036-21aten_rbpower_cu09:35:292
760699311,4cyclictest7950irq/56-dwc_otg_11:20:272
760699311,4cyclictest7950irq/56-dwc_otg_11:20:272
7606993112,12cyclictest1279-21taskset11:29:522
760799307,4cyclictest7750irq/56-dwc_otg10:45:153
760799306,6cyclictest7850irq/56-dwc_otg_08:10:153
760799306,14cyclictest0-21swapper/307:55:213
760799306,14cyclictest0-21swapper/307:55:213
760799304,4cyclictest7850irq/56-dwc_otg_10:15:393
760799301,6cyclictest128-21systemd-journal11:28:523
7607993012,5cyclictest16452-21taskset10:36:303
7607993010,6cyclictest27944-21uptime11:10:423
760699307,21cyclictest0-21swapper/209:15:272
760699303,4cyclictest0-21swapper/209:15:162
760699303,4cyclictest0-21swapper/209:15:162
760699303,15cyclictest7850irq/56-dwc_otg_12:00:352
760699302,11cyclictest7950irq/56-dwc_otg_11:50:362
7606993013,6cyclictest301ktimersoftd/210:23:442
760699301,18cyclictest6218-21ls11:40:402
760799298,9cyclictest7950irq/56-dwc_otg_12:39:053
760799297,4cyclictest7950irq/56-dwc_otg_11:30:293
7607992922,5cyclictest7198-21fschecks_time08:40:323
760799291,9cyclictest19333-21grep09:15:163
760799291,9cyclictest19333-21grep09:15:163
760799291,10cyclictest0-21swapper/308:10:313
760699295,4cyclictest7750irq/56-dwc_otg09:50:342
760699294,5cyclictest7850irq/56-dwc_otg_08:20:252
760699293,10cyclictest7950irq/56-dwc_otg_09:20:332
760699292,17cyclictest0-21swapper/207:30:352
760699291,5cyclictest7630-21latency11:45:352
760699291,3cyclictest7950irq/56-dwc_otg_11:10:272
7606992912,9cyclictest301ktimersoftd/210:30:292
7606992912,9cyclictest301ktimersoftd/210:30:292
760699291,25cyclictest25895-21missed_timers12:35:372
7606992911,4cyclictest26059-21unixbench_multi11:05:442
7606992910,10cyclictest20153-21latency_hist07:45:162
760799285,4cyclictest7850irq/56-dwc_otg_11:58:573
760799285,20cyclictest30967-21missed_timers11:20:363
760799285,20cyclictest30967-21missed_timers11:20:363
760799281,5cyclictest1124-21snmpd08:48:203
760799281,13cyclictest7950irq/56-dwc_otg_10:05:293
760799281,13cyclictest7950irq/56-dwc_otg_08:05:153
760699281,3cyclictest7950irq/56-dwc_otg_11:15:252
760699281,17cyclictest0-21swapper/210:10:372
760799275,4cyclictest7750irq/56-dwc_otg10:50:383
760799272,7cyclictest7850irq/56-dwc_otg_12:20:283
760699277,18cyclictest0-21swapper/212:25:282
760699273,3cyclictest7750irq/56-dwc_otg09:05:372
760699271,9cyclictest7950irq/56-dwc_otg_10:58:462
7606992717,6cyclictest23640-21fschecks_count09:25:322
760799261,3cyclictest7950irq/56-dwc_otg_10:55:543
64732263,16sleep30-21swapper/307:05:403
760699251,3cyclictest0-21swapper/209:58:332
760799222,5cyclictest0-21swapper/309:57:403
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional