You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-04 - 08:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Fri Sep 04, 2026 00:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1129421730,5sleep1670199cyclictest21:40:361
2451721050,2sleep20-21swapper/200:34:052
6700991023,58cyclictest30790-21ssh21:25:280
285862860,2sleep30-21swapper/323:20:303
670099834,44cyclictest60-21kswapd023:15:270
265612800,2sleep10-21swapper/122:38:211
670099789,53cyclictest5611-21fschecks_time20:35:340
670099778,55cyclictest19653-21tune2fs19:45:340
290932770,2sleep10-21swapper/122:01:251
285692750,3sleep0670099cyclictest22:40:380
670099749,47cyclictest17068-21sort21:05:420
6700997327,16cyclictest60-21kswapd023:05:250
670099727,46cyclictest29324-21sort00:00:330
6700997224,11cyclictest0-21swapper/000:20:280
670099714,56cyclictest0-21swapper/023:40:410
6700997014,42cyclictest0-21swapper/022:00:270
670099695,41cyclictest974-21zcat23:25:400
6700996910,43cyclictest9335-21rm22:17:010
670099686,49cyclictest14165-21latency_hist21:00:160
670099684,37cyclictest29303-21latency_hist00:40:150
6700996810,42cyclictest0-21swapper/022:10:480
245702680,3sleep20-21swapper/223:55:162
93632670,4sleep39367-21kernelversion20:45:353
670099677,47cyclictest11003-21ntpq19:20:380
6700996712,30cyclictest17999-21rm21:10:250
6700996710,34cyclictest3596-21switchtime21:30:460
6700996710,34cyclictest3596-21switchtime21:30:460
50392670,6sleep30-21swapper/319:05:303
670199663,3cyclictest7750irq/56-dwc_otg23:59:081
307052660,2sleep30-21swapper/322:43:273
197402660,21sleep27950irq/56-dwc_otg_23:49:082
121992660,1sleep30-21swapper/323:40:153
670099648,42cyclictest8260-21rm00:14:420
670099644,34cyclictest15775-21proc_pri21:45:400
6700996414,40cyclictest472-21ssh22:05:560
6700996414,40cyclictest472-21ssh22:05:560
147862640,2sleep30-21swapper/321:00:373
670099637,39cyclictest3468-21aten_rbpower_po20:30:280
670099634,43cyclictest15495-21chrt22:25:200
670099628,28cyclictest17697-21timerwakeupswit23:45:420
670099626,43cyclictest13766-21ssh22:22:340
670099625,40cyclictest25516-21rm22:36:260
670099624,43cyclictest0-21swapper/023:58:090
670099624,40cyclictest0-21swapper/022:55:560
6700996219,24cyclictest9095-21/usr/sbin/munin23:35:390
112302620,1sleep30-21swapper/322:20:163
670099617,29cyclictest16282-21latency19:35:340
670099614,42cyclictest22523-21ssh00:30:460
6700996116,5cyclictest0-21swapper/022:30:290
6700996116,5cyclictest0-21swapper/022:30:290
670099611,46cyclictest0-21swapper/021:24:550
6700996112,39cyclictest24263-21ssh23:54:540
670099609,38cyclictest0-21swapper/020:27:160
670099606,39cyclictest24118-21ssh21:16:400
6700996015,24cyclictest22802-21sh21:55:130
6700996013,33cyclictest20236-21ssh00:28:580
6700996013,33cyclictest20236-21ssh00:28:580
6700996010,31cyclictest0-21swapper/019:55:390
670099592,43cyclictest0-21swapper/020:15:390
670099591,9cyclictest23290-21seq21:55:290
6700995912,36cyclictest29944-21ssh23:21:040
670099589,34cyclictest15050-21pluginstate21:00:410
670099588,39cyclictest0-21swapper/000:09:240
670099588,39cyclictest0-21swapper/000:09:240
670099586,35cyclictest0-21swapper/023:04:180
670099586,35cyclictest0-21swapper/023:04:180
670099583,41cyclictest0-21swapper/020:13:380
670099583,41cyclictest0-21swapper/020:13:380
6700995813,34cyclictest1919-21ssh22:47:360
83652570,2sleep30-21swapper/300:14:543
670099577,11cyclictest0-21swapper/019:15:150
670099573,46cyclictest9897-21taskset19:17:470
6700995711,42cyclictest4661-21awk22:50:450
670099563,33cyclictest0-21swapper/021:35:520
670099562,46cyclictest0-21swapper/020:41:040
670099562,30cyclictest25767-21grep20:00:420
6700995614,39cyclictest101ktimersoftd/019:45:160
670099561,34cyclictest27690-21grep20:05:440
670099558,33cyclictest0-21swapper/019:26:090
670099558,33cyclictest0-21swapper/019:26:090
6700995510,37cyclictest0-21swapper/000:17:350
95602540,2sleep10-21swapper/122:17:241
6700995426,14cyclictest7850irq/56-dwc_otg_20:50:290
670099541,39cyclictest0-21swapper/023:32:200
670099536,11cyclictest0-21swapper/019:30:280
6700995327,5cyclictest11898-21timerandwakeup21:40:440
670099531,26cyclictest496-21cat20:20:420
670099525,34cyclictest20775-21ssh23:10:390
670099522,5cyclictest8989-21aten_rbpower_po20:45:290
63512523,12sleep00-21swapper/019:09:210
670099512,36cyclictest0-21swapper/019:51:360
670199503,8cyclictest7950irq/56-dwc_otg_21:05:261
670199501,10cyclictest0-21swapper/100:00:281
670199493,28cyclictest7950irq/56-dwc_otg_00:10:161
670199493,28cyclictest7950irq/56-dwc_otg_00:10:161
6701994925,7cyclictest3341-21aten_rbpower_vo22:10:331
670199491,5cyclictest7750irq/56-dwc_otg20:15:281
670299487,26cyclictest0-21swapper/221:55:212
6702994723,5cyclictest32237-21sh23:25:152
670199451,10cyclictest12750-21ls19:25:371
670199451,10cyclictest12750-21ls19:25:371
670399447,24cyclictest0-21swapper/320:25:213
670399436,26cyclictest0-21swapper/321:43:363
6702994323,4cyclictest3892-21sh23:30:152
6702994319,5cyclictest29093-21aten_rbpower_en00:00:302
670199431,4cyclictest16231-21users21:45:441
670199431,14cyclictest21-21rcuc/100:25:261
670199431,14cyclictest21-21rcuc/100:25:261
670199421,20cyclictest9255-21tune2fs00:15:341
6701994210,22cyclictest0-21swapper/121:55:211
277652420,2sleep20-21swapper/220:05:452
6703994113,12cyclictest0-21swapper/300:25:403
6703994113,12cyclictest0-21swapper/300:25:403
670299406,19cyclictest19812-21kworker/u8:220:25:212
6702994023,6cyclictest20953-21idleruntime-cro00:30:152
6702994023,6cyclictest20953-21idleruntime-cro00:30:152
670299401,25cyclictest0-21swapper/219:50:302
6701994015,15cyclictest21-21rcuc/119:50:451
274982400,2sleep30-21swapper/323:19:103
670299394,5cyclictest7950irq/56-dwc_otg_21:30:372
670299394,5cyclictest7950irq/56-dwc_otg_21:30:372
6702993914,9cyclictest17687-21sh21:10:162
670299391,3cyclictest7950irq/56-dwc_otg_23:10:312
670199398,5cyclictest7850irq/56-dwc_otg_20:40:151
670199397,8cyclictest7950irq/56-dwc_otg_20:35:151
6701993914,7cyclictest14405-21aten_rbpower_po21:00:311
670399386,23cyclictest23202-21ssh21:55:213
6703993813,10cyclictest7850irq/56-dwc_otg_21:20:413
6702993818,5cyclictest10815-21uname19:20:362
670199386,18cyclictest0-21swapper/120:41:521
670199384,7cyclictest2963-21latency_hist22:10:161
670199384,7cyclictest2963-21latency_hist22:10:161
6701993819,10cyclictest11923-21idleruntime-cro19:25:161
6701993814,5cyclictest8536-21netstat22:55:361
670199381,22cyclictest0-21swapper/119:45:501
670199381,11cyclictest0-21swapper/122:51:091
670399371,4cyclictest7950irq/56-dwc_otg_22:30:433
670399371,4cyclictest7950irq/56-dwc_otg_22:30:433
670299375,17cyclictest0-21swapper/219:25:372
670299375,17cyclictest0-21swapper/219:25:372
6702993721,8cyclictest7750irq/56-dwc_otg20:50:352
670299371,5cyclictest28333-21fschecks_time22:40:352
6702993714,5cyclictest4341-21aten_rbpower_po23:30:342
670299371,24cyclictest0-21swapper/223:15:292
670199371,9cyclictest20952-21ssh23:50:341
6701993717,13cyclictest27098-21sh00:36:161
6701993712,6cyclictest20495-21ssh23:10:361
670199371,25cyclictest21318-21ssh00:30:221
670199371,16cyclictest26191-21chrt21:20:151
6701993711,5cyclictest17343-21awk19:40:151
8422360,2sleep30-21swapper/322:45:443
6703993618,5cyclictest2939-21cut21:30:373
6703993618,5cyclictest2939-21cut21:30:373
670299369,17cyclictest301ktimersoftd/222:00:282
6702993627,5cyclictest301ktimersoftd/220:10:262
6702993627,5cyclictest301ktimersoftd/220:10:262
6702993619,15cyclictest2545650ssh22:36:202
670299361,22cyclictest0-21swapper/200:20:282
670199364,5cyclictest7950irq/56-dwc_otg_21:30:371
670199364,5cyclictest7950irq/56-dwc_otg_21:30:371
6701993617,6cyclictest14966-21sshd23:42:561
6701993615,4cyclictest221ktimersoftd/122:30:331
6701993615,4cyclictest221ktimersoftd/122:30:331
6701993614,6cyclictest14245-21fschecks_time19:30:331
670199361,16cyclictest0-21swapper/120:50:261
670199361,10cyclictest30514-21latency_hist21:25:161
670399357,3cyclictest0-21swapper/321:25:213
670299357,14cyclictest19134-21latency_hist19:45:152
670299352,17cyclictest7950irq/56-dwc_otg_00:18:242
670299351,5cyclictest0-21swapper/223:58:142
670299351,27cyclictest0-21swapper/222:51:102
670299351,21cyclictest0-21swapper/220:15:232
670299351,12cyclictest10058-21head20:45:442
670199357,6cyclictest9450irq/48-DMA08:56:171
670199357,5cyclictest0-21swapper/122:20:341
6701993515,11cyclictest7950irq/56-dwc_otg_21:35:301
670199351,23cyclictest0-21swapper/119:55:301
6701993511,7cyclictest0-21swapper/121:50:371
19202350,2sleep20-21swapper/222:47:362
160262350,8sleep2301ktimersoftd/223:44:532
670399346,13cyclictest7950irq/56-dwc_otg_23:05:423
6703993428,4cyclictest9235-21fschecks_time00:15:343
6703993414,5cyclictest30692-21ssh00:01:253
670299345,19cyclictest10812-21/usr/sbin/munin21:40:342
670299342,9cyclictest7950irq/56-dwc_otg_22:10:302
6702993420,5cyclictest14354-21if_err_wlan019:30:352
6702993414,13cyclictest106382chrt23:37:322
6702993413,13cyclictest16415-21/usr/sbin/munin19:35:362
670199348,6cyclictest15868-21fschecks_count22:25:331
670199346,3cyclictest0-21swapper/123:20:421
670199346,16cyclictest0-21swapper/120:25:211
670199345,7cyclictest0-21swapper/100:10:421
670199345,10cyclictest7950irq/56-dwc_otg_23:15:441
670199343,11cyclictest7850irq/56-dwc_otg_20:55:361
670199342,4cyclictest7750irq/56-dwc_otg21:10:431
670199342,16cyclictest7850irq/56-dwc_otg_23:05:391
670199341,5cyclictest19334-21latency_hist19:45:161
6701993412,10cyclictest9819-21wc20:45:421
6701993411,14cyclictest0-21swapper/123:25:441
151922340,3sleep30-21swapper/323:43:213
90522330,6sleep3670399cyclictest22:55:433
670399336,5cyclictest17722-21latency_hist21:10:163
670399332,22cyclictest7750irq/56-dwc_otg19:20:423
670399332,18cyclictest7950irq/56-dwc_otg_00:20:433
670399331,23cyclictest31906-21uniq22:05:353
670399331,23cyclictest31906-21uniq22:05:353
670399331,22cyclictest4950-21latency_hist20:35:153
670299337,5cyclictest29-21rcuc/221:25:322
670299336,9cyclictest7950irq/56-dwc_otg_20:35:152
670299335,5cyclictest27771-21tail21:20:462
670299332,21cyclictest16242-21snmpd19:45:162
670299331,9cyclictest21553-21ssh22:31:442
670299331,9cyclictest21553-21ssh22:31:442
670299331,26cyclictest0-21swapper/221:19:062
670299331,12cyclictest7950irq/56-dwc_otg_20:38:022
670299330,16cyclictest7850irq/56-dwc_otg_22:15:302
670199336,4cyclictest0-21swapper/100:20:291
670199332,11cyclictest0-21swapper/122:40:341
6701993316,4cyclictest21-21rcuc/122:45:331
670199331,3cyclictest7750irq/56-dwc_otg23:00:311
670199331,3cyclictest7750irq/56-dwc_otg23:00:311
670399329,10cyclictest0-21swapper/321:13:483
670399327,10cyclictest0-21swapper/322:22:253
670399323,3cyclictest7750irq/56-dwc_otg00:30:333
6703993223,5cyclictest30412-21wget20:15:303
670399321,19cyclictest24555-21ssh23:55:163
670299327,7cyclictest0-21swapper/200:37:422
670299326,13cyclictest0-21swapper/222:56:012
670299322,15cyclictest0-21swapper/200:15:162
6702993216,9cyclictest1180-21runrttasks20:59:552
6702993215,5cyclictest8761-21fschecks_count19:15:312
6702993213,4cyclictest74350wget00:05:292
6702993213,4cyclictest74350wget00:05:292
670299321,22cyclictest32639-21timerwakeupswit22:05:452
670299321,22cyclictest32639-21timerwakeupswit22:05:452
670299321,19cyclictest15719-21ssh21:45:392
670299321,18cyclictest0-21swapper/221:10:402
670199327,14cyclictest23-21ksoftirqd/123:35:381
670199324,4cyclictest7850irq/56-dwc_otg_23:45:281
670199324,4cyclictest0-21swapper/120:10:291
670199324,4cyclictest0-21swapper/120:10:291
670199321,9cyclictest396-21grep20:20:401
670199321,6cyclictest6962-21cut19:10:311
670199321,17cyclictest0-21swapper/121:25:321
670399316,12cyclictest0-21swapper/323:46:523
670399313,3cyclictest0-21swapper/323:01:213
670399313,3cyclictest0-21swapper/323:01:213
6703993120,8cyclictest23198-21rm21:15:413
670299314,6cyclictest6666-21munin-node21:35:262
670299311,15cyclictest0-21swapper/219:55:302
670299311,12cyclictest7950irq/56-dwc_otg_22:20:382
6702993110,11cyclictest16712-21sh23:05:352
231532310,6sleep3670399cyclictest23:13:463
89452300,1sleep10-21swapper/119:15:331
670399309,7cyclictest15033-21aten_rbpower_po21:45:293
670399307,5cyclictest7750irq/56-dwc_otg23:30:353
670399307,4cyclictest7750irq/56-dwc_otg21:50:423
670399307,10cyclictest0-21swapper/300:38:543
670399306,4cyclictest25739-21smartpower23:55:423
670399304,12cyclictest7950irq/56-dwc_otg_19:10:423
670399302,7cyclictest7750irq/56-dwc_otg22:00:433
670399301,5cyclictest24912-21ls22:35:433
670399301,24cyclictest0-21swapper/322:25:313
670299302,13cyclictest7950irq/56-dwc_otg_19:10:412
670299301,5cyclictest20346-21sed21:50:472
670299301,5cyclictest0-21swapper/223:00:362
670299301,5cyclictest0-21swapper/223:00:362
670199302,11cyclictest0-21swapper/120:00:411
670399297,10cyclictest0-21swapper/322:12:333
670399296,5cyclictest7850irq/56-dwc_otg_21:35:323
670399293,7cyclictest24081-21unixbench_multi19:55:453
670399293,5cyclictest7850irq/56-dwc_otg_20:10:323
670399293,5cyclictest7850irq/56-dwc_otg_20:10:323
6703992920,4cyclictest1180-21runrttasks19:30:373
670399291,5cyclictest11843-21head20:50:433
670299294,5cyclictest7850irq/56-dwc_otg_21:00:312
670299293,7cyclictest7750irq/56-dwc_otg20:00:432
670299291,15cyclictest1821850ssh22:28:102
670199291,11cyclictest0-21swapper/120:09:531
48802296,15sleep14908-21aten_rbpower_en19:05:291
670399288,10cyclictest7950irq/56-dwc_otg_20:35:173
670399286,4cyclictest0-21swapper/300:08:163
670399286,4cyclictest0-21swapper/300:08:163
670399283,13cyclictest7950irq/56-dwc_otg_22:50:403
670399281,17cyclictest0-21swapper/319:18:263
670299282,5cyclictest7850irq/56-dwc_otg_20:20:232
49192286,10sleep20-21swapper/219:05:302
670399276,4cyclictest0-21swapper/323:28:443
670399273,4cyclictest7850irq/56-dwc_otg_20:05:293
670399272,4cyclictest7750irq/56-dwc_otg20:20:273
6703992721,4cyclictest21414-21fschecks_count19:50:333
6703992716,8cyclictest12408-21fschecks_count19:25:333
6703992716,8cyclictest12408-21fschecks_count19:25:333
670399271,4cyclictest7750irq/56-dwc_otg20:55:383
6703992616,7cyclictest25113-21fschecks_time20:00:333
670399261,5cyclictest7850irq/56-dwc_otg_19:40:403
670299261,3cyclictest0-21swapper/220:40:282
670399256,3cyclictest7950irq/56-dwc_otg_20:40:273
670399252,11cyclictest7750irq/56-dwc_otg19:45:413
670399243,18cyclictest16903-21grep19:35:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional