You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 23:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Fri May 02, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
239852920,2sleep20-21swapper/210:40:302
1516699894,26cyclictest0-21swapper/012:25:280
188002870,2sleep00-21swapper/008:45:430
226442860,2sleep30-21swapper/311:24:523
1516699846,50cyclictest26429-21kworker/u8:109:20:210
15175997915,46cyclictest0-21swapper/107:35:331
235672770,6sleep21518199cyclictest12:06:382
1516699771,59cyclictest24629-21wget07:35:340
135032750,3sleep0101ktimersoftd/012:34:340
43912720,2sleep10-21swapper/110:58:371
1516699721,23cyclictest1058-21apt-config08:00:160
1516699711,53cyclictest30693-21munin-run10:50:160
1516699701,56cyclictest31041-21sh10:50:310
1516699682,55cyclictest137-21systemd-journal09:10:160
1516699682,55cyclictest137-21systemd-journal09:10:160
15186996625,24cyclictest9887-21date12:30:163
1516699668,54cyclictest0-21swapper/008:05:270
1516699661,52cyclictest0-21swapper/007:30:340
15166996444,11cyclictest101ktimersoftd/011:30:360
1516699642,50cyclictest20513-21aten_rbpower_cu10:35:310
15166996419,34cyclictest26619-21/usr/sbin/munin12:10:370
256192630,2sleep10-21swapper/111:28:161
15186996326,18cyclictest28337-21/usr/sbin/munin07:45:423
15186996326,18cyclictest28337-21/usr/sbin/munin07:45:423
15166996329,13cyclictest7950irq/56-dwc_otg_08:45:160
15166996317,35cyclictest13838-21fschecks_count11:55:350
15186996223,25cyclictest29054-21rm10:46:323
15166996225,26cyclictest9289-21fschecks_time11:05:350
15166996219,30cyclictest0-21swapper/008:55:320
145802623,13sleep20-21swapper/207:06:222
145802623,13sleep20-21swapper/207:06:222
1516699617,8cyclictest0-21swapper/012:20:300
15166996115,33cyclictest5756-21timerwakeupswit09:25:460
15186996020,22cyclictest7850irq/56-dwc_otg10:25:373
15186996015,20cyclictest0-21swapper/311:35:373
1516699608,37cyclictest0-21swapper/007:20:310
1516699608,37cyclictest0-21swapper/007:20:310
15186995928,11cyclictest19688-21/usr/sbin/munin08:50:303
15186995923,21cyclictest24384-21taskset09:02:143
15186995923,13cyclictest12456-21ssh11:54:003
15186995922,21cyclictest8198-21ssh09:30:363
15186995921,13cyclictest7950irq/56-dwc_otg_09:10:313
15186995916,17cyclictest3144-21grep11:40:473
1516699593,42cyclictest0-21swapper/008:50:320
15166995925,20cyclictest0-21swapper/012:05:300
1516699592,47cyclictest18236-21wget09:45:340
15166995911,28cyclictest9-21ksoftirqd/011:40:360
15186995819,25cyclictest23069-21sh11:25:193
1516699581,6cyclictest3514-21ssh10:10:380
1516699581,48cyclictest26600-21fschecks_time07:40:330
15166995814,31cyclictest603-21grep10:05:460
15166995814,31cyclictest603-21grep10:05:460
1518699572,9cyclictest0-21swapper/311:45:273
15186995724,16cyclictest4372-21ssh09:25:113
15186995721,13cyclictest30554-21cut09:15:333
15186995627,12cyclictest26511-21cpuspeed07:40:303
15186995621,19cyclictest21975-21ssh10:36:143
15186995620,20cyclictest7950irq/56-dwc_otg_12:30:173
15186995616,26cyclictest8050irq/56-dwc_otg_12:35:353
1516699566,4cyclictest0-21swapper/010:25:290
1516699564,35cyclictest21604-21ssh09:50:290
1516699564,35cyclictest21604-21ssh09:50:290
268822550,5sleep226889-21missed_timers07:40:362
15186995522,18cyclictest20180-21chrt09:47:263
15186995521,12cyclictest19386-21ssh12:01:333
15166995532,6cyclictest7950irq/56-dwc_otg_11:20:310
1516699551,41cyclictest0-21swapper/007:45:320
1516699551,41cyclictest0-21swapper/007:45:320
15186995417,25cyclictest8050irq/56-dwc_otg_07:15:193
1516699542,45cyclictest15589-21kworker/0:210:41:310
1516699541,42cyclictest0-21swapper/007:25:350
15186995327,12cyclictest14341-21ssh11:12:493
15186995325,8cyclictest975-21apt-get08:00:193
15186995324,12cyclictest25629-21ssh10:41:403
15186995323,12cyclictest3890-21ssh10:56:543
15186995323,12cyclictest29409-21ssh12:14:463
15186995322,15cyclictest5263-21ssh09:25:393
15186995319,10cyclictest25481-21if_wlan009:55:403
1516699535,28cyclictest137-21systemd-journal08:30:160
1516699533,35cyclictest0-21swapper/008:15:370
15166995316,30cyclictest17841-21netstat07:15:380
1516699531,38cyclictest0-21swapper/008:35:400
15166995313,27cyclictest0-21swapper/009:17:550
1516699531,25cyclictest0-21swapper/011:00:270
1516699531,17cyclictest0-21swapper/008:00:220
15166995311,23cyclictest0-21swapper/010:55:310
147742533,43sleep30-21swapper/307:08:543
147742533,43sleep30-21swapper/307:08:543
1518699526,15cyclictest0-21swapper/307:10:213
15186995223,17cyclictest1917-21ssh10:08:443
15186995223,17cyclictest1917-21ssh10:08:443
15186995223,12cyclictest8366-21ssh10:17:223
15186995220,11cyclictest30852-21rm10:04:023
15186995215,24cyclictest7950irq/56-dwc_otg_10:31:073
15186995215,23cyclictest8608-21ssh11:05:123
15186995214,14cyclictest30800-21awk12:15:403
15186995214,14cyclictest30800-21awk12:15:403
15166995217,28cyclictest9-21ksoftirqd/011:45:450
15166995211,38cyclictest13-21rcuc/011:30:150
15166995210,7cyclictest9550irq/48-DMA08:47:300
15186995124,12cyclictest15720-21ssh11:15:183
15186995124,11cyclictest11849-21seq08:29:333
15186995121,15cyclictest11100-21ssh11:07:443
15186995120,18cyclictest3091-21ssh12:20:513
15186995115,27cyclictest17743-21sh09:45:013
15186995115,22cyclictest0-21swapper/309:54:413
15186995115,22cyclictest0-21swapper/309:54:413
1517599513,13cyclictest7950irq/56-dwc_otg_10:05:291
1517599513,13cyclictest7950irq/56-dwc_otg_10:05:291
1516699512,24cyclictest27863-21grep09:10:440
15166995122,3cyclictest8050irq/56-dwc_otg_11:50:290
15166995115,17cyclictest17255-21fschecks_count10:30:380
1516699511,40cyclictest30156-21wget07:50:320
1516699511,28cyclictest18179-21wc12:40:160
1518699506,16cyclictest0-21swapper/308:45:293
15186995023,13cyclictest16537-21ssh11:58:523
15186995020,11cyclictest29219-21rm11:33:023
15186995017,14cyclictest13961-21runrttasks09:39:133
15186995015,12cyclictest8648-21tr08:20:333
1517599504,26cyclictest0-21swapper/110:25:291
1516699507,9cyclictest7950irq/56-dwc_otg_08:10:280
1516699503,28cyclictest0-21swapper/010:15:330
1516699502,22cyclictest0-21swapper/012:15:420
1516699502,22cyclictest0-21swapper/012:15:420
1516699501,3cyclictest0-21swapper/009:35:310
1516699501,36cyclictest15290-21cat09:40:390
1516699501,30cyclictest0-21swapper/009:30:220
15186994916,11cyclictest14669-21ntp_states08:35:413
1518199492,10cyclictest0-21swapper/212:25:282
1518699487,25cyclictest0-21swapper/307:20:293
1518699487,25cyclictest0-21swapper/307:20:293
15186994822,15cyclictest9957-21/usr/sbin/munin10:20:283
15186994821,11cyclictest8091-21chrt08:17:333
15186994818,19cyclictest25022-21ssh12:09:183
15186994817,12cyclictest1118-21dpkg08:00:163
15186994816,10cyclictest7950irq/56-dwc_otg_10:11:593
1517599487,26cyclictest0-21swapper/107:50:211
1516699484,36cyclictest16718-21ls11:15:450
15186994715,12cyclictest31292-21chrt07:51:123
15186994712,23cyclictest8050irq/56-dwc_otg_08:14:433
1516699472,38cyclictest0-21swapper/010:21:000
15166994719,13cyclictest8050irq/56-dwc_otg_10:05:150
15186994618,11cyclictest7950irq/56-dwc_otg_07:40:163
15186994617,9cyclictest0-21swapper/308:58:573
1517599468,25cyclictest0-21swapper/108:50:291
15186994513,17cyclictest8050irq/56-dwc_otg_07:31:563
15186994512,11cyclictest7950irq/56-dwc_otg_10:51:483
1518199457,24cyclictest0-21swapper/207:30:302
1517599457,28cyclictest0-21swapper/107:47:361
1517599457,28cyclictest0-21swapper/107:47:361
1517599452,4cyclictest7850irq/56-dwc_otg09:55:341
1517599452,27cyclictest0-21swapper/108:05:271
15175994520,4cyclictest878-21runrttasks10:00:461
1516699452,4cyclictest0-21swapper/011:40:160
15166994522,5cyclictest16034-21ps07:10:430
1516699451,26cyclictest15512-21latency_hist11:15:160
1516699451,24cyclictest0-21swapper/012:00:450
15166994510,18cyclictest9-21ksoftirqd/008:20:390
147562452,14sleep00-21swapper/007:08:380
147562452,14sleep00-21swapper/007:08:380
15186994417,13cyclictest0-21swapper/308:44:163
1518199447,23cyclictest0-21swapper/209:00:322
1517599447,24cyclictest0-21swapper/109:25:301
1517599447,23cyclictest0-21swapper/108:10:311
1517599445,25cyclictest6391-21ssh10:15:321
15166994415,4cyclictest13723-21apt08:35:160
243822430,3sleep215158-21cyclictest09:54:412
243822430,3sleep215158-21cyclictest09:54:412
15186994328,12cyclictest4181-21ls08:05:443
15186994314,10cyclictest8050irq/56-dwc_otg_08:32:193
15181994319,5cyclictest6351-21/usr/sbin/munin11:00:432
1517599436,26cyclictest7850irq/56-dwc_otg12:00:161
1517599436,24cyclictest7850irq/56-dwc_otg12:15:161
1517599434,26cyclictest0-21swapper/111:50:291
15175994317,13cyclictest21-21rcuc/110:20:281
15166994314,25cyclictest24173-21unixbench_multi09:00:460
161332420,2sleep20-21swapper/211:58:082
15186994213,10cyclictest7850irq/56-dwc_otg07:26:233
15181994235,5cyclictest8050irq/56-dwc_otg_09:40:282
1517599427,22cyclictest0-21swapper/111:35:311
1517599427,22cyclictest0-21swapper/107:40:291
1517599426,27cyclictest7850irq/56-dwc_otg08:20:161
1517599426,25cyclictest0-21swapper/111:40:321
224052410,2sleep1221ktimersoftd/111:24:221
1518199417,23cyclictest0-21swapper/209:15:292
1517599417,23cyclictest0-21swapper/109:00:321
1517599416,21cyclictest0-21swapper/107:30:301
1517599414,9cyclictest7950irq/56-dwc_otg_07:25:311
15181994024,5cyclictest32720-21sensors10:05:432
15181994024,5cyclictest32720-21sensors10:05:432
15175994019,6cyclictest878-21runrttasks11:15:161
1518199392,8cyclictest26442-21cron09:10:162
1518199392,8cyclictest26442-21cron09:10:162
15181993920,10cyclictest9953-21ssh09:32:382
15181993914,5cyclictest11340-21sh11:51:552
15175993925,5cyclictest18825-21latency_hist07:20:161
15175993912,25cyclictest0-21swapper/109:05:311
15175993912,25cyclictest0-21swapper/109:05:311
1517599391,16cyclictest30268-21sh09:15:291
15186993822,13cyclictest25575-21ntpq09:05:403
15186993822,13cyclictest25575-21ntpq09:05:403
1518199387,21cyclictest0-21swapper/210:50:312
1518199386,25cyclictest0-21swapper/207:10:312
15181993817,5cyclictest26447-21diskmemload10:18:362
1517599387,22cyclictest0-21swapper/108:35:321
1517599386,22cyclictest0-21swapper/110:40:331
15175993818,5cyclictest878-21runrttasks10:45:331
15175993814,4cyclictest6789-21/usr/sbin/munin12:25:441
204712370,1sleep20-21swapper/207:24:182
204712370,1sleep20-21swapper/207:24:182
1518199372,7cyclictest8050irq/56-dwc_otg_12:15:382
1518199372,7cyclictest8050irq/56-dwc_otg_12:15:382
15181993722,8cyclictest7950irq/56-dwc_otg_11:05:412
1518199371,25cyclictest2958-21ls08:05:162
1517599377,21cyclictest14894-21kworker/u8:408:45:221
1517599373,6cyclictest0-21swapper/111:30:311
15175993722,5cyclictest3127-21ssh09:21:451
1518199368,7cyclictest8050irq/56-dwc_otg_09:20:442
1518199362,8cyclictest8050irq/56-dwc_otg_07:55:302
1518199362,5cyclictest0-21swapper/210:50:002
1518199361,24cyclictest5924-21timerwakeupswit08:10:462
1517599361,8cyclictest8050irq/56-dwc_otg_11:00:441
1518199354,5cyclictest28143-21grep11:30:472
15181993522,10cyclictest0-21swapper/211:35:462
1518199351,5cyclictest24603-21aten_rbpower_po07:35:342
15181993511,5cyclictest14409-21if_wlan008:35:372
1517599356,6cyclictest7850irq/56-dwc_otg09:30:331
1517599355,17cyclictest7850irq/56-dwc_otg10:10:351
15175993525,8cyclictest8050irq/56-dwc_otg_12:00:421
1518199346,19cyclictest7950irq/56-dwc_otg_12:33:122
1518199344,10cyclictest8050irq/56-dwc_otg_08:45:172
1518199343,9cyclictest28477-21ssh10:00:282
15181993415,4cyclictest26447-21diskmemload11:14:212
15181993414,5cyclictest26588-21modprobe11:30:152
15181993412,7cyclictest11252-21/usr/sbin/munin10:20:472
1517599348,13cyclictest18571-21fschecks_time09:45:381
1517599348,11cyclictest8050irq/56-dwc_otg_12:05:441
1517599346,5cyclictest7950irq/56-dwc_otg_12:15:341
1517599346,5cyclictest7950irq/56-dwc_otg_12:15:341
1517599346,5cyclictest0-21swapper/107:10:311
1517599341,5cyclictest0-21swapper/111:19:011
15175993413,6cyclictest15164-21ssh12:35:441
1517599341,24cyclictest32465-21missed_timers07:55:401
1517599341,21cyclictest32008-21ls10:50:441
1518199338,15cyclictest0-21swapper/212:35:442
1518199336,14cyclictest0-21swapper/211:40:322
15181993321,6cyclictest28167-21ssh12:12:312
1518199331,6cyclictest30127-21wget07:50:302
1518199331,5cyclictest7269-21ls08:15:372
1518199331,23cyclictest0-21swapper/209:25:302
15181993311,5cyclictest0-21swapper/210:55:222
1517599334,5cyclictest8949-21/usr/sbin/munin11:05:371
1517599331,22cyclictest10652-21uniq08:25:361
1517599331,12cyclictest0-21swapper/112:35:161
15175993310,4cyclictest0-21swapper/109:35:371
1518199329,4cyclictest21415-21rm12:05:152
1518199328,5cyclictest18267-21cut09:45:342
1518199321,7cyclictest19734-21tune2fs08:50:322
15181993213,7cyclictest0-21swapper/208:55:452
1518199321,20cyclictest8050irq/56-dwc_otg_11:20:312
15181993210,14cyclictest0-21swapper/212:20:332
1517599327,16cyclictest0-21swapper/111:45:291
1517599325,5cyclictest8050irq/56-dwc_otg_09:10:341
1517599321,5cyclictest7850irq/56-dwc_otg12:25:151
1518199319,11cyclictest0-21swapper/210:31:492
1518199317,14cyclictest7850irq/56-dwc_otg07:25:342
1518199316,18cyclictest7950irq/56-dwc_otg_09:58:482
1518199316,14cyclictest31-21ksoftirqd/210:25:222
1518199315,6cyclictest8050irq/56-dwc_otg_10:10:402
1518199311,8cyclictest10380-21aten_rbpower_cu08:25:322
1518199311,12cyclictest11450-21ssh09:35:312
1517599312,6cyclictest7850irq/56-dwc_otg09:50:461
1517599312,6cyclictest7850irq/56-dwc_otg09:50:461
1517599311,8cyclictest15049-21fschecks_time09:40:361
1517599311,12cyclictest3074-21ls08:05:161
1518199305,5cyclictest7950irq/56-dwc_otg_07:45:342
1518199305,5cyclictest7950irq/56-dwc_otg_07:45:342
15181993020,7cyclictest8795-21fschecks_count08:20:362
15181993012,10cyclictest232842chrt10:39:262
1518199301,14cyclictest2729650ssh09:10:372
1517599304,8cyclictest20573-21munin-run07:25:161
1517599304,8cyclictest20573-21munin-run07:25:161
1517599303,14cyclictest8050irq/56-dwc_otg_08:30:341
15175993024,4cyclictest21144-21ls10:35:401
15175993011,12cyclictest0-21swapper/110:30:351
132192300,2sleep20-21swapper/208:30:442
15181992918,4cyclictest5971-21idleruntime11:45:342
1518199291,21cyclictest0-21swapper/208:05:272
1517599292,18cyclictest0-21swapper/108:56:171
1518199287,12cyclictest0-21swapper/211:16:292
1518199281,6cyclictest17866-21ntpq07:15:382
1518199281,21cyclictest0-21swapper/208:46:492
1517599284,11cyclictest8050irq/56-dwc_otg_08:40:461
1517599272,5cyclictest0-21swapper/108:20:301
146462274,15sleep10-21swapper/107:07:141
146462274,15sleep10-21swapper/107:07:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional