You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 23:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Sun Nov 23, 2025 12:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2047121740,2sleep30-21swapper/307:05:283
226821520,6sleep12172899cyclictest07:45:251
226821520,6sleep12172899cyclictest07:45:251
2061421260,5sleep20-21swapper/207:05:312
21727999316,61cyclictest14346-21wc11:00:340
112812890,4sleep32173099cyclictest10:11:563
112812890,4sleep32173099cyclictest10:11:563
21727998824,48cyclictest27709-21aten_rbpower_po12:05:220
21727998823,50cyclictest10273-21ssh09:24:450
21727998823,50cyclictest10273-21ssh09:24:450
21727998522,41cyclictest7353-21tr08:00:170
207232850,2sleep0101ktimersoftd/009:40:130
15862850,4sleep21589-21unixbench_singl09:10:362
21727998424,47cyclictest0-21swapper/008:55:180
21727998419,44cyclictest13964-21awk09:30:060
149412840,3sleep014944-21idleruntime08:20:260
21727998321,45cyclictest32602-21grep12:10:370
21727998223,40cyclictest27605-21aten_rbpower_vo09:50:260
21727998217,50cyclictest25797-21ssh09:46:220
21727998126,41cyclictest32144-21chrt09:05:590
21727998117,45cyclictest0-21swapper/011:20:250
146092810,5sleep22172999cyclictest12:30:272
2172799807,63cyclictest0-21swapper/010:05:130
21727997921,40cyclictest9597-21if_wlan008:05:250
2172799784,38cyclictest0-21swapper/010:30:180
21727997817,44cyclictest28821-21wc10:35:370
21727997814,42cyclictest0-21swapper/009:37:560
173822780,3sleep20-21swapper/209:35:072
127972780,2sleep20-21swapper/209:27:222
285622770,5sleep22172999cyclictest10:35:342
21727997716,50cyclictest17595-21missed_timers11:50:300
21727997619,41cyclictest14225-21ssh12:30:220
21727997619,30cyclictest23613-21latency_hist08:45:080
21727997617,42cyclictest2249-21ssh10:00:190
21727997518,39cyclictest19598-21taskset11:09:050
21727997518,31cyclictest5466-21rm11:34:300
21727997319,38cyclictest3163-21ssh09:14:180
21727997316,40cyclictest0-21swapper/010:55:250
21727997316,40cyclictest0-21swapper/008:50:240
21727997316,40cyclictest0-21swapper/008:50:240
21727997314,56cyclictest0-21swapper/011:40:060
21727997311,36cyclictest140-21systemd-journal11:14:470
21727997219,50cyclictest25331-21cat08:50:070
21727997219,50cyclictest25331-21cat08:50:070
21727997217,42cyclictest16052-21sh11:48:510
21727997217,40cyclictest28457-21taskset07:26:490
21727997215,43cyclictest0-21swapper/012:35:570
21727997214,46cyclictest15144-21awk09:30:330
21727997213,41cyclictest0-21swapper/012:16:260
21729997122,6cyclictest7950irq/56-dwc_otg_11:15:192
21727997123,39cyclictest23705-21latency_hist12:00:070
21727997123,39cyclictest23705-21latency_hist12:00:070
21727997118,39cyclictest22339-21ssh10:27:060
21727997117,41cyclictest0-21swapper/011:25:260
21727997116,40cyclictest0-21swapper/010:10:460
21727997116,40cyclictest0-21swapper/010:10:460
21727997115,41cyclictest5651-21rm09:17:010
120162710,2sleep30-21swapper/308:10:353
21727997020,34cyclictest16646-21chrt10:20:180
21727997017,40cyclictest6950-21ssh10:50:320
21727997015,44cyclictest1825-21latency_hist07:45:080
21727997015,44cyclictest0-21swapper/007:34:520
21727997015,43cyclictest0-21swapper/010:18:540
2172899692,6cyclictest7750irq/56-dwc_otg09:30:211
21727996916,39cyclictest0-21swapper/007:55:040
21727996914,42cyclictest0-21swapper/011:40:100
21727996913,43cyclictest0-21swapper/009:00:350
21727996912,49cyclictest0-21swapper/008:36:550
269562680,3sleep3381ktimersoftd/307:25:073
21727996822,36cyclictest5865-21tune2fs07:55:260
21727996816,37cyclictest31387-21rm10:40:210
80312670,1sleep20-21swapper/210:51:562
21727996716,38cyclictest639-21ssh09:57:150
21727996716,38cyclictest639-21ssh09:57:150
2172799671,49cyclictest7519-21grep12:20:350
211982670,2sleep30-21swapper/312:39:263
21727996621,24cyclictest21742-21/usr/sbin/munin07:10:280
21727996617,35cyclictest12769-21ssh12:27:560
21727996616,36cyclictest0-21swapper/010:49:260
21727996614,35cyclictest24850-21sshd07:16:560
21727996613,40cyclictest0-21swapper/007:37:240
21727996612,38cyclictest0-21swapper/008:26:240
2172899656,10cyclictest7950irq/56-dwc_otg_08:00:281
2172899653,5cyclictest7750irq/56-dwc_otg10:35:071
2172799656,51cyclictest0-21swapper/008:15:210
307472640,3sleep20-21swapper/211:24:412
21727996417,32cyclictest26212-21rm11:17:360
21727996416,35cyclictest0-21swapper/008:10:250
2172899635,5cyclictest10998-21unixbench_singl11:40:371
2172899632,5cyclictest0-21swapper/108:50:081
2172899632,5cyclictest0-21swapper/108:50:081
21727996323,21cyclictest3485-21sshd07:49:550
21727996323,21cyclictest3485-21sshd07:49:550
21727996314,33cyclictest0-21swapper/007:22:010
21727996313,36cyclictest0-21swapper/012:02:300
21727996313,29cyclictest1120-21runrttasks08:34:170
2172899605,26cyclictest7750irq/56-dwc_otg08:40:201
2172899604,11cyclictest31350-21ssh12:10:201
21728996017,35cyclictest10312-21cat10:55:291
21728996014,11cyclictest7950irq/56-dwc_otg_09:55:191
21728996014,11cyclictest7950irq/56-dwc_otg_09:55:191
2172899592,6cyclictest31331-21snmpd09:03:461
21728995914,7cyclictest20737-21ssh11:55:241
21728995914,7cyclictest20737-21ssh11:55:241
2172899588,4cyclictest0-21swapper/109:27:131
2172899585,4cyclictest7950irq/56-dwc_otg_10:05:351
2172899583,11cyclictest0-21swapper/110:04:321
2172899582,6cyclictest14977-21ntp_states12:30:321
21728995819,4cyclictest20626-21latency_hist09:40:071
2172899581,5cyclictest0-21swapper/112:37:571
2172899581,2cyclictest0-21swapper/108:50:171
2172899581,2cyclictest0-21swapper/108:50:171
2172899577,10cyclictest0-21swapper/111:33:421
2172899574,8cyclictest7750irq/56-dwc_otg12:15:201
2172899574,16cyclictest7850irq/56-dwc_otg_07:15:181
2172899571,6cyclictest7750irq/56-dwc_otg08:55:531
2172899566,4cyclictest7750irq/56-dwc_otg08:31:431
2172899564,4cyclictest0-21swapper/109:50:271
2172899562,12cyclictest7950irq/56-dwc_otg_07:55:371
2172899562,12cyclictest28899-21grep11:20:361
2172899561,20cyclictest0-21swapper/112:05:211
2172899557,4cyclictest7750irq/56-dwc_otg11:04:011
2172899556,5cyclictest7850irq/56-dwc_otg_10:44:051
2172899556,5cyclictest21362-21grep09:40:281
2172899555,4cyclictest0-21swapper/111:50:311
2172899553,4cyclictest7950irq/56-dwc_otg_07:37:151
2172899552,6cyclictest31331-21snmpd10:14:001
2172899552,6cyclictest31331-21snmpd10:14:001
2172899552,5cyclictest0-21swapper/111:06:281
2172899548,3cyclictest0-21swapper/109:13:541
2172899542,9cyclictest7850irq/56-dwc_otg_12:28:031
2172899542,6cyclictest7950irq/56-dwc_otg_11:15:251
2172899541,13cyclictest0-21swapper/110:35:431
2172999523,9cyclictest7950irq/56-dwc_otg_08:35:192
2172899527,9cyclictest7950irq/56-dwc_otg_10:30:071
2172899524,11cyclictest7950irq/56-dwc_otg_12:02:421
2172899522,5cyclictest0-21swapper/112:22:321
2172899522,14cyclictest0-21swapper/108:13:481
2172899516,5cyclictest24526-1kworker/u9:209:46:491
2172899513,5cyclictest0-21swapper/111:47:581
269692500,5sleep32173099cyclictest12:04:323
2172899507,11cyclictest0-21swapper/111:26:331
2172899501,6cyclictest0-21swapper/110:24:011
2173099493,28cyclictest0-21swapper/307:10:093
2172899485,6cyclictest7950irq/56-dwc_otg_09:15:181
2172899484,10cyclictest7750irq/56-dwc_otg10:15:201
2173099477,26cyclictest0-21swapper/308:30:203
2172899473,13cyclictest0-21swapper/107:10:091
2173099467,25cyclictest0-21swapper/308:40:203
2173099467,24cyclictest16497-21chrt08:25:203
2172899467,25cyclictest0-21swapper/111:35:221
2173099457,25cyclictest0-21swapper/309:45:223
2173099457,23cyclictest0-21swapper/308:50:203
2173099457,23cyclictest0-21swapper/308:50:203
2172999457,24cyclictest0-21swapper/208:20:222
2172899459,6cyclictest0-21swapper/108:35:181
21728994516,6cyclictest21-21rcuc/108:05:181
198922452,13sleep00-21swapper/007:05:140
2172999445,7cyclictest7850irq/56-dwc_otg_12:20:192
3962430,2sleep3394-21ssh09:10:213
2172999437,24cyclictest0-21swapper/209:00:202
2172899437,24cyclictest0-21swapper/107:20:191
2173099427,22cyclictest0-21swapper/308:20:223
21729994222,4cyclictest7387-21wget08:00:202
2173099415,21cyclictest20259-21cpuspeed08:35:213
2173099411,10cyclictest27969-21cut12:05:253
21729994117,14cyclictest21726-21cyclictest11:35:192
2172899412,8cyclictest7850irq/56-dwc_otg_07:25:171
2172999401,3cyclictest7850irq/56-dwc_otg_09:45:202
2172899406,3cyclictest7950irq/56-dwc_otg_08:25:241
2172899402,14cyclictest0-21swapper/107:40:201
2173099391,6cyclictest2715-21diskstats11:30:243
21730993916,4cyclictest2880-21fschecks_count10:45:233
2172999396,21cyclictest61-21kswapd008:25:142
155672390,2sleep10-21swapper/108:20:351
2172999387,22cyclictest0-21swapper/210:30:202
2172999384,25cyclictest0-21swapper/208:40:202
21729993813,5cyclictest2445-21/usr/sbin/munin11:30:362
21352380,2sleep20-21swapper/210:44:482
2173099377,21cyclictest0-21swapper/309:25:223
2173099376,7cyclictest0-21swapper/310:35:223
2173099371,23cyclictest16342-21latency_hist10:20:073
2172999373,27cyclictest7950irq/56-dwc_otg_10:25:202
21729993716,9cyclictest301ktimersoftd/207:51:532
2172999371,24cyclictest7950irq/56-dwc_otg_10:45:162
2172899379,16cyclictest0-21swapper/110:45:191
323712360,5sleep32173099cyclictest10:40:343
2173099362,6cyclictest32361-21timerwakeupswit11:25:353
21730993614,5cyclictest31312-21sh12:10:183
2173099361,24cyclictest7950irq/56-dwc_otg_11:05:183
2172999366,14cyclictest0-21swapper/207:10:092
2172999362,7cyclictest7850irq/56-dwc_otg_08:15:192
21729993610,7cyclictest10942-21grep11:40:362
2173099357,15cyclictest0-21swapper/307:30:203
21730993518,5cyclictest7082-21seq07:59:533
2172999358,15cyclictest14268-21rm11:00:342
2172999353,8cyclictest0-21swapper/212:02:572
21729993514,15cyclictest29-21rcuc/207:35:182
21729993510,4cyclictest0-21swapper/209:35:222
2172899351,16cyclictest8541-21ssh09:20:361
2172899351,16cyclictest8541-21ssh09:20:361
103462350,1sleep20-21swapper/209:24:572
103462350,1sleep20-21swapper/209:24:572
2173099344,19cyclictest37-21rcuc/311:10:193
2173099341,4cyclictest0-21swapper/310:00:083
2173099341,4cyclictest0-21swapper/310:00:083
2173099341,23cyclictest17896-21switchtime11:50:353
2173099341,20cyclictest31067-21ls11:25:083
2173099341,15cyclictest7950irq/56-dwc_otg_10:25:253
21730993411,4cyclictest0-21swapper/311:35:233
2172999348,5cyclictest7950irq/56-dwc_otg_07:20:062
21729993413,5cyclictest29-21rcuc/209:15:202
2173099337,11cyclictest7850irq/56-dwc_otg_09:30:233
2173099332,3cyclictest0-21swapper/311:40:343
21730993312,9cyclictest20010-21sh10:25:033
2173099331,23cyclictest7883-21ls08:00:263
2173099331,15cyclictest23827-21wget08:45:233
2173099331,15cyclictest23827-21wget08:45:233
2172999336,13cyclictest7950irq/56-dwc_otg_09:55:322
2172999336,13cyclictest7950irq/56-dwc_otg_09:55:322
2172999332,17cyclictest0-21swapper/211:05:582
21729993316,5cyclictest13632-21if_wlan010:15:282
21729993313,4cyclictest9656-21fschecks_count10:10:272
21729993313,4cyclictest9656-21fschecks_count10:10:272
21729993311,5cyclictest7850irq/56-dwc_otg_10:20:262
2172899333,7cyclictest4858-21grep07:50:351
2172899331,8cyclictest6542-21fschecks_time10:50:261
2173099327,22cyclictest0-21swapper/307:35:203
21730993226,4cyclictest4349-21ls09:15:273
21730993213,7cyclictest0-21swapper/312:30:193
2173099321,24cyclictest0-21swapper/310:00:203
2173099321,17cyclictest0-21swapper/311:20:083
21730993211,14cyclictest381ktimersoftd/309:25:083
21730993211,14cyclictest381ktimersoftd/309:25:083
2172999327,15cyclictest7950irq/56-dwc_otg_07:40:222
2172999326,12cyclictest7850irq/56-dwc_otg_12:15:482
2172999324,13cyclictest7750irq/56-dwc_otg07:30:072
2172999322,21cyclictest0-21swapper/212:26:442
21729993220,4cyclictest31239-21/usr/sbin/munin11:25:232
21729993212,5cyclictest25834-1kworker/u9:212:10:272
21729993211,13cyclictest110512chrt08:10:222
2172899326,13cyclictest0-21swapper/111:10:281
2173099319,5cyclictest7850irq/56-dwc_otg_10:55:253
2173099315,7cyclictest0-21swapper/310:50:283
2173099314,13cyclictest7850irq/56-dwc_otg_11:55:373
2173099314,13cyclictest7850irq/56-dwc_otg_11:55:373
2173099313,14cyclictest7950irq/56-dwc_otg_11:48:083
21730993118,5cyclictest381ktimersoftd/309:37:353
2173099311,5cyclictest10923-21ls12:25:253
21730993114,9cyclictest30379-21ssh09:54:213
21730993112,6cyclictest13573-21cpuspeed11:00:233
2173099311,21cyclictest29384-21uniq09:00:243
2172999317,4cyclictest0-21swapper/211:10:192
2172999316,16cyclictest0-21swapper/212:36:112
2172999314,17cyclictest14942-21ssh11:46:102
2172999313,21cyclictest7850irq/56-dwc_otg_10:05:202
2172999311,5cyclictest31-21ksoftirqd/208:50:072
2172999311,5cyclictest31-21ksoftirqd/208:50:072
21729993112,5cyclictest7850irq/56-dwc_otg_11:51:212
21729993110,5cyclictest0-21swapper/209:42:072
2172899314,5cyclictest7950irq/56-dwc_otg_08:15:361
2173099309,5cyclictest0-21swapper/310:05:263
2173099306,22cyclictest27557-21ls10:35:083
2173099302,10cyclictest23537-21/usr/sbin/munin07:15:263
2173099301,17cyclictest3817-21aten_rbpower_en07:50:213
2173099301,14cyclictest0-21swapper/309:40:343
2172999305,6cyclictest7850irq/56-dwc_otg_10:00:252
2172999304,5cyclictest7850irq/56-dwc_otg_08:55:472
2172999303,6cyclictest23282-21kworker/2:008:50:172
2172999303,6cyclictest23282-21kworker/2:008:50:172
21729993022,6cyclictest27905-21fschecks_count12:05:252
21729993011,9cyclictest0-21swapper/211:55:372
21729993011,9cyclictest0-21swapper/211:55:372
2173099294,20cyclictest7950irq/56-dwc_otg_12:20:183
21730992915,4cyclictest3193-21cat12:15:273
2173099291,16cyclictest3261-21kworker/u8:407:50:073
2173099291,16cyclictest3261-21kworker/u8:407:50:073
2172999296,4cyclictest0-21swapper/209:50:092
2172999296,10cyclictest7850irq/56-dwc_otg_08:30:302
2172999295,5cyclictest7950irq/56-dwc_otg_07:30:202
2172999294,10cyclictest7850irq/56-dwc_otg_09:05:212
2172999292,6cyclictest9812-21wget10:55:222
21729992911,5cyclictest7750irq/56-dwc_otg07:55:232
2172899294,6cyclictest31421-21latency09:05:271
2172899291,19cyclictest29476-21latency07:30:431
274162280,6sleep32173099cyclictest08:55:393
2173099284,13cyclictest7850irq/56-dwc_otg_07:25:283
2173099282,18cyclictest0-21swapper/309:09:023
21730992821,5cyclictest12878-21aten_rbpower_vo08:15:233
2173099282,15cyclictest0-21swapper/308:05:203
21729992813,5cyclictest2604-21/usr/sbin/munin07:45:302
21729992813,5cyclictest2604-21/usr/sbin/munin07:45:302
2173099271,6cyclictest32573-21/usr/sbin/munin07:40:223
2172999276,9cyclictest7950irq/56-dwc_otg_07:20:362
2172999276,5cyclictest0-21swapper/208:05:222
201122274,15sleep120124-21diskstats07:05:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional