You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 12:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Tue Nov 25, 2025 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2807221020,5sleep01314099cyclictest22:39:120
1314099991,74cyclictest3263-21latency_hist21:25:080
1314099991,74cyclictest3263-21latency_hist21:25:080
120142960,4sleep3381ktimersoftd/319:05:333
61232950,4sleep3381ktimersoftd/322:52:443
90482940,3sleep19046-21grep20:25:361
1314099945,81cyclictest7714-21ssh00:20:220
1314099945,81cyclictest7714-21ssh00:20:220
1314099941,70cyclictest19120-21apt-get23:10:180
1314099931,77cyclictest13904-21rm22:20:240
1314099931,77cyclictest0-21swapper/021:55:320
1314099921,74cyclictest1014-21fschecks_time20:05:240
1314099911,80cyclictest362-21df_inode22:45:250
1314099911,79cyclictest0-21swapper/020:35:260
1314099911,79cyclictest0-21swapper/020:35:260
1314099901,76cyclictest11405-21aten_rbpower_cu00:25:220
1314099901,75cyclictest0-21swapper/020:27:080
1314099896,75cyclictest0-21swapper/020:55:220
1314099881,77cyclictest18077-21sh21:43:510
1314099876,53cyclictest0-21swapper/020:40:220
1314099871,70cyclictest14961-21sh23:05:070
1314099871,69cyclictest8594-21grep21:30:380
1314099861,74cyclictest23028-21aten_rbpower_vo21:50:210
1314099861,70cyclictest10676-21sh22:59:140
82002850,3sleep2301ktimersoftd/220:25:252
1314099851,73cyclictest25461-21apt-get00:00:200
1314099851,65cyclictest0-21swapper/021:00:180
13140998432,20cyclictest7850irq/56-dwc_otg_23:30:230
1314099841,71cyclictest15062-21unixbench-2d23:45:360
1314099821,67cyclictest1744-21rm23:29:440
1314099821,67cyclictest0-21swapper/000:11:450
1314099821,61cyclictest0-21swapper/000:39:130
1314099821,56cyclictest32100-21rm00:08:580
1314099811,65cyclictest18899-21ssh22:25:570
1314099811,62cyclictest1120-21runrttasks21:27:520
1314099811,62cyclictest0-21swapper/021:19:530
1314099811,58cyclictest0-21swapper/022:15:120
13140998052,18cyclictest0-21swapper/023:15:230
1314099801,62cyclictest0-21swapper/019:35:380
1314099801,59cyclictest31417-21/usr/sbin/munin20:00:230
1314099792,67cyclictest6000-21ssh00:16:570
1314099791,65cyclictest5284-21taskset22:50:570
1314099791,63cyclictest14171-21ssh21:38:570
1314099791,57cyclictest19833-21ssh23:52:190
1314099791,38cyclictest6178-21ssh23:35:220
1314099791,38cyclictest6178-21ssh23:35:220
1314099791,36cyclictest1120-21runrttasks00:30:520
1314099781,65cyclictest0-21swapper/023:24:120
1314099781,60cyclictest6085-21df_inode22:10:240
1314099781,59cyclictest21371-21ls19:30:350
13140997712,48cyclictest27293-21sh21:14:000
139572760,5sleep21314299cyclictest21:38:312
1314299761,42cyclictest7950irq/56-dwc_otg_19:20:212
1314099761,56cyclictest20202-21sed21:45:360
1314099761,56cyclictest11444-21rm23:40:500
104252760,5sleep31314399cyclictest23:40:263
1314099753,57cyclictest0-21swapper/019:45:240
1314099751,63cyclictest2340-21rm22:05:250
1314099751,56cyclictest30782-21ssh22:42:000
1314099751,51cyclictest24015-21diskmemload22:30:360
1314099751,51cyclictest24015-21diskmemload22:30:360
1314099741,62cyclictest0-21swapper/020:18:560
1314099741,54cyclictest1036-21chrt22:03:490
1314099732,62cyclictest0-21swapper/020:22:030
1314099732,37cyclictest10013-1kworker/0:2H23:55:210
1314099731,53cyclictest18642-21/usr/sbin/munin19:25:370
1314099721,59cyclictest17508-21ssh23:07:310
1314099711,67cyclictest0-21swapper/019:10:360
1314099711,60cyclictest0-21swapper/021:05:490
1314099701,55cyclictest18350-21runrttasks20:52:030
1314099701,54cyclictest31111-21runrttasks19:59:590
49242690,2sleep20-21swapper/223:33:172
1314099695,49cyclictest9832-21apt-get20:30:190
1314099691,55cyclictest0-21swapper/019:41:220
1314099681,55cyclictest0-21swapper/020:46:290
1314099671,55cyclictest0-21swapper/019:22:150
1314199666,25cyclictest0-21swapper/120:55:221
1314099661,53cyclictest3970-21chrt20:13:310
55612650,2sleep229-21rcuc/200:15:562
1314099651,51cyclictest29280-21runrttasks19:54:360
13140996330,4cyclictest0-21swapper/019:15:190
13140996330,4cyclictest0-21swapper/019:15:190
13141996119,18cyclictest19738-21ssh00:36:251
13141996113,25cyclictest0-21swapper/122:25:201
1314299605,10cyclictest7750irq/56-dwc_otg21:50:182
13141995817,12cyclictest8889-21ssh23:38:291
13141995817,12cyclictest8889-21ssh23:38:291
1314299564,11cyclictest7850irq/56-dwc_otg_20:20:202
13141995622,20cyclictest7750irq/56-dwc_otg23:15:231
116732566,31sleep10-21swapper/119:05:281
159032550,2sleep20-21swapper/219:15:332
159032550,2sleep20-21swapper/219:15:332
13141995518,12cyclictest5428-21ssh22:51:161
13141995514,11cyclictest31630-21ntp_states23:25:331
13141995220,9cyclictest7850irq/56-dwc_otg_00:00:221
13141995215,22cyclictest13721-21sh00:28:051
13141995115,19cyclictest18724-21cat20:55:081
13141995115,15cyclictest14812-21sh23:04:501
13141995115,11cyclictest16275-21sh00:30:511
13141995113,13cyclictest4135-21ls21:25:291
13141995111,20cyclictest7066-21ssh00:19:231
13141995111,12cyclictest16518-21ssh23:48:231
13141995015,12cyclictest558-21/usr/sbin/munin22:45:271
13141995011,11cyclictest2420-21ssh00:11:571
1314399496,30cyclictest25092-21taskset22:35:213
13141994912,7cyclictest7850irq/56-dwc_otg_20:30:211
1314399487,26cyclictest0-21swapper/319:35:123
1314199489,21cyclictest28805-21ssh22:40:161
1314399479,5cyclictest7750irq/56-dwc_otg23:55:203
1314199477,26cyclictest0-21swapper/121:45:141
13141994725,9cyclictest9777-21apt-get22:15:201
1314199472,26cyclictest0-21swapper/119:10:221
13141994716,20cyclictest17206-21ssh21:42:041
13141994714,15cyclictest20445-21ssh23:10:391
13141994714,12cyclictest11650-21rm23:41:151
13141994713,12cyclictest30090-21ssh21:16:491
13141994713,12cyclictest12124-21cat20:35:271
13141994713,12cyclictest12124-21cat20:35:271
13141994712,12cyclictest7950irq/56-dwc_otg_23:55:141
1314399467,26cyclictest0-21swapper/300:30:233
1314399467,24cyclictest0-21swapper/319:10:223
1314399462,14cyclictest24015-21diskmemload22:45:223
1314199467,24cyclictest0-21swapper/121:35:231
13141994614,18cyclictest10477-21ssh00:23:521
13141994614,18cyclictest10477-21ssh00:23:521
13141994614,13cyclictest27518-21ntp_states21:55:311
13141994610,12cyclictest5058-21ssh22:09:141
13141994610,11cyclictest20737-21wget21:00:211
1314399453,33cyclictest7850irq/56-dwc_otg_23:10:203
1314299457,23cyclictest0-21swapper/220:50:212
1314399448,25cyclictest0-21swapper/300:00:223
1314399447,25cyclictest0-21swapper/300:25:223
1314399446,24cyclictest0-21swapper/320:25:223
13143994421,17cyclictest0-21swapper/323:20:193
13141994419,22cyclictest71950ssh22:03:101
13141994417,11cyclictest10221-21ssh21:33:471
13141994413,17cyclictest23188-21fschecks_count21:50:241
246272431,3sleep20-21swapper/222:35:082
246272431,3sleep20-21swapper/222:35:082
205732430,2sleep30-21swapper/322:29:373
1314399434,24cyclictest0-21swapper/323:49:353
1314399433,19cyclictest0-21swapper/323:05:193
1314199439,20cyclictest7587-21rm22:55:171
1314199437,24cyclictest0-21swapper/122:20:201
1314199437,24cyclictest0-21swapper/119:45:211
1314199436,10cyclictest0-21swapper/123:20:191
1314199433,24cyclictest7950irq/56-dwc_otg_19:30:211
13141994315,16cyclictest31950-21ssh00:08:381
13141994310,16cyclictest18642-21munin-node19:25:191
125892432,34sleep00-21swapper/019:07:050
1314399427,22cyclictest0-21swapper/300:15:223
1314199427,22cyclictest0-21swapper/120:00:221
13141994226,8cyclictest7850irq/56-dwc_otg_21:14:471
13141994217,13cyclictest23807-21mv19:38:431
13141994213,14cyclictest3311-21latency_hist21:25:081
13141994213,14cyclictest3311-21latency_hist21:25:081
13143994115,14cyclictest0-21swapper/300:05:223
13141994116,9cyclictest7750irq/56-dwc_otg22:35:191
1314399407,25cyclictest0-21swapper/322:00:203
1314399406,24cyclictest0-21swapper/321:00:203
1314399405,16cyclictest37-21rcuc/320:00:203
13143994010,26cyclictest0-21swapper/320:55:223
13142994021,4cyclictest1120-21runrttasks19:55:222
1314199408,5cyclictest6398-21latency22:10:281
1314199407,10cyclictest7950irq/56-dwc_otg_20:05:351
13141994012,11cyclictest24204-21ssh22:34:161
13141994012,11cyclictest24204-21ssh22:34:161
13141994010,12cyclictest17731-21ssh23:07:591
240212390,1sleep20-21swapper/223:57:522
1314399399,13cyclictest381ktimersoftd/321:25:303
1314399397,22cyclictest0-21swapper/320:30:223
1314399396,24cyclictest0-21swapper/323:25:133
1314399393,16cyclictest0-21swapper/319:25:223
1314399392,26cyclictest7850irq/56-dwc_otg_21:55:213
13143993918,11cyclictest7750irq/56-dwc_otg22:58:143
1314399391,17cyclictest381ktimersoftd/322:20:313
13143993911,14cyclictest39-21ksoftirqd/322:12:053
1314299393,8cyclictest0-21swapper/221:40:202
13142993913,6cyclictest12708-21unixbench_singl23:00:382
1314199399,11cyclictest19647-21ssh23:51:561
1314199397,10cyclictest30927-21chrt19:57:431
1314199396,9cyclictest7850irq/56-dwc_otg_23:30:231
1314199392,23cyclictest7950irq/56-dwc_otg_19:44:111
1314199391,8cyclictest15874-21ls20:45:311
13141993913,18cyclictest14693-21taskset20:41:191
183092380,3sleep318308-21ssh23:50:293
1314399388,8cyclictest37-21rcuc/323:05:083
1314399384,18cyclictest381ktimersoftd/321:44:333
13143993814,14cyclictest7950irq/56-dwc_otg_21:30:383
1314299387,21cyclictest0-21swapper/219:30:232
1314299386,22cyclictest0-21swapper/220:15:232
1314299384,18cyclictest0-21swapper/220:45:082
1314399378,8cyclictest39-21ksoftirqd/321:50:013
1314399378,17cyclictest37-21rcuc/320:05:223
1314399377,17cyclictest381ktimersoftd/321:11:403
13143993729,6cyclictest381ktimersoftd/300:35:183
1314399372,3cyclictest7850irq/56-dwc_otg_20:50:203
13143993718,4cyclictest29735-21cpu19:55:223
13143993716,9cyclictest381ktimersoftd/320:35:303
13143993716,9cyclictest381ktimersoftd/320:35:303
13143993714,8cyclictest7950irq/56-dwc_otg_19:54:403
1314399371,25cyclictest11554-21fschecks_time21:35:273
1314299379,6cyclictest7950irq/56-dwc_otg_23:50:372
1314299374,6cyclictest7750irq/56-dwc_otg22:20:352
38092361,4sleep20-21swapper/222:06:382
224702360,1sleep20-21swapper/221:49:542
13143993617,7cyclictest7850irq/56-dwc_otg_20:50:083
13143993615,9cyclictest31607-21sh21:19:483
1314399361,24cyclictest819-21fschecks_count00:10:263
1314399361,21cyclictest27638-21ls19:50:083
1314299368,24cyclictest0-21swapper/223:15:142
1314299368,15cyclictest7950irq/56-dwc_otg_22:55:082
1314299366,5cyclictest7750irq/56-dwc_otg00:05:082
1314299362,8cyclictest0-21swapper/222:25:192
13142993616,5cyclictest20562-21sh23:10:542
13142993610,12cyclictest0-21swapper/223:40:312
1314199368,9cyclictest0-21swapper/120:24:261
1314199366,17cyclictest0-21swapper/119:51:561
1314199366,15cyclictest0-21swapper/121:05:191
24592350,3sleep337-21rcuc/321:23:383
24592350,3sleep337-21rcuc/321:23:383
1314399352,16cyclictest7104-21grep20:20:363
1314399352,15cyclictest37-21rcuc/322:40:143
1314399351,4cyclictest23000-21wget21:50:213
1314299359,5cyclictest7850irq/56-dwc_otg_21:10:272
1314299357,25cyclictest0-21swapper/220:45:232
1314299355,23cyclictest0-21swapper/221:55:372
1314299352,8cyclictest13338-21latency_hist20:40:092
1314299352,8cyclictest13338-21latency_hist20:40:092
1314299351,5cyclictest0-21swapper/222:45:202
13142993512,6cyclictest31331-21snmpd22:12:232
1314299351,13cyclictest7750irq/56-dwc_otg20:10:362
1314199352,10cyclictest0-21swapper/120:10:191
13141993520,13cyclictest17479-21ls19:20:291
1314399348,18cyclictest0-21swapper/322:33:583
1314399348,18cyclictest0-21swapper/322:33:583
1314399344,12cyclictest37-21rcuc/323:35:353
1314399344,12cyclictest37-21rcuc/323:35:353
13143993415,12cyclictest16879-21aten_rbpower_cu19:20:223
1314399341,20cyclictest39-21ksoftirqd/323:16:153
1314299346,6cyclictest7750irq/56-dwc_otg23:25:352
1314299342,7cyclictest7950irq/56-dwc_otg_21:15:202
1314299342,14cyclictest7401-21ssh21:30:222
13142993421,11cyclictest7950irq/56-dwc_otg_23:35:312
13142993421,11cyclictest7950irq/56-dwc_otg_23:35:312
13142993417,9cyclictest1120-21runrttasks23:23:542
1314299341,6cyclictest12591-21unixbench_singl00:25:372
13141993410,5cyclictest16614-21apt19:20:081
13141993410,5cyclictest16614-21apt19:20:081
1314399337,11cyclictest7850irq/56-dwc_otg_19:30:203
13143993318,9cyclictest381ktimersoftd/322:17:423
1314399331,14cyclictest0-21swapper/300:20:343
1314399331,14cyclictest0-21swapper/300:20:343
1314299338,5cyclictest7750irq/56-dwc_otg22:45:082
1314299338,5cyclictest0-21swapper/222:40:082
1314299334,5cyclictest7750irq/56-dwc_otg21:25:232
1314299332,6cyclictest11124-21idleruntime-cro23:00:082
13142993326,5cyclictest1879-21latency_hist22:05:082
1314299331,8cyclictest1496650ssh23:45:352
13142993313,4cyclictest1120-21runrttasks20:06:432
13142993310,5cyclictest19426-21unixbench_singl00:35:372
13141993310,17cyclictest221ktimersoftd/120:19:201
1314399328,13cyclictest4177-21rm22:07:223
1314399324,11cyclictest37-21rcuc/323:32:383
1314399322,22cyclictest0-21swapper/320:12:453
13143993212,4cyclictest37-21rcuc/321:08:283
1314299328,12cyclictest7850irq/56-dwc_otg_00:10:352
1314299327,5cyclictest16327-21timerandwakeup23:05:342
1314299327,12cyclictest7750irq/56-dwc_otg19:40:352
1314299325,6cyclictest19962-21unixbench_multi20:55:372
1314299324,15cyclictest7850irq/56-dwc_otg_21:00:212
1314299322,6cyclictest0-21swapper/222:19:362
1314299321,8cyclictest17506-21ssh00:34:092
13142993212,5cyclictest22379-21aten_rbpower_cu19:35:242
1314399316,15cyclictest0-21swapper/319:15:213
1314399316,15cyclictest0-21swapper/319:15:213
1314299316,16cyclictest7950irq/56-dwc_otg_21:20:232
1314299316,16cyclictest7950irq/56-dwc_otg_21:20:232
1314299314,13cyclictest7950irq/56-dwc_otg_19:50:312
1314299312,7cyclictest8525-21ssh00:20:352
1314299312,7cyclictest8525-21ssh00:20:352
1314299311,23cyclictest13460-21cut19:10:262
1314399304,9cyclictest37-21rcuc/319:41:213
1314299309,11cyclictest7950irq/56-dwc_otg_21:05:312
1314299307,15cyclictest0-21swapper/219:25:222
1314299304,16cyclictest30184-21sensors00:05:352
13142993013,9cyclictest26338-21/usr/sbin/munin19:45:252
1314299301,12cyclictest32474-21grep20:00:372
1314399281,18cyclictest7750irq/56-dwc_otg20:40:253
1314299282,17cyclictest0-21swapper/220:31:382
1314399271,13cyclictest7950irq/56-dwc_otg_20:15:303
112522253,11sleep20-21swapper/219:05:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional