You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 08:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Wed Nov 19, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
756721760,1sleep10-21swapper/123:57:321
2795799918,40cyclictest0-21swapper/000:15:180
27957998919,53cyclictest24702-21latency00:20:270
27957998522,37cyclictest5454-21aten_rbpower_vo23:55:220
27957998519,51cyclictest30132-21grep19:15:240
27957998517,54cyclictest3776-21wc21:00:300
27957998515,41cyclictest23272-21latency_hist22:55:050
215952840,2sleep30-21swapper/323:34:273
27957998318,56cyclictest7959-21latency_hist22:35:050
27957998315,50cyclictest22080-21latency_hist23:35:050
27957998119,45cyclictest16125-21fschecks_count22:45:260
27957998118,49cyclictest25613-21unixbench_multi20:30:330
27957998113,53cyclictest5472-21chrt21:50:060
27957998019,53cyclictest14858-21ssh00:06:300
27957998010,56cyclictest0-21swapper/023:23:590
27957997922,11cyclictest18522-21ssh21:25:220
27957997920,42cyclictest31744-21tr20:50:180
27957997916,36cyclictest5407-21latency_hist21:50:050
27957997829,8cyclictest7950irq/56-dwc_otg_23:25:180
27957997817,52cyclictest12848-21unixbench-2d19:55:340
2795799773,36cyclictest0-21swapper/023:05:190
27957997721,30cyclictest22830-21ssh23:35:280
27957997716,38cyclictest2524850ssh22:15:220
27957997714,47cyclictest9370-21grep22:35:360
27957997627,8cyclictest0-21swapper/022:10:190
27957997613,38cyclictest22697-21cut20:25:050
2795799753,57cyclictest17753-21proc_pri00:10:370
2795799753,57cyclictest17753-21proc_pri00:10:370
27957997521,42cyclictest27699-21ssh23:41:110
27957997521,42cyclictest27699-21ssh23:41:110
27957997517,49cyclictest27377-21/usr/sbin/munin23:00:300
27957997515,34cyclictest5411-21ssh00:39:310
27957997514,34cyclictest3945-21ssh23:53:010
318592740,4sleep131864-21aten_rbpower_vo20:50:211
27957997417,43cyclictest9926-21latency_hist19:50:050
27957997416,12cyclictest10677-21grep21:55:340
27957997318,37cyclictest917-21rm23:49:440
27957997312,37cyclictest0-21swapper/023:20:040
27957997215,30cyclictest12281-21ssh00:04:060
27957997213,48cyclictest16895-21idleruntime-cro22:05:040
27957997119,27cyclictest30406-21ssh22:20:560
27957997118,31cyclictest20794-21apt22:10:040
27957997116,33cyclictest20687-21munin-plugin-st20:20:040
27957997115,33cyclictest28580-21if_wlan020:40:280
27957997114,38cyclictest7933-21rm21:10:340
27957997021,37cyclictest29263-21ssh21:39:110
27957997021,33cyclictest3506-21ssh22:29:200
27957997020,25cyclictest23996-21sh22:55:250
27957997020,25cyclictest23996-21sh22:55:250
27957997018,33cyclictest10640-21nfs4_client19:50:280
27957997018,25cyclictest18570-21runrttasks20:11:400
27957997016,37cyclictest29116-21unixbench_singl19:10:320
27957997015,41cyclictest13346-21rm21:18:230
2795799701,40cyclictest0-21swapper/020:00:200
27957996919,30cyclictest32126-21rm21:41:540
27957996813,38cyclictest29090-21rm00:25:560
27957996812,46cyclictest0-21swapper/019:20:490
2795799672,51cyclictest0-21swapper/023:10:310
27957996714,38cyclictest389-21sh00:31:270
27957996714,35cyclictest14743-21ssh22:43:360
27957996712,39cyclictest0-21swapper/020:55:250
27958996617,19cyclictest0-21swapper/119:20:201
27957996615,12cyclictest25102-21ssh21:33:400
27957996610,42cyclictest4888-21wget21:05:210
64842650,2sleep30-21swapper/322:32:323
91572640,2sleep20-21swapper/219:45:342
153782640,1sleep10-21swapper/120:05:051
27957996317,10cyclictest0-21swapper/020:45:190
27957996313,36cyclictest23330-21missed_timers20:25:260
27957996214,37cyclictest8111-21latency_hist19:45:050
27957996210,41cyclictest0-21swapper/021:20:330
27957996210,41cyclictest0-21swapper/021:20:330
27957996110,36cyclictest0-21swapper/020:06:150
2795799609,37cyclictest0-21swapper/019:31:230
2795799609,37cyclictest0-21swapper/019:31:230
27957996012,37cyclictest0-21swapper/019:39:250
27957995916,31cyclictest2413-21taskset19:28:420
274322572,48sleep00-21swapper/019:06:510
242212570,2sleep10-21swapper/123:36:591
27957995615,33cyclictest27553-21runrttasks20:35:590
27958995415,23cyclictest0-21swapper/120:45:201
2796099535,3cyclictest0-21swapper/322:45:223
2795899537,29cyclictest26131-21ssh23:40:211
2795899537,29cyclictest26131-21ssh23:40:211
225492530,1sleep30-21swapper/321:30:253
2795999515,15cyclictest10488-21ssh23:20:182
2795899515,4cyclictest16209-21grep22:45:261
27958995115,24cyclictest0-21swapper/121:00:111
2795999506,30cyclictest7950irq/56-dwc_otg_00:25:202
2795899509,22cyclictest23484-21processes20:25:291
27958995025,8cyclictest7950irq/56-dwc_otg_22:55:301
27958995025,8cyclictest7950irq/56-dwc_otg_22:55:301
121062500,1sleep20-21swapper/221:15:582
2796099493,21cyclictest0-21swapper/321:00:183
2795899499,21cyclictest5140-21apt-get21:50:101
27958994913,20cyclictest20985-21latency_hist22:10:051
2796099487,23cyclictest0-21swapper/323:40:213
2796099487,23cyclictest0-21swapper/323:40:213
2795899487,27cyclictest0-21swapper/123:05:191
291772470,2sleep20-21swapper/223:44:132
291772470,2sleep20-21swapper/223:44:132
2796099478,25cyclictest0-21swapper/321:50:093
2796099477,25cyclictest0-21swapper/319:40:183
27958994719,6cyclictest0-21swapper/120:40:371
2796099467,25cyclictest0-21swapper/320:45:203
2796099467,25cyclictest0-21swapper/319:15:203
2795899461,27cyclictest0-21swapper/100:10:211
2795899461,27cyclictest0-21swapper/100:10:211
2796099451,21cyclictest7950irq/56-dwc_otg_22:55:243
2796099451,21cyclictest7950irq/56-dwc_otg_22:55:243
2795999456,24cyclictest0-21swapper/220:20:192
2795899459,6cyclictest7850irq/56-dwc_otg_22:10:191
2795899457,24cyclictest0-21swapper/120:15:221
2796099448,11cyclictest39-21ksoftirqd/323:35:303
2796099447,28cyclictest24172-21sh00:20:193
2796099447,24cyclictest0-21swapper/321:55:193
27960994411,8cyclictest39-21ksoftirqd/322:55:053
2796099441,15cyclictest0-21swapper/319:50:303
2795899447,25cyclictest0-21swapper/121:30:211
2795899441,27cyclictest0-21swapper/100:05:181
2796099437,25cyclictest0-21swapper/319:25:203
2796099437,23cyclictest0-21swapper/322:15:203
2796099437,23cyclictest0-21swapper/320:00:093
27960994322,14cyclictest39-21ksoftirqd/321:28:213
2795999437,25cyclictest0-21swapper/220:10:222
2795899437,24cyclictest0-21swapper/123:45:191
2795899437,24cyclictest0-21swapper/123:30:101
2795899437,24cyclictest0-21swapper/119:15:201
2795899436,24cyclictest0-21swapper/119:40:181
27958994324,13cyclictest7950irq/56-dwc_otg_22:15:201
2795899431,5cyclictest7950irq/56-dwc_otg_00:15:201
2796099426,23cyclictest0-21swapper/321:10:193
2795999427,23cyclictest0-21swapper/222:20:202
2795999427,23cyclictest0-21swapper/221:25:212
2795899427,20cyclictest1441-21fschecks_time20:55:231
2795899426,24cyclictest0-21swapper/123:20:201
2795899426,22cyclictest0-21swapper/120:05:191
2795899426,22cyclictest0-21swapper/119:25:201
27958994225,8cyclictest7850irq/56-dwc_otg_21:41:581
27958994218,14cyclictest304-21/usr/sbin/munin22:25:271
27958994211,15cyclictest7850irq/56-dwc_otg_22:35:271
2796099417,25cyclictest6018-21ssh23:14:413
27960994121,4cyclictest37-21rcuc/320:57:493
2795899413,24cyclictest16394-21ssh22:04:041
2795899412,15cyclictest30295-21taskset00:28:451
2795899411,29cyclictest5385-21ssh21:50:051
2796099406,25cyclictest0-21swapper/323:50:183
2796099406,21cyclictest27956-21cyclictest23:45:103
2796099406,20cyclictest39-21ksoftirqd/320:25:053
27960994021,5cyclictest39-21ksoftirqd/322:21:133
2796099401,28cyclictest26747-21if_eth020:35:263
2795899408,16cyclictest28060-21ssh21:36:471
2795899404,22cyclictest0-21swapper/119:45:211
27958994023,9cyclictest7750irq/56-dwc_otg23:53:001
27958994017,10cyclictest7750irq/56-dwc_otg00:20:521
27958994016,9cyclictest221ktimersoftd/121:25:391
148432400,1sleep30-21swapper/322:43:473
306862390,2sleep3381ktimersoftd/300:29:433
2796099396,23cyclictest0-21swapper/322:35:203
27960993915,6cyclictest39-21ksoftirqd/321:50:043
27960993913,15cyclictest7850irq/56-dwc_otg_23:55:243
2795999399,14cyclictest7850irq/56-dwc_otg_21:40:352
2795999392,8cyclictest1539350ssh22:44:542
27959993911,9cyclictest22754-21ssh22:54:152
2795899399,7cyclictest0-21swapper/119:55:171
2795899396,21cyclictest7680-21ssh23:15:351
27958993923,8cyclictest12190-21ssh21:58:261
27958993917,9cyclictest7750irq/56-dwc_otg21:14:221
27958993915,13cyclictest221ktimersoftd/100:34:261
27958993910,19cyclictest7750irq/56-dwc_otg22:21:191
253562390,2sleep30-21swapper/320:30:313
212282390,2sleep121227-21cut20:20:231
27960993815,4cyclictest32088-21/usr/sbin/munin00:30:333
27960993811,9cyclictest29928-21latency_hist21:40:053
2796099381,16cyclictest7750irq/56-dwc_otg23:25:183
27960993811,12cyclictest39-21ksoftirqd/322:10:043
2795999389,14cyclictest7950irq/56-dwc_otg_23:15:042
2795999383,29cyclictest5585-21ssh21:50:182
2795999381,31cyclictest16902-21ls22:45:362
27959993812,12cyclictest0-21swapper/223:15:312
2795899387,5cyclictest7750irq/56-dwc_otg23:10:331
27958993820,8cyclictest19765-21aten_rbpower_po22:50:201
2795899381,21cyclictest19043-21ls20:15:051
2796099378,16cyclictest37-21rcuc/320:14:233
2796099375,13cyclictest20517-21ssh00:15:183
2796099372,9cyclictest7750irq/56-dwc_otg23:20:043
27960993727,7cyclictest381ktimersoftd/320:25:173
2795999372,7cyclictest7850irq/56-dwc_otg_22:30:182
27959993723,4cyclictest7950irq/56-dwc_otg_19:30:382
27959993723,4cyclictest7950irq/56-dwc_otg_19:30:382
2795999371,26cyclictest23229-21ls23:35:342
2795999371,23cyclictest13498-21ls00:05:242
2795899378,13cyclictest4863-21aten_rbpower_cu21:05:201
2795899372,10cyclictest0-21swapper/123:00:181
27958993718,8cyclictest7750irq/56-dwc_otg20:30:291
27958993713,16cyclictest221ktimersoftd/121:22:351
27958993713,16cyclictest221ktimersoftd/121:22:351
27958993712,5cyclictest221ktimersoftd/123:30:051
2796099364,18cyclictest0-21swapper/322:04:213
27960993619,8cyclictest37-21rcuc/321:17:223
2795999369,6cyclictest0-21swapper/222:20:042
2795999368,8cyclictest31205-21sed23:45:342
2795999368,19cyclictest0-21swapper/223:29:312
2795999367,25cyclictest0-21swapper/221:35:192
2795999366,21cyclictest18400-21ssh23:30:192
2795999366,10cyclictest7850irq/56-dwc_otg_00:20:172
27959993618,11cyclictest1120-21runrttasks23:08:152
2795999361,24cyclictest4576-21latency_hist21:05:042
2795999361,22cyclictest10308-21uniq19:50:242
2795899365,19cyclictest13526-21ssh22:41:071
27958993610,17cyclictest11031-21ls19:50:341
27958993610,13cyclictest7750irq/56-dwc_otg21:18:201
19412360,3sleep20-21swapper/221:45:252
2796099354,21cyclictest7750irq/56-dwc_otg20:44:253
27960993514,4cyclictest37-21rcuc/319:50:043
2795999357,14cyclictest7850irq/56-dwc_otg_23:00:332
2795999351,20cyclictest4777-21ls19:35:212
27959993511,11cyclictest0-21swapper/200:30:362
2795899358,18cyclictest221ktimersoftd/122:32:181
2795899352,24cyclictest0-21swapper/119:10:161
231432350,3sleep231-21ksoftirqd/200:18:262
2796099346,24cyclictest0-21swapper/323:20:203
27960993419,6cyclictest381ktimersoftd/320:09:163
27960993413,15cyclictest381ktimersoftd/319:55:353
27960993412,5cyclictest32405-21unixbench-2d23:05:343
2796099341,24cyclictest24752-21ssh22:15:053
2796099341,18cyclictest37-21rcuc/319:10:303
27959993416,10cyclictest7850irq/56-dwc_otg_22:05:272
27959993414,5cyclictest2982450wget19:15:212
27959993410,21cyclictest1788-21fschecks_time23:50:222
2795899344,5cyclictest0-21swapper/119:34:081
2795899344,5cyclictest0-21swapper/119:34:081
27958993413,12cyclictest0-21swapper/119:35:251
27958993412,17cyclictest9268-21ssh00:00:191
2796099337,18cyclictest7850irq/56-dwc_otg_00:00:233
2796099337,15cyclictest0-21swapper/319:35:203
2796099334,21cyclictest32621-21ls19:20:343
2796099333,17cyclictest11-21rcu_preempt21:23:003
2796099333,17cyclictest11-21rcu_preempt21:23:003
2796099332,16cyclictest381ktimersoftd/323:00:513
2796099331,5cyclictest30997-21/usr/sbin/munin21:40:313
2796099331,20cyclictest13660-21ssh00:05:263
2795999335,5cyclictest7950irq/56-dwc_otg_20:05:342
2795999332,21cyclictest11089-21ssh00:01:392
2795999331,23cyclictest32628-21ls20:50:332
2795999331,23cyclictest26088-21latency_hist20:35:052
2795999331,15cyclictest7750irq/56-dwc_otg22:35:242
27959993311,5cyclictest2999-21missed_timers00:35:292
2795899331,10cyclictest0-21swapper/120:35:191
2796099321,4cyclictest380-21aten_rbpower_en22:25:193
27960993212,9cyclictest0-21swapper/319:34:023
27960993212,9cyclictest0-21swapper/319:34:023
2796099321,21cyclictest3403-21timerandwakeup00:35:353
27960993211,6cyclictest7750irq/56-dwc_otg20:55:053
2795999329,6cyclictest7750irq/56-dwc_otg00:00:052
2795999328,12cyclictest0-21swapper/222:01:222
2795999324,13cyclictest7850irq/56-dwc_otg_21:35:042
2795999323,5cyclictest27063-21basename23:00:052
2795999323,5cyclictest27063-21basename23:00:052
2795999323,16cyclictest21763-21ls22:10:282
27959993223,7cyclictest301ktimersoftd/219:10:252
2795999322,15cyclictest7850irq/56-dwc_otg_21:55:242
2795999322,15cyclictest7750irq/56-dwc_otg19:20:342
27959993216,5cyclictest13356-21latency_hist20:00:042
27958993210,5cyclictest221ktimersoftd/100:36:541
265902326,12sleep20-21swapper/219:05:272
2796099313,13cyclictest37-21rcuc/320:17:203
2795999318,3cyclictest0-21swapper/222:26:282
2795999317,7cyclictest7850irq/56-dwc_otg_20:55:232
2795999317,12cyclictest7950irq/56-dwc_otg_20:40:382
2795999314,5cyclictest986-21aten_rbpower_cu19:25:202
2795999311,7cyclictest451-21rs:main2
2795999311,21cyclictest0-21swapper/220:35:212
27959993110,5cyclictest7950irq/56-dwc_otg_20:25:272
2796099303,20cyclictest7850irq/56-dwc_otg_00:10:263
2796099303,20cyclictest7850irq/56-dwc_otg_00:10:263
2796099301,18cyclictest0-21swapper/321:05:253
2795999302,4cyclictest7950irq/56-dwc_otg_20:45:212
2795999301,5cyclictest1749350ssh00:10:332
2795999301,5cyclictest1749350ssh00:10:332
27959993010,12cyclictest0-21swapper/221:22:532
27959993010,12cyclictest0-21swapper/221:22:532
2795999293,5cyclictest7950irq/56-dwc_otg_21:10:342
2795999292,9cyclictest7941-21idleruntime-cro19:45:042
2795999291,16cyclictest0-21swapper/221:06:422
2795999291,15cyclictest0-21swapper/220:05:052
274082284,14sleep10-21swapper/119:06:361
273382273,15sleep30-21swapper/319:05:413
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional