You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-01 - 21:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Sat Nov 01, 2025 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1811422820,6sleep10-21swapper/107:05:431
907921810,5sleep01983199cyclictest09:25:570
19831991056,74cyclictest0-21swapper/007:30:400
92742950,1sleep20-21swapper/211:00:402
56682930,2sleep20-21swapper/211:40:332
274122910,2sleep30-21swapper/309:00:443
324792820,2sleep10-21swapper/110:45:571
103252820,4sleep110318-21ssh11:45:521
1983199807,41cyclictest0-21swapper/008:50:410
19831998067,4cyclictest9-21ksoftirqd/011:18:540
19831997921,23cyclictest0-21swapper/009:50:380
19833997715,50cyclictest25674-21fschecks_count08:55:452
1983199774,37cyclictest0-21swapper/009:45:370
19831997534,26cyclictest0-21swapper/011:05:400
1983199746,50cyclictest0-21swapper/012:00:410
19833997313,4cyclictest8050irq/56-dwc_otg_10:25:372
1983199733,55cyclictest0-21swapper/011:55:390
320082710,1sleep10-21swapper/112:15:351
302422710,1sleep30-21swapper/312:12:113
19834997116,48cyclictest25357-21cat08:55:263
1983199701,51cyclictest7694-21fschecks_time08:05:460
1983199686,42cyclictest0-21swapper/011:20:390
1983199681,47cyclictest805-21grep12:15:560
19834996615,36cyclictest21668-21sh12:00:533
19833996410,9cyclictest8050irq/56-dwc_otg_09:00:492
1983199641,46cyclictest5367-21ssh11:40:250
1983499631,47cyclictest0-21swapper/312:26:203
1983499631,47cyclictest0-21swapper/308:15:263
1983399638,5cyclictest10882-21apt-get09:30:262
1983199637,40cyclictest0-21swapper/011:10:380
1983199632,43cyclictest0-21swapper/010:20:540
1983499624,48cyclictest24635-21unixbench_singl07:20:553
1983399626,9cyclictest8050irq/56-dwc_otg_11:15:582
19831996242,13cyclictest101ktimersoftd/010:30:560
1983199623,43cyclictest24807-21ssh12:05:460
1983499617,44cyclictest0-21swapper/308:55:453
1983499611,46cyclictest8050irq/56-dwc_otg_09:25:263
1983299615,5cyclictest0-21swapper/109:40:391
19831996139,6cyclictest2659-21latency_hist10:05:250
19831996128,18cyclictest0-21swapper/009:10:490
1983199612,41cyclictest0-21swapper/012:25:390
1983499602,46cyclictest0-21swapper/310:50:413
1983499601,17cyclictest7950irq/56-dwc_otg_07:40:413
1983199602,35cyclictest11271-21fschecks_time08:15:440
1983199601,8cyclictest2020-21ssh10:50:260
1983499591,12cyclictest0-21swapper/312:15:383
19831995943,6cyclictest7850irq/56-dwc_otg08:01:040
1983199592,41cyclictest0-21swapper/007:20:520
19831995913,28cyclictest9-21ksoftirqd/010:50:470
1983499589,41cyclictest9649-21ls11:45:423
1983499581,46cyclictest7950irq/56-dwc_otg_10:22:493
1983499581,41cyclictest0-21swapper/309:05:513
1983399588,6cyclictest7950irq/56-dwc_otg_10:00:392
1983399585,6cyclictest500-21rs:main2
1983199588,4cyclictest101ktimersoftd/008:55:410
1983199583,4cyclictest101ktimersoftd/010:25:380
1983199582,40cyclictest0-21swapper/011:25:500
1983199582,39cyclictest0-21swapper/009:40:420
1983199581,41cyclictest0-21swapper/010:37:000
194132583,14sleep30-21swapper/307:08:293
1983499577,4cyclictest7950irq/56-dwc_otg_10:10:253
1983499572,46cyclictest28112-21rm09:55:073
1983499571,43cyclictest0-21swapper/310:48:143
1983499571,10cyclictest0-21swapper/312:38:023
1983399574,9cyclictest7950irq/56-dwc_otg_12:28:412
1983399573,4cyclictest8050irq/56-dwc_otg_10:14:312
1983399571,4cyclictest13060-21grep08:20:442
1983399571,4cyclictest0-21swapper/210:54:522
1983299572,8cyclictest0-21swapper/109:55:391
19831995733,7cyclictest7950irq/56-dwc_otg_12:10:520
19831995732,10cyclictest101ktimersoftd/008:25:550
1983199571,36cyclictest7508-21sed10:10:560
1983499563,41cyclictest7850irq/56-dwc_otg11:38:453
1983499561,9cyclictest7950irq/56-dwc_otg_10:30:443
1983499561,5cyclictest18964-21/usr/sbin/munin09:40:533
1983499561,44cyclictest15075-21taskset11:07:493
1983499561,43cyclictest0-21swapper/310:39:503
1983499561,12cyclictest25679-21ssh11:21:553
1983399566,4cyclictest7950irq/56-dwc_otg_11:59:092
1983399566,4cyclictest7950irq/56-dwc_otg_10:39:312
1983399563,4cyclictest7850irq/56-dwc_otg10:20:262
1983399561,7cyclictest3578-21egrep12:20:422
1983399561,7cyclictest3578-21egrep12:20:422
1983399561,5cyclictest7950irq/56-dwc_otg_09:55:302
1983199561,32cyclictest30871-21apt10:45:250
1983499552,9cyclictest7950irq/56-dwc_otg_08:15:463
1983499551,10cyclictest0-21swapper/307:31:153
1983399553,5cyclictest25706-21if_eth007:25:452
1983399552,12cyclictest0-21swapper/209:16:052
1983399552,10cyclictest8050irq/56-dwc_otg_11:35:372
19833995518,34cyclictest8050irq/56-dwc_otg_10:45:402
1983399551,3cyclictest0-21swapper/209:47:592
19831995535,12cyclictest21428-21munin-run07:15:250
19831995533,5cyclictest8050irq/56-dwc_otg_09:05:250
1983199552,37cyclictest0-21swapper/008:35:540
1983199551,39cyclictest9-21ksoftirqd/009:35:460
1983499542,9cyclictest7850irq/56-dwc_otg11:13:083
1983499542,6cyclictest8050irq/56-dwc_otg_10:00:413
1983499541,45cyclictest0-21swapper/310:25:533
1983499541,44cyclictest0-21swapper/309:50:023
1983499541,42cyclictest0-21swapper/312:21:013
1983499541,42cyclictest0-21swapper/312:21:013
1983399542,12cyclictest0-21swapper/211:11:542
1983399541,6cyclictest7950irq/56-dwc_otg_12:15:262
1983399541,3cyclictest0-21swapper/207:59:232
19831995441,4cyclictest0-21swapper/009:15:400
1983199541,31cyclictest20325-21fschecks_time08:40:420
19831995412,5cyclictest101ktimersoftd/009:05:540
1983499531,9cyclictest0-21swapper/308:44:393
1983499531,44cyclictest12236-21sh11:04:423
1983499531,44cyclictest0-21swapper/312:06:343
1983499531,43cyclictest0-21swapper/312:32:003
1983499531,43cyclictest0-21swapper/311:44:003
1983499531,43cyclictest0-21swapper/309:19:573
1983499531,41cyclictest0-21swapper/310:56:183
1983499531,41cyclictest0-21swapper/310:56:183
1983499531,40cyclictest0-21swapper/309:38:433
1983499531,39cyclictest0-21swapper/310:20:223
1983499531,10cyclictest7950irq/56-dwc_otg_09:30:313
1983499531,10cyclictest28500-21runrttasks09:55:283
1983499531,10cyclictest2059-21wget07:50:403
1983399536,8cyclictest8050irq/56-dwc_otg_07:20:552
1983399531,11cyclictest0-21swapper/210:23:102
1983199539,29cyclictest0-21swapper/009:55:540
1983199532,32cyclictest0-21swapper/011:30:530
1983199531,37cyclictest0-21swapper/012:35:440
1983199531,33cyclictest10675-21/usr/sbin/munin10:15:550
1983499526,37cyclictest0-21swapper/308:29:023
1983499522,8cyclictest22911-21grep08:45:553
1983499521,43cyclictest9294-21ssh10:14:513
1983499521,3cyclictest0-21swapper/307:25:433
1983499521,38cyclictest0-21swapper/311:55:383
1983499521,11cyclictest15301-21ssh11:52:293
1983399527,8cyclictest0-21swapper/212:36:012
1983399523,4cyclictest0-21swapper/211:08:072
1983399521,9cyclictest8050irq/56-dwc_otg_12:34:542
1983399521,9cyclictest7950irq/56-dwc_otg_09:44:352
1983399521,6cyclictest8050irq/56-dwc_otg_12:18:162
1983399521,6cyclictest31798-21memory11:30:482
1983399521,5cyclictest7950irq/56-dwc_otg_09:25:252
1983399521,3cyclictest0-21swapper/208:55:252
1983399521,19cyclictest0-21swapper/208:42:012
1983199525,31cyclictest6128-21latency_hist10:10:260
19831995231,13cyclictest4517-21grep07:55:510
1983499516,3cyclictest0-21swapper/308:02:403
1983499513,8cyclictest8050irq/56-dwc_otg_09:30:243
1983499511,9cyclictest178692chrt08:34:193
1983499511,11cyclictest21177-21/usr/sbin/munin11:15:553
1983399518,5cyclictest28647-21apt-get07:35:262
1983399517,4cyclictest0-21swapper/209:39:312
1983399511,4cyclictest27569-21apt-get11:25:412
1983399511,4cyclictest0-21swapper/212:00:552
1983399511,10cyclictest0-21swapper/210:55:372
1983399511,10cyclictest0-21swapper/210:55:372
1983299516,22cyclictest14758-21ssh09:35:381
1983199516,14cyclictest0-21swapper/009:30:310
1983199512,27cyclictest0-21swapper/007:45:390
19831995118,23cyclictest0-21swapper/009:23:420
1983199511,5cyclictest9801-21grep11:00:460
1983499502,6cyclictest31199-21fschecks_count09:10:433
1983399508,8cyclictest12241-21sed09:30:532
1983399506,7cyclictest8050irq/56-dwc_otg_07:49:152
1983399505,4cyclictest7950irq/56-dwc_otg_10:42:182
1983399501,4cyclictest0-21swapper/209:51:232
1983399501,4cyclictest0-21swapper/208:15:462
1983199502,7cyclictest0-21swapper/011:40:570
19831995018,5cyclictest0-21swapper/007:50:370
1983499491,7cyclictest7439-21aten_rbpower_en08:05:433
1983499491,7cyclictest0-21swapper/311:27:313
1983499491,7cyclictest0-21swapper/310:42:173
1983499491,7cyclictest0-21swapper/308:00:033
1983499491,6cyclictest21458-21latency_hist07:15:253
1983499491,5cyclictest140-21systemd-journal07:18:123
1983499491,39cyclictest0-21swapper/311:30:273
1983499491,12cyclictest8050irq/56-dwc_otg_08:20:463
1983399493,4cyclictest7950irq/56-dwc_otg_09:10:552
1983399492,3cyclictest8050irq/56-dwc_otg_10:07:292
1983399492,3cyclictest0-21swapper/212:05:552
1983399491,4cyclictest0-21swapper/208:25:402
1983199499,26cyclictest9-21ksoftirqd/012:20:470
1983199499,26cyclictest9-21ksoftirqd/012:20:470
1983199492,35cyclictest11337-21ssh11:47:120
1983199492,24cyclictest11056-21apt-get12:30:390
186242494,33sleep00-21swapper/007:05:510
1983399484,10cyclictest8050irq/56-dwc_otg_11:50:242
1983399481,4cyclictest10852-21ls08:15:262
1983299481,11cyclictest7850irq/56-dwc_otg10:50:411
19831994824,4cyclictest13955-21grep08:20:550
19831994811,4cyclictest9-21ksoftirqd/007:35:520
19831994811,4cyclictest9-21ksoftirqd/007:35:520
1983499471,7cyclictest1192-21timerwakeupswit07:45:593
1983399474,12cyclictest15512-21ssh11:52:552
1983399472,4cyclictest7950irq/56-dwc_otg_07:38:162
1983399472,4cyclictest7950irq/56-dwc_otg_07:38:162
19832994725,6cyclictest28672-21latency_hist07:35:261
19831994740,5cyclictest101ktimersoftd/007:25:380
1983199472,31cyclictest0-21swapper/010:59:260
1983199472,31cyclictest0-21swapper/010:59:260
1983499462,6cyclictest0-21swapper/308:35:423
1983399461,9cyclictest0-21swapper/211:21:352
1983299462,11cyclictest7950irq/56-dwc_otg_10:00:371
19831994616,16cyclictest13-21rcuc/008:47:250
1983499451,6cyclictest29482-21cat07:35:453
1983499451,6cyclictest29482-21cat07:35:453
1983299455,21cyclictest7950irq/56-dwc_otg_11:20:391
1983199452,23cyclictest0-21swapper/011:52:500
1983199451,22cyclictest31892-21sed07:40:550
1983199451,17cyclictest7850irq/56-dwc_otg08:30:400
1983299447,24cyclictest7850irq/56-dwc_otg12:35:381
1983299442,28cyclictest0-21swapper/111:25:471
1983199441,23cyclictest0-21swapper/008:10:400
1983299434,9cyclictest0-21swapper/112:20:391
1983299434,9cyclictest0-21swapper/112:20:391
1983199436,24cyclictest0-21swapper/007:15:390
277942420,5sleep11983299cyclictest09:00:501
19832994222,5cyclictest7950irq/56-dwc_otg_09:25:401
102042420,2sleep10-21swapper/112:29:241
19832994130,4cyclictest8050irq/56-dwc_otg_09:45:461
1983299406,6cyclictest12933-21ssh11:05:401
19832994020,5cyclictest2464850wget12:05:431
19832994019,5cyclictest0-21swapper/110:35:591
19832994010,23cyclictest0-21swapper/109:10:401
1983299393,10cyclictest7950irq/56-dwc_otg_11:40:391
1983299383,20cyclictest20193-21fschecks_time07:10:461
19832993813,8cyclictest18329-21cut08:35:421
1983299381,27cyclictest28270-21latency_hist09:55:261
1983299381,26cyclictest1082-21ls07:45:591
19832993810,7cyclictest0-21swapper/110:55:451
19832993810,7cyclictest0-21swapper/110:55:451
1983299378,4cyclictest7950irq/56-dwc_otg_10:15:451
1983299378,14cyclictest8050irq/56-dwc_otg_11:05:251
19833993616,5cyclictest30604-21latency_hist09:10:262
1983299367,23cyclictest0-21swapper/107:50:401
1983299366,21cyclictest7850irq/56-dwc_otg12:10:421
1983299363,20cyclictest0-21swapper/108:55:481
1983299362,8cyclictest13560-21basename10:20:401
1983299362,5cyclictest14469-21grep11:50:571
1983299361,21cyclictest25175-21latency_hist08:55:261
1983299357,14cyclictest7950irq/56-dwc_otg_09:05:471
1983299351,8cyclictest23721-21fschecks_count07:20:441
19832993517,4cyclictest31222-21cut11:30:411
19832993515,18cyclictest0-21swapper/111:10:391
19832993514,6cyclictest22414-21proc_pri07:15:501
19832993512,6cyclictest11396-21aten_rbpower_po09:30:401
1983299351,18cyclictest0-21swapper/110:40:451
1983399349,5cyclictest7950irq/56-dwc_otg_08:35:382
1983399341,17cyclictest30787-21/usr/sbin/munin07:40:492
1983299347,8cyclictest0-21swapper/110:30:531
1983299343,6cyclictest0-21swapper/108:20:401
1983299342,7cyclictest7950irq/56-dwc_otg_10:25:441
1983299341,19cyclictest0-21swapper/109:15:431
19832993411,4cyclictest9556-21latency_hist10:15:251
19832993410,5cyclictest11324-21fschecks_count12:30:451
1983399337,11cyclictest8050irq/56-dwc_otg_08:30:492
1983299338,3cyclictest0-21swapper/109:20:261
1983299336,6cyclictest17516-21grep08:30:541
1983299336,14cyclictest0-21swapper/108:40:391
1983299334,8cyclictest7950irq/56-dwc_otg_07:25:551
1983299334,5cyclictest7950irq/56-dwc_otg_08:45:551
19832993311,7cyclictest3354-21ssh10:05:441
190172336,10sleep20-21swapper/207:05:572
1983399325,12cyclictest8050irq/56-dwc_otg_07:15:452
1983399321,25cyclictest20105-21uniq07:10:462
1983299327,14cyclictest0-21swapper/108:05:421
19832993214,5cyclictest6043-21uname08:00:511
19832993210,3cyclictest0-21swapper/111:35:531
211382310,6sleep11983299cyclictest12:00:461
1983399314,12cyclictest21810-21/usr/sbin/munin08:45:452
19833993110,5cyclictest2633-21ntp_kernel_err07:50:482
1983299315,5cyclictest7950irq/56-dwc_otg_07:55:411
1983299313,3cyclictest7850irq/56-dwc_otg08:10:391
1983299312,4cyclictest7850irq/56-dwc_otg08:25:391
1983299311,19cyclictest21135-21switchtime11:15:551
19832993013,5cyclictest18789-21sh11:57:071
19833992911,10cyclictest6369-21processes08:01:052
19833992811,5cyclictest7369-21/usr/sbin/munin08:05:572
1983299286,13cyclictest8050irq/56-dwc_otg_07:40:461
1983299285,6cyclictest29272-21fschecks_time07:35:431
1983299285,6cyclictest29272-21fschecks_time07:35:431
1983299262,9cyclictest11311-21fschecks_time08:15:441
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional