You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-24 - 19:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Fri Oct 24, 2025 12:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3177025400,7sleep00-21swapper/007:10:230
138812970,2sleep30-21swapper/307:50:353
305822890,2sleep30-21swapper/307:05:443
174602850,2sleep30-21swapper/311:38:243
110572840,2sleep20-21swapper/207:40:442
116922810,6sleep03183599cyclictest09:10:510
31835997551,3cyclictest0-21swapper/009:19:010
3183699742,66cyclictest318382chrt07:10:241
31836997210,48cyclictest22138-21sh11:00:151
242602720,1sleep20-21swapper/209:30:212
3183599713,40cyclictest0-21swapper/007:10:240
286612710,1sleep10-21swapper/108:30:381
3183599701,19cyclictest8778-21grep09:05:380
3183699693,49cyclictest7850irq/56-dwc_otg10:35:211
3183699692,27cyclictest0-21swapper/110:25:321
3183699692,27cyclictest0-21swapper/110:25:321
31835996928,4cyclictest0-21swapper/010:55:350
3183699687,44cyclictest0-21swapper/111:05:221
3183699673,20cyclictest0-21swapper/108:05:331
3183599671,50cyclictest0-21swapper/011:20:460
31835996713,32cyclictest9-21ksoftirqd/009:55:400
3183699665,46cyclictest24433-1kworker/u9:211:06:211
49502640,2sleep00-21swapper/009:48:230
3183699643,5cyclictest7950irq/56-dwc_otg_09:25:221
3183699637,4cyclictest0-21swapper/111:29:151
3183699635,43cyclictest8050irq/56-dwc_otg_12:35:211
3183699633,56cyclictest30783-21tr11:55:471
3183699622,10cyclictest13221-21ssh09:14:141
3183599621,30cyclictest0-21swapper/008:40:350
3183699611,45cyclictest10902-21ssh10:43:241
3183699611,45cyclictest10902-21ssh10:43:241
31835996113,25cyclictest11576-21idleruntime-cro10:45:210
31835996113,25cyclictest11576-21idleruntime-cro10:45:210
3183699605,5cyclictest7950irq/56-dwc_otg_08:00:431
3183699605,5cyclictest7950irq/56-dwc_otg_08:00:431
3183599605,40cyclictest0-21swapper/010:35:370
3183599602,6cyclictest17809-21grep09:20:390
3183599602,26cyclictest0-21swapper/007:15:310
3183799598,11cyclictest7850irq/56-dwc_otg12:25:342
3183699595,40cyclictest7850irq/56-dwc_otg11:50:431
3183699592,11cyclictest7850irq/56-dwc_otg09:00:331
3183599596,4cyclictest9-21ksoftirqd/011:30:500
3183599591,33cyclictest12344-21fschecks_time07:45:380
3183699582,5cyclictest5207-21apt-get12:05:331
3183699581,5cyclictest12680-21open_inodes10:45:461
3183699581,4cyclictest7850irq/56-dwc_otg07:55:441
31835995824,18cyclictest0-21swapper/011:10:400
31835995813,6cyclictest0-21swapper/008:30:330
3183699572,6cyclictest24682-21/usr/sbin/munin10:15:461
3183699572,4cyclictest8050irq/56-dwc_otg_12:35:441
3183599574,10cyclictest0-21swapper/010:00:360
3183599572,33cyclictest140-21systemd-journal11:30:210
3183799563,38cyclictest0-21swapper/207:10:242
3183699565,3cyclictest0-21swapper/111:45:281
3183699564,4cyclictest0-21swapper/110:12:591
3183699563,41cyclictest7850irq/56-dwc_otg11:39:151
3183699562,6cyclictest5920-21sed09:50:221
3183699561,4cyclictest0-21swapper/109:41:471
3183699561,43cyclictest0-21swapper/109:58:411
31836995610,9cyclictest7950irq/56-dwc_otg_10:20:321
31835995627,16cyclictest7850irq/56-dwc_otg07:40:210
31835995610,22cyclictest31908-21/usr/sbin/munin09:40:400
3183699555,4cyclictest3698-1kworker/u9:112:11:041
3183699553,44cyclictest0-21swapper/111:31:081
3183699553,43cyclictest21605-21missed_timers09:25:421
3183699552,7cyclictest24530-21latency_hist08:20:211
3183699552,4cyclictest0-21swapper/110:02:321
3183699552,41cyclictest0-21swapper/108:44:061
3183699551,9cyclictest20600-21chrt11:42:411
3183699551,7cyclictest24960-21ssh12:29:241
31836995516,36cyclictest26281-21sh09:32:051
3183699551,45cyclictest0-21swapper/110:07:031
31835995530,5cyclictest4797-21latency_hist08:55:210
31835995527,15cyclictest9-21ksoftirqd/011:05:370
3183599552,30cyclictest9-21ksoftirqd/012:15:350
3183599551,32cyclictest28290-21fschecks_time09:35:410
3183599551,28cyclictest21923-21ssh11:45:210
3183599551,22cyclictest30122-21cpuspeed12:35:370
31835995510,41cyclictest9-21ksoftirqd/010:20:460
3183699542,10cyclictest8050irq/56-dwc_otg_09:17:021
3183699541,5cyclictest30692-21sed11:10:501
3183599548,30cyclictest9-21ksoftirqd/009:30:390
3183599548,25cyclictest0-21swapper/009:25:380
31835995429,6cyclictest8050irq/56-dwc_otg_08:05:490
31835995420,4cyclictest21697-21/usr/sbin/munin12:25:390
3183699537,38cyclictest8050irq/56-dwc_otg_12:02:161
3183699537,38cyclictest8050irq/56-dwc_otg_12:02:161
3183699536,6cyclictest0-21swapper/108:45:551
3183699536,6cyclictest0-21swapper/108:45:551
3183699534,8cyclictest8050irq/56-dwc_otg_11:17:061
3183699532,6cyclictest0-21swapper/108:35:501
31836995314,36cyclictest8216-21latency_hist07:35:221
3183599534,9cyclictest17618-21ssh12:20:360
3183599533,30cyclictest0-21swapper/009:00:350
31835995324,6cyclictest3136-21timerwakeupswit10:30:510
3183599532,30cyclictest5300-21aten_rbpower_po12:05:340
3183699524,4cyclictest0-21swapper/110:50:531
3183699522,7cyclictest11173-21timerandwakeup07:40:461
3183699521,9cyclictest20744-21diskmemload12:24:131
3183699521,10cyclictest0-21swapper/110:37:471
31836995210,29cyclictest16191-1kworker/u9:012:18:511
31835995226,5cyclictest1177-21grep11:15:430
31835995222,9cyclictest13108-21sed10:45:500
3183599522,29cyclictest0-21swapper/009:50:410
3183599522,29cyclictest0-21swapper/009:50:410
31835995219,24cyclictest0-21swapper/012:32:530
31835995217,4cyclictest10063-21diskmemload11:45:430
3183599521,28cyclictest25465-21proc_pri08:20:510
3183599521,28cyclictest15935-21cut07:55:400
3183699511,8cyclictest7850irq/56-dwc_otg09:50:241
3183699511,8cyclictest7850irq/56-dwc_otg09:50:241
3183699511,3cyclictest7850irq/56-dwc_otg08:25:381
3183699511,2cyclictest0-21swapper/109:36:241
3183599518,19cyclictest0-21swapper/008:45:330
3183599518,19cyclictest0-21swapper/008:45:330
3183599515,8cyclictest140-21systemd-journal11:40:210
3183599514,26cyclictest5494-21sshd07:25:430
31835995125,12cyclictest1582-21tune2fs12:00:420
31835995125,12cyclictest1582-21tune2fs12:00:420
3183599512,34cyclictest0-21swapper/010:10:060
3183599511,30cyclictest3792-21/usr/sbin/munin07:20:530
3183599511,27cyclictest22550-21grep11:00:330
3183599511,23cyclictest2458350ssh10:15:450
3183699506,3cyclictest7850irq/56-dwc_otg11:20:371
3183699501,3cyclictest0-21swapper/108:14:421
31835995031,6cyclictest7950irq/56-dwc_otg_10:52:100
31835995026,5cyclictest27373-21switchtime08:25:470
3183599502,29cyclictest9440-21aten_rbpower_po12:10:350
3183699492,8cyclictest4653-21sh08:55:201
3183699491,9cyclictest0-21swapper/108:24:541
3183699491,7cyclictest6459-21modprobe09:00:211
3183699491,6cyclictest2369-21unixbench_singl07:15:471
3183699491,4cyclictest0-21swapper/107:50:041
3183599491,32cyclictest0-21swapper/010:13:300
3183599491,28cyclictest21140-21aten_rbpower_en08:10:370
31835994910,17cyclictest0-21swapper/010:25:350
31835994910,17cyclictest0-21swapper/010:25:350
3183699488,38cyclictest97592chrt07:37:211
3183699482,7cyclictest8784-21fschecks_time09:05:381
3183699481,9cyclictest0-21swapper/107:21:261
3183699481,6cyclictest14715-21switchtime07:50:471
3183599484,31cyclictest0-21swapper/008:16:310
3183599484,24cyclictest13-21rcuc/008:35:360
31835994720,20cyclictest0-21swapper/007:41:240
3183999467,25cyclictest0-21swapper/310:10:273
3183999453,33cyclictest8050irq/56-dwc_otg_12:25:333
3183699452,7cyclictest4874-21aten_rbpower_cu07:25:351
3183599456,18cyclictest9-21ksoftirqd/011:55:410
31835994521,5cyclictest26138-21/usr/sbin/munin11:50:350
3183999446,23cyclictest0-21swapper/311:05:333
31839994412,19cyclictest0-21swapper/307:10:243
3183799447,24cyclictest0-21swapper/209:00:362
31837994422,6cyclictest1111-21runrttasks07:15:332
31837994417,8cyclictest7850irq/56-dwc_otg10:00:362
243112440,13sleep33183999cyclictest11:47:223
3183999437,7cyclictest7850irq/56-dwc_otg08:00:213
3183999437,25cyclictest0-21swapper/311:55:383
3183999437,22cyclictest0-21swapper/312:00:373
3183999437,22cyclictest0-21swapper/312:00:373
31837994322,5cyclictest25870-21latency_hist11:50:212
3183799431,9cyclictest7950irq/56-dwc_otg_10:25:342
3183799431,9cyclictest7950irq/56-dwc_otg_10:25:342
3183999427,22cyclictest0-21swapper/309:25:353
3183799427,22cyclictest0-21swapper/209:10:272
3183799427,22cyclictest0-21swapper/208:45:352
3183799427,22cyclictest0-21swapper/208:45:352
31835994213,3cyclictest0-21swapper/008:55:490
3183599421,21cyclictest0-21swapper/008:00:480
3183599421,21cyclictest0-21swapper/008:00:480
31839994131,6cyclictest1111-21runrttasks07:30:323
31839994115,17cyclictest0-21swapper/312:35:333
31839994113,16cyclictest24586-21wget09:30:373
3183999411,13cyclictest7950irq/56-dwc_otg_09:45:353
3183999406,21cyclictest6136-21ssh09:50:363
3183999406,21cyclictest6136-21ssh09:50:363
3183799407,19cyclictest0-21swapper/208:40:372
31837994020,4cyclictest20528-21uniq10:10:402
3183799401,4cyclictest8050irq/56-dwc_otg_10:30:472
31835994018,13cyclictest0-21swapper/007:50:440
3183999395,25cyclictest9352-21ssh12:10:343
31839993917,4cyclictest2594050wget12:30:373
3183999391,26cyclictest11247-21ls07:40:463
31837993922,5cyclictest7950irq/56-dwc_otg_09:10:222
3183999381,26cyclictest3462-21ls08:50:563
3183799387,21cyclictest0-21swapper/212:20:372
3183799383,12cyclictest7850irq/56-dwc_otg11:00:322
3183799381,24cyclictest5588-21uniq12:05:382
3183799381,24cyclictest13732-21ls07:50:222
3183599386,5cyclictest7806-21grep07:30:480
3183999379,7cyclictest8050irq/56-dwc_otg_10:45:503
3183999377,20cyclictest0-21swapper/309:00:363
3183799372,15cyclictest7950irq/56-dwc_otg_11:35:382
31837993714,5cyclictest2640-21aten_rbpower_en09:45:362
3183999367,26cyclictest0-21swapper/307:15:323
3183999366,25cyclictest0-21swapper/308:40:373
3183999362,18cyclictest19584-21ssh09:22:373
3183999361,24cyclictest11640-21ls10:45:223
3183999361,24cyclictest11640-21ls10:45:223
31837993619,6cyclictest7950irq/56-dwc_otg_11:25:212
31837993614,15cyclictest19879-21ssh09:23:232
3183999357,10cyclictest8050irq/56-dwc_otg_08:15:513
3183999354,17cyclictest8050irq/56-dwc_otg_11:40:513
3183799357,14cyclictest8050irq/56-dwc_otg_11:10:362
31837993512,4cyclictest27229-21ssh10:20:222
3183799351,22cyclictest7950irq/56-dwc_otg_10:05:382
3183799351,22cyclictest19604-21missed_timers10:55:452
3183999348,16cyclictest8050irq/56-dwc_otg_09:05:353
3183999347,15cyclictest0-21swapper/308:45:353
3183999347,15cyclictest0-21swapper/308:45:353
3183999347,15cyclictest0-21swapper/307:25:343
3183999343,22cyclictest1140-21snmpd09:13:503
3183999342,21cyclictest15096-21ls09:15:513
3183799345,14cyclictest8050irq/56-dwc_otg_09:40:392
3183799342,22cyclictest7950irq/56-dwc_otg_10:20:432
3183799341,6cyclictest30706-21ntp_states12:35:452
31837993412,6cyclictest1111-21runrttasks12:18:482
3183999337,14cyclictest0-21swapper/312:05:333
3183999335,5cyclictest8050irq/56-dwc_otg_09:40:363
3183999335,19cyclictest17632-21ssh12:20:373
3183999331,17cyclictest8050irq/56-dwc_otg_10:00:363
3183999331,16cyclictest14296-21grep12:15:463
3183799339,5cyclictest0-21swapper/210:45:322
3183799339,12cyclictest0-21swapper/211:40:392
3183799338,16cyclictest0-21swapper/210:35:432
3183799337,23cyclictest0-21swapper/211:05:362
3183799337,11cyclictest0-21swapper/209:50:472
3183799337,11cyclictest0-21swapper/209:50:472
3183799335,6cyclictest7850irq/56-dwc_otg11:25:332
3183799332,21cyclictest7950irq/56-dwc_otg_07:30:472
144712330,2sleep30-21swapper/311:35:063
3183999329,4cyclictest0-21swapper/310:36:433
3183999325,19cyclictest9437-21ssh09:55:343
3183999325,16cyclictest8050irq/56-dwc_otg_10:15:383
3183999322,4cyclictest0-21swapper/310:05:383
3183999321,25cyclictest614-21ssh11:15:223
3183799328,5cyclictest0-21swapper/211:55:412
3183799324,5cyclictest27062-21/usr/sbin/munin08:25:422
3183799322,8cyclictest874-21aten_rbpower_en11:15:392
31837993216,9cyclictest11688-21ssh12:12:452
31837993215,9cyclictest2814-21ssh12:01:202
31837993215,9cyclictest2814-21ssh12:01:202
31837993212,5cyclictest28160-21sh12:32:442
3183999317,17cyclictest27922-21ssh09:35:343
3183999315,13cyclictest8050irq/56-dwc_otg_08:20:443
3183999314,6cyclictest7850irq/56-dwc_otg11:20:373
3183999312,9cyclictest26825-21fschecks_time08:25:393
31839993115,4cyclictest1111-21runrttasks10:50:373
3183999311,18cyclictest7850irq/56-dwc_otg08:05:403
3183999311,15cyclictest0-21swapper/308:05:223
3183999311,15cyclictest0-21swapper/308:05:223
3183799319,4cyclictest0-21swapper/210:42:432
3183799319,4cyclictest0-21swapper/210:42:432
3183799317,5cyclictest28889-21ssh11:53:472
3183799315,12cyclictest7950irq/56-dwc_otg_11:30:472
3183999307,5cyclictest7950irq/56-dwc_otg_10:25:483
3183999307,5cyclictest7950irq/56-dwc_otg_10:25:483
3183999305,15cyclictest8050irq/56-dwc_otg_11:25:413
3183999304,13cyclictest8050irq/56-dwc_otg_10:20:403
31839993010,8cyclictest0-21swapper/311:50:483
3183799309,4cyclictest0-21swapper/207:25:492
3183799304,5cyclictest7950irq/56-dwc_otg_08:20:472
3183799303,7cyclictest28592-21fschecks_count08:30:382
3183799302,15cyclictest7950irq/56-dwc_otg_08:50:552
3183799301,5cyclictest7950irq/56-dwc_otg_09:55:392
3183799301,18cyclictest21346-21cut08:10:422
3183799301,15cyclictest7950irq/56-dwc_otg_07:55:382
3183999299,11cyclictest8050irq/56-dwc_otg_08:32:483
3183999298,4cyclictest0-21swapper/311:05:213
3183999297,13cyclictest8050irq/56-dwc_otg_08:55:483
3183999291,3cyclictest0-21swapper/310:59:243
3183799299,4cyclictest0-21swapper/210:50:532
3183799296,13cyclictest8050irq/56-dwc_otg_08:10:212
3183799294,6cyclictest8050irq/56-dwc_otg_09:36:142
3183799293,4cyclictest7950irq/56-dwc_otg_07:50:512
3183799292,5cyclictest0-21swapper/209:15:342
3183799292,20cyclictest0-21swapper/207:21:502
31837992919,8cyclictest22916-21aten_rbpower_cu08:15:352
3183799291,14cyclictest7850irq/56-dwc_otg09:31:442
37932280,1sleep30-21swapper/307:20:533
3183999286,4cyclictest8050irq/56-dwc_otg_10:30:433
3183999282,16cyclictest0-21swapper/308:15:143
3183799285,10cyclictest7850irq/56-dwc_otg08:35:462
3183999276,5cyclictest1099-21ssh11:15:433
3183799277,11cyclictest8050irq/56-dwc_otg_08:55:362
3183799272,15cyclictest0-21swapper/207:35:312
3183799271,13cyclictest17753-21seq08:00:432
3183799271,13cyclictest17753-21seq08:00:432
315612276,13sleep231604-21ls07:10:222
312362273,14sleep10-21swapper/107:06:161
127622270,5sleep33183999cyclictest07:45:443
31839992519,4cyclictest30803-21ntp_offset08:35:513
3183999251,14cyclictest0-21swapper/307:35:473
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional