You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-18 - 14:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Fri Apr 18, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1855421020,2sleep20-21swapper/211:34:342
290772950,2sleep10-21swapper/111:01:561
25512940,3sleep12556-21aten_rbpower_po08:20:251
912399926,78cyclictest0-21swapper/007:15:240
42742890,2sleep30-21swapper/308:25:233
9123998418,25cyclictest0-21swapper/008:25:220
128972820,1sleep00-21swapper/007:20:230
28592800,6sleep0912399cyclictest09:32:020
912399766,41cyclictest18128-21latency_hist07:35:080
9123997231,17cyclictest0-21swapper/008:40:200
9123997231,17cyclictest0-21swapper/008:40:200
912399708,41cyclictest10251-21grep07:10:370
9123997040,4cyclictest25234-21idleruntime-cro12:25:080
912699692,6cyclictest0-21swapper/308:00:223
912699692,6cyclictest0-21swapper/308:00:223
912399697,44cyclictest19769-21grep09:05:370
912399694,48cyclictest23272-21idleruntime-cro10:55:080
9125996810,41cyclictest7950irq/56-dwc_otg_12:35:212
912399681,52cyclictest0-21swapper/011:40:250
75872680,1sleep20-21swapper/207:05:292
44232680,2sleep00-21swapper/011:57:270
912399678,39cyclictest29300-21ps12:30:290
912399678,39cyclictest29300-21ps12:30:290
912399656,50cyclictest11783-21fschecks_time08:45:290
912399654,40cyclictest1971-21fschecks_count10:20:280
912399652,55cyclictest0-21swapper/011:00:280
912399652,43cyclictest0-21swapper/011:30:290
192052650,2sleep00-21swapper/007:35:360
912599645,16cyclictest0-21swapper/211:15:202
912599645,16cyclictest0-21swapper/211:15:202
9123996444,7cyclictest21368-21sed07:40:390
912399635,40cyclictest512-21cron08:40:090
9123996336,11cyclictest0-21swapper/009:50:160
9123996332,6cyclictest7950irq/56-dwc_otg_08:35:090
9123996332,14cyclictest28316-21ssh10:10:340
9123996332,14cyclictest28316-21ssh10:10:340
61202630,2sleep20-21swapper/210:28:202
912699625,6cyclictest2188-21aten_rbpower_en11:55:243
912399623,35cyclictest31224-21open_files11:50:290
912399616,39cyclictest18670-21unixbench_singl09:55:410
912399613,28cyclictest9298-21unixbench_multi09:40:500
912699601,5cyclictest7950irq/56-dwc_otg_12:03:583
912699601,5cyclictest7950irq/56-dwc_otg_12:03:583
9123996033,6cyclictest7950irq/56-dwc_otg_12:10:080
9123996030,16cyclictest29310-21latency_hist08:05:090
9123996030,16cyclictest29310-21latency_hist08:05:090
9123996027,18cyclictest0-21swapper/008:55:080
912399601,45cyclictest2221-21idleruntime-cro08:20:080
912699594,6cyclictest0-21swapper/311:40:093
912699594,6cyclictest0-21swapper/311:40:093
912399598,25cyclictest0-21swapper/007:50:240
9123995933,6cyclictest9541-21grep11:20:390
912399592,48cyclictest0-21swapper/010:40:290
912399592,35cyclictest0-21swapper/008:10:080
912399587,38cyclictest21137-21ssh12:20:080
9123995829,14cyclictest0-21swapper/012:37:150
912399582,49cyclictest0-21swapper/009:14:060
912699572,6cyclictest7850irq/56-dwc_otg09:36:393
912399575,6cyclictest0-21swapper/011:10:160
9123995739,5cyclictest7850irq/56-dwc_otg10:15:300
912399573,39cyclictest27533-21latency_hist08:00:090
912399572,36cyclictest0-21swapper/010:00:370
912399571,34cyclictest6869-21modprobe10:30:080
912699563,8cyclictest7850irq/56-dwc_otg09:55:243
912399563,35cyclictest0-21swapper/011:00:080
9123995627,13cyclictest7950irq/56-dwc_otg_12:00:340
9123995627,13cyclictest7950irq/56-dwc_otg_12:00:340
912399561,30cyclictest7683-21latency10:30:400
912699556,8cyclictest6375-21ssh10:28:543
912399559,32cyclictest14868-21rm12:11:090
912399554,40cyclictest12987-21seq11:26:510
9123995531,9cyclictest0-21swapper/010:36:390
9123995524,18cyclictest0-21swapper/007:45:330
912399552,35cyclictest0-21swapper/007:25:400
912399551,29cyclictest16227-21unixbench_singl08:55:410
9126995411,4cyclictest0-21swapper/309:49:273
912599544,8cyclictest7950irq/56-dwc_otg_07:25:252
115192540,2sleep10-21swapper/111:24:311
912499539,25cyclictest0-21swapper/107:10:221
9124995314,16cyclictest16644-21sshd12:14:271
912399535,35cyclictest5732-21ssh11:15:370
912399535,35cyclictest5732-21ssh11:15:370
912399533,36cyclictest27030-21munin-run09:20:090
912399533,36cyclictest27030-21munin-run09:20:090
9126995217,6cyclictest27484-21aten_rbpower_vo09:20:263
912399527,32cyclictest26446-21diskmemload10:09:300
912399524,33cyclictest32139-21rm09:28:080
9123995237,8cyclictest0-21swapper/011:08:310
912399522,24cyclictest522-21latency_hist08:15:090
912699516,6cyclictest39-21ksoftirqd/312:22:513
912599512,12cyclictest0-21swapper/208:00:212
912599512,12cyclictest0-21swapper/208:00:212
9124995121,14cyclictest0-21swapper/112:20:221
9124995111,22cyclictest16663-21ls09:00:081
912399515,27cyclictest6152-21sshd09:36:270
9123995130,11cyclictest0-21swapper/010:47:140
9123995130,11cyclictest0-21swapper/010:47:140
9123995127,9cyclictest101ktimersoftd/009:20:290
912399512,30cyclictest28148-21ssh12:30:090
9124995017,22cyclictest0-21swapper/107:25:141
9124995010,23cyclictest1975-21grep10:20:281
912699491,3cyclictest0-21swapper/310:39:283
912599496,4cyclictest8050irq/56-dwc_otg_10:55:222
912599492,11cyclictest0-21swapper/212:20:222
912499492,12cyclictest0-21swapper/108:40:211
912499492,12cyclictest0-21swapper/108:40:211
9124994915,12cyclictest29670-21ssh09:22:551
9124994911,5cyclictest8050irq/56-dwc_otg_12:15:211
9123994930,4cyclictest101ktimersoftd/011:47:550
9123994916,15cyclictest0-21swapper/011:35:290
9123994916,15cyclictest0-21swapper/011:35:290
912699488,37cyclictest20286-21seq07:40:233
912699481,16cyclictest14442-21latency09:50:253
912499487,29cyclictest10619-21ssh10:35:211
912499487,23cyclictest32625-21ssh12:35:231
912499485,16cyclictest11276-21ssh09:45:061
912499485,14cyclictest26900-21ssh12:27:491
912499484,10cyclictest30410-21ssh10:15:221
9124994811,19cyclictest7541-21users12:00:401
9124994811,19cyclictest7541-21users12:00:401
912699475,9cyclictest617-21ssh11:10:173
912599477,25cyclictest0-21swapper/210:40:242
912599477,25cyclictest0-21swapper/209:10:252
912599477,25cyclictest0-21swapper/207:50:132
912499479,23cyclictest0-21swapper/107:55:221
912499477,25cyclictest0-21swapper/109:55:251
912499474,13cyclictest3518-21ssh11:13:541
9124994713,16cyclictest27658-21ssh11:45:371
9124994710,22cyclictest21624-21ssh11:37:391
9124994710,22cyclictest21624-21ssh11:37:391
912399471,32cyclictest13609-21idleruntime-cro09:50:080
912699467,36cyclictest0-21swapper/310:20:313
912699467,24cyclictest0-21swapper/308:35:223
912599466,25cyclictest0-21swapper/208:25:222
912499467,25cyclictest0-21swapper/111:40:141
912499461,20cyclictest8050irq/56-dwc_otg_07:35:321
912699459,22cyclictest0-21swapper/312:30:183
912699459,22cyclictest0-21swapper/312:30:183
912599457,24cyclictest0-21swapper/209:40:232
912599454,25cyclictest0-21swapper/207:15:242
912499458,25cyclictest0-21swapper/107:40:141
912499457,23cyclictest0-21swapper/108:30:251
9124994518,4cyclictest30444-21aten_rbpower_cu11:50:181
9124994510,16cyclictest24357-21ntpq10:55:321
9123994511,4cyclictest0-21swapper/009:00:230
912599447,25cyclictest0-21swapper/209:15:232
912599447,25cyclictest0-21swapper/209:15:232
912599447,24cyclictest0-21swapper/207:30:262
912599441,34cyclictest0-21swapper/211:00:392
912499448,23cyclictest0-21swapper/108:10:261
912499448,13cyclictest29619-21grep12:30:321
912499448,13cyclictest29619-21grep12:30:321
912499447,12cyclictest22167-21ssh09:11:221
912499446,25cyclictest0-21swapper/109:30:181
912499442,10cyclictest0-21swapper/110:40:241
9124994412,17cyclictest6187-21ssh11:16:311
9124994412,17cyclictest6187-21ssh11:16:311
9124994411,13cyclictest20073-21ssh10:50:141
85772442,35sleep00-21swapper/007:06:370
9125994321,4cyclictest7850irq/56-dwc_otg12:10:222
912499438,23cyclictest0-21swapper/111:55:241
912499437,24cyclictest0-21swapper/110:10:141
912499437,24cyclictest0-21swapper/110:10:141
912499437,23cyclictest0-21swapper/108:05:141
912499434,20cyclictest0-21swapper/108:35:211
912499434,12cyclictest12051-21wc08:45:321
9124994319,10cyclictest8050irq/56-dwc_otg_11:08:321
9123994320,16cyclictest0-21swapper/008:20:320
912699425,24cyclictest0-21swapper/308:05:233
912599426,23cyclictest6253-21ssh12:00:242
912599426,23cyclictest6253-21ssh12:00:242
9125994218,4cyclictest7850irq/56-dwc_otg12:15:232
9125994210,5cyclictest0-21swapper/210:35:272
912499429,22cyclictest11101-21ssh12:06:461
912499421,14cyclictest22798-21ssh10:01:381
912499421,11cyclictest14113-21/usr/sbin/munin09:50:211
912599416,7cyclictest8050irq/56-dwc_otg_09:30:172
912599413,25cyclictest0-21swapper/208:05:232
912499417,22cyclictest0-21swapper/107:50:131
9124994115,14cyclictest221ktimersoftd/110:33:551
9124994114,4cyclictest8050irq/56-dwc_otg_09:20:091
9124994114,4cyclictest8050irq/56-dwc_otg_09:20:091
912699407,25cyclictest0-21swapper/308:55:233
912699406,20cyclictest13244-21ssh12:10:183
912499408,22cyclictest0-21swapper/108:50:251
912499406,8cyclictest13220-21ssh11:27:181
912499404,10cyclictest19691-21ssh10:49:531
912499404,10cyclictest19691-21ssh10:49:531
9124994016,8cyclictest7850irq/56-dwc_otg11:34:471
912599395,7cyclictest7950irq/56-dwc_otg_11:40:242
9125993914,7cyclictest878-21runrttasks11:35:372
9125993914,7cyclictest878-21runrttasks11:35:372
912499397,26cyclictest0-21swapper/107:15:131
9124993913,12cyclictest12129-21ssh09:46:261
912599388,5cyclictest0-21swapper/211:51:002
912499384,19cyclictest31499-21ssh09:26:401
9124993810,11cyclictest25037-21ssh10:06:211
912699376,22cyclictest0-21swapper/309:30:183
9126993716,5cyclictest978650wget08:40:243
9126993716,5cyclictest978650wget08:40:243
9126993715,5cyclictest27213-21aten_rbpower_en11:00:263
9126993715,11cyclictest31223-21sh09:26:023
912599374,17cyclictest8050irq/56-dwc_otg_08:35:362
912599371,5cyclictest8908-21ls11:20:312
912599371,26cyclictest20004-21ls07:40:092
912499377,17cyclictest0-21swapper/107:20:231
912599366,15cyclictest0-21swapper/207:45:222
912599363,7cyclictest0-21swapper/212:35:102
912599363,7cyclictest0-21swapper/212:35:102
9125993615,5cyclictest2118050wget10:00:252
912599361,25cyclictest11237-21latency_hist08:45:082
912599361,25cyclictest11237-21latency_hist08:45:082
9125993611,6cyclictest7850irq/56-dwc_otg10:30:342
912499362,11cyclictest0-21swapper/110:26:201
9124993616,5cyclictest221ktimersoftd/109:05:321
912699357,15cyclictest0-21swapper/311:50:183
9126993513,5cyclictest37-21rcuc/312:35:233
912699351,20cyclictest0-21swapper/307:10:223
912599357,4cyclictest0-21swapper/212:29:162
912599357,23cyclictest0-21swapper/207:40:142
912599356,6cyclictest7850irq/56-dwc_otg10:20:092
912599356,15cyclictest8050irq/56-dwc_otg_08:30:262
9125993518,5cyclictest18618-21sort09:05:092
9125993515,4cyclictest21311-21unixbench_singl10:50:442
912599351,23cyclictest2222-21latency10:20:312
912499355,13cyclictest23314-21runrttasks07:46:121
9124993516,9cyclictest8050irq/56-dwc_otg_09:36:491
912699347,5cyclictest0-21swapper/307:25:233
912699345,8cyclictest8446-21fschecks_count09:40:283
912699341,21cyclictest26696-21timerandwakeup07:55:353
9125993426,5cyclictest18732-21aten_rbpower_en09:05:232
912599342,20cyclictest0-21swapper/209:54:292
9125993413,6cyclictest7950irq/56-dwc_otg_11:29:522
912599341,22cyclictest16105-21ls08:55:402
912499349,5cyclictest18327-21sed07:35:091
912499344,20cyclictest0-21swapper/108:15:211
912499341,12cyclictest4670-21if_eth008:25:281
912699337,11cyclictest8050irq/56-dwc_otg_07:50:083
912699335,13cyclictest8050irq/56-dwc_otg_08:20:253
912599338,13cyclictest8050irq/56-dwc_otg_07:55:312
912599332,10cyclictest0-21swapper/208:20:252
912599331,5cyclictest30225-21latency_hist10:15:092
912599331,5cyclictest30225-21latency_hist10:15:092
912599331,4cyclictest8050irq/56-dwc_otg_09:55:232
9125993314,9cyclictest11966-21sh12:08:232
912599331,25cyclictest3169-21ssh11:55:382
9125993312,5cyclictest20245-21diskmemload09:25:392
23232330,6sleep2912599cyclictest11:11:272
912699327,5cyclictest7950irq/56-dwc_otg_11:25:083
912699326,11cyclictest7950irq/56-dwc_otg_12:10:093
912699324,5cyclictest12580-21unixbench_multi08:45:393
912699323,13cyclictest18379-21ssh10:46:523
912699323,13cyclictest18379-21ssh10:46:523
912699322,8cyclictest30224-21ssh11:50:093
912699322,21cyclictest0-21swapper/312:15:403
912699321,8cyclictest24090-21cut09:15:283
912699321,8cyclictest24090-21cut09:15:283
9126993214,7cyclictest0-21swapper/310:30:433
9126993214,4cyclictest37-21rcuc/310:00:293
912699321,23cyclictest0-21swapper/311:05:143
9126993210,5cyclictest14914-21latency_hist11:30:093
912599322,4cyclictest7950irq/56-dwc_otg_09:50:082
9125993214,4cyclictest13441-21/usr/sbin/munin08:50:262
9125993210,13cyclictest2145-21taskset08:19:212
912499323,4cyclictest0-21swapper/108:05:101
912499323,4cyclictest0-21swapper/108:05:101
76892325,15sleep37769-21/usr/sbin/munin07:05:313
311952320,2sleep20-21swapper/211:06:482
912699317,5cyclictest0-21swapper/311:40:403
912699315,6cyclictest13184-21fschecks_time07:20:273
912699314,6cyclictest8050irq/56-dwc_otg_10:10:093
9126993121,7cyclictest23468-21idleruntime-cro09:15:083
9126993115,4cyclictest7605-21cron08:35:093
9126993111,4cyclictest14470-21sh10:40:293
912599316,6cyclictest17701-21sh10:45:152
912599316,6cyclictest17701-21sh10:45:152
912599316,5cyclictest8050irq/56-dwc_otg_09:20:292
912599313,13cyclictest7950irq/56-dwc_otg_10:05:282
9125993121,4cyclictest8050irq/56-dwc_otg_11:45:562
912699305,5cyclictest7950irq/56-dwc_otg_08:10:253
912699304,12cyclictest7950irq/56-dwc_otg_11:33:513
912699301,9cyclictest31137-21ntpq10:15:323
9126993013,15cyclictest878-21runrttasks11:15:293
9126993013,15cyclictest878-21runrttasks11:15:293
9126993011,4cyclictest18558-21latency_hist09:05:093
912599305,7cyclictest7950irq/56-dwc_otg_07:10:302
912599302,14cyclictest0-21swapper/208:13:432
9125993015,5cyclictest5630-21/usr/sbin/munin09:35:402
912699298,4cyclictest0-21swapper/310:11:093
912699298,4cyclictest0-21swapper/310:11:093
912699298,10cyclictest0-21swapper/310:55:233
912699291,6cyclictest2578650ssh12:25:463
9126992914,10cyclictest0-21swapper/307:15:143
9126992910,4cyclictest0-21swapper/310:54:123
912599294,5cyclictest8050irq/56-dwc_otg_07:21:362
912499293,10cyclictest0-21swapper/109:00:221
85452294,16sleep10-21swapper/107:06:141
912699282,4cyclictest0-21swapper/308:50:263
912699281,4cyclictest0-21swapper/308:15:213
912699281,18cyclictest18400-21wget07:35:253
912599283,15cyclictest0-21swapper/208:45:392
912699276,18cyclictest19594-21timerandwakeup09:05:353
912699271,6cyclictest0-21swapper/307:50:273
9126992714,4cyclictest16566-21/usr/sbin/munin07:30:393
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional