You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-27 - 09:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Wed Aug 27, 2025 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
155722920,2sleep20-21swapper/219:05:352
129321000,3sleep30-21swapper/319:05:323
2310999358,7cyclictest8050irq/56-dwc_otg_22:05:180
231099921,65cyclictest0-21swapper/021:00:170
236452840,6sleep3231399cyclictest22:48:203
285312810,2sleep10-21swapper/120:20:321
231099803,55cyclictest0-21swapper/021:30:170
231099796,32cyclictest0-21swapper/000:35:180
2310997939,10cyclictest0-21swapper/022:10:170
214202780,3sleep20-21swapper/222:02:422
291052770,2sleep129104-21ssh22:12:431
2310997754,5cyclictest8150irq/56-dwc_otg_21:55:170
2310997741,18cyclictest0-21swapper/021:20:180
231099772,57cyclictest0-21swapper/020:50:170
231199761,58cyclictest24499-21basename22:50:031
231099741,64cyclictest0-21swapper/020:20:330
231099741,56cyclictest0-21swapper/020:55:170
139742740,4sleep213463-21/usr/sbin/munin21:10:242
91622730,6sleep3231399cyclictest21:45:483
244662720,2sleep2231299cyclictest00:15:352
231099721,51cyclictest0-21swapper/022:15:170
231099702,50cyclictest0-21swapper/020:30:170
231199697,41cyclictest8150irq/56-dwc_otg_19:40:331
231099692,49cyclictest0-21swapper/021:45:160
231199687,44cyclictest8150irq/56-dwc_otg_20:35:331
231199687,44cyclictest8150irq/56-dwc_otg_20:35:331
2310996841,16cyclictest19694-21unixbench_multi19:55:340
2310996841,16cyclictest19694-21unixbench_multi19:55:340
231099681,51cyclictest0-21swapper/021:05:170
231099681,49cyclictest0-21swapper/021:10:170
8482675,21sleep0936-21/usr/sbin/munin19:05:260
2310996743,14cyclictest0-21swapper/019:30:040
2310996735,17cyclictest6313-21ssh23:50:310
231099671,45cyclictest6228-21fschecks_count19:20:220
231099662,52cyclictest0-21swapper/023:30:290
231199655,45cyclictest964-21ssh23:00:321
2311996510,46cyclictest25919-21sensors_temp21:25:321
2310996545,3cyclictest0-21swapper/021:25:190
231099651,52cyclictest25124-21grep22:50:230
231099651,51cyclictest0-21swapper/022:30:200
2312996417,36cyclictest709-21/usr/sbin/munin20:35:322
2312996417,36cyclictest709-21/usr/sbin/munin20:35:322
231199645,45cyclictest8150irq/56-dwc_otg_23:08:291
231199641,4cyclictest13564-21basename22:35:201
231099641,49cyclictest13900-21ps00:00:300
178822640,1sleep30-21swapper/322:40:313
178822640,1sleep30-21swapper/322:40:313
103942640,6sleep3231399cyclictest23:13:483
103942640,6sleep3231399cyclictest23:13:483
231299634,5cyclictest22445-21idleruntime-cro23:30:042
231299634,4cyclictest8150irq/56-dwc_otg_22:40:362
231299634,4cyclictest8150irq/56-dwc_otg_22:40:362
231299632,3cyclictest0-21swapper/221:25:312
2310996339,12cyclictest11994-21ssh23:15:290
2310996333,16cyclictest13575-21aten_rbpower_vo22:35:200
231099632,45cyclictest0-21swapper/022:41:260
231099632,45cyclictest0-21swapper/022:41:260
231099631,55cyclictest0-21swapper/022:45:200
231099631,53cyclictest22807-21cat20:05:280
231299622,4cyclictest8050irq/56-dwc_otg_23:40:262
231299622,4cyclictest8050irq/56-dwc_otg_23:40:262
231199627,38cyclictest0-21swapper/123:10:581
231199627,38cyclictest0-21swapper/123:10:581
231199625,46cyclictest8150irq/56-dwc_otg_23:25:341
231199625,41cyclictest0-21swapper/120:30:181
231099622,52cyclictest10701-21unixbench_multi19:30:340
231099622,49cyclictest0-21swapper/000:05:310
231099622,48cyclictest0-21swapper/023:00:250
231099622,48cyclictest0-21swapper/022:00:210
2310996218,15cyclictest0-21swapper/023:45:240
231299616,26cyclictest0-21swapper/219:40:182
231299611,47cyclictest0-21swapper/222:10:152
231199612,3cyclictest8150irq/56-dwc_otg_22:56:061
231199611,50cyclictest23137-21ssh00:15:121
231099616,14cyclictest0-21swapper/020:35:120
231099616,14cyclictest0-21swapper/020:35:120
231099612,50cyclictest24108-21ls00:15:310
231099612,40cyclictest0-21swapper/020:10:240
231099611,50cyclictest0-21swapper/021:40:320
231199603,5cyclictest9550irq/48-DMA09:15:451
2311996011,33cyclictest17962-21/usr/sbin/munin22:40:321
2311996011,33cyclictest17962-21/usr/sbin/munin22:40:321
231099609,47cyclictest9-21ksoftirqd/023:05:250
231099606,41cyclictest0-21swapper/020:00:290
231099602,7cyclictest17959-21grep19:50:330
231099601,42cyclictest2810-21proc_pri22:20:330
231099601,42cyclictest2810-21proc_pri22:20:330
231099601,41cyclictest27657-21switchtime00:20:310
43852590,1sleep30-21swapper/322:22:563
43852590,1sleep30-21swapper/322:22:563
231199599,40cyclictest18208-21taskset21:58:501
231199599,30cyclictest0-21swapper/100:32:351
231199598,39cyclictest8150irq/56-dwc_otg_22:54:321
231199592,5cyclictest10876-21tail21:00:341
231199591,8cyclictest8150irq/56-dwc_otg_21:11:401
231199591,7cyclictest4044-21ssh23:47:431
231099596,36cyclictest2047-21proc_pri00:30:310
231099592,48cyclictest0-21swapper/023:25:290
231299586,4cyclictest7950irq/56-dwc_otg00:10:262
231299586,4cyclictest7950irq/56-dwc_otg00:10:262
231299584,4cyclictest8150irq/56-dwc_otg_22:45:312
231299583,9cyclictest8050irq/56-dwc_otg_23:15:212
231299582,8cyclictest0-21swapper/221:39:562
2312995817,38cyclictest9438-21latency_hist22:30:042
231299581,4cyclictest9373-21sed19:30:042
231299581,4cyclictest0-21swapper/223:50:242
231199586,4cyclictest7950irq/56-dwc_otg23:41:091
231199586,4cyclictest7950irq/56-dwc_otg23:41:091
231199584,50cyclictest3952-21ssh21:40:201
231199583,4cyclictest8050irq/56-dwc_otg_21:19:571
231199583,4cyclictest8050irq/56-dwc_otg_21:19:571
231199583,4cyclictest24745-21ssh21:25:041
231199583,3cyclictest0-21swapper/121:45:141
231199582,6cyclictest22866-21awk20:05:281
231199582,5cyclictest0-21swapper/119:35:181
231199581,47cyclictest13137-21diskmemload00:23:301
231199581,3cyclictest0-21swapper/122:26:501
2310995847,4cyclictest7950irq/56-dwc_otg23:35:300
231099581,7cyclictest30899-21grep00:25:280
231099581,48cyclictest0-21swapper/020:25:170
231099581,46cyclictest0-21swapper/022:55:240
231299577,4cyclictest0-21swapper/223:24:322
231299573,3cyclictest7950irq/56-dwc_otg19:48:572
231299571,7cyclictest12249-21sed21:50:302
231299571,5cyclictest8068-21hostname20:55:192
231299571,3cyclictest0-21swapper/221:23:042
231199574,43cyclictest8150irq/56-dwc_otg_21:50:331
231199572,42cyclictest0-21swapper/100:25:221
231199571,41cyclictest14612-21ssh23:19:361
231099572,37cyclictest2579-21grep20:40:180
231299563,5cyclictest8050irq/56-dwc_otg_21:45:212
231299562,4cyclictest0-21swapper/200:29:252
2312995614,39cyclictest27383-21sh22:53:162
231299561,3cyclictest0-21swapper/222:34:432
231199565,4cyclictest0-21swapper/123:20:211
231199565,36cyclictest8150irq/56-dwc_otg_00:09:401
231199562,3cyclictest0-21swapper/123:39:231
231199562,39cyclictest8150irq/56-dwc_otg_00:37:231
231199562,10cyclictest0-21swapper/123:50:291
231099563,5cyclictest0-21swapper/000:10:170
231099563,5cyclictest0-21swapper/000:10:170
231099562,42cyclictest0-21swapper/023:20:270
2310995618,28cyclictest0-21swapper/023:10:290
2310995618,28cyclictest0-21swapper/023:10:290
231299556,16cyclictest0-21swapper/200:35:112
231299555,4cyclictest8050irq/56-dwc_otg_19:15:202
231299554,43cyclictest8150irq/56-dwc_otg_23:00:282
231299553,4cyclictest8050irq/56-dwc_otg_22:15:322
231299553,4cyclictest8050irq/56-dwc_otg_22:07:102
231299551,4cyclictest15982-21ssh22:38:072
231199552,3cyclictest0-21swapper/121:34:081
2310995529,4cyclictest0-21swapper/021:35:190
231099552,36cyclictest0-21swapper/023:55:310
231099551,46cyclictest30974-21uname23:40:290
231099551,46cyclictest30974-21uname23:40:290
231099551,44cyclictest11852-21kernelversion19:35:250
231299545,6cyclictest7950irq/56-dwc_otg20:05:302
231299543,3cyclictest7950irq/56-dwc_otg23:45:562
231299542,5cyclictest0-21swapper/219:35:182
231299541,4cyclictest10665-21ssh23:14:222
231299541,4cyclictest10665-21ssh23:14:222
231299541,3cyclictest0-21swapper/221:17:312
231299541,3cyclictest0-21swapper/221:17:312
231299541,3cyclictest0-21swapper/220:50:342
231199549,41cyclictest12347-21processes21:05:301
231199546,4cyclictest8150irq/56-dwc_otg_22:15:321
231199542,4cyclictest24303-21ssh23:31:031
231199541,6cyclictest3485-21ssh22:21:041
231199541,6cyclictest3485-21ssh22:21:041
231199541,6cyclictest23994-21timerwakeupswit22:05:331
231199541,5cyclictest13028-21apt-get00:00:181
2310995428,6cyclictest16626-21latency_hist19:50:040
231099542,44cyclictest0-21swapper/021:19:090
231099542,44cyclictest0-21swapper/021:19:090
231099541,41cyclictest0-21swapper/020:15:230
231299534,7cyclictest8150irq/56-dwc_otg_23:39:262
231299531,5cyclictest136-21systemd-journal19:10:432
231299531,5cyclictest136-21systemd-journal19:10:432
2312995310,40cyclictest19269-21ntp_kernel_pll_19:55:282
2312995310,40cyclictest19269-21ntp_kernel_pll_19:55:282
231199538,3cyclictest25019-21sed20:10:351
231199535,32cyclictest0-21swapper/123:58:271
231199534,5cyclictest0-21swapper/122:01:561
231199533,43cyclictest8150irq/56-dwc_otg_19:48:371
231199532,8cyclictest30050-21missed_timers20:25:261
231199532,8cyclictest0-21swapper/119:20:161
231199532,5cyclictest0-21swapper/120:02:431
231099531,44cyclictest0-21swapper/019:15:320
231399529,14cyclictest8150irq/56-dwc_otg_22:10:173
231299523,46cyclictest26989-21cut00:20:212
231299521,8cyclictest8150irq/56-dwc_otg_23:33:542
231299521,3cyclictest0-21swapper/221:59:102
2312995212,37cyclictest26313-21latency20:15:272
231199527,29cyclictest0-21swapper/121:35:211
231099522,36cyclictest0-21swapper/019:40:190
231099521,37cyclictest5760-21/usr/sbin/munin22:25:170
231299515,6cyclictest7950irq/56-dwc_otg22:21:262
231299515,6cyclictest7950irq/56-dwc_otg22:21:262
231299515,41cyclictest0-21swapper/220:25:332
231299512,8cyclictest20663-21fschecks_count20:00:222
231299511,8cyclictest27947-21fschecks_time20:20:232
231299511,8cyclictest0-21swapper/222:57:192
231299511,6cyclictest11321-21latency_hist19:35:042
231299511,4cyclictest9759-21/usr/sbin/munin21:00:312
231299511,4cyclictest31470-21df_inode20:30:202
231299511,3cyclictest0-21swapper/200:09:592
231199517,29cyclictest0-21swapper/119:50:191
231199514,7cyclictest8150irq/56-dwc_otg_19:15:171
231199512,6cyclictest0-21swapper/120:45:211
231199511,4cyclictest7950irq/56-dwc_otg19:10:181
231199511,4cyclictest7950irq/56-dwc_otg19:10:181
231199511,4cyclictest21840-21ssh00:12:451
231199511,4cyclictest21840-21ssh00:12:451
231299507,39cyclictest912-21runrttasks00:00:252
231299507,38cyclictest0-21swapper/221:40:222
231299501,3cyclictest31819-21ssh21:34:112
231199503,44cyclictest10963-21grep22:30:371
231199501,7cyclictest18708-21basename19:55:201
231199501,7cyclictest18708-21basename19:55:201
231099501,41cyclictest136-21systemd-journal19:10:430
231099501,41cyclictest136-21systemd-journal19:10:430
231299491,6cyclictest12356-21processes21:05:302
2312994915,31cyclictest24009-21/usr/sbin/munin20:10:262
2312994911,36cyclictest13137-21diskmemload23:06:182
231199491,7cyclictest26110-21fschecks_time20:15:241
231099492,37cyclictest0-21swapper/020:49:120
231099491,26cyclictest0-21swapper/021:51:570
231399482,8cyclictest0-21swapper/322:05:183
231399481,4cyclictest8150irq/56-dwc_otg_20:10:193
231299481,7cyclictest9293-21ssh23:55:192
231299481,6cyclictest17891-21grep19:50:332
231299481,6cyclictest1534-21fschecks_count00:30:242
231199483,4cyclictest8050irq/56-dwc_otg_20:50:341
231199481,8cyclictest4122-21munin-run20:45:031
231199481,6cyclictest11284-21latency_hist19:35:041
231199481,4cyclictest0-21swapper/119:30:041
231299472,7cyclictest5467-21sed20:45:372
231299471,8cyclictest6067-21aten_rbpower_vo19:20:192
2312994711,33cyclictest4161-21latency_hist20:45:042
231399457,23cyclictest0-21swapper/300:35:183
231399455,16cyclictest912-21runrttasks21:35:193
2313994527,5cyclictest29363-21ssh22:55:293
231399451,17cyclictest26180-21idleruntime20:15:253
231399447,25cyclictest0-21swapper/321:55:203
2313994311,16cyclictest37-21rcuc/322:15:203
231399417,25cyclictest0-21swapper/319:45:203
2313994119,5cyclictest37-21rcuc/320:30:043
231399407,23cyclictest13137-21diskmemload23:35:163
231399407,22cyclictest0-21swapper/321:40:223
2313994015,4cyclictest11371-21latency_hist21:05:043
231399397,22cyclictest0-21swapper/323:50:193
231399395,22cyclictest19644-21ssh00:10:183
231399395,22cyclictest19644-21ssh00:10:183
2313993919,4cyclictest19492-21irqstats23:25:283
231399391,30cyclictest5993-21ls22:25:213
231399387,26cyclictest8150irq/56-dwc_otg_21:15:183
231399387,26cyclictest8150irq/56-dwc_otg_21:15:183
231399385,6cyclictest0-21swapper/323:15:203
231399374,8cyclictest8150irq/56-dwc_otg_20:05:283
2313993718,4cyclictest24827-21latency_hist21:25:043
231399371,19cyclictest0-21swapper/300:20:173
231399369,13cyclictest39-21ksoftirqd/323:00:243
231399368,11cyclictest8050irq/56-dwc_otg_19:55:043
231399367,4cyclictest8150irq/56-dwc_otg_23:55:193
231399363,4cyclictest7950irq/56-dwc_otg23:40:233
231399363,4cyclictest7950irq/56-dwc_otg23:40:233
231399363,18cyclictest8150irq/56-dwc_otg_19:20:273
231399361,24cyclictest25230-21cpuspeed21:25:213
231399361,17cyclictest0-21swapper/323:45:173
231399355,15cyclictest8150irq/56-dwc_otg_21:50:223
231399352,8cyclictest5253-21sed19:15:343
2313993516,13cyclictest65872chrt23:08:493
231399351,4cyclictest0-21swapper/300:25:173
291282340,1sleep30-21swapper/321:30:223
231399346,20cyclictest39-21ksoftirqd/323:30:343
231399346,15cyclictest39-21ksoftirqd/300:15:233
231399343,18cyclictest8050irq/56-dwc_otg_21:11:513
231399342,8cyclictest9440-21latency_hist19:30:043
231399342,13cyclictest7950irq/56-dwc_otg23:25:043
231399341,22cyclictest13144-21latency_hist21:10:053
231399337,15cyclictest0-21swapper/320:35:123
231399337,15cyclictest0-21swapper/320:35:123
231399334,6cyclictest8150irq/56-dwc_otg_22:30:233
2313993327,4cyclictest39-21ksoftirqd/300:35:043
231399331,5cyclictest8263-21grep20:55:223
231399331,14cyclictest39-21ksoftirqd/319:40:233
231399322,6cyclictest0-21swapper/322:00:183
231399321,7cyclictest19713-21tail19:55:343
231399321,7cyclictest19713-21tail19:55:343
2313993210,10cyclictest0-21swapper/322:36:143
3392311,26sleep3362-21munin-run20:35:043
231399316,6cyclictest7950irq/56-dwc_otg00:05:343
231399314,18cyclictest0-21swapper/322:54:423
231399309,6cyclictest8150irq/56-dwc_otg_00:01:403
231399306,10cyclictest37-21rcuc/320:46:483
231399302,7cyclictest0-21swapper/320:40:213
231399302,10cyclictest8050irq/56-dwc_otg_19:32:213
2313993016,4cyclictest39-21ksoftirqd/319:35:173
231399301,5cyclictest8050irq/56-dwc_otg_20:20:263
231399298,7cyclictest35812chrt19:11:143
231399298,7cyclictest35812chrt19:11:143
231399291,8cyclictest6222-21wget20:50:193
20042294,17sleep10-21swapper/119:09:401
231399281,8cyclictest21456-21grep20:00:323
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional