You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-11 - 22:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: reduced 
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot8s.osadl.org (updated Mon Dec 11, 2023 12:44:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
240232699742739,2cyclictest2662138-21turbostat11:04:573
240232699742739,2cyclictest2448941-21turbostat07:54:563
240232699741738,2cyclictest2756794-21turbostat12:29:553
240232199737734,2cyclictest2420980-21turbostat07:29:551
240232699733730,2cyclictest2628778-21turbostat10:34:553
240232699731728,2cyclictest2678853-21turbostat11:19:553
240232699730727,2cyclictest2701135-21turbostat11:39:553
240232699727724,2cyclictest2606147-21turbostat10:14:553
240232699725723,1cyclictest2527394-21turbostat09:04:553
240232699724717,6cyclictest2544509-21turbostat09:19:563
240232199724721,2cyclictest2437791-21turbostat07:44:561
240232699721718,2cyclictest2499634-21turbostat08:39:553
240232699717714,2cyclictest2717831-21turbostat11:54:553
240232199717714,2cyclictest2734576-21turbostat12:09:551
240232699716714,1cyclictest2465685-21turbostat08:09:553
240232199716710,5cyclictest2477271-21turbostat08:19:551
240232699715708,6cyclictest2651048-21turbostat10:54:553
240232199715705,6cyclictest2443361-21turbostat07:49:561
240232699713710,2cyclictest2656590-21turbostat10:59:563
240232199713710,2cyclictest2426536-21turbostat07:34:561
240232699710707,2cyclictest2583868-21turbostat09:54:553
240232399710707,2cyclictest2745635-21turbostat12:19:552
240232699708705,2cyclictest2494070-21turbostat08:34:553
240232699707704,2cyclictest2762335-21turbostat12:34:553
240232699706704,1cyclictest2516312-21turbostat08:54:553
240232699705702,2cyclictest2578303-21turbostat09:49:553
240232699703700,2cyclictest2600563-21turbostat10:09:553
240232699701698,2cyclictest2594972-21turbostat10:04:553
240232699700694,5cyclictest2611746-21turbostat10:19:553
240232199700698,1cyclictest2505159-21turbostat08:44:551
240232199699693,5cyclictest2667727-21turbostat11:09:561
240232199698695,2cyclictest2432152-21turbostat07:39:561
240232399696693,2cyclictest2471333-21turbostat08:14:552
240232399694691,2cyclictest2533325-21turbostat09:09:552
240232199694687,6cyclictest2706692-21turbostat11:44:551
240232699691688,2cyclictest2712291-21turbostat11:49:553
240232699691688,2cyclictest2712291-21turbostat11:49:553
240232699691688,2cyclictest2407748-21turbostat07:19:553
240232199691688,2cyclictest2510743-21turbostat08:49:551
240232199689686,2cyclictest2639899-21turbostat10:44:561
240232699688685,2cyclictest2482877-21turbostat08:24:553
240232199688686,1cyclictest2572725-21turbostat09:44:551
240232699687684,2cyclictest2556041-21turbostat09:29:563
240232199687684,2cyclictest2460104-21turbostat08:04:551
240231699686679,6cyclictest2634345-21turbostat10:39:550
240232699685683,1cyclictest2645507-21turbostat10:49:553
240232699685683,1cyclictest2589410-21turbostat09:59:553
240232399685683,1cyclictest2415368-21turbostat07:24:552
240231699683680,2cyclictest2550420-21turbostat09:24:560
240231699682679,2cyclictest2402131-21turbostat07:14:550
240232699681679,1cyclictest2488464-21turbostat08:29:553
240232399681679,1cyclictest2684407-21turbostat11:24:552
240231699680678,1cyclictest2538941-21turbostat09:14:550
240232699679677,1cyclictest2751236-21turbostat12:24:553
240232699679677,1cyclictest2561581-21turbostat09:34:563
240232699678675,2cyclictest2673286-21turbostat11:14:553
240232699678675,2cyclictest2521836-21turbostat08:59:553
240232699677674,2cyclictest2723423-21turbostat11:59:563
240232199675673,1cyclictest2689990-21turbostat11:29:551
240232699674671,2cyclictest2728996-21turbostat12:04:553
240232699673670,2cyclictest2454538-21turbostat07:59:553
240232199668666,1cyclictest2622871-21turbostat10:29:551
240231699660657,2cyclictest2695586-21turbostat11:34:550
240232699656653,2cyclictest2768205-21turbostat12:39:553
240232399653651,1cyclictest2740118-21turbostat12:14:552
240232199652649,2cyclictest2567119-21turbostat09:39:551
240232399650647,2cyclictest2617285-21turbostat10:24:552
240232199642640,1cyclictest2432371-21kworker/1:1+pm08:20:121
240232199642639,2cyclictest2711958-21kworker/1:1+pm11:45:101
240232199642639,2cyclictest2711958-21kworker/1:1+pm11:45:101
240232199627625,1cyclictest2600529-21kworker/1:2+pm10:45:111
240231699600598,1cyclictest2628883-21kworker/0:0+pm10:40:100
240231699595593,1cyclictest2396708-21kworker/0:1+pm07:15:120
240232699593591,1cyclictest2611063-21kworker/3:1+pm10:20:123
240232699588585,2cyclictest2667692-21kworker/3:2+pm11:05:113
240232699586581,4cyclictest2415338-21kworker/3:1+pm09:05:123
240232199586584,1cyclictest2432371-21kworker/1:1+pm08:50:101
240232699579576,2cyclictest2415338-21kworker/3:1+pm07:20:113
240232399577575,1cyclictest2617403-21kworker/2:2+pm11:25:112
240232699574572,1cyclictest2544480-21kworker/3:2+pm09:35:103
240232699573571,1cyclictest2667692-21kworker/3:2+pm11:20:103
240232199564561,2cyclictest2432371-21kworker/1:1+pm09:45:111
240232199345342,2cyclictest2600529-21kworker/1:2+pm10:30:101
240232399339337,1cyclictest2617403-21kworker/2:2+pm10:25:122
240232199318315,2cyclictest2711958-21kworker/1:1+pm12:10:111
240232699304301,2cyclictest2695556-21kworker/3:0+pm12:05:123
240232699296293,2cyclictest2611063-21kworker/3:1+pm10:35:123
240232699294291,2cyclictest2415338-21kworker/3:1+pm08:40:103
240232199287284,2cyclictest2432371-21kworker/1:1+pm07:50:111
240232199279276,2cyclictest2600529-21kworker/1:2+pm11:10:121
240232399270266,2cyclictest2200232-21kworker/2:0+pm07:25:132
240232399269266,2cyclictest2717854-21kworker/2:0+pm12:20:112
240232199261258,2cyclictest2390962-21kworker/1:2+pm07:10:111
240232699260256,2cyclictest2415338-21kworker/3:1+pm08:10:113
240232199257254,2cyclictest2600529-21kworker/1:2+pm11:30:111
240232699239236,2cyclictest2544480-21kworker/3:2+pm09:20:123
240231699218215,2cyclictest2510717-21kworker/0:2+pm09:15:120
240231699217215,1cyclictest2701098-21kworker/0:0+pm11:35:120
240231699209206,2cyclictest2510717-21kworker/0:2+pm09:25:120
240232399207204,2cyclictest2454627-21kworker/2:1+pm08:15:122
240232699199196,2cyclictest2415338-21kworker/3:1+pm08:00:113
240232399192189,2cyclictest2510769-21kworker/2:0+pm09:10:122
240232699186183,2cyclictest2695556-21kworker/3:0+pm11:40:113
240232199165163,1cyclictest2432371-21kworker/1:1+pm08:05:121
2402323996965,3cyclictest2662168-21kworker/u8:2+flush-8:011:10:002
2402323996460,3cyclictest2701099-21kworker/u8:4+flush-8:012:00:252
2402323996359,2cyclictest2594944-21kworker/u8:1+flush-8:010:00:002
2402316996054,4cyclictest2578291-21latency_hist09:45:000
2402323995955,3cyclictest2701099-21kworker/u8:4+flush-8:012:05:382
2402323995753,3cyclictest2622834-21kworker/u8:3+flush-8:010:55:312
2402323995652,3cyclictest2407723-21kworker/u8:3+flush-8:007:15:012
2402321995651,3cyclictest2606191-21apt-get10:10:001
2402323995551,2cyclictest2533553-21kworker/u8:3+flush-8:009:15:212
2402326995347,3cyclictest2628766-21latency_hist10:30:013
2402323995349,3cyclictest2407723-21kworker/u8:3+flush-8:007:30:372
2402323995248,3cyclictest2488425-21kworker/u8:0+events_unbound08:30:282
2402323995246,4cyclictest2494038-21kworker/u8:1+events_unbound08:50:362
2402316995245,5cyclictest2622857-21latency_hist10:25:010
2402323995045,3cyclictest2533553-21kworker/u8:3+flush-8:009:20:262
2402323994845,2cyclictest2407723-21kworker/u8:3+events_unbound07:35:202
2402323994843,3cyclictest2622834-21kworker/u8:3+flush-8:010:45:312
2402316994842,4cyclictest2437777-21latency_hist07:40:010
2402316994744,2cyclictest2690015-21kworker/u8:1+flush-8:011:45:280
2402316994744,2cyclictest2690015-21kworker/u8:1+flush-8:011:45:280
2402323994642,3cyclictest2651030-21kworker/u8:0+events_unbound11:30:222
2402323994541,3cyclictest2673309-21kworker/u8:1+flush-8:011:15:242
2402323994541,3cyclictest2662168-21kworker/u8:2+events_unbound11:10:232
2402323994540,3cyclictest2723444-21kworker/u8:1+flush-8:012:35:262
2402321994541,3cyclictest2701099-21kworker/u8:4+flush-8:011:35:321
2402323994338,4cyclictest2578283-21kworker/u8:1+events_unbound09:45:262
2402323994238,3cyclictest2690015-21kworker/u8:1+flush-8:011:40:242
2402323994238,3cyclictest2460138-21kworker/u8:2+flush-8:008:20:352
2402323994237,4cyclictest2706758-21kworker/u8:0+events_unbound12:25:342
2402323994237,4cyclictest2567097-21kworker/u8:2+events_unbound09:35:272
2402323994235,4cyclictest2728977-21latency_hist12:00:012
2402323994137,3cyclictest2622834-21kworker/u8:3+events_unbound10:35:302
2402323994137,3cyclictest2443380-21kworker/u8:0+flush-8:007:55:302
2402323994137,2cyclictest2505126-21kworker/u8:2+flush-8:008:55:312
2402316994138,2cyclictest2701099-21kworker/u8:4+events_unbound11:50:320
2402316994137,2cyclictest2533553-21kworker/u8:3+flush-8:009:30:260
2402323994036,3cyclictest2651030-21kworker/u8:0+flush-8:010:50:312
2402323994036,3cyclictest2443380-21kworker/u8:0+events_unbound08:00:342
2402321994036,3cyclictest2583908-21kworker/u8:2+events_unbound10:20:251
2402321994034,4cyclictest2684392-21latency_hist11:20:001
2402321994033,5cyclictest2762334-21apt-get12:30:001
2402316994034,4cyclictest482-21rs:main0
2402326993935,3cyclictest2505126-21kworker/u8:2+events_unbound09:10:343
2402323993936,2cyclictest2606549-21kworker/u8:1+flush-8:010:10:262
2402323993835,2cyclictest2594944-21kworker/u8:1+events_unbound10:00:232
2402323993834,2cyclictest2583908-21kworker/u8:2+events_unbound10:15:242
2402321993834,3cyclictest2505126-21kworker/u8:2+flush-8:008:55:281
2402316993834,3cyclictest2673309-21kworker/u8:1+events_unbound11:15:230
2402323993734,2cyclictest2505126-21kworker/u8:2+events_unbound09:00:372
2402323993733,3cyclictest2690015-21kworker/u8:1+flush-8:011:45:332
2402323993733,3cyclictest2690015-21kworker/u8:1+flush-8:011:45:332
2402323993733,3cyclictest2465650-21kworker/u8:4+events_unbound08:27:412
2402321993733,3cyclictest2533553-21kworker/u8:3+events_unbound09:30:311
2402321993732,4cyclictest2488425-21kworker/u8:0+flush-8:008:35:361
2402316993731,4cyclictest2706972-21apt-get11:40:200
2402323993633,2cyclictest2494038-21kworker/u8:1+flush-8:008:45:262
2402323993629,4cyclictest2573085-21diskstats09:40:212
2402316993633,2cyclictest2706758-21kworker/u8:0+flush-8:012:15:330
2402323993527,3cyclictest2465650-21kworker/u8:4+events_unbound08:40:252
2402321993532,2cyclictest2583908-21kworker/u8:2+events_unbound10:10:341
2402321993531,3cyclictest2594944-21kworker/u8:1+events_unbound10:00:211
2402316993529,4cyclictest2516298-21latency_hist08:50:010
2402326993429,3cyclictest3974348-21systemd-journal07:40:383
2402323993431,2cyclictest2544538-21kworker/u8:0+events_unbound09:50:222
2402323993430,3cyclictest2690015-21kworker/u8:1+flush-8:011:35:232
2402321993431,2cyclictest2723444-21kworker/u8:1+events_unbound12:35:311
2402321993431,2cyclictest2701099-21kworker/u8:4+flush-8:011:55:321
2402321993431,2cyclictest2701099-21kworker/u8:4+flush-8:011:50:331
2402321993429,4cyclictest2544512-21apt-get09:15:011
2402316993431,2cyclictest2432171-21kworker/u8:1+flush-8:008:00:000
2402316993430,3cyclictest2622834-21kworker/u8:3+flush-8:010:30:210
2402316993430,2cyclictest2527446-21kworker/u8:0+events_unbound09:05:010
2402321993330,2cyclictest2544538-21kworker/u8:0+events_unbound09:50:241
2402316993329,3cyclictest2662168-21kworker/u8:2+events_unbound11:00:320
2402316993329,3cyclictest2544538-21kworker/u8:0+events_unbound09:50:000
2402323993228,3cyclictest2706758-21kworker/u8:0+events_unbound12:30:292
2402323993228,3cyclictest2488425-21kworker/u8:0+events_unbound08:35:352
2402321993228,3cyclictest2583908-21kworker/u8:2+flush-8:009:55:381
2402321993226,4cyclictest2634603-21apt-get10:35:221
2402316993226,4cyclictest2488448-21latency_hist08:25:010
240232699310,20cyclictest0-21swapper/307:35:183
240232699310,19cyclictest0-21swapper/307:30:223
2402323993128,2cyclictest2407723-21kworker/u8:3+flush-8:008:05:322
2402323993127,3cyclictest2407723-21kworker/u8:3+events_unbound07:40:352
2402321993127,3cyclictest2583908-21kworker/u8:2+events_unbound10:15:181
2402321993123,6cyclictest3974348-21systemd-journal12:35:001
2402316993127,3cyclictest3974348-21systemd-journal10:55:010
2402326993024,4cyclictest2367940-21kworker/u8:2+flush-8:007:10:303
2402321993026,3cyclictest2460138-21kworker/u8:2+events_unbound08:10:551
2402321993022,5cyclictest2533587-21apt-get09:05:211
2402316993027,2cyclictest2407723-21kworker/u8:3+flush-8:008:05:310
2402323992926,2cyclictest2407723-21kworker/u8:3+events_unbound07:50:272
2402323992925,3cyclictest2533553-21kworker/u8:3+events_unbound10:05:302
2402321992925,3cyclictest2533553-21kworker/u8:3+flush-8:009:25:321
2402321992823,3cyclictest2533553-21kworker/u8:3+events_unbound09:20:241
2402316992823,4cyclictest2460138-21kworker/u8:2+flush-8:008:25:340
2402316992822,4cyclictest2454537-21latency_hist07:55:000
2402316992723,3cyclictest2762811-21gpgv12:30:010
2402316992723,3cyclictest2622834-21kworker/u8:3+flush-8:010:25:260
2402323992622,3cyclictest2662168-21kworker/u8:2+events_unbound11:01:402
2402321992622,3cyclictest2443380-21kworker/u8:0+events_unbound07:55:241
2402316992620,4cyclictest2494322-21apt-get08:30:220
2402323992522,2cyclictest2701099-21kworker/u8:4+events_unbound11:50:342
2402321992521,3cyclictest2706758-21kworker/u8:0+flush-8:012:15:351
240232199251,21cyclictest2415642-21apt-get07:20:191
2402316992521,3cyclictest2443380-21kworker/u8:0+flush-8:008:00:380
2402316992520,4cyclictest2701099-21kworker/u8:4+events_unbound12:05:350
2402316992520,4cyclictest2533553-21kworker/u8:3+flush-8:009:05:310
2402316992520,4cyclictest2465650-21kworker/u8:4+flush-8:008:15:260
2402316992519,4cyclictest2421235-21apt-get07:25:220
2402316992517,5cyclictest2740371-21apt-get12:10:220
2402326992419,4cyclictest2567097-21kworker/u8:2+flush-8:009:40:233
2402321992419,3cyclictest2533553-21kworker/u8:3+flush-8:009:15:251
2402316992420,3cyclictest2701099-21kworker/u8:4+flush-8:012:00:290
2402316992419,4cyclictest2494038-21kworker/u8:1+events_unbound08:50:380
2402316992419,3cyclictest2527405-21apt-get09:00:000
2402321992319,3cyclictest2651030-21kworker/u8:0+flush-8:011:05:011
2402316992320,2cyclictest2606452-21apt-get10:10:190
2402316992319,3cyclictest2651030-21kworker/u8:0+events_unbound11:25:320
2402321992218,2cyclictest2734567-21munin-run12:05:001
2402321992217,4cyclictest2684374-21kworker/u8:3+flush-8:011:20:201
240232199220,19cyclictest0-21swapper/108:25:211
2402316992217,4cyclictest2426509-21kworker/u8:2+flush-8:007:45:350
2402323992119,1cyclictest0-21swapper/210:40:192
240232399211,19cyclictest0-21swapper/210:30:192
2402321992118,2cyclictest2745610-21kworker/u8:2+events_unbound12:20:241
2402321992118,2cyclictest2533325-21turbostat09:05:001
2402321992118,2cyclictest0-21swapper/108:30:231
2402321992117,3cyclictest2407723-21kworker/u8:3+flush-8:007:15:311
2402316992117,3cyclictest2651030-21kworker/u8:0+events_unbound10:55:230
240231699211,19cyclictest0-21swapper/012:20:210
240231699210,20cyclictest2673297-21basename11:10:000
240231699210,1cyclictest0-21swapper/008:40:220
240231699210,19cyclictest2606191-21apt-get10:10:010
2402326992018,1cyclictest0-21swapper/311:25:183
2402326992018,1cyclictest0-21swapper/308:15:223
240232699200,19cyclictest0-21swapper/312:15:213
2402323992018,1cyclictest60-21kcompactd009:25:182
2402323992016,3cyclictest2533553-21kworker/u8:3+flush-8:009:30:312
240232399200,19cyclictest0-21swapper/207:15:192
2402321992016,3cyclictest2622834-21kworker/u8:3+flush-8:010:55:191
2402321992016,3cyclictest2622834-21kworker/u8:3+events_unbound10:50:231
2402316992018,1cyclictest2499890-21apt-get08:35:230
2402316992016,2cyclictest3974348-21systemd-journal12:30:000
2402316992016,2cyclictest2584980-21sed09:50:260
240231699200,1cyclictest0-21swapper/007:40:250
2402326991916,2cyclictest0-21swapper/307:50:013
2402326991915,3cyclictest2651030-21kworker/u8:0+flush-8:011:30:343
240232699190,18cyclictest0-21swapper/312:10:233
2402316991913,4cyclictest2600547-21latency_hist10:05:010
2402316991816,1cyclictest0-21swapper/011:10:190
2402316991814,3cyclictest2471898-21kworker/u8:1+flush-8:008:10:410
2402323991714,2cyclictest2432171-21kworker/u8:1+events_unbound07:45:212
2402316991715,1cyclictest0-21swapper/010:15:180
2402326991611,4cyclictest2494038-21kworker/u8:1+flush-8:008:45:263
2402316991614,1cyclictest0-21swapper/011:55:200
2402316991612,3cyclictest2723444-21kworker/u8:1+flush-8:012:35:230
2402326991512,2cyclictest2426528-21cron07:30:013
240232699150,15cyclictest0-21swapper/310:44:383
2402316991514,1cyclictest2396708-21kworker/0:1+kacpid07:34:100
2402316991411,2cyclictest2572713-21idleruntime-cro09:40:000
2402316991410,3cyclictest2634317-21kworker/u8:2+flush-8:010:45:340
2402316991310,2cyclictest2583908-21kworker/u8:2+events_unbound09:55:250
28269890,5rtkit-daemon3680404-21snmpd07:08:143
24023169997,1cyclictest0-21swapper/007:20:200
24023219971,4cyclictest2402320-21sed07:10:011
24023239960,4cyclictest2402332-21latency_hist07:10:012
24023169961,4cyclictest2402118-21latency_hist07:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional