You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 23:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot4s.osadl.org (updated Fri May 02, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2570269993125,4cyclictest2868008-21latency_hist11:55:013
2570264993025,3cyclictest3000860-21systemd-journal12:39:321
2570269992923,4cyclictest2675208-21/usr/sbin/munin08:50:133
2570264992926,2cyclictest2888414-21fwupd12:11:331
2570264992925,3cyclictest2816106-21kworker/u8:3+flush-259:011:10:161
2570264992719,6cyclictest2622118-21latency_hist08:00:011
2570259992723,3cyclictest2585850-21kworker/u8:1+events_unbound08:45:010
257025999270,26cyclictest1-21systemd10:15:010
2570266992621,4cyclictest2570196-21latency_hist07:10:002
2570259992621,4cyclictest3000860-21systemd-journal07:15:000
2570266992521,2cyclictest2758541-21kworker/u8:0+flush-259:010:15:202
2570264992520,3cyclictest2847514-21/usr/sbin/munin11:35:161
2570264992519,4cyclictest3000860-21systemd-journal12:00:001
2570264992519,4cyclictest3000860-21systemd-journal07:36:311
2570264992518,5cyclictest3000860-21systemd-journal07:13:311
2570269992418,4cyclictest2915174-21latency_hist12:40:013
2570266992420,3cyclictest3000860-21systemd-journal08:55:312
2570259992419,4cyclictest2586052-21/usr/sbin/munin07:25:160
2570259992418,4cyclictest3000860-21systemd-journal09:46:310
2570259992318,3cyclictest3000860-21systemd-journal11:35:010
2570259992316,5cyclictest2914278-21ntpq12:35:240
2570266992219,2cyclictest2816106-21kworker/u8:3+flush-259:011:30:332
2570266992218,3cyclictest2899576-21latency_hist12:25:012
2570264992219,2cyclictest2816106-21kworker/u8:3+flush-259:012:10:001
2570259992218,3cyclictest2606571-21kworker/u8:2+flush-259:008:25:010
2570266992116,4cyclictest2795324-21latency_hist10:45:012
2570264992116,4cyclictest2585850-21kworker/u8:1+flush-259:008:20:001
257026999205,5cyclictest2874420-21irqstats12:00:173
257026999204,6cyclictest2674359-21sed08:45:243
2570269992013,5cyclictest2695807-21kworker/u8:0+flush-259:009:20:183
2570269992013,5cyclictest2580622-21latency_hist07:20:003
2570264992017,2cyclictest2732383-21kworker/u8:2+flush-259:009:45:341
2570259992014,5cyclictest2873602-21diskstats12:00:140
257026999194,4cyclictest2684451-21ntpq08:55:243
257026999193,6cyclictest2700048-21ldconfig09:10:223
2570269991913,4cyclictest3000860-21systemd-journal12:07:323
2570269991913,4cyclictest2769318-21latency_hist10:20:013
2570269991912,5cyclictest3000860-21systemd-journal12:15:003
2570269991912,5cyclictest3000860-21systemd-journal08:00:003
2570266991916,2cyclictest2758541-21kworker/u8:0+flush-259:010:30:312
2570266991916,2cyclictest2585850-21kworker/u8:1+events_unbound07:55:212
2570266991915,3cyclictest2810858-21kworker/u8:0+flush-259:012:14:592
2570266991915,3cyclictest2585850-21kworker/u8:1+events_unbound08:00:022
2570266991915,3cyclictest2554378-21kworker/u8:1+events_unbound07:15:002
2570264991915,3cyclictest2831987-21apt-get11:20:091
2570264991914,4cyclictest3000860-21systemd-journal10:15:011
2570264991914,4cyclictest2842348-21/usr/sbin/munin11:30:171
2570259991916,2cyclictest447041-21kworker/u8:0+flush-259:007:15:180
257025999191,17cyclictest492-21dbus-daemon09:20:000
257026999184,4cyclictest2590199-21ld07:25:243
257026999183,4cyclictest2893539-21ld12:15:233
2570269991812,4cyclictest3000860-21systemd-journal09:05:003
2570266991814,3cyclictest2643299-21apt-get08:20:022
2570266991812,4cyclictest2785157-21/usr/sbin/munin10:35:162
2570264991814,3cyclictest2758541-21kworker/u8:0+flush-259:010:15:261
2570259991815,2cyclictest2774536-21kworker/u8:1+flush-259:010:30:200
2570266991714,2cyclictest2737764-21kworker/u8:1+flush-259:010:00:182
2570266991713,3cyclictest2674104-21kworker/u8:3+flush-259:008:50:162
2570266991713,3cyclictest2585850-21kworker/u8:1+flush-259:008:17:332
2570264991711,4cyclictest2643808-21cat08:20:011
2570264991710,5cyclictest2790111-21latency_hist10:40:001
2570264991710,5cyclictest2596194-21latency_hist07:35:011
2570259991714,2cyclictest61450irq/134-enp0s29f1:tx-007:35:250
2570259991713,3cyclictest2888742-21gpgv12:15:000
2570259991713,3cyclictest2743185-21kworker/u8:3+events_unbound09:55:230
2570259991713,3cyclictest2743185-21kworker/u8:3+events_unbound09:55:220
257026999169,5cyclictest2873200-21kworker/u8:1+flush-259:012:35:003
257026999166,7cyclictest2643552-21wc08:20:013
257026999165,9cyclictest2857657-21latency_hist11:45:013
257026999164,9cyclictest2831730-21latency_hist11:20:013
257026999163,8cyclictest2794933-21users10:40:293
2570269991613,2cyclictest2821302-21latency_hist11:10:003
2570269991611,3cyclictest3000860-21systemd-journal10:46:333
2570269991611,3cyclictest3000063-21rs:main3
2570269991611,3cyclictest3000063-21rs:main3
2570266991614,1cyclictest2810858-21kworker/u8:0+events_unbound12:40:012
2570266991613,2cyclictest2701184-21kworker/u8:3+flush-259:009:15:222
2570266991613,2cyclictest2586487-21irqcore07:25:182
2570266991613,2cyclictest2585850-21kworker/u8:1+flush-259:007:50:182
2570264991613,2cyclictest447041-21kworker/u8:0+flush-259:007:15:161
2570264991613,2cyclictest2743185-21kworker/u8:3+flush-259:009:55:221
2570264991613,2cyclictest2743185-21kworker/u8:3+flush-259:009:55:211
2570264991611,4cyclictest3000860-21systemd-journal09:02:321
2570264991611,3cyclictest2742956-21latency_hist09:55:011
257025999169,5cyclictest2858076-21diskstats11:45:140
2570259991613,2cyclictest60650irq/133-enp0s29f1:rx-010:50:160
2570259991613,2cyclictest2810858-21kworker/u8:0+events_unbound11:20:010
2570259991613,2cyclictest2759168-21kworker/u8:4+flush-259:010:20:170
257026999158,5cyclictest2899580-21munin-run12:25:003
257026999157,6cyclictest2622902-21tr08:00:143
257026999156,8cyclictest2616453-21unixbench_multi07:50:223
257026999156,7cyclictest2659880-21perl08:35:173
257026999154,8cyclictest2733188-21irqcore09:45:213
257026999153,2cyclictest2785342-21diskstats10:35:163
2570269991512,2cyclictest2832407-21irqcore11:20:143
2570269991511,3cyclictest2664857-21apt-get08:40:163
2570269991510,3cyclictest2899840-21apt-get12:25:133
2570266991513,1cyclictest2835537-21kthreadcore11:20:162
2570266991512,2cyclictest2737728-21idleruntime-cro09:50:002
257026499159,4cyclictest2622789-21/usr/sbin/munin08:00:201
2570264991512,2cyclictest2711341-21kworker/u8:2+flush-259:009:29:591
2570264991511,3cyclictest2769195-21kworker/u8:2+flush-259:010:25:001
2570264991511,3cyclictest2653655-21unixbench_multi08:25:281
2570259991513,2cyclictest60650irq/133-enp0s29f1:rx-008:15:220
2570259991513,1cyclictest60650irq/133-enp0s29f1:rx-012:25:170
2570259991513,1cyclictest60650irq/133-enp0s29f1:rx-011:55:150
2570259991513,1cyclictest60650irq/133-enp0s29f1:rx-010:00:470
2570259991513,1cyclictest60650irq/133-enp0s29f1:rx-009:25:260
2570259991513,1cyclictest60650irq/133-enp0s29f1:rx-009:03:140
2570259991513,1cyclictest60650irq/133-enp0s29f1:rx-007:40:020
2570259991512,2cyclictest60650irq/133-enp0s29f1:rx-008:55:260
2570259991512,2cyclictest2606819-21apt-get07:45:160
2570259991511,2cyclictest2857659-21turbostat11:45:000
2570259991510,4cyclictest3000860-21systemd-journal11:02:310
257026999148,4cyclictest3000860-21systemd-journal07:43:323
257026999148,4cyclictest2805275-21awk10:50:263
257026999148,4cyclictest2716919-21idleruntime-cro09:29:593
257026999147,5cyclictest2758719-21latency_hist10:10:013
257026999147,5cyclictest2732491-21idleruntime-cro09:44:593
257026999146,5cyclictest2633564-21/usr/sbin/munin08:10:183
257026999145,7cyclictest2816190-21date11:05:013
257026999145,7cyclictest2693189-21cut09:05:203
257026999145,6cyclictest2717201-21chrt09:30:163
257026999145,6cyclictest2591288-21/usr/sbin/munin07:30:123
257026999143,2cyclictest2868435-21diskstats11:55:153
257026999143,2cyclictest2780140-21df10:30:173
257026999143,2cyclictest2723344-21irqstats09:35:193
257026999143,2cyclictest2649354-21diskstats08:25:173
257026999143,2cyclictest2570641-21df07:10:163
2570269991411,2cyclictest2837250-21cpuspeed_turbos11:25:173
257026699148,4cyclictest3000860-21systemd-journal11:05:002
257026699141,5cyclictest2883528-21awk12:10:002
2570266991412,1cyclictest0-21swapper/211:05:012
2570266991411,2cyclictest2711341-21kworker/u8:2+flush-259:009:25:252
2570266991411,2cyclictest2580338-21kworker/u8:3+flush-259:008:05:152
2570266991411,2cyclictest2580338-21kworker/u8:3+flush-259:007:45:232
2570266991410,3cyclictest2858113-21fschecks_count11:45:142
2570266991410,3cyclictest2795669-21cut10:45:142
2570266991410,3cyclictest2773660-21ntpq10:20:222
257026499149,3cyclictest2654143-21kworker/u8:0+events_unbound08:35:211
257026499142,5cyclictest2695762-21munin-run09:10:001
2570264991410,3cyclictest2820895-21unixbench_singl11:05:271
2570264991410,2cyclictest2900094-21fschecks_time12:25:161
2570264991410,2cyclictest2810858-21kworker/u8:0+flush-259:011:45:171
257025999149,4cyclictest2863058-21/usr/sbin/munin11:50:150
257025999149,4cyclictest2622644-21cat08:00:010
257025999148,5cyclictest2732509-21latency_hist09:45:010
2570259991413,1cyclictest60650irq/133-enp0s29f1:rx-012:20:120
2570259991413,1cyclictest60650irq/133-enp0s29f1:rx-012:18:500
2570259991413,1cyclictest60650irq/133-enp0s29f1:rx-010:45:210
2570259991413,1cyclictest60650irq/133-enp0s29f1:rx-009:37:430
2570259991413,1cyclictest60650irq/133-enp0s29f1:rx-009:23:040
2570259991413,1cyclictest60650irq/133-enp0s29f1:rx-008:45:240
2570259991413,1cyclictest60650irq/133-enp0s29f1:rx-008:32:190
2570259991413,1cyclictest60650irq/133-enp0s29f1:rx-007:57:260
2570259991412,2cyclictest60650irq/133-enp0s29f1:rx-010:26:310
2570259991412,1cyclictest60650irq/133-enp0s29f1:rx-011:25:210
2570259991412,1cyclictest60650irq/133-enp0s29f1:rx-010:05:170
2570259991412,1cyclictest60650irq/133-enp0s29f1:rx-007:20:170
2570259991412,1cyclictest2909977-21turbostat12:35:000
2570259991411,2cyclictest60650irq/133-enp0s29f1:rx-010:45:010
2570259991411,2cyclictest60650irq/133-enp0s29f1:rx-008:51:190
2570259991411,2cyclictest2826049-21unixbench_singl11:10:250
257026999137,5cyclictest2759327-21grep10:10:143
257026999137,4cyclictest2858008-21grep11:45:123
257026999137,4cyclictest2821649-21tr11:10:143
257026999137,4cyclictest2809922-21tail10:55:223
257026999137,4cyclictest2701622-21irqcore09:15:153
257026999135,6cyclictest2776602-21grep10:25:183
257026999135,6cyclictest2738187-21grep09:50:173
257026999135,6cyclictest2738187-21grep09:50:163
257026999135,6cyclictest2607339-21cut07:45:213
257026999134,2cyclictest2644269-21fw_forwarded_lo08:20:183
257026999133,2cyclictest2847823-21fw_conntrack11:35:173
257026999133,2cyclictest2769868-21fw_conntrack10:20:163
257026999133,2cyclictest2655363-21irqstats08:30:173
257026999133,2cyclictest2628161-21diskstats08:05:143
257026999132,4cyclictest2596621-21diskstats07:35:193
2570266991310,2cyclictest503-21systemd-logind11:59:592
2570266991310,2cyclictest2695807-21kworker/u8:0+flush-259:009:50:242
2570266991310,2cyclictest2695807-21kworker/u8:0+flush-259:009:50:242
2570266991310,2cyclictest2580338-21kworker/u8:3+events_unbound07:30:252
2570266991310,2cyclictest2570255-21kworker/u8:2+flush-259:007:17:332
257026699130,4cyclictest2810900-21awk11:00:012
257026499139,3cyclictest2674104-21kworker/u8:3+flush-259:008:50:251
257026499139,3cyclictest2585850-21kworker/u8:1+flush-259:008:05:251
257026499139,3cyclictest2580338-21kworker/u8:3+flush-259:007:50:301
2570264991310,2cyclictest2805222-21unixbench_multi10:50:271
2570264991310,2cyclictest2796006-21irqcore10:45:161
2570264991310,2cyclictest2606571-21kworker/u8:2+flush-259:008:10:271
257025999139,3cyclictest2790131-21kworker/u8:1+events_unbound11:05:160
257025999139,3cyclictest2591289-21apt-get07:30:120
257025999139,2cyclictest2759168-21kworker/u8:4+flush-259:010:55:210
2570259991312,1cyclictest60650irq/133-enp0s29f1:rx-008:26:350
2570259991312,1cyclictest60650irq/133-enp0s29f1:rx-008:09:350
2570259991311,1cyclictest60650irq/133-enp0s29f1:rx-010:15:210
2570259991311,1cyclictest60650irq/133-enp0s29f1:rx-009:50:160
2570259991311,1cyclictest60650irq/133-enp0s29f1:rx-009:50:150
257026999127,4cyclictest2842783-21sed11:30:183
257026999125,5cyclictest2748739-21/usr/sbin/munin10:00:163
257026699129,2cyclictest2790131-21kworker/u8:1+flush-259:010:55:002
257026699129,2cyclictest2745536-21kthreadcore09:55:242
257026699129,2cyclictest2745536-21kthreadcore09:55:232
257026699129,2cyclictest2722431-21cpuspeed_turbos09:35:152
257026699129,2cyclictest2669387-21unixbench_singl08:40:292
257026699127,4cyclictest2732478-21cat09:44:542
257026699126,4cyclictest2664583-21sh08:40:002
257026499129,2cyclictest2794369-21cut10:40:241
257026499129,2cyclictest0-21swapper/110:05:011
257026499129,2cyclictest0-21swapper/107:25:001
257026499128,3cyclictest2695807-21kworker/u8:0+flush-259:009:30:311
257026499128,3cyclictest2680200-21kworker/u8:1+flush-259:009:35:261
257026499127,3cyclictest2842082-21cat11:29:541
257025999129,2cyclictest2695807-21kworker/u8:0+flush-259:009:35:000
257025999128,3cyclictest2664610-21kworker/u8:2+events_unbound09:05:220
257025999127,3cyclictest3000860-21systemd-journal11:25:000
2570259991211,1cyclictest60650irq/133-enp0s29f1:rx-010:39:290
2570259991211,1cyclictest60650irq/133-enp0s29f1:rx-008:35:050
257026699119,1cyclictest2900412-21irqcore12:25:192
257026699118,2cyclictest2606571-21kworker/u8:2+flush-259:008:30:162
257026699118,2cyclictest2570250-21cyclictest11:35:272
257026699118,1cyclictest0-21swapper/211:15:172
257026699116,4cyclictest2759179-21/usr/sbin/munin10:10:162
257026699110,4cyclictest2701090-21cat09:15:002
257026499119,1cyclictest2570250-21cyclictest11:00:181
257026499118,2cyclictest2603936-21kthreadcore07:40:171
257026499115,4cyclictest2889377-21/usr/sbin/munin12:15:211
257025999118,2cyclictest60650irq/133-enp0s29f1:rx-009:12:080
257026699107,2cyclictest2889468-21cpuspeed_turbos12:15:142
257026699107,2cyclictest2606571-21kworker/u8:2+flush-259:008:30:012
257026699104,5cyclictest2853117-21grep11:40:192
257026499107,2cyclictest2873662-21fschecks_time12:00:151
257026499106,3cyclictest2680200-21kworker/u8:1+flush-259:009:40:181
25702669996,2cyclictest2710423-21ntp_kernel_pll_09:20:222
25702649995,3cyclictest2769195-21kworker/u8:2+flush-259:010:25:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional