You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-16 - 15:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot4s.osadl.org (updated Mon Jun 16, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2424106994135,4cyclictest2614569-21ntpq22:10:230
2424117993629,4cyclictest2501799-21latency_hist20:25:013
2424106993632,3cyclictest2559075-21apt-get21:20:120
2424113993530,3cyclictest2678285-21kworker/u8:1+flush-259:023:15:202
2424110993530,3cyclictest2668026-21/usr/sbin/munin23:05:221
2424113993428,4cyclictest2672897-21latency_hist23:10:012
2424117993328,3cyclictest2620791-21gpgv22:20:003
2424117993328,3cyclictest2574518-21/usr/sbin/munin21:35:253
2424106993329,3cyclictest2470600-21gpgv19:55:000
2424117993225,4cyclictest2750949-21latency_hist00:25:013
2424113993228,3cyclictest2698845-21kworker/u8:4+flush-259:023:45:302
2424110993226,4cyclictest2429418-21/usr/sbin/munin19:15:161
2424113993026,3cyclictest2573263-21kworker/u8:3+events_unbound21:35:202
2424106993025,4cyclictest2693764-21/usr/sbin/munin23:30:190
2424106993025,3cyclictest2642329-21/usr/sbin/munin22:40:200
2424110992925,3cyclictest2444667-21kworker/u8:2+flush-259:019:45:001
2424117992824,3cyclictest2429199-21kworker/u8:1+flush-259:019:20:003
2424113992821,5cyclictest2657510-21latency_hist22:55:012
2424113992723,3cyclictest3754036-21systemd-journal22:07:342
2424113992721,4cyclictest2756049-21awk00:29:592
2424110992723,2cyclictest3754036-21systemd-journal00:33:401
2424117992621,4cyclictest2600028-21kworker/u8:1+flush-259:022:35:003
2424117992619,5cyclictest2766421-21latency_hist00:40:013
2424117992618,6cyclictest2573368-21ntpq21:30:213
2424113992622,3cyclictest2563996-21kworker/u8:0+flush-259:021:55:002
2424110992620,4cyclictest2476333-21apt-get20:00:121
2424110992619,5cyclictest2500942-21ntpq20:20:221
2424106992620,4cyclictest2481635-21diskstats20:05:150
2424117992521,3cyclictest2652389-21kworker/u8:2+events_unbound23:05:253
2424117992520,3cyclictest2600028-21kworker/u8:1+flush-259:022:35:183
2424117992520,3cyclictest2496667-21kworker/u8:1+flush-259:021:15:173
2424117992421,2cyclictest2563996-21kworker/u8:0+events_unbound22:50:233
2424117992420,3cyclictest2573263-21kworker/u8:3+events_unbound22:00:243
2424117992420,3cyclictest2573263-21kworker/u8:3+events_unbound22:00:233
2424117992420,2cyclictest2366202-21kworker/u8:0+flush-259:019:40:173
2424117992419,3cyclictest2713807-21kworker/u8:0+events_unbound23:50:193
2424117992418,4cyclictest2434608-21apt-get19:20:123
2424117992416,6cyclictest2667784-21latency_hist23:05:003
2424113992421,2cyclictest2527922-21kworker/u8:2+flush-259:021:20:002
2424113992420,3cyclictest2414125-21kworker/u8:3+events_unbound19:30:202
2424113992419,4cyclictest2642114-21kworker/u8:3+flush-259:022:55:292
2424110992417,5cyclictest2656646-21ntpq22:50:291
2424110992417,5cyclictest2588883-21ntpq21:45:221
2424117992319,3cyclictest2725032-21kworker/u8:2+flush-259:000:10:213
2424117992319,3cyclictest2725032-21kworker/u8:2+flush-259:000:10:213
2424117992319,3cyclictest2678285-21kworker/u8:1+flush-259:023:15:233
2424117992319,3cyclictest2652389-21kworker/u8:2+flush-259:023:35:003
2424117992319,3cyclictest2527922-21kworker/u8:2+events_unbound21:00:263
2424117992319,3cyclictest2414125-21kworker/u8:3+events_unbound20:05:023
2424113992318,4cyclictest2692690-21ntpq23:25:282
2424110992318,3cyclictest2698845-21kworker/u8:4+flush-259:023:35:251
2424117992218,3cyclictest2642114-21kworker/u8:3+events_unbound22:40:233
2424117992218,3cyclictest2600028-21kworker/u8:1+events_unbound22:05:233
2424117992217,3cyclictest2698845-21kworker/u8:4+flush-259:023:40:213
2424117992217,3cyclictest2366202-21kworker/u8:0+events_unbound19:35:183
2424117992216,4cyclictest2527922-21kworker/u8:2+events_unbound21:50:213
2424117992215,5cyclictest2444650-21latency_hist19:30:003
2424113992219,2cyclictest2573263-21kworker/u8:3+flush-259:022:00:192
2424113992219,2cyclictest2573263-21kworker/u8:3+flush-259:022:00:182
2424110992219,2cyclictest2496667-21kworker/u8:1+flush-259:020:35:001
2424110992218,3cyclictest2745416-21gpgv00:20:001
2424106992218,3cyclictest2512311-21kworker/u8:0+flush-259:021:05:260
2424117992115,4cyclictest2731232-21kworker/u8:1+events_unbound00:30:193
2424113992118,2cyclictest2502032-21kworker/u8:4+flush-259:020:25:182
2424113992114,5cyclictest2637160-21cp22:35:022
2424110992117,3cyclictest2652389-21kworker/u8:2+flush-259:023:12:141
2424106992116,4cyclictest2449803-21latency_hist19:35:000
2424106992114,5cyclictest2702987-21ntpq23:35:290
2424106992114,5cyclictest2506968-21latency_hist20:30:010
2424117992016,3cyclictest2573263-21kworker/u8:3+events_unbound22:20:243
2424117992015,4cyclictest2563996-21kworker/u8:0+flush-259:021:40:253
2424117992015,3cyclictest2652389-21kworker/u8:2+flush-259:023:39:593
2424117992014,4cyclictest3754036-21systemd-journal23:29:333
2424117992013,5cyclictest2713197-21ntpq23:45:273
2424113992014,4cyclictest2714631-21cp23:50:002
2424110992016,2cyclictest0-21swapper/120:20:001
2424106992017,2cyclictest2527911-21idleruntime-cro20:50:000
2424106992017,2cyclictest0-21swapper/022:25:010
2424117991916,2cyclictest2719682-21kworker/u8:3+events_unbound00:05:163
2424117991916,2cyclictest2465525-21kworker/u8:4+flush-259:019:50:173
2424117991915,3cyclictest2522475-21gpgv20:45:003
2424117991915,3cyclictest2512311-21kworker/u8:0+events_unbound20:50:183
2424117991915,3cyclictest2444667-21kworker/u8:2+events_unbound20:15:243
2424117991914,4cyclictest2563996-21kworker/u8:0+flush-259:021:45:153
2424113991917,1cyclictest0-21swapper/223:39:592
2424113991913,4cyclictest2486794-21diskstats20:10:142
2424113991912,5cyclictest3754036-21systemd-journal19:15:002
2424110991915,2cyclictest2440815-21kthreadcore19:25:191
2424110991914,3cyclictest2735379-21apt-get00:10:161
2424110991914,3cyclictest2735379-21apt-get00:10:151
2424110991912,5cyclictest2619768-21ntpq22:15:181
2424110991912,5cyclictest2538183-21latency_hist21:00:011
2424106991916,2cyclictest2481198-21idleruntime-cro20:05:000
2424106991914,4cyclictest2564214-21/usr/sbin/munin21:25:240
2424106991913,5cyclictest2600493-21diskstats22:00:150
2424106991913,5cyclictest2600493-21diskstats22:00:150
2424117991814,3cyclictest2414125-21kworker/u8:3+flush-259:019:55:203
2424113991815,2cyclictest2766431-21turbostat00:40:002
2424113991815,2cyclictest2527922-21kworker/u8:2+flush-259:021:10:012
2424113991815,2cyclictest2496667-21kworker/u8:1+flush-259:020:45:232
2424113991814,2cyclictest2563996-21kworker/u8:0+flush-259:021:35:002
242411099187,9cyclictest2678297-21/usr/sbin/munin23:15:241
2424110991814,3cyclictest3000061-21in:imuxsock21:19:591
2424110991814,3cyclictest2444910-21apt-get19:30:131
242411099180,3cyclictest2647258-21latency_hist22:45:001
242411099180,16cyclictest2465302-21latency_hist19:50:001
242410699188,1cyclictest2761267-21turbostat00:35:000
2424106991816,1cyclictest2725032-21kworker/u8:2+flush-259:000:10:300
2424106991816,1cyclictest2725032-21kworker/u8:2+flush-259:000:10:300
2424106991815,2cyclictest503-21systemd-logind22:00:010
2424106991815,2cyclictest2652389-21kworker/u8:2+flush-259:023:20:260
2424106991815,2cyclictest2652389-21kworker/u8:2+flush-259:023:05:270
2424106991815,2cyclictest2522369-21cut20:45:010
2424106991814,3cyclictest2573263-21kworker/u8:3+flush-259:022:15:220
242410699180,16cyclictest3000063-21rs:main0
2424117991714,2cyclictest2512311-21kworker/u8:0+flush-259:021:05:263
2424117991713,3cyclictest2678285-21kworker/u8:1+events_unbound23:55:283
2424117991713,3cyclictest2652389-21kworker/u8:2+flush-259:023:25:013
2424117991712,3cyclictest2366202-21kworker/u8:0+flush-259:019:10:153
2424113991714,2cyclictest2414125-21kworker/u8:3+flush-259:019:25:182
2424113991714,2cyclictest0-21swapper/219:20:002
2424113991713,2cyclictest2698610-21sh23:35:002
242411099171,2cyclictest2532210-21ntpq20:50:231
2424110991711,4cyclictest2761536-21apt-get00:35:161
2424110991710,5cyclictest3754036-21systemd-journal23:25:001
2424110991710,5cyclictest3754036-21systemd-journal22:40:011
242411099170,5cyclictest2594936-21munin-run21:55:001
2424106991713,3cyclictest2414125-21kworker/u8:3+flush-259:019:20:170
2424106991712,4cyclictest2443777-21ntpq19:25:230
2424106991712,3cyclictest2512508-21diskstats20:35:160
2424117991612,3cyclictest2527922-21kworker/u8:2+flush-259:021:57:043
2424117991612,3cyclictest2496667-21kworker/u8:1+events_unbound21:20:243
2424117991612,3cyclictest2414125-21kworker/u8:3+events_unbound20:00:183
2424113991612,3cyclictest2636554-21gpgv22:35:002
242411099163,4cyclictest2610394-21cat22:10:001
242411099163,3cyclictest2705016-21cut23:40:221
242411099162,1cyclictest2662156-21grep22:55:291
242411099161,4cyclictest2615495-21cut22:15:011
2424110991612,3cyclictest2750041-21ntp_kernel_err00:20:251
2424110991611,4cyclictest2730216-21apt-get00:05:151
2424110991610,4cyclictest2652389-21kworker/u8:2+flush-259:023:00:291
242411099160,5cyclictest3754036-21systemd-journal23:55:001
242411099160,5cyclictest2636535-21dump-pmu-power22:35:001
242411099160,4cyclictest2714217-21latency_hist23:50:001
242411099160,4cyclictest2522380-21http20:45:001
242411099160,4cyclictest2434715-21tr19:20:141
242411099160,3cyclictest2574291-21cut21:34:591
242411099160,3cyclictest2454447-21unixbench-2d19:35:261
2424106991614,1cyclictest60650irq/133-enp0s29f1:rx-019:45:260
2424106991614,1cyclictest60650irq/133-enp0s29f1:rx-000:20:290
2424106991613,2cyclictest60650irq/133-enp0s29f1:rx-023:49:080
2424106991613,2cyclictest60650irq/133-enp0s29f1:rx-022:45:250
2424106991613,2cyclictest60650irq/133-enp0s29f1:rx-019:39:550
2424106991611,4cyclictest2708942-21latency_hist23:45:010
2424106991610,5cyclictest2459236-21ntpq19:40:220
242411799158,5cyclictest2465302-21latency_hist19:50:003
2424117991512,2cyclictest2502032-21kworker/u8:4+flush-259:020:30:213
2424117991511,3cyclictest2713807-21kworker/u8:0+events_unbound00:25:293
2424117991511,3cyclictest2600028-21kworker/u8:1+flush-259:022:30:013
2424117991510,4cyclictest2512311-21kworker/u8:0+events_unbound20:45:293
242411399155,6cyclictest2439540-21munin-plugin-st19:25:002
2424113991512,2cyclictest2414125-21kworker/u8:3+flush-259:020:05:012
2424113991511,2cyclictest0-21swapper/220:45:012
242411099155,9cyclictest0-21swapper/121:40:191
242411099152,2cyclictest0-21swapper/120:05:191
242411099151,4cyclictest2751533-21/usr/sbin/munin00:25:211
242411099151,3cyclictest2544050-21irqcore21:05:191
242411099151,3cyclictest2490538-21needreboot20:10:191
242411099151,2cyclictest2538584-21cut21:00:151
242411099151,2cyclictest2475141-21ntpq19:55:201
242411099151,1cyclictest2506069-21ntpq20:25:231
2424110991510,4cyclictest2620678-21apt-get22:20:011
2424110991510,4cyclictest2573263-21kworker/u8:3+flush-259:021:35:181
242411099150,5cyclictest2651502-21ld22:45:241
242411099150,5cyclictest2604815-21sed22:00:271
242411099150,5cyclictest2604815-21sed22:00:261
242411099150,4cyclictest2688787-21df_inode23:25:211
242411099150,4cyclictest2568165-21needreboot21:25:211
242411099150,4cyclictest2563967-21cron21:25:001
242411099150,4cyclictest2527047-21ntpq20:45:281
242411099150,4cyclictest2465972-21tr19:50:161
242411099150,4cyclictest2426600-21ps19:10:201
242411099150,3cyclictest2694201-21sed23:30:221
242411099150,3cyclictest2626909-21grep22:25:171
242411099150,3cyclictest2595286-21tr21:55:121
242411099150,2cyclictest2516667-21timerwakeupswit20:35:261
2424106991513,1cyclictest60650irq/133-enp0s29f1:rx-023:50:200
2424106991513,1cyclictest60650irq/133-enp0s29f1:rx-023:25:030
2424106991513,1cyclictest60650irq/133-enp0s29f1:rx-021:47:040
2424106991512,2cyclictest60650irq/133-enp0s29f1:rx-021:30:350
2424106991512,2cyclictest60650irq/133-enp0s29f1:rx-020:15:340
2424106991511,3cyclictest2745505-21gpgv00:20:010
2424106991511,3cyclictest2563996-21kworker/u8:0+flush-259:022:25:170
242410699151,12cyclictest3754036-21systemd-journal19:15:330
2424106991511,2cyclictest0-21swapper/021:10:220
2424117991412,1cyclictest2652355-21turbostat22:50:003
2424117991410,3cyclictest2444667-21kworker/u8:2+events_unbound20:10:193
2424113991412,1cyclictest2473471-21kthreadcore19:55:182
2424113991411,2cyclictest0-21swapper/221:00:012
242411099141,3cyclictest2720281-21if_enp0s30f423:55:201
242411099141,2cyclictest2725465-21ps00:00:211
242411099140,3cyclictest2548892-21tr21:10:151
242410699149,4cyclictest2631617-21/usr/sbin/munin22:30:230
2424106991413,1cyclictest60650irq/133-enp0s29f1:rx-023:17:430
2424106991413,1cyclictest60650irq/133-enp0s29f1:rx-022:50:300
2424106991413,1cyclictest60650irq/133-enp0s29f1:rx-021:50:270
2424106991412,2cyclictest60650irq/133-enp0s29f1:rx-023:10:530
2424106991412,2cyclictest60650irq/133-enp0s29f1:rx-021:15:280
2424106991412,2cyclictest60650irq/133-enp0s29f1:rx-019:57:040
2424106991412,1cyclictest60650irq/133-enp0s29f1:rx-023:56:350
2424106991412,1cyclictest60650irq/133-enp0s29f1:rx-021:40:160
2424106991412,1cyclictest60650irq/133-enp0s29f1:rx-020:10:240
2424106991411,2cyclictest60650irq/133-enp0s29f1:rx-022:09:560
2424106991410,3cyclictest60650irq/133-enp0s29f1:rx-020:30:130
2424106991410,3cyclictest2366202-21kworker/u8:0+flush-259:019:10:200
242411799137,4cyclictest2731232-21kworker/u8:1+flush-259:000:19:593
242411399139,3cyclictest3000063-21rs:main2
242411399139,3cyclictest2414125-21kworker/u8:3+flush-259:019:45:262
242411399138,3cyclictest3000063-21rs:main2
2424113991310,2cyclictest2725032-21kworker/u8:2+flush-259:000:10:262
2424113991310,2cyclictest2725032-21kworker/u8:2+flush-259:000:10:252
242411099131,2cyclictest2424257-21latency_hist19:10:001
2424106991310,2cyclictest60650irq/133-enp0s29f1:rx-022:55:200
2424106991310,2cyclictest60650irq/133-enp0s29f1:rx-022:36:370
2424106991310,2cyclictest60650irq/133-enp0s29f1:rx-000:02:250
2424106991310,2cyclictest2662959-21cpuspeed_turbos23:00:170
242411799128,2cyclictest2569136-21latency_hist21:30:013
242411799127,3cyclictest2719682-21kworker/u8:3+events_unbound00:00:023
242411399129,2cyclictest0-21swapper/221:15:002
242411399127,4cyclictest2563996-21kworker/u8:0+flush-259:022:25:052
242411399124,3cyclictest2600070-21apt22:00:002
2424113991210,1cyclictest0-21swapper/219:55:002
242410699129,2cyclictest60650irq/133-enp0s29f1:rx-021:36:120
242410699128,3cyclictest2713807-21kworker/u8:0+flush-259:000:35:210
2424106991210,1cyclictest60650irq/133-enp0s29f1:rx-021:00:220
2424106991210,1cyclictest60650irq/133-enp0s29f1:rx-020:54:300
242411799118,2cyclictest2512311-21kworker/u8:0+flush-259:021:10:173
242411799117,3cyclictest2414125-21kworker/u8:3+flush-259:019:34:393
242411799115,4cyclictest2615485-21latency_hist22:15:013
242411399119,1cyclictest0-21swapper/219:40:142
242411399118,2cyclictest2730310-21cpuspeed_turbos00:05:152
242411399118,2cyclictest2673393-21fschecks_time23:10:172
242411399118,2cyclictest0-21swapper/200:20:002
242411399117,2cyclictest2542497-21ntpq21:00:242
2424106991110,1cyclictest60650irq/133-enp0s29f1:rx-020:55:220
242411399107,2cyclictest2564307-21cpuspeed_turbos21:25:152
242411399107,2cyclictest2491972-21fschecks_count20:15:152
242411399107,1cyclictest0-21swapper/223:55:212
242411399106,3cyclictest2746059-21/usr/sbin/munin00:20:272
242410699107,2cyclictest60650irq/133-enp0s29f1:rx-020:20:160
24241179997,1cyclictest0-21swapper/320:35:163
24241139997,1cyclictest0-21swapper/222:40:162
24241139995,3cyclictest2414125-21kworker/u8:3+flush-259:020:09:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional