You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 09:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot4s.osadl.org (updated Fri Nov 28, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2710436994441,2cyclictest2933465-21gpgv22:45:013
2710419994236,4cyclictest2928189-21latency_hist22:40:000
2710432994033,5cyclictest2782771-21latency_hist20:20:002
2710436993328,4cyclictest2902265-21/usr/sbin/munin22:15:143
2710436993227,3cyclictest2767539-21/usr/sbin/munin20:05:163
2710427993227,3cyclictest2824352-21kworker/u8:3+flush-259:021:30:011
2710436993127,3cyclictest2886321-21kworker/u8:0+flush-259:022:05:263
2710427993128,2cyclictest2860408-21kworker/u8:4+flush-259:021:35:261
2710427993024,4cyclictest2814077-21latency_hist20:50:001
2710436992926,2cyclictest2672599-21kworker/u8:0+flush-259:019:15:003
2710427992825,2cyclictest2995987-21kworker/u8:0+flush-259:000:00:201
2710427992822,5cyclictest2746923-21kworker/u8:4+flush-259:019:50:301
2710427992724,2cyclictest2803795-21kworker/u8:2+events_unbound21:05:181
2710419992722,3cyclictest2870702-21latency_hist21:45:010
2710436992623,2cyclictest3032210-21kworker/u8:3+events_unbound00:35:003
2710427992623,2cyclictest2834469-21kworker/u8:0+flush-259:021:10:241
2710427992619,5cyclictest2698871-21systemd-journal20:30:001
2710436992521,3cyclictest2896898-21kworker/u8:1+flush-259:023:05:013
2710427992522,3cyclictest2902256-21kworker/u8:3+flush-259:022:35:001
2710427992521,3cyclictest2939126-21kworker/u8:4+flush-259:022:50:221
2710427992521,3cyclictest2939126-21kworker/u8:4+flush-259:022:50:211
2710427992521,2cyclictest0-21swapper/119:35:001
2710427992520,3cyclictest2757224-21kworker/u8:1+flush-259:020:00:151
2710427992519,4cyclictest3006203-21latency_hist23:55:011
2710419992519,5cyclictest2890876-21ntpq22:00:210
2710436992421,2cyclictest2824352-21kworker/u8:3+flush-259:021:20:013
2710432992417,5cyclictest3037135-21latency_hist00:25:012
2710427992421,2cyclictest3032210-21kworker/u8:3+events_unbound00:30:261
2710427992421,2cyclictest2855221-21kworker/u8:2+flush-259:021:45:161
2710419992419,3cyclictest2698871-21systemd-journal21:28:350
2710419992418,4cyclictest2912770-21diskstats22:25:140
2710436992317,5cyclictest2881408-21latency_hist21:55:013
2710432992319,3cyclictest2933346-21latency_hist22:45:002
2710427992320,2cyclictest2964932-21kworker/u8:1+events_unbound23:35:351
2710427992319,3cyclictest3032210-21kworker/u8:3+flush-259:000:25:201
2710427992319,3cyclictest3000061-21in:imuxsock19:25:001
2710427992318,4cyclictest641903-21kworker/u8:2+events_unbound19:35:171
2710427992317,4cyclictest2949243-21latency_hist23:00:011
2710436992217,4cyclictest2896898-21kworker/u8:1+events_unbound22:55:183
2710432992217,4cyclictest2859463-21ntpq21:30:222
2710432992216,4cyclictest2698871-21systemd-journal21:51:352
2710427992219,2cyclictest2818435-21kworker/u8:1+events_unbound20:57:431
2710427992218,3cyclictest2939126-21kworker/u8:4+flush-259:023:05:181
2710427992218,3cyclictest2896898-21kworker/u8:1+flush-259:022:45:001
2710427992215,5cyclictest2819216-21latency_hist20:55:001
2710419992217,4cyclictest2757234-21/usr/sbin/munin19:55:140
2710436992117,2cyclictest0-21swapper/321:50:003
2710436992116,3cyclictest2792660-21kworker/u8:4+flush-259:020:35:003
2710436992115,4cyclictest2917481-21apt-get22:30:003
271043699210,20cyclictest1-21systemd22:25:003
2710432992117,3cyclictest641903-21kworker/u8:2+flush-259:019:35:182
2710427992118,2cyclictest2964932-21kworker/u8:1+flush-259:023:20:161
2710427992118,2cyclictest2964932-21kworker/u8:1+flush-259:023:15:191
2710427992118,2cyclictest2886321-21kworker/u8:0+flush-259:022:20:221
2710427992118,2cyclictest2855221-21kworker/u8:2+events_unbound22:10:231
2710427992117,3cyclictest3032210-21kworker/u8:3+events_unbound00:23:371
2710427992117,3cyclictest2902256-21kworker/u8:3+flush-259:022:25:201
2710427992117,2cyclictest2850063-21munin-plugin-st21:25:001
2710419992116,3cyclictest2996205-21apt-get23:45:130
2710419992115,4cyclictest2757003-21latency_hist19:55:000
2710436992018,1cyclictest2844916-21kworker/u8:1+events_unbound21:35:013
2710436992017,2cyclictest2980166-21systemd-run23:30:003
2710432992016,3cyclictest2875872-21gpgv21:50:002
2710432992016,3cyclictest2803795-21kworker/u8:2+flush-259:021:00:232
2710432992014,4cyclictest2933944-21cat22:45:002
2710432992014,4cyclictest2698871-21systemd-journal23:43:152
2710427992016,3cyclictest2896898-21kworker/u8:1+flush-259:022:45:231
2710427992016,3cyclictest2767322-21kworker/u8:2+flush-259:020:20:271
2710427992014,4cyclictest2698871-21systemd-journal00:38:361
2710427992012,6cyclictest2798393-21cp20:35:001
2710436991915,3cyclictest2855183-21idleruntime-cro21:30:003
2710436991912,5cyclictest2698871-21systemd-journal21:40:003
2710432991916,2cyclictest3047418-21sh00:34:592
2710432991916,2cyclictest2939126-21kworker/u8:4+flush-259:023:05:172
2710432991915,3cyclictest2757224-21kworker/u8:1+flush-259:020:10:172
2710427991916,2cyclictest2939126-21kworker/u8:4+events_unbound23:02:361
2710427991916,2cyclictest2803795-21kworker/u8:2+events_unbound20:40:241
2710427991916,1cyclictest0-21swapper/123:30:011
2710427991911,6cyclictest3021696-21latency_hist00:10:021
2710419991915,2cyclictest0-21swapper/021:20:000
2710436991814,3cyclictest2964932-21kworker/u8:1+flush-259:000:20:213
271043299184,8cyclictest2756977-21cron19:55:002
2710432991814,3cyclictest2902290-21apt-get22:15:112
2710432991812,4cyclictest2839753-21latency_hist21:15:012
2710427991815,2cyclictest2980141-21kworker/u8:2+events_unbound23:33:121
2710427991815,2cyclictest2824352-21kworker/u8:3+events_unbound21:15:231
2710427991815,2cyclictest2792660-21kworker/u8:4+flush-259:020:35:211
2710427991815,2cyclictest2792660-21kworker/u8:4+flush-259:020:35:211
2710427991814,3cyclictest2964932-21kworker/u8:1+flush-259:000:10:231
2710427991814,3cyclictest2757224-21kworker/u8:1+events_unbound20:12:351
2710427991814,3cyclictest2757224-21kworker/u8:1+events_unbound19:55:231
2710427991813,3cyclictest2757040-21kworker/u8:0+flush-259:020:05:201
2710419991815,2cyclictest2860408-21kworker/u8:4+flush-259:021:45:250
2710419991814,3cyclictest2964932-21kworker/u8:1+flush-259:023:35:160
2710419991814,3cyclictest2824600-21apt-get21:00:170
2710419991813,3cyclictest2959782-21apt-get23:10:130
2710419991812,4cyclictest2943167-21ntpq22:50:210
2710419991812,4cyclictest2943167-21ntpq22:50:210
2710436991713,3cyclictest2757040-21kworker/u8:0+flush-259:020:45:173
2710432991713,3cyclictest2964932-21kworker/u8:1+flush-259:023:20:252
2710427991714,2cyclictest2855221-21kworker/u8:2+flush-259:022:00:241
2710427991714,2cyclictest2757040-21kworker/u8:0+flush-259:020:15:251
2710427991713,3cyclictest2855221-21kworker/u8:2+flush-259:021:55:161
2710419991714,2cyclictest2798208-21sh20:35:000
2710419991713,3cyclictest2896898-21kworker/u8:1+flush-259:023:00:010
2710419991713,3cyclictest2896898-21kworker/u8:1+flush-259:022:40:250
2710419991712,4cyclictest3016747-21/usr/sbin/munin00:05:190
2710419991711,4cyclictest2980809-21apt-get23:30:110
2710419991710,5cyclictest2798204-21apt-get20:35:010
2710419991710,5cyclictest2798204-21apt-get20:35:010
2710436991613,2cyclictest2767299-21cron20:05:003
2710436991612,3cyclictest2917481-21apt-get22:30:023
2710436991612,3cyclictest2824352-21kworker/u8:3+flush-259:021:15:003
2710436991612,3cyclictest2777806-21kworker/u8:3+flush-259:020:20:003
2710432991613,2cyclictest2995987-21kworker/u8:0+flush-259:000:15:182
2710432991612,3cyclictest2819450-21/usr/sbin/munin20:55:152
2710432991610,4cyclictest641903-21kworker/u8:2+flush-259:019:25:172
2710427991613,2cyclictest641903-21kworker/u8:2+events_unbound19:16:341
2710427991613,2cyclictest2896898-21kworker/u8:1+events_unbound22:35:261
2710419991613,2cyclictest60650irq/133-enp0s29f1:rx-021:50:190
2710419991611,4cyclictest2698871-21systemd-journal23:16:590
2710419991610,4cyclictest2937962-21ntpq22:45:220
2710436991512,2cyclictest0-21swapper/323:35:013
271043299159,5cyclictest2714880-21ntpq19:10:212
271043299159,4cyclictest2698871-21systemd-journal21:36:352
2710432991512,2cyclictest3017793-21kthreadcore00:05:172
2710432991511,3cyclictest2721240-21apt-get19:20:152
2710432991511,2cyclictest2995987-21kworker/u8:0+flush-259:023:45:182
271042799159,4cyclictest2907174-21latency_hist22:20:001
2710427991512,2cyclictest641903-21kworker/u8:2+flush-259:019:45:231
2710427991512,2cyclictest2860408-21kworker/u8:4+events_unbound21:44:381
2710427991512,2cyclictest2855221-21kworker/u8:2+events_unbound21:30:181
2710427991511,3cyclictest3011353-21idleruntime-cro00:00:001
2710427991510,4cyclictest2891978-21/usr/sbin/munin22:05:171
271041999159,5cyclictest2907594-21diskstats22:20:140
2710419991513,1cyclictest60650irq/133-enp0s29f1:rx-023:25:200
2710419991513,1cyclictest60650irq/133-enp0s29f1:rx-019:35:140
2710419991513,1cyclictest60650irq/133-enp0s29f1:rx-000:35:050
2710419991513,1cyclictest0-21swapper/021:40:000
2710419991512,2cyclictest60650irq/133-enp0s29f1:rx-021:55:140
2710419991512,2cyclictest60650irq/133-enp0s29f1:rx-020:13:420
2710419991512,2cyclictest60650irq/133-enp0s29f1:rx-020:05:140
2710419991512,2cyclictest60650irq/133-enp0s29f1:rx-000:15:170
2710419991510,4cyclictest2886321-21kworker/u8:0+flush-259:022:14:590
2710436991411,2cyclictest3017183-21irqcore00:05:153
2710436991410,3cyclictest3029387-21kthreadcore00:15:203
2710432991411,2cyclictest2975669-21irqcore23:25:182
2710432991411,2cyclictest2855221-21kworker/u8:2+flush-259:021:55:182
2710432991410,3cyclictest3032210-21kworker/u8:3+flush-259:000:25:272
2710427991411,2cyclictest2964932-21kworker/u8:1+events_unbound23:43:361
2710427991411,2cyclictest2855221-21kworker/u8:2+flush-259:021:50:271
2710419991413,1cyclictest60650irq/133-enp0s29f1:rx-023:00:230
2710419991413,1cyclictest60650irq/133-enp0s29f1:rx-019:31:010
2710419991413,1cyclictest60650irq/133-enp0s29f1:rx-000:14:130
2710419991412,2cyclictest60650irq/133-enp0s29f1:rx-021:08:460
2710419991412,2cyclictest60650irq/133-enp0s29f1:rx-000:31:170
2710419991412,1cyclictest60650irq/133-enp0s29f1:rx-023:40:250
2710419991412,1cyclictest60650irq/133-enp0s29f1:rx-021:20:270
2710419991412,1cyclictest60650irq/133-enp0s29f1:rx-020:15:170
2710419991412,1cyclictest60650irq/133-enp0s29f1:rx-000:25:270
2710419991411,2cyclictest60650irq/133-enp0s29f1:rx-023:55:140
2710419991411,2cyclictest60650irq/133-enp0s29f1:rx-021:10:130
2710419991411,2cyclictest2892202-21fschecks_count22:05:180
2710419991410,3cyclictest2741314-21kworker/u8:1+flush-259:019:40:240
271043699138,4cyclictest2720079-21ntpq19:15:223
271043699138,3cyclictest2741769-21/usr/sbin/munin19:40:143
271043699137,4cyclictest2698871-21systemd-journal20:35:023
271043699137,4cyclictest2698871-21systemd-journal20:35:013
2710436991310,2cyclictest3025853-21needreboot00:10:203
2710436991310,2cyclictest2832008-21kthreadcore21:05:213
271043299139,3cyclictest2767631-21cpuspeed_turbos20:05:142
271043299137,4cyclictest2698871-21systemd-journal20:55:002
271042799139,3cyclictest2964932-21kworker/u8:1+events_unbound00:17:351
271042799139,2cyclictest0-21swapper/123:45:191
271042799138,4cyclictest2741794-21apt-get19:40:111
271042799137,5cyclictest2710735-21cat19:10:011
271041999138,1cyclictest60650irq/133-enp0s29f1:rx-020:50:170
271041999137,4cyclictest2917952-21cp22:30:000
271041999137,4cyclictest2698871-21systemd-journal00:05:010
2710419991312,1cyclictest60650irq/133-enp0s29f1:rx-019:20:190
2710419991311,2cyclictest60650irq/133-enp0s29f1:rx-021:33:100
2710419991311,2cyclictest60650irq/133-enp0s29f1:rx-019:28:370
2710419991311,2cyclictest60650irq/133-enp0s29f1:rx-019:10:300
2710419991311,1cyclictest60650irq/133-enp0s29f1:rx-020:45:140
2710419991311,1cyclictest60650irq/133-enp0s29f1:rx-020:40:260
2710419991310,2cyclictest60650irq/133-enp0s29f1:rx-023:51:590
2710419991310,2cyclictest60650irq/133-enp0s29f1:rx-020:00:130
271043699129,2cyclictest2970308-21fschecks_count23:20:153
271043699129,2cyclictest2825615-21kthreadcore21:00:233
271043699126,4cyclictest1-21systemd20:25:003
271043699125,5cyclictest516-21gdbus00:00:013
271043699120,10cyclictest3016535-21munin-run00:05:003
271043299129,2cyclictest3011605-21kworker/u8:2+flush-259:000:05:002
271043299129,2cyclictest2809258-21cpuspeed_turbos20:45:132
271043299129,2cyclictest2747020-21cpuspeed_turbos19:45:112
271043299129,2cyclictest2741923-21cpuspeed_turbos19:40:132
271043299129,1cyclictest0-21swapper/223:55:002
271043299128,3cyclictest2959479-21kworker/u8:0+flush-259:023:10:252
271043299121,2cyclictest2720969-21awk19:20:002
271042799129,2cyclictest641903-21kworker/u8:2+events_unbound19:25:151
271042799129,2cyclictest2818435-21kworker/u8:1+flush-259:021:00:261
271041999129,2cyclictest60650irq/133-enp0s29f1:rx-023:10:000
271041999126,4cyclictest2788123-21munin-node20:25:130
2710419991211,1cyclictest60650irq/133-enp0s29f1:rx-020:20:320
2710419991210,2cyclictest60650irq/133-enp0s29f1:rx-022:15:200
2710419991210,2cyclictest60650irq/133-enp0s29f1:rx-000:20:460
2710419991210,1cyclictest60650irq/133-enp0s29f1:rx-019:47:520
271043699118,2cyclictest2749250-21kthreadcore19:45:173
271043699117,3cyclictest2964932-21kworker/u8:1+flush-259:023:40:183
271043699116,4cyclictest641903-21kworker/u8:2+flush-259:019:30:013
271043699115,4cyclictest2698871-21systemd-journal19:21:343
271043299119,1cyclictest2986081-21tr23:35:122
271043299118,2cyclictest2912675-21cpuspeed_turbos22:25:142
271043299118,2cyclictest2761691-21unixbench_multi19:55:252
271043299118,2cyclictest0-21swapper/220:35:002
271043299117,3cyclictest2824352-21kworker/u8:3+flush-259:021:20:252
271043299117,3cyclictest2787406-21unixbench_multi20:20:272
271041999119,2cyclictest60650irq/133-enp0s29f1:rx-019:16:110
271043699108,1cyclictest0-21swapper/321:55:133
271043699106,3cyclictest2818435-21kworker/u8:1+flush-259:020:56:063
271043299108,1cyclictest0-21swapper/221:25:212
271043299105,3cyclictest516-21gdbus23:20:012
27104369995,3cyclictest3047649-21kworker/u8:4+events_unbound00:35:173
27104329982,4cyclictest2833866-21/usr/sbin/munin21:05:272
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional