You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 10:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Tue Dec 02, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2847211994440,3cyclictest3059450-21kworker/u8:1+flush-259:023:15:000
2847211994137,3cyclictest3102482-21kworker/u8:0+flush-259:023:35:180
2847211994136,3cyclictest2698871-21systemd-journal22:24:570
2847221993429,4cyclictest3138948-21kworker/u8:4+flush-259:000:25:162
2847226993329,3cyclictest2925963-21kworker/u8:3+flush-259:020:30:003
2847226993328,4cyclictest3059450-21kworker/u8:1+flush-259:022:40:183
2847219993327,4cyclictest2698871-21systemd-journal23:15:141
2847219993226,4cyclictest2698871-21systemd-journal20:00:001
2847226993127,3cyclictest3040128-21kworker/u8:1+flush-259:022:20:003
2847211993125,4cyclictest2698871-21systemd-journal20:55:140
2847226993024,4cyclictest2867933-21/usr/sbin/munin19:30:163
2847211993027,2cyclictest3144375-21apt-get23:55:140
2847219992926,2cyclictest2831905-21kworker/u8:1+flush-259:019:55:001
2847211992923,4cyclictest3029166-21latency_hist22:04:590
2847226992825,2cyclictest3059450-21kworker/u8:1+flush-259:000:05:003
2847226992824,3cyclictest2894585-21apt-get19:55:143
2847226992822,4cyclictest2852284-21latency_hist19:15:003
2847226992822,4cyclictest2698871-21systemd-journal19:39:593
2847226992822,4cyclictest2698871-21systemd-journal00:19:593
2847219992822,4cyclictest3144116-21latency_hist23:55:011
2847226992619,5cyclictest3076423-21cp22:50:003
2847219992620,4cyclictest3034745-21latency_hist22:10:001
2847219992620,4cyclictest3000063-21rs:main1
2847211992623,2cyclictest2925963-21kworker/u8:3+flush-259:020:40:000
2847211992620,4cyclictest3122587-21ntpq23:30:220
2847211992620,4cyclictest3018831-21latency_hist21:55:000
284721999250,24cyclictest1-21systemd00:05:001
2847219992418,4cyclictest2961042-21ntpq20:55:231
2847219992417,5cyclictest2940392-21ntpq20:35:221
2847211992421,2cyclictest3128670-21kworker/u8:3+flush-259:023:50:010
2847211992421,2cyclictest1-21systemd23:30:000
2847226992319,3cyclictest3070238-21kworker/u8:0+flush-259:022:55:003
2847219992317,4cyclictest3076938-21cat22:50:011
2847211992318,4cyclictest2698871-21systemd-journal22:55:000
2847226992218,3cyclictest3059450-21kworker/u8:1+flush-259:023:10:003
2847226992216,4cyclictest3133807-21latency_hist23:45:013
2847219992217,3cyclictest2883666-21cat19:45:001
2847219992217,3cyclictest2698871-21systemd-journal22:40:141
2847211992219,2cyclictest2847104-21kworker/u8:2+events_unbound21:35:160
2847211992218,3cyclictest3000061-21in:imuxsock00:20:000
2847211992217,4cyclictest2909421-21ntpq20:05:270
2847226992117,3cyclictest2998433-21apt-get21:35:123
2847221992119,1cyclictest3092185-21sh23:05:002
2847221992118,2cyclictest3070238-21kworker/u8:0+flush-259:022:55:192
2847221992115,4cyclictest3117917-21awk23:30:002
2847219992117,3cyclictest2925963-21kworker/u8:3+flush-259:020:50:201
2847211992118,2cyclictest3059450-21kworker/u8:1+flush-259:000:00:170
2847226992016,3cyclictest2925963-21kworker/u8:3+flush-259:020:55:263
2847226992014,5cyclictest2966231-21ntpq21:00:223
2847221992017,2cyclictest3065668-21systemd-run22:40:002
2847221992017,2cyclictest0-21swapper/219:35:002
2847221992016,3cyclictest3102482-21kworker/u8:0+flush-259:000:00:012
2847219992017,2cyclictest3185698-21kworker/u8:2+flush-259:000:35:171
2847211992016,3cyclictest3059450-21kworker/u8:1+events_unbound22:35:180
2847211992015,3cyclictest2915421-21latency_hist20:15:000
2847211992014,4cyclictest2698871-21systemd-journal22:45:020
2847226991917,1cyclictest140-1kworker/3:1H+kblockd20:24:573
2847226991917,1cyclictest140-1kworker/3:1H+kblockd20:24:563
2847226991915,3cyclictest3034755-21kworker/u8:3+flush-259:022:10:243
2847226991915,3cyclictest2794166-21kworker/u8:0+flush-259:020:20:003
2847226991915,3cyclictest2794166-21kworker/u8:0+flush-259:020:19:593
2847226991915,3cyclictest2794166-21kworker/u8:0+flush-259:019:20:013
2847221991916,2cyclictest3128670-21kworker/u8:3+flush-259:023:45:002
2847221991916,2cyclictest2847104-21kworker/u8:2+flush-259:020:45:182
2847219991916,2cyclictest2888729-21kworker/u8:5+flush-259:020:05:211
2847219991916,2cyclictest2794166-21kworker/u8:0+flush-259:020:20:251
2847219991916,2cyclictest2794166-21kworker/u8:0+flush-259:020:20:241
2847219991915,3cyclictest2920565-21dump-pmu-power20:20:001
2847219991915,3cyclictest2920565-21dump-pmu-power20:20:001
2847219991915,2cyclictest3102482-21kworker/u8:0+flush-259:023:30:001
2847211991915,3cyclictest2967354-21kworker/u8:1+events_unbound21:20:130
2847226991816,1cyclictest0-21swapper/322:05:003
2847226991815,2cyclictest2847104-21kworker/u8:2+flush-259:020:05:003
2847221991814,3cyclictest2831905-21kworker/u8:1+flush-259:019:39:592
2847221991813,3cyclictest2972590-21apt-get21:10:112
2847221991812,4cyclictest3070766-21apt-get22:45:002
2847221991812,4cyclictest3019265-21diskstats21:55:162
284721999185,7cyclictest2992996-21munin-run21:30:011
2847219991814,3cyclictest2847104-21kworker/u8:2+events_unbound20:30:181
2847219991813,3cyclictest2857398-21sh19:20:001
2847211991816,1cyclictest2698871-21systemd-journal22:34:140
2847211991815,2cyclictest3059450-21kworker/u8:1+flush-259:023:20:170
2847211991815,2cyclictest2878701-21irqcore19:40:180
2847226991713,3cyclictest2794166-21kworker/u8:0+flush-259:019:45:003
2847226991712,4cyclictest2698871-21systemd-journal21:35:013
2847226991711,4cyclictest3065647-21latency_hist22:40:003
2847226991711,4cyclictest2977509-21munin-plugin-st21:15:003
2847221991713,3cyclictest3097841-21sshd23:10:142
2847221991713,3cyclictest2847104-21kworker/u8:2+flush-259:019:15:182
2847221991712,3cyclictest2988064-21kworker/u8:4+flush-259:021:25:202
2847221991711,4cyclictest2698871-21systemd-journal22:11:142
284722199170,10cyclictest3097329-21munin-run23:10:002
2847219991714,2cyclictest3165032-21kworker/u8:3+flush-259:000:25:001
2847219991714,2cyclictest3129938-21kthreadcore23:40:181
2847219991714,2cyclictest2847104-21kworker/u8:2+flush-259:021:00:191
2847219991713,3cyclictest3154740-21/usr/sbin/munin00:05:201
2847219991713,3cyclictest3118587-21/usr/sbin/munin23:30:241
2847219991713,3cyclictest3092206-21kworker/u8:2+flush-259:023:05:001
2847219991712,4cyclictest2862806-21/usr/sbin/munin19:25:171
2847211991714,2cyclictest61450irq/134-enp0s29f1:tx-021:05:160
2847211991713,3cyclictest3018877-21kworker/u8:0+flush-259:022:05:160
2847211991713,3cyclictest2925963-21kworker/u8:3+flush-259:020:35:000
2847226991613,2cyclictest3070238-21kworker/u8:0+flush-259:022:55:173
2847226991613,2cyclictest2893899-21cut19:55:003
2847226991612,3cyclictest3102482-21kworker/u8:0+flush-259:023:50:233
2847226991612,3cyclictest2888729-21kworker/u8:5+flush-259:020:05:183
2847226991611,4cyclictest492-21dbus-daemon23:20:013
2847226991610,4cyclictest2698871-21systemd-journal00:30:003
2847221991613,2cyclictest3018877-21kworker/u8:0+flush-259:022:25:002
2847221991612,3cyclictest2967354-21kworker/u8:1+flush-259:021:40:272
2847221991612,3cyclictest2831905-21kworker/u8:1+flush-259:019:50:172
2847221991611,3cyclictest2910525-21/usr/sbin/munin20:10:162
2847221991610,4cyclictest3029824-21/usr/sbin/munin22:05:142
2847219991614,2cyclictest3059450-21kworker/u8:1+flush-259:023:45:231
2847219991613,2cyclictest2882225-21ntp_kernel_err19:40:221
2847219991612,3cyclictest2977733-21kworker/u8:0+events_unbound21:40:181
2847211991614,1cyclictest60650irq/133-enp0s29f1:rx-022:10:170
2847211991613,2cyclictest2872850-21kworker/u8:3+events_unbound19:35:000
2847211991612,3cyclictest2941506-21apt-get20:40:120
2847226991511,3cyclictest2862120-21unixbench_singl19:20:253
2847221991511,3cyclictest3102482-21kworker/u8:0+flush-259:023:20:002
2847219991512,2cyclictest3138948-21kworker/u8:4+flush-259:000:15:151
2847219991512,2cyclictest0-21swapper/122:19:591
2847219991511,3cyclictest3034755-21kworker/u8:3+flush-259:022:10:241
2847219991511,3cyclictest2698871-21systemd-journal19:12:341
2847219991510,4cyclictest3112784-21latency_hist23:25:011
2847211991514,1cyclictest60650irq/133-enp0s29f1:rx-022:58:340
2847211991514,1cyclictest60650irq/133-enp0s29f1:rx-020:04:220
2847211991513,1cyclictest60650irq/133-enp0s29f1:rx-023:05:270
2847211991513,1cyclictest60650irq/133-enp0s29f1:rx-022:29:350
2847211991513,1cyclictest60650irq/133-enp0s29f1:rx-021:30:250
2847211991513,1cyclictest60650irq/133-enp0s29f1:rx-021:16:510
2847211991513,1cyclictest60650irq/133-enp0s29f1:rx-020:20:150
2847211991513,1cyclictest60650irq/133-enp0s29f1:rx-020:20:140
2847211991513,1cyclictest60650irq/133-enp0s29f1:rx-019:16:030
2847211991513,1cyclictest60650irq/133-enp0s29f1:rx-000:20:230
2847211991513,1cyclictest60650irq/133-enp0s29f1:rx-000:05:450
2847211991512,2cyclictest60650irq/133-enp0s29f1:rx-000:10:190
2847211991512,2cyclictest2888252-21unixbench_multi19:45:240
2847211991512,2cyclictest2847104-21kworker/u8:2+flush-259:021:25:190
2847211991511,3cyclictest3059450-21kworker/u8:1+events_unbound23:40:170
2847211991511,3cyclictest3008734-21kworker/u8:4+flush-259:021:45:190
2847211991511,3cyclictest2920549-21idleruntime-cro20:20:000
2847211991511,3cyclictest2920549-21idleruntime-cro20:20:000
284721199150,13cyclictest3107633-21turbostat23:20:000
284722699149,4cyclictest2698871-21systemd-journal23:59:553
284722699149,4cyclictest2698871-21systemd-journal21:25:013
284722699149,3cyclictest3034755-21kworker/u8:3+flush-259:022:30:183
2847226991412,1cyclictest3102482-21kworker/u8:0-ext4-rsv-conversion23:30:013
2847226991411,2cyclictest3135085-21kthreadcore23:45:173
2847226991411,2cyclictest2935747-21kworker/u8:0+flush-259:020:45:263
2847221991411,2cyclictest3160922-21kthreadcore00:10:192
2847221991411,2cyclictest2888298-21unixbench_singl19:45:242
2847221991410,3cyclictest2925963-21kworker/u8:3+flush-259:020:35:252
284722199140,12cyclictest492-21dbus-daemon22:55:002
2847219991412,1cyclictest2977733-21kworker/u8:0+events_unbound21:30:191
2847219991411,2cyclictest3160329-21irqcore00:10:181
2847219991411,2cyclictest3003328-21sh21:40:011
2847219991411,2cyclictest2951797-21kworker/u8:4+flush-259:021:15:011
2847219991410,3cyclictest2847104-21kworker/u8:2+flush-259:021:15:241
284721199142,11cyclictest3070797-21munin-run22:45:000
2847211991413,1cyclictest60650irq/133-enp0s29f1:rx-021:43:530
2847211991413,1cyclictest60650irq/133-enp0s29f1:rx-019:10:240
2847211991413,1cyclictest60650irq/133-enp0s29f1:rx-000:30:090
2847211991411,2cyclictest60650irq/133-enp0s29f1:rx-021:04:110
2847211991411,2cyclictest60650irq/133-enp0s29f1:rx-020:52:020
2847211991410,3cyclictest2967354-21kworker/u8:1+flush-259:021:55:200
2847211991410,3cyclictest2794166-21kworker/u8:0+flush-259:019:20:170
2847226991310,2cyclictest3102482-21kworker/u8:0+flush-259:023:35:263
2847226991310,2cyclictest3055368-21systemd-run22:30:003
2847226991310,2cyclictest2888693-21idleruntime-cro19:50:003
284722699130,12cyclictest1-21systemd20:35:003
284722199139,3cyclictest2888729-21kworker/u8:5+flush-259:020:05:222
284722199138,4cyclictest2698871-21systemd-journal19:44:122
284722199137,4cyclictest2971439-21ntpq21:05:222
2847221991311,1cyclictest0-21swapper/221:25:012
2847221991311,1cyclictest0-21swapper/200:40:002
284721999139,3cyclictest3024012-21latency_hist22:00:001
284721999139,3cyclictest2935747-21kworker/u8:0+flush-259:020:40:171
284721999137,4cyclictest3123934-21diskstats23:35:171
2847219991310,2cyclictest0-21swapper/122:45:211
284721199139,3cyclictest2899627-21gpgv19:59:590
2847211991312,1cyclictest60650irq/133-enp0s29f1:rx-023:50:260
2847211991312,1cyclictest60650irq/133-enp0s29f1:rx-022:15:160
2847211991311,2cyclictest60650irq/133-enp0s29f1:rx-019:28:020
2847211991311,1cyclictest2878004-21idleruntime-cro19:40:000
2847211991311,1cyclictest0-21swapper/020:29:590
2847211991310,2cyclictest61450irq/134-enp0s29f1:tx-019:50:170
2847211991310,2cyclictest60650irq/133-enp0s29f1:rx-023:00:260
2847211991310,2cyclictest3138948-21kworker/u8:4+flush-259:000:25:240
284722699129,2cyclictest3186149-21irqcore00:35:183
284722699129,2cyclictest3045722-21irqcore22:20:173
284722699129,2cyclictest3029979-21cpuspeed_turbos22:05:133
284722699129,2cyclictest2978179-21sed21:15:163
284722699127,3cyclictest3024005-21awk22:00:003
2847226991210,1cyclictest0-21swapper/320:40:233
284722199129,2cyclictest3134169-21cpuspeed_turbos23:45:132
284722199129,2cyclictest3076642-21latency_hist22:50:012
284722199129,2cyclictest3018877-21kworker/u8:0+flush-259:022:33:192
284722199128,3cyclictest2847104-21kworker/u8:2+flush-259:021:49:192
284722199128,3cyclictest2847104-21kworker/u8:2+flush-259:021:35:012
284722199125,4cyclictest492-21dbus-daemon20:25:002
284722199125,4cyclictest492-21dbus-daemon20:25:002
2847221991210,1cyclictest0-21swapper/200:10:002
284721999129,2cyclictest3059450-21kworker/u8:1+flush-259:023:55:181
284721999129,2cyclictest3050883-21sed22:25:181
284721999129,2cyclictest2927020-21kthreadcore20:25:201
284721999129,2cyclictest2868217-21fschecks_time19:30:151
284721999127,3cyclictest2698871-21systemd-journal20:01:121
2847219991210,1cyclictest3093468-21kthreadcore23:05:201
284721199129,3cyclictest2951797-21kworker/u8:4+events_unbound21:10:160
284722699118,2cyclictest3097650-21cpuspeed_turbos23:10:133
284722699115,4cyclictest3163940-21ntpq00:10:233
284722199119,1cyclictest0-21swapper/222:05:002
284722199119,1cyclictest0-21swapper/221:35:132
284722199117,3cyclictest3014145-21fschecks_count21:50:152
284722199117,2cyclictest2936057-21awk20:34:592
284722199115,4cyclictest2926290-21fw_forwarded_lo20:25:172
284722199115,4cyclictest2847571-21diskstats19:10:142
284722199110,9cyclictest2899944-21cat20:00:012
284721999117,3cyclictest3034755-21kworker/u8:3+flush-259:022:30:191
284722699108,1cyclictest0-21swapper/321:30:003
284722199107,2cyclictest2916098-21irqcore20:15:182
284722199107,2cyclictest2916098-21irqcore20:15:172
284722199106,2cyclictest2946419-21latency_hist20:45:002
284721999108,1cyclictest0-21swapper/122:20:181
284721999106,3cyclictest3059450-21kworker/u8:1+flush-259:023:10:161
284721999100,4cyclictest3185636-21cat00:35:011
28472219986,1cyclictest0-21swapper/223:35:152
28472219984,2cyclictest3122633-21ntpq23:30:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional