You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-22 - 14:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Dec 22, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2956969993730,4cyclictest2978025-21latency_hist19:30:003
2956958993530,3cyclictest3294923-21apt-get00:35:130
2956964993428,4cyclictest3284328-21latency_hist00:25:002
2956964993428,4cyclictest3284328-21latency_hist00:25:002
2956969993326,5cyclictest3071699-21latency_hist21:00:013
2956964993329,3cyclictest2951375-21kworker/u8:3+flush-259:019:30:172
2956962993327,4cyclictest3112984-21latency_hist21:40:001
2956969993227,3cyclictest3143559-21kworker/u8:1+flush-259:023:40:183
2956958993217,2cyclictest3253420-21cron23:55:000
2956958993125,4cyclictest3102958-21apt-get21:30:130
2956962993025,4cyclictest3050629-21gpgv20:40:011
2956964992924,3cyclictest3206711-21apt-key23:10:012
2956962992924,3cyclictest3143559-21kworker/u8:1+flush-259:023:15:261
2956962992923,4cyclictest3253667-21/usr/sbin/munin23:55:141
2956964992823,4cyclictest509-21rs:main2
2956958992822,3cyclictest252-21systemd-journal22:50:020
2956969992721,4cyclictest2998773-21latency_hist19:50:013
2956962992721,4cyclictest252-21systemd-journal19:39:591
2956962992620,4cyclictest3206710-21gpgv23:10:011
2956958992621,3cyclictest3077148-21apt-get21:05:140
2956958992620,4cyclictest3137859-21ntpq22:00:220
2956969992520,3cyclictest3056112-21latency_hist20:45:013
2956964992520,4cyclictest3024490-21kworker/u8:1+flush-259:020:25:172
2956958992520,3cyclictest2993839-21apt-get19:45:120
2956969992421,2cyclictest3107525-21kworker/u8:4+events_unbound22:34:563
2956964992421,2cyclictest507-21in:imuxsock21:00:012
2956964992419,4cyclictest3092433-21kworker/u8:3+flush-259:021:30:182
2956964992418,4cyclictest3288656-21ntpq00:25:242
2956964992418,4cyclictest3288656-21ntpq00:25:242
2956962992420,3cyclictest2967200-21kworker/u8:1+flush-259:019:30:261
2956958992419,3cyclictest3066771-21apt-get20:55:130
2956962992319,3cyclictest3179988-21gpgv22:45:001
2956962992319,3cyclictest3092433-21kworker/u8:3+events_unbound21:20:221
2956962992318,3cyclictest3279623-21diskstats00:20:161
2956962992318,3cyclictest3279623-21diskstats00:20:151
2956958992318,4cyclictest3051205-21apt-get20:40:120
2956958992318,3cyclictest3149330-21/usr/sbin/munin22:15:180
2956969992218,3cyclictest3263970-21kworker/u8:4+flush-259:000:25:263
2956969992218,3cyclictest3263970-21kworker/u8:4+flush-259:000:25:263
2956964992218,3cyclictest3024490-21kworker/u8:1+flush-259:020:40:012
2956964992217,4cyclictest3107525-21kworker/u8:4+flush-259:022:15:252
2956964992216,4cyclictest252-21systemd-journal21:49:082
2956962992219,2cyclictest3284356-21kworker/u8:0+flush-259:000:35:101
2956962992216,4cyclictest3102694-21latency_hist21:30:001
2956962992216,4cyclictest3040001-21kworker/u8:3+flush-259:021:00:261
2956958992217,3cyclictest3014542-21apt-get20:05:110
2956964992115,4cyclictest3092433-21kworker/u8:3+flush-259:021:50:452
2956962992116,3cyclictest3243355-21kworker/u8:3+flush-259:000:35:001
2956962992115,4cyclictest3082311-21/usr/sbin/munin21:10:161
2956962992114,5cyclictest252-21systemd-journal19:14:591
2956969992018,1cyclictest3076497-21kworker/u8:1+events_unbound21:25:013
2956964992017,2cyclictest3185559-21cut22:50:012
2956964992014,4cyclictest3212177-21latency_hist23:15:012
2956964992014,4cyclictest3158549-21ntpq22:20:232
2956964992014,4cyclictest3035012-21latency_hist20:25:012
2956962992015,3cyclictest3201356-21kworker/u8:2+flush-259:000:15:191
2956962992015,3cyclictest3056343-21kworker/u8:4+flush-259:020:55:181
2956962992015,3cyclictest2967200-21kworker/u8:1+flush-259:020:00:011
2956962992013,4cyclictest3237948-21latency_hist23:40:001
2956969991916,2cyclictest3279877-21irqcore00:20:183
2956969991916,2cyclictest3279877-21irqcore00:20:173
2956969991916,2cyclictest3179717-21kworker/u8:0+events_unbound22:50:373
2956969991910,1cyclictest191rcu_preempt23:15:013
2956964991916,2cyclictest3179717-21kworker/u8:0+flush-259:023:00:172
2956962991912,5cyclictest252-21systemd-journal00:15:001
2956962991911,5cyclictest3174827-21latency_hist22:40:011
2956958991915,3cyclictest3232542-21kworker/u8:3+flush-259:023:35:260
2956958991913,4cyclictest3060436-21ntpq20:45:230
2956969991813,4cyclictest3274323-21apt-get00:15:143
2956964991811,4cyclictest2967633-21base6419:20:012
2956962991814,3cyclictest3185758-21kworker/u8:3+flush-259:023:00:141
2956962991812,4cyclictest3243355-21kworker/u8:3+flush-259:023:45:191
2956962991812,4cyclictest3138763-21latency_hist22:05:011
2956958991814,3cyclictest60350irq/132-enp0s29f1:rx-021:55:200
2956958991814,3cyclictest3050501-21idleruntime-cro20:40:010
2956958991814,3cyclictest3024922-21apt-get20:15:130
2956958991813,3cyclictest3253693-21apt-get23:55:120
295696999173,3cyclictest50-21ksoftirqd/320:15:013
295696999173,2cyclictest491ktimers/320:50:223
2956969991714,2cyclictest3243355-21kworker/u8:3+events_unbound00:09:533
2956969991713,3cyclictest3243355-21kworker/u8:3+events_unbound00:31:473
2956964991711,4cyclictest2956879-21latency_hist19:10:012
2956962991714,2cyclictest3252468-21needreboot23:50:211
2956962991714,2cyclictest2956953-21cyclictest22:35:001
2956962991713,3cyclictest3118085-21kworker/u8:1+events_unbound21:50:001
2956962991713,3cyclictest3097791-21kworker/u8:2+flush-259:021:30:221
2956962991713,3cyclictest3050513-21apt-get20:40:031
2956962991712,3cyclictest2967200-21kworker/u8:1+flush-259:019:25:211
2956962991711,4cyclictest2967263-21apt-get19:20:001
2956958991713,3cyclictest3237711-21kworker/u8:0+flush-259:000:05:000
2956958991713,3cyclictest3217385-21kworker/u8:0+flush-259:023:25:160
2956969991613,2cyclictest3263970-21kworker/u8:4+events_unbound00:13:173
2956969991613,2cyclictest3056343-21kworker/u8:4+events_unbound21:06:323
2956969991612,3cyclictest3029821-21idleruntime-cro20:20:003
295696999161,13cyclictest1-21systemd20:35:013
2956969991610,1cyclictest191rcu_preempt19:31:233
295696499161,5cyclictest3243131-21munin-run23:45:012
2956964991613,2cyclictest3179966-21idleruntime-cro22:45:002
2956964991612,3cyclictest3201356-21kworker/u8:2+flush-259:023:33:492
2956964991612,3cyclictest3107525-21kworker/u8:4+flush-259:021:55:162
2956964991611,3cyclictest3258525-21cron00:00:002
2956964991611,3cyclictest3179717-21kworker/u8:0+flush-259:022:55:262
2956964991610,4cyclictest3201356-21kworker/u8:2+flush-259:023:20:162
2956964991610,4cyclictest3097554-21latency_hist21:25:002
2956964991610,4cyclictest3045340-21latency_hist20:35:002
295696299168,6cyclictest252-21systemd-journal22:20:001
2956962991613,2cyclictest3191141-21cron22:55:001
2956962991613,2cyclictest3143900-21sh22:10:001
2956962991613,2cyclictest3143900-21sh22:09:591
2956962991610,4cyclictest2956915-21kworker/u8:0+flush-259:019:15:191
2956958991614,1cyclictest60350irq/132-enp0s29f1:rx-021:10:260
2956958991614,1cyclictest60350irq/132-enp0s29f1:rx-021:00:270
2956958991614,1cyclictest60350irq/132-enp0s29f1:rx-020:11:030
2956958991614,1cyclictest60350irq/132-enp0s29f1:rx-019:35:160
2956958991614,1cyclictest0-21swapper/019:20:010
2956958991613,2cyclictest60350irq/132-enp0s29f1:rx-023:01:160
2956958991613,2cyclictest60350irq/132-enp0s29f1:rx-022:55:160
2956958991613,2cyclictest60350irq/132-enp0s29f1:rx-022:30:160
2956958991613,2cyclictest60350irq/132-enp0s29f1:rx-022:10:150
2956958991613,2cyclictest60350irq/132-enp0s29f1:rx-020:50:260
2956958991613,2cyclictest60350irq/132-enp0s29f1:rx-020:24:440
2956958991613,2cyclictest60350irq/132-enp0s29f1:rx-020:01:120
2956958991613,2cyclictest60350irq/132-enp0s29f1:rx-000:10:180
2956958991613,2cyclictest2956953-21cyclictest22:25:260
2956958991612,3cyclictest2967200-21kworker/u8:1+flush-259:019:55:260
2956958991610,5cyclictest3206596-21apt-get23:10:000
295696999158,1cyclictest191rcu_preempt23:29:013
295696999154,7cyclictest0-21swapper/321:50:133
295696999153,2cyclictest491ktimers/320:05:223
295696999152,3cyclictest50-21ksoftirqd/323:45:243
295696999152,3cyclictest50-21ksoftirqd/322:15:163
2956969991514,1cyclictest3243355-21kworker/u8:3+events_unbound00:38:363
2956969991513,2cyclictest3030072-21kworker/u8:2+events_unbound20:24:373
2956969991512,2cyclictest3237711-21kworker/u8:0+events_unbound23:37:503
2956969991512,2cyclictest3092433-21kworker/u8:3+events_unbound21:48:443
2956969991512,2cyclictest3030072-21kworker/u8:2+events_unbound20:45:093
2956969991512,2cyclictest2956915-21kworker/u8:0+events_unbound19:37:263
2956969991511,3cyclictest2962402-21apt-get19:15:143
2956964991512,2cyclictest3001353-21kthreadcore19:50:172
2956964991512,2cyclictest2993569-21sh19:45:002
2956964991510,4cyclictest3237711-21kworker/u8:0+flush-259:023:50:252
295696299159,4cyclictest3061570-21apt-get20:50:131
2956962991511,3cyclictest3217385-21kworker/u8:0+events_unbound23:25:221
2956962991511,2cyclictest0-21swapper/122:55:211
2956962991510,3cyclictest3263730-21latency_hist00:05:001
2956958991513,1cyclictest60350irq/132-enp0s29f1:rx-023:20:230
2956958991513,1cyclictest60350irq/132-enp0s29f1:rx-021:25:300
2956958991513,1cyclictest60350irq/132-enp0s29f1:rx-021:15:200
2956958991513,1cyclictest60350irq/132-enp0s29f1:rx-019:50:590
2956958991513,1cyclictest60350irq/132-enp0s29f1:rx-000:15:260
2956958991512,2cyclictest60350irq/132-enp0s29f1:rx-023:10:160
2956958991512,2cyclictest60350irq/132-enp0s29f1:rx-022:05:140
2956958991512,2cyclictest60350irq/132-enp0s29f1:rx-022:05:140
2956958991512,2cyclictest60350irq/132-enp0s29f1:rx-021:42:440
2956958991512,2cyclictest60350irq/132-enp0s29f1:rx-021:35:180
2956958991512,2cyclictest60350irq/132-enp0s29f1:rx-019:26:280
2956958991512,2cyclictest60350irq/132-enp0s29f1:rx-019:13:460
295696999149,1cyclictest191rcu_preempt23:08:223
295696999146,4cyclictest0-21swapper/322:25:013
295696999145,3cyclictest0-21swapper/323:50:263
295696999145,1cyclictest0-21swapper/322:45:223
295696999143,7cyclictest50-21ksoftirqd/323:15:173
295696999143,3cyclictest50-21ksoftirqd/321:35:173
295696999143,2cyclictest50-21ksoftirqd/322:00:213
295696999142,4cyclictest50-21ksoftirqd/319:40:213
295696999142,3cyclictest50-21ksoftirqd/321:55:103
295696999142,2cyclictest50-21ksoftirqd/321:15:213
295696999142,2cyclictest3179056-21ntpq22:40:213
295696999141,4cyclictest50-21ksoftirqd/322:55:193
2956969991413,0cyclictest0-21swapper/320:30:013
2956969991411,2cyclictest3261124-21kthreadcore00:00:193
2956969991411,2cyclictest3217385-21kworker/u8:0+events_unbound23:22:403
2956969991411,2cyclictest3092433-21kworker/u8:3+events_unbound21:30:353
2956969991411,2cyclictest3024490-21kworker/u8:1+events_unbound20:38:063
295696499149,4cyclictest3123740-21kworker/u8:0+flush-259:022:25:262
295696499148,4cyclictest3274298-21/usr/sbin/munin00:15:172
295696499148,4cyclictest3269137-21/usr/sbin/munin00:10:162
295696499143,6cyclictest3076885-21munin-run21:05:012
2956964991411,1cyclictest3051973-21irqcore20:40:182
2956964991410,3cyclictest3243355-21kworker/u8:3+flush-259:000:00:272
2956964991410,3cyclictest3056343-21kworker/u8:4+flush-259:020:48:202
295696299148,4cyclictest252-21systemd-journal20:34:071
2956962991410,3cyclictest2956915-21kworker/u8:0+flush-259:020:15:241
2956958991413,1cyclictest61150irq/133-enp0s29f1:tx-023:15:230
2956958991413,1cyclictest60350irq/132-enp0s29f1:rx-023:48:570
2956958991413,1cyclictest60350irq/132-enp0s29f1:rx-022:24:350
2956958991412,1cyclictest60350irq/132-enp0s29f1:rx-020:30:190
2956958991412,1cyclictest60350irq/132-enp0s29f1:rx-019:40:190
2956958991412,1cyclictest60350irq/132-enp0s29f1:rx-000:20:260
2956958991412,1cyclictest60350irq/132-enp0s29f1:rx-000:20:260
2956958991412,1cyclictest0-21swapper/022:50:000
2956958991411,2cyclictest60350irq/132-enp0s29f1:rx-023:40:090
2956958991411,2cyclictest60350irq/132-enp0s29f1:rx-021:51:030
2956958991411,2cyclictest60350irq/132-enp0s29f1:rx-000:25:260
2956958991411,2cyclictest60350irq/132-enp0s29f1:rx-000:25:260
2956958991411,2cyclictest60350irq/132-enp0s29f1:rx-000:07:280
2956958991410,3cyclictest2967200-21kworker/u8:1+flush-259:019:30:260
2956958991410,1cyclictest60350irq/132-enp0s29f1:rx-022:40:260
295696999135,1cyclictest0-21swapper/319:20:213
295696999132,7cyclictest50-21ksoftirqd/321:00:213
295696999132,3cyclictest50-21ksoftirqd/322:05:193
295696999132,3cyclictest50-21ksoftirqd/322:05:193
295696999132,2cyclictest50-21ksoftirqd/319:50:163
295696999131,8cyclictest50-21ksoftirqd/323:55:123
295696999131,4cyclictest50-21ksoftirqd/323:00:133
295696999131,3cyclictest50-21ksoftirqd/322:20:133
2956969991310,2cyclictest3169751-21kworker/u8:2+events_unbound22:39:063
2956969991310,2cyclictest3092433-21kworker/u8:3+events_unbound21:24:043
2956969991310,2cyclictest3030072-21kworker/u8:2+events_unbound21:13:233
295696999130,7cyclictest50-21ksoftirqd/322:10:223
295696499139,3cyclictest3056343-21kworker/u8:4+flush-259:020:53:112
295696499139,2cyclictest0-21swapper/222:35:182
295696499131,3cyclictest3029829-21awk20:19:592
2956964991310,2cyclictest3294628-21idleruntime-cro00:35:002
2956964991310,2cyclictest3009772-21sed20:00:142
2956964991310,2cyclictest3006513-21kthreadcore19:55:202
295696299139,3cyclictest3056343-21kworker/u8:4+flush-259:020:50:011
295696299137,4cyclictest513-21gdbus21:45:001
295696299136,5cyclictest509-21rs:main1
2956962991310,2cyclictest3228331-21sed23:30:161
2956962991310,2cyclictest3181673-21kthreadcore22:45:171
2956962991310,1cyclictest0-21swapper/119:50:171
295695899136,5cyclictest3173988-21ntpq22:35:240
2956958991311,2cyclictest60350irq/132-enp0s29f1:rx-021:21:340
2956958991311,1cyclictest60350irq/132-enp0s29f1:rx-020:25:240
2956958991310,2cyclictest60350irq/132-enp0s29f1:rx-023:32:000
2956958991310,2cyclictest60350irq/132-enp0s29f1:rx-000:30:160
295696999127,1cyclictest191rcu_preempt20:00:143
295696999125,1cyclictest191rcu_preempt21:40:193
295696999121,3cyclictest50-21ksoftirqd/319:55:143
295696999120,2cyclictest3231926-21ntpq23:30:213
295696499129,1cyclictest0-21swapper/220:10:212
295696499129,1cyclictest0-21swapper/219:39:592
295696499128,3cyclictest3092433-21kworker/u8:3+flush-259:022:05:262
295696499128,3cyclictest3092433-21kworker/u8:3+flush-259:022:05:262
295696499127,3cyclictest3112822-21kworker/u8:0+flush-259:021:40:192
295696499125,5cyclictest3097796-21/usr/sbin/munin21:25:162
295696499121,3cyclictest0-21swapper/223:30:002
2956964991210,1cyclictest0-21swapper/219:25:142
295696299129,2cyclictest3128877-21fschecks_count21:55:131
295696299129,2cyclictest252-21systemd-journal19:45:001
295696299129,1cyclictest0-21swapper/120:25:171
295696299121,8cyclictest3243123-21latency_hist23:45:001
295696499118,2cyclictest2968980-21tr19:20:192
295696299118,2cyclictest2993905-21cpuspeed_turbos19:45:141
295696299118,1cyclictest0-21swapper/120:05:001
295696299117,2cyclictest3092414-21awk21:20:001
295696299115,4cyclictest3269069-21cat00:10:011
295696299115,4cyclictest3164351-21sshd22:28:061
295696299115,4cyclictest252-21systemd-journal21:51:071
295696299107,2cyclictest3263970-21kworker/u8:4+events_unbound00:25:231
295696299107,2cyclictest3263970-21kworker/u8:4+events_unbound00:25:221
295696299107,1cyclictest0-21swapper/123:20:211
295696299107,1cyclictest0-21swapper/120:05:161
29569649995,2cyclictest3144540-21if_enp0s30f422:10:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional