You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-13 - 04:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot4s.osadl.org (updated Sat Sep 13, 2025 00:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3271170994841,5cyclictest3276787-21gpgv19:15:001
3271173994644,1cyclictest3395532-21fwupd21:06:313
3271173994335,5cyclictest3437572-21latency_hist21:50:013
3271172994238,3cyclictest3385575-21kworker/u8:2+events_unbound21:00:192
3271168994036,3cyclictest182790-21kworker/u8:0+flush-259:020:00:180
3271173993935,3cyclictest3526278-21kworker/u8:3+flush-259:023:15:263
3271168993630,4cyclictest3354286-21latency_hist20:30:010
3271172993531,3cyclictest3568295-21kworker/u8:0+flush-259:000:00:152
3271172993528,5cyclictest3473470-21ntpq22:20:212
3271172993429,3cyclictest3364624-21kworker/u8:3+events_unbound21:35:172
3271168993329,3cyclictest3364624-21kworker/u8:3+flush-259:021:30:160
3271168993326,5cyclictest3427262-21latency_hist21:40:010
3271172993226,4cyclictest3301969-21ntpq19:35:232
3271173993125,4cyclictest2698871-21systemd-journal20:30:003
3271168993125,4cyclictest3469414-21apt-get22:20:120
3271173993025,4cyclictest3562514-21kworker/u8:2+flush-259:023:50:003
3271173993025,4cyclictest3562514-21kworker/u8:2+flush-259:023:49:593
3271172993025,3cyclictest3458932-21kworker/u8:1+flush-259:023:10:162
3271168993024,4cyclictest3343139-21ntpq20:15:220
3271172992921,6cyclictest3589164-21latency_hist00:15:002
3271170992924,3cyclictest2698871-21systemd-journal21:38:281
3271173992823,3cyclictest3276575-21apt-get19:15:003
3271173992821,5cyclictest3453996-21diskstats22:05:153
3271172992823,4cyclictest3583602-21gpgv00:10:012
3271168992823,4cyclictest3385981-21diskstats21:00:160
3271168992823,4cyclictest3337931-21ntpq20:10:210
3271173992723,3cyclictest3364624-21kworker/u8:3+flush-259:021:35:233
3271173992720,5cyclictest3546634-21latency_hist23:35:013
3271172992621,3cyclictest3599682-21kworker/u8:1+events_unbound00:30:232
3271172992620,4cyclictest3270943-21latency_hist19:10:002
3271170992619,5cyclictest3583580-21latency_hist00:10:011
3271168992621,4cyclictest3603721-21ntpq00:25:190
3271168992621,4cyclictest3584238-21/usr/sbin/munin00:10:160
3271173992522,2cyclictest3385575-21kworker/u8:2+flush-259:021:55:003
3271173992520,4cyclictest2698871-21systemd-journal20:00:013
3271173992519,4cyclictest3453154-21apt-get22:05:003
3271173992519,4cyclictest3370250-21base6420:45:003
3271172992521,3cyclictest3448178-21kworker/u8:3+events_unbound22:56:302
3271172992521,3cyclictest182790-21kworker/u8:0+events_unbound20:00:282
3271168992521,3cyclictest3505656-21apt-get22:55:140
3271168992521,3cyclictest2698871-21systemd-journal19:36:290
3271173992419,3cyclictest3000063-21rs:main3
3271173992418,4cyclictest3354529-21/usr/sbin/munin20:30:163
3271173992418,3cyclictest3385575-21kworker/u8:2+flush-259:021:00:203
3271173992417,5cyclictest3599449-21latency_hist00:25:013
327117299249,10cyclictest3442748-21munin-run21:55:002
3271172992420,3cyclictest3260334-21kworker/u8:2+flush-259:019:20:002
3271172992418,4cyclictest3394941-21ntpq21:05:212
3271170992420,3cyclictest3448178-21kworker/u8:3+flush-259:022:30:001
3271173992318,3cyclictest3297913-21apt-get19:35:133
3271172992320,2cyclictest3448178-21kworker/u8:3+events_unbound22:00:132
3271172992319,3cyclictest3411823-21kworker/u8:1+flush-259:021:30:002
3271172992319,2cyclictest3584228-21kworker/u8:4+flush-259:000:20:242
3271172992316,5cyclictest3567205-21ntpq23:50:212
3271172992316,5cyclictest2698871-21systemd-journal20:10:012
3271170992319,3cyclictest2698871-21systemd-journal20:25:291
3271168992318,4cyclictest3504491-21ntpq22:50:180
3271168992318,3cyclictest2698871-21systemd-journal21:10:010
3271173992217,4cyclictest3260334-21kworker/u8:2+flush-259:019:35:003
3271173992217,4cyclictest2698871-21systemd-journal00:16:283
3271173992216,4cyclictest3422090-21latency_hist21:35:013
3271172992219,2cyclictest3458932-21kworker/u8:1+events_unbound22:10:212
3271172992219,2cyclictest3287532-21kworker/u8:1+flush-259:019:30:262
3271172992218,3cyclictest3411823-21kworker/u8:1+events_unbound21:48:302
3271172992218,3cyclictest3385575-21kworker/u8:2+flush-259:021:12:342
3271172992218,3cyclictest3113733-21kworker/u8:3+flush-259:019:10:212
3271172992217,4cyclictest3568295-21kworker/u8:0+events_unbound00:15:282
3271172992217,3cyclictest3526278-21kworker/u8:3+events_unbound23:30:132
3271170992218,3cyclictest3000061-21in:imuxsock22:15:001
3271170992217,3cyclictest3453757-21cat22:05:031
3271170992214,6cyclictest2698871-21systemd-journal23:30:001
3271168992218,3cyclictest3458932-21kworker/u8:1+flush-259:022:35:010
3271168992216,4cyclictest3583761-21cp00:10:000
3271168992215,5cyclictest3277423-21diskstats19:15:140
327116899220,20cyclictest1-21systemd23:09:590
3271173992117,3cyclictest3535393-21kworker/u8:1+flush-259:000:00:003
3271173992115,4cyclictest3384684-21ntpq20:55:223
3271173992114,5cyclictest3614095-21ntpq00:35:213
3271172992117,3cyclictest3458932-21kworker/u8:1+events_unbound23:00:232
3271172992117,3cyclictest182790-21kworker/u8:0+events_unbound19:55:282
3271172992115,4cyclictest3541495-21latency_hist23:30:012
3271172992115,4cyclictest3530371-21ntpq23:15:242
3271172992115,4cyclictest3474614-21apt-get22:25:112
3271170992115,4cyclictest3573261-21gzip00:00:001
3271170992115,4cyclictest2698871-21systemd-journal22:16:291
3271168992118,2cyclictest3401712-21apt-get21:15:120
3271168992117,3cyclictest3535393-21kworker/u8:1+flush-259:023:25:170
3271168992112,6cyclictest2698871-21systemd-journal23:05:000
3271173992016,3cyclictest3448178-21kworker/u8:3+events_unbound23:00:003
3271173992015,4cyclictest182790-21kworker/u8:0+flush-259:019:40:253
3271173992014,4cyclictest3396366-21base6421:10:003
3271173992013,5cyclictest2698871-21systemd-journal20:20:003
3271172992016,3cyclictest3354303-21kworker/u8:0+flush-259:020:38:352
3271172992016,3cyclictest3313363-21kworker/u8:3+flush-259:019:50:192
3271170992017,2cyclictest3307980-21cron19:45:001
3271170992015,3cyclictest3385575-21kworker/u8:2+flush-259:021:15:001
3271170992014,4cyclictest3390713-21latency_hist21:05:011
3271168992015,4cyclictest3375701-21diskstats20:50:130
3271168992014,4cyclictest2698871-21systemd-journal22:50:010
3271168992013,5cyclictest3495018-21latency_hist22:45:010
3271172991916,2cyclictest3448178-21kworker/u8:3+flush-259:022:05:202
3271172991915,3cyclictest3458932-21kworker/u8:1+events_unbound22:15:122
3271172991914,4cyclictest3339079-21kworker/u8:2+events_unbound20:44:552
3271172991913,4cyclictest2698871-21systemd-journal21:25:002
3271170991915,3cyclictest2698871-21systemd-journal23:10:291
3271168991913,3cyclictest3313363-21kworker/u8:3+flush-259:019:55:190
3271173991815,2cyclictest3568295-21kworker/u8:0+flush-259:000:04:593
3271173991814,3cyclictest3458932-21kworker/u8:1+flush-259:022:50:273
3271173991812,4cyclictest3286414-21ntpq19:20:233
3271172991814,3cyclictest3599682-21kworker/u8:1+events_unbound00:28:292
3271172991814,3cyclictest3458932-21kworker/u8:1+events_unbound22:44:282
3271172991814,3cyclictest3354303-21kworker/u8:0+events_unbound20:57:462
3271172991814,3cyclictest3313363-21kworker/u8:3+events_unbound20:26:592
3271172991814,3cyclictest3260334-21kworker/u8:2+flush-259:019:47:342
3271172991814,3cyclictest182790-21kworker/u8:0+events_unbound19:20:252
3271172991813,3cyclictest3458932-21kworker/u8:1+flush-259:022:50:112
3271172991813,3cyclictest3448178-21kworker/u8:3+flush-259:022:35:002
3271172991812,4cyclictest3552207-21latency_hist23:40:002
3271172991812,4cyclictest2698871-21systemd-journal20:16:292
3271170991815,2cyclictest3375266-21idleruntime-cro20:50:001
3271170991814,3cyclictest3458932-21kworker/u8:1+flush-259:023:00:181
3271170991814,3cyclictest3292724-21/usr/sbin/munin19:30:191
3271170991812,4cyclictest3416100-21ntpq21:25:251
3271170991811,5cyclictest2698871-21systemd-journal23:24:591
3271168991812,4cyclictest3370168-21cat20:45:000
3271173991714,2cyclictest3448178-21kworker/u8:3+flush-259:022:20:153
3271173991711,4cyclictest3287543-21/usr/sbin/munin19:25:163
3271173991711,4cyclictest2698871-21systemd-journal22:27:233
3271172991713,3cyclictest3448178-21kworker/u8:3+events_unbound22:37:302
3271172991713,3cyclictest3411823-21kworker/u8:1+flush-259:021:44:352
3271172991713,3cyclictest3313363-21kworker/u8:3+flush-259:020:10:162
3271172991713,3cyclictest182790-21kworker/u8:0+events_unbound19:40:212
3271170991713,3cyclictest3364596-21latency_hist20:40:001
3271170991711,4cyclictest2698871-21systemd-journal19:50:001
3271168991714,2cyclictest0-21swapper/022:20:010
3271168991713,3cyclictest60650irq/133-enp0s29f1:rx-019:10:120
3271168991711,4cyclictest3547318-21apt-get23:35:130
3271173991613,2cyclictest3511208-21irqcore23:00:163
3271173991613,2cyclictest3490464-21irqcore22:40:173
3271173991612,3cyclictest3458932-21kworker/u8:1+events_unbound22:35:253
3271173991611,4cyclictest3547284-21kworker/u8:0+flush-259:023:40:173
3271172991612,3cyclictest3568295-21kworker/u8:0+events_unbound23:58:132
3271172991612,3cyclictest3547284-21kworker/u8:0+events_unbound23:45:202
3271172991612,3cyclictest3547284-21kworker/u8:0+events_unbound23:45:192
3271172991612,3cyclictest3385575-21kworker/u8:2+flush-259:021:55:562
3271172991612,3cyclictest3354303-21kworker/u8:0+flush-259:020:46:342
3271172991612,3cyclictest3339079-21kworker/u8:2+events_unbound20:35:012
3271172991611,3cyclictest3515929-21kworker/u8:4+events_unbound23:20:132
3271168991613,2cyclictest60650irq/133-enp0s29f1:rx-022:39:400
3271168991613,2cyclictest60650irq/133-enp0s29f1:rx-022:28:590
3271168991613,2cyclictest60650irq/133-enp0s29f1:rx-021:56:120
3271168991613,2cyclictest60650irq/133-enp0s29f1:rx-019:30:120
3271168991613,2cyclictest60650irq/133-enp0s29f1:rx-000:30:190
3271168991611,4cyclictest3354303-21kworker/u8:0+flush-259:020:45:260
3271168991611,4cyclictest3313363-21kworker/u8:3+flush-259:020:10:000
3271172991511,3cyclictest3328800-21kworker/u8:1+events_unbound20:20:212
327117099159,5cyclictest3448186-21/usr/sbin/munin22:00:141
327117099159,4cyclictest2698871-21systemd-journal00:23:291
3271170991512,2cyclictest3557342-21latency_hist23:45:001
3271170991511,3cyclictest3364624-21kworker/u8:3+flush-259:021:20:231
3271170991510,4cyclictest2698871-21systemd-journal00:11:291
327116899159,4cyclictest3531451-21/usr/sbin/munin23:20:190
3271168991513,1cyclictest61450irq/134-enp0s29f1:tx-020:55:170
3271168991513,1cyclictest60650irq/133-enp0s29f1:rx-019:40:200
3271168991512,2cyclictest60650irq/133-enp0s29f1:rx-023:16:370
3271168991512,2cyclictest60650irq/133-enp0s29f1:rx-023:10:100
3271168991512,2cyclictest60650irq/133-enp0s29f1:rx-022:03:440
3271168991512,2cyclictest60650irq/133-enp0s29f1:rx-021:46:130
3271168991512,2cyclictest60650irq/133-enp0s29f1:rx-019:26:290
3271168991512,2cyclictest3395888-21systemd-run21:10:000
3271168991511,2cyclictest0-21swapper/019:10:000
3271168991510,3cyclictest3453836-21apt-get22:05:140
327117399149,4cyclictest3546629-21apt-get23:35:013
3271173991411,2cyclictest3408258-21kthreadcore21:20:163
3271173991410,3cyclictest3535393-21kworker/u8:1+flush-259:023:20:263
3271173991410,2cyclictest3312202-21ntp_kernel_err19:45:193
327117299148,4cyclictest3401473-21kworker/u8:0+events_unbound21:17:462
3271172991410,3cyclictest3547284-21kworker/u8:0+events_unbound23:40:142
3271170991411,2cyclictest0-21swapper/120:59:591
3271170991410,3cyclictest3599682-21kworker/u8:1+flush-259:000:35:161
3271170991410,3cyclictest3364624-21kworker/u8:3+flush-259:021:06:341
327116899149,3cyclictest0-21swapper/021:30:010
3271168991412,1cyclictest60650irq/133-enp0s29f1:rx-023:50:110
3271168991412,1cyclictest60650irq/133-enp0s29f1:rx-023:45:270
3271168991412,1cyclictest60650irq/133-enp0s29f1:rx-023:45:260
3271168991412,1cyclictest60650irq/133-enp0s29f1:rx-021:50:240
3271168991412,1cyclictest60650irq/133-enp0s29f1:rx-021:20:150
3271168991412,1cyclictest60650irq/133-enp0s29f1:rx-000:38:360
3271168991411,2cyclictest60650irq/133-enp0s29f1:rx-019:50:150
3271168991411,2cyclictest60650irq/133-enp0s29f1:rx-019:45:110
3271168991411,2cyclictest60650irq/133-enp0s29f1:rx-000:16:050
3271168991410,3cyclictest60650irq/133-enp0s29f1:rx-022:10:110
327117399139,3cyclictest3599682-21kworker/u8:1+flush-259:000:30:213
3271173991310,2cyclictest3406150-21unixbench_multi21:15:253
3271173991310,2cyclictest3369252-21unixbench_multi20:40:223
3271173991310,2cyclictest3338845-21sh20:15:003
327117299138,4cyclictest3599682-21kworker/u8:1+events_unbound00:38:312
327117099138,4cyclictest182790-21kworker/u8:0+flush-259:019:58:001
327117099138,3cyclictest3000063-21rs:main1
327117099137,4cyclictest2698871-21systemd-journal20:05:291
3271170991311,1cyclictest3547073-21wc23:35:011
327117099131,10cyclictest1-21systemd22:00:001
3271170991310,2cyclictest3549619-21tr23:35:181
3271170991310,2cyclictest3520443-21unixbench_multi23:05:271
3271170991310,2cyclictest3484662-21munin-plugin-st22:35:011
3271170991310,2cyclictest0-21swapper/119:30:001
327116899139,3cyclictest60650irq/133-enp0s29f1:rx-023:30:110
3271168991311,1cyclictest60650irq/133-enp0s29f1:rx-020:35:280
3271168991311,1cyclictest60650irq/133-enp0s29f1:rx-020:21:400
3271168991310,2cyclictest60650irq/133-enp0s29f1:rx-020:30:530
327117399129,2cyclictest3364624-21kworker/u8:3+flush-259:021:40:243
327117399129,2cyclictest0-21swapper/322:30:173
327117299127,3cyclictest3385575-21kworker/u8:2+events_unbound21:31:022
327117099129,2cyclictest3499728-21unixbench_singl22:45:251
327117099129,2cyclictest3358580-21ntp_kernel_pll_20:30:231
327117099129,2cyclictest3260334-21kworker/u8:2+flush-259:019:36:351
327117099128,3cyclictest3458932-21kworker/u8:1+flush-259:022:50:221
327117099128,3cyclictest3313363-21kworker/u8:3+flush-259:020:13:001
3271170991210,1cyclictest3442751-21munin-plugin-st21:55:011
3271168991211,1cyclictest60650irq/133-enp0s29f1:rx-000:00:020
327117399118,2cyclictest3516381-21irqcore23:05:193
327117399118,2cyclictest3442995-21acpi21:55:123
327117399118,2cyclictest3415951-21needreboot21:25:223
327117399118,2cyclictest3332739-21needreboot20:05:213
327117399110,4cyclictest3328661-21cat20:05:013
327117099119,1cyclictest0-21swapper/122:45:001
327117099119,1cyclictest0-21swapper/120:44:591
327117099117,3cyclictest3547284-21kworker/u8:0+flush-259:023:45:251
327117099117,3cyclictest3547284-21kworker/u8:0+flush-259:023:45:241
327117099108,1cyclictest0-21swapper/123:50:191
327117099108,1cyclictest0-21swapper/122:55:191
327117099105,3cyclictest3349168-21basename20:25:001
32711709997,1cyclictest0-21swapper/122:35:181
32711709997,1cyclictest0-21swapper/122:20:191
32711709997,1cyclictest0-21swapper/121:40:141
32711709997,1cyclictest0-21swapper/120:00:191
32711709996,2cyclictest3401835-21cpuspeed_turbos21:15:151
32711709996,2cyclictest3313500-21cpuspeed_turbos19:50:131
32711709996,1cyclictest0-21swapper/121:45:271
32711709996,1cyclictest0-21swapper/100:15:201
32711709994,3cyclictest3277770-21sed19:15:171
32711709993,5cyclictest3282827-21sed19:20:181
32711709990,7cyclictest3422090-21latency_hist21:35:001
32711709986,1cyclictest0-21swapper/123:20:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional