You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 11:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Tue Nov 18, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
41042993026,3cyclictest123756-21apt-get20:30:002
41039993025,3cyclictest155823-21/usr/sbin/munin21:00:161
41042992823,3cyclictest165892-21latency_hist21:10:012
41049992723,3cyclictest259558-21kworker/u8:4+flush-259:022:40:263
41049992721,4cyclictest176160-21latency_hist21:20:013
41036992623,2cyclictest2159342-21kworker/u8:4+events_unbound19:55:190
41036992621,4cyclictest171433-21diskstats21:15:150
41039992518,5cyclictest2698871-21systemd-journal20:05:001
41049992420,3cyclictest342210-21cp00:00:003
41036992421,2cyclictest192308-21kworker/u8:3+flush-259:021:35:190
41036992420,3cyclictest2159342-21kworker/u8:4+flush-259:019:25:180
41036992420,3cyclictest201991-21kworker/u8:2+events_unbound22:10:180
4103999235,5cyclictest191003-21memory21:30:201
41049992220,1cyclictest0-21swapper/321:55:003
41049992218,3cyclictest3000063-21rs:main3
41039992217,3cyclictest218230-21kworker/u8:3+flush-259:022:35:001
41049992118,2cyclictest92848-21munin-plugin-st20:00:003
41049992118,2cyclictest218230-21kworker/u8:3+flush-259:022:30:183
41042992118,2cyclictest97974-21kworker/u8:1+events_unbound20:05:192
41042992118,2cyclictest97974-21kworker/u8:1+events_unbound20:05:192
41039992118,2cyclictest316439-21cron23:35:001
41036992117,3cyclictest306399-21/usr/sbin/munin23:25:170
41049992017,2cyclictest300663-21gpgv23:20:013
41042992016,3cyclictest348042-21apt-get00:05:122
41036992016,3cyclictest134535-21gpgv20:40:000
41036992015,4cyclictest2698871-21systemd-journal22:09:550
41042991914,4cyclictest373511-21latency_hist00:30:002
4103999194,4cyclictest223230-21modprobe22:03:311
41039991914,3cyclictest342664-21diskstats00:00:151
41036991910,8cyclictest103326-21kworker/u8:2+flush-259:020:45:250
4104999181,5cyclictest577031irqcore19:25:183
41049991815,2cyclictest0-21swapper/321:25:003
41049991814,2cyclictest254854-21irqcore22:35:193
41049991812,4cyclictest40795-21apt-get19:10:023
41049991812,4cyclictest305335-21ntpq23:20:233
41042991815,2cyclictest201991-21kworker/u8:2+flush-259:022:20:002
4103999186,7cyclictest218214-21munin-run22:00:001
41039991815,2cyclictest3000061-21in:imuxsock21:19:591
41039991812,5cyclictest134620-21gpgv20:40:011
4103999181,10cyclictest274778-21munin-run22:55:001
41036991815,2cyclictest2159342-21kworker/u8:4+flush-259:020:00:180
41036991815,2cyclictest0-21swapper/020:15:000
41036991815,2cyclictest0-21swapper/000:05:000
41049991715,1cyclictest0-21swapper/300:30:003
41049991711,4cyclictest383825-21latency_hist00:40:003
41042991715,1cyclictest0-21swapper/200:00:002
41042991713,3cyclictest301208-21kworker/u8:2+flush-259:023:30:262
41042991713,3cyclictest130020-21grep20:35:212
41042991711,4cyclictest2698871-21systemd-journal21:55:002
4103999179,6cyclictest285112-21idleruntime-cro23:05:001
4103999177,8cyclictest268990-21awk22:45:301
4103999175,11cyclictest78581-21cut19:45:191
4103999173,4cyclictest246003-21tr22:25:201
4103999173,4cyclictest246003-21tr22:25:201
41039991714,2cyclictest72186-21sh19:40:001
41039991712,3cyclictest331878-21latency_hist23:50:001
41039991712,3cyclictest186498-21gpgv21:30:011
41039991712,3cyclictest0-21swapper/119:55:001
41039991711,4cyclictest337045-21kworker/u8:3+flush-259:023:55:171
4103999171,14cyclictest1-21systemd22:56:351
41039991710,5cyclictest2698871-21systemd-journal22:40:001
41049991613,2cyclictest2159342-21kworker/u8:4+flush-259:019:45:283
41049991613,2cyclictest187579-21irqcore21:30:173
41049991612,3cyclictest269481-21kworker/u8:0+flush-259:022:55:003
41049991610,4cyclictest377804-21ntpq00:30:223
41049991610,4cyclictest322011-21diskstats23:40:183
41042991613,2cyclictest269618-21cron22:50:002
41042991613,2cyclictest108239-21cron20:15:002
41042991612,3cyclictest363248-21kworker/u8:0+flush-259:000:20:262
41042991612,3cyclictest145484-21kworker/u8:0+flush-259:021:45:252
41042991611,4cyclictest40792-21latency_hist19:10:002
41042991611,3cyclictest2698871-21systemd-journal23:28:362
4103999169,5cyclictest358111-21latency_hist00:15:001
4103999169,5cyclictest322260-21sed23:40:211
4103999167,7cyclictest103146-21latency_hist20:10:001
4103999167,7cyclictest103146-21latency_hist20:09:591
4103999163,5cyclictest113362-21seq20:19:421
41039991612,3cyclictest87943-21apt-get19:55:151
41039991612,3cyclictest128414-21ntp_kernel_err20:30:211
41039991611,3cyclictest56955-21apt-get19:25:131
41039991610,4cyclictest2698871-21systemd-journal21:25:001
41036991613,2cyclictest60650irq/133-enp0s29f1:rx-021:30:180
41036991613,2cyclictest60650irq/133-enp0s29f1:rx-021:20:170
41036991613,2cyclictest60650irq/133-enp0s29f1:rx-020:30:160
41036991611,4cyclictest3000063-21rs:main0
41049991512,2cyclictest3000061-21in:imuxsock20:50:003
41049991512,2cyclictest196312-21ntpq21:35:233
41049991511,3cyclictest316678-21/usr/sbin/munin23:35:153
41049991511,3cyclictest311536-21/usr/sbin/munin23:30:243
4104999151,12cyclictest1-21systemd22:25:003
41049991510,4cyclictest280175-21/usr/sbin/munin23:00:153
41049991510,3cyclictest2698871-21systemd-journal23:46:363
41042991513,1cyclictest0-21swapper/221:15:002
41042991513,1cyclictest0-21swapper/220:35:002
41042991512,2cyclictest301344-21cpuspeed_turbos23:20:142
4104299150,14cyclictest145484-21kworker/u8:0+flush-259:021:25:002
4103999158,5cyclictest373500-21idleruntime-cro00:29:591
4103999158,5cyclictest285494-21tr23:05:161
4103999158,5cyclictest155150-21grep20:55:291
4103999158,5cyclictest113765-21tr20:20:131
4103999157,5cyclictest300940-21ls23:20:011
4103999156,6cyclictest363567-21tr00:20:141
4103999155,2cyclictest263654-21ntpq22:40:251
4103999154,2cyclictest50723-21ntpq19:15:231
41039991510,4cyclictest301238-21apt-get23:20:131
41036991514,1cyclictest60650irq/133-enp0s29f1:rx-022:25:170
41036991514,1cyclictest60650irq/133-enp0s29f1:rx-022:25:170
41036991513,2cyclictest60650irq/133-enp0s29f1:rx-023:41:040
41036991513,1cyclictest61450irq/134-enp0s29f1:tx-023:00:190
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-023:45:120
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-023:36:220
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-022:48:420
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-021:55:190
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-021:25:280
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-021:06:330
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-020:55:200
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-020:20:210
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-019:35:160
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-019:20:180
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-000:25:180
41036991513,1cyclictest60650irq/133-enp0s29f1:rx-000:05:210
41036991513,1cyclictest0-21swapper/023:15:010
41036991513,1cyclictest0-21swapper/021:55:000
41036991512,2cyclictest60650irq/133-enp0s29f1:rx-021:10:230
41036991512,2cyclictest60650irq/133-enp0s29f1:rx-000:15:260
41036991512,2cyclictest352420-21kworker/u8:4+flush-259:000:10:150
41036991512,2cyclictest337016-21cut23:55:010
4104999149,3cyclictest3000063-21rs:main3
4104999149,3cyclictest3000063-21rs:main3
41049991411,2cyclictest121144-21kthreadcore20:25:193
41049991410,3cyclictest238768-21kworker/u8:1+flush-259:022:45:213
41049991410,3cyclictest228747-21apt-get22:10:123
41049991410,3cyclictest145779-21fschecks_time20:50:163
41049991410,3cyclictest124443-21apt-get20:30:123
41042991411,2cyclictest54103-21kthreadcore19:20:202
41042991411,2cyclictest248038-21needreboot22:25:222
41042991411,2cyclictest248038-21needreboot22:25:212
41042991410,3cyclictest96980-21needreboot20:00:202
4103999149,4cyclictest311294-21cron23:30:011
4103999149,3cyclictest56687-21idleruntime-cro19:25:001
4103999149,3cyclictest45552-21ntpq19:10:241
4103999149,3cyclictest243879-21idleruntime-cro22:25:001
4103999149,3cyclictest212666-21dump-pmu-power21:55:001
4103999149,3cyclictest170208-21tr21:10:261
4103999148,5cyclictest66567-21head19:30:251
4103999148,4cyclictest72560-21cpuspeed_turbos19:40:161
4103999148,4cyclictest373873-21cpuspeed_turbos00:30:131
4103999148,4cyclictest3000063-21rs:main1
4103999148,4cyclictest228927-21uniq22:10:161
4103999148,4cyclictest197895-21grep21:40:171
4103999148,3cyclictest145611-21tr20:50:141
4103999147,5cyclictest379188-21tune2fs00:35:181
4103999147,5cyclictest290618-21cpuspeed_turbos23:10:141
4103999146,6cyclictest317282-21cut23:35:171
4103999146,6cyclictest224648-21kthreadcore22:05:201
4103999146,6cyclictest196209-21needreboot21:35:221
4103999146,6cyclictest160722-21latency_hist21:05:011
4103999144,3cyclictest234194-21hddtemp_smartct22:15:201
41039991410,3cyclictest144329-21ntp_kernel_err20:45:221
41036991413,1cyclictest60650irq/133-enp0s29f1:rx-020:19:010
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-023:15:420
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-023:05:170
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-022:50:190
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-022:40:270
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-021:43:320
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-020:40:190
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-020:05:170
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-020:05:160
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-019:50:040
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-019:11:580
41036991412,1cyclictest60650irq/133-enp0s29f1:rx-000:20:200
41036991412,1cyclictest0-21swapper/000:40:000
41036991411,2cyclictest60650irq/133-enp0s29f1:rx-022:55:260
41036991411,2cyclictest60650irq/133-enp0s29f1:rx-019:15:270
41036991411,2cyclictest342169-21idleruntime-cro23:59:590
4104999137,4cyclictest171011-21latency_hist21:15:013
41049991311,1cyclictest358112-21idleruntime-cro00:15:003
41049991311,1cyclictest246774-21kthreadcore22:25:203
41049991311,1cyclictest246774-21kthreadcore22:25:203
41049991311,1cyclictest115971-21tr20:20:193
41049991311,1cyclictest0-21swapper/321:45:003
41049991310,2cyclictest308710-21kthreadcore23:25:213
41049991310,2cyclictest107311-21ntp_kernel_err20:10:223
4104299139,3cyclictest269964-21cpuspeed_turbos22:50:142
41042991311,1cyclictest141357-21kthreadcore20:45:192
41042991310,2cyclictest62296-21fschecks_count19:30:162
41042991310,2cyclictest358437-21cpuspeed_turbos00:15:142
41042991310,2cyclictest326328-21unixbench_singl23:40:302
41042991310,2cyclictest145484-21kworker/u8:0+flush-259:022:00:002
41042991310,2cyclictest108571-21cpuspeed_turbos20:15:152
41042991310,2cyclictest0-21swapper/222:05:002
41042991310,1cyclictest0-21swapper/223:20:012
41042991310,1cyclictest0-21swapper/219:15:012
4103999138,3cyclictest203036-21irqcore21:45:161
4103999138,3cyclictest103464-21grep20:10:151
4103999137,4cyclictest348169-21grep00:05:141
4103999136,5cyclictest359973-21cut00:15:181
4103999136,5cyclictest334100-21cut23:50:211
4103699139,3cyclictest35735-21kworker/u8:2+events_unbound19:34:590
41036991311,1cyclictest60650irq/133-enp0s29f1:rx-022:20:050
41036991311,1cyclictest60650irq/133-enp0s29f1:rx-021:00:250
41036991311,1cyclictest60650irq/133-enp0s29f1:rx-020:50:170
41036991311,1cyclictest60650irq/133-enp0s29f1:rx-019:45:140
41036991311,1cyclictest60650irq/133-enp0s29f1:rx-019:40:450
41036991310,2cyclictest60650irq/133-enp0s29f1:rx-023:20:140
4104999129,2cyclictest71663-21unixbench_multi19:35:263
4104999127,4cyclictest336621-21sshd23:50:343
4104999125,5cyclictest516-21gdbus21:00:013
41049991210,1cyclictest360234-21kthreadcore00:15:183
4104299129,2cyclictest41475-21kworker/u8:3+flush-259:020:20:192
4104299129,2cyclictest41475-21kworker/u8:3+flush-259:019:45:272
4104299129,2cyclictest346290-21missed_timers00:00:202
4104299129,2cyclictest258431-21ntp_kernel_pll_22:35:232
4104299129,2cyclictest196100-21latency21:35:212
4104299129,2cyclictest0-21swapper/223:49:592
4103999125,6cyclictest135118-21cat20:40:011
41036991210,1cyclictest60650irq/133-enp0s29f1:rx-023:34:500
41036991210,1cyclictest60650irq/133-enp0s29f1:rx-022:35:180
41036991210,1cyclictest60650irq/133-enp0s29f1:rx-021:45:190
41036991210,1cyclictest60650irq/133-enp0s29f1:rx-020:25:180
4104999118,2cyclictest347381-21gpgv00:04:593
4104999117,3cyclictest129585-21/usr/sbin/munin20:35:193
4104999115,4cyclictest156744-21sed21:00:193
4104999111,8cyclictest368361-21awk00:24:593
4104299119,1cyclictest0-21swapper/222:25:002
4104299119,1cyclictest0-21swapper/220:45:002
4104299119,1cyclictest0-21swapper/219:30:002
4104299118,2cyclictest352420-21kworker/u8:4+flush-259:000:10:172
4104299118,2cyclictest150861-21fschecks_count20:55:192
4104299118,1cyclictest0-21swapper/223:10:012
4104299118,1cyclictest0-21swapper/221:20:012
4104299116,4cyclictest295046-21sshd23:10:352
4104999108,1cyclictest0-21swapper/319:50:203
4104999107,2cyclictest51885-21cpuspeed_turbos19:20:163
4104999105,4cyclictest77452-21cat19:45:003
4104999101,7cyclictest224043-21sed22:05:193
4104299108,1cyclictest230218-21kthreadcore22:10:182
4104299107,2cyclictest88011-21cpuspeed_turbos19:55:172
4104299107,2cyclictest269656-21kworker/u8:3+flush-259:023:00:192
4104299107,2cyclictest186465-21latency_hist21:30:012
4104299106,2cyclictest72582-21grep19:40:162
4104299103,5cyclictest337151-21latency_hist23:55:002
4104299103,5cyclictest276121-21kthreadcore22:55:192
410499995,3cyclictest145484-21kworker/u8:0+flush-259:021:58:363
410429997,1cyclictest0-21swapper/220:54:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional