You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-26 - 09:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot4s.osadl.org (updated Wed Nov 26, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
462295994632,9cyclictest555158-21ntpq20:35:253
462283994442,2cyclictest75-21kswapd021:59:580
462283993832,4cyclictest519570-21awk20:05:010
462283993831,5cyclictest2698871-21systemd-journal21:55:000
462290993420,11cyclictest732503-21latency_hist23:30:012
462295993330,2cyclictest75-21kswapd021:59:583
462288993326,5cyclictest560592-21ntpq20:40:201
462283993229,2cyclictest75-21kswapd022:45:040
462283993228,3cyclictest540586-21kworker/u8:0+flush-259:020:25:170
462283993226,4cyclictest747980-21latency_hist23:45:010
462295993128,2cyclictest75-21kswapd022:45:053
462288993125,4cyclictest633519-21latency_hist21:55:001
462283993126,3cyclictest617861-21kworker/u8:0+flush-259:021:40:180
462283993025,4cyclictest551072-21kworker/u8:1+events_unbound20:40:020
462283993024,4cyclictest717251-21/usr/sbin/munin23:15:240
462288992925,2cyclictest566695-21kworker/u8:0+flush-259:021:20:191
462295992812,14cyclictest768682-21latency_hist00:05:013
462290992721,4cyclictest550841-21latency_hist20:35:002
462283992721,4cyclictest499385-21diskstats19:45:140
462295992615,10cyclictest489968-21kthreadcore19:35:193
462290992620,4cyclictest727322-21latency_hist23:25:002
462283992623,2cyclictest1555393-21kworker/u8:1+flush-259:019:30:190
462295992522,2cyclictest685112-21kworker/u8:0+flush-259:022:50:163
462290992520,3cyclictest561250-21kworker/u8:2+flush-259:021:00:182
462288992522,2cyclictest664679-21kworker/u8:1+flush-259:023:35:201
462288992522,2cyclictest664679-21kworker/u8:1+flush-259:023:35:201
462288992520,4cyclictest561759-21/usr/sbin/munin20:45:151
462295992419,4cyclictest587478-21apt-get21:10:123
462295992419,3cyclictest701409-21latency_hist23:00:003
462295992411,1cyclictest0-21swapper/319:15:013
462290992420,3cyclictest617861-21kworker/u8:0+flush-259:021:55:012
462288992421,2cyclictest602448-21kworker/u8:3+events_unbound21:59:561
462283992419,4cyclictest654425-21apt-get22:15:120
462283992419,3cyclictest732739-21/usr/sbin/munin23:30:140
462283992418,4cyclictest2698871-21systemd-journal20:10:000
462295992320,2cyclictest701438-21kworker/u8:4+flush-259:023:09:593
462295992317,4cyclictest753162-21latency_hist23:50:003
462290992320,2cyclictest468364-21apt-get19:15:102
462290992317,4cyclictest561520-21latency_hist20:45:002
462288992320,2cyclictest462758-21apt-get19:10:131
462283992320,2cyclictest456676-21kworker/u8:0+flush-259:019:35:170
462283992319,3cyclictest514653-21kworker/u8:3+events_unbound20:30:220
462283992318,4cyclictest561250-21kworker/u8:2+flush-259:021:00:220
462283992317,4cyclictest534962-21apt-get20:20:010
462283992316,5cyclictest556145-21base6420:40:000
46229599228,4cyclictest675016-21/usr/sbin/munin22:35:153
462295992218,3cyclictest758580-21/usr/sbin/munin23:55:143
462295992216,4cyclictest602662-21latency_hist21:25:003
462290992218,3cyclictest473530-21apt-get19:20:122
462290992217,3cyclictest468045-21cp19:15:012
462288992218,3cyclictest75-21kswapd022:45:041
462288992217,4cyclictest540782-21/usr/sbin/munin20:25:201
462283992217,3cyclictest602448-21kworker/u8:3+events_unbound21:25:220
462295992116,4cyclictest592590-21/usr/sbin/munin21:15:203
462295992116,4cyclictest592590-21/usr/sbin/munin21:15:193
462295992116,4cyclictest494060-21/usr/sbin/munin19:40:153
462288992117,3cyclictest514653-21kworker/u8:3+flush-259:020:35:161
462288992116,4cyclictest608067-21/usr/sbin/munin21:30:161
462288992115,4cyclictest524682-21latency_hist20:10:011
462283992117,3cyclictest664679-21kworker/u8:1+events_unbound23:50:130
462283992117,3cyclictest551072-21kworker/u8:1+events_unbound20:50:150
462283992114,5cyclictest706619-21latency_hist23:05:000
462283992114,5cyclictest618089-21latency_hist21:40:000
462283992113,6cyclictest798643-21ntpq00:30:190
46229099209,1cyclictest0-21swapper/200:15:152
46229099208,2cyclictest664038-21unixbench_singl22:20:262
462290992017,2cyclictest679935-21sh22:40:002
462290992015,4cyclictest763516-21cp00:00:002
462290992014,4cyclictest519566-21latency_hist20:05:012
462288992016,3cyclictest706854-21kworker/u8:5+events_unbound23:15:181
462288992016,3cyclictest1555393-21kworker/u8:1+flush-259:019:25:181
462295991916,2cyclictest773858-21idleruntime-cro00:10:003
462295991916,2cyclictest0-21swapper/320:20:173
46229099198,2cyclictest716594-21unixbench_singl23:10:262
462290991916,2cyclictest566657-21sh20:50:002
462290991916,2cyclictest504069-21kworker/u8:1+flush-259:020:00:002
46228899199,9cyclictest685112-21kworker/u8:0+flush-259:022:50:121
462288991916,2cyclictest685112-21kworker/u8:0+flush-259:023:00:151
462288991916,2cyclictest488707-21kworker/u8:2+flush-259:019:55:161
462288991915,3cyclictest468332-21kworker/u8:2+events_unbound19:15:181
462288991913,4cyclictest710892-21ntpq23:05:231
462288991913,4cyclictest530239-21diskstats20:15:151
462288991913,4cyclictest504069-21kworker/u8:1+flush-259:019:55:001
462283991915,3cyclictest784379-21kworker/u8:1+events_unbound00:35:170
462283991915,3cyclictest669597-21sh22:30:010
462283991915,3cyclictest587238-21kworker/u8:2+events_unbound21:10:260
462283991914,3cyclictest2698871-21systemd-journal21:09:410
462290991815,2cyclictest768676-21idleruntime-cro00:05:002
462290991813,4cyclictest504100-21latency_hist19:50:012
462288991814,3cyclictest664679-21kworker/u8:1+flush-259:022:45:011
462288991814,3cyclictest468332-21kworker/u8:2+events_unbound19:20:151
462288991813,3cyclictest566695-21kworker/u8:0+flush-259:021:15:011
462288991812,5cyclictest628389-21latency_hist21:50:011
462283991814,3cyclictest664679-21kworker/u8:1+events_unbound23:10:010
462283991814,3cyclictest424904-21kworker/u8:4+flush-259:019:10:230
462295991714,2cyclictest501513-21kthreadcore19:45:183
462295991713,3cyclictest664679-21kworker/u8:1+flush-259:022:25:123
462295991713,3cyclictest551072-21kworker/u8:1+flush-259:020:40:163
462295991710,5cyclictest2698871-21systemd-journal23:11:423
462290991714,2cyclictest722157-21cron23:19:592
462290991714,2cyclictest603340-21irqcore21:25:152
462290991712,4cyclictest504338-21/usr/sbin/munin19:50:152
462288991714,2cyclictest587238-21kworker/u8:2+flush-259:021:15:171
462288991714,2cyclictest587238-21kworker/u8:2+flush-259:021:15:171
462288991713,3cyclictest602448-21kworker/u8:3+flush-259:021:35:281
462288991713,3cyclictest581940-21kworker/u8:3+flush-259:021:05:241
462288991713,3cyclictest456676-21kworker/u8:0+events_unbound19:45:251
462288991711,4cyclictest571793-21latency_hist20:55:001
462283991713,3cyclictest659319-21kworker/u8:0+events_unbound22:30:170
462283991713,3cyclictest602448-21kworker/u8:3+events_unbound21:23:410
462283991713,3cyclictest477458-21ntp_kernel_pll_19:20:220
462283991712,4cyclictest566695-21kworker/u8:0+events_unbound21:30:190
462283991712,4cyclictest551072-21kworker/u8:1+flush-259:020:59:570
462283991712,4cyclictest534591-21kworker/u8:2+events_unbound20:20:030
462295991613,1cyclictest0-21swapper/322:20:003
462290991613,2cyclictest0-21swapper/222:35:012
462290991612,3cyclictest794932-21fschecks_time00:30:142
462288991612,3cyclictest633569-21kworker/u8:1+flush-259:022:10:001
462288991611,3cyclictest727345-21kworker/u8:2+flush-259:000:30:161
462283991614,1cyclictest60650irq/133-enp0s29f1:rx-023:10:150
462283991614,1cyclictest60650irq/133-enp0s29f1:rx-020:45:180
462283991614,1cyclictest60650irq/133-enp0s29f1:rx-000:00:300
462283991613,2cyclictest60650irq/133-enp0s29f1:rx-000:20:150
462283991612,3cyclictest753180-21kworker/u8:3+events_unbound23:57:490
462283991612,3cyclictest727345-21kworker/u8:2+events_unbound00:10:190
462283991612,3cyclictest721807-21kworker/u8:0+events_unbound23:47:470
462283991612,3cyclictest509306-21kworker/u8:0+flush-259:020:10:260
462283991612,3cyclictest468332-21kworker/u8:2+flush-259:019:15:260
462283991612,2cyclictest597513-21sh21:20:000
462283991612,2cyclictest597513-21sh21:20:000
462295991512,2cyclictest717698-21irqcore23:15:183
462295991511,3cyclictest509306-21kworker/u8:0+flush-259:020:05:173
462295991510,3cyclictest763570-21kworker/u8:0+flush-259:000:15:133
462290991512,2cyclictest644548-21irqcore22:05:192
462290991512,2cyclictest602448-21kworker/u8:3+flush-259:021:35:262
462290991512,2cyclictest534955-21latency_hist20:20:012
46228899159,5cyclictest493818-21latency_hist19:40:011
462288991512,2cyclictest733185-21irqcore23:30:161
462288991512,2cyclictest514653-21kworker/u8:3+events_unbound20:30:251
462288991511,3cyclictest623266-21kworker/u8:2+flush-259:022:15:001
462283991513,1cyclictest60650irq/133-enp0s29f1:rx-022:20:160
462283991513,1cyclictest60650irq/133-enp0s29f1:rx-021:45:180
462283991513,1cyclictest60650irq/133-enp0s29f1:rx-019:40:240
462283991512,2cyclictest60650irq/133-enp0s29f1:rx-023:28:540
462283991512,2cyclictest60650irq/133-enp0s29f1:rx-022:10:290
462283991512,2cyclictest60650irq/133-enp0s29f1:rx-019:52:090
462295991412,1cyclictest0-21swapper/320:50:003
462295991411,2cyclictest743313-21fschecks_time23:40:143
462295991411,2cyclictest514429-21munin-plugin-st20:00:003
462295991410,3cyclictest602448-21kworker/u8:3+flush-259:021:35:283
462295991410,3cyclictest524720-21kworker/u8:1+flush-259:020:10:293
462290991411,2cyclictest654596-21fschecks_count22:15:142
462290991411,2cyclictest0-21swapper/223:34:592
462290991411,2cyclictest0-21swapper/220:35:182
46228899149,4cyclictest727563-21/usr/sbin/munin23:25:161
46228899149,4cyclictest618349-21apt-get21:40:121
462288991410,3cyclictest668671-21ntp_kernel_err22:25:191
462288991410,3cyclictest633569-21kworker/u8:1+events_unbound22:00:161
462283991411,2cyclictest61450irq/134-enp0s29f1:tx-019:55:240
462283991411,2cyclictest60650irq/133-enp0s29f1:rx-022:43:430
462283991411,2cyclictest60650irq/133-enp0s29f1:rx-000:25:190
462283991410,3cyclictest763570-21kworker/u8:0+events_unbound00:17:440
46229599138,3cyclictest509283-21latency_hist19:55:013
462295991311,1cyclictest0-21swapper/323:24:593
462295991310,2cyclictest632512-21needreboot21:50:223
462295991310,2cyclictest582532-21fschecks_time21:05:133
462295991310,2cyclictest484009-21fschecks_time19:30:153
46229099139,3cyclictest581940-21kworker/u8:3+flush-259:021:05:152
46229099139,3cyclictest566695-21kworker/u8:0+flush-259:021:12:462
46229099139,3cyclictest0-21swapper/221:15:252
46229099139,3cyclictest0-21swapper/221:15:242
46228899139,3cyclictest727345-21kworker/u8:2+flush-259:000:00:241
46228899139,3cyclictest701438-21kworker/u8:4+flush-259:023:10:171
462288991310,2cyclictest519562-21idleruntime-cro20:05:001
462283991310,2cyclictest60650irq/133-enp0s29f1:rx-000:05:160
462283991310,2cyclictest0-21swapper/023:40:010
462283991310,2cyclictest0-21swapper/023:40:000
46229599129,2cyclictest788819-21unixbench_multi00:20:253
46229599129,2cyclictest680698-21irqcore22:40:183
46229599129,2cyclictest545969-21apt-get20:30:123
46229599127,4cyclictest794455-21latency_hist00:30:003
46229599126,4cyclictest649015-21latency_hist22:10:013
462295991210,1cyclictest706613-21sh23:05:003
462295991210,1cyclictest0-21swapper/322:35:013
46229099129,2cyclictest691775-21irqcore22:50:142
46228899129,2cyclictest794011-21unixbench_singl00:25:261
46228899128,3cyclictest774585-21sed00:10:171
46229599119,1cyclictest0-21swapper/321:30:013
46229599118,2cyclictest742825-21sh23:40:003
46229599118,2cyclictest742825-21sh23:39:593
46229599118,2cyclictest649383-21tr22:10:153
46229599118,2cyclictest639032-21cpuspeed_turbos22:00:133
46229599115,4cyclictest572219-21diskstats20:55:143
46229599110,9cyclictest571793-21latency_hist20:55:003
46229099119,1cyclictest743505-21sed23:40:162
46229099119,1cyclictest508-21tpm2-abrmd22:15:002
46229099119,1cyclictest0-21swapper/221:25:002
46229099119,1cyclictest0-21swapper/220:50:172
46229099118,2cyclictest489053-21cpuspeed_turbos19:35:152
46229099118,2cyclictest488097-21timerwakeupswit19:30:252
46229099118,1cyclictest0-21swapper/219:40:242
46229099116,4cyclictest2698871-21systemd-journal19:27:412
46229099115,4cyclictest545688-21cron20:29:592
46229099111,9cyclictest753180-21kworker/u8:3+flush-259:023:55:012
46228899118,2cyclictest800072-21fschecks_count00:35:151
46228899116,3cyclictest696525-21apt-get22:55:121
46228899115,4cyclictest675202-21diskstats22:35:141
46229599108,1cyclictest0-21swapper/321:45:013
46229599108,1cyclictest0-21swapper/300:40:013
46229599108,1cyclictest0-21swapper/300:35:013
46229599107,2cyclictest478690-21cpuspeed_turbos19:25:153
46229599107,2cyclictest0-21swapper/319:15:123
46229599107,2cyclictest0-21swapper/300:10:163
46229599106,3cyclictest566695-21kworker/u8:0+flush-259:021:05:013
46229599106,2cyclictest736801-21ntpq23:30:223
46229599105,3cyclictest608238-21cut21:30:143
46229599101,2cyclictest628378-21idleruntime-cro21:50:003
46229599101,1cyclictest474469-21irqcore19:20:163
46229099108,1cyclictest685095-21munin-plugin-st22:45:002
46229099108,1cyclictest0-21swapper/222:25:172
46229099108,1cyclictest0-21swapper/221:40:182
46229099108,1cyclictest0-21swapper/221:30:172
46229099106,3cyclictest701003-21grep22:55:252
46229099106,3cyclictest685772-21apt-get22:45:112
46229099106,3cyclictest2698871-21systemd-journal00:30:012
46229099105,4cyclictest737916-21apt-get23:35:122
46229099105,4cyclictest737916-21apt-get23:35:122
46229099100,2cyclictest776416-21kthreadcore00:10:192
4622909997,1cyclictest0-21swapper/223:00:172
4622909997,1cyclictest0-21swapper/220:05:152
4622909996,2cyclictest804319-21unixbench_singl00:35:262
4622909996,2cyclictest0-21swapper/220:59:592
4622959984,3cyclictest753978-21sed23:50:163
4622959982,4cyclictest727677-21cpuspeed_turbos23:25:143
4622909984,3cyclictest639142-21cut22:00:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional