You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 09:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Thu Nov 27, 2025 00:39:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4003184995847,5cyclictest0-21swapper/221:44:212
4003184995847,5cyclictest0-21swapper/221:44:212
4003184995141,6cyclictest4039767-21/usr/sbin/munin21:14:122
4003184994823,22cyclictest4030695-21munin-plugin-st20:50:012
4003184994823,22cyclictest4030695-21munin-plugin-st20:50:012
4003184994818,27cyclictest4122321-21sh23:00:462
4003182994833,8cyclictest12450irq/32-eth021:34:130
4003182994833,8cyclictest12450irq/32-eth021:34:130
400318499469,18cyclictest4178507-21ssh00:14:162
400318499463,8cyclictest4036129-21ntpq21:09:152
4003184994616,27cyclictest4168653-21sh00:00:242
4003184994535,6cyclictest4098618-21kworker/u8:6-events_unbound22:59:142
4003184994534,7cyclictest4044537-21latency_hist21:20:022
4003184994523,19cyclictest0-21swapper/200:00:012
4003183994537,5cyclictest4172114-21ssh00:05:011
4003183994537,3cyclictest0-21swapper/121:59:121
4003183994535,6cyclictest4082123-21/usr/sbin/munin22:09:181
4003182994535,7cyclictest4049634-21sshd21:27:110
4003182994535,7cyclictest4049634-21sshd21:27:110
400318299452,35cyclictest2199103-21cups-browsed21:09:150
4003184994430,11cyclictest4152035-21mailstats23:39:212
4003184994425,14cyclictest0-21swapper/223:19:122
4003184994418,23cyclictest956-21mta-sts-daemon23:40:102
4003184994418,23cyclictest4113627-21timerwakeupswit22:49:272
4003184994416,25cyclictest4034491-21missed_timers21:04:142
4003183994436,5cyclictest4160498-21systemd-run23:50:011
4003183994435,6cyclictest4016677-21systemd-run19:55:011
4003183994435,6cyclictest4016677-21systemd-run19:55:011
4003182994435,7cyclictest4090884-21ssh22:20:010
4003182994431,7cyclictest4093789-21ssh22:24:150
400318499432,9cyclictest4185736-21ssh00:24:172
400318499432,9cyclictest4185736-21ssh00:24:172
4003184994323,17cyclictest0-21swapper/223:15:012
4003184994323,17cyclictest0-21swapper/223:15:012
4003183994336,4cyclictest4070339-21kernelversion21:54:091
4003183994336,4cyclictest4070339-21kernelversion21:54:091
4003183994335,5cyclictest4182985-21cron00:20:011
4003183994335,5cyclictest4182985-21cron00:20:011
4003182994338,3cyclictest0-21swapper/021:59:120
400318499422,18cyclictest4093753-21ssh22:24:142
400318499422,16cyclictest4101649-21diskmemload22:34:232
4003184994216,23cyclictest4005585-21ntp_kernel_pll_19:19:152
400318499421,15cyclictest4067075-21sensors21:49:262
400318499421,15cyclictest4067075-21sensors21:49:262
400318499421,14cyclictest4116917-21ssh22:54:152
4003182994225,12cyclictest151ktimers/020:59:130
400318499413,9cyclictest4147845-21ntpq23:34:132
400318499413,9cyclictest4147845-21ntpq23:34:132
400318499412,18cyclictest4097674-21ssh22:29:162
4003184994118,20cyclictest4008512-21needreboot19:29:112
400318499411,14cyclictest4070651-21ntpq21:54:162
400318499411,14cyclictest4070651-21ntpq21:54:162
400318499411,13cyclictest4055093-21ntpq21:34:122
400318499411,13cyclictest4055093-21ntpq21:34:122
4003183994134,3cyclictest0-21swapper/121:24:181
4003183994134,3cyclictest0-21swapper/121:24:181
4003183994133,5cyclictest1-21systemd19:15:011
4003183994123,3cyclictest171rcu_preempt21:49:181
4003183994123,3cyclictest171rcu_preempt21:49:181
4003182994134,4cyclictest4148938-21latency_hist23:35:020
4003182994134,4cyclictest4148938-21latency_hist23:35:020
4003182994132,7cyclictest266-21systemd-journal23:20:000
4003182994131,7cyclictest4121445-21vmstat22:59:280
4003182994128,11cyclictest421-21avahi-daemon00:24:070
4003182994128,11cyclictest421-21avahi-daemon00:24:070
400318499404,30cyclictest4078331-21ssh22:04:202
400318499402,18cyclictest956-21mta-sts-daemon23:54:182
4003184994017,2cyclictest0-21swapper/220:09:112
4003184994016,4cyclictest4017548-21ntp_kernel_pll_19:59:152
4003184994013,5cyclictest2194-21unixbench_multi00:39:302
4003184994013,5cyclictest2194-21unixbench_multi00:39:302
4003184994010,5cyclictest4036692-21diskmemload22:08:002
4003183994034,3cyclictest0-21swapper/100:00:031
4003183994033,3cyclictest0-21swapper/122:20:011
4003183994031,6cyclictest421-21avahi-daemon22:49:271
4003183994030,7cyclictest4027737-21latency_hist20:40:021
400318299405,26cyclictest4078426-21ssh22:04:230
4003182994029,7cyclictest422-21dbus-daemon21:44:090
4003182994029,7cyclictest422-21dbus-daemon21:44:090
4003182994022,4cyclictest0-21swapper/000:14:210
400318499392,16cyclictest4140115-21sh23:24:142
400318499392,16cyclictest4140115-21sh23:24:142
400318499391,9cyclictest4059043-21ssh21:39:142
400318499391,9cyclictest4059043-21ssh21:39:142
4003184993913,3cyclictest0-21swapper/220:20:012
400318499391,15cyclictest4144007-21ntpq23:29:172
400318499391,15cyclictest4144007-21ntpq23:29:172
400318499391,15cyclictest4105385-21rm22:39:202
4003184993911,5cyclictest4036692-21diskmemload21:23:122
4003184993911,5cyclictest4036692-21diskmemload21:23:122
4003184993911,5cyclictest2199103-21cups-browsed20:11:492
400318499391,13cyclictest4109043-21missed_timers22:44:102
400318499391,13cyclictest4074583-21ssh21:59:182
4003184993910,6cyclictest957-21mta-sts-daemon19:39:122
4003183993931,5cyclictest4059967-21ls21:40:001
4003183993931,5cyclictest4059967-21ls21:40:001
400318399391,27cyclictest4132387-21sh23:14:161
400318399391,27cyclictest4132387-21sh23:14:161
4003182993927,9cyclictest4192735-21timerwakeupswit00:34:270
4003182993927,9cyclictest4192735-21timerwakeupswit00:34:270
400318499388,8cyclictest4014608-21awk19:49:182
400318499383,7cyclictest4085949-21ntpq22:14:142
4003184993816,19cyclictest0-21swapper/220:40:012
400318499381,16cyclictest4159529-21ssh23:49:142
400318499381,13cyclictest4015677-21diskstats19:54:072
400318499381,13cyclictest4015677-21diskstats19:54:072
4003183993831,4cyclictest4032176-21cron20:55:001
4003183993830,3cyclictest0-21swapper/122:44:201
4003182993830,4cyclictest0-21swapper/023:14:200
4003182993830,4cyclictest0-21swapper/023:14:200
4003182993829,6cyclictest2199103-21cups-browsed23:46:330
4003182993825,7cyclictest0-21swapper/021:54:150
4003182993825,7cyclictest0-21swapper/021:54:150
4003182993824,9cyclictest12450irq/32-eth023:54:180
4003182993823,11cyclictest0-21swapper/023:25:010
4003182993823,11cyclictest0-21swapper/023:25:010
4003184993726,8cyclictest4010488-21timerwakeupswit19:34:242
400318499371,14cyclictest4182200-21ntpq00:19:242
400318499371,14cyclictest4182200-21ntpq00:19:242
4003183993729,5cyclictest4029216-21systemd-run20:45:011
4003183993729,5cyclictest4006193-21sendmail-msp19:20:011
4003183993728,4cyclictest0-21swapper/123:09:201
4003183993728,4cyclictest0-21swapper/123:09:201
4003183993725,9cyclictest4144126-21sh23:29:221
4003183993725,9cyclictest4144126-21sh23:29:221
4003182993728,5cyclictest0-21swapper/000:04:150
400318299372,29cyclictest4039931-21ssh21:14:190
4003182993721,12cyclictest151ktimers/020:24:150
4003184993628,5cyclictest4192636-21switchtime00:34:262
4003184993628,5cyclictest4192636-21switchtime00:34:262
4003184993628,5cyclictest4013662-21cron19:45:012
400318499362,10cyclictest657-21snmpd00:26:162
400318499362,10cyclictest657-21snmpd00:26:162
400318499362,10cyclictest2199103-21cups-browsed20:04:192
400318499361,14cyclictest4025047-21sshd20:24:112
4003184993611,3cyclictest0-21swapper/223:06:442
4003184993611,3cyclictest0-21swapper/223:06:442
4003184993610,7cyclictest4004024-21/usr/sbin/munin19:14:122
4003183993626,6cyclictest2199103-21cups-browsed19:39:221
4003182993627,7cyclictest2483-21cron00:40:000
4003182993627,7cyclictest2483-21cron00:40:000
4003182993627,6cyclictest421-21avahi-daemon23:08:230
4003182993627,6cyclictest421-21avahi-daemon23:08:230
4003182993627,6cyclictest4021080-21unixbench_multi20:09:250
400318299362,28cyclictest4155735-21ssh23:44:160
400318499354,9cyclictest4089929-21ntpq22:19:162
4003183993528,3cyclictest0-21swapper/100:29:251
4003183993528,3cyclictest0-21swapper/100:29:251
4003183993527,5cyclictest4102463-21systemd-run22:35:011
4003182993529,4cyclictest4035311-21latency_hist21:05:030
400318499341,16cyclictest4174968-21sh00:09:182
400318399344,19cyclictest4155735-21ssh23:44:161
4003183993428,2cyclictest0-21swapper/120:00:011
4003183993426,5cyclictest4141096-21systemd-run23:25:011
4003183993426,5cyclictest4141096-21systemd-run23:25:011
4003183993425,6cyclictest421-21avahi-daemon00:10:501
4003183993425,4cyclictest0-21swapper/122:14:201
4003183993424,7cyclictest422-21dbus-daemon21:26:221
4003183993424,7cyclictest422-21dbus-daemon21:26:221
4003183993424,7cyclictest421-21avahi-daemon22:21:451
4003183993424,7cyclictest4120757-21ntp_kernel_pll_22:59:161
4003183993421,10cyclictest0-21swapper/123:04:171
4003182993427,5cyclictest4015283-21kworker/u8:0-rpciod20:00:020
4003184993323,7cyclictest453-21systemd-logind21:00:022
4003184993311,6cyclictest599-21lldpd20:44:212
400318499331,13cyclictest4051597-21/usr/sbin/munin21:29:232
400318499331,13cyclictest4051597-21/usr/sbin/munin21:29:232
4003183993328,3cyclictest0-21swapper/122:40:011
4003183993327,2cyclictest0-21swapper/100:34:181
4003183993327,2cyclictest0-21swapper/100:34:181
4003183993325,5cyclictest4022657-21munin-plugin-st20:15:011
4003183993323,7cyclictest4147977-21ntp_offset23:34:181
4003183993323,7cyclictest4147977-21ntp_offset23:34:181
4003183993320,10cyclictest4163825-21switchtime23:54:281
4003182993326,5cyclictest4114064-21munin-run22:50:010
4003182993321,9cyclictest4004576-21unixbench_multi19:14:250
4003182993312,15cyclictest151ktimers/023:29:190
4003182993312,15cyclictest151ktimers/023:29:190
4003183993226,3cyclictest0-21swapper/123:20:011
4003183993226,3cyclictest0-21swapper/120:09:151
4003183993226,3cyclictest0-21swapper/100:24:291
4003183993226,3cyclictest0-21swapper/100:24:291
4003183993225,3cyclictest0-21swapper/123:39:161
4003183993224,5cyclictest4063999-21ssh21:45:021
4003183993224,5cyclictest4063999-21ssh21:45:021
4003183993224,5cyclictest4009730-21gltestperf19:34:071
4003183993223,4cyclictest0-21swapper/122:04:191
4003183993221,8cyclictest4036030-21needreboot21:09:121
4003182993227,3cyclictest959-21mta-sts-daemon22:44:300
4003182993226,4cyclictest4189668-21unixbench_multi00:29:310
4003182993226,4cyclictest4189668-21unixbench_multi00:29:310
4003182993223,7cyclictest4175377-21timerwakeupswit00:09:250
4003182993223,7cyclictest4009485-21aten2.4-expect19:34:030
400318299322,20cyclictest4036692-21diskmemload21:49:190
400318299322,20cyclictest4036692-21diskmemload21:49:190
4003183993126,3cyclictest0-21swapper/120:20:011
4003183993124,4cyclictest4007271-21sendmail_mailst19:24:241
4003183993123,5cyclictest4033771-21latency_hist21:00:021
4003183993117,11cyclictest4193590-21sh00:35:441
4003183993117,11cyclictest4193590-21sh00:35:441
4003182993125,3cyclictest0-21swapper/019:20:000
4003182993124,4cyclictest4096972-21aten_r2power_en22:29:040
4003182993124,4cyclictest4009142-21head19:29:250
4003182993122,7cyclictest4117559-21sh22:54:280
400318299312,20cyclictest4047553-21ssh21:24:190
400318299312,20cyclictest4047553-21ssh21:24:190
4003182993118,8cyclictest0-21swapper/021:39:160
4003182993118,8cyclictest0-21swapper/021:39:160
4003183993024,3cyclictest0-21swapper/121:34:151
4003183993024,3cyclictest0-21swapper/121:34:151
4003183993023,4cyclictest957-21mta-sts-daemon22:29:201
400318399302,21cyclictest4116989-21ssh22:54:171
4003183993021,5cyclictest956-21mta-sts-daemon21:15:181
4003182993022,5cyclictest421-21avahi-daemon22:39:550
4003182993017,9cyclictest12450irq/32-eth022:34:220
4003182993016,10cyclictest12450irq/32-eth023:04:140
400318399291,23cyclictest4039791-21ssh21:14:131
4003182992922,4cyclictest0-21swapper/000:20:010
4003182992922,4cyclictest0-21swapper/000:20:010
4003182992919,7cyclictest4026475-21fschecks_time20:34:030
4003182992916,9cyclictest422-21dbus-daemon21:20:020
4003183992821,4cyclictest959-21mta-sts-daemon00:08:051
4003183992818,7cyclictest4034373-21kernelversion21:04:131
4003182992822,4cyclictest4019661-21cron20:05:000
4003182992820,6cyclictest4024195-21systemd-run20:20:010
4003182992820,5cyclictest422-21dbus-daemon22:07:040
4003182992811,12cyclictest12450irq/32-eth023:39:150
4003183992719,5cyclictest4026469-21fschecks_count20:34:031
4003183992719,5cyclictest4025563-21sed20:24:241
4003183992719,5cyclictest4018719-21fschecks_time20:04:091
400318299273,19cyclictest4067805-21kworker/u8:17+rpciod22:14:200
4003182992619,4cyclictest4012962-21missed_timers19:44:100
4003182992618,6cyclictest2199103-21cups-browsed19:49:160
4003182992613,3cyclictest0-21swapper/020:40:010
400318499254,14cyclictest437-21mta-sts-daemon20:54:152
4003183992518,5cyclictest4013573-21unixbench_multi19:44:251
4003182992520,3cyclictest955-21mta-sts-daemon19:54:090
4003182992520,3cyclictest955-21mta-sts-daemon19:54:090
4003182992520,3cyclictest151ktimers/020:14:190
4003184992416,5cyclictest2199103-21cups-browsed20:28:082
4003184992415,7cyclictest4026486-21gltestperf20:34:032
4003183992419,2cyclictest0-21swapper/120:50:011
4003183992419,2cyclictest0-21swapper/120:50:011
4003183992415,6cyclictest4015062-21unixbench_multi19:49:271
400318299241,19cyclictest4166987-21kernelversion23:59:100
4003183992316,3cyclictest0-21swapper/119:30:011
4003183992314,7cyclictest1-21systemd20:30:011
4003182992318,3cyclictest151ktimers/020:54:120
4003182992317,4cyclictest955-21mta-sts-daemon20:49:040
4003182992317,4cyclictest955-21mta-sts-daemon20:49:040
4003182992317,4cyclictest4029189-21cron20:45:010
4003182992317,4cyclictest4007632-21sed19:24:290
4003184992212,7cyclictest4007110-21ntp_kernel_pll_19:24:182
400318299221,18cyclictest4011663-21ntpq19:39:200
400318599214,11cyclictest0-21swapper/323:34:133
400318599214,11cyclictest0-21swapper/323:34:133
400318599203,9cyclictest0-21swapper/300:24:183
400318599203,9cyclictest0-21swapper/300:24:183
400318599203,9cyclictest0-21swapper/300:04:153
400318599202,9cyclictest0-21swapper/322:04:213
400318299202,15cyclictest421-21avahi-daemon20:28:410
4003182992015,2cyclictest0-21swapper/019:10:010
4003182992015,2cyclictest0-21swapper/019:10:010
400318599193,8cyclictest0-21swapper/322:09:273
400318599193,7cyclictest0-21swapper/321:54:123
400318599193,7cyclictest0-21swapper/321:54:123
4003184991912,3cyclictest0-21swapper/219:10:012
4003184991912,3cyclictest0-21swapper/219:10:012
4003183991914,2cyclictest0-21swapper/119:10:021
4003183991914,2cyclictest0-21swapper/119:10:021
400318599183,8cyclictest0-21swapper/323:09:163
400318599183,8cyclictest0-21swapper/323:09:163
400318599183,8cyclictest0-21swapper/300:14:123
400318599183,7cyclictest0-21swapper/323:39:183
400318599183,6cyclictest0-21swapper/300:32:523
400318599183,6cyclictest0-21swapper/300:32:523
400318599182,9cyclictest0-21swapper/321:19:133
400318599182,7cyclictest0-21swapper/300:39:173
400318599182,7cyclictest0-21swapper/300:39:173
400318599181,9cyclictest0-21swapper/321:59:123
400318599181,9cyclictest0-21swapper/321:14:133
400318599181,8cyclictest0-21swapper/319:54:143
400318599181,8cyclictest0-21swapper/319:54:143
400318599174,7cyclictest0-21swapper/323:49:133
400318599174,7cyclictest0-21swapper/320:09:153
400318599173,9cyclictest0-21swapper/322:59:143
400318599173,6cyclictest0-21swapper/300:07:223
400318599172,9cyclictest0-21swapper/322:44:153
400318599172,8cyclictest0-21swapper/322:39:183
400318599172,8cyclictest0-21swapper/321:49:173
400318599172,8cyclictest0-21swapper/321:49:173
400318599172,7cyclictest0-21swapper/323:29:163
400318599172,7cyclictest0-21swapper/323:29:163
400318599172,6cyclictest0-21swapper/300:19:113
400318599172,6cyclictest0-21swapper/300:19:113
400318599171,7cyclictest0-21swapper/319:14:173
400318599164,6cyclictest0-21swapper/322:54:173
400318599163,7cyclictest0-21swapper/322:19:143
400318599163,7cyclictest0-21swapper/319:40:013
400318599163,6cyclictest0-21swapper/322:24:063
400318599163,6cyclictest0-21swapper/320:44:113
400318599162,9cyclictest0-21swapper/323:44:133
400318599162,9cyclictest0-21swapper/322:34:193
400318599162,8cyclictest0-21swapper/321:44:203
400318599162,8cyclictest0-21swapper/321:44:203
400318599162,8cyclictest0-21swapper/321:39:073
400318599162,8cyclictest0-21swapper/321:39:073
400318599162,8cyclictest0-21swapper/321:29:153
400318599162,8cyclictest0-21swapper/321:29:153
400318599162,7cyclictest0-21swapper/322:14:143
400318599161,9cyclictest0-21swapper/323:14:133
400318599161,9cyclictest0-21swapper/323:14:133
400318599161,8cyclictest0-21swapper/323:04:143
400318599161,8cyclictest0-21swapper/300:29:193
400318599161,8cyclictest0-21swapper/300:29:193
400318599161,7cyclictest0-21swapper/320:34:093
400318599155,6cyclictest0-21swapper/323:24:123
400318599155,6cyclictest0-21swapper/323:24:123
400318599153,8cyclictest0-21swapper/323:59:073
400318599153,7cyclictest0-21swapper/323:54:163
400318599153,7cyclictest0-21swapper/321:34:123
400318599153,7cyclictest0-21swapper/321:34:123
400318599153,6cyclictest0-21swapper/320:24:113
400318599152,6cyclictest0-21swapper/320:04:183
400318599151,9cyclictest0-21swapper/322:29:203
400318599151,8cyclictest0-21swapper/323:19:163
400318599151,8cyclictest0-21swapper/321:04:173
400318599151,7cyclictest0-21swapper/319:49:153
400318599144,6cyclictest0-21swapper/321:24:193
400318599144,6cyclictest0-21swapper/321:24:193
400318599144,4cyclictest0-21swapper/321:09:153
400318599142,7cyclictest0-21swapper/322:49:113
400318599133,6cyclictest0-21swapper/320:49:143
400318599133,6cyclictest0-21swapper/320:49:143
400318599133,5cyclictest0-21swapper/320:59:113
400318599133,5cyclictest0-21swapper/319:34:113
400318599132,6cyclictest0-21swapper/319:29:063
400318599131,7cyclictest0-21swapper/320:19:123
400318599131,6cyclictest0-21swapper/320:54:123
400318599131,6cyclictest0-21swapper/320:14:163
400318599121,7cyclictest0-21swapper/319:24:093
400318599121,6cyclictest0-21swapper/319:59:143
400318599120,5cyclictest0-21swapper/319:19:123
400318599111,6cyclictest0-21swapper/319:44:163
400318599111,5cyclictest0-21swapper/320:37:113
400318599101,5cyclictest0-21swapper/320:30:013
40031859991,4cyclictest0-21swapper/319:10:023
40031859991,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional