You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 03:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Thu Sep 18, 2025 00:40:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4065047996337,23cyclictest0-21swapper/000:00:010
4065048996246,12cyclictest19293-21timerandwakeup23:43:471
4065049995342,7cyclictest422-21dbus-daemon23:03:282
406504799522,24cyclictest4145740-21ssh22:13:350
406504799522,24cyclictest4145740-21ssh22:13:350
4065047995214,17cyclictest3932-21ssh23:23:390
4065047995214,17cyclictest3932-21ssh23:23:390
4065048995140,5cyclictest0-21swapper/100:23:361
4065048995140,5cyclictest0-21swapper/100:23:361
406504799502,41cyclictest4142059-21ssh22:08:380
406504799502,41cyclictest4142059-21ssh22:08:380
4065048994939,6cyclictest56068-21systemd-run00:40:011
4065047994924,22cyclictest46188-21sendmail_mailqu00:23:430
4065047994924,22cyclictest46188-21sendmail_mailqu00:23:430
4065049994840,5cyclictest4154507-21latency_hist22:25:022
4065048994839,5cyclictest0-21swapper/122:33:251
4065048994838,6cyclictest422-21dbus-daemon23:55:011
4065047994825,20cyclictest4102261-21latency21:13:290
406504799482,35cyclictest39729-21ssh00:13:460
4065047994738,4cyclictest0-21swapper/022:23:400
4065049994638,5cyclictest4069571-21systemd-run19:25:012
4065048994633,7cyclictest39602-21ssh00:13:411
406504799469,11cyclictest151ktimers/020:48:350
406504799463,23cyclictest4183062-21ssh23:03:380
406504799462,17cyclictest52249-21ssh00:33:420
4065047994620,3cyclictest0-21swapper/022:28:260
4065049994521,18cyclictest422-21dbus-daemon00:33:382
4065047994537,5cyclictest4111092-21cron21:25:010
4065047994537,5cyclictest4111092-21cron21:25:010
4065047994524,4cyclictest1457-21ssh23:20:020
4065047994519,4cyclictest4176846-21latency_hist22:55:020
4065047994434,5cyclictest0-21swapper/021:55:010
4065047994421,4cyclictest4171142-21aten_r2power_fr22:48:240
4065047994419,4cyclictest4122044-21grep21:40:010
4065047994418,7cyclictest266-21systemd-journal22:15:030
4065049994335,5cyclictest422-21dbus-daemon22:50:022
4065049994335,5cyclictest4111098-21sh21:25:012
4065049994335,5cyclictest4111098-21sh21:25:012
4065049994335,4cyclictest956-21mta-sts-daemon23:48:332
4065049994334,6cyclictest39190-21grep00:13:332
4065048994334,5cyclictest4154456-21ssh22:25:011
4065048994330,4cyclictest171rcu_preempt19:43:341
4065047994322,4cyclictest2329964-21cupsd21:28:340
4065047994322,4cyclictest2329964-21cupsd21:28:340
406504799432,17cyclictest33815-21ssh00:03:380
4065049994228,8cyclictest4102470-21sh21:13:342
406504999422,33cyclictest4090012-21sshd20:33:382
4065047994234,4cyclictest0-21swapper/023:53:380
406504799422,20cyclictest4124522-21ssh21:43:360
4065049994134,3cyclictest0-21swapper/223:33:432
4065049994132,6cyclictest4080051-21systemd-run20:00:012
4065049994132,5cyclictest15639-21unixbench_singl23:38:492
4065049994131,7cyclictest2329966-21cups-browsed23:08:362
4065049994130,7cyclictest4179870-21unixbench_multi22:58:482
4065049994124,12cyclictest5282-21sh23:25:032
4065049994124,12cyclictest5282-21sh23:25:032
4065048994133,5cyclictest5087-21munin-plugin-st23:25:011
4065048994133,5cyclictest5087-21munin-plugin-st23:25:011
406504899413,29cyclictest4171945-21ssh22:48:411
4065047994133,5cyclictest2329966-21cups-browsed22:43:060
4065047994133,4cyclictest4169312-21kworker/u8:18-events_unbound23:08:390
4065049994031,5cyclictest4092023-21systemctl20:40:012
406504999402,30cyclictest4128445-21ssh21:48:392
4065048994032,5cyclictest4138904-21unixbench_multi22:03:521
4065048994030,7cyclictest4069565-21systemd-run19:25:011
4065048994030,6cyclictest4113406-21missed_timers21:28:371
4065048994030,6cyclictest4113406-21missed_timers21:28:371
4065047994031,4cyclictest151ktimers/023:48:380
4065047994030,7cyclictest4135772-21ssh22:00:010
4065049993933,3cyclictest0-21swapper/222:00:012
4065049993931,5cyclictest31279-40ogrotate00:00:012
4065049993929,6cyclictest4139522-21cron22:05:012
4065049993929,4cyclictest0-21swapper/221:33:402
4065049993926,10cyclictest4113285-21kernelversion21:28:352
4065049993926,10cyclictest4113285-21kernelversion21:28:352
4065048993930,6cyclictest4096534-21systemd-run20:55:021
4065048993930,5cyclictest0-21swapper/121:40:011
4065048993929,5cyclictest0-21swapper/123:08:361
4065047993933,4cyclictest4097905-21tail20:58:450
4065049993831,4cyclictest4161246-21grep22:33:482
4065049993830,5cyclictest56035-21cat00:40:012
4065049993830,5cyclictest4089045-21true20:30:022
4065049993829,6cyclictest421-21avahi-daemon22:51:222
4065048993830,5cyclictest4169286-21latency_hist22:45:011
4065048993829,6cyclictest4141633-21fschecks_count22:08:311
4065048993829,6cyclictest4141633-21fschecks_count22:08:311
4065048993828,6cyclictest421-21avahi-daemon00:10:031
4065048993826,9cyclictest51437-21/usr/sbin/munin00:33:481
4065049993730,4cyclictest4063676-21kworker/u8:4-xprtiod21:20:012
4065049993729,5cyclictest4146996-21cron22:15:002
4065049993729,5cyclictest4146996-21cron22:15:002
4065048993730,4cyclictest2-21kthreadd19:15:021
4065048993729,5cyclictest4128165-21needreboot21:48:301
4065048993729,5cyclictest29690-21fschecks_time23:58:301
4065048993727,6cyclictest421-21avahi-daemon00:18:081
406504899372,27cyclictest4134561-21ssh21:58:331
4065049993627,4cyclictest0-21swapper/219:18:392
4065049993626,6cyclictest2329966-21cups-browsed23:53:052
4065049993624,9cyclictest4120840-21ntp_kernel_err21:38:402
4065049993623,6cyclictest0-21swapper/219:53:422
406504899363,24cyclictest4164355-21ssh22:38:381
4065048993630,3cyclictest261rcuc/122:58:401
4065048993618,9cyclictest171rcu_preempt20:08:351
4065048993614,9cyclictest28-21ksoftirqd/123:13:381
406504899361,29cyclictest4109955-21ssh21:23:381
406504899361,29cyclictest4109955-21ssh21:23:381
4065047993627,6cyclictest4087609-21latency_hist20:25:020
4065047993626,6cyclictest151ktimers/000:28:370
4065047993626,6cyclictest151ktimers/000:28:370
4065049993527,5cyclictest1392-21basename23:20:012
4065049993525,6cyclictest421-21avahi-daemon22:35:352
4065049993525,4cyclictest0-21swapper/223:43:372
406504999352,27cyclictest46068-21ssh00:23:402
406504999352,27cyclictest46068-21ssh00:23:402
4065049993520,10cyclictest4099790-21diskmemload23:28:352
4065049993520,10cyclictest4099790-21diskmemload23:28:352
4065048993527,5cyclictest22199-21fschecks_count23:48:291
4065048993527,5cyclictest1439-21latency_hist23:20:021
4065048993526,6cyclictest0-21swapper/122:20:011
4065048993525,6cyclictest4092027-21idleruntime-cro20:40:011
4065048993517,3cyclictest171rcu_preempt20:48:381
4065047993529,4cyclictest4065046-21cyclictest23:38:400
4065047993528,4cyclictest4077054-21systemd-run19:50:010
4065047993525,7cyclictest11085-21needreboot23:33:300
406504999346,17cyclictest4190439-21ssh23:13:372
4065049993424,6cyclictest0-21swapper/220:10:012
4065049993423,8cyclictest4075543-21cut19:45:022
406504899343,23cyclictest657-21snmpd00:28:391
406504899343,23cyclictest657-21snmpd00:28:391
4065048993428,3cyclictest0-21swapper/123:40:011
4065048993428,3cyclictest0-21swapper/121:35:001
4065048993425,6cyclictest453-21systemd-logind22:30:011
4065047993426,6cyclictest0-21swapper/020:05:010
4065047993425,6cyclictest31392-21cups-browsed00:17:480
4065047993417,12cyclictest12450irq/32-eth023:28:400
4065047993417,12cyclictest12450irq/32-eth023:28:400
4065049993326,5cyclictest0-21swapper/219:15:022
4065049993325,5cyclictest49598-21unixbench-2d00:28:482
4065049993325,5cyclictest49598-21unixbench-2d00:28:482
4065049993325,5cyclictest4070939-21unixbench_multi19:28:422
4065049993322,6cyclictest4149246-21kernelversion22:18:302
4065049993321,9cyclictest4073411-21ntp_kernel_pll_19:38:342
4065049993318,9cyclictest4084002-21sshd20:13:392
4065049993318,9cyclictest4084002-21sshd20:13:392
4065048993324,6cyclictest2329966-21cups-browsed23:31:451
4065048993323,6cyclictest4090062-21proc_pri20:33:401
4065048993322,8cyclictest4097394-21ntp_kernel_pll_20:58:341
4065048993320,3cyclictest171rcu_preempt21:18:371
4065048993315,6cyclictest171rcu_preempt22:53:401
4065048993312,16cyclictest422-21dbus-daemon21:10:011
4065047993327,4cyclictest4160215-21fschecks_time22:33:280
4065047993327,4cyclictest36414-21fschecks_time00:08:340
4065047993327,3cyclictest0-21swapper/023:43:290
4065047993325,5cyclictest2329966-21cups-browsed23:14:510
4065047993318,12cyclictest55369-21sendmail_mailqu00:38:450
4065047993317,11cyclictest151ktimers/022:58:420
4065049993225,3cyclictest0-21swapper/219:48:352
4065049993224,5cyclictest4157530-21unixbench_singl22:28:462
4065049993223,4cyclictest0-21swapper/222:08:402
4065049993223,4cyclictest0-21swapper/222:08:402
406504999321,21cyclictest33794-21ssh00:03:382
406504899326,12cyclictest31392-21cups-browsed00:00:381
4065048993226,3cyclictest0-21swapper/121:15:011
4065048993226,3cyclictest0-21swapper/119:35:021
4065048993219,3cyclictest171rcu_preempt19:40:011
4065048993214,3cyclictest171rcu_preempt23:28:341
4065048993214,3cyclictest171rcu_preempt23:28:341
4065048993212,11cyclictest957-21mta-sts-daemon20:23:391
406504999313,22cyclictest565-21gmain21:03:372
4065049993126,2cyclictest0-21swapper/221:10:012
4065049993124,3cyclictest0-21swapper/221:43:382
4065049993123,4cyclictest0-21swapper/222:43:322
4065048993126,3cyclictest4083045-21kworker/u8:3-events_unbound20:18:391
4065048993126,3cyclictest4083045-21kworker/u8:3-events_unbound20:18:391
4065048993125,3cyclictest0-21swapper/122:15:001
4065048993125,3cyclictest0-21swapper/122:15:001
4065048993123,4cyclictest0-21swapper/121:55:011
4065048993121,6cyclictest4078097-21processes19:53:421
4065048993117,3cyclictest171rcu_preempt19:28:351
406504799313,22cyclictest4164269-21ssh22:38:350
4065047993124,4cyclictest4074576-21fschecks_count19:43:270
4065047993120,8cyclictest0-21swapper/020:33:250
4065047993118,8cyclictest0-21swapper/019:18:390
406504999303,19cyclictest42819-21ssh00:18:412
4065049993022,4cyclictest0-21swapper/200:08:442
406504899305,5cyclictest4084026-21ntpq20:13:401
406504899305,5cyclictest4084026-21ntpq20:13:401
4065048993023,4cyclictest0-21swapper/120:05:011
4065048993022,5cyclictest956-21mta-sts-daemon21:03:431
4065048993022,4cyclictest0-21swapper/121:44:101
4065048993018,3cyclictest171rcu_preempt19:48:381
4065048993017,3cyclictest171rcu_preempt20:43:351
4065048993016,3cyclictest4182273-21ssh23:03:211
4065047993025,3cyclictest4118461-21systemd21:35:020
4065047993024,4cyclictest4085929-21tail20:18:460
4065047993024,4cyclictest4085929-21tail20:18:460
4065047993023,4cyclictest0-21swapper/021:48:340
4065047993021,7cyclictest4049464-21kworker/u8:13-xprtiod19:30:010
4065047993019,7cyclictest151ktimers/019:53:350
4065048992921,5cyclictest4067964-21unixbench_singl19:18:491
4065048992916,3cyclictest171rcu_preempt20:28:381
4065047992922,4cyclictest4082935-21tail20:08:460
4065047992921,5cyclictest4074073-21unin-run19:40:020
4065047992920,5cyclictest0-21swapper/021:18:410
4065047992919,8cyclictest4091341-21needreboot20:38:320
4065047992916,10cyclictest0-21swapper/020:30:010
4065049992818,6cyclictest0-21swapper/220:43:342
4065048992819,6cyclictest4079916-21unixbench_multi19:58:441
406504799282,20cyclictest4138379-21ssh22:03:420
4065047992821,5cyclictest959-21mta-sts-daemon19:23:450
4065049992717,6cyclictest4072605-21latency_hist19:35:022
406504999271,18cyclictest4087026-21sshd20:23:382
4065047992721,4cyclictest4099793-21munin-run21:10:010
4065047992721,4cyclictest4079052-21fschecks_count19:58:260
4065049992619,4cyclictest0-21swapper/220:05:012
4065049992619,3cyclictest0-21swapper/220:58:382
4065049992617,4cyclictest0-21swapper/221:50:462
4065047992621,2cyclictest0-21swapper/019:15:010
4065047992611,12cyclictest0-21swapper/020:43:450
406504999254,12cyclictest437-21mta-sts-daemon20:18:342
406504999254,12cyclictest437-21mta-sts-daemon20:18:342
406504999253,14cyclictest421-21avahi-daemon20:53:432
406504999251,16cyclictest4094501-21sshd20:48:382
4065047992419,3cyclictest957-21mta-sts-daemon19:35:000
406504799232,15cyclictest4098937-21ntpq21:03:400
4065047992312,7cyclictest4083285-21/usr/sbin/munin20:13:240
4065047992312,7cyclictest4083285-21/usr/sbin/munin20:13:240
4065047992215,4cyclictest2329966-21cups-browsed20:50:320
406504899217,2cyclictest171rcu_preempt19:10:021
406504999198,7cyclictest421-21avahi-daemon19:10:022
406505099183,9cyclictest0-21swapper/322:23:383
406505099182,9cyclictest0-21swapper/322:53:383
406505099182,9cyclictest0-21swapper/300:08:423
406505099182,8cyclictest0-21swapper/300:28:363
406505099182,8cyclictest0-21swapper/300:28:363
406504799188,6cyclictest422-21dbus-daemon19:10:020
406505099171,9cyclictest0-21swapper/322:48:383
406505099163,7cyclictest0-21swapper/323:18:353
406505099163,7cyclictest0-21swapper/300:33:433
406505099162,8cyclictest0-21swapper/323:13:383
406505099162,8cyclictest0-21swapper/321:58:393
406505099162,8cyclictest0-21swapper/300:18:433
406505099161,9cyclictest0-21swapper/323:48:363
406505099161,7cyclictest0-21swapper/322:38:263
406505099154,7cyclictest0-21swapper/300:03:323
406505099153,8cyclictest0-21swapper/323:38:293
406505099153,8cyclictest0-21swapper/322:03:513
406505099153,8cyclictest0-21swapper/321:33:373
406505099153,7cyclictest0-21swapper/322:13:403
406505099153,7cyclictest0-21swapper/322:13:403
406505099152,8cyclictest0-21swapper/321:28:433
406505099152,8cyclictest0-21swapper/321:28:433
406505099151,9cyclictest0-21swapper/321:48:353
406505099151,7cyclictest0-21swapper/320:03:323
406505099151,6cyclictest0-21swapper/319:58:333
406505099143,7cyclictest0-21swapper/321:23:383
406505099143,7cyclictest0-21swapper/321:23:383
406505099143,6cyclictest0-21swapper/321:43:423
406505099142,7cyclictest0-21swapper/321:13:393
406505099142,6cyclictest0-21swapper/323:53:363
406505099142,6cyclictest0-21swapper/320:53:373
406505099141,8cyclictest0-21swapper/322:58:423
406505099141,8cyclictest0-21swapper/300:38:413
406505099141,7cyclictest0-21swapper/323:33:303
406505099141,6cyclictest0-21swapper/320:58:333
406505099133,8cyclictest0-21swapper/322:43:303
406505099133,5cyclictest0-21swapper/320:43:343
406505099132,7cyclictest0-21swapper/323:23:393
406505099132,7cyclictest0-21swapper/323:23:393
406505099132,7cyclictest0-21swapper/320:08:333
406505099132,6cyclictest0-21swapper/319:53:423
406505099131,8cyclictest0-21swapper/323:28:323
406505099131,8cyclictest0-21swapper/323:28:323
406505099131,8cyclictest0-21swapper/323:03:253
406505099131,8cyclictest0-21swapper/322:08:403
406505099131,8cyclictest0-21swapper/322:08:403
406505099131,7cyclictest0-21swapper/323:43:343
406505099131,7cyclictest0-21swapper/323:08:373
406505099131,7cyclictest0-21swapper/322:28:273
406505099131,7cyclictest0-21swapper/322:18:243
406505099131,7cyclictest0-21swapper/321:55:013
406505099131,7cyclictest0-21swapper/321:38:423
406505099131,6cyclictest0-21swapper/320:38:373
406505099131,6cyclictest0-21swapper/319:33:343
406505099131,6cyclictest0-21swapper/319:23:343
406505099122,6cyclictest0-21swapper/323:58:353
406505099122,5cyclictest0-21swapper/319:48:383
406505099122,5cyclictest0-21swapper/319:18:363
406505099121,7cyclictest0-21swapper/322:33:343
406505099121,7cyclictest0-21swapper/321:18:323
406505099121,6cyclictest0-21swapper/321:10:013
406505099121,6cyclictest0-21swapper/320:23:333
406505099121,6cyclictest0-21swapper/320:18:403
406505099121,6cyclictest0-21swapper/320:18:403
406505099121,6cyclictest0-21swapper/300:23:283
406505099121,6cyclictest0-21swapper/300:23:283
406505099121,5cyclictest0-21swapper/320:33:323
406505099121,4cyclictest0-21swapper/321:03:373
406505099120,7cyclictest0-21swapper/319:38:323
406505099111,7cyclictest0-21swapper/300:13:303
406505099111,6cyclictest0-21swapper/320:28:363
406505099111,6cyclictest0-21swapper/319:15:013
406505099111,5cyclictest0-21swapper/320:13:333
406505099111,5cyclictest0-21swapper/320:13:333
406505099103,5cyclictest0-21swapper/319:43:343
406505099101,6cyclictest0-21swapper/319:28:333
406505099101,5cyclictest0-21swapper/320:47:383
40650509971,3cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional