You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-21 - 21:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack2slot7s.osadl.org (updated Tue Apr 21, 2026 12:43:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3677880996752,10cyclictest548-21gdbus12:00:451
3677880996752,10cyclictest548-21gdbus12:00:451
3677881994839,6cyclictest3786821-21munin-run10:40:002
3677880994834,11cyclictest3834890-21sshd11:39:331
3677880994636,6cyclictest3883650-21sendmail12:40:011
3677880994636,6cyclictest3799184-21latency_hist10:55:021
3677879994431,3cyclictest0-21swapper/010:00:010
3677879994430,10cyclictest2852470-21cups-browsed12:25:420
3677881994332,3cyclictest0-21swapper/207:26:372
3677881994332,3cyclictest0-21swapper/207:26:372
3677880994333,6cyclictest3754388-21cron10:00:001
3677881994235,4cyclictest0-21swapper/211:05:462
3677880994234,5cyclictest3828319-21netstat11:30:371
3677880994234,4cyclictest0-21swapper/108:35:391
3677880994230,9cyclictest3816823-21unixbench_multi11:15:521
3677880994230,9cyclictest3816823-21unixbench_multi11:15:521
3677879994231,4cyclictest3844695-21mailstats11:50:500
367788199416,28cyclictest3796180-21ssh10:50:422
3677881994135,3cyclictest0-21swapper/209:15:022
3677881994132,6cyclictest420-21avahi-daemon10:56:382
3677880994133,5cyclictest3746957-21aten_r2power_fr09:50:311
367788199401,33cyclictest3723189-21ssh09:20:402
367788099406,6cyclictest171rcu_preempt12:05:411
367788099406,6cyclictest171rcu_preempt12:05:411
3677880994030,7cyclictest421-21dbus-daemon11:50:011
3677880994029,7cyclictest3718981-21missed_timers09:15:371
3677881993931,5cyclictest3847206-21cron11:55:012
3677881993930,6cyclictest3852052-21fschecks_count12:00:312
3677881993930,6cyclictest3852052-21fschecks_count12:00:312
367788199392,26cyclictest659-21snmpd07:35:462
3677880993929,5cyclictest2852470-21cups-browsed10:40:411
3677879993931,5cyclictest3723796-21grep09:20:550
3677879993930,7cyclictest3851316-21systemd12:00:020
367788199383,28cyclictest3816309-21ssh11:15:422
367788199383,28cyclictest3816309-21ssh11:15:422
3677881993832,3cyclictest0-21swapper/209:10:012
3677881993830,5cyclictest421-21dbus-daemon09:55:012
3677881993828,7cyclictest3695875-21cut08:10:012
3677881993818,15cyclictest421-21dbus-daemon11:30:372
3677880993831,4cyclictest0-21swapper/111:40:451
3677879993832,4cyclictest3786849-21sendmail10:40:000
3677879993831,4cyclictest3832030-21fschecks_time11:35:350
367787999382,29cyclictest3868757-21sh12:20:430
367787999381,32cyclictest3747737-21rm09:50:510
3677881993729,5cyclictest3871545-21ssh12:25:012
3677881993728,5cyclictest3742366-21munin-run09:45:012
3677880993730,3cyclictest0-21swapper/110:35:001
3677880993730,3cyclictest0-21swapper/110:35:001
3677880993729,4cyclictest0-21swapper/110:40:001
3677880993727,7cyclictest3766571-21munin-plugin-st10:15:011
3677879993731,4cyclictest3731540-21sendmail_mailqu09:30:460
3677879993730,4cyclictest3746407-21ssh09:50:010
3677879993730,4cyclictest3746407-21ssh09:50:010
3677879993726,3cyclictest3684849-21ntpq07:30:440
3677881993628,5cyclictest3835154-21sendmail11:40:012
3677881993628,4cyclictest0-21swapper/210:45:442
3677881993626,6cyclictest1055-21mta-sts-daemon09:55:352
3677880993628,5cyclictest3865172-21grep12:15:511
3677880993628,5cyclictest3723285-21ntp_kernel_pll_09:20:431
3677880993628,3cyclictest0-21swapper/111:20:411
3677880993628,3cyclictest0-21swapper/111:20:411
3677880993626,7cyclictest3679380-21true07:15:011
367787999364,24cyclictest3780098-21ssh10:30:470
367787999364,24cyclictest3780098-21ssh10:30:470
367787999363,25cyclictest3836436-21ssh11:40:420
3677879993629,4cyclictest3735998-21unixbench_multi09:35:520
3677879993626,4cyclictest12350irq/32-eth012:00:430
3677879993626,4cyclictest12350irq/32-eth012:00:430
3677879993618,3cyclictest171rcu_preempt09:25:390
3677881993529,2cyclictest0-21swapper/212:15:332
3677881993527,5cyclictest3883674-21idleruntime-cro12:40:012
3677881993527,5cyclictest3730878-21fschecks_count09:30:322
3677881993527,4cyclictest3710896-21kworker/u8:5-xprtiod09:25:452
3677881993526,6cyclictest420-21avahi-daemon10:15:012
3677881993526,6cyclictest3836977-21unixbench_multi11:40:522
3677881993526,6cyclictest3722060-21latency_hist09:20:022
3677881993526,6cyclictest2852470-21cups-browsed11:55:592
3677881993524,4cyclictest351ktimers/212:09:352
3677881993524,4cyclictest351ktimers/212:09:352
3677881993523,9cyclictest3710862-21idleruntime-cro09:00:012
3677881993522,3cyclictest171rcu_preempt07:15:442
3677881993521,3cyclictest171rcu_preempt11:25:402
367788199351,27cyclictest3812280-21ssh11:10:442
3677881993511,10cyclictest36-21ksoftirqd/207:50:012
367788099355,22cyclictest3872823-21ssh12:25:451
367788099353,22cyclictest3808200-21ssh11:05:431
3677880993527,6cyclictest0-21swapper/112:35:011
3677880993526,6cyclictest3770671-21ssh10:20:021
3677880993525,7cyclictest421-21dbus-daemon09:15:011
3677880993525,7cyclictest1017-21sendmail-mta11:55:411
3677880993524,8cyclictest3727558-21mailstats09:25:471
3677880993523,8cyclictest3692859-21munin-run08:00:011
3677880993519,13cyclictest271ktimers/108:00:381
367788099351,26cyclictest3868725-21ssh12:20:421
3677879993528,5cyclictest420-21avahi-daemon10:18:520
3677879993528,5cyclictest3706366-21cron08:45:010
3677879993528,4cyclictest3717927-21ssh09:15:010
3677879993525,7cyclictest3799803-21fschecks_time10:55:370
3677881993427,3cyclictest0-21swapper/211:45:432
3677881993427,3cyclictest0-21swapper/211:20:422
3677881993427,3cyclictest0-21swapper/211:20:422
3677881993426,5cyclictest3735767-21switchtime09:35:512
3677881993426,5cyclictest3704438-21systemctl08:35:472
3677881993426,5cyclictest3693409-21fschecks_time08:00:332
3677881993425,6cyclictest3767827-21sendmail-mta10:15:412
3677881993424,6cyclictest3873117-21switchtime12:25:532
3677881993423,7cyclictest3876024-21/usr/sbin/munin12:30:332
3677881993421,5cyclictest171rcu_preempt09:45:342
3677881993421,5cyclictest171rcu_preempt09:45:342
3677880993428,3cyclictest0-21swapper/110:25:021
3677880993426,5cyclictest3847243-21systemd-run11:55:011
3677879993429,3cyclictest3721999-21grep09:20:010
3677879993419,10cyclictest12350irq/32-eth010:45:460
3677879993413,4cyclictest171rcu_preempt11:25:430
3677881993327,4cyclictest1055-21mta-sts-daemon10:25:452
3677881993326,4cyclictest0-21swapper/210:00:412
3677881993324,5cyclictest3779065-21sshd10:30:132
3677881993324,5cyclictest3779065-21sshd10:30:132
3677881993322,3cyclictest171rcu_preempt09:00:052
3677881993321,3cyclictest171rcu_preempt10:20:412
3677881993321,3cyclictest171rcu_preempt07:30:332
367788099333,24cyclictest3824400-21ssh11:25:421
3677880993327,3cyclictest0-21swapper/111:00:011
3677880993324,5cyclictest0-21swapper/109:35:021
367787999333,23cyclictest3776001-21ssh10:25:440
3677879993326,5cyclictest3796759-21unixbench_multi10:50:520
3677879993326,5cyclictest0-21swapper/012:20:000
3677879993325,5cyclictest2852470-21cups-browsed12:12:190
3677879993318,5cyclictest1054-21mta-sts-daemon11:48:050
3677879993315,11cyclictest3812907-21ssh11:10:560
3677881993224,5cyclictest3691362-21munin-run07:55:012
367788099327,6cyclictest3713865-21diskmemload10:00:441
3677880993226,3cyclictest0-21swapper/108:45:531
3677880993224,5cyclictest3859876-21ssh12:10:291
3677880993224,5cyclictest3776452-21unixbench-2d10:25:521
3677880993223,6cyclictest420-21avahi-daemon07:45:461
3677880993222,8cyclictest3700396-21sh08:25:011
3677880993222,8cyclictest3700396-21sh08:25:011
3677880993222,6cyclictest3791525-21aten_r2power_vo10:45:301
3677879993225,4cyclictest3713856-21cron09:10:000
3677879993224,6cyclictest1055-21mta-sts-daemon11:17:020
3677879993224,6cyclictest1055-21mta-sts-daemon11:17:020
3677879993221,8cyclictest3708673-21needreboot08:50:370
367788199318,8cyclictest36-21ksoftirqd/211:00:412
367788199318,7cyclictest36-21ksoftirqd/208:15:382
3677881993123,3cyclictest0-21swapper/208:45:442
3677881993119,3cyclictest171rcu_preempt07:10:392
3677881993117,3cyclictest171rcu_preempt08:20:502
3677881993117,3cyclictest171rcu_preempt08:20:502
3677880993125,3cyclictest0-21swapper/107:26:391
3677880993125,3cyclictest0-21swapper/107:26:391
3677880993123,5cyclictest3713856-21cron09:10:001
3677880993123,5cyclictest3688400-21systemd-run07:45:021
3677880993123,4cyclictest0-21swapper/111:10:311
3677880993121,8cyclictest0-21swapper/109:40:381
367787999313,22cyclictest3856582-21sh12:05:410
367787999313,22cyclictest3856582-21sh12:05:410
3677879993125,4cyclictest3713865-21diskmemload12:30:490
3677879993125,4cyclictest2852470-21cups-browsed08:55:300
3677879993123,6cyclictest3688396-21sh07:45:010
3677879993123,4cyclictest151ktimers/007:55:440
3677879993121,3cyclictest171rcu_preempt10:40:330
3677879993120,8cyclictest3763608-21missed_timers10:10:360
3677879993120,8cyclictest3698860-21cron08:20:000
3677879993112,12cyclictest1051-21mta-sts-daemon11:09:280
367788199309,5cyclictest36-21ksoftirqd/212:10:222
367788199307,13cyclictest1051-21mta-sts-daemon10:05:422
367788199306,9cyclictest36-21ksoftirqd/208:30:412
367788199305,6cyclictest36-21ksoftirqd/208:40:462
3677881993022,5cyclictest3682307-21unixbench_singl07:20:502
3677880993022,5cyclictest3712363-21cron09:05:011
3677880993022,3cyclictest0-21swapper/109:35:401
3677880993017,10cyclictest271ktimers/107:20:441
367787999305,10cyclictest191rcuc/008:35:020
3677879993023,3cyclictest0-21swapper/010:05:330
3677879993020,7cyclictest3738806-21aten_r2power_cu09:40:280
3677879993019,5cyclictest3807259-21kworker/u8:9+xprtiod11:30:440
367788199298,5cyclictest351ktimers/207:55:392
367788199297,8cyclictest36-21ksoftirqd/210:41:142
3677880992923,3cyclictest0-21swapper/109:50:011
3677880992923,3cyclictest0-21swapper/109:50:011
3677880992922,3cyclictest0-21swapper/110:05:381
3677880992920,6cyclictest420-21avahi-daemon08:40:581
3677880992912,12cyclictest624-21lldpd11:02:261
3677879992920,7cyclictest3673884-21kworker/u8:0-rpciod07:26:360
3677879992920,7cyclictest3673884-21kworker/u8:0-rpciod07:26:360
3677879992920,2cyclictest171rcu_preempt11:00:320
3677879992918,2cyclictest0-21swapper/008:30:000
3677879992917,3cyclictest3821630-21ssh11:22:070
3677879992917,3cyclictest3821630-21ssh11:22:070
367788199282,6cyclictest36-21ksoftirqd/208:25:362
3677881992820,5cyclictest3677884-21systemd-run07:10:012
3677881992818,3cyclictest171rcu_preempt08:50:382
3677881992810,14cyclictest341rcuc/207:40:272
3677880992822,3cyclictest0-21swapper/109:00:011
3677880992820,5cyclictest3703207-21unixbench-2d08:30:521
367787999286,11cyclictest14-21ksoftirqd/012:35:420
3677879992820,6cyclictest3694928-21sshd08:05:330
3677879992818,3cyclictest171rcu_preempt08:35:370
367788199274,8cyclictest36-21ksoftirqd/208:10:382
3677880992722,2cyclictest0-21swapper/108:20:011
3677880992721,3cyclictest0-21swapper/108:15:011
3677880992719,5cyclictest3695892-21systemd-run08:10:011
3677880992717,7cyclictest3680865-21idleruntime-cro07:20:001
3677879992718,3cyclictest3686909-21kworker/u8:3+xprtiod08:00:400
3677879992717,3cyclictest3758253-21ssh10:04:350
3677879992717,3cyclictest3711998-21sensors_fan09:00:490
367787999271,12cyclictest3762551-21kworker/u8:16+rpciod10:20:420
3677880992621,2cyclictest0-21swapper/108:30:001
3677880992621,2cyclictest0-21swapper/107:40:011
3677880992610,12cyclictest421-21dbus-daemon08:50:441
367787999262,8cyclictest3680843-21kworker/0:0+events_freezable_power_07:45:360
3677879992621,2cyclictest0-21swapper/007:20:010
3677879992619,2cyclictest171rcu_preempt07:50:420
3677879992613,2cyclictest171rcu_preempt07:20:410
3677880992517,5cyclictest1051-21mta-sts-daemon07:10:021
3677879992517,5cyclictest3696480-21if_wlan008:10:330
3677879992513,3cyclictest171rcu_preempt07:35:450
3677880992418,3cyclictest0-21swapper/107:50:411
3677879992414,3cyclictest171rcu_preempt07:10:010
3677879992413,3cyclictest171rcu_preempt08:20:390
3677879992413,3cyclictest171rcu_preempt08:20:390
3677879992412,3cyclictest171rcu_preempt07:15:010
367787999239,8cyclictest171rcu_preempt08:45:290
367788099221,16cyclictest2852467-21cupsd07:30:441
367788299193,11cyclictest0-21swapper/311:30:433
367788299193,10cyclictest0-21swapper/309:45:443
367788299193,10cyclictest0-21swapper/309:45:443
367788299183,10cyclictest0-21swapper/310:00:383
367788299182,12cyclictest0-21swapper/309:55:403
367788299181,12cyclictest0-21swapper/312:10:443
367788299181,12cyclictest0-21swapper/312:05:393
367788299181,12cyclictest0-21swapper/312:05:393
367788299181,12cyclictest0-21swapper/312:00:443
367788299181,12cyclictest0-21swapper/312:00:443
367788299181,12cyclictest0-21swapper/311:20:403
367788299181,12cyclictest0-21swapper/311:20:403
367788299173,9cyclictest0-21swapper/310:10:343
367788299172,12cyclictest0-21swapper/310:40:413
367788299172,12cyclictest0-21swapper/309:10:343
367788299172,11cyclictest0-21swapper/310:20:433
367788299171,12cyclictest0-21swapper/312:25:473
367788299171,12cyclictest0-21swapper/311:45:483
367788299171,12cyclictest0-21swapper/311:40:433
367788299171,12cyclictest0-21swapper/311:10:413
367788299171,12cyclictest0-21swapper/311:05:463
367788299171,12cyclictest0-21swapper/310:45:393
367788299171,12cyclictest0-21swapper/310:35:443
367788299171,12cyclictest0-21swapper/310:30:393
367788299171,12cyclictest0-21swapper/310:30:393
367788299171,12cyclictest0-21swapper/307:30:383
367788299171,11cyclictest0-21swapper/312:35:333
367788299171,11cyclictest0-21swapper/308:40:403
367788299171,11cyclictest0-21swapper/308:20:373
367788299171,11cyclictest0-21swapper/308:20:373
367788299171,11cyclictest0-21swapper/308:10:383
367788299171,11cyclictest0-21swapper/307:55:403
367788299171,11cyclictest0-21swapper/307:50:393
367788299171,11cyclictest0-21swapper/307:10:403
367788299170,13cyclictest0-21swapper/309:50:453
367788299164,8cyclictest0-21swapper/312:30:423
367788299162,11cyclictest0-21swapper/311:15:373
367788299162,11cyclictest0-21swapper/311:15:373
367788299162,10cyclictest0-21swapper/309:40:283
367788299162,10cyclictest0-21swapper/308:45:433
367788299161,12cyclictest0-21swapper/311:36:143
367788299161,12cyclictest0-21swapper/307:26:383
367788299161,12cyclictest0-21swapper/307:26:383
367788299161,12cyclictest0-21swapper/307:15:413
367788299161,11cyclictest0-21swapper/312:20:303
367788299161,11cyclictest0-21swapper/312:15:393
367788299161,11cyclictest0-21swapper/310:25:393
367788299161,11cyclictest0-21swapper/310:05:383
367788299161,11cyclictest0-21swapper/309:35:393
367788299161,11cyclictest0-21swapper/309:30:383
367788299161,11cyclictest0-21swapper/309:20:333
367788299161,11cyclictest0-21swapper/309:15:333
367788299161,11cyclictest0-21swapper/309:05:403
367788299161,11cyclictest0-21swapper/308:55:403
367788299161,11cyclictest0-21swapper/308:50:383
367788299161,11cyclictest0-21swapper/308:25:403
367788299161,11cyclictest0-21swapper/308:15:383
367788299161,11cyclictest0-21swapper/307:45:393
367788299161,11cyclictest0-21swapper/307:40:403
367788299161,11cyclictest0-21swapper/307:35:403
367788299160,12cyclictest0-21swapper/311:25:453
367788299160,12cyclictest0-21swapper/309:25:373
367788299151,11cyclictest0-21swapper/311:50:333
367788299151,11cyclictest0-21swapper/310:50:313
367788299151,11cyclictest0-21swapper/310:15:313
367788299151,11cyclictest0-21swapper/309:00:393
367788299151,11cyclictest0-21swapper/308:35:333
367788299151,11cyclictest0-21swapper/308:00:303
367788299151,10cyclictest0-21swapper/310:55:233
367788299150,12cyclictest0-21swapper/307:10:013
367788299150,11cyclictest0-21swapper/311:55:443
367788299150,11cyclictest0-21swapper/311:00:403
367788299150,11cyclictest0-21swapper/308:30:413
367788299150,11cyclictest0-21swapper/308:05:373
367788299150,11cyclictest0-21swapper/307:20:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional