You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 02:36
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7.osadl.org (updated Fri Nov 28, 2025 00:43:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
371307099525,44cyclictest3923938-21/usr/sbin/munin00:23:100
371307099512,6cyclictest3753641-21diskmemload22:38:070
371307199422,7cyclictest3862584-21ntpq23:13:081
371307099410,28cyclictest0-21swapper/023:54:480
371307199392,10cyclictest3788577-21ntpq21:53:111
3713070993934,4cyclictest0-21swapper/023:08:100
3713070993724,3cyclictest0-21swapper/000:13:000
371307099362,31cyclictest0-21swapper/000:06:080
3713070993534,1cyclictest0-21swapper/020:08:090
371307099351,23cyclictest3936078-21grep00:38:030
371307199347,22cyclictest3938377-21apt-get00:41:071
371307199332,13cyclictest3830664-21ntpq22:38:071
371307199331,8cyclictest3881007-21sed23:33:071
371307199324,20cyclictest3743425-21sendmail-msp20:51:071
371307199323,11cyclictest3800978-21ssh22:06:051
371307199322,14cyclictest3906103-21packagekitd00:01:031
371307199322,14cyclictest3906103-21packagekitd00:01:031
3713071993211,20cyclictest3742295-21fschecks_count20:43:021
371307099323,11cyclictest0-21swapper/023:13:080
3713070993226,3cyclictest0-21swapper/020:28:140
371307199312,6cyclictest3825833-21sed22:33:061
3713071993113,17cyclictest3848352-21cut22:58:031
371307099312,13cyclictest0-21swapper/000:01:060
3713072993013,13cyclictest111788-21systemd-journal23:26:592
371307199305,3cyclictest0-21swapper/121:33:021
371307199304,3cyclictest3858028-21ntpq23:08:101
371307199303,12cyclictest3737329-21/usr/sbin/munin20:28:031
371307199302,6cyclictest3791389-21ssh21:56:061
371307099302,26cyclictest3753719-21cat21:16:040
371307099301,19cyclictest0-21swapper/023:23:010
371307299294,21cyclictest111788-21systemd-journal20:48:452
371307199298,3cyclictest11050irq/107-eth%d22:13:091
371307199298,3cyclictest11050irq/107-eth%d22:13:091
371307199294,19cyclictest111788-21systemd-journal00:06:141
371307199292,4cyclictest3887986-21latency_hist23:41:031
3713071992915,13cyclictest0-21swapper/123:48:011
371307299280,6cyclictest171rcu_preempt22:58:422
371307199289,15cyclictest619-21mta-sts-daemon23:51:091
371307199284,5cyclictest3798066-21ntp_states22:03:071
371307199283,9cyclictest3722962-21sendmail-msp19:46:071
371307199283,8cyclictest3906155-21ssh00:01:061
371307199281,6cyclictest3752515-21fschecks_count21:13:031
371307199281,6cyclictest3735708-21grep20:23:041
3713071992816,11cyclictest3917224-21unixbench_singl00:13:161
3713071992811,16cyclictest3843762-21cut22:53:031
371307199280,11cyclictest3817365-21grep22:23:131
371307299273,3cyclictest171rcu_preempt19:33:072
371307199275,3cyclictest0-21swapper/100:23:031
371307199273,3cyclictest3779409-21/usr/sbin/munin21:43:091
371307199271,9cyclictest3715321-21tune2fs19:23:051
371307199270,9cyclictest3814543-21cat22:21:031
3713070992713,7cyclictest3835301-21/usr/sbin/munin22:43:080
371307099271,19cyclictest0-21swapper/023:37:590
371307199265,4cyclictest0-21swapper/123:23:091
371307199262,9cyclictest3875975-21sed23:28:051
371307199262,6cyclictest3721802-21grep19:43:011
371307199261,5cyclictest3755528-21tail21:18:021
371307199261,11cyclictest3740692-21tune2fs20:38:021
371307199260,8cyclictest3752109-21cat21:11:031
371307199260,6cyclictest3936440-21needreboot00:38:051
3713070992619,4cyclictest3880881-21memory23:33:050
371307099261,11cyclictest3763055-21sendmail-msp21:30:440
371307199253,3cyclictest3928767-21/usr/sbin/munin00:28:091
371307199252,6cyclictest111788-21systemd-journal19:41:021
371307199252,5cyclictest3734155-21grep20:18:021
3713071992518,6cyclictest3750245-21unixbench_multi21:03:171
3713071992518,6cyclictest3750245-21unixbench_multi21:03:171
371307199251,7cyclictest764-21rs:main1
371307199251,5cyclictest11050irq/107-eth%d21:30:441
371307099250,5cyclictest3781601-21sendmail-msp21:48:060
371307099250,5cyclictest3781601-21sendmail-msp21:48:060
371307299246,3cyclictest11050irq/107-eth%d23:50:152
371307299244,7cyclictest171rcu_preempt22:53:072
371307299241,4cyclictest171rcu_preempt23:33:412
3713072992413,10cyclictest3782125-21kworker/u8:0+events_unbound22:06:052
371307199242,5cyclictest3867368-21/usr/sbin/munin23:18:101
371307199242,3cyclictest3920593-21sh00:18:091
371307199242,13cyclictest3932898-21ssh00:33:181
371307199241,6cyclictest3761009-21ssh21:23:121
3713071992411,11cyclictest3727577-21gltestperf19:58:021
371307099247,10cyclictest1-21systemd00:01:030
371307099247,10cyclictest1-21systemd00:01:030
3713070992416,0cyclictest0-21swapper/022:58:000
371307099241,21cyclictest0-21swapper/022:31:030
371307199235,16cyclictest0-21swapper/120:08:091
371307199234,6cyclictest0-21swapper/122:43:021
371307199233,5cyclictest3738776-21aten_r2power_en20:33:011
371307199232,5cyclictest3890772-21ssh23:43:141
371307199232,13cyclictest11050irq/107-eth%d19:16:031
371307199231,8cyclictest3852881-21fschecks_time23:03:031
371307199231,10cyclictest0-21swapper/121:46:541
371307199231,10cyclictest0-21swapper/121:46:541
371307099235,12cyclictest762-21in:imuxsock22:51:030
371307099232,15cyclictest3889260-21ssh23:43:010
3713070992317,1cyclictest0-21swapper/021:43:420
3713070992313,9cyclictest3727449-21fschecks_count19:58:020
3713070992313,8cyclictest3940983-21threads00:43:130
371307299222,6cyclictest11050irq/107-eth%d00:13:002
371307299222,3cyclictest171rcu_preempt00:01:052
371307199221,7cyclictest3733671-21grep20:16:031
3713071992210,10cyclictest0-21swapper/122:48:021
3713071992210,10cyclictest0-21swapper/122:48:021
371307099225,14cyclictest3736815-21awk20:26:030
371307099224,13cyclictest3743425-21sendmail-msp20:48:130
371307099222,5cyclictest3802454-21/usr/sbin/munin22:08:050
371307099222,16cyclictest3828363-21munin-run22:36:030
3713070992216,4cyclictest3726462-21grep19:53:150
3713070992210,4cyclictest0-21swapper/019:28:580
3713072992115,5cyclictest3871257-21fschecks_time23:23:032
371307199214,7cyclictest3774583-21ntpq21:38:081
371307199212,2cyclictest3714341-21/usr/sbin/munin19:18:111
371307099219,2cyclictest0-21swapper/021:20:230
371307099210,21cyclictest0-21swapper/000:33:160
371307299206,10cyclictest11050irq/107-eth%d23:55:102
371307199202,5cyclictest3748090-21ntpq20:58:061
371307199202,4cyclictest3729792-21/usr/sbin/munin20:03:091
371307199201,6cyclictest3717445-21awk19:28:091
371307099203,14cyclictest3809123-21ssh22:14:450
371307099203,14cyclictest3809123-21ssh22:14:450
3713070992016,2cyclictest1005-21mta-sts-daemon20:35:130
371307099201,15cyclictest3763029-21munin-plugin-st21:26:030
371307299197,8cyclictest3792805-21/usr/sbin/munin21:58:022
371307299194,11cyclictest1-21systemd00:41:032
371307299193,13cyclictest3889282-21/usr/sbin/munin23:43:012
371307299193,12cyclictest3924919-21/usr/sbin/munin00:23:102
371307299192,6cyclictest171rcu_preempt22:18:062
371307299192,12cyclictest764-21rs:main2
371307299192,12cyclictest764-21rs:main2
371307299192,11cyclictest3783928-21ssh21:48:092
371307299192,11cyclictest3783928-21ssh21:48:092
3713072991915,3cyclictest3733456-21unixbench_singl20:13:142
371307299191,15cyclictest3929069-21ps00:28:132
371307099196,2cyclictest3893705-21/usr/sbin/munin23:48:010
371307099196,11cyclictest0-21swapper/020:51:070
371307099194,7cyclictest3869652-21latency_hist23:21:030
371307099192,15cyclictest0-21swapper/022:18:000
371307099192,14cyclictest3774014-21grep21:38:040
371307099192,13cyclictest3876488-21ntp_states23:28:100
3713070991917,2cyclictest3792997-21gltestperf21:58:030
371307099191,17cyclictest3928041-21/usr/sbin/munin00:28:040
371307299184,7cyclictest3760939-21/usr/sbin/munin21:23:122
371307299184,12cyclictest3932889-21ntpq00:33:182
371307299183,12cyclictest1006-21mta-sts-daemon22:31:032
371307299182,7cyclictest3787730-21/usr/sbin/munin21:53:032
371307299181,4cyclictest171rcu_preempt22:23:082
3713072991813,3cyclictest3740027-21unixbench_multi20:33:152
3713072991812,4cyclictest0-21swapper/222:38:062
371307199184,3cyclictest3745054-21latency_hist20:51:041
371307199181,5cyclictest3719563-21/usr/sbin/munin19:33:131
371307099184,10cyclictest764-21rs:main0
371307099182,14cyclictest3713069-21systemd-run19:16:030
3713070991810,1cyclictest0-21swapper/021:33:100
371307099180,1cyclictest3853589-21/usr/sbin/munin23:03:100
371307299174,9cyclictest3910456-21ssh00:06:082
371307299174,7cyclictest3836323-21ssh22:44:022
371307299173,9cyclictest3903340-21ssh23:58:052
371307299173,9cyclictest3903340-21ssh23:58:052
3713072991714,2cyclictest3723174-21latency_hist19:46:032
371307299170,6cyclictest171rcu_preempt00:41:082
371307199170,8cyclictest3819209-21sendmail-msp22:27:261
371307099178,8cyclictest3822158-1kworker/0:2H+kblockd22:48:020
371307099178,8cyclictest3822158-1kworker/0:2H+kblockd22:48:020
371307099172,10cyclictest3734532-21/usr/sbin/munin20:18:060
3713070991710,1cyclictest3720613-21cut19:38:050
371307099170,14cyclictest3797122-21aten_r2power_cu22:02:590
371307399163,9cyclictest0-21swapper/322:03:073
371307399161,6cyclictest0-21swapper/300:41:083
371307299164,6cyclictest3755918-21ntpq21:18:062
371307299161,9cyclictest3819209-21sendmail-msp22:28:102
3713072991611,3cyclictest0-21swapper/221:13:162
371307099163,8cyclictest3816889-21ntpq22:23:080
371307099163,6cyclictest111788-21systemd-journal19:16:040
371307099161,7cyclictest3788063-21gltestperf21:53:060
3713070991614,1cyclictest0-21swapper/020:38:060
3713070991613,2cyclictest3751027-21gltestperf21:08:060
371307099160,13cyclictest0-21swapper/000:18:110
371307399152,6cyclictest0-21swapper/321:48:093
371307399152,6cyclictest0-21swapper/321:48:093
371307299155,9cyclictest3918818-21journal-offline00:16:042
371307299155,6cyclictest3774743-21/usr/sbin/munin21:38:102
371307299154,8cyclictest762-21in:imuxsock22:49:152
371307299154,8cyclictest762-21in:imuxsock22:49:152
371307299154,6cyclictest3740591-21/usr/sbin/munin20:38:012
371307299153,8cyclictest3763055-21sendmail-msp21:28:082
371307299152,12cyclictest3715268-21cut19:23:052
371307299151,9cyclictest3852764-21/usr/sbin/munin23:03:022
3713072991512,2cyclictest3727589-21gltestperf19:58:022
371307099156,8cyclictest11650irq/162-mmc019:48:160
3713070991513,1cyclictest0-21swapper/020:03:010
371307099151,2cyclictest11050irq/107-eth%d20:13:050
3713070991512,2cyclictest3722872-21unixbench_singl19:43:110
3713070991511,2cyclictest0-21swapper/021:03:130
3713070991511,2cyclictest0-21swapper/021:03:130
371307399144,6cyclictest0-21swapper/323:55:103
371307399144,6cyclictest0-21swapper/322:06:053
371307399143,5cyclictest0-21swapper/322:38:073
371307399143,4cyclictest0-21swapper/322:43:023
371307399142,8cyclictest0-21swapper/300:33:183
371307399142,7cyclictest0-21swapper/322:18:053
371307399142,6cyclictest0-21swapper/320:23:093
371307399142,2cyclictest0-21swapper/322:33:093
371307399141,7cyclictest0-21swapper/323:08:093
371307299146,5cyclictest0-21swapper/219:48:112
371307299145,7cyclictest3867304-21/usr/sbin/munin23:18:102
371307299144,8cyclictest0-21swapper/220:43:022
371307299144,5cyclictest3884882-21aten_r2power_fr23:37:592
371307299143,7cyclictest3737391-21fschecks_time20:28:072
371307299141,7cyclictest3855941-21ssh23:06:042
3713072991412,1cyclictest0-21swapper/220:58:132
371307299141,10cyclictest3797439-21cut22:03:012
371307299140,8cyclictest3778937-21awk21:43:042
371307299140,10cyclictest111788-21systemd-journal21:08:182
371307399134,5cyclictest0-21swapper/300:23:103
371307399133,8cyclictest0-21swapper/300:28:133
371307399133,6cyclictest0-21swapper/322:28:103
371307399133,5cyclictest0-21swapper/320:48:133
371307399133,4cyclictest0-21swapper/322:11:553
371307399133,4cyclictest0-21swapper/322:11:553
371307399132,4cyclictest0-21swapper/300:06:063
371307399131,7cyclictest0-21swapper/321:30:443
371307399131,5cyclictest0-21swapper/321:41:543
371307399130,8cyclictest0-21swapper/320:28:143
371307399130,8cyclictest0-21swapper/300:01:033
371307399130,8cyclictest0-21swapper/300:01:033
371307299136,4cyclictest0-21swapper/220:06:512
371307299132,9cyclictest3734033-21fschecks_count20:18:022
371307299131,9cyclictest3769454-21grep21:33:052
371307299131,8cyclictest3746993-21grep20:53:202
3713072991310,2cyclictest1005-21mta-sts-daemon20:23:082
371307299130,9cyclictest3862282-21if_wlan023:13:052
371307299130,10cyclictest3729972-21wc20:03:122
371307099130,8cyclictest3715373-21grep19:23:050
371307399125,5cyclictest0-21swapper/323:38:323
371307399124,4cyclictest0-21swapper/323:26:593
371307399124,4cyclictest0-21swapper/321:58:483
371307399123,6cyclictest0-21swapper/323:13:083
371307399123,6cyclictest0-21swapper/322:55:163
371307399123,6cyclictest0-21swapper/319:33:073
371307399123,5cyclictest0-21swapper/320:53:163
371307399123,5cyclictest0-21swapper/319:43:103
371307399122,7cyclictest0-21swapper/323:21:033
371307399122,7cyclictest0-21swapper/322:58:103
371307399122,6cyclictest0-21swapper/323:03:043
371307399122,6cyclictest0-21swapper/321:33:023
371307399122,5cyclictest0-21swapper/323:23:123
371307399122,5cyclictest0-21swapper/320:06:513
371307399122,5cyclictest0-21swapper/319:23:063
371307399120,6cyclictest0-21swapper/323:43:353
371307299127,2cyclictest3725353-21/usr/sbin/munin19:53:032
371307299125,5cyclictest3714224-21/usr/sbin/munin19:18:082
371307299124,5cyclictest3720034-21/usr/sbin/munin19:38:022
371307299123,7cyclictest3749842-21/usr/sbin/munin21:03:132
371307299123,7cyclictest3749842-21/usr/sbin/munin21:03:132
371307299120,9cyclictest3718004-21apt19:31:032
371307099124,7cyclictest3747563-21fschecks_count20:58:020
371307099121,7cyclictest3742264-21/usr/sbin/munin20:43:020
112699120,5rtkit-daemon1006-21mta-sts-daemon19:12:532
371307399114,3cyclictest0-21swapper/319:46:073
371307399113,5cyclictest0-21swapper/319:38:033
371307399113,5cyclictest0-21swapper/300:13:443
371307399113,4cyclictest0-21swapper/320:13:073
371307399112,6cyclictest0-21swapper/322:23:403
371307399112,5cyclictest0-21swapper/323:47:473
371307399111,6cyclictest0-21swapper/321:18:063
371307399103,4cyclictest0-21swapper/319:56:063
371307399102,5cyclictest0-21swapper/321:51:223
371307399102,5cyclictest0-21swapper/321:08:083
371307399102,5cyclictest0-21swapper/320:38:013
371307399102,5cyclictest0-21swapper/320:02:593
371307399102,5cyclictest0-21swapper/300:18:093
371307399102,3cyclictest0-21swapper/322:46:113
371307399102,3cyclictest0-21swapper/322:46:113
371307399101,6cyclictest0-21swapper/319:16:043
371307399100,4cyclictest0-21swapper/300:36:173
37130739994,3cyclictest0-21swapper/320:33:013
37130739994,3cyclictest0-21swapper/300:01:053
37130739993,4cyclictest0-21swapper/321:38:023
37130739992,4cyclictest0-21swapper/319:53:033
37130739991,5cyclictest0-21swapper/323:33:083
37130739991,5cyclictest0-21swapper/321:13:163
37130739991,4cyclictest0-21swapper/320:43:003
37130739983,2cyclictest0-21swapper/319:28:093
37130739982,4cyclictest0-21swapper/321:22:263
37130739982,4cyclictest0-21swapper/321:03:053
37130739982,4cyclictest0-21swapper/321:03:053
37130739981,5cyclictest0-21swapper/320:18:053
37130739981,3cyclictest0-21swapper/320:58:003
37130739973,3cyclictest0-21swapper/319:16:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional