You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 16:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Nov 17, 2025 00:45:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3143481998320,24cyclictest3143939-21munin-run19:20:000
3143481997540,5cyclictest3157235-21munin-run23:35:000
3143481997220,19cyclictest3144967-21cut19:35:020
314348899714,7cyclictest131rcu_preempt20:20:022
3143488997120,14cyclictest3150312-21cat21:20:022
3143481997135,32cyclictest3152575-21systemd-run22:05:000
3143481997134,33cyclictest3147099-21/usr/sbin/munin20:20:010
3143488997015,3cyclictest131rcu_preempt20:55:022
3143481997024,13cyclictest3153603-21cron22:25:000
314349399683,16cyclictest3149436-21modprobe21:05:003
314349399663,19cyclictest3149692-21modprobe21:10:003
314349399653,19cyclictest3152025-21modprobe21:55:003
314349399643,20cyclictest3158572-21munin-run00:00:013
3143493996325,7cyclictest37-21ksoftirqd/323:20:013
3143493996321,7cyclictest37-21ksoftirqd/322:40:003
314349399626,14cyclictest0-21swapper/320:20:003
314348499621,53cyclictest272-21dbus-daemon19:30:011
3143493996130,27cyclictest3145299-21sh19:45:003
3143488996050,5cyclictest30-21ksoftirqd/223:45:022
3143488996040,16cyclictest3150481-21sshd21:24:482
3143488995945,11cyclictest3149132-21sshd20:56:482
3143488995941,15cyclictest3148635-21sshd20:47:482
3143488995917,13cyclictest1-21systemd20:45:002
3143493995819,10cyclictest37-21ksoftirqd/319:15:003
3143481995840,14cyclictest3148644-21sshd20:49:480
3143481995837,17cyclictest3154915-21munin-run22:50:010
314349399572,19cyclictest3147363-21modprobe20:25:003
3143493995721,7cyclictest37-21ksoftirqd/320:55:023
3143488995739,15cyclictest3157212-21sshd23:32:482
3143488995739,14cyclictest3145766-21sshd19:51:562
3143488995739,14cyclictest3145766-21sshd19:51:562
3143488995739,14cyclictest3144472-21sshd19:25:492
3143481995719,34cyclictest0-21swapper/022:30:030
3143488995643,9cyclictest3145516-21sshd19:47:482
3143488995641,11cyclictest3147607-21cron20:30:002
3143488995640,12cyclictest3144994-21sshd19:39:472
3143488995640,11cyclictest3146578-21munin-run20:10:012
3143488995640,11cyclictest3146578-21munin-run20:10:012
3143488995638,14cyclictest3155116-21sshd22:50:482
3143481995642,11cyclictest3152267-21sshd21:57:480
3143481995642,11cyclictest3152267-21sshd21:57:480
3143481995638,14cyclictest3153315-21sshd22:18:480
3143481995638,13cyclictest3144188-21sshd19:23:480
3143481995624,28cyclictest3158403-21latency_hist23:55:020
3143481995624,28cyclictest3158403-21latency_hist23:55:020
3143481995623,29cyclictest3143605-21latency_hist19:10:010
3143493995524,8cyclictest37-21ksoftirqd/323:15:013
3143493995517,12cyclictest37-21ksoftirqd/323:35:013
3143481995541,11cyclictest3145770-21sshd19:52:480
3143481995541,11cyclictest3145770-21sshd19:52:480
3143481995541,10cyclictest3157780-21munin-run23:45:010
3143481995539,11cyclictest1-21systemd00:10:010
3143481995525,24cyclictest1-21systemd23:50:010
3143481995515,5cyclictest131rcu_preempt20:42:480
3143493995422,11cyclictest37-21ksoftirqd/322:55:023
3143493995419,9cyclictest306-21in:imuxsock22:16:483
3143488995441,8cyclictest3151509-21sshd21:42:482
3143488995440,11cyclictest3159026-21sshd00:07:162
3143488995440,11cyclictest3144192-21sshd19:24:482
3143488995439,10cyclictest3147874-21munin-run20:35:002
3143488995436,13cyclictest1-21systemd21:50:012
3143488995427,22cyclictest306-21in:imuxsock23:20:002
3143481995441,9cyclictest3155917-21sshd23:08:490
3143481995441,10cyclictest3153039-21sshd22:10:480
3143481995441,10cyclictest3148883-21sshd20:52:100
3143481995441,10cyclictest3147335-21sshd20:20:480
3143481995439,11cyclictest3160071-21sshd00:27:160
3143481995439,11cyclictest3145053-21munin-run19:40:010
3143481995438,13cyclictest3156154-21sshd23:10:470
3143481995438,13cyclictest3149145-21sshd20:59:480
314348199542,19cyclictest1-21systemd21:10:000
3143493995327,7cyclictest37-21ksoftirqd/320:45:023
3143488995341,9cyclictest3156159-21sshd23:11:492
3143488995340,10cyclictest3153293-21sshd22:16:482
3143488995335,14cyclictest363-21snmpd00:18:272
314348499530,34cyclictest0-21swapper/123:20:011
3143481995340,9cyclictest3146542-21sshd20:05:480
3143481995340,9cyclictest3146542-21sshd20:05:480
3143481995340,10cyclictest3154336-21sshd22:36:480
3143481995339,11cyclictest3154097-21sshd22:33:480
3143481995339,10cyclictest3146015-21sshd19:56:480
3143481995337,12cyclictest3151264-21munin-run21:40:000
3143493995237,13cyclictest37-21ksoftirqd/321:50:033
3143488995240,9cyclictest3153840-21sshd22:27:282
3143488995239,10cyclictest3155920-21sshd23:09:482
3143488995239,10cyclictest3152038-21munin-run21:55:002
3143488995238,11cyclictest3158500-21sshd23:58:392
3143488995238,10cyclictest3147353-21sshd20:24:482
3143488995237,10cyclictest3154118-21munin-run22:35:002
3143488995232,15cyclictest363-21snmpd20:04:592
3143481995240,9cyclictest3149936-21sshd21:13:480
3143481995239,10cyclictest3155370-21sshd22:55:480
3143481995239,10cyclictest3155370-21sshd22:55:480
3143481995239,10cyclictest3151759-21sshd21:47:210
3143481995238,10cyclictest3148169-21munin-run20:40:010
3143481995227,21cyclictest3144454-21apt-config19:25:300
3143493995134,12cyclictest235-21systemd-journal19:50:003
3143488995139,9cyclictest3152791-21sshd22:06:482
3143488995139,9cyclictest3149660-21sshd21:05:482
3143488995138,9cyclictest3151221-21sshd21:36:482
3143488995138,10cyclictest363-21snmpd19:40:572
3143488995137,11cyclictest3159270-21sshd00:10:162
3143488995137,10cyclictest3155698-21munin-run23:05:012
3143488995133,14cyclictest363-21snmpd22:55:212
3143488995133,14cyclictest363-21snmpd22:55:212
3143488995132,15cyclictest363-21snmpd00:23:252
3143484995129,16cyclictest235-21systemd-journal22:20:001
3143481995142,5cyclictest3160364-21munin-run00:35:020
3143481995140,8cyclictest3159849-21munin-run00:25:000
3143481995139,9cyclictest3150473-21sshd21:22:480
3143481995138,9cyclictest3157460-21sshd23:37:410
3143481995138,10cyclictest3156407-21sshd23:15:480
3143481995137,11cyclictest3145527-21sshd19:49:480
3143481995136,11cyclictest3146809-21sshd20:13:480
3143481995132,15cyclictest3155650-21sshd23:00:480
3143481995131,15cyclictest3156718-21cron23:25:000
3143481995113,33cyclictest3158738-21latency_hist00:00:020
3143481995112,35cyclictest0-21swapper/021:41:480
3143488995038,9cyclictest363-21snmpd22:11:272
3143488995038,9cyclictest3158787-21sshd00:04:162
3143488995038,9cyclictest3146030-21sshd19:59:482
3143488995037,10cyclictest363-21snmpd23:21:422
3143488995037,10cyclictest363-21snmpd22:46:352
3143488995027,19cyclictest3152443-21cat22:00:022
3143488995027,19cyclictest3152443-21cat22:00:022
3143481995038,9cyclictest3160566-21sshd00:35:160
3143481995038,9cyclictest3149419-21sshd21:02:480
3143481995033,13cyclictest363-21snmpd21:27:190
3143481995017,29cyclictest0-21swapper/022:50:480
3143488994940,5cyclictest3148169-21munin-run20:40:012
3143488994937,9cyclictest0-21swapper/222:41:332
3143488994937,8cyclictest3150763-21munin-run21:30:012
3143488994936,9cyclictest363-21snmpd19:11:092
3143488994936,10cyclictest363-21snmpd21:30:462
3143488994936,10cyclictest3158271-21munin-run23:55:002
3143488994936,10cyclictest3158271-21munin-run23:55:002
3143488994934,11cyclictest363-21snmpd00:36:342
3143488994925,20cyclictest3160289-21latency_hist00:30:022
314348499496,32cyclictest3155687-21cut23:05:011
3143484994934,10cyclictest3155696-21kworker/1:100:00:001
3143484994933,12cyclictest235-21systemd-journal22:45:021
3143481994939,7cyclictest3150985-21sshd21:34:480
3143481994934,12cyclictest3152786-21sshd22:05:480
3143481994930,14cyclictest3146294-21sshd20:01:480
3143481994924,21cyclictest306-21in:imuxsock21:20:010
3143481994913,7cyclictest12-21ksoftirqd/022:42:480
3143488994840,5cyclictest3149940-21sshd21:14:492
3143488994836,9cyclictest363-21snmpd22:36:342
3143488994836,9cyclictest363-21snmpd19:30:552
3143484994833,12cyclictest3146546-21sshd20:06:481
3143484994833,12cyclictest3146546-21sshd20:06:481
3143484994833,11cyclictest3152292-21cron22:00:001
3143484994833,11cyclictest3152292-21cron22:00:001
3143481994841,4cyclictest3147847-21sshd20:32:050
3143481994838,7cyclictest3143643-21sshd19:12:480
3143481994834,10cyclictest363-21snmpd23:26:590
3143488994735,9cyclictest363-21snmpd23:46:172
3143488994734,10cyclictest363-21snmpd22:01:262
3143488994733,10cyclictest363-21snmpd23:29:352
3143488994733,10cyclictest363-21snmpd21:00:092
3143488994727,16cyclictest3153772-21latency_hist22:25:022
3143481994739,5cyclictest3145275-21sshd19:44:480
3143481994734,10cyclictest363-21snmpd00:15:050
3143481994725,17cyclictest3159272-21sshd00:10:160
3143481994724,19cyclictest1-21systemd21:55:010
3143488994634,9cyclictest363-21snmpd20:13:562
3143488994634,9cyclictest363-21snmpd00:30:252
3143488994633,9cyclictest363-21snmpd19:16:332
3143484994633,7cyclictest271-21cron22:59:591
3143484994633,7cyclictest271-21cron22:59:591
3143484994630,11cyclictest3159576-21sendmail00:20:001
3143481994629,12cyclictest363-21snmpd20:28:290
3143488994533,9cyclictest363-21snmpd23:39:272
3143493994431,8cyclictest3148138-21systemctl20:40:003
3143493994427,12cyclictest272-21dbus-daemon22:00:013
3143493994427,12cyclictest272-21dbus-daemon22:00:013
314349399442,4cyclictest131rcu_preempt21:30:003
3143493994424,15cyclictest1-21systemd21:35:013
3143484994431,9cyclictest1-21systemd20:15:001
3143484994430,7cyclictest3147893-21latency_hist20:35:011
314348499442,4cyclictest131rcu_preempt21:55:001
3143493994330,11cyclictest37-21ksoftirqd/322:50:013
3143484994330,10cyclictest3150730-21sshd21:28:481
314348499432,14cyclictest3158032-21unin-run23:50:011
314348499432,12cyclictest235-21systemd-journal21:20:001
3143481994330,10cyclictest363-21snmpd00:02:060
3143493994232,6cyclictest3155704-21latency_hist23:05:013
3143493994225,10cyclictest3157494-21munin-plugin-st23:40:003
3143493994224,12cyclictest37-21ksoftirqd/322:45:023
314348499426,25cyclictest3151266-21grep21:40:001
314348499424,4cyclictest131rcu_preempt00:10:001
3143484994227,11cyclictest235-21systemd-journal19:46:551
3143484994225,13cyclictest3149148-21cron21:00:001
3143484994225,13cyclictest23-21ksoftirqd/122:15:011
3143484994220,15cyclictest3145016-21sendmail19:40:011
3143484994218,19cyclictest235-21systemd-journal23:35:001
3143493994122,14cyclictest272-21dbus-daemon22:05:003
314348499414,24cyclictest235-21systemd-journal21:10:001
3143484994128,10cyclictest235-21systemd-journal21:43:481
3143484994120,14cyclictest272-21dbus-daemon22:55:011
3143493994027,9cyclictest37-21ksoftirqd/323:00:003
3143493994027,9cyclictest37-21ksoftirqd/323:00:003
3143493994027,10cyclictest37-21ksoftirqd/321:00:003
3143493994024,12cyclictest272-21dbus-daemon23:30:003
314348499404,24cyclictest3148115-21cron20:39:591
3143484994029,8cyclictest3156696-21sshd23:20:481
3143484994029,7cyclictest3160586-21sshd00:39:061
314349399398,7cyclictest131rcu_preempt00:20:003
3143493993928,7cyclictest306-21in:imuxsock19:55:013
3143493993928,7cyclictest306-21in:imuxsock19:55:013
3143493993927,9cyclictest3144733-21sshd19:31:503
314349399392,6cyclictest3145018-21munin-run19:40:013
3143493993911,6cyclictest131rcu_preempt23:25:003
3143484993926,8cyclictest306-21in:imuxsock21:05:001
3143484993926,10cyclictest3158989-21sed00:05:031
3143484993926,10cyclictest3148647-21cron20:50:001
3143484993925,11cyclictest23-21ksoftirqd/119:55:011
3143484993925,11cyclictest23-21ksoftirqd/119:55:011
3143484993924,11cyclictest235-21systemd-journal23:29:481
3143484993924,10cyclictest23-21ksoftirqd/120:00:011
3143484993913,18cyclictest272-21dbus-daemon19:25:011
3143493993828,7cyclictest3146305-21sshd20:03:483
3143493993828,6cyclictest3160604-21sh00:40:003
3143493993826,9cyclictest37-21ksoftirqd/319:20:023
3143493993826,9cyclictest37-21ksoftirqd/319:10:013
3143493993825,10cyclictest3146030-21sshd19:59:493
3143493993824,11cyclictest37-21ksoftirqd/323:55:003
3143493993824,11cyclictest37-21ksoftirqd/323:55:003
314348499387,9cyclictest23-21ksoftirqd/100:24:041
3143484993828,6cyclictest3158272-21grep23:55:001
3143484993828,6cyclictest3158272-21grep23:55:001
3143484993827,8cyclictest23-21ksoftirqd/122:35:001
3143484993827,8cyclictest23-21ksoftirqd/120:05:011
3143484993823,8cyclictest306-21in:imuxsock20:50:021
314348499381,33cyclictest3156273-21sed23:15:011
3143484993811,5cyclictest40850lldpd22:07:381
3143493993726,6cyclictest3153602-21cron22:25:003
3143493993726,5cyclictest306-21in:imuxsock20:50:013
3143493993725,8cyclictest37-21ksoftirqd/323:45:003
3143493993725,5cyclictest3159306-21munin-plugin-st00:15:003
3143493993724,10cyclictest272-21dbus-daemon19:30:013
3143493993717,15cyclictest272-21dbus-daemon21:20:013
3143493993711,8cyclictest131rcu_preempt19:25:023
314348499375,21cyclictest3149950-21modprobe21:15:001
3143484993725,9cyclictest235-21systemd-journal20:29:261
3143484993725,9cyclictest23-21ksoftirqd/122:21:481
3143484993724,9cyclictest23-21ksoftirqd/120:15:491
3143484993724,7cyclictest306-21in:imuxsock22:05:001
3143484993718,12cyclictest3144761-21munin-plugin-st19:35:001
3143484993713,18cyclictest306-21in:imuxsock23:45:011
3143493993627,6cyclictest341irq_work/320:30:023
3143493993625,8cyclictest3151228-21sshd21:37:493
3143493993623,9cyclictest37-21ksoftirqd/320:10:033
3143488993615,17cyclictest3143508-21latency_hist19:10:012
314348499368,17cyclictest3143927-21modprobe19:20:001
3143484993626,8cyclictest23-21ksoftirqd/122:50:001
3143484993624,7cyclictest3157488-21sh23:40:001
3143484993620,11cyclictest272-21dbus-daemon21:25:001
3143484993620,11cyclictest272-21dbus-daemon00:30:021
314349399357,9cyclictest3146576-21munin-plugin-st20:10:003
314348499358,17cyclictest3153868-21grep22:30:001
314348499352,4cyclictest131rcu_preempt21:35:011
3143484993520,10cyclictest0-21swapper/119:41:531
3143484993519,12cyclictest1-21systemd00:15:011
3143484993516,13cyclictest272-21dbus-daemon00:35:011
3143484993511,19cyclictest363-21snmpd23:10:011
3143493993426,4cyclictest37-21ksoftirqd/300:24:163
3143493993423,9cyclictest37-21ksoftirqd/320:10:033
3143493993422,10cyclictest37-21ksoftirqd/300:05:013
3143493993421,9cyclictest37-21ksoftirqd/321:45:023
3143493993416,12cyclictest235-21systemd-journal23:10:013
314348499343,4cyclictest131rcu_preempt21:50:001
3143493993325,5cyclictest37-21ksoftirqd/322:35:033
3143493993324,6cyclictest3149462-21kworker/3:221:22:163
3143493993322,7cyclictest3160087-21cron00:30:003
3143493993319,10cyclictest37-21ksoftirqd/322:26:483
314348499336,15cyclictest3154358-21cron22:40:001
3143484993326,4cyclictest3148408-21/usr/sbin/munin20:45:001
314348499332,3cyclictest131rcu_preempt20:25:011
3143484993320,9cyclictest23-21ksoftirqd/119:10:481
3143493993221,9cyclictest37-21ksoftirqd/322:10:033
314349399322,15cyclictest3149970-21unin-run21:15:013
3143493993220,10cyclictest37-21ksoftirqd/323:48:483
314349399321,13cyclictest3160352-21munin-run00:35:013
3143484993224,6cyclictest23-21ksoftirqd/119:10:011
3143493993121,8cyclictest37-21ksoftirqd/300:10:023
3143493993118,9cyclictest3148078-21cat20:35:023
3143493993020,8cyclictest37-21ksoftirqd/322:14:483
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional