You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-20 - 04:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack2slot7s.osadl.org (updated Sun Jul 20, 2025 00:40:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2606045995432,19cyclictest0-21swapper/022:37:530
2606045995321,28cyclictest2756123-21cups-browsed00:08:000
2606047995243,6cyclictest2708830-21latency_hist22:50:022
260604599522,21cyclictest657-21snmpd20:53:040
2606045995022,3cyclictest0-21swapper/022:18:060
2606045995017,30cyclictest2681127-21sendmail_mailqu22:08:100
2606046994939,7cyclictest2627034-21cut20:20:011
2606046994918,13cyclictest2780434-21ntpq00:38:071
2606045994923,22cyclictest0-21swapper/021:48:060
2606047994838,6cyclictest2701357-21/usr/sbin/munin22:38:172
2606045994822,7cyclictest2765642-21grep00:13:200
2606046994739,5cyclictest2642030-21grep21:10:011
2606046994718,7cyclictest2677633-21sh22:03:071
2606045994632,11cyclictest2639064-21latency_hist21:00:020
260604599462,5cyclictest171rcu_preempt21:13:040
2606045994621,20cyclictest12450irq/32-eth023:23:110
2606046994518,7cyclictest2713464-21sh22:57:361
2606045994540,3cyclictest957-21mta-sts-daemon19:43:060
2606045994523,19cyclictest0-21swapper/000:33:010
2606046994424,14cyclictest2642026-21diskmemload23:03:041
2606046994421,7cyclictest2708558-21sh22:49:241
2606045994437,5cyclictest421-21avahi-daemon20:08:050
2606045994436,6cyclictest2754543-21missed_timers23:58:010
260604599442,16cyclictest2635533-21modprobe20:48:120
2606047994334,6cyclictest955-21mta-sts-daemon21:38:122
2606047994334,6cyclictest2654760-21/usr/sbin/munin21:28:122
2606047994316,8cyclictest2779699-21sh00:37:452
2606046994335,5cyclictest2675547-21munin-run22:00:011
2606046994329,8cyclictest2761594-21ntpq00:08:041
2606045994336,3cyclictest0-21swapper/022:50:010
2606045994335,5cyclictest422-21dbus-daemon22:25:010
260604599432,20cyclictest2758227-21ssh00:03:050
2606046994228,8cyclictest2754658-21ntpq23:58:041
2606046994228,8cyclictest2635526-21ntpq20:48:111
2606046994226,12cyclictest2622766-21sshd20:05:121
2606046994224,12cyclictest2731375-21ntpq23:23:101
2606046994218,7cyclictest2649881-21sh21:21:141
2606046994217,19cyclictest2738040-21ssh23:33:161
2606046994217,10cyclictest2659505-21ssh21:35:341
2606046994216,7cyclictest2615958-21ntpq19:43:031
2606045994216,9cyclictest12450irq/32-eth022:58:070
2606047994116,8cyclictest2697815-21/usr/sbin/munin22:33:202
2606046994131,6cyclictest2621060-21true20:00:011
2606046994125,12cyclictest2701996-21ssh22:39:371
2606046994118,8cyclictest2704877-21sh22:43:391
2606046994118,7cyclictest2683469-21sh22:12:451
260604599412,18cyclictest2704152-21ssh22:43:080
260604799404,28cyclictest2651218-21ssh21:23:072
2606047994031,6cyclictest2692234-21diskmemload22:25:012
2606047994027,8cyclictest2618960-21ntpq19:53:032
2606047994015,9cyclictest2718955-21sh23:04:492
2606047994015,7cyclictest2771659-21ntpq00:23:092
2606047994013,8cyclictest2728521-21users23:18:172
2606046994032,5cyclictest2668963-21ls21:50:011
2606046994032,5cyclictest2619413-21unixbench_multi19:53:131
2606046994030,6cyclictest2778681-21munin-run00:35:011
2606046994028,8cyclictest2608531-21ntpq19:18:091
2606046994028,7cyclictest2629522-21ntpq20:28:131
2606046994027,8cyclictest2639893-21ntpq21:03:031
2606046994027,8cyclictest2639893-21ntpq21:03:031
2606046994026,11cyclictest2746815-21ssh23:46:381
2606046994026,11cyclictest2709626-21ssh22:51:331
2606046994026,11cyclictest2638390-21ntpq20:58:041
2606046994025,12cyclictest2773108-21ssh00:25:241
2606046994025,11cyclictest2763670-21ssh00:11:101
2606046994025,11cyclictest2735946-21ssh23:29:501
2606046994025,11cyclictest2654069-21diskstats21:27:561
2606046994024,12cyclictest2767577-21ssh00:17:231
2606046994024,12cyclictest2627988-21ntpq20:23:071
2606046994024,11cyclictest2726288-21ssh23:15:391
2606046994023,12cyclictest2646052-21ssh21:15:121
2606046994018,16cyclictest422-21dbus-daemon22:08:071
2606045994033,4cyclictest2612108-21latency_hist19:30:030
2606045994033,4cyclictest12450irq/32-eth023:48:040
2606045994031,7cyclictest2699051-21latency_hist22:35:030
2606045994031,6cyclictest2727698-21memory23:18:000
2606047993931,5cyclictest2712268-21systemd-run22:55:012
2606047993931,5cyclictest2688917-21cut22:20:012
2606047993916,7cyclictest2613176-21mailstats19:33:122
2606047993914,8cyclictest2627846-21sed20:23:012
2606046993930,4cyclictest2687603-21ntpq22:18:061
2606046993928,7cyclictest2609946-21ntpq19:23:031
2606046993927,8cyclictest2670901-21ntpq21:53:041
2606046993927,8cyclictest2617397-21ntpq19:48:021
2606046993926,8cyclictest2614414-21ntpq19:38:021
2606046993925,11cyclictest2771848-21/usr/sbin/munin00:23:131
2606046993925,10cyclictest2756783-21ssh00:01:091
2606046993924,12cyclictest2644371-21ntpq21:13:031
2606046993924,11cyclictest2745686-21ssh23:44:201
2606046993924,11cyclictest2719825-21ssh23:06:291
2606046993924,11cyclictest2693159-21ssh22:26:581
2606046993924,11cyclictest2665605-21ssh21:44:461
2606046993923,10cyclictest2656147-21ssh21:30:211
2606046993920,13cyclictest2612549-21diskstats19:32:551
2606045993930,6cyclictest2684434-21proc_pri22:13:100
2606045993920,3cyclictest0-21swapper/023:06:450
260604799383,27cyclictest2747975-21ssh23:48:042
2606047993831,4cyclictest2642058-21kworker/u8:7-xprtiod21:10:022
2606047993830,5cyclictest956-21mta-sts-daemon21:30:172
2606047993830,5cyclictest2674004-21kernelversion21:58:002
2606047993829,6cyclictest421-21avahi-daemon23:23:142
2606047993829,6cyclictest2681085-21ps22:08:102
2606047993828,6cyclictest2753998-21aten_r2power_cu23:57:542
2606047993823,12cyclictest2664435-21ntpq21:43:082
2606047993823,11cyclictest2719805-21ssh23:06:222
2606046993827,7cyclictest2636998-21ntpq20:53:081
2606046993827,7cyclictest2631002-21ntpq20:33:091
2606046993827,7cyclictest2624963-21ntpq20:13:071
2606046993826,9cyclictest2723060-21ssh23:11:071
2606046993826,8cyclictest2689387-21ssh22:20:411
2606046993825,8cyclictest2611429-21ntpq19:28:051
2606046993824,10cyclictest2750228-21ssh23:51:581
2606046993824,10cyclictest2742508-21ssh23:39:401
2606046993823,12cyclictest2696813-21ssh22:32:501
2606045993833,3cyclictest151ktimers/019:58:050
2606045993831,4cyclictest2738317-21mailstats23:33:240
2606047993729,5cyclictest2621076-21latency_hist20:00:022
2606047993726,7cyclictest2704122-21ntpq22:43:072
2606047993725,8cyclictest2768516-21ntpq00:18:062
2606047993724,8cyclictest2632456-21ntpq20:38:092
2606047993723,11cyclictest2714411-21ntpq22:58:092
2606047993723,10cyclictest2774573-21ntpq00:28:122
260604799372,29cyclictest2667671-21ssh21:48:042
2606047993722,11cyclictest2649040-21ssh21:19:442
2606047993721,11cyclictest2682843-21ssh22:11:022
2606047993720,12cyclictest2736330-21ssh23:30:172
2606047993718,9cyclictest2677117-21cpu22:02:562
2606046993724,9cyclictest2606807-21missed_timers19:13:011
2606046993723,9cyclictest2633951-21ntpq20:43:031
2606045993732,3cyclictest955-21mta-sts-daemon22:01:340
2606045993731,3cyclictest0-21swapper/023:28:080
2606045993728,6cyclictest12450irq/32-eth021:43:040
2606047993627,6cyclictest2626941-21unixbench_multi20:18:172
2606047993625,7cyclictest2614411-21ntp_kernel_pll_19:38:012
2606047993624,8cyclictest2629960-21unixbench_singl20:28:202
2606047993624,7cyclictest2633923-21ntpq20:43:022
2606047993623,8cyclictest2611418-21ntpq19:28:042
2606047993621,12cyclictest2742103-21ssh23:38:332
2606047993620,12cyclictest2692530-21ssh22:25:182
2606047993620,11cyclictest2722895-21ssh23:10:422
2606047993613,6cyclictest2642026-21diskmemload21:52:432
2606046993622,11cyclictest2621922-21ntpq20:03:021
2606045993628,5cyclictest2771595-21ntp_offset00:23:060
2606045993627,7cyclictest2744596-21needreboot23:43:050
2606045993625,8cyclictest2642107-21latency_hist21:10:020
2606047993529,4cyclictest2606044-21cyclictest21:13:062
2606047993527,5cyclictest2764658-21ssh00:12:592
2606047993526,6cyclictest2761596-21sh00:08:042
2606047993525,7cyclictest2615689-21irqstats19:42:562
2606047993524,7cyclictest2777470-21ntpq00:33:052
2606047993524,7cyclictest2623464-21ntpq20:08:062
2606047993524,7cyclictest2609601-21fw_packets19:22:562
2606047993524,7cyclictest2606962-21ntpq19:13:062
2606047993523,7cyclictest2734853-21ntpq23:28:132
2606047993523,7cyclictest2636888-21ntpq20:53:032
2606047993522,9cyclictest2624883-21nfs_client20:13:022
2606047993521,11cyclictest2751322-21ntpq23:53:062
2606047993521,11cyclictest2745750-21ssh23:44:282
2606047993521,11cyclictest2607784-21acpi19:17:522
2606047993521,10cyclictest2756637-21ssh00:00:482
2606046993524,7cyclictest2631749-21sshd20:35:111
260604599353,26cyclictest2661050-21rm21:38:050
2606045993527,5cyclictest2627071-21systemd-run20:20:020
2606045993526,6cyclictest2675538-21ssh22:00:010
2606045993524,8cyclictest2692887-21sshd22:26:120
2606045993520,12cyclictest12450irq/32-eth023:53:110
2606045993515,15cyclictest151ktimers/020:03:090
2606045993513,18cyclictest2647790-21ssh21:18:020
2606047993424,7cyclictest2635508-21ntpq20:48:102
2606047993424,7cyclictest2630927-21ntpq20:33:062
2606047993423,8cyclictest2639545-21df_inode21:02:572
2606047993423,8cyclictest2639545-21df_inode21:02:572
2606047993423,7cyclictest2617094-21/usr/sbin/munin19:47:572
2606047993422,9cyclictest2621560-21/usr/sbin/munin20:02:562
2606046993422,9cyclictest2606057-21munin-run19:10:011
2606046993422,9cyclictest2606057-21munin-run19:10:011
2606045993428,3cyclictest0-21swapper/000:38:070
2606045993427,5cyclictest0-21swapper/020:40:010
2606045993427,4cyclictest2774536-21ntp_kernel_err00:28:100
2606045993427,4cyclictest2659197-21latency_hist21:35:020
2606045993415,11cyclictest2741537-21ssh23:38:120
2606045993412,17cyclictest2670929-21ssh21:53:050
2606047993321,8cyclictest2638456-21ntpq20:58:072
260604599335,25cyclictest2717781-21ssh23:03:070
2606045993327,4cyclictest2606044-21cyclictest23:13:190
2606045993325,3cyclictest0-21swapper/021:23:300
2606045993224,3cyclictest0-21swapper/022:50:110
2606045993124,4cyclictest2768338-21missed_timers00:18:010
2606045993120,5cyclictest2625414-21unixbench_multi20:13:160
2606045993022,5cyclictest2652753-21sh21:25:110
2606045993019,8cyclictest151ktimers/020:23:070
2606045992924,2cyclictest0-21swapper/019:40:010
2606045992922,4cyclictest806984-21cups-browsed19:15:490
260604599291,25cyclictest421-21avahi-daemon19:47:000
2606047992812,12cyclictest2606221-21latency_hist19:10:022
2606047992812,12cyclictest2606221-21latency_hist19:10:022
2606045992714,9cyclictest421-21avahi-daemon21:03:070
2606045992714,9cyclictest421-21avahi-daemon21:03:070
2606045992712,12cyclictest0-21swapper/019:32:590
2606045992613,3cyclictest0-21swapper/020:43:030
260604599261,18cyclictest806984-21cups-browsed19:53:040
260604599255,9cyclictest12450irq/32-eth020:33:130
2606045992519,4cyclictest2628781-21/usr/sbin/munin20:28:100
2606045992514,3cyclictest0-21swapper/019:25:000
2606045992318,2cyclictest0-21swapper/019:10:010
2606045992318,2cyclictest0-21swapper/019:10:010
260604599211,9cyclictest2607470-21unixbench_singl19:13:150
260604899174,8cyclictest0-21swapper/322:28:063
260604899173,8cyclictest0-21swapper/323:23:113
260604899173,7cyclictest0-21swapper/321:58:053
260604899172,8cyclictest0-21swapper/320:03:043
260604899172,6cyclictest0-21swapper/319:53:023
260604899172,10cyclictest0-21swapper/321:33:083
260604899171,9cyclictest0-21swapper/322:53:043
260604899162,9cyclictest0-21swapper/322:08:073
260604899162,8cyclictest0-21swapper/323:13:073
260604899162,8cyclictest0-21swapper/322:58:053
260604899162,8cyclictest0-21swapper/322:48:053
260604899161,8cyclictest0-21swapper/323:48:123
260604899161,8cyclictest0-21swapper/323:07:593
260604899161,8cyclictest0-21swapper/322:43:153
260604899156,6cyclictest0-21swapper/321:18:073
260604899153,7cyclictest0-21swapper/323:38:123
260604899153,7cyclictest0-21swapper/300:03:053
260604899153,5cyclictest0-21swapper/319:28:093
260604899153,4cyclictest0-21swapper/320:13:053
260604899152,8cyclictest0-21swapper/322:23:083
260604899152,8cyclictest0-21swapper/300:33:063
260604899152,7cyclictest0-21swapper/321:52:533
260604899152,6cyclictest0-21swapper/320:53:043
260604899152,6cyclictest0-21swapper/319:23:033
260604899152,6cyclictest0-21swapper/319:13:033
260604899151,7cyclictest0-21swapper/319:18:103
260604899151,6cyclictest0-21swapper/320:18:053
260604899143,7cyclictest0-21swapper/300:08:123
260604899143,6cyclictest0-21swapper/323:03:043
260604899142,7cyclictest0-21swapper/322:38:063
260604899142,7cyclictest0-21swapper/320:08:053
260604899141,9cyclictest0-21swapper/321:28:103
260604899141,8cyclictest0-21swapper/323:53:043
260604899141,8cyclictest0-21swapper/323:43:083
260604899141,8cyclictest0-21swapper/322:03:023
260604899141,8cyclictest0-21swapper/321:37:553
260604899141,8cyclictest0-21swapper/321:13:083
260604899141,7cyclictest0-21swapper/321:08:193
260604899141,7cyclictest0-21swapper/321:03:073
260604899141,7cyclictest0-21swapper/321:03:073
260604899141,7cyclictest0-21swapper/320:28:103
260604899141,6cyclictest0-21swapper/319:58:043
260604899141,6cyclictest0-21swapper/319:38:023
260604899140,6cyclictest0-21swapper/319:48:023
260604899133,7cyclictest0-21swapper/323:17:523
260604899133,6cyclictest0-21swapper/320:23:073
260604899132,7cyclictest0-21swapper/323:33:123
260604899132,7cyclictest0-21swapper/322:13:043
260604899132,7cyclictest0-21swapper/300:38:023
260604899132,6cyclictest0-21swapper/322:33:013
260604899132,6cyclictest0-21swapper/320:33:043
260604899132,6cyclictest0-21swapper/300:22:523
260604899131,8cyclictest0-21swapper/300:17:543
260604899131,7cyclictest0-21swapper/322:17:593
260604899131,7cyclictest0-21swapper/321:23:033
260604899131,7cyclictest0-21swapper/319:43:073
260604899131,7cyclictest0-21swapper/300:13:133
260604899123,5cyclictest0-21swapper/321:00:013
260604899122,7cyclictest0-21swapper/300:28:053
260604899122,6cyclictest0-21swapper/321:48:063
260604899122,6cyclictest0-21swapper/320:47:593
260604899121,8cyclictest0-21swapper/321:42:543
260604899121,7cyclictest0-21swapper/319:33:043
260604899121,6cyclictest0-21swapper/323:57:533
260604899121,6cyclictest0-21swapper/323:25:343
260604899113,6cyclictest0-21swapper/320:38:033
260604899111,6cyclictest0-21swapper/320:42:583
26060489991,4cyclictest0-21swapper/319:10:023
26060489991,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional