You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-02 - 18:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Wed Sep 02, 2026 12:42:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2471888994940,5cyclictest2620305-21latency_hist12:15:011
2471889994736,8cyclictest1656768-21cups-browsed08:13:042
2471887994731,13cyclictest2473365-21cron07:15:010
2471889994637,5cyclictest2480862-21cron07:40:012
2471888994638,5cyclictest2520605-21/usr/sbin/munin09:31:011
2471888994535,7cyclictest2611272-21cron12:00:001
2471889994433,6cyclictest0-21swapper/211:16:082
247188999443,32cyclictest2597313-21ssh11:36:062
2471888994436,4cyclictest2541438-21kthreadd10:05:011
2471888994436,4cyclictest2541438-21kthreadd10:05:011
2471888994434,4cyclictest0-21swapper/112:36:111
2471887994436,5cyclictest1002-21mta-sts-daemon10:24:170
2471889994333,7cyclictest2611318-21sh12:00:012
2471889994328,11cyclictest2569640-21sh10:51:012
247188999432,32cyclictest2566917-21sshd10:46:052
2471889994234,5cyclictest2543256-21unixbench_singl10:06:192
2471889994234,5cyclictest2543256-21unixbench_singl10:06:192
2471889994231,8cyclictest2532297-21/usr/sbin/munin09:50:022
2471887994234,5cyclictest2480865-21sendmail07:40:010
2471889994130,8cyclictest2513986-21sendmail09:20:012
2471888994135,3cyclictest0-21swapper/112:21:051
2471889994027,5cyclictest0-21swapper/209:11:062
2471889994012,9cyclictest2549167-21cat10:16:192
2471887994033,4cyclictest2562662-21munin-plugin-st10:40:010
2471887994032,6cyclictest2602173-21munin-plugin-st11:45:010
247188999393,28cyclictest2515190-21ssh09:21:092
2471889993930,7cyclictest2471886-21cyclictest12:20:012
2471889993915,8cyclictest1656768-21cups-browsed11:06:112
2471889993915,8cyclictest1656768-21cups-browsed11:06:112
2471888993928,8cyclictest2513990-21/usr/sbin/munin09:20:021
2471887993931,5cyclictest2486952-21latency_hist08:00:010
2471887993931,5cyclictest1656768-21cups-browsed08:16:010
2471889993825,9cyclictest2493753-21ntpq08:21:042
2471889993825,9cyclictest2493753-21ntpq08:21:042
2471889993814,8cyclictest2546641-21ssh10:12:232
2471889993812,8cyclictest2572992-21ntpq10:56:062
2471888993830,5cyclictest2563312-21aten_r2power_po10:40:561
2471888993830,4cyclictest2547467-21cron10:15:011
2471888993829,6cyclictest439-21systemd-logind09:10:011
2471888993828,6cyclictest1656768-21cups-browsed11:35:451
2471888993827,7cyclictest2627901-21switchtime12:26:161
247188799383,30cyclictest2627498-21sh12:26:040
2471887993832,3cyclictest2550557-21systemd-run10:20:010
2471887993831,5cyclictest2498854-21cron08:40:010
2471887993830,5cyclictest421-21dbus-daemon11:16:070
2471887993830,4cyclictest0-21swapper/009:21:130
2471887993826,8cyclictest1656763-21cupsd07:41:080
2471887993826,10cyclictest1-21systemd08:21:040
2471887993826,10cyclictest1-21systemd08:21:040
247188999377,19cyclictest351ktimers/207:41:072
247188999376,20cyclictest428-21mta-sts-daemon08:01:122
247188999374,26cyclictest2524417-21ssh09:36:082
2471889993729,5cyclictest2486862-21grep08:00:002
2471889993729,5cyclictest0-21swapper/208:50:012
2471889993728,5cyclictest1-21systemd07:25:012
2471889993726,8cyclictest2520169-21latency_hist09:30:022
2471889993726,6cyclictest2615447-21needreboot12:06:032
2471889993718,12cyclictest2476886-21diskstats07:25:572
2471889993713,7cyclictest1002-21mta-sts-daemon12:13:012
2471888993729,5cyclictest2587040-21systemd-run11:20:011
2471888993729,5cyclictest2479382-21idleruntime-cro07:35:011
2471888993729,5cyclictest1-21systemd09:55:021
2471888993728,7cyclictest2574832-21grep11:00:011
2471888993728,6cyclictest2569564-21fschecks_count10:50:591
2471888993728,5cyclictest2623375-21latency_hist12:20:021
2471888993727,6cyclictest1656768-21cups-browsed08:40:591
2471888993725,7cyclictest420-21avahi-daemon09:22:151
247188799372,30cyclictest151ktimers/008:26:050
2471889993627,6cyclictest2596075-21sshd11:35:012
2471889993627,5cyclictest0-21swapper/207:16:112
2471889993622,8cyclictest2472807-21ntpq07:11:072
247188999362,25cyclictest2560827-21ssh10:36:052
2471889993618,10cyclictest2492331-21ntpq08:16:092
247188999361,30cyclictest2527577-21ssh09:41:122
2471888993629,4cyclictest1002-21mta-sts-daemon07:41:071
2471888993628,5cyclictest2550508-21ssh10:20:011
2471888993628,5cyclictest2497896-21fschecks_time08:36:001
2471888993628,5cyclictest1-21systemd10:25:021
2471887993629,4cyclictest2612698-21proc_pri12:01:140
2471887993628,5cyclictest567-21gdbus09:45:010
2471887993628,5cyclictest1656768-21cups-browsed09:01:090
2471887993627,6cyclictest694-21cron10:50:010
2471887993626,7cyclictest0-21swapper/010:35:010
2471887993617,14cyclictest12650irq/32-eth009:16:060
2471889993528,4cyclictest2626333-21munin-plugin-st12:25:012
2471889993527,5cyclictest2503241-21unixbench_multi08:51:172
2471889993525,6cyclictest1656768-21cups-browsed12:28:512
2471889993513,7cyclictest2540749-21sh10:02:562
2471889993513,7cyclictest2540749-21sh10:02:562
2471888993527,5cyclictest2602175-21idleruntime-cro11:45:011
2471888993527,5cyclictest2580959-21latency_hist11:10:011
2471888993527,5cyclictest2580959-21latency_hist11:10:011
2471888993527,3cyclictest0-21swapper/110:46:101
2471888993525,6cyclictest2503378-21systemd-run08:55:011
2471888993524,8cyclictest2504202-21ntp_kernel_err08:56:051
247188899352,25cyclictest2591394-21ssh11:26:171
247188899351,27cyclictest2493749-21sshd08:21:041
247188899351,27cyclictest2493749-21sshd08:21:041
2471887993529,4cyclictest2547471-21rm10:15:000
2471887993526,7cyclictest2510624-21sh09:14:070
2471887993526,6cyclictest1656768-21cups-browsed09:55:290
2471889993428,2cyclictest0-21swapper/210:45:012
2471889993427,4cyclictest2559616-21ssh10:35:012
2471889993427,3cyclictest0-21swapper/210:26:012
2471889993425,6cyclictest2590893-21/usr/sbin/munin11:26:032
2471889993424,5cyclictest2505846-21ntpq09:01:102
2471889993423,7cyclictest2495304-21ntpq08:26:092
2471889993418,11cyclictest2631821-21ssh12:33:212
2471889993414,13cyclictest0-21swapper/208:36:062
247188899343,24cyclictest2527536-21ssh09:41:091
2471888993428,3cyclictest0-21swapper/110:40:021
2471888993426,5cyclictest420-21avahi-daemon09:05:011
2471888993425,5cyclictest2544458-21sh10:10:011
2471888993425,5cyclictest2544458-21sh10:10:011
2471887993423,9cyclictest2583976-21systemd-run11:15:010
2471887993423,9cyclictest2583976-21systemd-run11:15:010
2471887993419,10cyclictest151ktimers/010:06:070
2471887993419,10cyclictest151ktimers/010:06:070
2471889993323,6cyclictest2504571-21/usr/sbin/munin08:56:172
2471889993322,7cyclictest421-21dbus-daemon09:34:112
2471889993322,7cyclictest2612823-21sensors_fan12:01:172
2471889993322,7cyclictest2507216-21ntpq09:06:032
2471889993321,8cyclictest2533491-21ntpq09:51:052
2471889993321,8cyclictest2478417-21/usr/sbin/munin07:30:592
2471889993321,7cyclictest2633678-21ntpq12:36:112
2471889993320,8cyclictest2499794-21ntpq08:41:062
2471888993327,3cyclictest0-21swapper/109:13:031
2471887993327,4cyclictest2590865-21fschecks_time11:26:030
2471887993326,4cyclictest0-21swapper/011:01:030
2471887993326,4cyclictest0-21swapper/011:01:030
2471887993325,6cyclictest2485437-21latency_hist07:55:040
2471887993325,6cyclictest2485437-21latency_hist07:55:040
2471887993325,5cyclictest2520646-21head09:30:560
247188999322,24cyclictest2603500-21ssh11:46:092
2471889993221,7cyclictest2536593-21ntpq09:56:102
2471889993219,10cyclictest2484743-21ntpq07:51:052
2471889993219,10cyclictest2484743-21ntpq07:51:052
2471889993215,12cyclictest2496767-21ntpq08:31:062
2471888993226,3cyclictest0-21swapper/109:46:131
2471888993225,4cyclictest2538351-21cron10:00:011
2471888993222,8cyclictest2606210-21latency11:51:011
2471888993222,4cyclictest1000-21mta-sts-daemon09:36:011
2471887993226,4cyclictest2635406-21latency_hist12:40:020
2471887993226,4cyclictest2596070-21sh11:35:010
2471887993225,4cyclictest420-21avahi-daemon10:44:420
2471887993224,5cyclictest420-21avahi-daemon08:01:130
2471887993211,6cyclictest12750irq/33-eth009:48:510
247188999313,21cyclictest566-21gmain11:11:092
247188999313,21cyclictest566-21gmain11:11:092
2471889993122,4cyclictest0-21swapper/211:43:392
2471889993120,8cyclictest2489076-21/usr/sbin/munin08:06:012
2471889993118,9cyclictest2576063-21ntpq11:01:052
2471889993118,9cyclictest2576063-21ntpq11:01:052
2471889993116,10cyclictest2606545-21ssh11:51:112
247188899318,4cyclictest171rcu_preempt12:01:131
2471888993123,5cyclictest420-21avahi-daemon07:56:111
2471888993121,7cyclictest2593681-21aten_r2power_po11:30:561
2471888993121,6cyclictest1656768-21cups-browsed07:36:101
2471887993126,2cyclictest0-21swapper/010:50:590
2471887993125,4cyclictest2616149-21head12:06:230
2471887993124,5cyclictest2434615-21kworker/u8:13-xprtiod07:46:180
2471887993123,6cyclictest2471909-21latency_hist07:10:020
2471887993123,5cyclictest439-21systemd-logind08:45:020
2471887993122,7cyclictest2618909-21switchtime12:11:240
2471887993116,11cyclictest2572296-21/usr/sbin/munin10:55:550
2471887993115,12cyclictest567-21gdbus10:00:320
2471887993115,12cyclictest567-21gdbus10:00:320
2471889993020,7cyclictest2588181-21ntpq11:21:042
2471889993020,6cyclictest2483309-21ntpq07:46:092
2471889993014,13cyclictest2551298-21ssh10:20:572
2471888993024,3cyclictest2580944-21kworker/u8:5-xprtiod11:46:011
2471888993023,4cyclictest2488914-21fschecks_time08:05:591
2471888993023,4cyclictest2476405-21kthreadd07:25:011
2471888993023,4cyclictest1000-21mta-sts-daemon12:07:101
2471888993022,5cyclictest420-21avahi-daemon11:05:011
2471888993022,5cyclictest420-21avahi-daemon11:05:011
2471888993022,5cyclictest2555342-21unixbench_multi10:26:181
2471888993021,6cyclictest439-21systemd-logind07:10:011
2471888993020,7cyclictest1656768-21cups-browsed11:10:041
2471888993020,7cyclictest1656768-21cups-browsed11:10:041
2471887993025,2cyclictest0-21swapper/009:40:010
2471887993024,3cyclictest2597278-21ntpq11:36:050
2471887993024,3cyclictest2590109-21latency_hist11:25:020
2471887993023,4cyclictest2507869-21munin-run09:10:010
247188799302,20cyclictest12650irq/32-eth010:26:100
2471887993021,7cyclictest485-21in:imuxsock09:00:010
2471887993021,6cyclictest1656768-21cups-browsed08:06:050
2471887993016,7cyclictest1656768-21cups-browsed11:56:070
2471888992924,2cyclictest0-21swapper/107:51:171
2471888992924,2cyclictest0-21swapper/107:51:171
2471888992923,3cyclictest0-21swapper/108:50:021
2471888992922,4cyclictest2474874-21munin-plugin-st07:20:011
247188899292,22cyclictest2588214-21ssh11:21:061
247188899292,20cyclictest2557943-21ssh10:31:111
2471888992916,9cyclictest567-21gdbus09:28:071
247188799293,22cyclictest2630642-21ssh12:31:100
247188799293,18cyclictest2621626-21ssh12:16:120
2471887992922,5cyclictest420-21avahi-daemon11:05:390
2471887992922,5cyclictest420-21avahi-daemon11:05:390
247188799292,22cyclictest2624550-21ntpq12:21:080
2471887992920,7cyclictest2606402-21ntp_kernel_pll_11:51:060
2471888992820,5cyclictest2491384-21systemd-run08:15:011
2471888992819,5cyclictest1656768-21cups-browsed07:14:091
2471889992713,10cyclictest2471898-21systemctl07:10:012
2471888992719,3cyclictest0-21swapper/108:26:171
2471888992718,7cyclictest2477124-21missed_timers07:26:011
247188799273,17cyclictest151ktimers/008:31:060
2471887992721,4cyclictest2603213-21missed_timers11:46:000
2471887992721,3cyclictest2510945-21kworker/u8:1-xprtiod09:55:010
2471887992721,3cyclictest1001-21mta-sts-daemon09:29:570
2471887992716,6cyclictest0-21swapper/008:50:010
247188899262,19cyclictest2629938-21/usr/sbin/munin12:30:561
247188799263,6cyclictest171rcu_preempt07:26:080
247188899258,14cyclictest2483103-21sed07:46:021
2471888992518,3cyclictest0-21swapper/108:16:051
247188899250,3cyclictest171rcu_preempt08:01:081
2471888992416,5cyclictest1656768-21cups-browsed08:34:331
2471887992418,3cyclictest0-21swapper/007:21:080
2471887992318,3cyclictest151ktimers/007:31:080
2471887992216,4cyclictest2473881-21fschecks_count07:16:000
2471887992216,3cyclictest0-21swapper/008:55:010
247188799221,16cyclictest2490716-21ntpq08:11:040
247189099193,9cyclictest0-21swapper/311:36:073
247189099191,11cyclictest0-21swapper/310:46:103
247189099181,9cyclictest0-21swapper/310:26:093
247189099181,9cyclictest0-21swapper/309:21:133
247189099181,7cyclictest0-21swapper/311:56:043
247189099171,8cyclictest0-21swapper/312:11:113
247189099171,8cyclictest0-21swapper/311:21:083
247189099171,8cyclictest0-21swapper/309:16:113
247189099171,8cyclictest0-21swapper/309:06:073
247189099171,8cyclictest0-21swapper/308:01:083
247189099163,7cyclictest0-21swapper/312:01:053
247189099161,8cyclictest0-21swapper/312:31:133
247189099161,8cyclictest0-21swapper/311:06:093
247189099161,8cyclictest0-21swapper/311:06:093
247189099161,8cyclictest0-21swapper/310:41:113
247189099161,8cyclictest0-21swapper/310:31:073
247189099161,8cyclictest0-21swapper/310:11:073
247189099161,8cyclictest0-21swapper/310:06:053
247189099161,8cyclictest0-21swapper/310:06:053
247189099161,8cyclictest0-21swapper/309:51:053
247189099161,8cyclictest0-21swapper/307:51:043
247189099161,8cyclictest0-21swapper/307:51:043
247189099161,10cyclictest0-21swapper/309:41:053
247189099152,7cyclictest0-21swapper/312:26:043
247189099151,9cyclictest0-21swapper/310:21:083
247189099151,8cyclictest0-21swapper/311:01:053
247189099151,8cyclictest0-21swapper/311:01:053
247189099151,7cyclictest0-21swapper/308:26:083
247189099151,7cyclictest0-21swapper/308:16:053
247189099151,6cyclictest0-21swapper/310:56:063
247189099151,6cyclictest0-21swapper/308:06:043
247189099143,7cyclictest0-21swapper/311:26:053
247189099143,6cyclictest0-21swapper/308:50:013
247189099142,7cyclictest0-21swapper/310:36:053
247189099142,6cyclictest0-21swapper/309:36:043
247189099142,5cyclictest0-21swapper/308:11:063
247189099141,8cyclictest0-21swapper/310:01:033
247189099141,8cyclictest0-21swapper/310:01:033
247189099141,8cyclictest0-21swapper/309:11:133
247189099141,7cyclictest0-21swapper/311:51:053
247189099141,7cyclictest0-21swapper/309:31:133
247189099141,7cyclictest0-21swapper/308:31:063
247189099141,7cyclictest0-21swapper/307:56:073
247189099141,7cyclictest0-21swapper/307:36:093
247189099141,6cyclictest0-21swapper/307:46:083
247189099141,6cyclictest0-21swapper/307:26:043
247189099132,7cyclictest0-21swapper/307:45:013
247189099132,6cyclictest0-21swapper/312:15:563
247189099131,7cyclictest0-21swapper/311:46:043
247189099131,7cyclictest0-21swapper/311:41:093
247189099131,7cyclictest0-21swapper/311:11:173
247189099131,7cyclictest0-21swapper/311:11:173
247189099131,7cyclictest0-21swapper/310:16:113
247189099131,7cyclictest0-21swapper/309:56:123
247189099131,7cyclictest0-21swapper/309:46:063
247189099131,6cyclictest0-21swapper/312:36:073
247189099131,6cyclictest0-21swapper/311:16:073
247189099131,6cyclictest0-21swapper/307:16:083
247189099122,6cyclictest0-21swapper/312:21:053
247189099121,7cyclictest0-21swapper/310:55:013
247189099121,7cyclictest0-21swapper/307:35:023
247189099121,6cyclictest0-21swapper/312:05:593
247189099121,6cyclictest0-21swapper/309:26:073
247189099121,6cyclictest0-21swapper/309:01:083
247189099121,6cyclictest0-21swapper/309:00:023
247189099121,6cyclictest0-21swapper/308:41:113
247189099121,5cyclictest0-21swapper/308:21:043
247189099121,5cyclictest0-21swapper/308:21:043
247189099112,6cyclictest0-21swapper/311:30:573
247189099111,4cyclictest0-21swapper/307:21:083
247189099110,6cyclictest0-21swapper/308:51:043
247189099101,5cyclictest0-21swapper/308:36:073
247189099101,5cyclictest0-21swapper/307:11:043
24718909991,4cyclictest0-21swapper/307:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional