You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-23 - 16:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Thu Oct 23, 2025 12:40:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2231534996039,16cyclictest0-21swapper/012:19:000
2231534995729,25cyclictest2312902-21latency_hist10:20:030
2231534995729,25cyclictest2312902-21latency_hist10:20:030
2231536995042,5cyclictest2254063-21systemd-run08:25:022
2231534995030,12cyclictest12450irq/32-eth009:59:000
2231535994939,6cyclictest1435145-21cups-browsed10:35:011
2231535994939,6cyclictest1435145-21cups-browsed10:35:011
2231534994941,5cyclictest2336194-21systemd-run10:55:010
2231534994941,5cyclictest2336194-21systemd-run10:55:010
2231534994919,8cyclictest12550irq/33-eth011:58:550
2231536994637,6cyclictest2283980-21true09:35:012
2231536994636,6cyclictest2349157-21latency_hist11:15:022
2231536994635,6cyclictest959-21mta-sts-daemon08:53:582
2231535994638,4cyclictest2355445-21sed11:24:131
2231534994628,12cyclictest12450irq/32-eth012:13:540
2231536994538,4cyclictest0-21swapper/211:54:012
2231536994537,5cyclictest266-21systemd-journal11:20:012
2231536994437,3cyclictest0-21swapper/212:40:012
2231535994437,3cyclictest0-21swapper/110:14:001
2231535994336,5cyclictest2341403-21missed_timers11:03:551
2231536994234,3cyclictest0-21swapper/211:04:052
2231536994233,6cyclictest2390977-21systemd-run12:15:022
2231536994233,5cyclictest2293354-21unixbench_singl09:49:072
2231535994233,5cyclictest2287155-21ls09:40:011
2231535994231,7cyclictest0-21swapper/109:10:011
2231534994235,3cyclictest0-21swapper/007:14:010
2231534994232,4cyclictest0-21swapper/011:19:010
2231534994219,13cyclictest12450irq/32-eth011:33:580
2231536994133,5cyclictest2270930-21unixbench_multi09:14:112
2231535994134,3cyclictest0-21swapper/109:00:011
2231535994133,5cyclictest2254048-21idleruntime-cro08:25:011
2231534994133,5cyclictest2347476-21aten_r2power_fr11:13:490
2231536994032,5cyclictest2335711-21unixbench-2d10:54:112
2231536994032,5cyclictest2335711-21unixbench-2d10:54:112
2231535994031,6cyclictest2279328-21aten_r2power_vo09:28:461
2231535994031,5cyclictest2373128-21cron11:50:011
223153599401,32cyclictest2311881-21ssh10:19:041
223153599401,32cyclictest2311881-21ssh10:19:041
2231534994032,5cyclictest2309048-21unixbench_multi10:14:070
223153499402,31cyclictest2331932-21ssh10:49:050
2231536993932,4cyclictest2312759-21cron10:20:012
2231536993932,4cyclictest2312759-21cron10:20:012
2231536993932,4cyclictest2287196-21kworker/u8:6+xprtiod09:40:012
2231536993831,4cyclictest2260036-21munin-plugin-st08:45:012
2231536993817,16cyclictest351ktimers/207:33:582
2231536993817,16cyclictest351ktimers/207:33:582
2231535993831,4cyclictest0-21swapper/110:38:461
2231535993829,6cyclictest2319742-21ssh10:30:011
2231535993829,6cyclictest2319742-21ssh10:30:011
223153499385,29cyclictest2279907-21sh09:28:570
2231534993831,4cyclictest2252562-21systemd-run08:20:010
2231534993823,11cyclictest2231763-21kworker/u8:1-nfsiod08:02:040
223153699373,24cyclictest2331755-21ssh10:48:582
2231536993731,3cyclictest0-21swapper/209:25:012
2231536993730,4cyclictest0-21swapper/209:55:022
2231535993729,6cyclictest0-21swapper/109:59:081
2231535993729,5cyclictest2385805-21grep12:08:491
2231535993729,5cyclictest2246552-21systemd-run08:00:021
2231535993728,6cyclictest421-21avahi-daemon09:16:231
2231535993726,8cyclictest2292570-21missed_timers09:48:521
2231534993731,4cyclictest2231533-21cyclictest11:23:480
2231536993628,5cyclictest2316260-21cron10:25:002
2231536993628,5cyclictest2316260-21cron10:25:002
2231536993628,5cyclictest2267704-21latency_hist09:10:022
2231536993628,4cyclictest0-21swapper/207:50:012
2231536993627,7cyclictest2302009-21needreboot10:03:562
2231536993627,6cyclictest422-21dbus-daemon11:09:142
223153699362,26cyclictest2322002-21ssh10:34:032
223153699362,26cyclictest2322002-21ssh10:34:032
2231535993630,3cyclictest0-21swapper/110:25:001
2231535993630,3cyclictest0-21swapper/110:25:001
2231535993629,4cyclictest2393045-21fschecks_time12:18:491
2231535993627,6cyclictest421-21avahi-daemon11:53:491
2231535993627,5cyclictest2258511-21cron08:40:011
2231535993626,6cyclictest422-21dbus-daemon10:50:011
2231535993624,9cyclictest2259395-21ntp_kernel_err08:43:521
2231534993627,7cyclictest1-21systemd10:32:480
2231534993627,7cyclictest1-21systemd10:32:480
2231534993625,8cyclictest2365758-21unixbench_multi11:39:070
2231534993623,2cyclictest0-21swapper/009:40:010
2231534993618,12cyclictest2273681-21sh09:19:010
223153499361,30cyclictest2386228-21ssh12:08:580
223153499361,27cyclictest2404331-21ssh12:33:570
2231536993528,4cyclictest955-21mta-sts-daemon09:15:482
2231536993527,5cyclictest2235085-21grep07:23:562
2231536993524,7cyclictest421-21avahi-daemon10:26:072
2231536993524,7cyclictest421-21avahi-daemon10:26:072
223153599353,23cyclictest2296139-21ssh09:54:011
2231535993527,5cyclictest2390999-21latency_hist12:15:021
2231535993527,5cyclictest2231712-21latency_hist07:10:031
2231535993527,4cyclictest0-21swapper/112:24:001
2231535993523,7cyclictest0-21swapper/112:39:011
223153599352,24cyclictest2348059-21ssh11:13:581
223153599351,29cyclictest2378816-21ssh11:59:011
2231534993527,5cyclictest2237545-21systemd-run07:30:010
2231534993526,6cyclictest151ktimers/009:53:590
2231534993524,7cyclictest12450irq/32-eth011:28:580
2231534993518,12cyclictest12450irq/32-eth010:24:000
2231534993518,12cyclictest12450irq/32-eth010:24:000
2231536993426,5cyclictest2240522-21munin-plugin-st07:40:012
2231535993427,3cyclictest0-21swapper/110:08:491
2231535993426,5cyclictest2401664-21ssh12:30:011
2231535993426,5cyclictest2245018-21munin-run07:55:011
223153499343,25cyclictest2408036-21ssh12:38:570
2231534993428,4cyclictest2267704-21latency_hist09:10:020
2231534993428,3cyclictest2319553-21kworker/u8:14+xprtiod11:50:010
2231534993427,5cyclictest2341103-21df11:03:490
2231534993426,5cyclictest12550irq/33-eth010:08:550
2231534993425,6cyclictest421-21avahi-daemon09:13:120
2231534993423,7cyclictest151ktimers/007:53:550
2231534993422,4cyclictest956-21mta-sts-daemon12:30:020
2231534993416,11cyclictest151ktimers/012:23:580
2231536993327,4cyclictest2231533-21cyclictest08:14:002
2231536993327,4cyclictest0-21swapper/212:03:582
2231536993326,4cyclictest2234513-21sh07:20:012
2231536993325,5cyclictest2246516-21basename08:00:012
2231536993324,6cyclictest2355816-21cut11:25:012
2231536993322,8cyclictest2404793-21timerwakeupswit12:34:072
2231536993322,8cyclictest2338052-21needreboot10:58:552
2231536993322,8cyclictest2338052-21needreboot10:58:552
2231536993322,8cyclictest2279181-21basename09:28:442
2231535993327,3cyclictest2297261-21kworker/u8:8-nfsiod10:05:021
2231535993327,3cyclictest0-21swapper/108:08:481
2231535993327,3cyclictest0-21swapper/108:08:481
2231535993324,5cyclictest0-21swapper/109:15:011
2231534993327,3cyclictest0-21swapper/010:45:010
2231534993327,3cyclictest0-21swapper/010:45:010
2231534993325,5cyclictest474-21in:imuxsock08:50:010
2231534993324,7cyclictest2239906-21ntp_kernel_pll_07:38:540
2231534993321,9cyclictest2292925-21find09:49:020
2231534993319,12cyclictest151ktimers/007:43:560
2231534993319,12cyclictest151ktimers/007:43:560
223153699326,4cyclictest0-21swapper/207:13:592
2231536993226,3cyclictest0-21swapper/212:20:012
2231536993225,4cyclictest2263926-21ntp_kernel_err08:58:542
2231536993225,3cyclictest0-21swapper/212:08:512
2231536993224,5cyclictest2305963-21unixbench_multi10:09:072
2231536993223,5cyclictest2252426-21unixbench_multi08:19:072
2231535993225,4cyclictest2369837-21ssh11:45:011
2231535993225,4cyclictest2336173-21sh10:55:011
2231535993225,4cyclictest2336173-21sh10:55:011
2231535993221,7cyclictest2328737-21sh10:44:021
2231535993221,7cyclictest2328737-21sh10:44:021
2231535993221,7cyclictest0-21swapper/111:38:551
2231534993227,2cyclictest0-21swapper/010:40:000
2231534993226,3cyclictest151ktimers/009:03:520
2231534993221,8cyclictest2228880-21kworker/u8:3-xprtiod07:20:010
223153699318,15cyclictest959-21mta-sts-daemon07:53:562
2231536993125,2cyclictest0-21swapper/212:25:012
2231536993123,5cyclictest0-21swapper/211:58:452
2231536993122,6cyclictest421-21avahi-daemon11:42:582
2231536993121,7cyclictest2299928-21sh10:00:022
2231536993120,8cyclictest2358133-21missed_timers11:28:522
223153599312,23cyclictest2276866-21ssh09:23:591
223153599312,21cyclictest2358353-21awk11:28:581
223153599312,20cyclictest698-21cron10:59:001
223153599312,20cyclictest698-21cron10:59:001
2231535993118,9cyclictest2403880-21fschecks_count12:33:481
2231534993125,2cyclictest0-21swapper/007:50:010
2231534993123,6cyclictest2257040-21sh08:35:020
2231534993123,5cyclictest421-21avahi-daemon12:02:010
2231534993118,8cyclictest12450irq/32-eth011:09:140
2231534993118,10cyclictest698-21cron08:10:010
2231534993118,10cyclictest698-21cron08:10:010
2231536993023,3cyclictest0-21swapper/210:13:522
2231536993022,5cyclictest2241065-21fschecks_count07:43:482
2231536993022,5cyclictest2241065-21fschecks_count07:43:482
2231536993022,4cyclictest0-21swapper/212:28:472
2231536993019,8cyclictest2372743-21timerwakeupswit11:49:072
2231535993024,3cyclictest0-21swapper/107:40:011
2231535993022,5cyclictest2242857-21needreboot07:48:531
2231535993022,5cyclictest2234471-21unixbench_singl07:19:091
2231535993021,7cyclictest2382440-21ntp_kernel_pll_12:03:561
2231535993019,8cyclictest0-21swapper/111:09:121
223153499308,12cyclictest12450irq/32-eth008:23:550
223153499302,23cyclictest2289482-21ssh09:43:570
2231534993019,8cyclictest266-21systemd-journal10:30:010
2231534993019,8cyclictest266-21systemd-journal10:30:010
2231534993015,13cyclictest2337427-21/usr/sbin/munin10:59:080
2231534993015,13cyclictest2337427-21/usr/sbin/munin10:59:080
2231534993015,12cyclictest0-21swapper/007:34:060
2231534993015,12cyclictest0-21swapper/007:34:060
2231536992923,3cyclictest0-21swapper/209:43:552
2231536992923,3cyclictest0-21swapper/209:05:012
2231536992923,2cyclictest0-21swapper/210:45:012
2231536992923,2cyclictest0-21swapper/210:45:012
2231536992922,4cyclictest0-21swapper/208:40:012
223153599292,21cyclictest2361531-21ssh11:33:541
2231535992920,6cyclictest421-21avahi-daemon07:27:541
223153499292,22cyclictest2276872-21ssh09:23:590
2231536992821,3cyclictest0-21swapper/211:34:072
2231536992819,6cyclictest2325134-21ntpq10:38:522
2231535992823,2cyclictest0-21swapper/107:25:011
2231535992822,3cyclictest956-21mta-sts-daemon08:04:031
2231535992821,3cyclictest0-21swapper/108:30:021
2231535992820,5cyclictest2242036-21cut07:45:011
2231535992820,5cyclictest2242036-21cut07:45:011
223153599281,9cyclictest171rcu_preempt08:52:361
223153499282,20cyclictest2368838-21ssh11:43:580
2231536992721,3cyclictest0-21swapper/211:39:072
2231536992720,3cyclictest0-21swapper/208:30:012
2231536992719,6cyclictest2237558-21latency_hist07:30:022
223153599273,12cyclictest2267580-21diskmemload09:43:101
2231535992720,3cyclictest0-21swapper/111:19:121
2231535992717,6cyclictest0-21swapper/109:34:011
2231535992715,3cyclictest171rcu_preempt08:19:071
223153499279,13cyclictest421-21avahi-daemon09:33:560
2231534992721,4cyclictest2262996-21unixbench_singl08:54:100
2231534992721,3cyclictest0-21swapper/008:45:020
2231534992720,4cyclictest2258556-21systemd-run08:40:010
2231534992718,5cyclictest0-21swapper/007:24:030
2231534992712,9cyclictest12450irq/32-eth010:04:010
2231536992619,4cyclictest957-21mta-sts-daemon08:01:392
223153599262,4cyclictest171rcu_preempt07:33:581
223153599262,4cyclictest171rcu_preempt07:33:581
2231535992620,4cyclictest957-21mta-sts-daemon08:48:561
2231534992619,4cyclictest2246501-21cron08:00:010
2231536992519,3cyclictest0-21swapper/208:35:012
2231535992516,2cyclictest0-21swapper/108:33:531
223153499258,8cyclictest422-21dbus-daemon07:10:010
2231534992519,4cyclictest2255429-21unixbench_multi08:29:070
2231534992517,6cyclictest2264323-21timerandwakeup08:59:060
223153499251,20cyclictest2375465-21ntpq11:54:050
223153699241,19cyclictest1-21systemd07:10:012
2231535992417,4cyclictest2251017-21cron08:15:001
223153599241,17cyclictest2265566-21ntpq09:03:581
223153599232,16cyclictest2232561-21ps07:14:041
2231536992217,2cyclictest0-21swapper/208:48:432
2231536992216,3cyclictest0-21swapper/208:08:502
2231536992216,3cyclictest0-21swapper/208:08:502
2231534992216,4cyclictest1435145-21cups-browsed08:11:170
223153799183,8cyclictest0-21swapper/312:03:593
223153799172,8cyclictest0-21swapper/310:08:573
223153799171,8cyclictest0-21swapper/311:13:583
223153799162,9cyclictest0-21swapper/308:02:053
223153799162,8cyclictest0-21swapper/312:09:003
223153799162,8cyclictest0-21swapper/309:48:593
223153799162,8cyclictest0-21swapper/309:28:553
223153799162,7cyclictest0-21swapper/312:23:583
223153799162,7cyclictest0-21swapper/309:53:583
223153799161,9cyclictest0-21swapper/311:33:543
223153799161,9cyclictest0-21swapper/310:14:083
223153799161,7cyclictest0-21swapper/308:52:373
223153799154,7cyclictest0-21swapper/311:28:543
223153799153,6cyclictest0-21swapper/311:24:033
223153799152,8cyclictest0-21swapper/312:13:573
223153799152,8cyclictest0-21swapper/310:25:013
223153799152,8cyclictest0-21swapper/310:25:013
223153799152,8cyclictest0-21swapper/309:59:003
223153799152,5cyclictest0-21swapper/307:13:583
223153799151,8cyclictest0-21swapper/310:44:003
223153799151,8cyclictest0-21swapper/310:44:003
223153799151,7cyclictest0-21swapper/308:13:553
223153799143,8cyclictest0-21swapper/311:58:543
223153799143,6cyclictest0-21swapper/310:28:553
223153799143,6cyclictest0-21swapper/310:28:553
223153799142,7cyclictest0-21swapper/311:48:563
223153799142,7cyclictest0-21swapper/310:54:023
223153799142,7cyclictest0-21swapper/310:54:023
223153799142,7cyclictest0-21swapper/309:38:573
223153799142,7cyclictest0-21swapper/309:18:593
223153799142,6cyclictest0-21swapper/311:09:123
223153799141,8cyclictest0-21swapper/312:28:583
223153799141,8cyclictest0-21swapper/309:14:003
223153799141,7cyclictest0-21swapper/312:38:533
223153799141,7cyclictest0-21swapper/311:54:043
223153799141,7cyclictest0-21swapper/311:35:483
223153799141,7cyclictest0-21swapper/309:43:563
223153799141,7cyclictest0-21swapper/309:34:003
223153799141,6cyclictest0-21swapper/308:28:543
223153799134,6cyclictest0-21swapper/310:49:023
223153799133,6cyclictest0-21swapper/312:33:583
223153799133,6cyclictest0-21swapper/312:18:553
223153799132,6cyclictest0-21swapper/310:38:463
223153799132,6cyclictest0-21swapper/309:23:493
223153799131,8cyclictest0-21swapper/310:18:513
223153799131,8cyclictest0-21swapper/310:18:513
223153799131,8cyclictest0-21swapper/307:44:013
223153799131,8cyclictest0-21swapper/307:44:013
223153799131,7cyclictest0-21swapper/311:43:563
223153799131,7cyclictest0-21swapper/311:18:453
223153799131,7cyclictest0-21swapper/310:33:593
223153799131,7cyclictest0-21swapper/310:33:593
223153799131,7cyclictest0-21swapper/307:38:563
223153799131,6cyclictest0-21swapper/310:03:463
223153799131,6cyclictest0-21swapper/308:58:543
223153799131,6cyclictest0-21swapper/308:08:573
223153799131,6cyclictest0-21swapper/308:08:573
223153799131,6cyclictest0-21swapper/307:24:023
223153799130,6cyclictest0-21swapper/307:48:543
223153799123,4cyclictest0-21swapper/307:33:583
223153799123,4cyclictest0-21swapper/307:33:583
223153799122,6cyclictest0-21swapper/308:23:553
223153799122,5cyclictest0-21swapper/310:58:483
223153799122,5cyclictest0-21swapper/310:58:483
223153799122,5cyclictest0-21swapper/308:38:543
223153799122,5cyclictest0-21swapper/307:58:543
223153799122,4cyclictest0-21swapper/308:18:543
223153799121,7cyclictest0-21swapper/311:03:513
223153799121,6cyclictest0-21swapper/309:10:013
223153799111,6cyclictest0-21swapper/309:03:553
223153799111,6cyclictest0-21swapper/307:53:543
223153799111,6cyclictest0-21swapper/307:28:563
223153799101,6cyclictest0-21swapper/308:47:483
223153799101,6cyclictest0-21swapper/308:43:533
22315379991,5cyclictest0-21swapper/308:33:443
22315379991,5cyclictest0-21swapper/307:18:463
22315379982,5cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional