You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 22:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Thu Nov 20, 2025 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8313992623,2cyclictest32471-21awk08:00:001
832199251,23cyclictest27536-21cron11:10:003
8321992421,2cyclictest32474-21munin-run08:00:003
832199240,1cyclictest11756-21apt-get09:32:003
831799240,23cyclictest30358-21cron11:16:592
831799240,23cyclictest23382-21cron09:55:002
8313992422,1cyclictest6847-21sendmail-msp12:40:001
8313992421,2cyclictest313-21systemd-journal10:40:001
8309992422,1cyclictest14867-21perf08:30:000
8321992321,1cyclictest6840-21munin-run12:40:003
832199230,22cyclictest9557-21sendmail11:40:003
8317992321,1cyclictest575-21munin-run10:15:002
831799230,2cyclictest13018-21awk10:40:002
831799230,23cyclictest0-21swapper/209:46:592
8313992321,1cyclictest0-21swapper/109:40:011
8309992320,2cyclictest3421-21dump-pmu-power10:20:000
8309992320,2cyclictest3421-21dump-pmu-power10:20:000
8321992221,1cyclictest34-21ksoftirqd/309:04:463
832199220,21cyclictest10478-21apt-get08:21:593
832199220,21cyclictest0-21swapper/309:16:593
8317992222,0cyclictest0-21swapper/209:07:032
8317992220,1cyclictest29338-21munin-run09:00:012
8317992220,1cyclictest17943-21grep07:30:002
8317992220,1cyclictest0-21swapper/209:45:012
8317992220,1cyclictest0-21swapper/208:47:022
831799220,21cyclictest5654-21apt-get08:12:002
831799220,21cyclictest15472-21apt07:24:592
831799220,21cyclictest0-21swapper/209:22:002
831799220,21cyclictest0-21swapper/208:36:592
8313992222,0cyclictest22-21ksoftirqd/112:31:251
8313992221,1cyclictest0-21swapper/108:10:011
8313992220,1cyclictest4123-21apt09:14:591
8313992220,1cyclictest27563-21munin-run11:09:591
8313992220,1cyclictest0-21swapper/111:47:001
831399221,20cyclictest0-21swapper/109:07:031
831399220,21cyclictest26237-21apt-get10:02:001
831399220,21cyclictest0-21swapper/109:51:591
831399220,21cyclictest0-21swapper/109:32:001
831399220,1cyclictest9556-21basename11:40:001
8309992220,1cyclictest6811-21cron12:39:590
8309992220,1cyclictest21024-21munin-run09:50:000
8309992220,1cyclictest14780-21apt-get11:52:020
8309992219,3cyclictest0-21swapper/010:51:590
830999220,21cyclictest28629-21apt-get10:06:580
830999220,21cyclictest0-21swapper/011:02:010
830999220,1cyclictest22755-21wc10:59:590
830999220,1cyclictest10660-21turbostat.cron07:15:000
8321992121,0cyclictest0-21swapper/310:02:003
8321992120,1cyclictest9982-21apt-get11:41:593
8321992120,1cyclictest8938-21sh09:25:003
8321992120,1cyclictest2368-21perf11:25:003
8321992120,1cyclictest20749-21apt-get10:57:003
8321992120,1cyclictest14385-21perf11:50:003
8321992120,0cyclictest1408-21apt-get10:17:043
8321992120,0cyclictest1408-21apt-get10:17:043
8321992119,1cyclictest4100-21cron09:15:003
8321992119,1cyclictest30032-21sh07:55:003
832199210,20cyclictest32049-21apt-get12:26:593
832199210,20cyclictest27238-21apt-get12:17:033
832199210,20cyclictest16221-21sendmail-msp09:40:003
832199210,20cyclictest15954-21apt-get10:47:003
832199210,20cyclictest0-21swapper/310:42:033
832199210,1cyclictest17235-21apt-get11:57:043
8317992120,1cyclictest1625-21awk12:30:002
8317992120,1cyclictest0-21swapper/208:42:002
8317992119,1cyclictest4839-21apt-get08:10:012
831799210,20cyclictest7559-21apt-get11:36:592
831799210,20cyclictest25667-21apt-get07:47:022
831799210,20cyclictest0-21swapper/210:32:002
831799210,20cyclictest0-21swapper/208:31:582
831799210,1cyclictest13080-21dump-pmu-power07:20:012
831799210,0cyclictest4535-21apt-get09:16:592
8313992121,0cyclictest0-21swapper/112:07:001
8313992121,0cyclictest0-21swapper/111:12:011
8313992121,0cyclictest0-21swapper/110:12:021
8313992121,0cyclictest0-21swapper/108:01:581
8313992121,0cyclictest0-21swapper/107:21:571
8313992120,0cyclictest0-21swapper/108:21:591
8313992119,1cyclictest17288-21cron08:35:001
831399210,21cyclictest0-21swapper/109:04:461
831399210,1cyclictest8944-21turbostat09:25:001
831399210,1cyclictest6554-21awk09:19:591
831399210,0cyclictest20148-21apt-get08:42:001
831399210,0cyclictest0-21swapper/112:22:021
831399210,0cyclictest0-21swapper/110:42:031
831399210,0cyclictest0-21swapper/109:27:011
8309992120,1cyclictest6231-21apt-get10:27:000
8309992120,0cyclictest17729-21apt-get08:36:590
8309992119,1cyclictest25831-21awk09:59:590
830999210,21cyclictest0-21swapper/008:52:020
830999210,21cyclictest0-21swapper/007:26:590
830999210,20cyclictest22546-21apt-get08:47:030
830999210,1cyclictest612-21gdbus09:00:000
830999210,1cyclictest4839-21apt-get08:10:020
830999210,1cyclictest13454-21apt-get10:42:030
830999210,0cyclictest30354-21apt-get11:17:000
830999210,0cyclictest27958-21apt-get11:12:010
830999210,0cyclictest0-21swapper/007:56:580
8321992020,0cyclictest9363-21apt-get09:27:013
8321992020,0cyclictest11077-21apt-get07:16:593
8321992020,0cyclictest0-21swapper/312:22:013
8321992020,0cyclictest0-21swapper/309:51:593
8321992020,0cyclictest0-21swapper/308:10:023
8321992019,1cyclictest22044-21apt-get12:07:003
8321992019,0cyclictest414-21apt-get08:01:593
832199200,20cyclictest32155-21apt-get09:07:033
832199200,20cyclictest0-21swapper/309:45:003
832199200,0cyclictest31030-21apt-get10:12:033
832199200,0cyclictest19027-21apt-get09:46:593
832199200,0cyclictest0-21swapper/311:12:013
832199200,0cyclictest0-21swapper/308:56:573
8317992020,0cyclictest0-21swapper/212:06:582
8317992020,0cyclictest0-21swapper/210:57:002
8317992020,0cyclictest0-21swapper/210:47:002
8317992020,0cyclictest0-21swapper/209:56:592
8317992020,0cyclictest0-21swapper/208:01:592
8317992020,0cyclictest0-21swapper/207:32:012
8317992019,1cyclictest29949-21awk11:15:002
8317992018,1cyclictest0-21swapper/207:45:002
8313992020,0cyclictest0-21swapper/111:41:581
8313992020,0cyclictest0-21swapper/110:47:001
8313992020,0cyclictest0-21swapper/110:06:571
8313992020,0cyclictest0-21swapper/109:56:591
8313992019,0cyclictest0-21swapper/107:16:581
8313992018,1cyclictest17945-21perf07:30:001
8313992018,1cyclictest0-21swapper/111:54:591
831399200,20cyclictest0-21swapper/112:15:011
831399200,20cyclictest0-21swapper/108:26:581
831399200,20cyclictest0-21swapper/108:06:581
831399200,0cyclictest0-21swapper/109:45:011
831399200,0cyclictest0-21swapper/107:51:591
8309992020,0cyclictest0-21swapper/012:05:010
8309992020,0cyclictest0-21swapper/011:27:000
8309992020,0cyclictest0-21swapper/011:22:020
8309992020,0cyclictest0-21swapper/010:22:010
8309992020,0cyclictest0-21swapper/009:42:000
8309992020,0cyclictest0-21swapper/009:27:010
8309992020,0cyclictest0-21swapper/007:47:010
8309992020,0cyclictest0-21swapper/007:16:590
8309992018,1cyclictest0-21swapper/007:25:000
830999200,0cyclictest26419-21apt-get12:15:010
830999200,0cyclictest0-21swapper/008:42:000
830999200,0cyclictest0-21swapper/008:17:010
8321991919,0cyclictest2863-21apt-get08:06:593
8321991919,0cyclictest0-21swapper/308:26:583
8321991918,1cyclictest8634-21apt-get10:31:593
8321991917,1cyclictest593-21dbus-daemon08:35:003
8321991917,1cyclictest12993-21cron10:39:593
832199190,1cyclictest13495-21apt-get07:21:573
8317991919,0cyclictest0-21swapper/210:52:002
8317991919,0cyclictest0-21swapper/208:52:022
8317991919,0cyclictest0-21swapper/208:17:012
8317991919,0cyclictest0-21swapper/207:36:592
8317991918,1cyclictest3835-21apt-get10:22:002
8317991918,0cyclictest0-21swapper/207:56:592
8317991916,2cyclictest3381-21cron10:20:002
8317991916,2cyclictest3381-21cron10:19:592
831799190,0cyclictest0-21swapper/209:36:582
8313991919,0cyclictest0-21swapper/111:32:021
8313991919,0cyclictest0-21swapper/110:32:001
8309991919,0cyclictest0-21swapper/012:26:590
8309991919,0cyclictest0-21swapper/012:22:020
8309991919,0cyclictest0-21swapper/011:36:590
8309991919,0cyclictest0-21swapper/011:32:020
8309991919,0cyclictest0-21swapper/010:12:020
8309991919,0cyclictest0-21swapper/009:11:570
8309991919,0cyclictest0-21swapper/009:07:030
8309991919,0cyclictest0-21swapper/007:51:570
8309991917,1cyclictest22863-21sendmail-msp07:40:000
8321991818,0cyclictest0-21swapper/310:06:583
8321991818,0cyclictest0-21swapper/309:56:593
8321991818,0cyclictest0-21swapper/308:36:583
8321991816,1cyclictest16836-21munin-plugin-st11:55:003
8317991818,0cyclictest0-21swapper/211:22:012
8317991818,0cyclictest0-21swapper/208:21:592
8317991817,0cyclictest2764-21apt-get11:27:012
8317991817,0cyclictest0-21swapper/209:32:002
831399182,15cyclictest22760-21perf11:00:001
8313991818,0cyclictest0-21swapper/111:22:021
8313991818,0cyclictest0-21swapper/110:22:001
8309991818,0cyclictest25563-21apt-get11:07:000
8309991818,0cyclictest0-21swapper/010:36:590
8309991818,0cyclictest0-21swapper/009:22:010
8309991817,1cyclictest28239-21awk10:04:590
8309991817,1cyclictest0-21swapper/012:10:010
830999180,17cyclictest16225-21perf09:40:010
8321991717,0cyclictest5165-21apt-get11:32:023
8321991716,1cyclictest17952-21awk07:30:003
8317991717,0cyclictest0-21swapper/212:22:032
8317991717,0cyclictest0-21swapper/211:32:022
8317991717,0cyclictest0-21swapper/211:07:012
8317991717,0cyclictest0-21swapper/211:02:012
8317991717,0cyclictest0-21swapper/210:27:002
8317991717,0cyclictest0-21swapper/210:02:002
8313991717,0cyclictest0-21swapper/112:26:591
8313991717,0cyclictest0-21swapper/111:01:591
8313991717,0cyclictest0-21swapper/108:52:011
8313991717,0cyclictest0-21swapper/107:47:021
8313991717,0cyclictest0-21swapper/107:42:001
8313991717,0cyclictest0-21swapper/107:11:591
8313991716,0cyclictest19653-21apt-get12:02:041
8313991716,0cyclictest0-21swapper/108:36:591
8313991715,1cyclictest4764-21latency_hist11:30:001
8313991715,1cyclictest26396-21cron12:15:001
8309991717,0cyclictest0-21swapper/011:57:030
8309991717,0cyclictest0-21swapper/010:32:000
8309991716,1cyclictest23267-21apt-get07:42:000
8309991716,0cyclictest0-21swapper/011:47:000
8321991616,0cyclictest23167-21apt-get11:02:003
8321991616,0cyclictest0-21swapper/312:30:023
8321991616,0cyclictest0-21swapper/310:22:003
8321991616,0cyclictest0-21swapper/308:41:593
8321991615,1cyclictest8679-21apt-get07:11:593
8321991615,1cyclictest24941-21apt-get08:52:013
8321991614,2cyclictest26419-21apt-get12:15:003
8321991613,2cyclictest4763-21idleruntime-cro11:29:593
8317991616,0cyclictest0-21swapper/212:15:012
8317991615,0cyclictest0-21swapper/208:26:572
8317991614,1cyclictest0-21swapper/207:15:012
831799160,15cyclictest766-21snmpd09:03:002
831799160,15cyclictest0-21swapper/211:57:132
831399162,14cyclictest211rcuc/107:38:381
8313991615,1cyclictest0-21swapper/109:46:591
8313991614,1cyclictest593-21dbus-daemon12:00:001
8313991614,1cyclictest28225-21sed08:57:011
8313991613,2cyclictest3444-21ls10:20:001
8313991613,2cyclictest3444-21ls10:20:001
8309991616,0cyclictest0-21swapper/009:51:580
8321991515,0cyclictest0-21swapper/307:47:023
8321991515,0cyclictest0-21swapper/307:42:003
8321991514,1cyclictest0-21swapper/312:02:043
8321991514,0cyclictest0-21swapper/310:27:003
8321991513,1cyclictest24189-21unixbench_multi08:47:123
8317991515,0cyclictest0-21swapper/207:51:592
8313991513,1cyclictest0-21swapper/108:47:081
8313991512,2cyclictest19239-21sed07:32:051
8309991515,0cyclictest0-21swapper/008:21:590
8309991515,0cyclictest0-21swapper/008:02:000
832199146,7cyclictest21791-21ntp_states07:37:043
832199140,14cyclictest0-21swapper/310:52:583
832199140,0cyclictest0-21swapper/308:16:503
832199140,0cyclictest0-21swapper/307:30:353
8317991414,0cyclictest0-21swapper/212:00:472
8317991414,0cyclictest0-21swapper/210:06:582
8317991413,0cyclictest0-21swapper/212:30:022
831799140,14cyclictest0-21swapper/211:45:482
831799140,14cyclictest0-21swapper/210:40:252
831799140,14cyclictest0-21swapper/209:12:092
831799140,13cyclictest0-21swapper/212:37:022
831799140,0cyclictest0-21swapper/212:12:082
831799140,0cyclictest0-21swapper/209:26:292
8313991414,0cyclictest0-21swapper/108:17:311
8313991413,1cyclictest30321-21/usr/sbin/munin11:17:061
831399140,0cyclictest0-21swapper/110:50:091
8309991414,0cyclictest0-21swapper/009:32:000
8309991414,0cyclictest0-21swapper/007:30:040
830999140,14cyclictest0-21swapper/011:44:140
830999140,14cyclictest0-21swapper/010:45:410
830999140,14cyclictest0-21swapper/009:00:460
830999140,14cyclictest0-21swapper/008:32:130
830999140,14cyclictest0-21swapper/008:06:560
8321991312,1cyclictest0-21swapper/311:17:063
8317991212,0cyclictest0-21swapper/211:44:242
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional