You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 11:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot3s.osadl.org (updated Sat Jul 12, 2025 00:45:04)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
403727699113108,3cyclictest0-21swapper/000:05:020
4037276998274,6cyclictest12-21ksoftirqd/023:40:020
4037276997829,45cyclictest4043966-21sshd21:19:220
4037276997829,45cyclictest4043966-21sshd21:19:210
403727699731,5cyclictest0-21swapper/000:40:010
4037291997122,44cyclictest235-21systemd-journal21:45:013
403727699703,62cyclictest1-21systemd22:46:220
4037276997012,54cyclictest4048379-21sshd22:40:220
4037276996935,29cyclictest272-21dbus-daemon22:30:000
4037276996932,34cyclictest16950irq/224-800000020:54:210
403727699682,61cyclictest1-21systemd22:15:220
403727699682,32cyclictest4045972-21sed21:55:020
403727699682,32cyclictest4045972-21sed21:55:020
403727699673,59cyclictest4038535-21munin-run19:35:000
403727699671,11cyclictest4046782-21latency_hist22:10:020
403727699671,11cyclictest4046782-21latency_hist22:10:020
403727699661,2cyclictest0-21swapper/022:53:240
403727699661,13cyclictest4049976-21sshd23:12:380
403727699652,58cyclictest4046337-21munin-run22:05:000
403727699651,3cyclictest0-21swapper/020:10:030
4037291996430,30cyclictest37-21ksoftirqd/300:00:013
4037291996422,37cyclictest4050315-21latency_hist23:20:023
4037291996422,37cyclictest4050315-21latency_hist23:20:013
4037291996418,40cyclictest4038534-21systemd-run19:35:003
4037276996437,24cyclictest1-21systemd21:00:010
403727699642,57cyclictest4039463-21latency_hist19:50:020
403727699641,57cyclictest4050849-21latency_hist23:30:010
4037291996318,40cyclictest272-21dbus-daemon21:39:593
403727699632,56cyclictest4052348-40ogrotate00:00:000
403727699632,55cyclictest4042161-21cron20:45:010
4037276996318,40cyclictest3477409-1kworker/0:1H19:22:060
4037291996226,31cyclictest272-21dbus-daemon23:00:003
403727699622,55cyclictest4053881-21sshd00:25:330
403727699622,55cyclictest4043434-21modprobe21:07:210
403727699613,53cyclictest4041138-21munin-run20:25:000
403727699612,54cyclictest4041901-21munin-run20:40:000
403727699612,52cyclictest4048124-21cron22:40:000
4037276996124,34cyclictest4037219-21latency_hist19:10:020
403727699611,8cyclictest4053630-21sshd00:20:320
403727699611,7cyclictest4038756-21sshd19:39:210
403729199601,21cyclictest1-21systemd00:05:013
4037285996044,12cyclictest4041881-21cron20:40:002
403727699603,52cyclictest0-21swapper/019:30:020
403727699601,7cyclictest4049187-21sshd22:56:210
403727699601,3cyclictest0-21swapper/021:32:210
403729199596,9cyclictest4048125-21modprobe22:40:003
403729199592,20cyclictest4050729-21latency_hist23:25:023
403729199592,11cyclictest4049487-21cron23:05:013
4037291995917,8cyclictest4040319-21sshd20:07:213
4037285995911,6cyclictest131rcu_preempt23:50:002
403727699593,51cyclictest4040225-21date20:05:020
403727699592,6cyclictest4037713-21sshd19:17:210
403727699592,53cyclictest4051794-21sshd23:45:230
403727699592,53cyclictest4051047-21munin-run23:35:000
403727699592,52cyclictest363-21snmpd21:26:380
403727699592,13cyclictest4039792-21sshd19:58:200
403727699591,8cyclictest4045263-21sshd21:40:220
403727699591,8cyclictest4040813-21sshd20:15:220
403727699591,6cyclictest4039021-21sshd19:41:210
403729199582,19cyclictest4053148-21grep00:15:023
403729199582,14cyclictest4041449-21latency_hist20:30:013
403727699583,50cyclictest4041369-21sshd20:28:200
403727699582,6cyclictest4037491-21munin-run19:15:020
403727699581,5cyclictest4049714-21sshd23:05:210
403727699581,5cyclictest4044229-21sshd21:20:210
403727699581,53cyclictest1-21systemd21:02:200
403727699581,52cyclictest4050508-21modprobe23:20:230
403727699581,52cyclictest4046842-21munin-run22:15:010
403727699581,52cyclictest363-21snmpd23:02:140
403727699581,51cyclictest4041692-21sed20:35:020
403727699581,4cyclictest4045029-21munin-run21:40:000
4037276995812,42cyclictest4040609-21systemd-run20:15:000
403727699581,16cyclictest4053139-21systemd-run00:15:020
4037291995738,13cyclictest235-21systemd-journal23:09:593
403729199572,10cyclictest4044755-21cron21:35:003
4037291995718,34cyclictest4043007-21latency_hist21:00:023
4037291995717,4cyclictest271-21cron00:34:593
4037291995716,37cyclictest235-21systemd-journal22:52:353
4037291995715,36cyclictest0-21swapper/320:20:033
403727699571,5cyclictest0-21swapper/019:50:510
403727699571,4cyclictest4043687-21sshd21:10:210
403729199566,7cyclictest0-21swapper/321:10:013
403729199563,9cyclictest4039041-21modprobe19:45:003
403729199563,14cyclictest4037481-21grep19:15:013
403729199562,18cyclictest1-21systemd22:25:023
403729199562,11cyclictest4045863-21cat21:55:013
403729199562,11cyclictest4045863-21cat21:55:013
4037291995614,10cyclictest306-21in:imuxsock23:30:003
403729199561,15cyclictest4048849-21sort22:50:023
4037285995647,6cyclictest30-21ksoftirqd/223:40:002
403727699562,49cyclictest363-21snmpd22:33:240
403727699562,49cyclictest363-21snmpd00:30:490
403727699561,51cyclictest363-21snmpd20:47:080
403727699561,4cyclictest4051542-21sshd23:40:210
403729199555,7cyclictest4041630-21modprobe20:35:003
403729199555,13cyclictest272-21dbus-daemon22:35:013
403729199552,14cyclictest4051561-21sshd23:44:223
403729199552,14cyclictest4042605-21latency_hist20:50:023
403729199552,13cyclictest4040633-21latency_hist20:15:013
4037291995519,32cyclictest37-21ksoftirqd/300:40:013
4037291995514,4cyclictest4039267-21kworker/u8:120:25:003
4037291995510,6cyclictest4044799-21kworker/3:222:10:013
4037291995510,6cyclictest4044799-21kworker/3:222:10:013
403727699553,45cyclictest4045726-21latency_hist21:50:020
403727699553,45cyclictest4045726-21latency_hist21:50:020
403727699552,49cyclictest363-21snmpd23:50:060
403727699552,49cyclictest363-21snmpd00:17:190
403727699551,50cyclictest363-21snmpd00:06:270
403729199548,11cyclictest272-21dbus-daemon19:55:003
403729199546,12cyclictest4047092-21modprobe22:20:003
403729199542,16cyclictest4051045-21munin-plugin-st23:35:003
403729199542,14cyclictest4046039-21sshd21:59:223
4037291995418,7cyclictest37-21ksoftirqd/319:20:023
4037291995411,6cyclictest4041869-21sshd20:37:213
403729199541,13cyclictest4044279-21unin-run21:25:003
4037285995435,13cyclictest1-21systemd22:45:012
4037279995435,14cyclictest235-21systemd-journal00:30:001
403727699542,48cyclictest363-21snmpd21:56:330
403729199531,15cyclictest4051794-21sshd23:45:233
403729199531,13cyclictest4042679-21munin-run20:55:003
403729199531,12cyclictest4046864-21wc22:15:023
403728599535,6cyclictest131rcu_preempt00:00:012
4037285995328,17cyclictest272-21dbus-daemon21:55:012
4037285995328,17cyclictest272-21dbus-daemon21:55:002
403727699531,48cyclictest363-21snmpd23:19:030
403727699531,48cyclictest363-21snmpd23:19:020
403729199527,12cyclictest272-21dbus-daemon20:05:013
403729199525,9cyclictest0-21swapper/323:55:003
4037291995212,13cyclictest1-21systemd21:50:013
4037291995212,13cyclictest1-21systemd21:50:013
4037291995210,5cyclictest0-21swapper/300:19:593
4037285995244,5cyclictest30-21ksoftirqd/220:50:012
403729199515,7cyclictest4053911-21modprobe00:30:013
403729199511,14cyclictest4044529-21unin-run21:30:013
4037279995147,2cyclictest0-21swapper/121:25:011
4037279995126,18cyclictest272-21dbus-daemon23:50:021
403727699511,5cyclictest4047362-21sshd22:24:220
403729199502,13cyclictest4053628-21modprobe00:20:023
403729199502,13cyclictest4043995-21munin-run21:20:003
403729199502,13cyclictest4043995-21munin-run21:20:003
403729199502,13cyclictest4039318-21unin-run19:50:003
4037291995015,10cyclictest4043197-21cron21:05:003
403729199501,14cyclictest4042175-21munin-run20:45:013
403729199501,13cyclictest4038353-21latency_hist19:30:023
403729199501,12cyclictest4053044-21latency_hist00:10:023
4037285995027,17cyclictest272-21dbus-daemon21:40:002
403729199493,15cyclictest4051340-21munin-run23:40:013
403729199493,14cyclictest4039809-21modprobe20:00:003
403729199493,10cyclictest4050025-21wc23:15:013
403729199491,15cyclictest4047770-21grep22:30:023
4037291994834,11cyclictest1-21systemd21:15:003
403728599485,8cyclictest30-21ksoftirqd/219:10:012
4037285994838,7cyclictest30-21ksoftirqd/221:35:022
4037285994835,8cyclictest306-21in:imuxsock21:59:592
4037285994814,6cyclictest131rcu_preempt22:10:022
4037285994814,6cyclictest131rcu_preempt22:10:012
4037279994833,11cyclictest1-21systemd20:30:011
4037279994826,15cyclictest272-21dbus-daemon21:50:021
4037279994826,15cyclictest272-21dbus-daemon21:50:011
403729199472,10cyclictest1-21systemd19:40:013
4037285994729,13cyclictest1-21systemd23:10:012
4037279994729,6cyclictest306-21in:imuxsock22:40:021
4037279994728,15cyclictest235-21systemd-journal19:55:001
4037285994635,7cyclictest30-21ksoftirqd/222:30:022
4037279994633,9cyclictest23-21ksoftirqd/123:39:591
403729199451,15cyclictest4046302-21sshd22:01:213
4037285994532,9cyclictest4037745-21systemctl19:20:012
4037285994532,7cyclictest4045550-21munin-plugin-st21:50:012
4037285994532,7cyclictest4045550-21munin-plugin-st21:50:002
4037285994530,10cyclictest235-21systemd-journal19:40:012
4037279994533,7cyclictest4052351-21tr00:00:011
4037279994532,7cyclictest306-21in:imuxsock22:20:001
4037285994434,7cyclictest30-21ksoftirqd/200:25:022
4037285994432,7cyclictest306-21in:imuxsock19:15:012
4037285994430,10cyclictest30-21ksoftirqd/223:00:012
4037285994430,10cyclictest30-21ksoftirqd/221:10:012
403728599442,11cyclictest30-21ksoftirqd/200:10:032
4037279994438,2cyclictest0-21swapper/100:15:021
403729199432,14cyclictest4048474-21latency_hist22:45:013
403728599439,6cyclictest131rcu_preempt00:40:012
4037285994332,7cyclictest30-21ksoftirqd/220:35:012
4037285994331,9cyclictest30-21ksoftirqd/220:05:002
4037285994327,12cyclictest30-21ksoftirqd/219:30:012
4037279994332,6cyclictest4042931-21sendmail21:00:001
4037279994331,9cyclictest4038248-21sshd19:28:211
4037279994330,10cyclictest23-21ksoftirqd/100:25:021
4037279994325,13cyclictest235-21systemd-journal20:55:001
4037279994324,13cyclictest235-21systemd-journal19:45:001
403729199426,10cyclictest4038136-21date19:25:023
4037285994230,9cyclictest30-21ksoftirqd/221:20:002
4037285994230,9cyclictest30-21ksoftirqd/221:20:002
4037285994230,8cyclictest30-21ksoftirqd/219:25:012
4037285994227,12cyclictest272-21dbus-daemon19:50:002
4037285994226,12cyclictest272-21dbus-daemon23:25:002
4037285994212,11cyclictest30-21ksoftirqd/223:15:002
403728599421,10cyclictest30-21ksoftirqd/220:10:042
4037279994227,10cyclictest235-21systemd-journal00:34:321
4037279994217,18cyclictest272-21dbus-daemon00:05:001
4037285994132,6cyclictest30-21ksoftirqd/221:30:002
4037285994132,6cyclictest30-21ksoftirqd/219:55:022
4037285994131,7cyclictest30-21ksoftirqd/221:25:012
4037285994131,7cyclictest30-21ksoftirqd/219:45:032
4037285994130,8cyclictest30-21ksoftirqd/220:45:022
4037285994130,8cyclictest291rcuc/222:25:012
4037279994131,8cyclictest23-21ksoftirqd/121:10:001
4037285994030,7cyclictest30-21ksoftirqd/220:30:012
4037285994029,6cyclictest306-21in:imuxsock00:19:332
4037285994026,10cyclictest30-21ksoftirqd/220:00:022
4037285994012,6cyclictest131rcu_preempt22:39:592
403727999403,32cyclictest4047870-21cron22:35:011
4037279994027,9cyclictest235-21systemd-journal22:30:001
4037279994027,10cyclictest4048146-21systemctl22:39:591
4037285993930,6cyclictest30-21ksoftirqd/223:55:002
4037285993927,8cyclictest30-21ksoftirqd/222:48:222
4037285993926,9cyclictest30-21ksoftirqd/200:05:012
4037285993925,11cyclictest30-21ksoftirqd/221:15:002
4037285993924,11cyclictest4047179-21sed22:20:022
4037285993915,5cyclictest131rcu_preempt23:45:012
4037279993928,7cyclictest4041603-21sshd20:30:211
4037279993925,9cyclictest306-21in:imuxsock00:10:021
4037279993923,11cyclictest4047376-21cron22:25:001
403728599389,6cyclictest131rcu_preempt20:25:012
403728599386,8cyclictest30-21ksoftirqd/223:17:012
403728599386,8cyclictest30-21ksoftirqd/223:17:002
403728599384,8cyclictest30-21ksoftirqd/200:35:022
4037285993828,8cyclictest30-21ksoftirqd/222:00:222
4037285993828,6cyclictest4046836-21sh22:14:592
4037285993826,9cyclictest30-21ksoftirqd/223:35:022
4037285993825,10cyclictest235-21systemd-journal20:20:012
403727999385,9cyclictest23-21ksoftirqd/119:59:211
4037279993824,10cyclictest0-21swapper/121:20:011
4037279993824,10cyclictest0-21swapper/121:20:001
4037279993822,12cyclictest4051610-21cut23:45:011
4037279993815,17cyclictest235-21systemd-journal20:15:011
4037285993727,6cyclictest30-21ksoftirqd/223:05:012
4037285993725,8cyclictest30-21ksoftirqd/222:32:322
4037285993724,9cyclictest306-21in:imuxsock23:30:022
4037279993724,9cyclictest272-21dbus-daemon23:15:011
4037279993723,10cyclictest272-21dbus-daemon20:50:011
4037279993721,11cyclictest272-21dbus-daemon23:25:001
4037279993721,11cyclictest1-21systemd19:50:011
4037279993718,12cyclictest272-21dbus-daemon19:35:001
403728599367,6cyclictest131rcu_preempt19:35:022
4037285993627,6cyclictest30-21ksoftirqd/222:53:232
4037285993626,8cyclictest30-21ksoftirqd/200:15:012
4037285993626,7cyclictest30-21ksoftirqd/221:05:022
4037285993625,8cyclictest235-21systemd-journal20:12:112
4037279993625,9cyclictest23-21ksoftirqd/122:12:291
4037279993625,8cyclictest4043188-21sshd21:03:211
4037279993623,10cyclictest23-21ksoftirqd/122:10:011
4037279993623,10cyclictest23-21ksoftirqd/122:10:001
4037279993621,11cyclictest272-21dbus-daemon19:25:021
4037279993620,12cyclictest1-21systemd23:30:001
4037279993618,14cyclictest23-21ksoftirqd/120:20:011
4037285993525,8cyclictest30-21ksoftirqd/220:55:012
4037285993523,8cyclictest30-21ksoftirqd/221:45:022
403727999356,22cyclictest4044694-21grep21:30:021
403727999354,3cyclictest131rcu_preempt00:15:011
4037279993525,8cyclictest23-21ksoftirqd/120:20:211
4037279993524,9cyclictest23-21ksoftirqd/122:50:021
4037279993524,8cyclictest4044744-21sshd21:33:211
4037279993523,9cyclictest23-21ksoftirqd/120:05:011
4037279993523,9cyclictest23-21ksoftirqd/100:40:011
4037279993523,8cyclictest23-21ksoftirqd/122:05:001
4037279993521,10cyclictest272-21dbus-daemon19:20:001
4037279993519,12cyclictest272-21dbus-daemon23:20:011
4037279993519,12cyclictest272-21dbus-daemon23:20:001
403727999343,3cyclictest131rcu_preempt22:00:001
4037279993426,5cyclictest4042171-21latency_hist20:45:021
4037279993420,9cyclictest235-21systemd-journal19:10:211
4037279993417,12cyclictest272-21dbus-daemon23:05:001
403728599337,5cyclictest131rcu_preempt00:30:012
403727999335,5cyclictest131rcu_preempt23:00:011
4037279993327,4cyclictest4045023-21latency_hist21:40:021
4037279993324,6cyclictest23-21ksoftirqd/120:10:021
4037279993323,8cyclictest23-21ksoftirqd/123:35:031
403727999325,15cyclictest4038763-21modprobe19:40:001
403727999323,3cyclictest131rcu_preempt23:10:001
4037279993223,7cyclictest23-21ksoftirqd/120:40:011
4037279993223,5cyclictest4045398-21latency_hist21:45:011
4037279993222,7cyclictest23-21ksoftirqd/121:55:011
4037279993222,7cyclictest23-21ksoftirqd/121:55:001
4037279993221,9cyclictest23-21ksoftirqd/121:15:011
4037279993118,9cyclictest23-21ksoftirqd/122:55:001
403728599305,20cyclictest235-21systemd-journal21:00:002
4037279992920,7cyclictest23-21ksoftirqd/123:54:311
403729199282,4cyclictest131rcu_preempt19:10:023
4037279992719,5cyclictest4037364-21cat19:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional