You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 10:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Nov 06, 2025 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
177940899110102,6cyclictest23-21ksoftirqd/122:45:021
1779408999559,23cyclictest221rcuc/121:45:021
1779403999385,6cyclictest12-21ksoftirqd/020:55:020
1779418997511,38cyclictest235-21systemd-journal23:25:023
177940899732,66cyclictest1779415-21latency_hist19:10:011
1779403997210,8cyclictest1793083-21sshd23:31:430
1779403997210,8cyclictest1793083-21sshd23:31:420
1779418996932,32cyclictest1-21systemd19:30:013
1779418996927,38cyclictest1783030-21wc20:20:023
177940399698,37cyclictest19050irq/34-mmc120:20:020
177941899681,13cyclictest1781913-21sshd19:57:243
1779403996826,39cyclictest363-21snmpd19:56:580
177940399681,4cyclictest0-21swapper/022:30:000
177941899672,15cyclictest1785016-21sshd20:55:243
177941899671,15cyclictest1791548-21sshd23:04:243
177941899671,15cyclictest1791548-21sshd23:04:233
1779403996721,41cyclictest272-21dbus-daemon23:20:020
1779403996721,41cyclictest272-21dbus-daemon23:20:010
177940399671,62cyclictest3237915-1kworker/0:2H19:45:010
1779418996537,24cyclictest272-21dbus-daemon21:15:003
177941899652,16cyclictest1786636-21munin-run21:30:003
177941899652,15cyclictest1788970-21munin-run22:15:013
177941899651,14cyclictest1782440-21sshd20:06:253
1779403996561,2cyclictest0-21swapper/019:50:030
1779403996561,2cyclictest0-21swapper/019:50:020
177940399652,16cyclictest1789999-21munin-run22:35:010
177940399651,60cyclictest1786888-21/usr/sbin/munin21:35:010
177941899642,12cyclictest1787932-21munin-run21:55:013
177941899641,16cyclictest1789967-21sshd22:31:253
177941899641,15cyclictest1781150-21sshd19:40:593
177940399648,50cyclictest0-21swapper/021:45:000
177941899632,56cyclictest1784613-21ls20:50:023
177941899632,16cyclictest1786397-21munin-run21:25:013
177941899632,14cyclictest1781434-21munin-run19:50:013
177941899632,14cyclictest1781434-21munin-run19:50:003
177941899632,13cyclictest1795156-21sshd00:11:243
177941899632,12cyclictest1783276-21munin-run20:25:013
177941899632,12cyclictest1783276-21munin-run20:25:013
1779408996355,6cyclictest23-21ksoftirqd/120:20:031
1779408996355,6cyclictest23-21ksoftirqd/120:20:021
1779403996359,2cyclictest0-21swapper/023:10:010
177940399632,16cyclictest1789258-21munin-run22:20:020
177940399632,16cyclictest1789258-21munin-run22:20:010
1779403996312,6cyclictest1779819-21sshd19:15:240
1779403996312,6cyclictest1779819-21sshd19:15:230
177941899624,14cyclictest1793746-21sort23:45:023
1779418996216,40cyclictest272-21dbus-daemon19:20:013
1779418996216,40cyclictest272-21dbus-daemon19:20:003
1779403996217,5cyclictest3237915-1kworker/0:2H22:22:060
1779403996212,7cyclictest1786092-21cron21:20:000
177941899612,14cyclictest1796477-21sshd00:38:243
177941899612,14cyclictest1795741-21munin-run00:25:013
177941899612,13cyclictest1788134-21sshd21:55:243
177941899612,13cyclictest1782735-21munin-run20:15:013
177941899612,13cyclictest1782735-21munin-run20:15:003
177940399611,15cyclictest1782196-21sshd20:03:240
177940399611,15cyclictest1782196-21sshd20:03:230
177940399611,14cyclictest1790494-21sshd22:40:240
177941899603,14cyclictest1785330-21munin-run21:05:003
177941899602,17cyclictest1796333-21cat00:35:023
177941899602,17cyclictest1794454-21cat00:00:023
177941899602,17cyclictest1781685-21munin-run19:54:593
177941899602,13cyclictest1794698-21munin-run00:05:013
1779418996015,41cyclictest1-21systemd00:30:013
177941899601,14cyclictest1787120-21sshd21:39:243
177941899601,14cyclictest1785551-21sshd21:06:193
1779418996010,44cyclictest0-21swapper/320:30:003
177940399603,9cyclictest1785569-21cron21:10:000
1779403996026,31cyclictest0-21swapper/000:16:250
177940399602,17cyclictest1795176-21cron00:14:590
177940399602,13cyclictest1791046-21munin-run22:55:000
177940399602,13cyclictest1782722-21munin-run20:15:010
177940399602,13cyclictest1782722-21munin-run20:15:000
177940399602,12cyclictest1794110-21sshd23:50:230
177940399601,14cyclictest1784801-21munin-run20:55:010
1779418995925,28cyclictest1795438-21cron00:20:013
177941899592,15cyclictest1790784-21munin-run22:50:003
177941899592,14cyclictest1788472-21grep22:05:013
177941899592,14cyclictest1780212-21sed19:25:013
177941899592,13cyclictest363-21snmpd22:52:133
1779403995919,34cyclictest1796230-21latency_hist00:35:030
1779403995913,8cyclictest1793920-21sort23:50:020
177941899582,14cyclictest1782230-21/usr/sbin/munin20:05:023
177941899582,14cyclictest1782230-21/usr/sbin/munin20:05:013
177941899582,13cyclictest363-21snmpd19:13:303
177941899582,13cyclictest1794939-21munin-run00:10:013
177941899582,13cyclictest1794939-21munin-run00:10:003
177941899581,53cyclictest48-21kcompactd020:55:013
177941899581,16cyclictest1780889-21cron19:40:003
1779408995811,12cyclictest23-21ksoftirqd/123:40:001
177940399584,9cyclictest1792874-21systemd-run23:30:010
1779403995836,15cyclictest1-21systemd23:25:010
177940399583,15cyclictest1794495-21grep00:00:020
177940399582,14cyclictest1780153-21systemd-run19:25:000
177940399581,15cyclictest1790216-21sshd22:36:240
177940399581,14cyclictest363-21snmpd23:03:320
177940399581,14cyclictest363-21snmpd23:03:310
177941899572,16cyclictest1783778-21munin-run20:35:013
177941899571,12cyclictest363-21snmpd21:16:373
177941899571,12cyclictest1790515-21modprobe22:44:253
177940399576,17cyclictest272-21dbus-daemon19:35:010
177940399573,14cyclictest1780891-21cron19:40:000
1779403995726,28cyclictest0-21swapper/000:06:250
1779403995726,28cyclictest0-21swapper/000:06:240
177940399572,14cyclictest1788766-21latency_hist22:10:010
1779403995719,33cyclictest363-21snmpd23:42:430
177941899562,15cyclictest1793112-21munin-run23:35:013
177941899562,15cyclictest1793112-21munin-run23:35:003
177941899562,15cyclictest1792081-21munin-run23:15:003
1779418995614,36cyclictest0-21swapper/323:40:013
177941899561,12cyclictest363-21snmpd19:31:513
177940399562,14cyclictest1796511-21munin-run00:40:000
177940399562,14cyclictest1784263-21sshd20:41:110
177940399562,13cyclictest1783361-21latency_hist20:25:020
177940399562,13cyclictest1783361-21latency_hist20:25:020
177940399562,12cyclictest1788174-21modprobe22:00:000
177940399562,12cyclictest1786384-21modprobe21:25:000
177941899552,15cyclictest1791816-21munin-run23:10:003
177941899551,15cyclictest1786970-21cut21:35:023
1779413995510,6cyclictest131rcu_preempt00:00:002
1779408995511,20cyclictest1783746-21kworker/u8:121:05:021
177940399553,15cyclictest1781698-21munin-run19:55:000
177940399552,14cyclictest1791829-21munin-run23:10:010
177940399552,13cyclictest363-21snmpd20:45:140
177940399552,11cyclictest1793398-21true23:40:000
1779403995514,4cyclictest0-21swapper/021:02:110
177940399551,15cyclictest1779692-21sed19:15:020
177940399551,14cyclictest1795996-21unin-run00:30:010
177940399551,14cyclictest1-21systemd21:40:000
177941899544,8cyclictest1789768-21cat22:30:013
177941899543,12cyclictest1794146-21munin-run23:54:593
1779418995422,28cyclictest1787710-21latency_hist21:50:013
177941899541,12cyclictest363-21snmpd20:42:123
177940399543,21cyclictest1788959-21cut22:15:000
177940399543,12cyclictest1785981-21cat21:15:020
1779403995420,30cyclictest16850irq/134-800000022:00:240
1779403995417,33cyclictest1780477-21latency_hist19:30:020
177941899533,7cyclictest235-21systemd-journal23:30:013
177941899532,14cyclictest1790260-21munin-run22:39:593
177941899532,10cyclictest1793895-21munin-run23:50:013
177940899537,10cyclictest23-21ksoftirqd/122:10:021
1779408995313,6cyclictest23-21ksoftirqd/120:35:011
1779408995312,6cyclictest131rcu_preempt22:05:021
177940399532,14cyclictest1795904-21latency_hist00:25:020
1779403995318,31cyclictest1783492-21sshd20:28:250
177941899523,13cyclictest1792340-21cut23:20:013
177941899523,13cyclictest1792340-21cut23:20:003
177941899523,12cyclictest1789212-21modprobe22:20:013
177941899523,12cyclictest1789212-21modprobe22:20:003
177941399527,6cyclictest131rcu_preempt22:10:002
177940399523,10cyclictest1794682-21latency_hist00:05:010
177940399521,4cyclictest1787665-21latency_hist21:50:020
177941899514,8cyclictest0-21swapper/322:10:023
177941399518,8cyclictest30-21ksoftirqd/220:00:002
1779413995116,7cyclictest30-21ksoftirqd/223:40:002
177940899513,12cyclictest23-21ksoftirqd/100:00:011
177940399519,7cyclictest0-21swapper/021:28:240
177940399512,14cyclictest1787939-21cat21:55:010
177940399511,2cyclictest0-21swapper/022:58:250
1779418995010,7cyclictest1791306-21sendmail23:00:003
1779413995041,6cyclictest30-21ksoftirqd/221:20:002
1779413995032,14cyclictest1-21systemd23:50:022
177940899508,16cyclictest131rcu_preempt22:20:021
177940899508,16cyclictest131rcu_preempt22:20:021
177940899505,10cyclictest23-21ksoftirqd/122:50:001
177940399502,9cyclictest1782458-21cron20:09:590
177941899492,13cyclictest1789495-21munin-run22:25:003
177940899497,6cyclictest131rcu_preempt22:35:021
177940899493,8cyclictest23-21ksoftirqd/121:30:031
1779408994935,10cyclictest306-21in:imuxsock19:30:001
1779408994930,15cyclictest1-21systemd00:05:001
177940399492,20cyclictest1784025-21systemctl20:40:000
177940399491,22cyclictest12-21ksoftirqd/022:48:120
177941899489,6cyclictest1779524-21latency_hist19:10:023
177941899483,16cyclictest1784028-21munin-run20:40:003
1779413994833,11cyclictest30-21ksoftirqd/220:20:002
1779413994816,20cyclictest0-21swapper/222:25:012
177940899489,4cyclictest131rcu_preempt21:00:241
177940899486,6cyclictest131rcu_preempt21:20:021
1779408994820,17cyclictest1785092-21kworker/1:221:40:031
1779408994813,6cyclictest131rcu_preempt00:20:001
177941899472,12cyclictest1787417-21munin-run21:45:013
177940899477,6cyclictest131rcu_preempt20:30:011
1779408994738,6cyclictest23-21ksoftirqd/122:35:121
1779408994713,5cyclictest131rcu_preempt22:30:011
177941399464,6cyclictest131rcu_preempt19:30:002
177941399462,5cyclictest131rcu_preempt21:45:012
177940899467,5cyclictest131rcu_preempt19:50:031
177940899467,5cyclictest131rcu_preempt19:50:021
177940899466,6cyclictest23-21ksoftirqd/123:12:371
1779408994637,6cyclictest23-21ksoftirqd/100:10:011
1779408994637,6cyclictest23-21ksoftirqd/100:10:001
177940899462,11cyclictest23-21ksoftirqd/123:20:001
177940899462,11cyclictest23-21ksoftirqd/123:20:001
177940899461,5cyclictest0-21swapper/123:26:231
1779408994613,7cyclictest131rcu_preempt21:45:021
1779403994616,8cyclictest12-21ksoftirqd/020:35:010
177941399457,7cyclictest30-21ksoftirqd/221:15:012
177941399456,6cyclictest131rcu_preempt21:00:022
1779413994530,11cyclictest1-21systemd21:35:002
1779413994528,13cyclictest30-21ksoftirqd/200:20:012
177940899457,6cyclictest131rcu_preempt19:25:021
1779408994534,8cyclictest23-21ksoftirqd/121:15:021
1779408994534,8cyclictest23-21ksoftirqd/120:55:001
177940899453,10cyclictest23-21ksoftirqd/123:35:011
177940899453,10cyclictest23-21ksoftirqd/123:35:001
1779408994530,11cyclictest272-21dbus-daemon20:00:011
177941399449,6cyclictest131rcu_preempt21:10:022
1779413994435,7cyclictest30-21ksoftirqd/220:25:022
1779413994435,7cyclictest30-21ksoftirqd/220:25:012
1779413994435,6cyclictest30-21ksoftirqd/200:01:252
1779413994435,5cyclictest1786383-21munin-plugin-st21:25:012
1779413994431,7cyclictest30-21ksoftirqd/220:35:022
1779413994429,11cyclictest235-21systemd-journal19:14:242
177941399442,6cyclictest131rcu_preempt19:10:012
1779413994416,7cyclictest131rcu_preempt23:15:012
177940899445,10cyclictest23-21ksoftirqd/100:35:001
177940899444,6cyclictest131rcu_preempt23:55:001
1779408994431,10cyclictest23-21ksoftirqd/100:40:011
1779408994428,11cyclictest235-21systemd-journal00:25:021
177940899442,11cyclictest23-21ksoftirqd/121:35:011
177940899441,9cyclictest0-21swapper/119:40:231
1779408994413,6cyclictest131rcu_preempt21:00:001
1779413994333,8cyclictest30-21ksoftirqd/220:10:002
1779413994330,8cyclictest235-21systemd-journal23:05:012
1779413994330,8cyclictest235-21systemd-journal23:05:002
1779413994328,11cyclictest272-21dbus-daemon22:40:002
177941399431,31cyclictest3631snmpd22:12:252
177940899437,6cyclictest131rcu_preempt19:20:021
177940899437,6cyclictest131rcu_preempt19:20:021
1779408994331,8cyclictest23-21ksoftirqd/123:45:021
1779408994331,8cyclictest1788177-21run-parts21:59:591
177941399428,5cyclictest131rcu_preempt19:50:002
177941399428,5cyclictest131rcu_preempt19:50:002
1779413994229,10cyclictest1779882-21tr19:20:012
1779413994229,10cyclictest1779882-21tr19:20:002
177941399421,7cyclictest1789761-21unin-run22:30:002
1779413994217,6cyclictest131rcu_preempt21:59:592
177940899429,5cyclictest131rcu_preempt20:40:031
1779408994230,7cyclictest23-21ksoftirqd/120:15:011
1779408994230,7cyclictest23-21ksoftirqd/120:15:001
177941399418,6cyclictest131rcu_preempt23:25:002
177941399417,6cyclictest131rcu_preempt22:20:022
177941399417,6cyclictest131rcu_preempt22:20:012
177941399416,6cyclictest131rcu_preempt23:00:022
177941399416,10cyclictest30-21ksoftirqd/223:35:002
177941399416,10cyclictest30-21ksoftirqd/223:35:002
1779413994131,7cyclictest30-21ksoftirqd/219:36:242
1779413994124,12cyclictest272-21dbus-daemon21:05:012
1779413994113,23cyclictest291rcuc/200:15:002
1779413994111,6cyclictest131rcu_preempt20:40:002
1779408994111,6cyclictest131rcu_preempt20:40:021
177941399409,5cyclictest131rcu_preempt20:45:012
1779413994028,9cyclictest1792511-21cut23:20:032
1779413994028,9cyclictest1792511-21cut23:20:022
1779413994028,8cyclictest1795725-21modprobe00:25:012
1779413994027,9cyclictest1796016-21cat00:30:002
1779413994021,14cyclictest272-21dbus-daemon22:45:002
177940899406,5cyclictest131rcu_preempt22:25:001
177940899401,9cyclictest0-21swapper/100:15:001
177940899401,4cyclictest131rcu_preempt19:15:031
177941399394,6cyclictest131rcu_preempt19:25:012
177941399393,6cyclictest131rcu_preempt21:40:012
1779413993928,7cyclictest30-21ksoftirqd/222:35:032
1779413993924,11cyclictest30-21ksoftirqd/220:15:012
1779413993924,11cyclictest30-21ksoftirqd/220:15:002
1779413993923,12cyclictest1781233-21cut19:45:022
177941399391,5cyclictest131rcu_preempt21:50:032
1779413993914,17cyclictest30-21ksoftirqd/221:52:172
1779413993912,17cyclictest1792875-21munin-run23:30:022
177940899394,5cyclictest131rcu_preempt23:10:011
1779408993924,11cyclictest23-21ksoftirqd/120:20:001
177941399387,7cyclictest30-21ksoftirqd/222:50:022
177941399385,6cyclictest131rcu_preempt22:05:032
1779413993828,8cyclictest30-21ksoftirqd/200:35:022
1779413993824,10cyclictest272-21dbus-daemon20:05:012
1779413993824,10cyclictest272-21dbus-daemon20:05:002
177941399381,5cyclictest131rcu_preempt00:40:022
177940899385,7cyclictest23-21ksoftirqd/123:05:011
177940899385,7cyclictest23-21ksoftirqd/123:05:011
1779408993829,6cyclictest221rcuc/119:40:001
1779408993827,8cyclictest235-21systemd-journal22:10:021
1779408993824,11cyclictest23-21ksoftirqd/123:50:011
177940399381,14cyclictest1779559-21sed19:10:030
1779413993725,9cyclictest1-21systemd20:55:012
1779413993723,11cyclictest1791074-21cat22:55:002
1779413993712,6cyclictest131rcu_preempt19:35:012
177940899379,5cyclictest131rcu_preempt20:02:431
177940899379,5cyclictest131rcu_preempt20:02:431
1779408993725,9cyclictest23-21ksoftirqd/120:50:021
1779408993725,10cyclictest23-21ksoftirqd/121:55:021
1779413993617,14cyclictest235-21systemd-journal23:45:002
177940899369,5cyclictest131rcu_preempt19:50:401
177940899365,7cyclictest23-21ksoftirqd/100:26:311
177940899361,7cyclictest0-21swapper/123:24:091
1779413993525,7cyclictest30-21ksoftirqd/219:55:012
1779413993523,8cyclictest30-21ksoftirqd/221:30:012
1779413993520,10cyclictest306-21in:imuxsock23:55:002
1779408993525,7cyclictest23-21ksoftirqd/122:57:331
1779413993413,12cyclictest150-21pr/ttyS220:27:552
1779408993427,5cyclictest23-21ksoftirqd/121:22:151
177941399335,5cyclictest131rcu_preempt23:10:022
1779413993323,7cyclictest1784696-21sed20:50:022
1779413993312,12cyclictest30-21ksoftirqd/200:07:172
1779413993312,12cyclictest30-21ksoftirqd/200:07:172
1779408993318,11cyclictest235-21systemd-journal19:35:001
1779408993221,8cyclictest23-21ksoftirqd/122:55:011
177940899321,6cyclictest131rcu_preempt20:08:061
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional