You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Fri May 02, 2025 00:45:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
360288599963,3cyclictest131rcu_preempt20:10:023
360287699693,60cyclictest3602878-21ls19:10:021
360288599676,56cyclictest3608492-21/usr/sbin/munin21:00:023
360288599662,60cyclictest3618719-21sshd00:19:323
360288599661,60cyclictest3612038-21sshd22:07:323
360288599661,59cyclictest3619733-21sshd00:37:323
360288599651,60cyclictest3604219-21cat19:35:023
360288599642,58cyclictest3604624-21munin-run19:45:013
360288599642,58cyclictest3604624-21munin-run19:45:013
360288599642,57cyclictest3613347-21munin-run22:35:003
360288599641,59cyclictest3604315-21sshd19:36:323
360288599632,57cyclictest3606681-21munin-run20:25:003
360288599632,56cyclictest3610560-21munin-run21:40:013
360288599631,58cyclictest3608956-21sshd21:08:323
360288599631,58cyclictest3608178-21sshd20:53:323
360288599631,58cyclictest3607147-21sshd20:31:323
360288599631,57cyclictest3603304-21sshd19:19:323
360288599631,56cyclictest48-21kcompactd020:28:183
360288599622,54cyclictest3617949-21sshd00:00:323
360288599621,57cyclictest3614863-21sshd23:01:323
360288599621,57cyclictest3610240-21sshd21:31:323
360288599621,56cyclictest3612817-21sshd22:20:323
360288599621,56cyclictest3612533-21sshd22:16:323
360288599621,55cyclictest48-21kcompactd022:01:313
360288599621,55cyclictest3614098-21sshd22:48:323
360288599616,51cyclictest3606114-21sshd20:11:323
360288599616,51cyclictest3606114-21sshd20:11:323
360288599611,55cyclictest48-21kcompactd022:13:433
360288599611,55cyclictest3610796-21munin-run21:45:013
360288599611,55cyclictest3607404-21sshd20:38:323
360288599611,54cyclictest48-21kcompactd023:24:103
360288599611,54cyclictest48-21kcompactd021:55:583
3602876996140,15cyclictest235-21systemd-journal23:10:001
360288599602,54cyclictest3604871-21munin-run19:49:593
360288599602,53cyclictest3618242-21munin-run00:10:013
360288599602,53cyclictest3618242-21munin-run00:10:013
360288599602,53cyclictest3605629-21sshd20:04:313
360288599601,55cyclictest3617676-21sshd23:59:323
360288599601,55cyclictest3616639-21sshd23:35:323
360288599601,55cyclictest3616388-21sshd23:30:323
360288599601,55cyclictest3613079-21sshd22:29:323
360288599601,55cyclictest3611026-21sshd21:49:333
360288599601,55cyclictest3611026-21sshd21:49:323
360288599601,54cyclictest48-21kcompactd020:46:373
360288599601,54cyclictest48-21kcompactd020:18:193
360288599601,54cyclictest48-21kcompactd019:29:263
360288599601,54cyclictest3614627-21munin-run23:00:013
360288599601,54cyclictest3605094-21sshd19:52:323
360288599592,52cyclictest3609216-21sshd21:13:323
360288599591,54cyclictest3616150-21sshd23:27:323
360288599591,54cyclictest3614351-21sshd22:54:323
360288599591,54cyclictest3609480-21sshd21:19:323
360288599591,53cyclictest363-21snmpd23:05:533
360288599591,53cyclictest363-21snmpd00:10:523
360288599591,52cyclictest48-21kcompactd021:50:583
3602871995912,9cyclictest12-21ksoftirqd/021:35:010
360288599582,51cyclictest363-21snmpd21:23:433
360288599582,51cyclictest3603612-21munin-run19:25:013
360288599581,53cyclictest3619232-21sshd00:26:323
360288599581,53cyclictest3613603-21/usr/sbin/munin22:40:003
360288599581,52cyclictest363-21snmpd00:30:323
3602880995814,10cyclictest30-21ksoftirqd/221:34:592
360288599572,50cyclictest3610009-21cron21:30:003
360288599571,51cyclictest48-21kcompactd022:40:203
360288599562,49cyclictest363-21snmpd23:46:333
360288599551,49cyclictest3615612-21sshd23:16:323
3602885995510,41cyclictest0-21swapper/321:04:543
3602880995534,15cyclictest235-21systemd-journal20:30:002
360288599542,47cyclictest3618981-21sshd00:20:323
360288599541,49cyclictest363-21snmpd23:53:043
360288599541,47cyclictest48-21kcompactd019:57:143
3602871995438,13cyclictest1-21systemd00:20:010
360288599532,47cyclictest363-21snmpd23:40:303
360288599532,47cyclictest3603035-21sshd19:11:323
360288099535,5cyclictest30-21ksoftirqd/221:40:012
360288599522,45cyclictest0-21swapper/320:40:093
360288599521,46cyclictest363-21snmpd23:12:353
3602880995235,12cyclictest1-21systemd22:30:012
3602871995241,7cyclictest16950irq/224-800000020:00:010
360288099514,8cyclictest30-21ksoftirqd/200:20:012
3602876995135,12cyclictest23-21ksoftirqd/122:20:011
360287199515,9cyclictest12-21ksoftirqd/020:40:000
3602871995140,7cyclictest16950irq/224-800000023:40:020
360287199505,9cyclictest12-21ksoftirqd/019:20:000
3602871995034,11cyclictest1-21systemd23:50:010
360287699497,28cyclictest3615642-21modprobe23:20:001
360288099486,8cyclictest30-21ksoftirqd/220:55:012
360288099483,8cyclictest30-21ksoftirqd/223:20:012
3602876994833,10cyclictest3618745-21grep00:20:001
3602876994824,18cyclictest3608991-21sendmail-mta21:10:011
360287199489,9cyclictest12-21ksoftirqd/020:15:030
3602871994833,10cyclictest235-21systemd-journal21:25:000
360287199482,11cyclictest12-21ksoftirqd/022:35:020
360288099473,8cyclictest30-21ksoftirqd/220:00:012
3602880994729,15cyclictest272-21dbus-daemon00:00:002
3602880994717,10cyclictest30-21ksoftirqd/219:20:012
3602876994727,14cyclictest235-21systemd-journal00:10:011
3602876994727,14cyclictest235-21systemd-journal00:10:011
360287199473,7cyclictest12-21ksoftirqd/019:45:010
360287199473,7cyclictest12-21ksoftirqd/019:45:010
3602871994727,15cyclictest272-21dbus-daemon20:45:010
360287199471,13cyclictest12-21ksoftirqd/000:10:020
360287199471,13cyclictest12-21ksoftirqd/000:10:020
360288099463,11cyclictest30-21ksoftirqd/220:05:002
3602880994629,13cyclictest1-21systemd23:05:002
3602876994626,14cyclictest3613579-21cron22:40:001
360287199465,9cyclictest12-21ksoftirqd/023:55:000
360287199464,11cyclictest12-21ksoftirqd/022:45:010
360287199461,41cyclictest0-21swapper/022:25:020
360288099457,9cyclictest30-21ksoftirqd/223:35:022
360288099457,4cyclictest131rcu_preempt19:15:012
3602876994524,14cyclictest3606401-21modprobe20:20:001
3602871994538,4cyclictest16950irq/224-800000022:15:010
360287199453,12cyclictest12-21ksoftirqd/022:10:000
360287199452,39cyclictest3604883-21systemd-run19:50:000
360287199451,8cyclictest12-21ksoftirqd/020:04:590
3602876994434,7cyclictest23-21ksoftirqd/120:34:591
3602876994427,13cyclictest235-21systemd-journal21:17:321
3602876994425,4cyclictest0-21swapper/122:05:031
360287199447,6cyclictest12-21ksoftirqd/020:10:020
360287199446,5cyclictest131rcu_preempt19:35:020
360287199445,8cyclictest12-21ksoftirqd/022:50:010
360287199443,8cyclictest3617743-21munin-run00:00:020
3602871994436,5cyclictest12-21ksoftirqd/022:30:000
3602871994432,8cyclictest306-21in:imuxsock19:25:010
360287199441,11cyclictest12-21ksoftirqd/020:15:030
3602885994317,22cyclictest3602809-21latency_hist19:10:023
360288099435,7cyclictest30-21ksoftirqd/219:30:012
3602880994328,11cyclictest235-21systemd-journal22:54:322
360288099431,4cyclictest0-21swapper/221:05:002
3602876994326,12cyclictest3617724-21sendmail-msp00:00:001
3602876994323,16cyclictest235-21systemd-journal22:00:001
360287199435,8cyclictest12-21ksoftirqd/021:55:000
360287199433,9cyclictest12-21ksoftirqd/021:55:020
3602871994328,12cyclictest1-21systemd20:50:010
3602871994325,14cyclictest3615360-21sshd23:10:320
360287199431,5cyclictest12-21ksoftirqd/020:55:020
360288099426,5cyclictest131rcu_preempt23:29:322
360288099423,8cyclictest30-21ksoftirqd/222:50:012
360288099423,14cyclictest30-21ksoftirqd/222:35:012
3602876994229,9cyclictest306-21in:imuxsock22:55:001
3602876994229,10cyclictest23-21ksoftirqd/121:30:011
360287199426,7cyclictest12-21ksoftirqd/022:05:020
360287199426,30cyclictest3602872-21latency_hist19:10:010
360287199425,9cyclictest12-21ksoftirqd/023:30:010
360287199425,5cyclictest12-21ksoftirqd/021:50:000
360287199425,5cyclictest12-21ksoftirqd/021:50:000
360287199423,6cyclictest12-21ksoftirqd/019:55:010
3602871994228,11cyclictest12-21ksoftirqd/023:25:000
3602871994217,12cyclictest272-21dbus-daemon23:05:000
360287199421,36cyclictest3609500-21sendmail21:20:000
360288099414,9cyclictest30-21ksoftirqd/221:50:002
360288099414,9cyclictest30-21ksoftirqd/221:50:002
3602880994127,10cyclictest3617993-21pluginstate00:05:012
360288099411,7cyclictest30-21ksoftirqd/219:45:012
360288099411,7cyclictest30-21ksoftirqd/219:45:012
360288099411,10cyclictest30-21ksoftirqd/220:25:022
3602876994126,10cyclictest23-21ksoftirqd/121:25:001
3602876994122,14cyclictest1-21systemd22:45:011
360287199416,4cyclictest131rcu_preempt20:35:000
360287199414,7cyclictest12-21ksoftirqd/021:40:020
360287199413,6cyclictest12-21ksoftirqd/021:15:010
360287199412,7cyclictest3618640-21cut00:15:020
3602871994119,18cyclictest16850irq/134-800000019:35:320
3602871994117,19cyclictest16850irq/134-800000023:15:310
3602871994112,25cyclictest12-21ksoftirqd/020:24:400
360288099406,8cyclictest30-21ksoftirqd/200:15:002
360288099406,7cyclictest30-21ksoftirqd/222:40:002
360288099406,5cyclictest30-21ksoftirqd/200:40:012
3602880994025,9cyclictest3604343-21cron19:40:002
360287699407,20cyclictest221rcuc/100:30:021
3602876994030,6cyclictest3604858-21cron19:50:001
3602876994029,6cyclictest3607414-21cron20:40:001
3602876994027,7cyclictest271-21cron00:40:011
3602876994023,11cyclictest3610526-21systemctl21:40:001
3602871994031,6cyclictest12-21ksoftirqd/020:30:020
3602871994027,10cyclictest12-21ksoftirqd/021:10:020
360287199402,6cyclictest3615156-21munin-run23:10:010
360287199401,8cyclictest0-21swapper/019:28:330
360288099394,8cyclictest30-21ksoftirqd/221:25:002
3602880993928,9cyclictest30-21ksoftirqd/220:45:012
3602880993922,12cyclictest30-21ksoftirqd/223:44:322
360288099391,6cyclictest3611830-21munin-run22:05:022
360287699395,24cyclictest3616176-21cut23:30:001
3602876993927,7cyclictest306-21in:imuxsock20:55:011
3602876993924,11cyclictest23-21ksoftirqd/122:25:001
3602876993922,13cyclictest235-21systemd-journal00:35:011
360287199396,7cyclictest12-21ksoftirqd/023:00:010
360287199395,7cyclictest12-21ksoftirqd/023:45:010
360287199392,8cyclictest12-21ksoftirqd/022:40:000
3602871993925,10cyclictest12-21ksoftirqd/023:35:000
3602871993925,10cyclictest1-21systemd22:54:590
360287199391,34cyclictest0-21swapper/000:35:020
360288099386,7cyclictest30-21ksoftirqd/221:00:012
360288099383,9cyclictest30-21ksoftirqd/222:10:002
3602880993828,6cyclictest306-21in:imuxsock21:20:012
3602880993819,13cyclictest272-21dbus-daemon23:25:002
3602880993817,15cyclictest272-21dbus-daemon22:25:012
3602880993811,20cyclictest291rcuc/223:09:332
3602876993827,6cyclictest3616423-21modprobe23:35:001
3602876993823,10cyclictest306-21in:imuxsock23:40:321
3602876993822,11cyclictest235-21systemd-journal23:40:001
360287199386,6cyclictest12-21ksoftirqd/022:20:010
360288099374,8cyclictest30-21ksoftirqd/222:15:012
360288099373,6cyclictest30-21ksoftirqd/222:18:312
3602880993727,7cyclictest3608964-21cron21:10:002
3602880993723,10cyclictest272-21dbus-daemon21:45:012
3602880993719,14cyclictest272-21dbus-daemon23:00:012
3602880993719,13cyclictest272-21dbus-daemon20:15:012
3602880993719,13cyclictest272-21dbus-daemon20:15:012
3602876993728,5cyclictest306-21in:imuxsock19:35:011
3602876993727,5cyclictest306-21in:imuxsock19:40:011
3602876993724,7cyclictest306-21in:imuxsock21:05:001
360287699371,33cyclictest48-21kcompactd022:59:451
360287199376,4cyclictest131rcu_preempt21:03:320
360287199375,6cyclictest12-21ksoftirqd/021:00:010
360287199374,6cyclictest12-21ksoftirqd/021:42:320
3602871993730,4cyclictest16950irq/224-800000019:13:320
3602871993727,7cyclictest12-21ksoftirqd/000:30:010
3602871993725,9cyclictest3618724-21cron00:20:020
3602871993720,13cyclictest3619769-21systemctl00:40:010
360288099367,6cyclictest30-21ksoftirqd/219:30:032
360288099364,4cyclictest131rcu_preempt00:31:332
360288099363,8cyclictest30-21ksoftirqd/219:25:012
3602880993625,9cyclictest30-21ksoftirqd/200:10:022
3602880993625,9cyclictest30-21ksoftirqd/200:10:022
3602880993625,7cyclictest3611581-21pluginstate22:00:012
360288099361,6cyclictest0-21swapper/200:22:382
3602876993626,5cyclictest306-21in:imuxsock22:15:001
3602876993624,8cyclictest23-21ksoftirqd/100:25:001
3602876993623,9cyclictest1-21systemd21:35:011
3602876993621,12cyclictest3607851-21cut20:45:021
360287699361,3cyclictest131rcu_preempt22:30:001
360288099355,5cyclictest30-21ksoftirqd/220:10:022
360288099353,27cyclictest3607937-21cron20:50:002
360288099352,6cyclictest30-21ksoftirqd/221:51:402
3602880993526,5cyclictest235-21systemd-journal22:45:002
3602880993524,8cyclictest30-21ksoftirqd/200:30:012
3602880993518,12cyclictest1-21systemd23:50:012
3602876993524,9cyclictest3617955-21sshd00:02:321
360287699352,3cyclictest131rcu_preempt00:15:031
3602876993523,8cyclictest3608540-21grep21:00:011
3602876993522,9cyclictest1-21systemd22:35:011
3602876993522,10cyclictest23-21ksoftirqd/119:30:011
360287199351,31cyclictest0-21swapper/020:11:480
360288099345,5cyclictest30-21ksoftirqd/220:39:322
3602880993422,8cyclictest0-21swapper/221:30:022
3602880993421,9cyclictest272-21dbus-daemon19:55:002
3602876993425,6cyclictest271-21cron21:50:001
3602876993425,6cyclictest271-21cron21:50:001
3602876993425,6cyclictest23-21ksoftirqd/123:05:011
3602876993425,5cyclictest3605223-21latency_hist19:55:021
360287699342,4cyclictest131rcu_preempt20:00:001
3602876993423,9cyclictest23-21ksoftirqd/119:45:011
3602876993423,9cyclictest23-21ksoftirqd/119:45:011
3602876993423,8cyclictest3607121-21sed20:30:031
3602876993423,8cyclictest23-21ksoftirqd/119:25:011
3602876993422,10cyclictest23-21ksoftirqd/120:50:011
360287699341,3cyclictest131rcu_preempt20:10:001
3602871993413,5cyclictest16850irq/134-800000021:27:310
3602880993322,8cyclictest30-21ksoftirqd/223:55:022
3602880993319,10cyclictest30-21ksoftirqd/220:35:012
3602880993318,10cyclictest306-21in:imuxsock19:50:002
3602876993320,9cyclictest23-21ksoftirqd/121:50:321
3602876993317,12cyclictest272-21dbus-daemon23:55:001
3602876993316,13cyclictest272-21dbus-daemon19:15:011
3602880993225,4cyclictest3615388-21latency_hist23:15:022
3602880993224,6cyclictest30-21ksoftirqd/223:32:122
360288099321,28cyclictest272-21dbus-daemon20:20:012
3602876993221,9cyclictest23-21ksoftirqd/120:25:011
3602876993220,9cyclictest235-21systemd-journal23:15:001
3602871993220,9cyclictest408-21lldpd00:01:400
3602880993117,10cyclictest272-21dbus-daemon21:15:012
3602876993118,9cyclictest0-21swapper/123:46:481
3602876993117,10cyclictest306-21in:imuxsock20:10:021
3602876993117,10cyclictest306-21in:imuxsock20:10:021
360287699311,4cyclictest131rcu_preempt22:50:011
360287699311,28cyclictest48-21kcompactd023:21:571
360287699302,3cyclictest131rcu_preempt21:45:011
3602876993018,9cyclictest23-21ksoftirqd/121:14:191
3602876993018,7cyclictest306-21in:imuxsock19:20:001
360287699301,3cyclictest131rcu_preempt22:10:011
360287699294,19cyclictest3605644-21munin-plugin-st20:05:001
3602880992822,4cyclictest3602929-21latency_hist19:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional