You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-26 - 12:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Sep 26, 2023 00:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2259796998379,2cyclictest0-21swapper/020:15:020
2259804997024,6cyclictest2262298-21cron20:00:002
2259800997048,17cyclictest2264899-21sshd20:49:291
225980099692,3cyclictest131rcu_preempt23:20:021
2259804996815,12cyclictest2265469-21latency_hist21:00:012
2259800996853,11cyclictest2262804-21sshd20:05:291
2259804996719,11cyclictest274-21dbus-daemon19:35:002
225980099674,10cyclictest131rcu_preempt23:10:021
2259800996725,15cyclictest2268290-21systemd-run21:55:001
2259800996547,14cyclictest2267473-21sshd21:37:291
2259800996533,4cyclictest2265975-21munin-run21:10:021
2259804996328,8cyclictest30-21ksoftirqd/200:00:022
2259800996246,12cyclictest2270585-21sshd22:38:271
2259800996146,11cyclictest2275825-21munin-run00:20:001
2259800996137,17cyclictest2267767-21cron21:45:001
2259800996048,8cyclictest2266181-21sshd21:11:291
2259800996021,14cyclictest366-21snmpd23:26:431
2259804995923,8cyclictest304-21in:imuxsock22:40:002
2259804995923,7cyclictest2269046-21sshd22:09:282
2259804995916,12cyclictest274-21dbus-daemon00:35:012
2259800995952,4cyclictest2262060-21munin-run19:55:001
2259800995945,10cyclictest2271917-21munin-run23:05:001
2259800995944,11cyclictest2273954-21sshd23:41:271
2259800995944,11cyclictest2261011-21sshd19:34:291
2259800995943,12cyclictest2270079-21sshd22:27:391
2259800995938,15cyclictest2269839-21cron22:25:001
2259800995938,15cyclictest2269839-21cron22:25:001
2259804995825,11cyclictest30-21ksoftirqd/220:50:012
2259804995822,15cyclictest274-21dbus-daemon23:15:002
2259800995845,10cyclictest2261809-21munin-run19:50:001
2259800995842,13cyclictest2268006-21sshd21:47:331
2259800995842,13cyclictest2268006-21sshd21:47:331
2259796995849,4cyclictest16850irq/224-800000019:35:000
2259800995742,12cyclictest2262563-21sshd20:02:171
2259800995735,17cyclictest2272943-21cron23:24:591
225980499567,31cyclictest291rcuc/223:10:012
2259804995624,4cyclictest2273203-21latency_hist23:30:012
2259800995647,5cyclictest2275273-21munin-run00:10:001
2259800995647,5cyclictest2271135-21munin-run22:50:001
2259800995646,5cyclictest2268542-21munin-run22:00:001
2259800995643,10cyclictest2266979-21sshd21:27:291
2259800995643,10cyclictest2266979-21sshd21:27:291
2259800995641,11cyclictest2275023-21munin-run00:05:001
2259800995637,15cyclictest366-21snmpd20:43:071
2259800995637,15cyclictest366-21snmpd20:43:071
2259800995637,14cyclictest366-21snmpd22:43:291
2259800995637,14cyclictest366-21snmpd22:43:291
2259800995634,18cyclictest2273429-21sshd23:32:281
225979699562,4cyclictest131rcu_preempt00:00:010
2259800995543,9cyclictest2276279-21sshd00:26:281
2259800995542,10cyclictest2274462-21sshd23:53:061
2259800995541,11cyclictest366-21snmpd23:46:471
2259800995540,11cyclictest2272392-21sshd23:12:591
2259800995536,14cyclictest366-21snmpd21:23:391
2259796995513,33cyclictest12-21ksoftirqd/023:11:410
2259800995446,5cyclictest2263644-21munin-run20:25:011
2259800995446,5cyclictest2263644-21munin-run20:25:011
2259800995435,15cyclictest366-21snmpd20:25:091
2259800995431,17cyclictest2271604-21modprobe22:55:281
2259800995430,18cyclictest1-21systemd19:30:001
2259796995416,18cyclictest16750irq/134-800000019:27:140
2259800995345,4cyclictest2265697-21munin-run21:05:001
2259800995341,9cyclictest366-21snmpd19:20:281
2259800995341,9cyclictest2269533-21sshd22:15:291
2259800995340,9cyclictest366-21snmpd00:20:081
2259800995339,11cyclictest366-21snmpd23:56:501
2259800995339,10cyclictest366-21snmpd19:40:431
2259800995338,11cyclictest366-21snmpd19:10:261
2259800995338,10cyclictest366-21snmpd22:12:131
2259800995338,10cyclictest366-21snmpd20:54:041
2259800995338,10cyclictest366-21snmpd20:54:041
2259800995338,10cyclictest366-21snmpd20:19:301
2259804995221,5cyclictest2275019-21latency_hist00:05:012
2259800995243,5cyclictest2264427-21munin-run20:40:011
2259800995239,9cyclictest366-21snmpd19:59:101
2259800995237,12cyclictest366-21snmpd21:15:111
2259800995237,11cyclictest366-21snmpd21:31:351
2259800995235,13cyclictest366-21snmpd20:13:421
2259800995234,15cyclictest2276530-21sshd00:31:281
2259800995234,14cyclictest366-21snmpd19:37:291
2259800995231,17cyclictest2271370-21sshd22:54:291
2259796995235,13cyclictest1-21systemd19:50:010
2259804995124,7cyclictest30-21ksoftirqd/220:05:012
2259800995139,9cyclictest366-21snmpd00:10:121
2259800995139,9cyclictest366-21snmpd00:10:111
2259800995137,10cyclictest366-21snmpd23:35:071
2259800995137,10cyclictest366-21snmpd23:35:071
2259800995134,13cyclictest366-21snmpd19:15:551
2259800995126,20cyclictest1-21systemd22:10:021
2259804995033,12cyclictest235-21systemd-journal23:20:012
225980499501,3cyclictest0-21swapper/220:15:022
2259800995033,14cyclictest2264101-21sshd20:31:291
2259800995028,17cyclictest2268992-21grep22:05:021
2259796995042,4cyclictest16850irq/224-800000023:00:010
225979699502,13cyclictest2267665-21latency_hist21:40:020
2259806994933,12cyclictest1-21systemd00:10:013
2259800994937,9cyclictest366-21snmpd00:35:331
2259800994925,20cyclictest2265382-21sshd20:55:291
2259796994932,13cyclictest235-21systemd-journal19:45:000
2259796994925,18cyclictest1-21systemd22:00:000
225979699492,4cyclictest131rcu_preempt22:25:000
225979699492,4cyclictest131rcu_preempt22:25:000
2259796994913,17cyclictest16750irq/134-800000020:52:240
2259796994913,17cyclictest16750irq/134-800000020:52:240
225980699484,4cyclictest131rcu_preempt19:45:023
2259800994836,9cyclictest366-21snmpd22:31:311
2259796994833,11cyclictest1-21systemd19:55:000
2259804994734,9cyclictest274-21dbus-daemon22:15:012
2259804994730,13cyclictest274-21dbus-daemon21:05:022
2259796994726,15cyclictest274-21dbus-daemon22:20:010
2259806994628,13cyclictest1-21systemd21:10:013
2259804994634,8cyclictest30-21ksoftirqd/223:55:022
2259796994637,5cyclictest16850irq/224-800000022:10:020
2259796994629,13cyclictest274-21dbus-daemon00:15:020
2259796994629,13cyclictest274-21dbus-daemon00:15:010
2259804994532,10cyclictest274-21dbus-daemon23:50:002
2259804994530,10cyclictest235-21systemd-journal19:20:002
2259804994527,12cyclictest235-21systemd-journal23:24:592
2259796994538,4cyclictest16750irq/134-800000021:10:000
2259796994524,16cyclictest0-21swapper/000:05:020
2259796994524,15cyclictest274-21dbus-daemon22:55:000
2259800994433,9cyclictest0-21swapper/119:10:011
2259800994433,9cyclictest0-21swapper/119:10:011
2259796994433,6cyclictest2275788-21munin-plugin-st00:20:000
2259796994429,12cyclictest2274216-21sshd23:48:290
2259796994428,12cyclictest274-21dbus-daemon20:20:010
2259796994427,13cyclictest2262569-21sshd20:03:300
2259806994334,5cyclictest2261266-21cron19:40:003
2259806994324,13cyclictest274-21dbus-daemon20:30:003
2259796994329,10cyclictest2263603-21sshd20:22:200
2259796994329,10cyclictest2263603-21sshd20:22:200
2259796994328,12cyclictest2264648-21sshd20:44:290
2259796994328,12cyclictest2264648-21sshd20:44:290
2259796994324,14cyclictest235-21systemd-journal20:10:000
2259796994321,4cyclictest131rcu_preempt20:00:020
225980699425,3cyclictest131rcu_preempt00:30:023
2259806994228,11cyclictest37-21ksoftirqd/321:05:003
2259804994233,6cyclictest2264433-21latency_hist20:40:012
2259796994230,9cyclictest304-21in:imuxsock20:25:030
2259796994226,12cyclictest2265393-21sshd20:57:290
225979699422,35cyclictest2270648-21unin-run22:40:010
225979699421,37cyclictest2273959-21sshd23:42:280
225980699414,5cyclictest131rcu_preempt20:35:013
2259806994127,10cyclictest1-21systemd23:45:003
2259806994126,10cyclictest0-21swapper/323:55:003
225980499414,31cyclictest2269585-21cut22:20:002
2259804994131,8cyclictest30-21ksoftirqd/221:10:012
2259796994134,4cyclictest16850irq/224-800000021:16:290
2259796994134,4cyclictest16850irq/224-800000020:50:000
225979699412,7cyclictest2266255-21latency_hist21:15:030
2259796994124,14cyclictest366-21snmpd23:01:380
225979699411,11cyclictest2268827-21/usr/sbin/munin22:05:010
2259806994025,11cyclictest1-21systemd22:50:003
2259806994011,8cyclictest2272953-21modprobe23:24:593
2259804994030,7cyclictest2265142-21sshd20:52:292
2259804994030,7cyclictest2265142-21sshd20:52:292
2259804994023,13cyclictest1-21systemd00:25:002
2259796994029,7cyclictest18650irq/34-mmc119:15:000
2259796994028,9cyclictest2272137-21sshd23:07:280
2259796994026,11cyclictest2274470-21sshd23:54:280
2259796994024,12cyclictest12-21ksoftirqd/019:20:000
2259806993927,8cyclictest235-21systemd-journal19:54:293
2259806993927,8cyclictest235-21systemd-journal19:19:293
2259806993925,11cyclictest2268961-21sed22:05:023
225980699392,4cyclictest131rcu_preempt00:25:023
2259806993924,11cyclictest274-21dbus-daemon21:20:003
2259806993921,13cyclictest274-21dbus-daemon22:10:013
2259806993920,13cyclictest1-21systemd00:20:003
2259806993917,16cyclictest274-21dbus-daemon20:40:013
225980699391,3cyclictest131rcu_preempt22:00:013
225980699391,3cyclictest131rcu_preempt00:00:003
2259804993925,10cyclictest30-21ksoftirqd/220:30:012
2259804993922,13cyclictest30-21ksoftirqd/222:05:032
225979699399,5cyclictest131rcu_preempt22:25:280
2259796993932,4cyclictest16850irq/224-800000019:25:010
2259796993929,6cyclictest304-21in:imuxsock21:54:590
2259796993927,8cyclictest235-21systemd-journal22:50:000
2259796993927,8cyclictest2264394-21munin-plugin-st20:40:000
2259796993926,9cyclictest235-21systemd-journal21:26:300
2259796993926,9cyclictest235-21systemd-journal21:26:290
2259796993922,13cyclictest235-21systemd-journal21:40:020
2259806993830,5cyclictest2276816-21sendmail00:40:003
2259806993827,9cyclictest37-21ksoftirqd/323:40:023
2259806993827,9cyclictest37-21ksoftirqd/323:40:023
2259806993824,11cyclictest37-21ksoftirqd/321:55:013
2259806993824,10cyclictest274-21dbus-daemon21:25:013
225980699382,3cyclictest131rcu_preempt20:20:003
2259806993823,10cyclictest1-21systemd21:50:013
2259806993823,10cyclictest1-21systemd21:50:013
2259806993822,12cyclictest235-21systemd-journal20:25:003
2259806993822,12cyclictest235-21systemd-journal20:25:003
2259806993821,12cyclictest274-21dbus-daemon20:15:003
2259806993819,13cyclictest235-21systemd-journal20:55:003
2259806993819,13cyclictest235-21systemd-journal20:55:003
225980499385,3cyclictest131rcu_preempt22:25:002
225980499385,3cyclictest131rcu_preempt22:25:002
2259804993828,7cyclictest2276791-21sshd00:38:292
2259804993827,8cyclictest2274030-21sed23:45:012
2259804993827,8cyclictest2263605-21sshd20:22:202
2259804993827,8cyclictest2263605-21sshd20:22:202
2259804993824,11cyclictest30-21ksoftirqd/219:50:002
225979699388,3cyclictest131rcu_preempt23:40:000
225979699388,3cyclictest131rcu_preempt23:40:000
2259796993829,5cyclictest304-21in:imuxsock00:10:000
2259796993823,11cyclictest1-21systemd00:40:000
225979699381,7cyclictest2268102-21latency_hist21:50:020
225979699381,7cyclictest2268102-21latency_hist21:50:020
225980699374,3cyclictest131rcu_preempt23:35:003
2259806993728,5cyclictest2272680-21sendmail23:20:003
2259806993728,5cyclictest2261026-21munin-plugin-st19:35:003
2259806993727,7cyclictest2275503-21sshd00:13:103
2259806993727,7cyclictest2275503-21sshd00:13:093
2259806993724,9cyclictest274-21dbus-daemon23:30:003
2259806993724,10cyclictest2260037-21sed19:15:013
2259806993722,12cyclictest1-21systemd22:55:003
2259806993720,13cyclictest235-21systemd-journal19:25:013
2259806993719,13cyclictest274-21dbus-daemon23:00:023
2259806993710,20cyclictest2262300-21modprobe20:00:003
2259804993730,5cyclictest30-21ksoftirqd/221:55:012
2259804993729,5cyclictest2263087-21latency_hist20:15:002
225980499372,3cyclictest131rcu_preempt19:15:012
2259804993721,12cyclictest2273465-21sh23:35:002
2259804993720,11cyclictest30-21ksoftirqd/223:00:002
2259796993730,4cyclictest16850irq/224-800000023:35:010
2259796993710,21cyclictest2267246-21modprobe21:34:590
2259806993628,6cyclictest37-21ksoftirqd/323:10:013
2259806993628,6cyclictest37-21ksoftirqd/319:30:013
2259806993628,5cyclictest2266185-21sshd21:12:273
2259806993627,6cyclictest2262685-21latency_hist20:05:023
2259806993626,7cyclictest2265392-21sshd20:57:263
2259804993628,6cyclictest30-21ksoftirqd/221:40:022
2259804993626,7cyclictest30-21ksoftirqd/219:25:012
2259804993624,10cyclictest30-21ksoftirqd/220:10:012
2259804993620,12cyclictest235-21systemd-journal19:40:022
2259804993620,10cyclictest30-21ksoftirqd/219:30:002
225980499361,3cyclictest131rcu_preempt21:30:002
225980499361,3cyclictest131rcu_preempt21:30:002
225979699366,24cyclictest2273207-21munin-run23:30:000
225979699365,23cyclictest2259795-21latency_hist19:10:010
225979699365,23cyclictest2259795-21latency_hist19:10:010
225979699365,22cyclictest2261269-21modprobe19:40:000
2259796993623,10cyclictest12-21ksoftirqd/000:30:010
2259796993620,12cyclictest2272631-21sshd23:15:280
225979699361,4cyclictest2264101-21sshd20:31:280
225979699361,3cyclictest0-21swapper/023:24:280
225980699356,16cyclictest2269566-21modprobe22:20:003
2259806993524,5cyclictest0-21swapper/322:25:003
2259806993524,5cyclictest0-21swapper/322:25:003
225980699352,3cyclictest131rcu_preempt20:45:013
225980699352,3cyclictest131rcu_preempt20:45:013
225980699352,3cyclictest131rcu_preempt19:50:023
2259806993523,9cyclictest2276768-21sed00:35:033
2259806993523,10cyclictest37-21ksoftirqd/322:30:013
225980699351,4cyclictest131rcu_preempt21:35:013
2259806993513,9cyclictest2274237-21munin-run23:50:013
225980499356,21cyclictest291rcuc/222:50:022
2259804993526,7cyclictest30-21ksoftirqd/219:40:002
2259804993524,9cyclictest30-21ksoftirqd/222:35:002
2259804993523,9cyclictest2268548-21sendmail22:00:002
2259804993522,9cyclictest2264833-21sed20:45:022
2259804993522,9cyclictest2264833-21sed20:45:022
2259804993522,10cyclictest1-21systemd21:20:012
2259804993517,13cyclictest274-21dbus-daemon00:20:012
225979699355,4cyclictest131rcu_preempt21:25:000
2259796993522,10cyclictest12-21ksoftirqd/000:35:020
2259806993425,5cyclictest2270450-21latency_hist22:35:013
225980699342,3cyclictest131rcu_preempt21:40:293
225980699342,3cyclictest131rcu_preempt00:05:013
2259806993423,8cyclictest2267098-21wc21:30:033
2259806993423,8cyclictest2267098-21wc21:30:023
2259806993422,9cyclictest235-21systemd-journal21:40:003
225980699341,3cyclictest131rcu_preempt23:05:003
2259806993412,18cyclictest2272430-21unin-run23:15:013
2259804993423,8cyclictest30-21ksoftirqd/200:30:002
2259804993422,9cyclictest2270193-21sed22:30:012
2259804993415,15cyclictest235-21systemd-journal20:35:002
2259796993424,7cyclictest12-21ksoftirqd/022:42:430
2259796993424,7cyclictest12-21ksoftirqd/022:42:430
2259796993420,11cyclictest401-21lldpd21:01:560
225979699341,4cyclictest2270319-21sshd22:30:280
2259806993327,4cyclictest413-21sshd20:09:293
2259806993324,5cyclictest2270612-21munin-plugin-st22:40:003
225980699332,3cyclictest131rcu_preempt22:15:033
2259806993322,8cyclictest37-21ksoftirqd/320:50:013
225980499333,5cyclictest131rcu_preempt00:15:002
225980499333,5cyclictest131rcu_preempt00:15:002
2259804993323,7cyclictest2270850-21sshd22:40:282
2259804993323,7cyclictest2270850-21sshd22:40:282
2259804993322,8cyclictest235-21systemd-journal00:10:012
225980499331,3cyclictest131rcu_preempt19:10:012
225980499331,3cyclictest131rcu_preempt19:10:012
225980499323,5cyclictest131rcu_preempt21:25:002
2259804993222,7cyclictest2267248-21cron21:35:002
225979699324,23cyclictest2276063-21munin-run00:25:000
2259806993119,8cyclictest2271025-21wc22:45:033
2259806993119,8cyclictest2271025-21wc22:45:023
225980499315,3cyclictest131rcu_preempt23:05:012
2259804993019,8cyclictest30-21ksoftirqd/221:15:022
2259804993019,8cyclictest235-21systemd-journal22:51:282
225979699301,27cyclictest0-21swapper/022:11:320
225980499293,3cyclictest131rcu_preempt23:40:022
225980499293,3cyclictest131rcu_preempt23:40:022
2259804992914,11cyclictest274-21dbus-daemon21:50:012
2259804992914,11cyclictest274-21dbus-daemon21:50:012
225980499282,8cyclictest0-21swapper/219:55:032
2259804992816,8cyclictest2267899-21latency_hist21:45:022
2259806992719,6cyclictest37-21ksoftirqd/319:10:013
2259806992719,6cyclictest37-21ksoftirqd/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional