You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 07:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Nov 24, 2025 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
401196399961,3cyclictest0-21swapper/023:15:020
4011963999283,4cyclictest0-21swapper/021:15:010
4011968997122,10cyclictest23-21ksoftirqd/120:55:011
401196899677,22cyclictest4012846-21cut19:25:031
401197799642,10cyclictest37-21ksoftirqd/321:50:003
401197399634,6cyclictest131rcu_preempt22:15:002
4011973996216,12cyclictest30-21ksoftirqd/200:00:002
4011973996216,12cyclictest30-21ksoftirqd/200:00:002
401196899629,15cyclictest4026459-21cut23:45:021
4011968996216,8cyclictest23-21ksoftirqd/100:10:001
4011968996215,8cyclictest23-21ksoftirqd/123:50:011
4011968996116,9cyclictest23-21ksoftirqd/119:10:011
4011968996115,10cyclictest23-21ksoftirqd/120:25:021
401197799608,15cyclictest37-21ksoftirqd/300:00:003
401197799608,15cyclictest37-21ksoftirqd/300:00:003
4011968996017,9cyclictest23-21ksoftirqd/122:35:001
4011968996017,9cyclictest23-21ksoftirqd/122:35:001
4011968996010,17cyclictest4024832-21latency_hist23:15:021
4011977995911,5cyclictest131rcu_preempt20:55:013
4011968995917,9cyclictest23-21ksoftirqd/121:00:001
4011968995917,9cyclictest23-21ksoftirqd/121:00:001
4011968995915,9cyclictest23-21ksoftirqd/120:50:021
4011968995915,10cyclictest23-21ksoftirqd/119:55:021
401197399583,7cyclictest30-21ksoftirqd/200:25:002
401196899588,16cyclictest4027797-21latency_hist00:15:021
401196899588,16cyclictest4027797-21latency_hist00:15:021
401196899588,13cyclictest4013907-21date19:45:021
401196899588,13cyclictest4013907-21date19:45:011
4011968995818,7cyclictest23-21ksoftirqd/120:35:011
4011968995810,14cyclictest4020860-21wc22:00:011
4011977995711,6cyclictest131rcu_preempt21:10:013
4011968995739,14cyclictest4027280-21sshd00:04:371
4011968995716,9cyclictest23-21ksoftirqd/120:20:021
401197399569,5cyclictest131rcu_preempt22:05:012
4011968995638,13cyclictest1-21systemd20:45:011
4011968995638,13cyclictest1-21systemd20:45:011
4011968995615,9cyclictest23-21ksoftirqd/119:15:001
401197799556,7cyclictest37-21ksoftirqd/322:56:593
4011977995535,15cyclictest1-21systemd20:20:003
4011973995544,8cyclictest30-21ksoftirqd/221:15:012
4011973995510,5cyclictest131rcu_preempt22:45:002
4011968995511,14cyclictest4027063-21munin-run00:00:011
4011968995511,14cyclictest4027063-21munin-run00:00:011
4011963995549,3cyclictest0-21swapper/023:30:020
401197399542,5cyclictest131rcu_preempt23:25:022
401197399542,5cyclictest131rcu_preempt23:25:022
401196899547,14cyclictest4018841-21sed21:20:021
401196899546,14cyclictest4022802-21apt-helper22:39:231
4011977995320,12cyclictest37-21ksoftirqd/323:35:273
4011977995315,9cyclictest131rcu_preempt20:30:003
4011977995315,9cyclictest131rcu_preempt20:30:003
401197399538,5cyclictest131rcu_preempt20:40:012
4011973995312,5cyclictest131rcu_preempt23:30:012
401196899536,16cyclictest4028410-21sed00:25:011
401196899534,15cyclictest4026123-21latency_hist23:40:021
4011968995313,8cyclictest23-21ksoftirqd/100:20:011
401197799525,8cyclictest37-21ksoftirqd/300:05:003
4011977995224,19cyclictest37-21ksoftirqd/319:15:023
4011977995215,5cyclictest131rcu_preempt23:15:023
401197399524,5cyclictest30-21ksoftirqd/220:10:002
401196899523,17cyclictest0-21swapper/119:50:011
401196399525,34cyclictest4011966-21latency_hist19:10:000
401197799518,5cyclictest131rcu_preempt20:15:023
401197799516,9cyclictest37-21ksoftirqd/323:18:233
401197799513,9cyclictest37-21ksoftirqd/319:20:023
401197799513,6cyclictest131rcu_preempt21:25:023
4011977995111,5cyclictest131rcu_preempt19:30:003
4011973995141,8cyclictest291rcuc/219:15:012
4011973995112,5cyclictest131rcu_preempt22:40:002
401196899515,13cyclictest4020975-21modprobe22:00:021
4011963995143,4cyclictest16950irq/224-800000000:00:010
4011963995143,4cyclictest16950irq/224-800000000:00:010
4011977995015,6cyclictest131rcu_preempt19:45:013
4011977995015,6cyclictest131rcu_preempt19:45:003
4011977995010,5cyclictest131rcu_preempt20:10:003
401197399504,12cyclictest30-21ksoftirqd/221:00:012
401197399504,12cyclictest30-21ksoftirqd/221:00:012
401197399502,11cyclictest30-21ksoftirqd/219:30:002
4011973995011,6cyclictest131rcu_preempt22:55:002
4011973995011,5cyclictest131rcu_preempt00:15:372
4011973995010,6cyclictest131rcu_preempt23:15:002
401196899506,16cyclictest1-21systemd21:24:191
401197799499,5cyclictest131rcu_preempt00:15:003
401197799499,5cyclictest131rcu_preempt00:15:003
401197799498,5cyclictest131rcu_preempt19:25:023
401197799497,8cyclictest37-21ksoftirqd/320:50:023
4011977994914,6cyclictest37-21ksoftirqd/319:55:023
4011977994914,6cyclictest131rcu_preempt22:00:023
4011977994912,5cyclictest131rcu_preempt23:10:023
4011977994911,5cyclictest131rcu_preempt23:45:023
401197799491,11cyclictest37-21ksoftirqd/323:55:013
401197399499,6cyclictest131rcu_preempt00:15:022
401197399499,6cyclictest131rcu_preempt00:15:022
401197399498,7cyclictest131rcu_preempt21:05:022
401197399498,7cyclictest131rcu_preempt21:05:022
401197399494,6cyclictest131rcu_preempt23:50:012
401197399493,10cyclictest30-21ksoftirqd/223:05:002
401197399491,5cyclictest131rcu_preempt21:50:012
401197399491,5cyclictest131rcu_preempt00:40:012
4011973994915,6cyclictest131rcu_preempt00:05:002
4011973994914,8cyclictest131rcu_preempt19:25:022
4011963994937,8cyclictest19050irq/34-mmc123:50:030
4011963994933,11cyclictest272-21dbus-daemon00:20:000
401196399491,3cyclictest131rcu_preempt20:10:020
401197799488,5cyclictest131rcu_preempt22:49:233
401197799488,5cyclictest131rcu_preempt22:49:233
401197799487,9cyclictest37-21ksoftirqd/321:45:013
401197799486,11cyclictest37-21ksoftirqd/322:39:243
401197799484,8cyclictest37-21ksoftirqd/320:01:403
401197799482,8cyclictest37-21ksoftirqd/321:00:023
401197799482,8cyclictest37-21ksoftirqd/321:00:013
4011977994813,6cyclictest131rcu_preempt00:40:023
4011977994813,5cyclictest131rcu_preempt22:30:003
4011977994813,4cyclictest131rcu_preempt20:25:023
4011977994811,6cyclictest131rcu_preempt23:50:023
401197399489,6cyclictest131rcu_preempt23:10:012
401197399485,6cyclictest131rcu_preempt20:25:022
4011973994821,7cyclictest291rcuc/223:55:022
4011968994830,13cyclictest272-21dbus-daemon21:55:011
4011968994810,9cyclictest23-21ksoftirqd/122:50:011
4011968994810,9cyclictest23-21ksoftirqd/122:50:011
401196399481,28cyclictest0-21swapper/019:34:140
401197799472,13cyclictest37-21ksoftirqd/322:55:013
4011977994712,6cyclictest131rcu_preempt00:16:423
4011973994714,7cyclictest131rcu_preempt19:35:002
4011973994712,5cyclictest131rcu_preempt22:35:022
4011973994712,5cyclictest131rcu_preempt22:35:022
4011973994711,5cyclictest131rcu_preempt21:58:202
401196899475,12cyclictest1-21systemd23:55:011
401196899473,13cyclictest4019157-21modprobe21:25:021
4011968994726,15cyclictest235-21systemd-journal19:20:001
401196399476,35cyclictest4022875-21modprobe22:40:000
4011963994735,6cyclictest4013755-21latency_hist19:45:010
4011963994735,6cyclictest4013755-21latency_hist19:45:000
401197799468,5cyclictest131rcu_preempt20:40:013
401197799467,8cyclictest37-21ksoftirqd/319:50:013
401197799463,7cyclictest4028590-21cron00:30:003
4011977994635,9cyclictest37-21ksoftirqd/319:35:143
401197799462,7cyclictest37-21ksoftirqd/320:35:003
401197799462,5cyclictest131rcu_preempt22:20:023
4011977994610,5cyclictest131rcu_preempt23:03:233
401197399469,9cyclictest131rcu_preempt21:25:022
401197399467,5cyclictest131rcu_preempt19:58:152
401197399467,5cyclictest131rcu_preempt19:58:152
401197399463,8cyclictest30-21ksoftirqd/221:10:002
4011973994636,7cyclictest30-21ksoftirqd/220:55:012
401197399462,5cyclictest131rcu_preempt20:05:012
4011973994613,6cyclictest131rcu_preempt22:10:032
401196899465,14cyclictest4022186-21latency_hist22:25:021
4011968994633,10cyclictest4022278-21sshd22:26:221
4011968994615,24cyclictest272-21dbus-daemon22:10:011
4011963994631,12cyclictest12-21ksoftirqd/023:55:000
4011963994631,10cyclictest272-21dbus-daemon19:15:010
401197799454,12cyclictest37-21ksoftirqd/319:35:013
401197799453,11cyclictest37-21ksoftirqd/321:20:003
4011977994517,17cyclictest131rcu_preempt00:21:373
4011977994512,5cyclictest131rcu_preempt00:10:003
401197799451,11cyclictest37-21ksoftirqd/322:15:003
4011977994510,4cyclictest131rcu_preempt23:22:233
4011977994510,4cyclictest131rcu_preempt23:22:233
401197399456,6cyclictest131rcu_preempt20:35:012
401197399456,5cyclictest30-21ksoftirqd/200:35:002
401197399456,4cyclictest131rcu_preempt21:40:012
401197399452,5cyclictest131rcu_preempt22:20:012
4011973994513,11cyclictest30-21ksoftirqd/222:30:022
4011973994511,5cyclictest131rcu_preempt23:00:002
4011968994531,10cyclictest235-21systemd-journal23:10:001
4011968994530,11cyclictest4019668-21sshd21:36:141
4011968994521,18cyclictest4028839-21cron00:35:011
401197799449,5cyclictest131rcu_preempt22:35:023
401197799449,5cyclictest131rcu_preempt22:35:023
401197799445,8cyclictest37-21ksoftirqd/321:05:023
401197799445,8cyclictest37-21ksoftirqd/321:05:023
401197799443,8cyclictest37-21ksoftirqd/322:45:003
401197799441,9cyclictest4022103-21latency_hist22:25:013
4011977994415,6cyclictest131rcu_preempt00:35:013
4011977994410,5cyclictest131rcu_preempt21:36:143
401197399448,6cyclictest131rcu_preempt23:40:032
401197399448,6cyclictest131rcu_preempt21:55:022
401197399448,10cyclictest30-21ksoftirqd/221:20:002
401197399445,8cyclictest30-21ksoftirqd/221:44:142
401197399445,5cyclictest131rcu_preempt23:20:022
401197399443,8cyclictest30-21ksoftirqd/219:50:002
4011973994434,7cyclictest30-21ksoftirqd/223:40:002
4011968994429,12cyclictest4024920-21sshd23:18:231
4011968994428,12cyclictest4018379-21sshd21:13:141
4011968994426,14cyclictest1-21systemd21:05:001
4011968994426,14cyclictest1-21systemd21:05:001
4011968994423,17cyclictest4019448-21munin-run21:35:001
4011968994423,16cyclictest4012939-21cron19:30:001
401196899441,15cyclictest363-21snmpd23:26:551
4011963994433,6cyclictest4018668-21sendmail21:20:000
4011963994432,10cyclictest12-21ksoftirqd/019:50:010
4011963994429,11cyclictest235-21systemd-journal23:40:000
4011963994429,11cyclictest1-21systemd20:35:000
4011963994427,11cyclictest12-21ksoftirqd/021:05:010
4011963994427,11cyclictest12-21ksoftirqd/021:05:010
4011977994332,8cyclictest37-21ksoftirqd/323:30:013
4011977994327,12cyclictest4021252-21cron22:10:003
4011977994310,5cyclictest131rcu_preempt20:45:013
4011977994310,5cyclictest131rcu_preempt20:45:013
4011973994331,8cyclictest4027527-21sshd00:08:452
401197399432,9cyclictest30-21ksoftirqd/222:50:012
401197399432,9cyclictest30-21ksoftirqd/222:50:012
4011973994310,5cyclictest131rcu_preempt19:55:022
4011968994328,12cyclictest4023118-21sshd22:42:231
4011968994328,11cyclictest4029150-21munin-run00:40:011
4011968994318,20cyclictest4013441-21sshd19:39:141
4011968994317,21cyclictest1-21systemd23:05:011
401196399432,24cyclictest151rcuc/019:30:020
401196399432,11cyclictest4016134-21sort20:30:020
401196399432,11cyclictest4016134-21sort20:30:020
401197799426,6cyclictest131rcu_preempt21:35:003
4011977994233,6cyclictest37-21ksoftirqd/321:30:003
4011977994227,11cyclictest1-21systemd23:35:003
401197799421,8cyclictest37-21ksoftirqd/319:10:003
401197399426,6cyclictest131rcu_preempt20:49:142
401197399426,6cyclictest131rcu_preempt19:45:022
401197399426,6cyclictest131rcu_preempt19:45:022
401197399424,7cyclictest30-21ksoftirqd/220:15:012
401197399424,6cyclictest131rcu_preempt23:35:002
401197399424,6cyclictest131rcu_preempt20:20:002
4011973994212,6cyclictest131rcu_preempt22:25:002
4011973994210,6cyclictest30-21ksoftirqd/221:35:002
4011968994233,6cyclictest23-21ksoftirqd/120:14:591
4011968994229,10cyclictest4016544-21sshd20:36:461
4011968994227,9cyclictest4014533-21munin-run20:00:011
4011968994227,9cyclictest4014533-21munin-run20:00:011
4011968994227,11cyclictest4028595-21munin-run00:30:011
4011968994227,11cyclictest4019980-21munin-run21:45:001
4011968994226,13cyclictest4023878-21sshd22:58:231
4011968994221,16cyclictest363-21snmpd23:32:151
401196899420,13cyclictest363-21snmpd21:46:351
401196399425,3cyclictest131rcu_preempt19:40:010
4011963994227,12cyclictest4017088-21sshd20:49:140
4011963994227,10cyclictest272-21dbus-daemon20:39:590
4011963994227,10cyclictest0-21swapper/023:05:010
4011973994131,7cyclictest30-21ksoftirqd/220:45:012
4011973994131,7cyclictest30-21ksoftirqd/220:45:012
401197399412,6cyclictest30-21ksoftirqd/219:37:142
401196899414,6cyclictest23-21ksoftirqd/121:05:141
4011968994127,10cyclictest4023661-21munin-run22:55:011
4011968994127,10cyclictest4015006-21sshd20:09:151
4011968994126,9cyclictest272-21dbus-daemon20:05:001
4011968994125,13cyclictest4016032-21sshd20:25:141
4011968994125,13cyclictest4016032-21sshd20:25:141
4011968994119,18cyclictest4025234-21/usr/sbin/munin23:25:011
4011968994119,18cyclictest4025234-21/usr/sbin/munin23:25:011
4011963994134,4cyclictest16950irq/224-800000022:40:230
4011963994130,6cyclictest4029110-21sh00:40:000
4011963994126,12cyclictest12-21ksoftirqd/000:05:010
4011963994117,3cyclictest131rcu_preempt00:35:020
401197399404,7cyclictest30-21ksoftirqd/200:30:022
4011973994029,8cyclictest4012398-21sshd19:18:142
401197399402,6cyclictest131rcu_preempt21:30:022
4011968994025,10cyclictest363-21snmpd19:30:491
4011963994026,9cyclictest4015541-21cat20:20:000
4011963994024,12cyclictest4028309-21sshd00:21:440
4011963994021,13cyclictest272-21dbus-daemon20:25:010
4011963994013,3cyclictest0-21swapper/023:45:020
4011977993930,6cyclictest37-21ksoftirqd/320:00:003
4011977993930,6cyclictest37-21ksoftirqd/320:00:003
401197799391,9cyclictest37-21ksoftirqd/322:05:023
401197399395,5cyclictest30-21ksoftirqd/220:30:012
401197399395,5cyclictest30-21ksoftirqd/220:30:012
4011968993919,15cyclictest363-21snmpd22:11:481
4011963993928,8cyclictest12-21ksoftirqd/022:25:000
4011963993927,8cyclictest12-21ksoftirqd/019:20:000
4011963993919,15cyclictest235-21systemd-journal21:30:000
401197799384,9cyclictest37-21ksoftirqd/321:13:143
401196899385,22cyclictest4021772-21modprobe22:20:001
401196399383,4cyclictest4027538-21modprobe00:10:000
4011963993831,4cyclictest16850irq/134-800000021:08:140
401196399382,3cyclictest131rcu_preempt20:55:000
401196399382,10cyclictest12-21ksoftirqd/023:25:010
401196399382,10cyclictest12-21ksoftirqd/023:25:000
401196399381,3cyclictest131rcu_preempt22:20:010
401197799371,6cyclictest0-21swapper/321:52:233
4011963993727,6cyclictest4020722-21cron22:00:000
4011963993726,8cyclictest4016824-21sshd20:41:470
4011963993726,8cyclictest4016824-21sshd20:41:470
4011963993725,10cyclictest12-21ksoftirqd/021:50:020
4011963993724,10cyclictest4020459-21sshd21:52:150
4011963993724,10cyclictest12-21ksoftirqd/020:00:010
4011963993724,10cyclictest12-21ksoftirqd/020:00:010
401196399371,3cyclictest131rcu_preempt22:50:010
401196399371,3cyclictest131rcu_preempt22:50:010
401196399363,29cyclictest0-21swapper/021:44:240
4011963993627,6cyclictest12-21ksoftirqd/022:05:020
4011963993624,8cyclictest12-21ksoftirqd/023:00:020
4011963993623,10cyclictest12-21ksoftirqd/023:30:000
401196399361,31cyclictest1-21systemd22:30:230
401196399361,31cyclictest1-21systemd22:30:230
401196399361,31cyclictest0-21swapper/021:36:330
401196399361,31cyclictest0-21swapper/019:52:140
4011963993529,4cyclictest0-21swapper/000:26:190
4011963993528,5cyclictest19050irq/34-mmc123:05:520
4011963993523,8cyclictest235-21systemd-journal22:55:000
4011963993523,10cyclictest12-21ksoftirqd/022:10:030
4011963993522,10cyclictest12-21ksoftirqd/000:15:000
4011963993522,10cyclictest12-21ksoftirqd/000:15:000
4011963993520,11cyclictest4021261-21latency_hist22:10:020
4011963993425,5cyclictest4012839-21latency_hist19:25:030
4011963993423,7cyclictest12-21ksoftirqd/021:25:010
4011963993422,9cyclictest4014903-21date20:05:020
401196399341,5cyclictest4017594-21modprobe20:59:140
401196399341,5cyclictest4017594-21modprobe20:59:140
401196399341,29cyclictest0-21swapper/022:26:140
401196399341,29cyclictest0-21swapper/021:30:530
401196399334,26cyclictest12-21ksoftirqd/020:12:380
401196399311,3cyclictest131rcu_preempt23:10:020
4011973992719,5cyclictest4011900-21latency_hist19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional