You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 07:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Dec 08, 2025 00:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1545747991481,136cyclictest0-21swapper/021:25:030
154574799977,3cyclictest131rcu_preempt23:50:020
1545752997737,7cyclictest23-21ksoftirqd/100:00:011
1545747997431,6cyclictest306-21in:imuxsock22:30:010
1545747997431,6cyclictest306-21in:imuxsock22:30:010
154574799674,40cyclictest19050irq/34-mmc100:35:020
1545752996341,18cyclictest1-21systemd20:55:011
1545752996341,18cyclictest1-21systemd20:55:011
1545747996250,6cyclictest0-21swapper/021:45:020
1545747996227,9cyclictest1553245-21latency_hist21:35:010
1545747996227,9cyclictest1553245-21latency_hist21:35:010
1545752996040,16cyclictest1-21systemd20:20:011
1545752996037,13cyclictest1552431-21cron21:20:001
1545757995723,19cyclictest1553983-21kworker/u8:122:00:023
1545757995723,19cyclictest1553983-21kworker/u8:122:00:023
1545754995624,27cyclictest30-21ksoftirqd/221:10:002
1545754995624,27cyclictest30-21ksoftirqd/221:10:002
1545747995623,20cyclictest1547544-21kworker/0:020:30:020
1545757995523,19cyclictest1-21systemd19:25:013
1545752995442,8cyclictest23-21ksoftirqd/100:25:011
1545752995428,21cyclictest272-21dbus-daemon21:40:001
1545752995411,6cyclictest131rcu_preempt19:35:011
1545752995340,8cyclictest23-21ksoftirqd/121:35:011
1545752995340,8cyclictest23-21ksoftirqd/121:35:011
1545752995337,12cyclictest235-21systemd-journal19:14:591
154574799534,4cyclictest131rcu_preempt00:10:010
1545747995341,2cyclictest0-21swapper/021:15:020
1545757995234,15cyclictest1-21systemd20:45:013
1545752995211,7cyclictest131rcu_preempt22:25:021
1545747995246,4cyclictest0-21swapper/019:13:240
154575299519,7cyclictest23-21ksoftirqd/123:50:001
1545757995036,9cyclictest1-21systemd21:10:013
1545757995036,9cyclictest1-21systemd21:10:013
1545757995031,14cyclictest235-21systemd-journal20:15:003
1545754995036,10cyclictest1-21systemd22:55:012
1545754995035,11cyclictest235-21systemd-journal22:20:002
1545752995028,15cyclictest235-21systemd-journal23:15:011
1545747995042,5cyclictest16950irq/224-800000019:50:010
1545747995032,13cyclictest272-21dbus-daemon20:50:010
1545747994931,13cyclictest272-21dbus-daemon20:00:000
1545747994923,6cyclictest12-21ksoftirqd/021:20:020
154574799491,42cyclictest1559223-21kworker/0:200:00:020
154574799491,3cyclictest131rcu_preempt23:45:020
154574799491,3cyclictest131rcu_preempt23:45:020
1545752994813,5cyclictest131rcu_preempt20:15:001
154574799485,28cyclictest1550856-21kworker/u8:121:00:020
1545747994834,10cyclictest235-21systemd-journal00:35:030
1545757994734,10cyclictest1-21systemd22:40:013
1545757994732,12cyclictest235-21systemd-journal20:20:013
1545754994730,12cyclictest272-21dbus-daemon00:00:012
1545754994729,13cyclictest235-21systemd-journal21:25:012
154575499472,4cyclictest131rcu_preempt20:40:012
154575499472,4cyclictest131rcu_preempt20:40:012
154575499472,4cyclictest131rcu_preempt19:40:022
1545752994726,14cyclictest235-21systemd-journal23:45:011
1545752994726,14cyclictest235-21systemd-journal23:45:011
1545754994630,12cyclictest272-21dbus-daemon23:10:002
1545754994625,15cyclictest272-21dbus-daemon21:05:012
1545752994635,8cyclictest23-21ksoftirqd/122:50:001
1545752994618,6cyclictest131rcu_preempt23:00:011
1545752994618,6cyclictest131rcu_preempt23:00:011
1545752994612,6cyclictest131rcu_preempt22:00:001
1545752994612,6cyclictest131rcu_preempt22:00:001
154575299461,15cyclictest1551402-21munin-plugin-st21:00:001
1545752994611,5cyclictest131rcu_preempt22:30:001
1545752994611,5cyclictest131rcu_preempt22:30:001
154575299461,14cyclictest0-21swapper/100:26:001
1545747994631,12cyclictest1-21systemd23:55:010
1545747994631,11cyclictest12-21ksoftirqd/023:30:020
1545747994631,11cyclictest12-21ksoftirqd/023:30:020
154574799461,4cyclictest131rcu_preempt22:10:020
1545757994530,10cyclictest235-21systemd-journal21:45:013
1545757994528,14cyclictest272-21dbus-daemon22:25:013
154575799452,3cyclictest131rcu_preempt22:35:003
1545754994516,20cyclictest30-21ksoftirqd/223:20:012
1545752994533,6cyclictest23-21ksoftirqd/122:10:021
154575299452,6cyclictest131rcu_preempt21:00:021
1545752994511,6cyclictest131rcu_preempt22:45:011
1545752994511,6cyclictest131rcu_preempt22:45:011
154574799455,35cyclictest0-21swapper/021:40:010
1545747994530,11cyclictest235-21systemd-journal20:44:240
1545757994433,8cyclictest37-21ksoftirqd/322:30:023
1545757994433,8cyclictest37-21ksoftirqd/322:30:023
1545757994431,10cyclictest1559968-21sshd23:42:353
1545757994431,10cyclictest1559968-21sshd23:42:353
1545752994431,8cyclictest1546186-21cron19:20:011
1545752994413,6cyclictest131rcu_preempt20:40:011
1545752994413,6cyclictest131rcu_preempt20:40:011
1545757994329,11cyclictest37-21ksoftirqd/321:55:003
154575799431,3cyclictest131rcu_preempt19:40:023
1545754994336,5cyclictest30-21ksoftirqd/220:59:592
1545754994331,9cyclictest30-21ksoftirqd/200:25:012
1545754994323,14cyclictest272-21dbus-daemon00:20:012
1545752994332,6cyclictest306-21in:imuxsock21:55:001
1545752994331,7cyclictest1548259-21cron19:59:591
1545752994329,10cyclictest235-21systemd-journal20:45:261
1545752994314,6cyclictest131rcu_preempt19:39:591
1545747994331,7cyclictest1558417-21munin-plugin-st23:15:000
154574799432,8cyclictest1547183-21latency_hist19:35:030
1545754994231,7cyclictest1546406-21latency_hist19:20:032
154575299427,10cyclictest23-21ksoftirqd/120:30:171
154575299426,5cyclictest131rcu_preempt00:10:011
154575299425,6cyclictest131rcu_preempt19:55:011
154575299425,6cyclictest131rcu_preempt19:55:011
1545752994228,7cyclictest306-21in:imuxsock00:20:011
1545752994224,13cyclictest235-21systemd-journal23:30:011
1545752994224,13cyclictest235-21systemd-journal23:30:011
1545752994215,5cyclictest131rcu_preempt23:10:011
1545752994211,6cyclictest131rcu_preempt21:15:011
1545747994229,9cyclictest306-21in:imuxsock23:40:010
154574799421,13cyclictest363-21snmpd21:52:390
1545757994131,5cyclictest306-21in:imuxsock00:10:003
1545757994127,10cyclictest272-21dbus-daemon21:20:013
1545754994129,8cyclictest30-21ksoftirqd/220:00:022
1545754994129,7cyclictest306-21in:imuxsock00:40:012
1545754994128,10cyclictest1547749-21sshd19:48:162
1545754994126,11cyclictest1-21systemd22:30:012
1545754994126,11cyclictest1-21systemd22:30:012
154575299419,6cyclictest131rcu_preempt00:40:021
154575299418,6cyclictest131rcu_preempt22:05:001
154575299418,5cyclictest131rcu_preempt20:05:011
1545752994132,5cyclictest1558945-21cron23:25:001
1545752994126,9cyclictest306-21in:imuxsock23:40:001
1545752994110,6cyclictest131rcu_preempt22:15:021
1545752994110,6cyclictest131rcu_preempt22:15:011
1545752994110,11cyclictest131rcu_preempt22:20:011
1545747994128,10cyclictest235-21systemd-journal22:57:080
1545747994128,10cyclictest235-21systemd-journal22:57:080
1545747994124,13cyclictest1558938-21sshd23:23:350
154574799412,34cyclictest363-21snmpd23:06:590
1545757994026,11cyclictest37-21ksoftirqd/323:40:023
1545757994022,14cyclictest1551105-21sshd20:51:283
1545757994022,14cyclictest1551105-21sshd20:51:273
1545754994031,5cyclictest306-21in:imuxsock23:45:012
1545754994031,5cyclictest306-21in:imuxsock23:45:012
1545754994028,9cyclictest1551103-21sshd20:51:282
1545754994028,9cyclictest1551103-21sshd20:51:282
1545754994027,9cyclictest30-21ksoftirqd/223:55:012
1545754994022,13cyclictest272-21dbus-daemon23:30:012
1545754994022,13cyclictest272-21dbus-daemon23:30:012
1545754994019,16cyclictest272-21dbus-daemon20:25:012
1545754994015,14cyclictest30-21ksoftirqd/200:10:022
154575299407,10cyclictest23-21ksoftirqd/119:45:011
154575299406,4cyclictest131rcu_preempt00:35:011
1545752994029,8cyclictest23-21ksoftirqd/121:22:381
1545752994024,12cyclictest272-21dbus-daemon20:10:011
1545752994024,12cyclictest272-21dbus-daemon20:10:001
1545752994011,5cyclictest131rcu_preempt23:55:021
1545752994011,4cyclictest131rcu_preempt23:04:591
1545747994032,6cyclictest12-21ksoftirqd/022:40:010
1545747994032,5cyclictest19050irq/34-mmc121:59:320
1545747994032,5cyclictest19050irq/34-mmc121:59:320
1545747994029,6cyclictest1547223-21cron19:40:000
1545747994025,12cyclictest235-21systemd-journal22:15:020
1545747994025,12cyclictest235-21systemd-journal22:15:020
154574799401,34cyclictest408-21lldpd20:50:040
154574799401,34cyclictest408-21lldpd20:50:030
1545757993927,9cyclictest1553636-21sed21:40:023
1545757993925,11cyclictest1550847-21sshd20:45:263
1545757993923,12cyclictest37-21ksoftirqd/321:50:003
1545754993928,8cyclictest1553980-21sshd21:46:342
1545754993928,7cyclictest1549305-21cron20:20:002
1545754993928,5cyclictest306-21in:imuxsock19:55:012
1545754993928,5cyclictest306-21in:imuxsock19:55:012
1545754993927,7cyclictest1556331-21cron22:35:002
1545754993926,10cyclictest30-21ksoftirqd/222:40:352
1545754993926,10cyclictest30-21ksoftirqd/222:40:352
1545754993926,10cyclictest235-21systemd-journal00:11:462
1545754993922,5cyclictest30-21ksoftirqd/222:15:002
1545754993922,5cyclictest30-21ksoftirqd/222:14:592
154575299395,9cyclictest23-21ksoftirqd/121:10:021
154575299395,9cyclictest23-21ksoftirqd/121:10:021
154575299393,4cyclictest131rcu_preempt20:25:011
1545752993929,4cyclictest1556330-21cron22:35:001
154575299391,5cyclictest1558672-21modprobe23:20:001
154574799393,8cyclictest1549069-21munin-run20:15:010
1545747993931,5cyclictest1549320-21sendmail20:20:000
1545747993925,10cyclictest1-21systemd19:25:150
154574799392,33cyclictest235-21systemd-journal20:25:010
1545747993917,15cyclictest272-21dbus-daemon23:20:000
154574799391,34cyclictest0-21swapper/020:30:010
154575799383,4cyclictest131rcu_preempt21:05:023
1545757993828,7cyclictest37-21ksoftirqd/300:25:003
1545757993827,9cyclictest37-21ksoftirqd/320:25:023
1545757993827,8cyclictest37-21ksoftirqd/322:50:013
1545757993827,7cyclictest1561540-21munin-plugin-st00:15:003
1545757993825,9cyclictest306-21in:imuxsock23:05:003
1545757993825,9cyclictest1548738-21sed20:05:013
1545757993825,10cyclictest37-21ksoftirqd/320:00:003
1545757993825,10cyclictest37-21ksoftirqd/300:05:023
1545757993825,10cyclictest37-21ksoftirqd/300:05:023
1545757993824,11cyclictest37-21ksoftirqd/322:05:013
1545754993828,6cyclictest1552445-21modprobe21:20:002
1545754993826,8cyclictest1550632-21modprobe20:45:002
1545754993825,10cyclictest30-21ksoftirqd/220:30:012
1545754993817,3cyclictest30-21ksoftirqd/221:55:012
1545754993816,5cyclictest131rcu_preempt23:15:022
1545754993814,6cyclictest131rcu_preempt21:45:012
1545754993814,20cyclictest30-21ksoftirqd/223:50:002
154575299388,7cyclictest23-21ksoftirqd/120:42:241
1545752993826,9cyclictest23-21ksoftirqd/100:15:001
1545752993825,9cyclictest23-21ksoftirqd/121:41:351
1545752993825,8cyclictest23-21ksoftirqd/119:50:011
1545752993823,11cyclictest235-21systemd-journal23:35:001
1545752993810,5cyclictest131rcu_preempt22:40:011
1545747993827,8cyclictest1546445-21sshd19:20:150
154574799382,15cyclictest131rcu_preempt20:10:010
154574799382,15cyclictest131rcu_preempt20:10:010
1545747993819,15cyclictest1554761-21sshd22:00:350
1545747993810,24cyclictest0-21swapper/019:18:500
154575799373,3cyclictest131rcu_preempt23:50:013
1545757993728,5cyclictest1549834-21cron20:30:003
1545757993727,5cyclictest1560496-21munin-plugin-st23:55:003
1545757993724,9cyclictest235-21systemd-journal00:00:013
1545757993722,7cyclictest1544917-21kworker/3:019:20:023
154575499377,14cyclictest1556588-21modprobe22:40:002
1545754993727,8cyclictest30-21ksoftirqd/219:21:142
1545754993727,6cyclictest1557656-21sendmail23:00:002
1545754993727,6cyclictest1557656-21sendmail23:00:002
154575499372,3cyclictest131rcu_preempt20:10:022
154575499372,3cyclictest131rcu_preempt20:10:012
1545754993719,13cyclictest1-21systemd00:35:012
154575499371,3cyclictest131rcu_preempt23:25:002
1545752993726,8cyclictest1554017-21systemd-run21:50:011
1545752993723,11cyclictest23-21ksoftirqd/120:30:011
1545752993719,12cyclictest1546725-21modprobe19:30:001
1545752993710,5cyclictest131rcu_preempt22:55:001
154574799376,5cyclictest131rcu_preempt22:20:010
1545747993733,2cyclictest0-21swapper/021:30:000
1545747993729,5cyclictest1559432-21sshd23:31:350
1545747993727,6cyclictest12-21ksoftirqd/019:10:020
154574799372,32cyclictest235-21systemd-journal22:48:520
1545747993723,10cyclictest408-21lldpd21:45:350
1545747993721,13cyclictest1557350-21sshd22:50:350
154575799368,6cyclictest131rcu_preempt21:23:323
1545757993629,4cyclictest1557402-21latency_hist22:55:013
154575799362,4cyclictest131rcu_preempt00:40:003
1545757993623,9cyclictest37-21ksoftirqd/320:35:183
1545757993623,9cyclictest37-21ksoftirqd/320:35:183
1545757993622,11cyclictest37-21ksoftirqd/300:30:003
1545757993618,13cyclictest1-21systemd23:30:013
1545757993618,13cyclictest1-21systemd23:30:013
154575499365,5cyclictest131rcu_preempt22:20:352
1545754993626,6cyclictest1559719-21sendmail23:40:002
1545754993626,5cyclictest1550097-21munin-plugin-st20:35:002
1545754993625,8cyclictest30-21ksoftirqd/223:02:472
1545754993623,10cyclictest30-21ksoftirqd/220:00:022
1545752993627,7cyclictest23-21ksoftirqd/119:25:011
1545752993622,11cyclictest1545775-21sed19:10:021
1545747993631,3cyclictest0-21swapper/000:30:010
154574799362,5cyclictest1556846-21sshd22:41:350
154574799362,5cyclictest1556846-21sshd22:41:350
154575799358,3cyclictest131rcu_preempt19:12:163
154575799357,18cyclictest1549815-21kworker/u8:020:35:023
154575799353,10cyclictest37-21ksoftirqd/320:07:053
154575799353,10cyclictest37-21ksoftirqd/320:07:053
1545757993526,5cyclictest37-21ksoftirqd/323:20:003
154575799352,3cyclictest131rcu_preempt21:00:013
1545757993522,9cyclictest1-21systemd00:20:013
1545757993520,11cyclictest37-21ksoftirqd/322:05:343
154575499355,5cyclictest131rcu_preempt20:15:012
1545754993525,6cyclictest306-21in:imuxsock23:35:002
1545754993524,9cyclictest30-21ksoftirqd/221:30:012
1545754993523,8cyclictest1562314-21sshd00:29:002
1545754993520,11cyclictest30-21ksoftirqd/219:15:002
1545754993520,11cyclictest235-21systemd-journal21:30:022
1545754993520,11cyclictest235-21systemd-journal21:30:012
154575499351,3cyclictest131rcu_preempt19:10:012
154575299357,5cyclictest131rcu_preempt00:04:461
154575299357,5cyclictest131rcu_preempt00:04:461
1545752993510,4cyclictest131rcu_preempt21:27:291
154574799355,5cyclictest12-21ksoftirqd/019:55:010
154574799355,5cyclictest12-21ksoftirqd/019:55:010
154574799352,6cyclictest235-21systemd-journal22:32:410
1545747993525,7cyclictest12-21ksoftirqd/020:40:010
1545747993525,7cyclictest12-21ksoftirqd/020:40:000
1545747993520,11cyclictest1562048-21sshd00:20:570
154574799351,6cyclictest1548553-21munin-run20:05:000
154575799344,10cyclictest37-21ksoftirqd/321:14:553
1545757993427,4cyclictest1553226-21/usr/sbin/munin21:35:013
1545757993427,4cyclictest1553226-21/usr/sbin/munin21:35:013
1545757993426,6cyclictest37-21ksoftirqd/319:30:013
1545757993426,5cyclictest1558602-21latency_hist23:15:023
1545757993424,8cyclictest37-21ksoftirqd/323:35:013
1545757993421,8cyclictest341irq_work/323:00:023
1545757993421,8cyclictest341irq_work/323:00:023
1545757993421,10cyclictest37-21ksoftirqd/319:55:023
1545757993421,10cyclictest37-21ksoftirqd/319:55:023
154575499344,3cyclictest131rcu_preempt20:50:012
1545754993423,8cyclictest30-21ksoftirqd/200:05:022
1545754993423,8cyclictest30-21ksoftirqd/200:05:022
1545754993420,11cyclictest1554524-21systemctl22:00:002
1545754993420,11cyclictest1554524-21systemctl22:00:002
1545754993415,5cyclictest131rcu_preempt21:11:322
154574799342,29cyclictest0-21swapper/000:17:040
1545747993420,11cyclictest363-21snmpd19:41:160
1545747993418,13cyclictest1561505-21sshd00:10:460
154574799341,29cyclictest0-21swapper/023:00:350
154574799341,29cyclictest0-21swapper/021:05:280
154574799341,29cyclictest0-21swapper/021:05:280
1545757993322,8cyclictest37-21ksoftirqd/321:30:003
154575799332,10cyclictest37-21ksoftirqd/319:45:033
1545757993319,11cyclictest1547962-21sed19:50:023
154575499338,17cyclictest291rcuc/219:35:012
1545754993323,7cyclictest30-21ksoftirqd/222:50:012
1545754993323,5cyclictest30-21ksoftirqd/219:45:022
154574799333,5cyclictest235-21systemd-journal00:00:460
154574799333,5cyclictest235-21systemd-journal00:00:460
1545747993327,4cyclictest0-21swapper/022:22:090
154575799324,3cyclictest131rcu_preempt22:45:013
154575799324,3cyclictest131rcu_preempt22:45:013
1545757993222,7cyclictest37-21ksoftirqd/322:15:023
1545757993222,7cyclictest37-21ksoftirqd/322:15:023
1545757993220,9cyclictest37-21ksoftirqd/323:25:003
154575799321,3cyclictest131rcu_preempt22:20:003
1545754993225,3cyclictest30-21ksoftirqd/219:30:012
1545754993223,5cyclictest30-21ksoftirqd/222:10:012
1545754993221,6cyclictest30-21ksoftirqd/221:40:002
1545754993214,5cyclictest131rcu_preempt22:05:002
154574799325,24cyclictest12-21ksoftirqd/021:01:070
154575799312,4cyclictest131rcu_preempt19:35:013
1545757993122,7cyclictest37-21ksoftirqd/300:35:013
1545757993120,8cyclictest37-21ksoftirqd/323:10:003
154575799261,3cyclictest131rcu_preempt19:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional