You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-09 - 03:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Apr 09, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
20156839912448,9cyclictest37-21ksoftirqd/322:40:013
2015666991103,3cyclictest131rcu_preempt21:45:020
2015666991103,3cyclictest131rcu_preempt21:45:020
20156669910417,11cyclictest2076043-21cat23:15:020
2015666999628,13cyclictest2042615-21ssh21:57:380
2015666999628,13cyclictest2042615-21ssh21:57:370
2015678998656,20cyclictest30-21ksoftirqd/221:10:022
2015678998656,20cyclictest30-21ksoftirqd/221:10:012
2015666998428,12cyclictest2033903-21ssh21:37:260
2015666998326,13cyclictest2090849-21ssh23:49:210
2015666998227,12cyclictest2110361-21ssh00:34:230
2015666998127,17cyclictest2104196-21ssh00:20:010
2015666997827,15cyclictest2022138-21ssh21:10:090
2015666997727,10cyclictest2071954-21ssh23:05:160
2015666997726,12cyclictest2049284-21ssh22:13:070
201566699772,39cyclictest2054830-21ssh22:25:410
201566699762,38cyclictest2063357-21ssh22:45:300
2015666997535,4cyclictest16950irq/224-800000022:50:280
2015666997032,4cyclictest16950irq/224-800000022:33:370
2015666997032,4cyclictest16950irq/224-800000022:33:370
2015666997028,10cyclictest2094700-21ssh23:58:020
2015666997026,14cyclictest2111027-21ssh00:35:330
2015666996820,11cyclictest2061099-21ssh22:40:120
2015666996430,13cyclictest2041225-21ssh21:54:320
201566699641,3cyclictest131rcu_preempt23:40:000
2015666996324,10cyclictest16950irq/224-800000023:55:010
2015666996226,12cyclictest2088939-21ssh23:44:550
2015666996226,12cyclictest2088939-21ssh23:44:540
201566699614,21cyclictest16850irq/134-800000021:19:280
201566699614,21cyclictest16850irq/134-800000021:19:270
2015666996126,13cyclictest2070771-21ssh23:02:480
2015666996034,4cyclictest16950irq/224-800000021:44:540
201566699603,3cyclictest131rcu_preempt19:10:030
201566699603,3cyclictest131rcu_preempt19:10:020
2015666996028,11cyclictest2028410-21ssh21:24:590
2015666995931,13cyclictest2084496-21ssh23:34:400
2015666995931,13cyclictest2084496-21ssh23:34:400
2015683995834,15cyclictest1-21systemd00:00:003
2015678995843,7cyclictest2001902-21kworker/2:222:46:472
201568399577,32cyclictest3467602-21systemd-journal21:20:013
201568399577,32cyclictest3467602-21systemd-journal21:20:003
2015683995743,11cyclictest3467602-21systemd-journal19:55:013
2015683995731,20cyclictest275-21dbus-daemon20:10:013
2015678995748,5cyclictest2034933-21kworker/2:323:47:272
201568399562,49cyclictest3467602-21systemd-journal19:35:013
201568399558,28cyclictest2021307-21modprobe21:00:003
2015683995530,18cyclictest275-21dbus-daemon23:40:013
2015678995547,5cyclictest2034933-21kworker/2:322:17:082
2015666995541,9cyclictest16950irq/224-800000000:30:020
2015666995536,14cyclictest275-21dbus-daemon19:20:010
201567199549,10cyclictest1-21systemd20:40:011
2015671995443,8cyclictest23-21ksoftirqd/123:10:181
2015666995446,6cyclictest2015674-21latency_hist19:10:020
201566699544,32cyclictest16850irq/134-800000023:26:560
201566699542,43cyclictest0-21swapper/022:04:030
2015683995333,14cyclictest1-21systemd22:50:013
2015666995328,9cyclictest2059473-21ssh22:36:330
2015666995327,10cyclictest2030521-21ssh21:29:460
2015666995326,13cyclictest2069014-21ssh22:58:560
2015666995324,13cyclictest2098824-21ssh00:07:310
2015683995231,15cyclictest2039330-21cut21:50:023
2015683995231,15cyclictest2039330-21cut21:50:023
2015678995143,5cyclictest2034933-21kworker/2:323:10:012
2015678995143,5cyclictest2001902-21kworker/2:223:01:332
2015678995128,8cyclictest2034933-21kworker/2:323:14:562
2015671995135,11cyclictest275-21dbus-daemon22:50:011
201566699511,35cyclictest2051283-21ssh22:17:470
201568399506,32cyclictest2021595-21munin-plugin-st21:04:593
201568399501,5cyclictest131rcu_preempt00:15:023
201567899504,4cyclictest131rcu_preempt23:36:482
2015678995042,5cyclictest2001902-21kworker/2:222:02:312
2015678995042,4cyclictest2034933-21kworker/2:323:15:122
2015678995031,12cyclictest275-21dbus-daemon00:00:002
2015666995037,8cyclictest2017694-21munin-plugin-st19:50:000
2015683994931,13cyclictest3467602-21systemd-journal23:55:013
2015678994942,5cyclictest2019282-21kworker/2:021:24:052
2015678994941,5cyclictest2034933-21kworker/2:323:55:012
2015678994941,5cyclictest2001902-21kworker/2:222:05:302
2015678994940,6cyclictest2034933-21kworker/2:300:02:372
2015678994940,6cyclictest2001902-21kworker/2:221:16:022
2015678994940,6cyclictest2001902-21kworker/2:221:16:012
2015678994930,14cyclictest275-21dbus-daemon20:25:012
2015666994924,10cyclictest2046748-21ssh22:07:100
2015666994923,11cyclictest2080231-21ssh23:24:530
201568399486,7cyclictest131rcu_preempt22:10:013
201568399485,28cyclictest2078133-21modprobe23:20:003
2015683994833,12cyclictest1-21systemd00:05:013
201568399482,7cyclictest37-21ksoftirqd/319:15:013
201568399482,7cyclictest37-21ksoftirqd/319:15:013
2015683994823,17cyclictest1-21systemd22:15:013
2015678994840,5cyclictest2019282-21kworker/2:021:30:432
2015678994840,5cyclictest2001902-21kworker/2:221:42:082
2015678994840,5cyclictest2001902-21kworker/2:221:38:182
201567899482,4cyclictest131rcu_preempt22:40:022
201567199484,11cyclictest0-21swapper/123:45:011
201567199484,11cyclictest0-21swapper/123:45:011
201567199482,5cyclictest131rcu_preempt23:25:001
2015671994821,5cyclictest23-21ksoftirqd/122:01:231
2015666994838,7cyclictest12-21ksoftirqd/022:24:410
2015666994821,12cyclictest2096932-21ssh00:03:090
2015666994817,9cyclictest12-21ksoftirqd/021:32:240
201568399478,5cyclictest131rcu_preempt19:45:013
201568399476,28cyclictest2020274-21modprobe20:40:013
2015683994733,9cyclictest309-21in:imuxsock19:25:013
2015683994733,9cyclictest2020555-21munin-plugin-st20:45:003
2015683994730,12cyclictest1-21systemd00:40:013
2015683994728,14cyclictest37-21ksoftirqd/321:10:003
2015683994728,14cyclictest37-21ksoftirqd/321:09:593
201568399472,4cyclictest131rcu_preempt21:40:013
2015683994722,19cyclictest361rcuc/321:32:053
2015683994712,23cyclictest2019517-21modprobe20:25:013
2015678994740,4cyclictest2099842-21kworker/2:200:36:182
2015678994740,4cyclictest2034933-21kworker/2:300:08:452
2015678994739,5cyclictest2078200-21kworker/2:023:27:272
2015678994738,5cyclictest2001902-21kworker/2:221:51:342
2015678994733,9cyclictest3467602-21systemd-journal21:05:012
2015678994712,5cyclictest131rcu_preempt23:30:362
2015678994712,5cyclictest131rcu_preempt23:30:352
2015671994731,12cyclictest275-21dbus-daemon20:25:011
2015671994716,8cyclictest2081382-21ssh23:27:161
2015671994711,6cyclictest131rcu_preempt21:08:321
2015671994711,6cyclictest131rcu_preempt21:08:321
201566699473,39cyclictest2021283-21kworker/u8:321:10:000
201566699473,39cyclictest2021283-21kworker/u8:321:10:000
201566699472,40cyclictest275-21dbus-daemon20:40:010
201568399469,4cyclictest131rcu_preempt22:32:193
201568399469,4cyclictest131rcu_preempt22:32:193
2015683994626,13cyclictest3467602-21systemd-journal00:20:013
2015683994624,16cyclictest3467602-21systemd-journal00:24:593
2015683994621,21cyclictest37-21ksoftirqd/322:59:423
201568399461,5cyclictest131rcu_preempt00:10:023
2015678994638,5cyclictest2034933-21kworker/2:323:23:242
2015678994636,6cyclictest2034933-21kworker/2:300:14:472
2015678994619,6cyclictest131rcu_preempt20:30:002
201567199468,5cyclictest131rcu_preempt20:27:141
201566699465,4cyclictest131rcu_preempt20:55:010
2015666994629,13cyclictest275-21dbus-daemon20:15:010
2015666994629,13cyclictest275-21dbus-daemon20:15:010
2015666994629,13cyclictest2100847-21ssh00:12:080
201566699462,7cyclictest12-21ksoftirqd/021:00:020
2015666994622,17cyclictest275-21dbus-daemon20:05:010
2015666994619,23cyclictest12-21ksoftirqd/023:15:550
2015666994610,6cyclictest131rcu_preempt20:19:590
2015683994530,12cyclictest37-21ksoftirqd/323:33:203
2015683994530,12cyclictest37-21ksoftirqd/323:33:193
2015683994529,11cyclictest37-21ksoftirqd/322:43:053
2015683994528,11cyclictest1-21systemd23:30:013
2015683994523,15cyclictest275-21dbus-daemon23:15:013
2015683994514,5cyclictest131rcu_preempt21:55:003
201567899457,6cyclictest131rcu_preempt22:40:032
2015678994538,5cyclictest2078200-21kworker/2:023:42:432
2015678994538,5cyclictest2078200-21kworker/2:023:42:422
2015678994537,5cyclictest2001902-21kworker/2:222:53:342
2015678994537,5cyclictest2001902-21kworker/2:219:47:022
2015678994536,5cyclictest2034933-21kworker/2:300:15:042
201567899451,3cyclictest131rcu_preempt00:30:152
2015671994532,10cyclictest3467602-21systemd-journal21:53:021
201567199451,3cyclictest131rcu_preempt22:34:031
201567199451,3cyclictest131rcu_preempt22:34:021
2015671994512,6cyclictest0-21swapper/123:03:471
2015666994518,12cyclictest2104857-21ssh00:21:160
201568399443,8cyclictest37-21ksoftirqd/320:35:003
2015683994428,12cyclictest37-21ksoftirqd/320:50:003
2015683994424,14cyclictest1-21systemd22:20:013
201568399441,5cyclictest131rcu_preempt22:25:093
2015683994410,5cyclictest131rcu_preempt20:00:003
2015678994436,5cyclictest2099842-21kworker/2:200:26:512
2015678994436,5cyclictest2001902-21kworker/2:222:58:002
2015678994431,10cyclictest3467602-21systemd-journal20:57:282
201567899442,11cyclictest2057828-21ssh22:32:542
201567899442,11cyclictest2057828-21ssh22:32:532
2015671994433,8cyclictest2084053-21sshd23:33:291
2015671994433,8cyclictest2084053-21sshd23:33:291
2015671994432,9cyclictest1-21systemd21:15:001
2015671994424,14cyclictest275-21dbus-daemon23:20:011
2015671994421,5cyclictest23-21ksoftirqd/123:56:391
201567199442,11cyclictest2096165-21ssh00:01:111
2015666994424,15cyclictest2017172-21systemctl19:40:000
2015683994326,11cyclictest2028450-21systemd-run21:25:013
2015683994324,14cyclictest1-21systemd21:45:003
201568399431,5cyclictest131rcu_preempt23:43:433
201568399431,5cyclictest131rcu_preempt23:43:433
201568399431,10cyclictest2069892-21ssh23:00:303
201567899438,6cyclictest131rcu_preempt19:15:002
201567899438,6cyclictest131rcu_preempt19:14:592
201567899434,4cyclictest131rcu_preempt22:28:442
201567899434,4cyclictest131rcu_preempt00:20:112
201567899433,4cyclictest131rcu_preempt20:00:032
2015678994333,8cyclictest30-21ksoftirqd/221:30:032
2015678994328,9cyclictest309-21in:imuxsock20:20:012
2015671994326,12cyclictest1-21systemd22:45:011
201567199431,3cyclictest131rcu_preempt21:41:041
201568399423,5cyclictest131rcu_preempt20:19:593
2015683994230,9cyclictest37-21ksoftirqd/320:54:293
2015683994227,12cyclictest37-21ksoftirqd/322:55:013
2015678994234,5cyclictest2001902-21kworker/2:221:46:522
2015678994234,5cyclictest2001902-21kworker/2:221:46:512
201567899421,4cyclictest0-21swapper/221:12:182
201567899421,3cyclictest131rcu_preempt22:25:012
201567199426,7cyclictest23-21ksoftirqd/121:24:491
2015671994230,9cyclictest23-21ksoftirqd/122:22:441
2015671994229,10cyclictest2110215-21sshd00:34:021
2015671994226,12cyclictest3467602-21systemd-journal19:19:591
2015671994226,11cyclictest309-21in:imuxsock00:20:001
2015671994224,14cyclictest275-21dbus-daemon00:30:001
201567199421,3cyclictest131rcu_preempt00:14:181
201568399418,6cyclictest131rcu_preempt21:56:293
201568399418,6cyclictest131rcu_preempt21:56:293
2015683994124,12cyclictest3467602-21systemd-journal20:05:013
2015683994118,16cyclictest275-21dbus-daemon20:15:013
2015683994118,16cyclictest275-21dbus-daemon20:15:003
2015683994113,10cyclictest37-21ksoftirqd/323:20:243
2015683994110,6cyclictest131rcu_preempt22:04:593
2015678994125,11cyclictest275-21dbus-daemon19:40:002
2015671994133,5cyclictest2091132-21sh23:49:591
2015671994130,8cyclictest2065918-21sshd22:51:271
2015671994129,9cyclictest3467602-21systemd-journal20:55:011
2015671994123,13cyclictest275-21dbus-daemon20:10:011
2015671994114,5cyclictest346771850lldpd19:40:091
201567199411,3cyclictest131rcu_preempt23:06:041
201567199411,3cyclictest131rcu_preempt21:18:391
201567199411,3cyclictest131rcu_preempt21:18:391
201566699414,24cyclictest2015567-21kworker/u8:219:30:000
2015666994134,4cyclictest16950irq/224-800000021:05:000
2015666994130,8cyclictest2019491-21sshd20:21:290
2015666994127,10cyclictest3467602-21systemd-journal19:40:260
201568399408,5cyclictest131rcu_preempt22:20:283
201568399404,27cyclictest1-21systemd19:40:013
201568399403,5cyclictest131rcu_preempt00:28:433
2015683994022,13cyclictest2016150-21tr19:19:593
2015683994010,6cyclictest131rcu_preempt23:50:013
2015678994029,8cyclictest309-21in:imuxsock20:30:032
2015678994027,10cyclictest30-21ksoftirqd/219:55:002
2015678994025,11cyclictest2016728-21sed19:30:022
201567199405,6cyclictest131rcu_preempt21:35:011
2015671994030,7cyclictest2018170-21sshd19:57:271
2015671994029,8cyclictest2091962-21sshd23:51:381
2015671994027,10cyclictest2052223-21wc22:20:011
2015671994020,5cyclictest23-21ksoftirqd/120:05:011
201567199401,4cyclictest131rcu_preempt22:29:101
201567199401,3cyclictest131rcu_preempt21:39:061
2015666994028,9cyclictest1-21systemd20:30:000
201566699402,6cyclictest275-21dbus-daemon20:35:010
201566699401,11cyclictest2018381-21cat20:00:030
2015683993926,10cyclictest37-21ksoftirqd/321:30:023
2015683993921,13cyclictest3467602-21systemd-journal19:30:003
2015683993915,6cyclictest131rcu_preempt21:10:203
2015683993911,7cyclictest131rcu_preempt00:34:433
2015678993931,5cyclictest2021060-21munin-plugin-st20:55:002
2015678993931,5cyclictest2001902-21kworker/2:222:10:152
2015678993927,8cyclictest275-21dbus-daemon20:05:012
2015678993926,9cyclictest30-21ksoftirqd/219:18:142
2015678993926,9cyclictest275-21dbus-daemon20:50:012
201567899392,3cyclictest131rcu_preempt21:56:422
201567899392,3cyclictest131rcu_preempt21:56:412
2015678993910,6cyclictest131rcu_preempt20:40:002
201567199399,5cyclictest131rcu_preempt19:50:211
2015671993931,5cyclictest2060767-21cron22:40:001
2015671993928,8cyclictest23-21ksoftirqd/122:12:001
2015671993927,9cyclictest23-21ksoftirqd/100:35:031
2015671993926,10cyclictest2016696-21sed19:30:011
2015671993925,10cyclictest3467602-21systemd-journal19:35:001
201567199392,3cyclictest131rcu_preempt20:31:281
2015671993920,14cyclictest1-21systemd21:00:021
201566699391,35cyclictest0-21swapper/019:35:000
201568399388,5cyclictest131rcu_preempt23:10:023
201568399381,5cyclictest131rcu_preempt20:30:013
201567199389,5cyclictest131rcu_preempt00:20:231
2015671993825,9cyclictest2017682-21cron19:49:591
2015671993823,11cyclictest23-21ksoftirqd/121:55:201
2015671993823,11cyclictest23-21ksoftirqd/121:55:191
2015671993817,6cyclictest131rcu_preempt22:59:191
201568399371,3cyclictest131rcu_preempt19:49:273
2015678993729,5cyclictest2018716-21cron20:09:592
2015678993727,6cyclictest2016929-21latency_hist19:35:022
2015678993725,9cyclictest30-21ksoftirqd/220:15:022
2015678993725,9cyclictest30-21ksoftirqd/220:15:012
2015678993725,9cyclictest30-21ksoftirqd/219:45:002
2015678993723,10cyclictest30-21ksoftirqd/219:25:002
201567199377,7cyclictest23-21ksoftirqd/120:13:271
201567199377,7cyclictest23-21ksoftirqd/120:13:261
201567199377,5cyclictest131rcu_preempt21:49:261
201567199377,5cyclictest131rcu_preempt21:49:251
201567199375,3cyclictest131rcu_preempt21:27:051
2015671993726,9cyclictest23-21ksoftirqd/119:40:021
2015671993726,8cyclictest2015844-21sshd19:14:271
2015671993726,8cyclictest2015844-21sshd19:14:261
2015671993726,8cyclictest1-21systemd20:50:001
2015671993725,9cyclictest23-21ksoftirqd/100:05:291
2015671993723,10cyclictest23-21ksoftirqd/122:05:491
2015671993723,10cyclictest2016385-21sshd19:24:281
2015671993720,4cyclictest23-21ksoftirqd/123:35:471
2015666993721,11cyclictest275-21dbus-daemon19:25:010
2015671993625,6cyclictest309-21in:imuxsock20:20:011
201566699362,9cyclictest3467602-21systemd-journal19:50:310
2015666993627,6cyclictest0-21swapper/020:45:000
2015666993623,9cyclictest12-21ksoftirqd/020:10:010
201566699361,3cyclictest131rcu_preempt20:50:020
201567899353,3cyclictest131rcu_preempt19:10:022
2015678993520,11cyclictest2020542-21sshd20:44:292
2015671993521,10cyclictest1-21systemd21:05:001
2015671993418,12cyclictest2020567-21sh20:45:011
2015683993222,8cyclictest37-21ksoftirqd/319:10:013
201567199274,17cyclictest2015674-21latency_hist19:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional