You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-01 - 21:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Jun 01, 2026 12:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2226147997820,53cyclictest12-21ksoftirqd/010:00:000
2226162997732,12cyclictest1-21systemd09:15:013
2226155997625,10cyclictest275-21dbus-daemon11:00:002
2226155997224,12cyclictest3467602-21systemd-journal07:55:002
2226162997163,3cyclictest37-21ksoftirqd/312:20:013
2226155996923,12cyclictest2272942-21cron10:39:592
2226147996735,6cyclictest131rcu_preempt11:20:010
2226147996612,44cyclictest12-21ksoftirqd/011:46:410
222615599651,3cyclictest131rcu_preempt07:50:012
2226147996517,39cyclictest12-21ksoftirqd/012:36:390
222614799635,53cyclictest2268399-21latency_hist10:30:010
2226147996354,5cyclictest16950irq/224-800000007:10:010
222615599627,39cyclictest291rcuc/212:35:032
2226147996224,33cyclictest2286602-21latency_hist11:10:030
222616299612,20cyclictest2225667-21kworker/u8:107:20:023
222614799617,15cyclictest2230762-21modprobe08:40:000
2226147996129,16cyclictest2284081-21ssh11:04:440
222614799612,31cyclictest131rcu_preempt12:30:010
2226147996119,17cyclictest16850irq/134-800000010:52:310
2226147996116,30cyclictest12-21ksoftirqd/009:54:260
2226147996116,30cyclictest12-21ksoftirqd/009:54:260
2226147996114,40cyclictest12-21ksoftirqd/010:40:000
2226147996022,34cyclictest2259363-21cron10:09:590
2226155995934,8cyclictest2266189-21kworker/2:212:06:002
2226147995854,2cyclictest0-21swapper/007:45:020
2226147995854,2cyclictest0-21swapper/007:45:020
2226147995818,6cyclictest16850irq/134-800000012:05:180
2226147995818,5cyclictest16850irq/134-800000012:13:030
2226147995818,31cyclictest16850irq/134-800000011:52:270
2226147995818,31cyclictest16850irq/134-800000010:11:150
2226147995818,31cyclictest16850irq/134-800000009:20:350
2226147995817,17cyclictest16850irq/134-800000010:47:500
222614799576,32cyclictest12-21ksoftirqd/008:20:010
2226147995720,9cyclictest12-21ksoftirqd/011:34:290
2226147995719,32cyclictest0-21swapper/010:30:520
2226147995719,32cyclictest0-21swapper/010:30:520
2226147995718,6cyclictest16850irq/134-800000011:44:090
2226147995649,4cyclictest16950irq/224-800000011:10:030
2226147995634,17cyclictest2266196-21ssh10:25:020
222616299552,4cyclictest131rcu_preempt11:00:023
2226147995535,14cyclictest3467602-21systemd-journal08:00:000
2226147995535,14cyclictest3467602-21systemd-journal08:00:000
222614799553,15cyclictest2226614-21cron07:20:000
2226147995528,12cyclictest2323373-21ssh12:32:040
2226147995519,18cyclictest16850irq/134-800000011:24:050
2226147995518,6cyclictest16850irq/134-800000010:42:360
2226147995518,6cyclictest16850irq/134-800000010:42:360
2226147995518,5cyclictest2257205-21latency_hist10:05:020
2226147995517,18cyclictest16850irq/134-800000011:27:260
2226147995517,18cyclictest16850irq/134-800000009:38:290
2226147995418,6cyclictest16850irq/134-800000009:34:040
222616299532,4cyclictest131rcu_preempt11:57:443
2226147995349,2cyclictest0-21swapper/008:55:020
2226147995349,2cyclictest0-21swapper/008:55:020
222614799534,42cyclictest2236878-21sendmail09:20:000
222614799533,18cyclictest3467602-21systemd-journal12:01:080
222614799533,18cyclictest3467602-21systemd-journal12:01:080
2226162995236,12cyclictest1-21systemd08:30:013
2226162995236,12cyclictest1-21systemd08:30:013
2226155995243,6cyclictest2266189-21kworker/2:212:14:132
222615099529,6cyclictest131rcu_preempt10:20:011
2226150995234,14cyclictest1-21systemd11:35:011
2226147995232,4cyclictest12-21ksoftirqd/010:57:360
2226147995218,6cyclictest16850irq/134-800000010:17:550
2226147995217,7cyclictest16850irq/134-800000009:47:150
2226147995217,6cyclictest16850irq/134-800000009:42:290
2226147995216,6cyclictest16850irq/134-800000009:11:540
2226155995143,5cyclictest2266189-21kworker/2:210:47:492
2226155995143,5cyclictest2254884-21kworker/2:110:07:562
2226155995142,5cyclictest2229015-21kworker/2:010:28:192
2226155995134,12cyclictest30-21ksoftirqd/208:35:022
222614799515,40cyclictest2232338-21munin-run09:10:000
2226147995126,9cyclictest2239708-21ssh09:25:500
222614799512,12cyclictest2227693-21latency_hist07:40:010
2226147995118,16cyclictest16850irq/134-800000012:24:170
2226147995117,17cyclictest16850irq/134-800000011:59:520
2226155995043,4cyclictest2229015-21kworker/2:010:13:132
2226155995042,5cyclictest2286451-21kworker/2:011:47:202
2226155995041,6cyclictest2254884-21kworker/2:111:10:482
2226155995041,6cyclictest2254884-21kworker/2:111:05:122
2226155995041,5cyclictest2286451-21kworker/2:012:01:092
2226155995041,5cyclictest2286451-21kworker/2:012:01:082
222616299491,6cyclictest0-21swapper/312:13:253
222616299491,5cyclictest0-21swapper/310:14:303
2226155994941,5cyclictest2229015-21kworker/2:008:30:022
2226155994941,5cyclictest2229015-21kworker/2:008:30:022
2226155994940,6cyclictest2266189-21kworker/2:212:16:242
2226155994935,10cyclictest1-21systemd12:30:002
222615599492,7cyclictest131rcu_preempt08:15:032
2226150994934,11cyclictest3467602-21systemd-journal10:35:011
2226150994934,11cyclictest3467602-21systemd-journal10:35:011
2226150994932,13cyclictest3467602-21systemd-journal10:25:011
222614799496,14cyclictest2299962-21cat11:40:020
2226147994933,12cyclictest3467602-21systemd-journal07:15:000
2226147994916,17cyclictest16850irq/134-800000012:19:570
2226162994831,14cyclictest1-21systemd09:30:013
2226155994841,4cyclictest2229015-21kworker/2:009:19:142
2226155994840,5cyclictest2254884-21kworker/2:110:40:082
2226155994840,5cyclictest2254884-21kworker/2:110:40:072
2226155994840,5cyclictest2229015-21kworker/2:010:23:552
2226155994840,5cyclictest2229015-21kworker/2:009:32:152
2226155994839,6cyclictest2286451-21kworker/2:012:23:142
2226155994839,6cyclictest2218190-21kworker/2:007:20:022
2226155994829,14cyclictest275-21dbus-daemon09:00:002
222615099487,9cyclictest131rcu_preempt12:00:001
222616299473,4cyclictest131rcu_preempt11:27:313
2226162994711,6cyclictest2232328-21diskmemload10:03:353
2226155994740,5cyclictest2229015-21kworker/2:009:39:442
2226155994739,5cyclictest2266189-21kworker/2:211:31:302
2226155994739,5cyclictest2234975-21kworker/2:209:50:222
2226155994739,5cyclictest2234975-21kworker/2:209:50:222
2226150994733,10cyclictest3467602-21systemd-journal11:53:231
222615099472,8cyclictest2248119-21modprobe09:45:001
222614799471,43cyclictest0-21swapper/008:05:020
2226162994629,13cyclictest3467602-21systemd-journal09:55:013
2226162994629,13cyclictest3467602-21systemd-journal09:55:013
222616299462,7cyclictest2300589-21ssh11:41:063
222616299462,4cyclictest131rcu_preempt12:21:463
222616299462,12cyclictest2312233-21ssh12:07:193
2226155994638,6cyclictest2266189-21kworker/2:211:38:472
2226155994638,5cyclictest2266189-21kworker/2:211:55:272
2226155994637,5cyclictest2286451-21kworker/2:012:39:452
2226150994636,7cyclictest2315709-21grep12:15:031
222615099461,3cyclictest131rcu_preempt07:45:021
222615099461,3cyclictest131rcu_preempt07:45:021
222614799463,10cyclictest3467602-21systemd-journal07:30:000
222614799462,10cyclictest2230465-21latency_hist08:30:030
222616299454,7cyclictest37-21ksoftirqd/311:33:293
222616299454,4cyclictest131rcu_preempt11:49:293
222616299454,4cyclictest131rcu_preempt10:41:403
222616299454,4cyclictest131rcu_preempt10:41:393
222616299454,4cyclictest131rcu_preempt10:18:513
2226162994533,9cyclictest1-21systemd11:15:013
222616299452,4cyclictest131rcu_preempt10:37:203
222616299451,11cyclictest2249592-21ssh09:48:043
2226155994537,5cyclictest2254884-21kworker/2:110:34:082
2226155994537,5cyclictest2254884-21kworker/2:110:34:072
2226155994537,5cyclictest2254884-21kworker/2:110:05:012
222615599451,6cyclictest0-21swapper/209:13:012
222615099456,4cyclictest131rcu_preempt09:55:021
222615099456,4cyclictest131rcu_preempt09:55:021
2226150994533,8cyclictest23-21ksoftirqd/110:40:011
2226150994531,11cyclictest1-21systemd11:40:021
2226150994529,11cyclictest2290926-21systemctl11:20:001
2226150994527,11cyclictest275-21dbus-daemon10:10:001
222615099452,3cyclictest131rcu_preempt09:31:161
222616299443,4cyclictest131rcu_preempt09:15:163
222616299443,3cyclictest150-21pr/ttyS209:34:023
222616299443,3cyclictest131rcu_preempt11:35:213
2226162994430,10cyclictest275-21dbus-daemon08:05:013
222616299442,4cyclictest131rcu_preempt11:04:133
222616299441,8cyclictest275-21dbus-daemon09:25:013
222616299441,4cyclictest131rcu_preempt11:08:003
222616299441,4cyclictest131rcu_preempt10:23:573
2226155994436,5cyclictest2286451-21kworker/2:011:20:142
2226155994436,5cyclictest2286451-21kworker/2:011:19:562
2226155994436,5cyclictest2254884-21kworker/2:111:02:342
2226155994410,5cyclictest131rcu_preempt11:54:182
222615099446,4cyclictest131rcu_preempt11:45:001
222615099443,3cyclictest131rcu_preempt08:55:001
222615099443,3cyclictest131rcu_preempt08:55:001
2226150994432,9cyclictest2291320-21sshd11:20:231
2226150994428,13cyclictest23-21ksoftirqd/112:40:011
2226150994427,12cyclictest275-21dbus-daemon12:30:001
2226150994423,15cyclictest3467602-21systemd-journal07:55:001
2226150994423,15cyclictest275-21dbus-daemon11:30:021
2226147994429,10cyclictest275-21dbus-daemon08:25:010
222616299436,9cyclictest2306146-21sh11:53:473
222616299435,6cyclictest37-21ksoftirqd/309:10:033
222616299433,5cyclictest0-21swapper/309:35:073
222616299433,3cyclictest131rcu_preempt10:48:273
222616299433,3cyclictest131rcu_preempt08:53:183
222616299433,3cyclictest131rcu_preempt08:53:183
222616299432,4cyclictest131rcu_preempt12:03:503
222616299432,4cyclictest131rcu_preempt12:03:503
2226155994335,5cyclictest2229015-21kworker/2:009:47:142
2226155994335,5cyclictest2225321-21kworker/2:207:45:022
2226155994335,5cyclictest2225321-21kworker/2:207:45:022
2226155994331,8cyclictest30-21ksoftirqd/210:17:282
2226155994325,12cyclictest275-21dbus-daemon07:40:002
222615599432,3cyclictest131rcu_preempt07:10:012
2226150994335,5cyclictest2236861-21latency_hist09:20:001
222615099432,8cyclictest23-21ksoftirqd/109:10:031
2226150994325,12cyclictest309-21in:imuxsock08:25:011
222615099431,3cyclictest131rcu_preempt11:50:021
222615099431,3cyclictest131rcu_preempt09:45:121
2226147994336,4cyclictest16950irq/224-800000008:45:010
222614799433,35cyclictest3468525-21sendmail-mta08:49:520
2226147994328,9cyclictest309-21in:imuxsock07:45:030
222616299425,3cyclictest150-21pr/ttyS212:27:383
222616299424,8cyclictest0-21swapper/310:26:363
222616299424,3cyclictest131rcu_preempt09:56:153
2226162994229,9cyclictest1-21systemd08:40:013
2226162994226,12cyclictest37-21ksoftirqd/310:34:543
2226162994226,12cyclictest37-21ksoftirqd/310:34:543
222616299421,4cyclictest131rcu_preempt09:42:163
222615599427,5cyclictest131rcu_preempt09:25:432
2226155994233,6cyclictest2266189-21kworker/2:211:29:412
2226155994233,5cyclictest309-21in:imuxsock08:00:012
2226155994233,5cyclictest309-21in:imuxsock08:00:012
222615599423,10cyclictest2246325-21ssh09:40:402
2226150994230,9cyclictest23-21ksoftirqd/109:26:061
2226150994229,10cyclictest3467602-21systemd-journal08:17:441
2226150994229,10cyclictest23-21ksoftirqd/110:05:011
222616299411,3cyclictest131rcu_preempt11:21:383
222615599413,4cyclictest131rcu_preempt07:35:022
222615599413,4cyclictest131rcu_preempt07:35:022
222615599411,6cyclictest0-21swapper/209:24:232
222615599411,3cyclictest131rcu_preempt11:45:022
2226150994129,10cyclictest23-21ksoftirqd/111:11:451
2226150994128,10cyclictest3467602-21systemd-journal07:46:441
2226150994126,11cyclictest275-21dbus-daemon08:10:011
2226150994126,11cyclictest2231801-21cat09:00:001
2226150994126,10cyclictest309-21in:imuxsock07:25:001
2226150994125,12cyclictest23-21ksoftirqd/108:00:031
2226150994125,12cyclictest23-21ksoftirqd/108:00:021
2226150994125,11cyclictest309-21in:imuxsock10:55:011
222615099411,3cyclictest131rcu_preempt12:00:411
222615099411,3cyclictest131rcu_preempt12:00:401
2226147994126,7cyclictest309-21in:imuxsock07:35:010
2226147994126,7cyclictest309-21in:imuxsock07:35:010
222616299404,7cyclictest37-21ksoftirqd/308:50:023
2226162994024,12cyclictest2228313-21wc07:50:013
222616299401,6cyclictest131rcu_preempt10:53:023
222616299401,5cyclictest2324045-21ssh12:33:433
2226155994033,5cyclictest2229015-21kworker/2:008:15:242
222615599401,6cyclictest0-21swapper/210:54:362
2226150994030,5cyclictest309-21in:imuxsock08:30:011
2226150994030,5cyclictest309-21in:imuxsock08:30:011
2226150994025,11cyclictest3467602-21systemd-journal10:45:021
222615099401,4cyclictest131rcu_preempt10:28:041
2226150994011,21cyclictest2284189-21modprobe11:05:001
222616299396,3cyclictest131rcu_preempt12:35:183
222616299395,6cyclictest37-21ksoftirqd/310:08:513
2226162993923,12cyclictest37-21ksoftirqd/311:16:073
2226155993931,5cyclictest2229015-21kworker/2:008:40:012
2226155993931,5cyclictest2218190-21kworker/2:007:25:022
222615599392,4cyclictest131rcu_preempt08:10:022
222615099398,6cyclictest131rcu_preempt12:09:441
222615099394,3cyclictest131rcu_preempt12:24:471
2226150993928,8cyclictest2231018-21sshd08:42:451
2226150993927,9cyclictest23-21ksoftirqd/112:15:041
2226150993927,9cyclictest23-21ksoftirqd/111:10:021
2226150993927,8cyclictest275-21dbus-daemon07:40:011
2226150993926,10cyclictest23-21ksoftirqd/110:55:051
222615099392,3cyclictest131rcu_preempt09:21:341
222615099391,3cyclictest131rcu_preempt09:40:001
222615099391,3cyclictest131rcu_preempt07:10:011
2226147993928,8cyclictest12-21ksoftirqd/009:05:030
2226147993911,20cyclictest2230260-21munin-run08:30:000
2226162993823,11cyclictest309-21in:imuxsock07:30:013
222615599383,3cyclictest131rcu_preempt08:55:022
222615599383,3cyclictest131rcu_preempt08:55:022
2226155993829,5cyclictest309-21in:imuxsock08:25:012
2226150993826,7cyclictest309-21in:imuxsock08:50:001
2226150993825,10cyclictest23-21ksoftirqd/108:05:001
222615099382,4cyclictest131rcu_preempt09:58:531
2226150993824,10cyclictest23-21ksoftirqd/110:45:001
2226150993824,10cyclictest23-21ksoftirqd/110:45:001
222615099382,3cyclictest131rcu_preempt07:30:021
2226150993822,12cyclictest23-21ksoftirqd/107:11:421
2226147993826,9cyclictest12-21ksoftirqd/008:10:000
2226147993816,3cyclictest12-21ksoftirqd/008:15:030
222616299375,3cyclictest131rcu_preempt08:15:013
2226162993724,9cyclictest2227632-21cron07:40:003
2226155993727,7cyclictest30-21ksoftirqd/208:45:002
222615599372,6cyclictest131rcu_preempt09:10:012
2226155993725,9cyclictest2231421-21cut08:50:022
2226155993725,10cyclictest30-21ksoftirqd/209:56:172
2226150993727,8cyclictest23-21ksoftirqd/112:30:571
2226150993724,9cyclictest23-21ksoftirqd/107:32:351
2226150993724,9cyclictest23-21ksoftirqd/107:32:351
2226150993723,10cyclictest275-21dbus-daemon08:40:011
222614799374,4cyclictest131rcu_preempt07:55:000
222614799372,4cyclictest2231790-21modprobe09:00:000
2226147993718,15cyclictest275-21dbus-daemon07:25:000
222616299362,4cyclictest131rcu_preempt07:20:033
2226162993624,9cyclictest37-21ksoftirqd/307:35:023
2226162993624,9cyclictest37-21ksoftirqd/307:35:023
222616299361,5cyclictest131rcu_preempt07:45:003
222616299361,5cyclictest131rcu_preempt07:45:003
222615599361,3cyclictest131rcu_preempt07:30:012
222615099364,3cyclictest131rcu_preempt10:14:231
2226150993627,7cyclictest23-21ksoftirqd/108:35:031
2226150993625,8cyclictest23-21ksoftirqd/107:20:021
2226150993623,10cyclictest2232089-21pluginstate09:05:011
2226150993621,11cyclictest3467602-21systemd-journal09:11:021
222614799363,9cyclictest0-21swapper/008:32:050
222616299353,3cyclictest131rcu_preempt08:42:453
2226162993520,11cyclictest37-21ksoftirqd/307:15:013
2226162993519,12cyclictest275-21dbus-daemon09:05:013
222616299351,6cyclictest0-21swapper/308:10:023
222615599352,5cyclictest131rcu_preempt07:12:342
2226162993419,12cyclictest275-21dbus-daemon08:00:003
2226162993419,12cyclictest275-21dbus-daemon08:00:003
2226155993426,4cyclictest2232072-21latency_hist09:05:022
2226155993418,11cyclictest309-21in:imuxsock08:05:002
2226150993423,8cyclictest2229436-21sshd08:11:441
2226162993326,4cyclictest2231775-21sshd08:59:183
2226162993319,10cyclictest2230114-21cut08:25:023
2226162993219,10cyclictest2229744-21sendmail08:20:003
2226162993120,8cyclictest1-21systemd08:35:003
222616299311,3cyclictest131rcu_preempt07:10:013
222616299301,4cyclictest131rcu_preempt07:55:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional