You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 08:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Mar 26, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1925161998662,6cyclictest23-21ksoftirqd/100:30:031
1925158997049,11cyclictest1937417-21sh21:25:030
1925158996945,15cyclictest1-21systemd22:10:020
192516199676,24cyclictest131rcu_preempt21:50:021
1925166996540,16cyclictest1964409-21kworker/2:122:35:042
1925168996227,7cyclictest37-21ksoftirqd/323:12:403
1925168996216,8cyclictest37-21ksoftirqd/300:09:233
1925166996213,18cyclictest131rcu_preempt19:35:012
1925158996231,7cyclictest1956606-21ssh22:10:510
1925168996111,11cyclictest37-21ksoftirqd/320:50:003
1925168996111,11cyclictest37-21ksoftirqd/320:50:003
1925168996110,9cyclictest37-21ksoftirqd/321:10:023
1925168996110,9cyclictest37-21ksoftirqd/321:10:023
1925166996141,7cyclictest1964409-21kworker/2:122:56:212
192515899601,38cyclictest131rcu_preempt22:02:430
192516899599,12cyclictest37-21ksoftirqd/323:31:013
192516899594,12cyclictest37-21ksoftirqd/300:28:283
1925168995915,7cyclictest37-21ksoftirqd/322:10:023
1925168995914,8cyclictest37-21ksoftirqd/300:35:493
1925166995938,7cyclictest1951994-21kworker/2:023:54:292
1925166995925,6cyclictest131rcu_preempt19:45:012
1925158995931,9cyclictest12-21ksoftirqd/023:30:030
1925168995815,8cyclictest37-21ksoftirqd/322:05:023
1925168995811,10cyclictest37-21ksoftirqd/321:28:033
1925166995835,6cyclictest1964409-21kworker/2:122:43:402
1925158995828,26cyclictest16950irq/224-800000021:15:010
1925168995734,14cyclictest1968532-21cron22:40:003
1925168995714,8cyclictest37-21ksoftirqd/321:21:343
1925168995711,9cyclictest37-21ksoftirqd/322:15:023
1925166995737,14cyclictest1-21systemd21:20:012
1925161995727,17cyclictest23-21ksoftirqd/100:22:161
1925158995729,13cyclictest1951556-21ssh21:59:040
1925158995727,15cyclictest1938903-21ssh21:28:370
192516899569,11cyclictest37-21ksoftirqd/300:15:303
192516899562,10cyclictest1973366-21ssh22:51:003
1925168995617,9cyclictest37-21ksoftirqd/323:00:023
1925168995615,8cyclictest37-21ksoftirqd/321:44:533
1925166995624,8cyclictest1951994-21kworker/2:022:30:082
1925158995626,3cyclictest131rcu_preempt21:19:030
192516899559,6cyclictest37-21ksoftirqd/300:11:433
192516899557,9cyclictest2000978-21ssh23:57:393
192516899557,9cyclictest2000978-21ssh23:57:393
192516899554,11cyclictest37-21ksoftirqd/321:36:063
192516699558,8cyclictest30-21ksoftirqd/200:40:012
1925166995531,9cyclictest1-21systemd21:10:012
1925166995531,9cyclictest1-21systemd21:10:012
1925161995535,15cyclictest275-21dbus-daemon22:40:001
192515899559,33cyclictest19350irq/34-mmc120:15:020
1925158995526,13cyclictest1945537-21ssh21:44:350
1925158995524,14cyclictest1983144-21ssh23:15:000
1925158995519,32cyclictest16850irq/134-800000023:52:250
192515899551,35cyclictest131rcu_preempt22:21:530
192516899548,10cyclictest37-21ksoftirqd/300:32:083
192516899545,12cyclictest37-21ksoftirqd/321:14:183
192516899545,10cyclictest37-21ksoftirqd/323:40:253
192516899544,13cyclictest37-21ksoftirqd/323:39:353
1925168995436,10cyclictest309-21in:imuxsock23:50:023
1925168995436,10cyclictest309-21in:imuxsock23:50:013
1925168995419,8cyclictest2000001-21rm23:55:023
1925168995415,6cyclictest0-21swapper/323:15:223
1925168995413,8cyclictest37-21ksoftirqd/319:45:013
1925168995410,10cyclictest37-21ksoftirqd/321:50:003
1925166995445,6cyclictest1951994-21kworker/2:023:22:522
192515899543,3cyclictest131rcu_preempt23:28:400
192515899543,3cyclictest131rcu_preempt23:09:300
1925158995418,31cyclictest16850irq/134-800000000:17:110
1925158995417,32cyclictest16850irq/134-800000021:39:320
192516899537,9cyclictest37-21ksoftirqd/321:53:173
1925168995315,6cyclictest37-21ksoftirqd/323:09:363
1925166995345,5cyclictest1951994-21kworker/2:022:53:512
1925166995345,5cyclictest1928215-21kworker/2:121:39:572
1925166995344,6cyclictest1964409-21kworker/2:123:16:382
1925166995344,6cyclictest1928215-21kworker/2:121:20:092
1925166995337,5cyclictest1928215-21kworker/2:120:45:022
1925161995325,12cyclictest1-21systemd19:50:001
192515899534,7cyclictest131rcu_preempt22:35:000
192515899534,35cyclictest12-21ksoftirqd/023:39:410
192515899531,37cyclictest1931323-21diskmemload00:40:020
192516899528,12cyclictest1935279-21latency_hist21:20:023
192516899527,9cyclictest37-21ksoftirqd/322:17:213
1925166995244,5cyclictest1951994-21kworker/2:000:03:402
1925166995243,5cyclictest1951994-21kworker/2:023:59:312
1925166995243,5cyclictest1951994-21kworker/2:023:59:312
1925166995242,6cyclictest1964409-21kworker/2:123:13:022
1925166995235,7cyclictest1928215-21kworker/2:121:12:142
1925166995217,10cyclictest30-21ksoftirqd/220:20:012
192516199527,29cyclictest0-21swapper/119:30:021
1925158995233,15cyclictest16850irq/134-800000023:50:000
1925158995233,15cyclictest16850irq/134-800000023:50:000
1925158995222,15cyclictest1940472-21ssh21:32:140
1925158995220,6cyclictest12-21ksoftirqd/000:31:480
1925158995219,28cyclictest12-21ksoftirqd/023:23:570
1925158995218,30cyclictest16850irq/134-800000000:07:200
1925158995217,31cyclictest16850irq/134-800000021:48:150
1925158995217,31cyclictest16850irq/134-800000000:24:430
1925158995217,19cyclictest16850irq/134-800000000:10:020
192516899519,26cyclictest1929746-21modprobe20:40:013
192516899518,10cyclictest37-21ksoftirqd/323:20:243
192516899516,8cyclictest37-21ksoftirqd/322:25:333
192516899515,6cyclictest37-21ksoftirqd/322:42:003
192516899515,10cyclictest37-21ksoftirqd/320:10:023
192516899514,7cyclictest37-21ksoftirqd/323:26:553
1925168995119,10cyclictest37-21ksoftirqd/322:30:053
1925168995118,8cyclictest37-21ksoftirqd/322:24:543
1925168995115,9cyclictest37-21ksoftirqd/300:05:013
1925168995113,8cyclictest37-21ksoftirqd/322:45:363
1925166995143,5cyclictest1964409-21kworker/2:122:47:092
1925166995143,5cyclictest1951994-21kworker/2:023:30:162
1925166995143,5cyclictest1951994-21kworker/2:000:19:492
1925166995143,5cyclictest1941530-21kworker/2:321:47:482
1925166995143,5cyclictest1928215-21kworker/2:121:41:142
1925166995143,5cyclictest1928215-21kworker/2:121:26:172
1925166995142,6cyclictest1941530-21kworker/2:322:09:282
1925166995142,5cyclictest1911102-21kworker/2:320:05:012
1925166995141,7cyclictest1951994-21kworker/2:023:49:562
1925166995141,7cyclictest1951994-21kworker/2:023:49:562
1925166995140,7cyclictest30-21ksoftirqd/200:22:572
1925161995138,9cyclictest275-21dbus-daemon22:35:001
1925158995124,12cyclictest1967168-21ssh22:36:230
1925158995118,19cyclictest16850irq/134-800000022:55:500
1925158995117,31cyclictest16850irq/134-800000000:29:280
1925158995116,31cyclictest16850irq/134-800000022:26:200
1925158995116,31cyclictest16850irq/134-800000000:04:210
192515899511,35cyclictest1969887-21ssh22:42:560
192516899504,12cyclictest37-21ksoftirqd/323:00:063
1925168995040,8cyclictest37-21ksoftirqd/321:00:013
1925168995030,16cyclictest275-21dbus-daemon21:35:013
1925168995014,9cyclictest37-21ksoftirqd/300:23:533
192516899501,31cyclictest0-21swapper/320:00:023
1925166995043,4cyclictest1951994-21kworker/2:023:39:352
1925166995042,5cyclictest2002152-21kworker/2:300:29:332
1925166995042,5cyclictest1964409-21kworker/2:123:08:082
1925166995042,5cyclictest1951994-21kworker/2:023:42:402
1925166995042,5cyclictest1951994-21kworker/2:023:25:562
1925166995042,5cyclictest1951994-21kworker/2:000:09:082
1925166995042,5cyclictest1941530-21kworker/2:322:21:362
1925166995042,5cyclictest1941530-21kworker/2:322:12:592
1925166995042,5cyclictest1941530-21kworker/2:321:53:572
192516199506,6cyclictest131rcu_preempt00:15:031
1925161995033,12cyclictest1-21systemd00:10:011
1925161995032,13cyclictest1956124-21munin-run22:10:011
1925158995017,30cyclictest16850irq/134-800000023:18:500
1925158995017,30cyclictest16850irq/134-800000022:18:250
1925158995017,18cyclictest16850irq/134-800000022:47:240
1925158995016,30cyclictest16850irq/134-800000022:50:230
1925158995015,31cyclictest16850irq/134-800000023:41:060
1925166994942,4cyclictest1941530-21kworker/2:321:57:392
1925166994942,4cyclictest1924348-21kworker/2:221:30:122
1925166994941,5cyclictest2002152-21kworker/2:300:31:462
1925166994941,5cyclictest1964409-21kworker/2:123:01:542
1925166994941,5cyclictest1941530-21kworker/2:322:17:162
1925166994940,6cyclictest2002152-21kworker/2:300:13:162
1925166994940,6cyclictest1951994-21kworker/2:022:26:342
1925166994940,6cyclictest1941530-21kworker/2:322:00:322
192516199494,4cyclictest131rcu_preempt21:25:341
1925161994933,12cyclictest1-21systemd23:50:011
1925161994933,12cyclictest1-21systemd23:50:011
1925161994931,14cyclictest1-21systemd22:05:011
1925161994923,21cyclictest23-21ksoftirqd/100:38:501
1925158994924,10cyclictest1977531-21ssh23:01:040
192515899491,44cyclictest1949486-21rm21:54:020
192515899491,38cyclictest19350irq/34-mmc120:20:020
1925158994913,32cyclictest16850irq/134-800000023:59:200
1925158994913,32cyclictest16850irq/134-800000023:59:190
192516899489,8cyclictest37-21ksoftirqd/320:45:013
192516899489,10cyclictest37-21ksoftirqd/321:57:203
1925168994830,14cyclictest37-21ksoftirqd/321:05:013
1925161994827,8cyclictest23-21ksoftirqd/100:33:461
1925161994824,19cyclictest23-21ksoftirqd/121:10:211
1925168994737,7cyclictest37-21ksoftirqd/320:25:023
1925166994740,4cyclictest1911102-21kworker/2:319:57:012
1925166994739,5cyclictest1928215-21kworker/2:120:35:112
1925166994739,5cyclictest1928215-21kworker/2:120:26:112
1925166994739,5cyclictest1928215-21kworker/2:120:26:112
1925166994739,5cyclictest1928215-21kworker/2:120:23:222
192516199474,4cyclictest131rcu_preempt21:56:571
1925161994723,20cyclictest23-21ksoftirqd/120:40:021
1925161994722,5cyclictest23-21ksoftirqd/100:00:011
1925161994722,5cyclictest23-21ksoftirqd/100:00:011
1925161994721,21cyclictest23-21ksoftirqd/123:10:511
1925161994721,16cyclictest23-21ksoftirqd/121:20:061
1925161994710,5cyclictest131rcu_preempt20:50:001
1925161994710,5cyclictest131rcu_preempt20:50:001
192515899475,37cyclictest0-21swapper/019:45:020
192515899471,42cyclictest1928219-21munin-run20:10:010
192516899466,10cyclictest37-21ksoftirqd/319:25:013
192516899465,6cyclictest37-21ksoftirqd/320:15:023
192516899463,5cyclictest37-21ksoftirqd/319:10:023
192516899463,5cyclictest37-21ksoftirqd/319:10:023
1925168994624,14cyclictest1925611-21systemctl19:20:003
1925168994613,4cyclictest0-21swapper/319:55:013
1925166994638,5cyclictest1928215-21kworker/2:121:03:012
1925166994638,5cyclictest1928215-21kworker/2:120:31:162
1925166994631,11cyclictest3467602-21systemd-journal19:17:002
192516199465,4cyclictest150-21pr/ttyS223:40:561
192516199463,3cyclictest131rcu_preempt21:37:211
1925161994623,19cyclictest23-21ksoftirqd/119:20:011
1925161994612,5cyclictest131rcu_preempt00:15:161
1925158994630,13cyclictest275-21dbus-daemon20:35:010
192515899462,40cyclictest1925948-21latency_hist19:25:020
192516899456,9cyclictest37-21ksoftirqd/320:05:013
192516899456,11cyclictest37-21ksoftirqd/320:55:023
1925166994537,5cyclictest1924348-21kworker/2:220:09:332
1925166994537,5cyclictest1911102-21kworker/2:319:24:142
192516199455,5cyclictest131rcu_preempt21:10:011
192516199455,5cyclictest131rcu_preempt21:10:011
1925161994532,10cyclictest1-21systemd20:15:011
1925161994530,11cyclictest23-21ksoftirqd/123:05:001
1925161994522,20cyclictest23-21ksoftirqd/119:45:011
1925158994531,10cyclictest1929742-21cron20:40:010
1925158994529,12cyclictest275-21dbus-daemon19:55:010
1925158994528,12cyclictest1925481-21latency_hist19:15:020
192515899452,5cyclictest1927171-21modprobe19:50:000
1925168994430,11cyclictest37-21ksoftirqd/319:40:013
1925168994423,15cyclictest1-21systemd20:20:003
1925166994436,5cyclictest1928215-21kworker/2:120:57:572
1925166994436,5cyclictest1928215-21kworker/2:120:50:472
1925166994436,5cyclictest1911102-21kworker/2:319:49:312
1925166994436,5cyclictest1911102-21kworker/2:319:39:202
1925166994436,5cyclictest1911102-21kworker/2:319:26:462
1925166994436,5cyclictest1911102-21kworker/2:319:13:392
192516199443,4cyclictest131rcu_preempt22:53:361
1925161994429,10cyclictest1977037-21sed23:00:021
1925161994422,5cyclictest23-21ksoftirqd/122:48:031
1925161994421,19cyclictest23-21ksoftirqd/122:17:001
1925161994420,20cyclictest23-21ksoftirqd/120:57:211
192516199441,3cyclictest131rcu_preempt21:54:311
1925158994439,3cyclictest0-21swapper/020:00:000
1925158994436,5cyclictest16950irq/224-800000021:00:000
192516899434,8cyclictest37-21ksoftirqd/319:30:003
192516899433,10cyclictest37-21ksoftirqd/319:35:023
1925166994336,5cyclictest1928215-21kworker/2:120:49:542
1925166994336,5cyclictest1928215-21kworker/2:120:49:542
1925166994335,5cyclictest1924348-21kworker/2:220:10:412
1925166994335,5cyclictest1911102-21kworker/2:319:50:262
192516199436,4cyclictest131rcu_preempt23:24:111
192516199433,6cyclictest131rcu_preempt22:25:021
1925161994320,6cyclictest23-21ksoftirqd/122:45:001
1925161994320,19cyclictest23-21ksoftirqd/122:12:281
1925161994320,19cyclictest23-21ksoftirqd/120:20:011
1925161994319,6cyclictest23-21ksoftirqd/123:26:471
1925161994319,20cyclictest23-21ksoftirqd/121:15:471
1925161994319,20cyclictest23-21ksoftirqd/119:40:011
1925161994314,21cyclictest0-21swapper/123:08:381
1925161994310,6cyclictest131rcu_preempt00:00:191
192516899424,8cyclictest37-21ksoftirqd/319:50:023
192516199426,8cyclictest23-21ksoftirqd/121:33:241
192516199423,5cyclictest131rcu_preempt23:53:121
192516199423,4cyclictest131rcu_preempt19:35:011
1925161994222,5cyclictest23-21ksoftirqd/123:36:491
1925161994218,19cyclictest23-21ksoftirqd/123:16:531
192516199421,7cyclictest131rcu_preempt19:10:021
192516199421,7cyclictest131rcu_preempt19:10:011
192515899425,30cyclictest19350irq/34-mmc120:20:010
1925158994225,13cyclictest3467602-21systemd-journal20:45:000
1925158994222,14cyclictest3467602-21systemd-journal19:40:010
1925168994130,8cyclictest37-21ksoftirqd/320:35:003
192516199414,8cyclictest23-21ksoftirqd/120:01:501
1925161994120,4cyclictest23-21ksoftirqd/120:35:011
1925161994119,7cyclictest23-21ksoftirqd/123:33:451
1925161994117,6cyclictest23-21ksoftirqd/121:40:261
192515899419,28cyclictest0-21swapper/020:02:320
192515899412,5cyclictest131rcu_preempt19:20:000
1925158994115,6cyclictest131rcu_preempt21:02:210
1925161994020,5cyclictest23-21ksoftirqd/120:25:011
1925161994020,5cyclictest23-21ksoftirqd/120:10:011
1925161994019,5cyclictest23-21ksoftirqd/120:45:011
1925161994018,4cyclictest23-21ksoftirqd/119:59:211
1925161994017,5cyclictest23-21ksoftirqd/120:55:001
1925158994028,6cyclictest1931405-21grep21:10:020
1925158994028,6cyclictest1931405-21grep21:10:020
192516899396,6cyclictest0-21swapper/320:30:013
192516899396,6cyclictest0-21swapper/320:30:013
192516899395,8cyclictest37-21ksoftirqd/319:11:153
1925161993922,12cyclictest284-21systemd-logind20:27:591
1925161993922,12cyclictest284-21systemd-logind20:27:591
1925161993916,4cyclictest23-21ksoftirqd/122:29:041
1925161993818,5cyclictest23-21ksoftirqd/121:02:211
1925158993832,4cyclictest16950irq/224-800000020:30:010
1925158993832,4cyclictest16950irq/224-800000020:30:010
1925158993832,3cyclictest16950irq/224-800000020:55:000
192516699373,6cyclictest131rcu_preempt19:10:022
192516699373,6cyclictest131rcu_preempt19:10:022
1925161993716,5cyclictest23-21ksoftirqd/119:21:181
1925158993725,8cyclictest309-21in:imuxsock20:47:210
1925158993725,8cyclictest309-21in:imuxsock20:47:210
192515899371,4cyclictest1926399-21systemd-run19:35:010
1925158993613,19cyclictest16850irq/134-800000019:28:210
1925161993517,4cyclictest23-21ksoftirqd/119:51:291
192515899351,3cyclictest131rcu_preempt19:10:020
192515899351,3cyclictest131rcu_preempt19:10:010
1925161993317,4cyclictest23-21ksoftirqd/119:11:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional