You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-11 - 23:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon May 11, 2026 12:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3752431998126,11cyclictest3809413-21ssh10:55:020
375243199801,42cyclictest3814244-21grep11:05:010
3752431997926,11cyclictest3764982-21ssh09:22:460
3752431997719,5cyclictest16850irq/134-800000011:50:410
3752431997517,32cyclictest16850irq/134-800000010:28:150
3752431997517,32cyclictest16850irq/134-800000010:28:150
3752431997517,31cyclictest16850irq/134-800000012:01:520
3752431997420,37cyclictest12-21ksoftirqd/007:45:020
3752431997418,32cyclictest16850irq/134-800000009:49:470
3752431997316,6cyclictest16850irq/134-800000009:36:390
3752431997316,32cyclictest16850irq/134-800000011:43:190
3752431997216,31cyclictest16850irq/134-800000010:35:210
3752431997019,30cyclictest16850irq/134-800000011:36:420
3752431997016,30cyclictest16850irq/134-800000011:19:450
3752431997015,31cyclictest16850irq/134-800000009:25:390
3752441996929,12cyclictest37-21ksoftirqd/311:03:333
3752431996918,31cyclictest16850irq/134-800000012:08:140
3752431996817,30cyclictest16850irq/134-800000010:40:150
3752431996815,31cyclictest16850irq/134-800000009:58:070
3752436996743,8cyclictest3782862-21kworker/2:110:30:032
3752436996743,8cyclictest3782862-21kworker/2:110:30:022
3752431996722,36cyclictest12-21ksoftirqd/011:30:020
3752431996616,6cyclictest16850irq/134-800000010:58:110
3752431996517,32cyclictest16850irq/134-800000010:19:180
3752431996515,32cyclictest16850irq/134-800000009:10:460
3752441995912,11cyclictest0-21swapper/308:45:033
3752436995950,5cyclictest3739602-21kworker/2:212:28:242
3752431995950,6cyclictest12-21ksoftirqd/010:00:040
3752431995832,11cyclictest3779362-21ssh09:52:340
3752431995727,15cyclictest3818043-21ssh11:12:420
3752431995716,38cyclictest16850irq/134-800000010:25:020
3752436995647,6cyclictest3739602-21kworker/2:210:07:532
3752436995636,12cyclictest275-21dbus-daemon08:50:002
3752434995638,14cyclictest3467602-21systemd-journal08:35:001
3752431995628,11cyclictest3816244-21ssh11:09:070
3752431995627,14cyclictest3847636-21ssh12:14:100
3752431995615,21cyclictest12-21ksoftirqd/011:25:020
3752436995547,5cyclictest3739602-21kworker/2:211:10:232
3752434995516,7cyclictest3754505-21latency_hist07:50:021
3752434995516,7cyclictest3754505-21latency_hist07:50:021
3752431995528,19cyclictest3753517-21latency_hist07:30:020
3752431995528,11cyclictest3849960-21ssh12:19:020
3752431995525,15cyclictest3806237-21ssh10:48:210
3752431995519,32cyclictest16850irq/134-800000011:50:000
3752431995519,32cyclictest16850irq/134-800000011:49:590
3752434995416,9cyclictest309-21in:imuxsock11:15:011
3752431995419,6cyclictest16850irq/134-800000012:33:250
3752431995419,17cyclictest16850irq/134-800000011:57:360
3752431995419,17cyclictest16850irq/134-800000011:57:360
3752431995418,6cyclictest16850irq/134-800000009:43:390
3752431995418,32cyclictest16850irq/134-800000010:11:200
3752431995418,31cyclictest16850irq/134-800000011:26:450
375243199541,24cyclictest131rcu_preempt12:25:030
3752431995410,38cyclictest12-21ksoftirqd/010:08:520
3752436995345,5cyclictest3739602-21kworker/2:212:19:142
3752436995345,5cyclictest3739602-21kworker/2:211:19:452
3752436995344,6cyclictest3739602-21kworker/2:209:43:012
3752436995344,5cyclictest3739602-21kworker/2:211:42:412
3752434995334,14cyclictest3467602-21systemd-journal09:25:001
3752434995324,21cyclictest275-21dbus-daemon11:55:011
3752431995325,13cyclictest3858513-21ssh12:36:270
3752431995319,31cyclictest16850irq/134-800000011:31:110
3752431995319,30cyclictest16850irq/134-800000009:33:000
3752431995318,31cyclictest16850irq/134-800000009:15:200
3752431995318,30cyclictest16850irq/134-800000012:26:460
3752436995244,5cyclictest3782862-21kworker/2:110:55:472
3752436995244,5cyclictest3782862-21kworker/2:110:35:012
3752436995243,5cyclictest3739602-21kworker/2:211:06:372
3752434995232,15cyclictest275-21dbus-daemon09:55:011
3752431995218,30cyclictest16850irq/134-800000010:30:250
3752441995138,9cyclictest37-21ksoftirqd/312:30:033
3752441995135,12cyclictest1-21systemd10:25:013
3752436995143,5cyclictest3782862-21kworker/2:110:53:172
3752436995143,5cyclictest3739602-21kworker/2:212:02:372
3752436995142,6cyclictest3758417-21kworker/2:009:38:172
3752436995141,5cyclictest3739602-21kworker/2:208:35:012
3752434995135,11cyclictest3467602-21systemd-journal09:40:011
3752434995132,15cyclictest3802145-21cron10:40:001
3752434995127,18cyclictest275-21dbus-daemon12:00:001
3752434995127,18cyclictest275-21dbus-daemon12:00:001
375243199516,38cyclictest3754706-21unin-run07:55:000
3752436995044,3cyclictest3739602-21kworker/2:211:36:322
3752436995042,5cyclictest3811868-21kworker/2:311:29:172
3752436995042,5cyclictest3811868-21kworker/2:311:29:162
3752436995042,5cyclictest3739602-21kworker/2:212:31:522
3752436995042,5cyclictest3739602-21kworker/2:211:21:132
3752436995042,5cyclictest3739602-21kworker/2:210:15:502
3752436995041,6cyclictest3782862-21kworker/2:110:23:212
3752436995041,5cyclictest3739602-21kworker/2:212:10:252
3752436995032,13cyclictest1-21systemd08:45:012
3752436995024,19cyclictest291rcuc/208:30:022
3752434995035,11cyclictest3467602-21systemd-journal12:05:001
3752434995031,13cyclictest3467602-21systemd-journal11:10:001
3752434995023,13cyclictest275-21dbus-daemon09:00:011
3752436994941,5cyclictest3739602-21kworker/2:210:00:502
3752436994941,5cyclictest3739602-21kworker/2:209:59:392
3752436994941,5cyclictest3739602-21kworker/2:209:49:392
3752436994917,8cyclictest30-21ksoftirqd/210:50:022
3752434994933,11cyclictest275-21dbus-daemon11:30:021
3752434994933,11cyclictest275-21dbus-daemon11:30:011
375244199482,14cyclictest37-21ksoftirqd/310:00:023
3752436994841,5cyclictest3739602-21kworker/2:212:22:182
3752436994840,5cyclictest3739602-21kworker/2:209:51:532
3752436994832,11cyclictest3467602-21systemd-journal11:34:592
375243499487,8cyclictest23-21ksoftirqd/110:07:381
3752434994823,5cyclictest23-21ksoftirqd/112:11:311
3752441994733,9cyclictest3467602-21systemd-journal12:20:003
3752436994739,5cyclictest3758417-21kworker/2:009:29:522
3752436994739,5cyclictest3739602-21kworker/2:209:17:262
3752436994738,5cyclictest3758417-21kworker/2:009:32:362
375243499471,5cyclictest131rcu_preempt10:50:021
3752431994716,8cyclictest12-21ksoftirqd/008:01:370
3752441994632,11cyclictest37-21ksoftirqd/309:48:203
3752441994629,13cyclictest3467602-21systemd-journal09:55:003
375244199461,4cyclictest131rcu_preempt12:40:023
3752434994619,6cyclictest23-21ksoftirqd/109:30:031
375244199459,6cyclictest131rcu_preempt10:18:163
3752441994512,6cyclictest131rcu_preempt09:40:003
3752436994537,5cyclictest3739602-21kworker/2:211:50:532
3752436994536,5cyclictest3739602-21kworker/2:212:35:452
3752436994536,5cyclictest3739602-21kworker/2:211:56:312
3752436994536,5cyclictest3739602-21kworker/2:211:56:312
3752436994529,12cyclictest275-21dbus-daemon08:40:012
375243499455,3cyclictest131rcu_preempt10:15:581
3752434994534,7cyclictest23-21ksoftirqd/111:39:361
3752434994531,11cyclictest3467602-21systemd-journal07:29:021
3752434994530,10cyclictest0-21swapper/109:05:011
3752434994523,16cyclictest275-21dbus-daemon11:45:011
375244199444,3cyclictest131rcu_preempt10:55:003
375244199441,22cyclictest3758813-21ssh09:10:023
3752436994438,3cyclictest3739602-21kworker/2:212:08:202
3752436994436,5cyclictest3739602-21kworker/2:211:46:032
3752436994436,5cyclictest3739602-21kworker/2:211:46:032
3752436994436,5cyclictest3739602-21kworker/2:209:11:502
3752436994436,5cyclictest3739602-21kworker/2:208:02:202
3752436994434,7cyclictest3739602-21kworker/2:211:01:372
3752436994434,7cyclictest30-21ksoftirqd/208:00:022
3752434994433,7cyclictest3775546-21cron09:45:001
3752434994427,8cyclictest23-21ksoftirqd/112:40:031
3752434994422,16cyclictest3467602-21systemd-journal11:25:001
3752431994431,10cyclictest3467602-21systemd-journal07:48:320
3752431994431,10cyclictest3467602-21systemd-journal07:48:320
3752431994428,12cyclictest3758138-21sendmail-msp09:00:000
3752441994330,10cyclictest275-21dbus-daemon08:26:363
3752441994327,12cyclictest3831113-21pluginstate11:40:023
3752441994326,12cyclictest275-21dbus-daemon09:25:013
3752436994335,5cyclictest3739602-21kworker/2:208:53:152
3752436994335,5cyclictest3739602-21kworker/2:208:53:152
3752436994334,6cyclictest30-21ksoftirqd/209:10:012
3752436994334,5cyclictest3739602-21kworker/2:209:23:132
3752436994329,10cyclictest3467602-21systemd-journal10:45:002
375243499434,6cyclictest131rcu_preempt08:15:001
3752434994334,5cyclictest3850376-21latency_hist12:20:021
3752434994331,9cyclictest3813199-21sshd11:02:361
3752434994331,7cyclictest23-21ksoftirqd/110:45:021
3752434994330,9cyclictest1-21systemd08:00:011
3752434994328,11cyclictest3782857-21sh10:00:011
3752434994325,12cyclictest275-21dbus-daemon08:40:011
3752434994319,5cyclictest23-21ksoftirqd/111:47:031
3752434994319,5cyclictest23-21ksoftirqd/111:47:031
3752431994334,5cyclictest3753152-21munin-plugin-st07:25:000
3752431994334,5cyclictest3753152-21munin-plugin-st07:25:000
3752431994327,12cyclictest3467602-21systemd-journal09:10:010
3752431994311,3cyclictest131rcu_preempt08:10:020
375244199428,6cyclictest131rcu_preempt10:02:243
375244199425,8cyclictest37-21ksoftirqd/312:07:483
3752441994229,9cyclictest37-21ksoftirqd/311:26:173
3752441994229,9cyclictest37-21ksoftirqd/311:26:173
3752441994229,10cyclictest3753699-21cat07:35:013
375244199422,8cyclictest3797403-21latency_hist10:30:023
375244199422,8cyclictest3797403-21latency_hist10:30:023
3752441994228,11cyclictest37-21ksoftirqd/310:30:493
3752441994228,10cyclictest275-21dbus-daemon11:00:023
3752441994226,12cyclictest3848130-21cat12:15:033
375244199421,3cyclictest131rcu_preempt11:15:013
375244199421,3cyclictest131rcu_preempt09:10:273
375244199421,22cyclictest0-21swapper/311:56:223
375244199421,22cyclictest0-21swapper/311:56:223
375243699424,3cyclictest131rcu_preempt07:25:022
375243699424,3cyclictest131rcu_preempt07:25:022
3752436994234,5cyclictest3782862-21kworker/2:110:36:062
3752436994232,7cyclictest3467602-21systemd-journal08:07:362
375243699422,3cyclictest131rcu_preempt08:25:022
3752434994231,8cyclictest23-21ksoftirqd/110:10:041
3752434994228,8cyclictest3828645-21cron11:34:591
3752434994225,12cyclictest23-21ksoftirqd/107:15:021
3752434994217,5cyclictest23-21ksoftirqd/112:06:521
3752434994210,6cyclictest131rcu_preempt08:10:021
375244199414,7cyclictest131rcu_preempt07:40:023
375244199414,5cyclictest131rcu_preempt09:29:143
375244199413,34cyclictest150-21pr/ttyS208:56:363
3752441994130,8cyclictest3774552-21sshd09:42:363
3752441994129,9cyclictest37-21ksoftirqd/311:41:203
3752441994128,10cyclictest37-21ksoftirqd/310:37:013
3752441994127,11cyclictest37-21ksoftirqd/308:10:023
3752441994126,11cyclictest3467602-21systemd-journal11:45:363
3752441994126,11cyclictest3467602-21systemd-journal11:45:363
375244199412,4cyclictest131rcu_preempt11:31:113
3752441994124,13cyclictest275-21dbus-daemon08:15:013
375244199411,5cyclictest0-21swapper/312:23:333
375244199411,3cyclictest131rcu_preempt09:17:553
3752436994132,5cyclictest3739602-21kworker/2:209:00:002
3752436994129,9cyclictest3467602-21systemd-journal10:11:372
375243699412,32cyclictest30-21ksoftirqd/208:15:022
3752434994131,6cyclictest3857692-21latency_hist12:35:021
3752434994129,9cyclictest23-21ksoftirqd/109:20:031
3752434994121,16cyclictest275-21dbus-daemon09:30:011
375243499411,3cyclictest131rcu_preempt12:29:111
3752431994134,4cyclictest16950irq/224-800000008:40:010
3752431994130,9cyclictest12-21ksoftirqd/007:20:020
375243199412,7cyclictest3753909-21modprobe07:40:000
375244199405,3cyclictest131rcu_preempt09:31:063
375244199404,4cyclictest131rcu_preempt12:31:173
3752441994032,6cyclictest37-21ksoftirqd/311:50:323
3752441994030,6cyclictest3757848-21cron08:55:003
3752441994030,6cyclictest3757848-21cron08:55:003
3752441994030,6cyclictest309-21in:imuxsock08:00:003
3752441994029,7cyclictest3467602-21systemd-journal10:06:363
375244199402,8cyclictest364-21snmpd10:48:003
375244199401,5cyclictest0-21swapper/311:17:483
3752436994031,5cyclictest3739602-21kworker/2:208:20:002
3752436994031,5cyclictest3739602-21kworker/2:207:30:012
375243499408,5cyclictest131rcu_preempt10:30:001
375243499408,5cyclictest131rcu_preempt10:30:001
375243499404,26cyclictest3785431-21ls10:05:021
375243499403,27cyclictest3468525-21sendmail-mta07:40:021
3752434994027,10cyclictest1-21systemd09:50:011
3752434994026,11cyclictest3754298-21sed07:45:011
3752434994024,11cyclictest23-21ksoftirqd/107:20:001
3752434994023,12cyclictest3821465-21sh11:20:001
3752434994022,14cyclictest3757843-21sshd08:54:361
3752434994022,14cyclictest3757843-21sshd08:54:361
3752431994030,8cyclictest12-21ksoftirqd/008:35:010
375243199402,34cyclictest3755727-21munin-plugin-st08:15:000
375243199401,36cyclictest0-21swapper/007:10:020
375244199395,6cyclictest131rcu_preempt11:09:513
375244199393,8cyclictest131rcu_preempt10:14:363
375244199393,4cyclictest131rcu_preempt07:20:023
3752441993927,9cyclictest37-21ksoftirqd/311:21:413
3752441993920,13cyclictest3467602-21systemd-journal08:35:003
3752436993926,10cyclictest30-21ksoftirqd/207:20:022
375243699392,3cyclictest131rcu_preempt07:55:012
375243499396,22cyclictest3758636-21modprobe09:10:001
3752434993925,10cyclictest23-21ksoftirqd/108:25:001
3752434993921,13cyclictest3467718-21lldpd10:34:241
375243199393,32cyclictest0-21swapper/008:00:020
3752431993925,9cyclictest3467602-21systemd-journal08:55:010
3752431993925,9cyclictest3467602-21systemd-journal08:55:000
3752431993918,6cyclictest16850irq/134-800000008:25:280
375244199385,3cyclictest131rcu_preempt12:04:323
3752441993827,8cyclictest1-21systemd08:20:013
3752441993826,9cyclictest3758432-21cat09:05:013
375244199381,3cyclictest131rcu_preempt10:45:003
3752436993830,5cyclictest3739602-21kworker/2:207:50:022
3752436993830,5cyclictest3739602-21kworker/2:207:50:022
375243499386,4cyclictest150-21pr/ttyS209:15:011
375243499385,28cyclictest3757354-21munin-run08:45:011
3752434993828,7cyclictest23-21ksoftirqd/108:26:381
3752434993827,8cyclictest3754676-21sshd07:54:361
375243499382,3cyclictest131rcu_preempt10:50:321
3752434993823,8cyclictest310-21in:imklog07:25:011
3752434993823,8cyclictest310-21in:imklog07:25:011
3752434993822,10cyclictest0-21swapper/112:25:001
3752434993821,9cyclictest3753670-21systemd-run07:35:011
3752434993818,16cyclictest275-21dbus-daemon10:25:021
3752434993813,5cyclictest23-21ksoftirqd/110:56:331
3752431993827,8cyclictest3756248-21sshd08:23:360
375243199382,4cyclictest3755979-21modprobe08:20:000
375243699372,4cyclictest131rcu_preempt07:15:012
3752431993726,8cyclictest12-21ksoftirqd/009:05:020
375243199371,3cyclictest0-21swapper/007:15:010
375244199363,3cyclictest131rcu_preempt07:48:363
375244199363,3cyclictest131rcu_preempt07:48:363
375244199363,3cyclictest131rcu_preempt07:10:023
375244199362,6cyclictest131rcu_preempt08:20:363
3752441993624,8cyclictest3757084-21run-parts08:40:003
3752441993620,13cyclictest275-21dbus-daemon07:30:003
375244199361,3cyclictest131rcu_preempt07:15:033
3752436993625,8cyclictest3753898-21cron07:40:012
3752436993623,8cyclictest30-21ksoftirqd/207:42:012
3752434993628,4cyclictest3757606-21latency_hist08:50:021
3752431993628,4cyclictest309-21in:imuxsock08:40:370
375244199351,3cyclictest131rcu_preempt08:00:033
375243499354,4cyclictest131rcu_preempt08:05:021
3752431993522,10cyclictest3757586-21sshd08:48:360
375243199351,31cyclictest0-21swapper/007:31:310
3752441993423,8cyclictest37-21ksoftirqd/307:45:003
3752441993423,8cyclictest37-21ksoftirqd/307:25:023
3752441993423,8cyclictest37-21ksoftirqd/307:25:023
3752436993422,9cyclictest30-21ksoftirqd/209:00:082
375243699341,3cyclictest131rcu_preempt07:35:022
375244199333,3cyclictest131rcu_preempt07:52:363
3752434993323,4cyclictest23-21ksoftirqd/108:17:001
375244199322,3cyclictest131rcu_preempt08:46:363
3752436993122,7cyclictest30-21ksoftirqd/207:10:022
3752434993021,7cyclictest23-21ksoftirqd/107:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional