You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-06 - 10:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Sat Jun 06, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3653782998723,19cyclictest3696523-21ssh22:31:522
3653782998628,20cyclictest3703621-21ssh22:47:432
3653782997831,8cyclictest3743307-21ssh00:15:582
3653782997825,19cyclictest3656632-21sed20:05:022
3653782997824,19cyclictest3678965-21ssh21:52:362
3653782997823,14cyclictest3736129-21ssh00:00:042
3653782997630,15cyclictest3670664-21ssh21:34:032
3653782997630,13cyclictest3722078-21sh23:29:202
3653782997525,19cyclictest3467602-21systemd-journal21:37:462
3653782997525,19cyclictest3467602-21systemd-journal21:37:452
3653782997523,14cyclictest3728458-21ssh23:43:182
3653782997522,19cyclictest3659905-21sed21:05:032
3653782997522,19cyclictest3658259-21cut20:35:032
3653782997522,18cyclictest3689187-21ssh22:15:162
3653782997423,19cyclictest3708197-21ssh22:58:122
3653782997423,17cyclictest3747672-21rm00:25:422
3653782997422,20cyclictest3658998-21latency_hist20:50:022
3653782997422,19cyclictest3698789-21sh22:36:552
3653782997422,15cyclictest3732383-21ssh23:51:532
3653782997323,16cyclictest3739029-21rm00:06:372
3653782997321,17cyclictest3658742-21latency_hist20:45:012
3653782997224,18cyclictest3686523-21rm22:09:382
3653782997219,18cyclictest3674742-21ssh21:43:032
3653782997118,13cyclictest3717481-21ssh23:18:502
3653782997025,15cyclictest3467602-21systemd-journal23:03:512
3653782997024,16cyclictest3683800-21sh22:03:322
3653782997020,15cyclictest3725937-21ssh23:37:482
3653782997020,15cyclictest3725937-21ssh23:37:472
3653775997031,9cyclictest12-21ksoftirqd/022:57:270
3653782996922,15cyclictest3742350-21ssh00:14:102
3653782996922,15cyclictest3687194-21ssh22:10:442
3653782996830,15cyclictest3712175-21ssh23:06:592
3653782996825,15cyclictest3750412-21ssh00:32:072
3653782996823,14cyclictest3752949-21ssh00:37:502
3653782996822,13cyclictest3666869-21ssh21:25:042
3653782996821,17cyclictest3655189-21sed19:35:022
3653782996819,14cyclictest3693898-21ssh22:25:462
3653782996759,6cyclictest30-21ksoftirqd/223:25:022
3653782996650,13cyclictest3665255-21ssh21:21:352
3653782996627,9cyclictest3724624-21munin-run23:35:012
3653782996620,17cyclictest3661304-21ssh21:12:472
365377599668,28cyclictest131rcu_preempt22:00:040
3653775996657,6cyclictest16950irq/224-800000022:05:030
3653782996542,19cyclictest3467602-21systemd-journal21:20:012
3653782996449,12cyclictest3729993-21ssh23:46:342
3653782996443,16cyclictest3701239-21ssh22:42:222
3653782996423,14cyclictest3735026-21rm23:57:592
3653782996423,14cyclictest3735026-21rm23:57:592
3653782996344,16cyclictest3714912-21ssh23:13:072
3653782996342,17cyclictest3680526-21ssh21:55:582
365377599638,29cyclictest131rcu_preempt21:10:120
365377599634,9cyclictest12-21ksoftirqd/023:50:000
3653775996341,19cyclictest16850irq/134-800000000:30:000
3653782996142,16cyclictest3704784-21ssh22:50:122
3653782996140,17cyclictest3745627-21ssh00:21:092
365377599618,5cyclictest131rcu_preempt23:22:300
3653782996045,12cyclictest3691501-21ssh22:20:242
3653782996045,12cyclictest3676301-21ssh21:46:202
3653782996042,15cyclictest3657856-21sshd20:27:272
3653778996038,9cyclictest23-21ksoftirqd/122:59:041
365377599602,9cyclictest12-21ksoftirqd/000:00:020
365377599602,9cyclictest12-21ksoftirqd/000:00:020
365378799593,3cyclictest131rcu_preempt22:02:533
3653775995925,31cyclictest12-21ksoftirqd/023:04:080
3653775995914,6cyclictest131rcu_preempt21:30:550
365378799582,41cyclictest3700159-21ssh22:40:023
3653782995844,11cyclictest3657607-21sshd20:22:282
365377599577,9cyclictest12-21ksoftirqd/022:25:020
3653775995727,14cyclictest3680799-21ssh21:56:460
3653775995716,20cyclictest16850irq/134-800000022:20:010
3653775995713,10cyclictest12-21ksoftirqd/021:27:510
3653775995712,5cyclictest131rcu_preempt20:35:030
3653787995632,18cyclictest275-21dbus-daemon23:20:003
3653782995643,9cyclictest3658366-21sshd20:38:292
3653782995640,10cyclictest3653963-21cron19:15:012
3653782995636,16cyclictest364-21snmpd19:47:412
3653782995634,17cyclictest364-21snmpd19:51:082
365377899561,9cyclictest3659960-21diskmemload23:30:001
365377599568,6cyclictest131rcu_preempt22:13:280
3653775995627,14cyclictest3727799-21ssh23:41:420
3653775995622,31cyclictest12-21ksoftirqd/023:28:430
365377599561,38cyclictest3723257-21ssh23:31:390
3653782995548,4cyclictest3656853-21munin-run20:10:012
3653782995546,5cyclictest3659145-21sshd20:51:272
3653782995544,8cyclictest3655518-21sshd19:40:272
3653782995542,10cyclictest3659928-21sshd21:06:272
3653782995540,12cyclictest3654734-21sshd19:25:262
3653782995533,16cyclictest3655243-21modprobe19:36:282
365377899555,6cyclictest131rcu_preempt22:10:301
365377599553,21cyclictest12-21ksoftirqd/023:54:120
3653775995528,11cyclictest3712014-21ssh23:06:360
3653775995527,12cyclictest3716446-21ssh23:16:280
3653775995527,11cyclictest3749913-21ssh00:30:500
3653775995511,6cyclictest131rcu_preempt22:52:300
3653775995510,5cyclictest131rcu_preempt22:33:010
3653782995437,12cyclictest364-21snmpd19:15:332
3653782995424,25cyclictest3657361-21munin-plugin-st20:20:012
3653778995437,13cyclictest1-21systemd21:20:011
365377599548,18cyclictest131rcu_preempt22:35:520
3653775995418,8cyclictest131rcu_preempt23:11:560
3653775995411,6cyclictest131rcu_preempt21:20:530
3653782995346,4cyclictest3657063-21sshd20:11:272
3653782995339,11cyclictest364-21snmpd19:56:342
3653782995338,11cyclictest3659396-21sshd20:56:272
365377899531,20cyclictest0-21swapper/100:15:001
3653775995335,15cyclictest1-21systemd23:40:010
3653775995335,15cyclictest1-21systemd23:40:010
365377599532,7cyclictest12-21ksoftirqd/000:10:020
3653775995319,29cyclictest16850irq/134-800000021:19:180
3653775995314,6cyclictest131rcu_preempt21:42:280
3653787995233,14cyclictest275-21dbus-daemon23:55:013
3653778995226,10cyclictest23-21ksoftirqd/122:39:591
3653775995240,8cyclictest151rcuc/020:00:010
3653775995223,13cyclictest3741809-21ssh00:12:580
3653775995210,5cyclictest131rcu_preempt21:46:150
3653782995132,15cyclictest364-21snmpd19:20:372
365377899514,7cyclictest131rcu_preempt00:30:011
3653778995122,21cyclictest23-21ksoftirqd/122:43:581
3653775995143,5cyclictest16850irq/134-800000022:47:260
3653775995143,4cyclictest16950irq/224-800000020:10:010
365377599513,8cyclictest3673301-21systemctl21:40:010
365377599513,8cyclictest3673301-21systemctl21:40:000
365377599512,17cyclictest3753832-21sendmail-msp00:40:000
3653775995120,26cyclictest310-21in:imklog22:41:420
365377599511,33cyclictest12-21ksoftirqd/020:58:530
3653775995110,5cyclictest131rcu_preempt21:54:040
3653787995023,22cyclictest37-21ksoftirqd/320:10:013
3653778995021,16cyclictest23-21ksoftirqd/122:27:301
3653778995019,22cyclictest23-21ksoftirqd/122:15:441
365377899501,17cyclictest0-21swapper/121:51:491
365377599509,8cyclictest12-21ksoftirqd/019:50:010
3653787994940,5cyclictest37-21ksoftirqd/322:30:023
3653787994934,11cyclictest275-21dbus-daemon23:00:013
3653787994931,8cyclictest37-21ksoftirqd/321:10:243
3653787994931,14cyclictest275-21dbus-daemon00:00:013
3653787994931,14cyclictest275-21dbus-daemon00:00:003
3653778994937,8cyclictest23-21ksoftirqd/100:00:011
3653778994937,8cyclictest23-21ksoftirqd/100:00:011
3653778994931,13cyclictest275-21dbus-daemon00:40:001
3653778994921,20cyclictest23-21ksoftirqd/100:33:401
3653778994921,19cyclictest23-21ksoftirqd/121:33:031
3653778994920,25cyclictest23-21ksoftirqd/122:48:311
3653778994920,20cyclictest23-21ksoftirqd/123:35:041
3653778994920,20cyclictest23-21ksoftirqd/123:35:041
3653778994919,26cyclictest23-21ksoftirqd/120:50:001
365377899491,8cyclictest0-21swapper/122:54:011
3653778994916,19cyclictest23-21ksoftirqd/121:24:271
365377599499,5cyclictest131rcu_preempt22:28:090
365377599495,7cyclictest12-21ksoftirqd/000:05:010
365377599494,10cyclictest12-21ksoftirqd/019:45:000
3653787994838,6cyclictest3753807-21latency_hist00:40:003
3653787994815,7cyclictest0-21swapper/300:33:313
3653782994824,19cyclictest3653860-21latency_hist19:10:022
3653778994822,15cyclictest23-21ksoftirqd/123:24:161
3653778994821,18cyclictest23-21ksoftirqd/121:44:541
3653778994820,19cyclictest23-21ksoftirqd/123:05:361
3653778994819,20cyclictest23-21ksoftirqd/121:26:021
3653778994815,17cyclictest23-21ksoftirqd/121:39:181
3653778994815,17cyclictest23-21ksoftirqd/121:39:181
365377899481,19cyclictest0-21swapper/122:20:221
365377599483,38cyclictest3659956-21sh21:10:010
3653775994830,14cyclictest1-21systemd00:25:010
3653775994821,6cyclictest151rcuc/000:16:530
365378799474,26cyclictest3738219-21ssh00:05:023
3653787994731,13cyclictest3673281-21cron21:40:003
3653787994731,13cyclictest3673281-21cron21:40:003
3653787994722,20cyclictest37-21ksoftirqd/323:32:583
3653787994722,20cyclictest37-21ksoftirqd/300:15:383
365378799471,5cyclictest3659420-21modprobe21:00:013
3653778994730,13cyclictest3467602-21systemd-journal20:40:001
3653778994719,20cyclictest23-21ksoftirqd/123:33:471
3653778994711,5cyclictest131rcu_preempt21:45:211
365377599477,10cyclictest12-21ksoftirqd/019:16:270
3653775994710,4cyclictest131rcu_preempt20:45:020
365378799463,6cyclictest131rcu_preempt23:45:003
3653787994632,10cyclictest3467602-21systemd-journal20:20:003
3653787994624,16cyclictest3467602-21systemd-journal00:30:003
3653787994622,19cyclictest37-21ksoftirqd/322:09:363
365377899467,6cyclictest131rcu_preempt00:01:241
3653778994629,13cyclictest275-21dbus-daemon00:25:001
3653778994619,19cyclictest23-21ksoftirqd/121:12:011
3653778994619,18cyclictest23-21ksoftirqd/122:02:101
3653778994617,21cyclictest131rcu_preempt23:11:561
365377899461,15cyclictest0-21swapper/123:46:591
365377899461,15cyclictest0-21swapper/122:32:151
365378799454,3cyclictest131rcu_preempt19:45:033
3653778994526,15cyclictest275-21dbus-daemon19:30:001
3653778994522,19cyclictest23-21ksoftirqd/122:10:001
3653778994511,6cyclictest131rcu_preempt20:20:011
3653775994519,23cyclictest12-21ksoftirqd/019:30:020
365377599451,8cyclictest0-21swapper/020:00:320
365377599451,7cyclictest3654152-21latency_hist19:15:030
365378799443,3cyclictest131rcu_preempt21:25:133
3653787994427,12cyclictest3467602-21systemd-journal21:55:033
3653787994426,13cyclictest275-21dbus-daemon20:15:013
3653787994421,19cyclictest37-21ksoftirqd/322:25:053
3653787994419,21cyclictest37-21ksoftirqd/321:34:213
3653787994416,18cyclictest37-21ksoftirqd/300:05:033
3653778994428,12cyclictest3654246-21systemctl19:20:011
3653778994418,6cyclictest131rcu_preempt00:17:021
3653778994418,22cyclictest23-21ksoftirqd/123:18:431
3653778994418,18cyclictest23-21ksoftirqd/123:50:301
3653778994417,20cyclictest23-21ksoftirqd/100:07:001
3653778994411,14cyclictest23-21ksoftirqd/123:41:511
365377599441,7cyclictest0-21swapper/019:34:280
3653787994323,15cyclictest275-21dbus-daemon23:50:013
365377899439,13cyclictest23-21ksoftirqd/121:58:291
3653778994329,11cyclictest23-21ksoftirqd/120:12:281
365377599439,6cyclictest131rcu_preempt20:40:000
3653775994330,8cyclictest1-21systemd19:40:020
365377599432,7cyclictest12-21ksoftirqd/021:05:000
365378799422,6cyclictest131rcu_preempt21:25:003
3653778994226,12cyclictest3467602-21systemd-journal21:10:011
365377899422,5cyclictest131rcu_preempt19:35:011
3653775994230,7cyclictest16950irq/224-800000020:20:010
365377599421,8cyclictest12-21ksoftirqd/020:55:020
365377599421,13cyclictest12-21ksoftirqd/019:25:010
3653787994127,11cyclictest37-21ksoftirqd/319:25:033
3653787994118,19cyclictest37-21ksoftirqd/321:45:503
3653775994110,7cyclictest12-21ksoftirqd/020:29:590
365378799404,31cyclictest361rcuc/322:45:023
3653787994023,12cyclictest275-21dbus-daemon20:45:013
3653787994016,19cyclictest37-21ksoftirqd/300:20:083
3653787994015,9cyclictest37-21ksoftirqd/322:10:423
3653787994014,19cyclictest131rcu_preempt19:15:003
365377899407,12cyclictest3639564-21munin-node19:10:011
365377899402,10cyclictest150-21pr/ttyS219:45:251
365377599402,9cyclictest12-21ksoftirqd/020:50:020
3653787993930,5cyclictest309-21in:imuxsock21:55:003
3653787993930,5cyclictest309-21in:imuxsock21:05:003
3653787993923,12cyclictest3467602-21systemd-journal22:46:283
3653778993927,9cyclictest3467602-21systemd-journal20:03:321
3653778993911,6cyclictest131rcu_preempt23:03:291
365377599392,18cyclictest12-21ksoftirqd/020:22:420
3653787993828,7cyclictest3655779-21sshd19:47:293
3653787993825,10cyclictest3722526-21sed23:30:013
3653787993824,10cyclictest309-21in:imuxsock19:35:013
365378799382,3cyclictest131rcu_preempt22:50:293
3653787993816,8cyclictest131rcu_preempt23:37:233
3653787993816,8cyclictest131rcu_preempt23:37:223
365378799381,5cyclictest131rcu_preempt23:22:203
3653778993830,5cyclictest23-21ksoftirqd/119:55:001
3653778993828,6cyclictest3657619-21cron20:24:591
3653778993827,8cyclictest3656814-21sshd20:07:281
3653778993825,10cyclictest23-21ksoftirqd/119:40:301
365377899381,8cyclictest0-21swapper/120:26:281
365377899381,28cyclictest3654733-21kworker/u8:119:40:011
3653775993825,11cyclictest12-21ksoftirqd/019:55:010
365377599381,6cyclictest0-21swapper/020:11:010
3653775993811,6cyclictest131rcu_preempt19:10:020
3653787993729,4cyclictest3712363-21sh23:07:243
3653787993728,5cyclictest37-21ksoftirqd/300:15:013
3653787993727,6cyclictest3691307-21latency_hist22:20:023
365378799372,4cyclictest131rcu_preempt21:19:143
3653787993724,10cyclictest37-21ksoftirqd/321:10:013
365378799372,3cyclictest131rcu_preempt21:42:333
3653787993723,11cyclictest37-21ksoftirqd/322:23:353
3653787993715,15cyclictest131rcu_preempt23:01:543
365377899378,4cyclictest131rcu_preempt19:55:461
3653778993727,7cyclictest23-21ksoftirqd/119:20:381
3653778993726,8cyclictest3467602-21systemd-journal20:56:281
3653778993725,10cyclictest23-21ksoftirqd/119:14:271
3653787993626,8cyclictest37-21ksoftirqd/319:40:023
365378799362,5cyclictest131rcu_preempt23:13:283
3653778993618,14cyclictest275-21dbus-daemon20:35:001
365377899361,11cyclictest23-21ksoftirqd/120:50:441
3653787993525,7cyclictest37-21ksoftirqd/320:40:013
3653787993521,11cyclictest275-21dbus-daemon20:00:023
3653778993523,9cyclictest23-21ksoftirqd/120:45:011
365377899351,10cyclictest23-21ksoftirqd/121:01:281
3653787993424,7cyclictest37-21ksoftirqd/320:26:423
3653787993419,11cyclictest37-21ksoftirqd/320:50:313
3653787993415,7cyclictest131rcu_preempt20:50:003
3653787993322,9cyclictest37-21ksoftirqd/319:20:013
3653787993319,11cyclictest3658187-21sed20:35:013
3653787993226,4cyclictest0-21swapper/319:10:013
3653787993220,10cyclictest37-21ksoftirqd/319:55:003
365378799316,5cyclictest131rcu_preempt20:04:273
365378799313,21cyclictest3657633-21munin-run20:25:003
3653787993021,4cyclictest37-21ksoftirqd/319:24:283
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional