You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-03 - 08:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Sep 03, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2781229999578,14cyclictest37-21ksoftirqd/323:40:023
2781218999527,14cyclictest2872031-21ssh00:08:360
2781218999029,13cyclictest2821219-21ssh22:20:460
2781218998926,11cyclictest2883278-21ssh00:32:130
2781218998427,15cyclictest2795940-21ssh21:27:460
2781218998426,11cyclictest2874834-21ssh00:14:380
278121899842,30cyclictest48-21kcompactd022:35:450
2781218998227,12cyclictest2868578-21ssh00:01:040
278121899821,36cyclictest2879300-21ssh00:24:010
2781218998128,12cyclictest2861226-21ssh23:45:200
2781218998126,11cyclictest2823686-21ssh22:26:110
2781218998126,11cyclictest2823686-21ssh22:26:110
2781218998024,11cyclictest2864498-21ssh23:52:330
2781218998024,11cyclictest2864498-21ssh23:52:330
2781218997942,32cyclictest16850irq/134-800000000:00:010
2781218997924,9cyclictest2843789-21ssh23:08:540
2781218997924,9cyclictest2843789-21ssh23:08:540
2781218997621,9cyclictest2820825-21sort22:20:020
2781218997620,13cyclictest2853366-21ssh23:29:150
278121899753,26cyclictest16950irq/224-800000022:00:020
2781218997526,12cyclictest2807478-21ssh21:51:550
2781218997525,14cyclictest2855185-21ssh23:32:550
2781218997525,14cyclictest2806285-21ssh21:49:420
2781218997425,13cyclictest2832906-21ssh22:45:310
2781218997423,12cyclictest2831859-21ssh22:43:380
2781218997423,12cyclictest2831859-21ssh22:43:370
2781218997422,14cyclictest2860148-21ssh23:43:250
2781218997422,14cyclictest2860148-21ssh23:43:240
2781229997319,16cyclictest131rcu_preempt22:20:023
278121899733,7cyclictest2781671-21sh19:20:000
2781218997327,12cyclictest2825588-21ssh22:30:030
2781218997327,12cyclictest2825588-21ssh22:30:030
2781218997327,11cyclictest2793468-21ssh21:22:340
2781218997325,13cyclictest2838064-21ssh22:56:300
2781218997320,12cyclictest2800369-21ssh21:37:040
2781227997213,19cyclictest2839716-21ssh23:00:022
2781218997228,5cyclictest2782692-21cron19:40:000
2781218997135,31cyclictest16850irq/134-800000021:20:000
2781218997124,13cyclictest2886488-21ssh00:39:150
2781218997123,11cyclictest2844885-21ssh23:10:520
2781218997122,10cyclictest2813246-21ssh22:04:140
2781218997024,14cyclictest2788882-21ssh21:12:590
2781218996826,11cyclictest2881974-21ssh00:29:380
2781218996824,11cyclictest2837152-21ssh22:54:540
2781218996812,6cyclictest16850irq/134-800000021:33:280
2781218996731,33cyclictest16950irq/224-800000020:30:030
278121899671,4cyclictest2781922-21modprobe19:21:290
2781218996662,2cyclictest0-21swapper/023:35:020
2781218996616,16cyclictest16850irq/134-800000022:07:580
2781218996519,15cyclictest2802479-21ssh21:41:250
278121899651,11cyclictest2849045-21sh23:20:010
2781218996426,7cyclictest16950irq/224-800000000:20:010
278121899642,4cyclictest2818331-21munin-run22:15:010
2781227996340,5cyclictest2832498-21kworker/2:223:08:282
2781227996340,5cyclictest2832498-21kworker/2:223:08:272
2781218996328,6cyclictest2784015-21latency_hist20:05:020
2781218996221,10cyclictest1-21systemd21:00:000
278122999616,9cyclictest37-21ksoftirqd/321:25:033
278122999609,12cyclictest1-21systemd19:40:003
2781229996016,21cyclictest361rcuc/321:10:023
278122999596,9cyclictest37-21ksoftirqd/321:45:013
2781229995911,15cyclictest37-21ksoftirqd/322:10:033
278121899599,45cyclictest16850irq/134-800000023:24:260
278122999586,10cyclictest37-21ksoftirqd/322:05:023
2781227995845,8cyclictest2784532-21kworker/2:122:10:002
2781218995817,17cyclictest16850irq/134-800000023:02:270
278122999576,10cyclictest37-21ksoftirqd/321:00:023
278122999575,11cyclictest37-21ksoftirqd/300:35:023
2781229995741,13cyclictest37-21ksoftirqd/300:00:003
278121899571,30cyclictest0-21swapper/019:25:030
278121899571,30cyclictest0-21swapper/019:25:030
278122999561,8cyclictest150-21pr/ttyS223:15:023
2781229995610,6cyclictest131rcu_preempt21:51:453
278122799569,33cyclictest310-21in:imklog23:40:032
2781222995639,13cyclictest3467602-21systemd-journal19:20:011
278122999557,11cyclictest37-21ksoftirqd/322:40:333
278122999557,11cyclictest37-21ksoftirqd/322:40:333
278122999555,9cyclictest37-21ksoftirqd/321:14:113
278122999552,8cyclictest37-21ksoftirqd/322:40:003
2781218995527,5cyclictest2785570-21munin-run20:35:000
278122999549,6cyclictest131rcu_preempt23:16:493
278122999546,7cyclictest37-21ksoftirqd/322:06:493
278122999546,10cyclictest37-21ksoftirqd/300:39:563
278122999545,10cyclictest37-21ksoftirqd/300:12:523
2781229995413,6cyclictest131rcu_preempt00:15:593
2781227995445,5cyclictest2784532-21kworker/2:121:49:372
2781227995442,8cyclictest2784532-21kworker/2:123:44:072
2781227995442,8cyclictest2784532-21kworker/2:123:44:072
2781227995430,5cyclictest2784532-21kworker/2:122:41:002
2781227995430,5cyclictest2784532-21kworker/2:122:41:002
278122299544,11cyclictest0-21swapper/100:00:021
2781229995318,11cyclictest37-21ksoftirqd/323:45:013
2781229995318,11cyclictest37-21ksoftirqd/323:45:013
2781229995313,4cyclictest131rcu_preempt22:30:013
2781229995313,4cyclictest131rcu_preempt22:30:003
2781227995344,6cyclictest2784532-21kworker/2:122:36:192
278122999528,10cyclictest37-21ksoftirqd/323:08:423
278122999528,10cyclictest37-21ksoftirqd/323:08:423
278122999525,8cyclictest37-21ksoftirqd/323:30:523
278122999524,10cyclictest37-21ksoftirqd/321:20:003
278122999522,11cyclictest37-21ksoftirqd/300:05:013
2781229995211,5cyclictest131rcu_preempt22:58:133
2781227995237,3cyclictest0-21swapper/220:15:012
2781227995232,15cyclictest275-21dbus-daemon19:20:012
278122999518,12cyclictest37-21ksoftirqd/320:55:023
278122999518,12cyclictest37-21ksoftirqd/320:55:023
278122999517,14cyclictest37-21ksoftirqd/323:27:583
278122999517,10cyclictest37-21ksoftirqd/323:24:173
278122999515,9cyclictest37-21ksoftirqd/320:00:003
278122999515,11cyclictest37-21ksoftirqd/322:31:393
278122999515,11cyclictest37-21ksoftirqd/322:31:383
278122999512,11cyclictest37-21ksoftirqd/321:58:553
278122999512,10cyclictest37-21ksoftirqd/323:00:593
2781229995117,11cyclictest37-21ksoftirqd/321:35:023
2781227995143,5cyclictest2858480-21kworker/2:023:46:192
2781227995143,5cyclictest2790005-21kworker/2:221:56:052
2781227995143,5cyclictest2784532-21kworker/2:121:38:412
2781227995143,5cyclictest2784532-21kworker/2:100:15:472
2781227995142,6cyclictest2832498-21kworker/2:223:01:172
2781227995142,6cyclictest2784532-21kworker/2:123:28:262
2781227995132,15cyclictest275-21dbus-daemon21:55:012
2781222995135,12cyclictest1-21systemd19:30:001
2781222995135,12cyclictest1-21systemd19:30:001
2781222995133,14cyclictest3467602-21systemd-journal23:46:401
2781218995124,7cyclictest2783786-21wc20:00:010
278122999506,12cyclictest37-21ksoftirqd/300:27:453
2781229995012,5cyclictest131rcu_preempt23:51:583
2781229995012,5cyclictest131rcu_preempt23:51:583
2781227995042,5cyclictest2874992-21kworker/2:200:23:112
2781227995042,5cyclictest2784532-21kworker/2:123:25:022
2781227995042,5cyclictest2784532-21kworker/2:121:10:172
2781227995041,5cyclictest2784532-21kworker/2:121:17:362
2781227995036,10cyclictest2787372-21cron21:10:002
2781227995022,7cyclictest2784532-21kworker/2:122:52:032
278122799502,13cyclictest2863646-21sh23:50:352
278122799502,13cyclictest2863646-21sh23:50:342
2781227995010,6cyclictest131rcu_preempt00:33:262
278122999496,9cyclictest37-21ksoftirqd/319:30:013
278122999496,9cyclictest37-21ksoftirqd/319:30:003
278122999496,10cyclictest37-21ksoftirqd/322:25:003
278122999495,9cyclictest37-21ksoftirqd/321:47:013
278122999495,7cyclictest37-21ksoftirqd/300:22:373
278122999493,7cyclictest37-21ksoftirqd/321:37:103
2781229994934,11cyclictest3467602-21systemd-journal20:50:003
278122999492,6cyclictest37-21ksoftirqd/300:06:493
2781227994941,5cyclictest2820706-21kworker/2:322:47:002
2781227994941,5cyclictest2784532-21kworker/2:122:33:352
2781227994941,5cyclictest2784532-21kworker/2:122:33:352
2781227994941,5cyclictest2784532-21kworker/2:121:20:102
2781227994941,5cyclictest2784532-21kworker/2:100:35:372
2781227994940,5cyclictest2784532-21kworker/2:100:08:392
278122799493,4cyclictest131rcu_preempt22:26:522
278122799493,4cyclictest131rcu_preempt22:26:522
2781227994934,11cyclictest1-21systemd20:20:012
2781227994914,6cyclictest131rcu_preempt20:55:002
2781227994914,6cyclictest131rcu_preempt20:55:002
2781222994935,11cyclictest3467602-21systemd-journal21:30:331
2781222994933,11cyclictest23-21ksoftirqd/123:45:021
2781222994933,11cyclictest23-21ksoftirqd/123:45:021
2781222994932,13cyclictest3467602-21systemd-journal21:09:351
278122999484,9cyclictest37-21ksoftirqd/320:25:023
278122999484,7cyclictest37-21ksoftirqd/322:45:383
278122999481,7cyclictest361rcuc/322:51:443
278122799484,3cyclictest131rcu_preempt23:15:392
2781227994841,4cyclictest2781962-21kworker/2:219:46:312
2781227994840,5cyclictest2832498-21kworker/2:223:12:072
2781227994840,5cyclictest2784532-21kworker/2:121:32:572
278122299487,32cyclictest2791263-21kworker/u8:121:55:031
2781229994726,14cyclictest275-21dbus-daemon20:15:003
278122999471,8cyclictest0-21swapper/320:40:083
2781227994739,6cyclictest2858480-21kworker/2:000:03:102
2781227994738,6cyclictest2784532-21kworker/2:100:13:542
2781227994738,5cyclictest2784532-21kworker/2:122:21:512
2781227994713,6cyclictest131rcu_preempt23:33:172
2781227994713,6cyclictest131rcu_preempt22:00:152
2781227994711,5cyclictest131rcu_preempt21:44:542
2781222994735,9cyclictest23-21ksoftirqd/123:30:011
2781222994734,7cyclictest2801637-21cron21:40:001
2781222994733,11cyclictest23-21ksoftirqd/120:30:011
2781222994729,13cyclictest275-21dbus-daemon20:20:001
2781222994725,15cyclictest275-21dbus-daemon20:25:001
278122999462,5cyclictest0-21swapper/323:48:193
278122999461,12cyclictest37-21ksoftirqd/321:26:353
278122999461,12cyclictest37-21ksoftirqd/320:20:003
2781227994638,5cyclictest2858480-21kworker/2:023:58:042
2781227994638,5cyclictest2784532-21kworker/2:100:29:252
2781227994637,6cyclictest2542792-21kworker/2:019:37:202
2781227994636,6cyclictest2542792-21kworker/2:021:00:002
2781222994627,14cyclictest23-21ksoftirqd/120:45:021
278122999458,8cyclictest37-21ksoftirqd/320:35:013
2781229994532,7cyclictest309-21in:imuxsock19:25:013
2781229994530,11cyclictest309-21in:imuxsock21:05:003
2781229994530,11cyclictest309-21in:imuxsock21:05:003
2781229994520,20cyclictest361rcuc/320:39:343
2781227994540,3cyclictest2784532-21kworker/2:122:14:312
2781227994537,5cyclictest2542792-21kworker/2:019:41:082
2781227994537,5cyclictest2542792-21kworker/2:019:41:082
2781227994537,5cyclictest2542792-21kworker/2:019:30:022
2781227994537,5cyclictest2542792-21kworker/2:019:30:022
2781227994536,5cyclictest2790005-21kworker/2:221:27:152
2781227994530,11cyclictest3467602-21systemd-journal19:14:592
2781222994528,11cyclictest275-21dbus-daemon23:40:011
2781222994527,14cyclictest3467602-21systemd-journal22:55:031
2781218994511,4cyclictest131rcu_preempt20:55:020
2781218994511,4cyclictest131rcu_preempt20:55:020
278122999444,10cyclictest37-21ksoftirqd/320:30:013
2781229994431,9cyclictest1-21systemd19:20:003
2781222994432,9cyclictest2850814-21sshd23:23:381
2781222994428,11cyclictest309-21in:imuxsock21:55:001
2781222994410,5cyclictest131rcu_preempt23:20:021
278121899442,22cyclictest0-21swapper/020:25:000
278121899441,39cyclictest16950irq/224-800000021:10:000
2781227994331,9cyclictest30-21ksoftirqd/220:05:002
278122799431,13cyclictest2786098-21modprobe20:45:012
278122299438,31cyclictest364-21snmpd00:05:151
2781222994330,9cyclictest1-21systemd21:20:001
2781222994329,11cyclictest23-21ksoftirqd/122:40:001
2781222994326,13cyclictest23-21ksoftirqd/122:49:491
2781218994336,4cyclictest16950irq/224-800000020:15:000
278121899433,36cyclictest2784994-21sort20:20:020
278122999423,7cyclictest37-21ksoftirqd/320:05:023
2781229994226,12cyclictest2782434-21sshd19:33:293
278122999421,5cyclictest0-21swapper/319:13:153
278122999421,10cyclictest37-21ksoftirqd/319:55:023
2781227994225,11cyclictest275-21dbus-daemon20:35:002
278122799421,3cyclictest131rcu_preempt22:15:252
2781222994231,8cyclictest1-21systemd23:15:001
2781222994229,10cyclictest23-21ksoftirqd/122:35:001
2781222994229,10cyclictest23-21ksoftirqd/122:35:001
2781218994225,11cyclictest309-21in:imuxsock19:15:000
278122799413,8cyclictest2783490-21munin-run19:55:002
2781227994133,5cyclictest2542792-21kworker/2:019:10:012
2781227994133,5cyclictest2542792-21kworker/2:019:10:012
278122299411,4cyclictest131rcu_preempt23:52:301
278122299411,4cyclictest131rcu_preempt23:52:301
278122299411,4cyclictest131rcu_preempt22:01:501
2781218994131,6cyclictest2785825-21latency_hist20:40:000
2781218994127,9cyclictest1-21systemd19:55:010
2781227994026,10cyclictest1-21systemd19:25:012
278122299405,31cyclictest3467602-21systemd-journal19:45:001
278122299405,31cyclictest3467602-21systemd-journal19:45:001
2781222994029,8cyclictest2794717-21sed21:25:021
2781222994028,10cyclictest23-21ksoftirqd/120:00:011
2781222994027,10cyclictest23-21ksoftirqd/123:33:461
2781222994026,11cyclictest275-21dbus-daemon22:20:011
2781222994026,10cyclictest309-21in:imuxsock23:10:021
2781222994026,10cyclictest309-21in:imuxsock23:10:011
2781222994023,12cyclictest275-21dbus-daemon21:00:001
2781222994022,11cyclictest311-21rs:main1
278122299401,3cyclictest131rcu_preempt22:22:191
278122299401,3cyclictest131rcu_preempt00:20:321
278121899404,3cyclictest131rcu_preempt21:05:000
278121899404,3cyclictest131rcu_preempt21:05:000
278121899403,4cyclictest2782469-21latency_hist19:35:010
2781218994032,5cyclictest12-21ksoftirqd/019:45:020
2781218994032,5cyclictest12-21ksoftirqd/019:45:020
2781218994027,10cyclictest2784251-21modprobe20:08:310
278121899402,33cyclictest2786072-21sshd20:41:340
278122999396,5cyclictest131rcu_preempt20:10:013
278122999394,8cyclictest37-21ksoftirqd/319:44:393
278122999394,8cyclictest37-21ksoftirqd/319:44:393
2781227993927,8cyclictest3467602-21systemd-journal20:40:002
2781227993922,12cyclictest275-21dbus-daemon19:35:002
2781222993927,9cyclictest23-21ksoftirqd/122:42:041
2781222993927,9cyclictest23-21ksoftirqd/122:42:031
2781222993927,9cyclictest23-21ksoftirqd/119:11:391
278122299392,6cyclictest131rcu_preempt19:25:031
2781222993926,11cyclictest23-21ksoftirqd/121:28:231
2781222993926,10cyclictest23-21ksoftirqd/122:55:031
2781222993925,11cyclictest23-21ksoftirqd/100:15:541
2781222993921,14cyclictest0-21swapper/100:40:011
278122299391,3cyclictest131rcu_preempt22:25:181
278122299391,3cyclictest131rcu_preempt22:25:181
278122299391,3cyclictest131rcu_preempt00:03:301
2781218993926,10cyclictest2781334-21wc19:10:020
2781218993926,10cyclictest2781334-21wc19:10:020
278122999381,7cyclictest150-21pr/ttyS219:45:353
2781227993825,10cyclictest1-21systemd20:30:012
2781222993827,8cyclictest23-21ksoftirqd/100:34:301
2781222993826,9cyclictest23-21ksoftirqd/121:49:051
278122299381,5cyclictest131rcu_preempt22:10:011
278122299381,4cyclictest131rcu_preempt23:01:561
278122799374,4cyclictest131rcu_preempt20:10:022
2781227993729,5cyclictest1-21systemd20:00:002
2781227993722,12cyclictest2786512-21sed20:50:022
2781222993727,7cyclictest23-21ksoftirqd/120:50:011
2781222993726,5cyclictest274-21cron19:40:001
2781222993723,11cyclictest23-21ksoftirqd/100:11:361
278122299371,9cyclictest2784529-21true20:15:001
2781218993725,9cyclictest12-21ksoftirqd/020:46:560
2781218993723,10cyclictest2783210-21sshd19:47:310
2781227993624,8cyclictest3467602-21systemd-journal21:00:022
2781227993624,8cyclictest3467602-21systemd-journal21:00:022
2781222993624,9cyclictest23-21ksoftirqd/100:27:151
2781222993621,11cyclictest23-21ksoftirqd/120:31:371
2781222993620,11cyclictest275-21dbus-daemon20:05:011
2781222993528,4cyclictest2786571-21sshd20:51:061
2781222993528,4cyclictest2786571-21sshd20:51:061
278122299351,4cyclictest131rcu_preempt19:55:011
278122299351,3cyclictest131rcu_preempt21:14:081
2781222993420,10cyclictest2782529-21sed19:35:011
2781222993416,12cyclictest3467602-21systemd-journal21:40:021
278122799333,8cyclictest30-21ksoftirqd/220:21:592
2781222993324,4cyclictest23-21ksoftirqd/119:10:011
2781222993324,4cyclictest23-21ksoftirqd/119:10:011
2781222993323,8cyclictest23-21ksoftirqd/121:05:021
2781222993323,8cyclictest23-21ksoftirqd/121:05:021
2781222993323,8cyclictest23-21ksoftirqd/120:37:341
2781222993320,8cyclictest23-21ksoftirqd/119:50:021
2781229993220,8cyclictest0-21swapper/319:10:023
2781229993220,8cyclictest0-21swapper/319:10:023
2781222993213,14cyclictest3467602-21systemd-journal20:10:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional