You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-20 - 18:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Sep 20, 2026 12:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3775221998273,3cyclictest0-21swapper/008:15:020
3775221998066,8cyclictest0-21swapper/007:25:010
3775225997129,38cyclictest3869189-21sh12:23:391
377522599711,13cyclictest3804748-21ssh10:01:081
377522199712,67cyclictest0-21swapper/009:00:010
3775230997019,11cyclictest30-21ksoftirqd/212:10:352
3775230997019,11cyclictest30-21ksoftirqd/212:10:352
377522599703,20cyclictest3813133-21true10:20:021
3775225997028,37cyclictest3817453-21sh10:29:381
3775230996920,12cyclictest30-21ksoftirqd/211:00:022
377522599691,12cyclictest3811101-21cat10:15:021
3775230996830,12cyclictest0-21swapper/211:25:232
3775230996819,12cyclictest30-21ksoftirqd/209:14:412
3775230996819,12cyclictest30-21ksoftirqd/209:14:412
3775230996816,11cyclictest30-21ksoftirqd/210:41:412
3775230996816,11cyclictest30-21ksoftirqd/210:41:402
3775225996823,37cyclictest309-21in:imuxsock11:20:011
3775225996823,37cyclictest309-21in:imuxsock11:20:011
377522599672,14cyclictest3865282-21munin-run12:15:011
377522599672,14cyclictest3865282-21munin-run12:15:001
377522599662,17cyclictest3777479-21sshd07:53:241
377522599655,51cyclictest3781151-21munin-plugin-st09:05:001
377522599655,51cyclictest3781151-21munin-plugin-st09:05:001
377522599652,13cyclictest3777263-21munin-run07:50:011
3775225996434,23cyclictest309-21in:imuxsock11:45:011
377522599643,19cyclictest3798957-21ssh09:48:251
377522599643,19cyclictest3798957-21ssh09:48:251
377522599642,21cyclictest3797396-21munin-run09:45:001
377522599641,14cyclictest3803296-21ssh09:58:161
377522599641,13cyclictest3871778-21ssh12:29:141
377522599641,13cyclictest3779303-21sshd08:28:241
377522599641,10cyclictest3807560-21sshd10:07:251
377522599641,10cyclictest3807560-21sshd10:07:241
3775230996310,14cyclictest3847676-21ssh11:35:522
3775230996310,14cyclictest3847676-21ssh11:35:512
3775225996325,34cyclictest23-21ksoftirqd/112:02:471
3775225996325,34cyclictest23-21ksoftirqd/112:02:461
377522599632,13cyclictest3861504-21ssh12:06:101
3775225996320,39cyclictest23-21ksoftirqd/111:51:551
377522599631,13cyclictest3837007-21ssh11:12:381
377522599631,13cyclictest3789044-21ssh09:26:221
377522599631,10cyclictest3793475-21ssh09:36:031
377522599622,14cyclictest3775723-21munin-run07:20:011
3775225996221,37cyclictest23-21ksoftirqd/109:12:041
3775225996221,37cyclictest23-21ksoftirqd/109:12:031
377522599622,10cyclictest48-21kcompactd010:35:081
377522599621,9cyclictest3857023-21ssh11:56:341
377522599621,7cyclictest3778266-21sshd08:09:201
3775234996110,6cyclictest131rcu_preempt12:40:033
3775230996115,10cyclictest30-21ksoftirqd/210:05:042
3775230996115,10cyclictest30-21ksoftirqd/210:05:032
377522599614,50cyclictest3849295-21ssh11:40:001
377522599614,50cyclictest3849295-21ssh11:40:001
3775225996122,35cyclictest3825256-21rm10:46:131
377522599612,17cyclictest3800104-21ssh09:50:391
3775225996121,35cyclictest23-21ksoftirqd/111:07:561
377522599611,9cyclictest3865863-21ssh12:16:001
377522599611,9cyclictest3840903-21ssh11:20:521
377522599611,9cyclictest3822610-21ssh10:40:331
377522599611,9cyclictest3822610-21ssh10:40:331
377522599611,12cyclictest3830792-21ssh10:58:521
377522599611,10cyclictest3875619-21ssh12:37:311
377522599611,10cyclictest3815225-21ssh10:24:431
3775221996124,34cyclictest12-21ksoftirqd/009:28:130
377522599602,7cyclictest3780839-21sshd08:57:241
377522599602,16cyclictest3872959-21ssh12:31:421
377522599602,15cyclictest3818956-21ssh10:32:451
377522599602,15cyclictest3818956-21ssh10:32:451
377522599602,15cyclictest3783950-21ssh09:15:031
377522599601,14cyclictest3775974-21munin-run07:25:011
377523499597,11cyclictest37-21ksoftirqd/311:20:003
377523499597,11cyclictest37-21ksoftirqd/311:20:003
3775234995912,6cyclictest131rcu_preempt08:00:023
3775230995917,9cyclictest30-21ksoftirqd/212:05:582
3775230995916,10cyclictest30-21ksoftirqd/212:26:422
377522599593,14cyclictest3778033-21munin-run08:05:001
377522199591,2cyclictest0-21swapper/012:15:020
377522199591,2cyclictest0-21swapper/012:15:010
377523499587,5cyclictest131rcu_preempt10:55:003
3775230995849,5cyclictest3831240-21kworker/2:311:40:232
3775230995816,9cyclictest30-21ksoftirqd/212:31:372
3775230995815,9cyclictest30-21ksoftirqd/210:20:052
3775230995815,8cyclictest30-21ksoftirqd/210:30:012
3775230995813,10cyclictest30-21ksoftirqd/210:19:062
377522599585,4cyclictest0-21swapper/107:29:151
377522599582,11cyclictest3776434-21modprobe07:31:241
377522599582,11cyclictest3776434-21modprobe07:31:241
3775225995819,33cyclictest275-21dbus-daemon08:40:001
3775225995818,35cyclictest275-21dbus-daemon09:25:021
3775225995818,35cyclictest275-21dbus-daemon09:25:011
377522599581,15cyclictest3833518-21ssh11:05:001
3775221995827,27cyclictest3838077-21ssh11:15:010
3775221995827,16cyclictest3815606-21ssh10:25:020
377523499576,9cyclictest37-21ksoftirqd/310:29:303
3775234995711,5cyclictest131rcu_preempt11:10:003
3775230995713,8cyclictest30-21ksoftirqd/209:53:132
377522599572,7cyclictest3779040-21sshd08:20:241
377522599572,21cyclictest3828327-21ssh10:53:121
377522599572,12cyclictest3780076-21sshd08:40:241
377522599571,9cyclictest3845423-21ssh11:30:491
3775225995715,34cyclictest309-21in:imuxsock08:35:001
377522599571,13cyclictest364-21snmpd07:36:021
377522599571,11cyclictest3852072-21sh11:45:351
377522599571,10cyclictest3781410-21ssh09:10:001
3775221995731,11cyclictest3784503-21ssh09:16:300
3775221995720,32cyclictest16850irq/134-800000009:15:020
3775221995720,32cyclictest16850irq/134-800000009:15:020
377523499562,14cyclictest37-21ksoftirqd/312:35:033
3775234995614,24cyclictest3795135-21kworker/3:110:25:023
3775230995614,8cyclictest30-21ksoftirqd/210:50:542
377522599563,12cyclictest3467602-21systemd-journal07:15:001
377522599561,19cyclictest3780366-21cut08:50:001
3775221995625,14cyclictest3797621-21ssh09:45:020
3775221995624,29cyclictest12-21ksoftirqd/009:37:000
377523499558,7cyclictest131rcu_preempt10:38:053
377523499557,6cyclictest131rcu_preempt10:14:593
3775230995516,8cyclictest30-21ksoftirqd/211:19:092
3775230995516,8cyclictest30-21ksoftirqd/211:19:082
3775230995516,10cyclictest30-21ksoftirqd/210:36:382
3775230995514,9cyclictest30-21ksoftirqd/212:35:422
3775230995514,8cyclictest30-21ksoftirqd/209:24:532
3775230995514,8cyclictest30-21ksoftirqd/209:24:522
3775230995512,12cyclictest3791015-21ssh09:30:382
3775230995511,9cyclictest30-21ksoftirqd/212:04:042
3775230995511,9cyclictest30-21ksoftirqd/212:04:032
3775230995511,7cyclictest3776466-21/usr/sbin/munin07:35:012
3775230995511,7cyclictest3776466-21/usr/sbin/munin07:35:012
3775225995525,24cyclictest0-21swapper/108:15:011
377522599552,11cyclictest364-21snmpd07:41:171
377522599551,8cyclictest3467602-21systemd-journal09:32:121
377522599551,11cyclictest3843848-21ssh11:27:261
3775221995529,11cyclictest3845659-21ssh11:31:360
3775221995528,11cyclictest3838458-21ssh11:15:310
3775221995528,11cyclictest3838458-21ssh11:15:300
3775221995527,12cyclictest3818180-21ssh10:30:510
3775221995527,12cyclictest3818180-21ssh10:30:500
3775221995525,14cyclictest3871892-21ssh12:29:280
3775221995516,11cyclictest3781552-21latency_hist09:10:020
3775234995420,16cyclictest37-21ksoftirqd/311:41:563
3775234995417,10cyclictest37-21ksoftirqd/309:50:023
3775234995417,10cyclictest37-21ksoftirqd/309:50:023
3775234995412,6cyclictest131rcu_preempt12:09:593
3775234995410,4cyclictest131rcu_preempt10:07:493
3775234995410,4cyclictest131rcu_preempt10:07:483
377522599542,5cyclictest364-21snmpd08:17:211
377522199547,10cyclictest12-21ksoftirqd/010:36:100
3775221995427,12cyclictest3843649-21ssh11:27:020
3775221995425,25cyclictest12-21ksoftirqd/011:06:150
377522199542,45cyclictest3856133-21modprobe11:55:000
3775221995424,14cyclictest3848645-21ssh11:38:240
3775221995424,14cyclictest3848645-21ssh11:38:230
3775221995418,32cyclictest16850irq/134-800000011:49:460
3775221995418,31cyclictest12-21ksoftirqd/012:20:020
377523499539,5cyclictest131rcu_preempt09:51:293
3775234995331,15cyclictest1-21systemd11:05:013
3775230995314,10cyclictest30-21ksoftirqd/211:05:522
377522599531,8cyclictest3780622-21systemd-run08:55:011
3775221995349,2cyclictest0-21swapper/007:40:020
3775221995327,11cyclictest3805976-21ssh10:04:030
3775221995314,10cyclictest12-21ksoftirqd/011:57:240
3775234995210,5cyclictest131rcu_preempt09:31:213
377522599522,14cyclictest275-21dbus-daemon08:00:011
377522199529,33cyclictest12-21ksoftirqd/012:32:440
3775221995232,16cyclictest16850irq/134-800000012:40:000
3775221995225,13cyclictest38289511ssh10:54:440
3775221995223,13cyclictest3831710-21ssh11:00:330
3775221995216,31cyclictest16850irq/134-800000009:23:380
3775221995216,31cyclictest16850irq/134-800000009:23:380
377523499519,6cyclictest131rcu_preempt10:46:023
377523499517,6cyclictest37-21ksoftirqd/307:45:013
377523499517,6cyclictest131rcu_preempt09:10:163
377523499517,6cyclictest131rcu_preempt09:10:163
377523499517,5cyclictest131rcu_preempt12:13:293
377523499517,5cyclictest131rcu_preempt12:13:283
3775234995113,6cyclictest131rcu_preempt11:29:003
3775234995112,6cyclictest131rcu_preempt11:40:013
3775234995112,6cyclictest131rcu_preempt11:40:003
3775234995112,6cyclictest131rcu_preempt09:16:193
3775234995110,6cyclictest131rcu_preempt11:55:033
3775230995142,6cyclictest3831240-21kworker/2:311:54:252
3775230995141,5cyclictest3831240-21kworker/2:311:21:252
3775230995136,12cyclictest3846880-21ssh11:34:392
3775230995113,7cyclictest30-21ksoftirqd/209:44:082
3775230995112,8cyclictest30-21ksoftirqd/211:49:572
377522199518,9cyclictest12-21ksoftirqd/012:19:590
3775221995126,11cyclictest3842487-21ssh11:24:410
3775221995123,12cyclictest3811998-21ssh10:17:190
3775221995122,13cyclictest3798679-21ssh09:47:510
3775221995122,13cyclictest3798679-21ssh09:47:500
3775221995120,16cyclictest3824962-21ssh10:45:350
3775221995114,33cyclictest16850irq/134-800000009:51:290
377523499509,6cyclictest131rcu_preempt12:02:403
377523499509,6cyclictest131rcu_preempt12:02:393
377523499509,6cyclictest131rcu_preempt08:35:013
377523499505,6cyclictest131rcu_preempt10:42:433
377523499505,6cyclictest131rcu_preempt10:42:433
3775234995010,7cyclictest131rcu_preempt09:58:033
3775234995010,6cyclictest131rcu_preempt08:44:593
377523099508,8cyclictest30-21ksoftirqd/210:03:422
3775221995026,9cyclictest3850544-21ssh11:42:220
3775221995015,31cyclictest16850irq/134-800000010:44:560
3775221995015,31cyclictest16850irq/134-800000010:44:560
3775221995013,21cyclictest16850irq/134-800000011:00:000
377523499498,5cyclictest131rcu_preempt12:16:053
377523499497,5cyclictest131rcu_preempt09:38:393
377523499497,4cyclictest131rcu_preempt09:27:523
377523499495,6cyclictest131rcu_preempt09:40:303
3775234994935,10cyclictest3831213-21readlink11:00:013
3775234994913,6cyclictest131rcu_preempt09:20:173
3775234994913,6cyclictest131rcu_preempt09:20:173
3775234994912,5cyclictest131rcu_preempt11:30:333
3775234994912,5cyclictest131rcu_preempt07:15:003
3775234994910,6cyclictest131rcu_preempt10:04:283
3775234994910,5cyclictest131rcu_preempt12:29:173
3775230994940,5cyclictest3776680-21kworker/2:109:17:582
3775221994915,9cyclictest12-21ksoftirqd/010:23:410
377522199491,44cyclictest38025141ssh09:56:070
3775221994914,31cyclictest16850irq/134-800000010:05:130
3775221994914,31cyclictest16850irq/134-800000010:05:130
3775221994913,32cyclictest16850irq/134-800000012:01:310
3775221994913,32cyclictest16850irq/134-800000012:01:300
3775221994911,34cyclictest12-21ksoftirqd/007:55:020
377523499488,5cyclictest131rcu_preempt10:17:523
377523499487,7cyclictest37-21ksoftirqd/311:20:473
377523499487,7cyclictest37-21ksoftirqd/311:13:033
377523499484,5cyclictest131rcu_preempt10:34:553
377523499484,5cyclictest131rcu_preempt10:34:553
3775234994833,11cyclictest1-21systemd09:10:003
377523099488,8cyclictest30-21ksoftirqd/212:16:192
3775230994839,6cyclictest3776680-21kworker/2:109:39:342
377522599482,14cyclictest3775310-21latency_hist07:10:031
377522199482,7cyclictest12-21ksoftirqd/008:44:590
3775221994813,31cyclictest16850irq/134-800000010:12:120
3775221994812,32cyclictest16850irq/134-800000009:32:470
377523499476,5cyclictest131rcu_preempt12:23:083
377523499473,8cyclictest37-21ksoftirqd/308:55:013
3775234994711,5cyclictest131rcu_preempt11:47:363
3775230994731,12cyclictest30-21ksoftirqd/209:30:002
377522199471,41cyclictest38614031ssh12:05:560
3775234994632,10cyclictest3467602-21systemd-journal07:35:013
3775234994632,10cyclictest3467602-21systemd-journal07:35:003
3775234994615,5cyclictest131rcu_preempt09:05:033
3775234994615,5cyclictest131rcu_preempt09:05:023
3775234994611,5cyclictest131rcu_preempt11:56:133
3775230994626,15cyclictest3797670-21ssh09:45:072
3775230994626,15cyclictest3797670-21ssh09:45:072
3775230994535,7cyclictest3776999-21/usr/sbin/munin07:45:022
3775230994529,14cyclictest3833013-21ssh11:03:362
3775230994529,13cyclictest3856637-21ssh11:55:452
3775230994529,12cyclictest3837469-21ssh11:13:362
3775230994528,13cyclictest3818313-21ssh10:31:072
3775230994528,13cyclictest3818313-21ssh10:31:072
3775230994528,13cyclictest3802297-21ssh09:55:262
3775230994510,6cyclictest30-21ksoftirqd/212:24:132
377522199455,7cyclictest12-21ksoftirqd/007:50:000
3775221994528,9cyclictest12-21ksoftirqd/008:28:510
377523499448,5cyclictest131rcu_preempt08:26:493
3775234994421,6cyclictest361rcuc/308:25:023
3775230994433,9cyclictest3809461-21ssh10:11:372
3775230994431,9cyclictest309-21in:imuxsock09:00:012
3775230994430,11cyclictest275-21dbus-daemon08:15:012
3775230994427,14cyclictest3826648-21ssh10:49:312
3775230994425,13cyclictest30-21ksoftirqd/207:55:002
3775230994416,22cyclictest3779258-21latency_hist08:25:022
3775221994431,10cyclictest3467602-21systemd-journal08:21:240
377522199441,4cyclictest131rcu_preempt07:34:170
377522199441,4cyclictest131rcu_preempt07:34:170
377522199441,38cyclictest3781112-21sshd09:00:240
377522199441,38cyclictest3781112-21sshd09:00:240
377523499438,6cyclictest131rcu_preempt08:10:243
377522199433,9cyclictest1-21systemd07:20:010
3775221994324,15cyclictest3776185-21sshd07:27:250
3775234994211,6cyclictest131rcu_preempt08:40:013
3775230994233,7cyclictest30-21ksoftirqd/209:05:002
3775230994233,7cyclictest30-21ksoftirqd/209:05:002
3775230994229,9cyclictest284-21systemd-logind08:50:002
3775230994227,12cyclictest3775931-21sshd07:21:242
3775230994227,11cyclictest30-21ksoftirqd/208:40:002
3775230994219,17cyclictest3778751-21modprobe08:15:042
3775230994216,22cyclictest3781543-21sed09:10:012
377523499415,5cyclictest131rcu_preempt08:05:013
3775234994110,7cyclictest131rcu_preempt07:15:193
3775234994110,5cyclictest131rcu_preempt07:37:303
3775230994129,7cyclictest309-21in:imuxsock07:40:002
3775230994128,10cyclictest3777736-21sshd07:59:242
377522199412,35cyclictest3778273-21cron08:10:000
3775221994121,13cyclictest0-21swapper/008:51:580
377523499404,7cyclictest37-21ksoftirqd/307:54:203
377523499404,5cyclictest131rcu_preempt08:15:243
3775230994027,10cyclictest3779564-21sshd08:34:272
3775230994021,11cyclictest30-21ksoftirqd/207:30:012
3775221994027,8cyclictest12-21ksoftirqd/008:00:010
377522199401,34cyclictest3779829-21kworker/u8:408:40:000
377523499396,8cyclictest37-21ksoftirqd/307:50:013
3775221993932,4cyclictest16950irq/224-800000008:04:240
3775221993932,4cyclictest16950irq/224-800000007:15:010
3775221993928,8cyclictest12-21ksoftirqd/008:20:000
377522199392,7cyclictest1-21systemd07:40:010
377522199392,6cyclictest131rcu_preempt07:10:020
377523499384,4cyclictest131rcu_preempt08:49:023
3775234993815,6cyclictest131rcu_preempt08:05:243
3775230993830,4cyclictest3780617-21munin-run08:55:002
3775230993829,6cyclictest3778286-21munin-run08:10:002
3775230993825,10cyclictest3775340-21cat07:10:032
3775230993822,11cyclictest3780126-21munin-run08:45:002
377522199381,8cyclictest3779570-21modprobe08:35:000
3775234993710,5cyclictest131rcu_preempt08:57:063
377523499361,9cyclictest0-21swapper/307:26:243
3775230993628,4cyclictest3777249-21munin-run07:50:002
3775230993626,7cyclictest3775394-21sshd07:12:242
3775230993618,14cyclictest1-21systemd08:30:012
3775230993520,11cyclictest275-21dbus-daemon07:20:002
377522199359,5cyclictest131rcu_preempt08:46:540
377523499345,5cyclictest131rcu_preempt07:21:043
3775230993420,10cyclictest364-21snmpd08:00:472
3775234993020,7cyclictest37-21ksoftirqd/307:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional