You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-13 - 20:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Apr 13, 2026 12:45:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3354611999888,6cyclictest0-21swapper/009:10:020
3354611999888,6cyclictest0-21swapper/009:10:020
3354625998035,40cyclictest3467602-21systemd-journal12:09:573
3354625997330,38cyclictest309-21in:imuxsock09:00:003
3354625996728,35cyclictest3359292-21tr08:40:003
3354625996724,36cyclictest3396225-21sh10:31:533
3354625996723,5cyclictest3432721-21sh11:56:283
3354611996726,36cyclictest309-21in:imuxsock10:00:430
3354625996624,38cyclictest3362994-21cat09:15:013
3354625996526,35cyclictest37-21ksoftirqd/311:53:523
3354625996514,10cyclictest3401873-21sed10:45:023
3354625996425,35cyclictest3382315-21latency_hist10:00:033
3354625996328,31cyclictest3425251-21sh11:39:353
3354625996323,9cyclictest37-21ksoftirqd/312:29:493
3354625996321,11cyclictest37-21ksoftirqd/311:45:013
335461199634,50cyclictest3373857-21ssh09:40:010
3354625996223,6cyclictest3376368-21sh09:45:443
3354611996223,34cyclictest12-21ksoftirqd/011:12:550
3354625996132,23cyclictest3444903-21munin-run12:25:003
3354625996125,33cyclictest37-21ksoftirqd/308:05:013
3354625996121,4cyclictest3367734-21sh09:25:423
3354611996136,19cyclictest3408212-21rm11:00:000
3354611996136,19cyclictest3408212-21rm11:00:000
335461199613,39cyclictest19350irq/34-mmc112:35:020
335461199613,39cyclictest19350irq/34-mmc112:35:020
3354611996127,26cyclictest3444933-21sort12:25:020
3354611996121,36cyclictest12-21ksoftirqd/009:19:060
335461199611,57cyclictest0-21swapper/010:10:010
3354611996112,45cyclictest12-21ksoftirqd/010:39:180
3354625996025,31cyclictest3358683-21sshd08:25:573
3354625996025,31cyclictest3358683-21sshd08:25:573
3354625996024,5cyclictest3380106-21sh09:54:523
3354611996046,8cyclictest16950irq/224-800000009:11:590
335462599593,36cyclictest3378123-21kworker/u8:010:15:023
3354625995926,29cyclictest3355906-21sed07:35:013
3354625995923,7cyclictest37-21ksoftirqd/309:16:413
3354625995920,11cyclictest37-21ksoftirqd/309:33:303
335461199595,38cyclictest12-21ksoftirqd/009:48:490
3354611995939,16cyclictest16850irq/134-800000012:10:010
3354611995927,16cyclictest3375549-21ssh09:44:090
3354611995927,16cyclictest3375549-21ssh09:44:090
3354611995911,29cyclictest16950irq/224-800000011:15:040
3354611995911,29cyclictest16950irq/224-800000011:15:030
3354611995822,7cyclictest12-21ksoftirqd/011:45:020
3354625995718,7cyclictest3357651-21sshd08:08:573
3354625995713,9cyclictest3399631-21systemd-run10:40:013
3354611995733,18cyclictest3442727-21ssh12:20:000
3354611995726,15cyclictest3439574-21ssh12:12:260
3354611995726,14cyclictest3428372-21ssh11:46:150
335461199572,4cyclictest131rcu_preempt12:29:240
3354625995625,27cyclictest3358188-21sendmail08:20:003
3354611995627,13cyclictest3435872-21ssh12:03:590
3354611995623,17cyclictest3394585-21ssh10:28:110
3354611995614,38cyclictest12-21ksoftirqd/010:51:350
335462599553,18cyclictest3359551-21munin-plugin-st08:45:003
3354625995523,4cyclictest3360841-21diskmemload09:41:323
3354625995523,4cyclictest3360841-21diskmemload09:41:323
3354625995522,8cyclictest37-21ksoftirqd/311:28:163
3354625995521,9cyclictest3373793-21sed09:40:013
3354625995518,9cyclictest3357167-21sendmail-msp08:00:003
3354625995515,9cyclictest3467602-21systemd-journal07:55:013
3354611995535,14cyclictest3397449-21latency_hist10:35:010
335462599549,14cyclictest3416865-21munin-plugin-st11:20:003
335462599549,14cyclictest3416865-21munin-plugin-st11:20:003
3354625995425,26cyclictest37-21ksoftirqd/312:20:023
3354625995421,5cyclictest3355256-21latency_hist07:20:023
3354625995417,9cyclictest37-21ksoftirqd/310:49:483
3354625995416,10cyclictest3467602-21systemd-journal08:35:003
3354621995446,5cyclictest3357180-21kworker/2:010:00:002
3354621995444,6cyclictest3378360-21kworker/2:210:42:362
3354621995444,6cyclictest3378360-21kworker/2:210:21:372
3354611995418,7cyclictest12-21ksoftirqd/008:40:010
3354625995331,17cyclictest275-21dbus-daemon08:50:003
3354625995327,23cyclictest37-21ksoftirqd/311:06:133
3354625995321,5cyclictest3360555-21sshd09:01:573
3354625995318,7cyclictest37-21ksoftirqd/312:40:003
3354625995318,7cyclictest37-21ksoftirqd/312:39:593
3354625995318,5cyclictest3390543-21sh10:19:003
3354625995317,6cyclictest309-21in:imuxsock10:30:003
3354625995315,6cyclictest3406085-21latency_hist10:55:013
3354621995345,5cyclictest3357180-21kworker/2:008:40:022
3354621995344,6cyclictest3378360-21kworker/2:211:01:282
3354621995344,6cyclictest3357180-21kworker/2:009:23:512
3354621995344,5cyclictest3395316-21kworker/2:311:46:572
3354621995342,7cyclictest3395316-21kworker/2:311:51:312
335461199538,6cyclictest131rcu_preempt11:27:340
335461199533,45cyclictest3425455-21systemctl11:40:000
3354611995326,11cyclictest3389273-21ssh10:15:420
3354611995321,27cyclictest12-21ksoftirqd/010:00:000
3354611995317,14cyclictest12-21ksoftirqd/011:35:010
3354625995222,5cyclictest3440498-21sh12:15:013
3354625995221,9cyclictest37-21ksoftirqd/310:55:163
3354625995221,9cyclictest37-21ksoftirqd/310:55:163
3354625995211,8cyclictest309-21in:imuxsock07:45:003
3354621995244,5cyclictest3425512-21kworker/2:012:20:022
3354621995244,5cyclictest3357180-21kworker/2:009:32:162
3354621995243,6cyclictest3395316-21kworker/2:312:37:122
3354621995243,6cyclictest3395316-21kworker/2:312:37:112
3354621995243,6cyclictest3395316-21kworker/2:312:31:182
3354621995243,6cyclictest3378360-21kworker/2:210:33:102
3354621995243,6cyclictest3357180-21kworker/2:009:10:432
3354621995242,7cyclictest3378360-21kworker/2:210:40:012
3354616995234,14cyclictest275-21dbus-daemon12:05:011
3354611995242,7cyclictest12-21ksoftirqd/010:45:020
3354611995226,11cyclictest3387572-21ssh10:11:490
335461199522,14cyclictest3358893-21cat08:30:020
335461199522,14cyclictest3358893-21cat08:30:020
3354611995210,6cyclictest131rcu_preempt09:55:010
335462599512,18cyclictest3429477-21ssh11:49:143
3354621995143,5cyclictest3357180-21kworker/2:010:01:402
3354621995142,6cyclictest3378360-21kworker/2:210:48:412
3354621995141,6cyclictest3357180-21kworker/2:009:50:502
335461699511,5cyclictest131rcu_preempt07:30:021
335461199512,19cyclictest275-21dbus-daemon12:00:020
3354611995115,31cyclictest12-21ksoftirqd/010:50:020
3354611995113,8cyclictest12-21ksoftirqd/011:20:580
335461199511,26cyclictest131rcu_preempt10:20:570
3354625995020,9cyclictest37-21ksoftirqd/309:25:003
3354625995011,12cyclictest37-21ksoftirqd/311:00:113
3354621995044,4cyclictest3395316-21kworker/2:311:45:032
3354621995042,5cyclictest3395316-21kworker/2:311:57:042
3354621995042,5cyclictest3395316-21kworker/2:311:19:392
3354621995042,5cyclictest3395316-21kworker/2:311:19:392
3354621995042,5cyclictest3378360-21kworker/2:211:09:082
3354621995041,6cyclictest3378360-21kworker/2:210:16:532
3354621995041,6cyclictest3357180-21kworker/2:009:15:552
3354616995033,12cyclictest275-21dbus-daemon08:20:001
335461199502,9cyclictest3360314-21munin-run09:00:000
3354611995014,32cyclictest151rcuc/009:30:090
3354621994941,5cyclictest3425512-21kworker/2:012:29:122
3354621994941,5cyclictest3425512-21kworker/2:012:11:392
3354621994941,5cyclictest3395316-21kworker/2:312:02:542
3354621994941,5cyclictest3378360-21kworker/2:211:11:152
3354621994940,6cyclictest3395316-21kworker/2:311:31:092
3354621994940,6cyclictest3357180-21kworker/2:009:10:022
3354621994940,6cyclictest3357180-21kworker/2:009:10:022
3354621994937,8cyclictest3467602-21systemd-journal08:10:002
3354621994933,13cyclictest30-21ksoftirqd/208:00:002
335461699495,29cyclictest3410361-21kworker/u8:211:55:021
3354611994937,8cyclictest12-21ksoftirqd/008:35:020
3354611994929,16cyclictest3409612-21sshd11:02:570
3354611994917,28cyclictest12-21ksoftirqd/011:50:590
3354621994840,5cyclictest3378360-21kworker/2:210:54:442
3354621994839,6cyclictest3357180-21kworker/2:009:35:142
3354621994839,5cyclictest3357180-21kworker/2:008:55:012
3354621994827,5cyclictest30-21ksoftirqd/210:26:172
335461199482,4cyclictest131rcu_preempt08:10:010
3354611994821,23cyclictest12-21ksoftirqd/009:28:000
3354625994718,7cyclictest37-21ksoftirqd/311:21:333
3354621994739,5cyclictest3395316-21kworker/2:312:09:092
3354621994739,5cyclictest3357180-21kworker/2:009:42:182
3354621994739,5cyclictest3357180-21kworker/2:009:42:182
3354621994739,5cyclictest3357180-21kworker/2:009:25:072
3354621994737,6cyclictest3395316-21kworker/2:311:24:122
3354611994716,7cyclictest131rcu_preempt07:15:000
3354611994716,7cyclictest131rcu_preempt07:15:000
3354611994714,5cyclictest131rcu_preempt08:20:000
3354625994620,5cyclictest3423435-21sed11:35:013
3354625994618,7cyclictest37-21ksoftirqd/310:25:033
3354621994638,5cyclictest3378360-21kworker/2:210:58:172
3354621994638,5cyclictest3378360-21kworker/2:210:58:172
3354621994638,5cyclictest3357180-21kworker/2:009:47:352
3354621994638,5cyclictest3357180-21kworker/2:009:02:332
3354621994638,5cyclictest3342044-21kworker/2:107:21:102
3354621994630,12cyclictest275-21dbus-daemon08:35:002
3354616994628,14cyclictest1-21systemd11:20:021
3354616994628,14cyclictest1-21systemd11:20:011
335461699462,3cyclictest131rcu_preempt12:25:021
335461199463,11cyclictest12-21ksoftirqd/007:45:000
3354611994620,11cyclictest3411372-21ssh11:07:040
335461199461,41cyclictest3365407-21ssh09:20:140
3354621994536,5cyclictest3357180-21kworker/2:008:25:002
3354616994531,10cyclictest3372006-21sshd09:35:391
3354616994530,11cyclictest3389752-21cron10:17:001
3354616994529,11cyclictest309-21in:imuxsock10:40:031
335461199458,5cyclictest131rcu_preempt07:47:520
3354611994536,6cyclictest19350irq/34-mmc112:30:220
3354621994436,5cyclictest3378360-21kworker/2:210:15:032
3354621994436,5cyclictest3378360-21kworker/2:210:08:302
3354621994436,5cyclictest3342044-21kworker/2:107:20:022
3354616994430,11cyclictest3371676-21sed09:35:021
3354616994426,12cyclictest275-21dbus-daemon07:45:001
335461699442,4cyclictest131rcu_preempt10:23:271
3354611994435,6cyclictest12-21ksoftirqd/008:25:010
3354611994433,7cyclictest19350irq/34-mmc109:05:000
3354611994428,13cyclictest12-21ksoftirqd/007:35:010
3354625994329,10cyclictest1-21systemd07:50:013
3354625994326,13cyclictest37-21ksoftirqd/310:06:183
3354621994335,5cyclictest3395316-21kworker/2:311:37:482
3354621994335,5cyclictest3357180-21kworker/2:008:49:572
3354621994335,5cyclictest3342044-21kworker/2:107:15:022
3354621994335,5cyclictest3342044-21kworker/2:107:15:022
3354621994334,6cyclictest3342044-21kworker/2:108:05:022
3354621994334,5cyclictest3425512-21kworker/2:012:22:542
3354621994324,13cyclictest275-21dbus-daemon07:40:012
3354621994324,13cyclictest275-21dbus-daemon07:40:012
335461699434,3cyclictest131rcu_preempt10:50:011
3354616994335,5cyclictest3410371-21modprobe11:05:001
335461699433,3cyclictest131rcu_preempt09:13:431
3354616994327,12cyclictest23-21ksoftirqd/110:30:001
335461699432,26cyclictest3359181-21kworker/u8:209:15:021
3354616994310,10cyclictest131rcu_preempt10:59:541
3354616994310,10cyclictest131rcu_preempt10:59:541
3354611994336,4cyclictest16950irq/224-800000008:05:010
3354611994323,15cyclictest3467602-21systemd-journal07:20:000
3354611994315,7cyclictest131rcu_preempt07:55:020
3354611994315,24cyclictest12-21ksoftirqd/008:50:010
3354625994230,10cyclictest37-21ksoftirqd/312:34:193
3354621994235,4cyclictest3357180-21kworker/2:008:56:332
3354621994226,12cyclictest30-21ksoftirqd/211:26:432
335462199421,3cyclictest131rcu_preempt08:30:032
335462199421,3cyclictest131rcu_preempt08:30:022
335461699424,3cyclictest131rcu_preempt12:32:581
3354616994231,8cyclictest3412489-21sshd11:09:571
3354616994230,9cyclictest1-21systemd07:35:001
3354616994227,11cyclictest23-21ksoftirqd/112:16:131
3354616994227,10cyclictest3406046-21cron10:55:001
3354616994227,10cyclictest309-21in:imuxsock08:45:021
335461699421,6cyclictest131rcu_preempt12:00:001
335461699421,3cyclictest131rcu_preempt11:40:501
335461699421,3cyclictest131rcu_preempt11:32:161
3354616994212,10cyclictest23-21ksoftirqd/109:25:011
3354621994131,6cyclictest3358186-21sh08:20:002
335461699415,3cyclictest131rcu_preempt10:40:001
335461699413,3cyclictest131rcu_preempt08:05:011
3354616994130,8cyclictest23-21ksoftirqd/110:05:511
3354616994129,9cyclictest23-21ksoftirqd/112:27:011
335461699412,3cyclictest131rcu_preempt09:56:141
3354611994131,7cyclictest12-21ksoftirqd/007:25:010
3354611994129,8cyclictest12-21ksoftirqd/008:55:000
3354611994127,10cyclictest309-21in:imuxsock07:40:000
3354611994127,10cyclictest309-21in:imuxsock07:40:000
335462599402,8cyclictest3360843-21modprobe09:10:003
335462599402,8cyclictest3360843-21modprobe09:10:003
335461699403,4cyclictest131rcu_preempt09:43:351
335461699403,4cyclictest131rcu_preempt09:43:341
3354616994031,6cyclictest23-21ksoftirqd/107:55:021
3354616994030,6cyclictest309-21in:imuxsock09:05:011
3354616994027,10cyclictest23-21ksoftirqd/108:10:011
335461699401,3cyclictest131rcu_preempt12:13:271
3354616994012,5cyclictest131rcu_preempt09:27:141
335461699401,17cyclictest0-21swapper/109:49:111
3354611994025,12cyclictest3357901-21sshd08:12:570
3354625993925,10cyclictest3435000-21ssh12:01:453
3354625993917,9cyclictest3384587-21ssh10:05:023
3354621993926,9cyclictest30-21ksoftirqd/208:45:012
3354621993924,11cyclictest30-21ksoftirqd/207:45:002
3354621993924,11cyclictest1-21systemd08:15:012
3354616993926,10cyclictest23-21ksoftirqd/111:50:021
3354616993926,10cyclictest23-21ksoftirqd/111:24:041
335461699392,4cyclictest131rcu_preempt12:10:001
3354616993923,12cyclictest3467602-21systemd-journal08:30:021
3354611993926,10cyclictest12-21ksoftirqd/008:00:000
3354611993911,5cyclictest131rcu_preempt08:42:480
3354625993827,8cyclictest37-21ksoftirqd/311:13:183
335462199381,3cyclictest131rcu_preempt07:10:022
335461699383,3cyclictest131rcu_preempt09:10:001
335461699383,3cyclictest131rcu_preempt09:10:001
3354616993824,10cyclictest3467602-21systemd-journal08:42:311
335461699382,3cyclictest131rcu_preempt11:10:561
3354616993822,12cyclictest23-21ksoftirqd/108:22:541
3354616993821,13cyclictest275-21dbus-daemon08:40:001
335461699381,3cyclictest131rcu_preempt09:53:021
335461699381,11cyclictest131rcu_preempt09:00:001
3354621993728,6cyclictest30-21ksoftirqd/207:50:012
3354621993719,4cyclictest30-21ksoftirqd/207:30:022
335461699373,3cyclictest131rcu_preempt10:05:021
335461699373,12cyclictest131rcu_preempt11:38:321
3354616993728,6cyclictest23-21ksoftirqd/112:40:021
3354616993728,6cyclictest23-21ksoftirqd/112:40:021
335461699372,4cyclictest131rcu_preempt10:31:481
335461699372,3cyclictest131rcu_preempt11:27:221
335461699371,4cyclictest131rcu_preempt08:15:021
335461199373,9cyclictest12-21ksoftirqd/007:30:010
3354611993713,5cyclictest131rcu_preempt07:10:020
335462599365,9cyclictest37-21ksoftirqd/307:40:003
335462599365,9cyclictest37-21ksoftirqd/307:40:003
335461699363,3cyclictest131rcu_preempt07:25:011
3354616993624,8cyclictest3356808-21sed07:50:021
3354616993620,12cyclictest310-21in:imklog10:11:021
335462599352,19cyclictest3355604-21munin-run07:30:003
3354621993528,4cyclictest3356895-21sh07:55:012
335461699353,3cyclictest131rcu_preempt07:40:001
335461699353,3cyclictest131rcu_preempt07:40:001
335461699352,3cyclictest131rcu_preempt08:55:021
335461699345,23cyclictest3354618-21latency_hist07:10:011
335461699343,3cyclictest131rcu_preempt07:20:001
3354616993422,10cyclictest23-21ksoftirqd/108:30:021
3354616993422,10cyclictest23-21ksoftirqd/108:30:021
3354621993325,5cyclictest3342044-21kworker/2:107:35:022
335461699332,10cyclictest131rcu_preempt08:00:011
3354625993225,4cyclictest3355348-21latency_hist07:25:023
3354625993220,10cyclictest37-21ksoftirqd/308:55:003
335461699321,4cyclictest131rcu_preempt07:14:561
335461699321,4cyclictest131rcu_preempt07:14:561
3354625993123,4cyclictest3354687-21latency_hist07:10:023
3354625993118,9cyclictest3354983-21date07:15:023
3354625993118,9cyclictest3354983-21date07:15:023
3354625993020,7cyclictest3358449-21sshd08:24:573
335462599283,12cyclictest150-21pr/ttyS208:15:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional