You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-03 - 11:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Fri Jul 03, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3887540999470,15cyclictest23-21ksoftirqd/123:20:031
3887535998779,6cyclictest12-21ksoftirqd/022:10:020
3887535998729,18cyclictest3967633-21ssh23:45:040
3887535998729,18cyclictest3967633-21ssh23:45:030
3887535998628,12cyclictest3933843-21ssh22:34:290
3887535998427,12cyclictest3908628-21ssh21:41:070
3887535998425,12cyclictest3911616-21ssh21:47:310
3887535998326,12cyclictest3964067-21ssh23:37:500
3887535998326,12cyclictest3964067-21ssh23:37:490
3887535998126,13cyclictest3987445-21ssh00:26:500
3887535998028,11cyclictest3993106-21ssh00:38:530
3887535998028,11cyclictest3993106-21ssh00:38:530
3887535997924,15cyclictest3916660-21ssh21:58:100
388753599782,38cyclictest3931826-21latency_hist22:30:020
388753599781,42cyclictest3956316-21ssh23:21:260
3887545997756,4cyclictest30-21ksoftirqd/220:20:022
3887535997734,37cyclictest12-21ksoftirqd/000:00:000
3887535997729,43cyclictest309-21in:imuxsock19:40:010
3887535997724,47cyclictest3976126-21sh00:03:070
3887535997724,47cyclictest3976126-21sh00:03:060
3887535997721,12cyclictest3927618-21ssh22:20:560
3887535997624,15cyclictest3919077-21ssh22:03:090
3887535997527,13cyclictest3906646-21ssh21:37:150
388753599751,38cyclictest3981310-21ssh00:14:120
388753599751,38cyclictest3981310-21ssh00:14:110
3887535997426,13cyclictest3898035-21ssh21:19:140
3887535997423,14cyclictest3913084-21ssh21:50:270
388753599731,42cyclictest3982149-21ssh00:15:400
3887535997218,14cyclictest3970224-21ssh23:50:290
3887535997126,9cyclictest3467602-21systemd-journal22:51:320
3887535997126,12cyclictest3977328-21ssh00:05:320
3887535997126,12cyclictest3953613-21ssh23:15:440
3887535997126,12cyclictest3953613-21ssh23:15:430
3887535997026,13cyclictest3960807-21ssh23:30:500
3887535997021,11cyclictest3985945-21ssh00:23:490
388753599701,5cyclictest3467562-21sshd23:08:110
388755099681,6cyclictest284-21systemd-logind23:25:013
3887535996623,13cyclictest3959309-21ssh23:27:510
388753599661,6cyclictest3890784-21latency_hist20:10:030
3887550996514,11cyclictest37-21ksoftirqd/321:50:013
388753599652,4cyclictest3901685-21sshd21:26:450
388755099644,13cyclictest37-21ksoftirqd/323:40:013
388755099644,13cyclictest37-21ksoftirqd/323:40:013
3887550996415,10cyclictest37-21ksoftirqd/323:10:013
388753599641,5cyclictest3893122-21ls20:55:030
3887535996414,6cyclictest16850irq/134-800000000:30:120
3887535996325,8cyclictest16950irq/224-800000022:38:100
3887535996321,13cyclictest0-21swapper/023:01:160
3887535996319,17cyclictest16850irq/134-800000022:19:070
3887545996252,7cyclictest3893484-21kworker/2:223:04:002
3887535996227,12cyclictest12-21ksoftirqd/021:30:150
3887535996224,5cyclictest309-21in:imuxsock20:15:000
3887535996222,9cyclictest275-21dbus-daemon20:05:010
3887535996222,9cyclictest275-21dbus-daemon20:05:010
388753599621,31cyclictest3923870-21sh22:13:140
388753599612,5cyclictest3900558-21ssh21:24:340
388755099607,11cyclictest37-21ksoftirqd/321:57:123
388754099609,5cyclictest131rcu_preempt22:59:491
3887540996014,6cyclictest131rcu_preempt23:29:151
3887535996049,7cyclictest16950irq/224-800000022:59:590
3887535996018,17cyclictest16850irq/134-800000022:42:580
3887550995916,5cyclictest131rcu_preempt20:59:593
388754099596,6cyclictest131rcu_preempt21:22:161
388754099591,10cyclictest23-21ksoftirqd/122:40:011
3887545995848,7cyclictest3886873-21kworker/2:021:22:512
388754099584,7cyclictest23-21ksoftirqd/123:40:021
388754099584,7cyclictest23-21ksoftirqd/123:40:011
3887535995821,33cyclictest12-21ksoftirqd/019:30:440
388755099574,11cyclictest37-21ksoftirqd/323:04:583
3887550995721,5cyclictest131rcu_preempt21:14:153
3887550995716,5cyclictest131rcu_preempt21:30:353
3887550995714,8cyclictest37-21ksoftirqd/321:28:453
3887550995713,6cyclictest131rcu_preempt21:16:383
3887545995749,5cyclictest3948677-21kworker/2:023:41:332
3887545995749,5cyclictest3893484-21kworker/2:222:03:022
388754099575,12cyclictest23-21ksoftirqd/122:00:271
388754099574,12cyclictest23-21ksoftirqd/122:30:011
3887540995733,15cyclictest23-21ksoftirqd/121:45:021
3887535995725,8cyclictest12-21ksoftirqd/020:00:020
388753599572,31cyclictest3953232-21latency_hist23:15:020
3887535995721,9cyclictest12-21ksoftirqd/021:10:020
388755099567,11cyclictest37-21ksoftirqd/323:19:573
388755099567,11cyclictest37-21ksoftirqd/323:19:573
388755099565,11cyclictest37-21ksoftirqd/321:20:543
388755099564,7cyclictest37-21ksoftirqd/322:15:023
388755099564,15cyclictest37-21ksoftirqd/300:39:153
388755099564,15cyclictest37-21ksoftirqd/300:39:143
3887550995614,5cyclictest131rcu_preempt00:22:423
3887550995613,5cyclictest131rcu_preempt22:55:273
3887540995633,8cyclictest23-21ksoftirqd/122:16:301
3887540995612,7cyclictest131rcu_preempt19:10:021
3887540995612,6cyclictest131rcu_preempt22:52:111
3887550995515,5cyclictest131rcu_preempt22:41:183
3887550995513,5cyclictest131rcu_preempt00:29:243
3887550995511,6cyclictest131rcu_preempt21:50:233
3887545995547,5cyclictest3948677-21kworker/2:023:15:582
3887545995547,5cyclictest3948677-21kworker/2:023:15:572
3887545995546,6cyclictest3948677-21kworker/2:000:31:362
3887545995546,5cyclictest3893484-21kworker/2:222:19:102
3887545995544,7cyclictest3893484-21kworker/2:222:41:452
388754099556,14cyclictest23-21ksoftirqd/121:55:011
388754099554,7cyclictest23-21ksoftirqd/122:15:001
3887540995514,16cyclictest131rcu_preempt21:35:561
388755099548,9cyclictest37-21ksoftirqd/323:25:463
388755099548,8cyclictest37-21ksoftirqd/300:06:543
388755099548,7cyclictest37-21ksoftirqd/300:14:353
388755099548,7cyclictest37-21ksoftirqd/300:14:343
388755099547,8cyclictest37-21ksoftirqd/323:10:573
388755099547,7cyclictest37-21ksoftirqd/322:53:093
388755099547,10cyclictest37-21ksoftirqd/322:50:013
388755099545,7cyclictest37-21ksoftirqd/323:40:523
388755099545,7cyclictest37-21ksoftirqd/321:40:023
3887550995420,11cyclictest37-21ksoftirqd/300:03:423
3887550995420,11cyclictest37-21ksoftirqd/300:03:413
3887550995416,5cyclictest131rcu_preempt23:35:023
3887550995415,10cyclictest37-21ksoftirqd/300:00:003
3887550995413,5cyclictest131rcu_preempt22:17:043
3887550995410,5cyclictest131rcu_preempt23:51:203
3887545995446,5cyclictest3948677-21kworker/2:000:27:012
3887545995446,5cyclictest3948677-21kworker/2:000:11:252
3887545995446,5cyclictest3948677-21kworker/2:000:11:252
3887545995446,5cyclictest3893484-21kworker/2:222:57:142
3887545995440,5cyclictest3886873-21kworker/2:021:16:472
3887540995441,10cyclictest23-21ksoftirqd/121:50:021
3887535995423,28cyclictest12-21ksoftirqd/021:12:160
388755099532,10cyclictest37-21ksoftirqd/322:09:403
3887550995314,6cyclictest131rcu_preempt22:39:333
3887550995313,6cyclictest131rcu_preempt19:10:033
3887550995313,5cyclictest131rcu_preempt22:26:463
3887550995311,6cyclictest131rcu_preempt00:15:353
3887550995310,6cyclictest131rcu_preempt22:01:113
3887545995345,5cyclictest3948677-21kworker/2:023:56:492
3887545995345,5cyclictest3948677-21kworker/2:023:53:042
3887545995345,5cyclictest3948677-21kworker/2:023:39:052
3887545995345,5cyclictest3948677-21kworker/2:023:39:052
3887545995345,5cyclictest3948677-21kworker/2:023:24:192
3887545995344,6cyclictest3886873-21kworker/2:021:11:402
388754099534,5cyclictest131rcu_preempt22:10:031
3887540995311,6cyclictest131rcu_preempt23:04:121
388755099528,7cyclictest37-21ksoftirqd/322:34:373
388755099527,7cyclictest37-21ksoftirqd/322:22:173
388755099526,9cyclictest37-21ksoftirqd/300:33:293
388755099525,7cyclictest37-21ksoftirqd/320:08:443
388755099522,13cyclictest37-21ksoftirqd/323:50:033
388755099522,13cyclictest37-21ksoftirqd/323:50:023
3887545995244,5cyclictest3948677-21kworker/2:023:29:472
3887545995244,5cyclictest3893484-21kworker/2:222:54:402
3887545995244,5cyclictest3893484-21kworker/2:222:14:152
3887545995243,6cyclictest3948677-21kworker/2:023:46:452
3887545995243,6cyclictest3948677-21kworker/2:023:46:442
3887545995243,5cyclictest3948677-21kworker/2:000:09:222
3887545995243,5cyclictest3893484-21kworker/2:222:37:522
3887545995242,6cyclictest3948677-21kworker/2:000:20:032
3887545995242,6cyclictest3946095-21kworker/2:123:13:272
3887540995229,10cyclictest23-21ksoftirqd/123:20:011
3887540995229,10cyclictest23-21ksoftirqd/123:20:001
3887540995228,19cyclictest275-21dbus-daemon00:19:591
3887540995210,5cyclictest131rcu_preempt00:05:011
3887540995210,5cyclictest131rcu_preempt00:05:001
3887550995115,4cyclictest131rcu_preempt19:40:023
3887545995143,5cyclictest3893484-21kworker/2:221:43:032
3887545995142,6cyclictest3948677-21kworker/2:000:22:012
3887545995142,6cyclictest3893484-21kworker/2:221:56:212
3887545995141,6cyclictest3893484-21kworker/2:222:49:132
388754099518,5cyclictest131rcu_preempt21:05:021
388754099515,11cyclictest23-21ksoftirqd/100:26:451
3887540995132,15cyclictest3467602-21systemd-journal22:31:451
3887540995112,5cyclictest131rcu_preempt00:35:001
388754099511,19cyclictest0-21swapper/100:36:321
388754099511,19cyclictest0-21swapper/100:36:321
3887540995111,11cyclictest23-21ksoftirqd/121:30:021
3887540995110,6cyclictest131rcu_preempt21:58:411
388753599511,6cyclictest3966555-21ssh23:43:060
388755099507,10cyclictest37-21ksoftirqd/321:10:013
3887550995035,11cyclictest3890056-21cron20:00:003
3887545995042,5cyclictest3946095-21kworker/2:123:32:002
3887545995042,5cyclictest3893484-21kworker/2:222:26:052
3887545995042,5cyclictest3893484-21kworker/2:221:53:492
3887545995041,6cyclictest3893484-21kworker/2:222:21:322
3887545995041,5cyclictest3946095-21kworker/2:123:05:312
388754099509,5cyclictest131rcu_preempt23:35:011
388754099506,11cyclictest23-21ksoftirqd/120:00:001
388754099505,9cyclictest23-21ksoftirqd/121:35:021
388754099502,13cyclictest23-21ksoftirqd/123:10:561
388754099501,7cyclictest23-21ksoftirqd/123:41:061
388754099501,4cyclictest131rcu_preempt22:22:121
3887540995011,6cyclictest131rcu_preempt23:46:591
3887540995011,6cyclictest131rcu_preempt23:46:591
3887550994918,5cyclictest131rcu_preempt21:41:053
3887545994941,5cyclictest3948677-21kworker/2:000:35:242
3887545994941,5cyclictest3948677-21kworker/2:000:35:232
3887545994941,5cyclictest3893484-21kworker/2:222:32:122
3887545994941,5cyclictest3893484-21kworker/2:221:48:202
3887545994940,5cyclictest3893484-21kworker/2:222:07:242
3887545994940,5cyclictest3886873-21kworker/2:021:26:282
3887545994933,9cyclictest3893484-21kworker/2:221:38:192
388754099499,5cyclictest131rcu_preempt22:40:351
3887540994937,9cyclictest23-21ksoftirqd/121:10:561
3887540994930,14cyclictest3467602-21systemd-journal19:20:011
3887540994930,14cyclictest3467602-21systemd-journal19:20:011
3887540994928,16cyclictest275-21dbus-daemon00:00:011
3887540994914,10cyclictest23-21ksoftirqd/100:11:511
3887540994914,10cyclictest23-21ksoftirqd/100:11:511
3887540994911,6cyclictest131rcu_preempt00:09:411
388755099488,9cyclictest37-21ksoftirqd/319:17:013
388755099488,9cyclictest37-21ksoftirqd/319:17:003
388755099487,10cyclictest37-21ksoftirqd/319:55:013
388755099486,9cyclictest37-21ksoftirqd/320:40:573
3887550994812,6cyclictest131rcu_preempt19:15:013
388755099481,17cyclictest0-21swapper/320:32:553
3887545994840,5cyclictest3886873-21kworker/2:021:33:522
3887545994840,5cyclictest3886873-21kworker/2:020:50:012
3887545994839,5cyclictest3886873-21kworker/2:020:45:012
388754099487,8cyclictest23-21ksoftirqd/121:17:551
388754099486,5cyclictest131rcu_preempt22:45:121
3887535994831,14cyclictest12-21ksoftirqd/020:59:590
3887535994827,6cyclictest3939626-21ssh22:46:090
3887550994712,5cyclictest131rcu_preempt19:29:593
3887545994739,5cyclictest3948677-21kworker/2:000:02:032
3887545994739,5cyclictest3948677-21kworker/2:000:02:032
3887545994739,5cyclictest3886873-21kworker/2:019:52:582
388754099478,6cyclictest131rcu_preempt21:10:021
388754099476,9cyclictest23-21ksoftirqd/120:27:421
388754099476,5cyclictest131rcu_preempt20:24:441
3887540994731,11cyclictest1-21systemd20:55:011
3887540994730,5cyclictest0-21swapper/100:20:031
388754099471,8cyclictest3893358-21latency_hist21:00:021
3887540994717,15cyclictest23-21ksoftirqd/123:52:501
388753599473,10cyclictest3889625-21grep19:50:030
388753599473,10cyclictest3889625-21grep19:50:020
388755099466,12cyclictest37-21ksoftirqd/321:05:013
388755099465,7cyclictest37-21ksoftirqd/319:45:023
3887545994639,4cyclictest3886873-21kworker/2:020:29:472
3887545994638,5cyclictest3886873-21kworker/2:020:15:012
3887545994638,5cyclictest3886873-21kworker/2:019:45:002
3887545994636,5cyclictest3886873-21kworker/2:020:40:012
388754099465,8cyclictest23-21ksoftirqd/120:19:591
3887540994610,5cyclictest131rcu_preempt20:10:011
3887535994633,9cyclictest3891106-21cron20:19:590
388755099458,5cyclictest131rcu_preempt20:25:013
388755099456,8cyclictest37-21ksoftirqd/320:50:013
3887550994516,4cyclictest131rcu_preempt20:25:543
3887545994537,5cyclictest3886873-21kworker/2:019:40:002
3887545994537,5cyclictest3886873-21kworker/2:019:30:012
388754099454,9cyclictest23-21ksoftirqd/123:09:411
3887540994534,9cyclictest23-21ksoftirqd/119:30:011
3887535994528,12cyclictest3467602-21systemd-journal20:27:410
388755099445,8cyclictest3891129-21munin-run20:19:593
388755099444,8cyclictest37-21ksoftirqd/319:50:033
388755099444,8cyclictest37-21ksoftirqd/319:50:023
388755099443,10cyclictest37-21ksoftirqd/320:15:033
388755099441,7cyclictest0-21swapper/320:00:513
388755099441,7cyclictest0-21swapper/320:00:503
3887545994437,4cyclictest3886873-21kworker/2:019:33:492
3887545994436,5cyclictest3886873-21kworker/2:021:07:452
3887545994436,5cyclictest3886873-21kworker/2:020:51:412
3887545994436,5cyclictest3886873-21kworker/2:020:31:102
3887545994436,5cyclictest3886873-21kworker/2:019:11:282
3887545994426,13cyclictest275-21dbus-daemon19:20:022
3887545994426,13cyclictest275-21dbus-daemon19:20:012
388754099449,6cyclictest131rcu_preempt19:55:011
3887540994427,12cyclictest1-21systemd19:35:001
3887545994336,4cyclictest3886873-21kworker/2:020:59:502
3887535994335,3cyclictest0-21swapper/019:20:020
3887535994335,3cyclictest0-21swapper/019:20:020
3887535994327,12cyclictest3888499-21sshd19:27:320
3887545994235,4cyclictest3886873-21kworker/2:021:00:282
3887545994235,4cyclictest3886873-21kworker/2:019:24:082
3887545994235,4cyclictest3886873-21kworker/2:019:24:082
3887545994234,5cyclictest3886873-21kworker/2:020:20:002
3887545994234,5cyclictest3886873-21kworker/2:019:47:492
3887545994234,5cyclictest3886873-21kworker/2:019:47:482
3887545994218,20cyclictest30-21ksoftirqd/219:10:022
388754099424,7cyclictest23-21ksoftirqd/119:40:001
3887540994219,12cyclictest23-21ksoftirqd/120:35:011
3887540994212,6cyclictest131rcu_preempt19:24:591
3887540994212,6cyclictest131rcu_preempt19:24:591
388755099416,7cyclictest37-21ksoftirqd/320:55:033
3887545994134,4cyclictest3888812-21kworker/2:319:57:092
3887545994134,4cyclictest3886873-21kworker/2:020:05:492
3887540994129,9cyclictest3892136-21sshd20:38:431
3887540994114,7cyclictest131rcu_preempt19:45:011
3887535994135,4cyclictest16950irq/224-800000021:05:020
388754099409,6cyclictest131rcu_preempt19:47:501
388754099409,6cyclictest131rcu_preempt19:47:501
388754099401,7cyclictest23-21ksoftirqd/120:50:011
3887540994016,5cyclictest131rcu_preempt19:15:031
388753599401,4cyclictest0-21swapper/019:45:000
3887550993932,5cyclictest37-21ksoftirqd/319:20:423
3887550993932,5cyclictest37-21ksoftirqd/319:20:423
3887545993931,5cyclictest3886873-21kworker/2:020:02:562
3887545993931,5cyclictest3886873-21kworker/2:020:02:552
3887540993910,5cyclictest131rcu_preempt20:15:021
3887550993830,6cyclictest361rcuc/319:35:023
3887540993830,6cyclictest23-21ksoftirqd/120:01:451
3887540993830,6cyclictest23-21ksoftirqd/120:01:441
3887535993723,11cyclictest3889795-21sshd19:52:430
388755099361,10cyclictest37-21ksoftirqd/320:35:033
3887535993630,4cyclictest16950irq/224-800000019:10:020
3887535993629,4cyclictest16950irq/224-800000020:40:010
3887535993625,8cyclictest12-21ksoftirqd/019:25:030
3887535993625,8cyclictest12-21ksoftirqd/019:25:020
3887535993624,9cyclictest3887792-21latency_hist19:15:010
388753599352,4cyclictest0-21swapper/020:35:010
3887535993521,10cyclictest309-21in:imuxsock20:40:020
3887535993516,12cyclictest0-21swapper/020:25:030
388754099345,3cyclictest131rcu_preempt20:40:571
388753599341,12cyclictest131rcu_preempt20:50:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional