You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 18:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Tue Apr 28, 2026 00:45:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
36062869914926,6cyclictest16850irq/134-800000023:55:000
3606298999367,22cyclictest37-21ksoftirqd/322:30:033
3606286998115,45cyclictest12-21ksoftirqd/023:38:520
3606286997873,3cyclictest0-21swapper/000:00:040
3606286997733,40cyclictest3694475-21cron00:20:000
3606286996810,5cyclictest131rcu_preempt22:58:430
3606298996726,13cyclictest275-21dbus-daemon23:25:003
360628699666,39cyclictest12-21ksoftirqd/021:30:210
3606298996525,7cyclictest3642770-21latency_hist22:20:013
3606286996517,39cyclictest12-21ksoftirqd/023:32:580
3606295996444,7cyclictest3609906-21kworker/2:021:49:552
3606298996338,4cyclictest37-21ksoftirqd/322:25:023
3606295996341,7cyclictest3636232-21kworker/2:100:12:422
3606286996330,4cyclictest12-21ksoftirqd/000:07:110
3606286996328,7cyclictest131rcu_preempt22:05:000
360628699632,3cyclictest0-21swapper/023:40:030
3606286996316,4cyclictest12-21ksoftirqd/022:51:010
3606286996219,18cyclictest12-21ksoftirqd/023:16:250
3606286996216,42cyclictest12-21ksoftirqd/023:49:190
3606298996127,8cyclictest37-21ksoftirqd/322:01:033
3606295996141,14cyclictest3467602-21systemd-journal23:20:002
360628699618,31cyclictest131rcu_preempt22:13:330
3606286996114,43cyclictest12-21ksoftirqd/023:07:560
360628699611,42cyclictest3672875-21unin-run23:30:010
3606286996032,12cyclictest3626320-21ssh21:41:540
3606286996025,6cyclictest131rcu_preempt20:25:010
3606286996024,10cyclictest12-21ksoftirqd/021:20:180
3606295995927,5cyclictest3636232-21kworker/2:123:15:012
360628699596,7cyclictest131rcu_preempt22:50:010
3606286995918,33cyclictest16850irq/134-800000021:55:420
3606286995918,33cyclictest16850irq/134-800000021:55:410
3606286995916,33cyclictest16850irq/134-800000021:26:320
3606286995914,41cyclictest12-21ksoftirqd/022:36:360
3606298995828,4cyclictest3607269-21latency_hist19:29:593
3606298995727,4cyclictest3673091-21sh23:30:103
360629599575,6cyclictest131rcu_preempt00:10:022
3606295995746,6cyclictest3599052-21kworker/2:320:30:022
3606295995736,8cyclictest3599052-21kworker/2:321:25:442
3606286995738,15cyclictest3610939-21grep20:40:010
3606286995718,35cyclictest16850irq/134-800000022:30:100
3606286995718,34cyclictest16850irq/134-800000023:55:210
3606286995718,34cyclictest16850irq/134-800000023:55:210
3606286995718,34cyclictest16850irq/134-800000023:13:170
3606286995718,34cyclictest16850irq/134-800000023:01:420
3606286995718,34cyclictest16850irq/134-800000023:01:410
3606286995718,30cyclictest16850irq/134-800000023:21:470
3606286995717,32cyclictest16850irq/134-800000000:37:540
360629899567,24cyclictest0-21swapper/300:20:013
3606295995646,6cyclictest3599052-21kworker/2:322:28:172
3606286995619,33cyclictest16850irq/134-800000021:48:470
3606286995619,33cyclictest16850irq/134-800000021:14:330
3606286995619,33cyclictest16850irq/134-800000000:30:080
3606286995618,34cyclictest16850irq/134-800000022:40:410
3606286995618,34cyclictest16850irq/134-800000022:40:400
3606286995617,35cyclictest16850irq/134-800000021:38:090
3606286995617,35cyclictest16850irq/134-800000021:38:090
3606286995615,32cyclictest16850irq/134-800000021:16:190
3606298995523,11cyclictest3659285-21ssh22:58:303
3606295995546,6cyclictest3599052-21kworker/2:323:00:562
3606295995546,6cyclictest3599052-21kworker/2:323:00:552
3606295995546,5cyclictest3636232-21kworker/2:123:59:042
3606295995546,5cyclictest3636232-21kworker/2:123:59:032
3606295995546,5cyclictest3636232-21kworker/2:100:00:212
3606295995531,8cyclictest3636232-21kworker/2:100:21:162
3606286995517,6cyclictest16850irq/134-800000022:24:110
3606286995517,34cyclictest16850irq/134-800000022:16:120
3606286995516,31cyclictest16850irq/134-800000000:26:020
3606286995516,30cyclictest16850irq/134-800000000:24:300
3606286995515,32cyclictest16850irq/134-800000022:08:150
3606286995515,31cyclictest16850irq/134-800000022:26:310
3606286995515,31cyclictest16850irq/134-800000021:50:270
3606295995446,5cyclictest3636232-21kworker/2:123:47:462
3606295995446,5cyclictest3636232-21kworker/2:100:16:172
3606295995444,6cyclictest3636232-21kworker/2:123:31:302
3606286995417,33cyclictest16850irq/134-800000000:12:280
3606295995347,3cyclictest3599052-21kworker/2:322:22:432
3606295995345,5cyclictest3636232-21kworker/2:122:56:212
3606295995344,6cyclictest3636232-21kworker/2:123:37:042
3606295995344,5cyclictest3599052-21kworker/2:321:11:562
3606295995244,5cyclictest3636232-21kworker/2:123:23:342
3606295995244,5cyclictest3636232-21kworker/2:122:54:072
3606295995244,5cyclictest3609906-21kworker/2:021:51:242
3606295995244,5cyclictest3599052-21kworker/2:322:43:382
3606295995244,5cyclictest3599052-21kworker/2:322:43:372
3606295995244,5cyclictest3599052-21kworker/2:322:38:472
3606295995244,5cyclictest3599052-21kworker/2:322:12:552
3606295995244,5cyclictest3599052-21kworker/2:321:22:362
3606295995244,5cyclictest3599052-21kworker/2:319:41:442
3606295995242,7cyclictest3685774-21kworker/2:000:33:212
3606295995144,4cyclictest3599052-21kworker/2:321:18:262
3606295995143,5cyclictest3636232-21kworker/2:123:29:472
3606295995143,5cyclictest3599052-21kworker/2:322:48:382
3606295995143,5cyclictest3599052-21kworker/2:320:25:012
3606295995143,5cyclictest3599052-21kworker/2:320:05:012
3606295995142,6cyclictest3636232-21kworker/2:100:25:182
3606295995119,21cyclictest131rcu_preempt23:45:002
3606291995134,13cyclictest3467602-21systemd-journal22:49:591
3606295995042,5cyclictest3636232-21kworker/2:123:07:302
3606295995042,5cyclictest3609906-21kworker/2:022:02:502
3606295995042,5cyclictest3609906-21kworker/2:021:40:162
3606295995042,5cyclictest3599052-21kworker/2:320:40:022
3606295995042,5cyclictest3599052-21kworker/2:320:10:012
3606286995016,30cyclictest3612461-21latency_hist21:10:020
3606298994937,10cyclictest3680593-21sshd23:47:453
3606298994933,12cyclictest3467602-21systemd-journal00:29:443
3606295994942,5cyclictest3609906-21kworker/2:021:34:152
3606295994941,5cyclictest3609906-21kworker/2:021:55:192
3606295994941,5cyclictest3609906-21kworker/2:021:55:192
3606295994941,5cyclictest3609906-21kworker/2:021:37:212
3606295994941,5cyclictest3609906-21kworker/2:021:37:212
3606295994941,5cyclictest3599052-21kworker/2:322:32:302
3606295994941,5cyclictest3599052-21kworker/2:322:16:362
3606295994941,5cyclictest3599052-21kworker/2:322:09:332
3606286994913,31cyclictest12-21ksoftirqd/020:35:020
3606295994840,5cyclictest3636232-21kworker/2:123:51:182
3606295994840,5cyclictest3599052-21kworker/2:319:20:112
3606295994812,6cyclictest131rcu_preempt00:40:012
3606291994833,11cyclictest3467602-21systemd-journal23:59:591
3606291994833,11cyclictest3467602-21systemd-journal00:00:001
3606291994832,12cyclictest1-21systemd22:05:011
360628699481,7cyclictest3609895-21unin-run20:20:010
3606295994739,5cyclictest3599052-21kworker/2:320:56:352
3606295994739,5cyclictest3599052-21kworker/2:320:50:002
3606295994739,5cyclictest3599052-21kworker/2:319:25:282
3606291994736,7cyclictest309-21in:imuxsock23:30:011
3606286994739,4cyclictest16850irq/134-800000019:40:000
3606286994728,12cyclictest309-21in:imuxsock20:45:010
3606286994722,18cyclictest275-21dbus-daemon19:15:000
3606295994638,5cyclictest3599052-21kworker/2:321:06:002
3606295994638,5cyclictest3599052-21kworker/2:319:55:022
3606295994632,11cyclictest3609841-21cron20:20:002
3606291994632,9cyclictest23-21ksoftirqd/121:35:021
3606291994631,11cyclictest3467602-21systemd-journal19:11:451
3606291994629,13cyclictest275-21dbus-daemon23:15:011
360629199461,3cyclictest131rcu_preempt23:35:021
3606286994633,9cyclictest309-21in:imuxsock19:30:000
3606298994530,11cyclictest3677120-21cron23:40:013
3606295994537,5cyclictest3599052-21kworker/2:319:33:452
3606291994521,20cyclictest23-21ksoftirqd/119:38:441
3606291994518,8cyclictest0-21swapper/122:19:301
360629199451,3cyclictest131rcu_preempt21:40:001
360629199451,3cyclictest131rcu_preempt21:39:591
3606298994433,8cyclictest37-21ksoftirqd/300:00:023
3606298994433,8cyclictest37-21ksoftirqd/300:00:023
3606298994427,13cyclictest3467602-21systemd-journal21:37:453
3606298994427,13cyclictest3467602-21systemd-journal21:37:443
3606295994433,8cyclictest3606446-21sshd19:10:452
3606295994430,9cyclictest30-21ksoftirqd/220:15:012
360629599441,8cyclictest0-21swapper/219:10:002
360629199444,3cyclictest131rcu_preempt23:09:211
360629199443,3cyclictest131rcu_preempt22:53:431
3606291994431,9cyclictest1-21systemd21:10:001
360629199441,3cyclictest131rcu_preempt23:04:221
360629199441,3cyclictest131rcu_preempt23:04:221
360628699446,26cyclictest3606289-21grep19:10:010
360628699445,4cyclictest12-21ksoftirqd/019:56:180
360628699442,7cyclictest3467602-21systemd-journal21:00:000
360628699442,6cyclictest131rcu_preempt21:05:010
3606298994330,11cyclictest37-21ksoftirqd/321:44:063
3606298994330,10cyclictest1-21systemd23:55:013
3606298994326,14cyclictest3608797-21cron19:59:593
3606298994325,14cyclictest275-21dbus-daemon23:45:013
360629899432,3cyclictest131rcu_preempt21:05:023
3606295994335,5cyclictest3599052-21kworker/2:320:50:392
3606295994331,8cyclictest30-21ksoftirqd/219:54:592
360629599432,11cyclictest1-21systemd19:20:012
360629199434,3cyclictest131rcu_preempt20:10:031
360629199433,3cyclictest131rcu_preempt23:49:151
3606291994328,11cyclictest23-21ksoftirqd/100:20:491
360629199431,4cyclictest131rcu_preempt21:42:171
3606286994333,7cyclictest0-21swapper/019:54:590
3606286994327,13cyclictest3609558-21sshd20:10:450
3606298994232,7cyclictest37-21ksoftirqd/319:15:013
3606298994232,6cyclictest3629756-21grep21:50:013
3606298994228,11cyclictest37-21ksoftirqd/323:09:373
3606298994226,12cyclictest275-21dbus-daemon22:00:013
3606298994226,12cyclictest275-21dbus-daemon22:00:003
3606295994234,5cyclictest3599052-21kworker/2:320:45:022
3606295994234,5cyclictest3599052-21kworker/2:319:45:362
3606291994231,8cyclictest3703164-21systemctl00:40:011
3606291994229,9cyclictest275-21dbus-daemon20:00:001
3606291994226,12cyclictest275-21dbus-daemon22:30:011
3606291994226,12cyclictest275-21dbus-daemon19:55:011
360629199422,3cyclictest131rcu_preempt22:33:501
3606298994129,8cyclictest1-21systemd22:15:013
360629199415,8cyclictest23-21ksoftirqd/100:33:391
360629199413,3cyclictest131rcu_preempt23:21:311
3606291994131,6cyclictest3607292-21latency_hist19:30:021
3606291994126,10cyclictest23-21ksoftirqd/100:30:011
3606291994124,12cyclictest23-21ksoftirqd/122:40:001
360629199412,3cyclictest131rcu_preempt21:56:231
360629199412,3cyclictest131rcu_preempt21:56:221
3606291994115,17cyclictest23-21ksoftirqd/122:58:291
360629199411,3cyclictest131rcu_preempt23:38:141
3606286994121,4cyclictest12-21ksoftirqd/019:20:020
360628699411,36cyclictest3611411-21kworker/u8:120:50:000
3606286994110,5cyclictest131rcu_preempt20:10:000
3606298994026,9cyclictest3467602-21systemd-journal22:35:023
3606298994026,11cyclictest37-21ksoftirqd/321:13:153
3606298994026,11cyclictest37-21ksoftirqd/300:37:253
3606298994026,10cyclictest275-21dbus-daemon20:25:013
3606298994025,11cyclictest37-21ksoftirqd/321:30:343
3606298994012,6cyclictest131rcu_preempt22:46:383
3606298994011,7cyclictest131rcu_preempt22:33:163
3606298994010,6cyclictest131rcu_preempt23:20:023
3606295994033,4cyclictest3599052-21kworker/2:321:00:222
360629199403,3cyclictest131rcu_preempt23:53:391
360629199402,3cyclictest131rcu_preempt00:15:071
3606291994020,14cyclictest275-21dbus-daemon20:45:001
360629199401,3cyclictest131rcu_preempt23:40:181
3606291994010,15cyclictest131rcu_preempt22:06:191
3606286994029,7cyclictest12-21ksoftirqd/020:01:050
3606286994021,14cyclictest275-21dbus-daemon19:50:010
3606286994018,6cyclictest16850irq/134-800000020:50:440
360628699401,35cyclictest0-21swapper/019:25:000
3606298993928,9cyclictest37-21ksoftirqd/323:04:303
3606298993928,9cyclictest37-21ksoftirqd/323:04:303
3606298993927,9cyclictest37-21ksoftirqd/319:50:013
3606298993926,9cyclictest37-21ksoftirqd/323:26:353
3606298993926,10cyclictest37-21ksoftirqd/300:00:113
3606298993923,12cyclictest1-21systemd21:10:003
3606298993922,9cyclictest37-21ksoftirqd/320:10:033
360629899391,6cyclictest150-21pr/ttyS222:40:323
360629899391,6cyclictest150-21pr/ttyS222:40:323
3606298993913,18cyclictest0-21swapper/321:20:023
360629599391,3cyclictest131rcu_preempt19:40:002
3606291993928,9cyclictest23-21ksoftirqd/119:20:011
3606291993923,13cyclictest3467602-21systemd-journal19:50:001
360628699391,3cyclictest131rcu_preempt20:28:450
360629899386,4cyclictest131rcu_preempt19:55:023
360629899384,30cyclictest3619816-21sshd21:26:453
3606298993823,11cyclictest37-21ksoftirqd/300:12:313
360629199389,5cyclictest131rcu_preempt22:22:441
360629199385,8cyclictest23-21ksoftirqd/122:10:531
360629199383,3cyclictest131rcu_preempt21:52:241
360629199383,3cyclictest131rcu_preempt21:22:561
360629199382,7cyclictest23-21ksoftirqd/123:20:021
3606291993826,9cyclictest23-21ksoftirqd/119:25:021
3606291993824,11cyclictest1-21systemd21:15:011
360629199381,3cyclictest131rcu_preempt21:17:321
3606286993825,11cyclictest12-21ksoftirqd/019:45:000
360629899375,14cyclictest3696722-21ssh00:25:023
3606298993728,5cyclictest3656577-21sh22:51:583
3606298993726,9cyclictest37-21ksoftirqd/300:34:313
3606298993725,9cyclictest37-21ksoftirqd/322:10:013
3606298993724,10cyclictest275-21dbus-daemon21:55:013
3606298993722,11cyclictest3609980-21wc20:20:023
3606298993719,13cyclictest1-21systemd19:40:013
3606291993727,7cyclictest23-21ksoftirqd/122:40:521
3606291993727,7cyclictest23-21ksoftirqd/122:40:521
3606291993726,9cyclictest23-21ksoftirqd/119:44:591
3606291993725,9cyclictest23-21ksoftirqd/120:55:411
3606291993725,9cyclictest23-21ksoftirqd/119:30:021
3606291993725,10cyclictest23-21ksoftirqd/120:05:001
3606291993720,13cyclictest3610115-21sshd20:23:441
3606286993727,6cyclictest3607695-21latency_hist19:35:030
360629899366,8cyclictest37-21ksoftirqd/323:10:443
360629899366,8cyclictest37-21ksoftirqd/300:10:023
360629899365,27cyclictest3610897-21munin-run20:39:593
360629899363,9cyclictest37-21ksoftirqd/320:01:123
3606298993625,8cyclictest37-21ksoftirqd/320:35:033
3606298993622,9cyclictest37-21ksoftirqd/319:20:023
360629599362,5cyclictest131rcu_preempt20:35:002
360629199363,3cyclictest131rcu_preempt20:20:011
360629199362,9cyclictest131rcu_preempt00:05:201
3606291993625,7cyclictest23-21ksoftirqd/120:15:011
360629199362,4cyclictest131rcu_preempt21:27:031
360629199362,3cyclictest131rcu_preempt00:05:011
3606291993621,11cyclictest3467602-21systemd-journal20:35:021
3606291993612,16cyclictest23-21ksoftirqd/121:48:151
3606298993526,7cyclictest37-21ksoftirqd/320:55:013
3606298993526,7cyclictest37-21ksoftirqd/320:30:023
360629899352,4cyclictest131rcu_preempt20:14:593
3606298993524,9cyclictest37-21ksoftirqd/321:24:113
3606291993524,8cyclictest23-21ksoftirqd/120:55:011
3606291993522,9cyclictest23-21ksoftirqd/120:50:021
360629199351,3cyclictest131rcu_preempt00:12:551
3606298993423,7cyclictest3607009-21cron19:25:003
3606291993427,4cyclictest3610474-21latency_hist20:30:011
3606291993422,8cyclictest23-21ksoftirqd/120:40:021
360629899334,3cyclictest131rcu_preempt19:35:013
360629899332,3cyclictest131rcu_preempt21:00:013
3606291993319,9cyclictest23-21ksoftirqd/121:05:011
3606298993222,7cyclictest37-21ksoftirqd/320:42:523
360629899311,7cyclictest0-21swapper/320:50:023
3606298993023,4cyclictest3467562-21sshd19:42:433
360629899283,3cyclictest131rcu_preempt19:10:023
360629199283,3cyclictest131rcu_preempt19:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional