You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-01 - 08:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Jul 01, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
323180799952,86cyclictest3273924-21ssh22:25:023
323180799942,4cyclictest131rcu_preempt21:55:013
323179799821,3cyclictest0-21swapper/021:40:030
323180799715,33cyclictest3281059-21sshd22:40:023
323179799696,58cyclictest3271461-21ssh22:20:030
3231797996917,45cyclictest16950irq/224-800000022:35:020
3231807996826,12cyclictest1-21systemd23:50:013
3231805996738,6cyclictest3304730-21kworker/2:123:53:382
323179799679,5cyclictest131rcu_preempt21:57:280
3231797996761,3cyclictest0-21swapper/021:10:020
3231797996761,3cyclictest0-21swapper/021:10:020
323180799661,29cyclictest3304857-21ssh23:30:023
3231797996553,7cyclictest16950irq/224-800000021:50:010
3231797996513,5cyclictest131rcu_preempt23:09:550
323179799649,5cyclictest131rcu_preempt22:24:020
323179799646,8cyclictest121ksoftirqd/023:45:010
3231801996319,9cyclictest131rcu_preempt21:25:001
3231797996140,17cyclictest16850irq/134-800000023:00:010
323179799604,14cyclictest12-21ksoftirqd/000:10:000
3231797996027,17cyclictest3241417-21ssh21:16:570
3231797996017,39cyclictest12-21ksoftirqd/023:56:020
3231805995937,7cyclictest3240395-21kworker/2:022:04:572
3231797995947,7cyclictest16950irq/224-800000023:15:000
3231797995912,5cyclictest131rcu_preempt00:20:000
3231805995849,6cyclictest3319573-21kworker/2:200:19:482
323179799588,9cyclictest12-21ksoftirqd/022:36:440
323179799586,7cyclictest12-21ksoftirqd/022:55:000
3231797995828,26cyclictest3267891-21ssh22:12:320
3231797995828,14cyclictest3336814-21ssh00:37:080
3231797995828,14cyclictest3336814-21ssh00:37:080
3231797995822,6cyclictest151rcuc/019:20:010
3231805995749,5cyclictest3240395-21kworker/2:022:16:152
323179799573,8cyclictest12-21ksoftirqd/022:50:020
3231797995728,14cyclictest3305596-21ssh23:31:280
3231797995728,14cyclictest3305596-21ssh23:31:280
3231797995728,14cyclictest3248617-21ssh21:31:590
3231797995727,14cyclictest3291523-21ssh23:02:070
323179799571,40cyclictest3263630-21ssh22:03:390
3231805995647,5cyclictest3319573-21kworker/2:200:21:452
3231805995647,5cyclictest3210759-21kworker/2:319:55:022
323179799568,6cyclictest12-21ksoftirqd/023:48:370
3231797995627,13cyclictest3331297-21ssh00:25:290
3231797995623,17cyclictest3243153-21ssh21:20:320
323179799562,11cyclictest3254760-21latency_hist21:45:010
3231797995619,12cyclictest12-21ksoftirqd/021:30:000
3231797995610,5cyclictest131rcu_preempt22:43:060
3231805995546,6cyclictest3240395-21kworker/2:021:59:452
3231797995523,16cyclictest3240354-21ssh21:14:590
3231797995522,29cyclictest12-21ksoftirqd/023:22:460
3231797995522,29cyclictest12-21ksoftirqd/023:22:460
323179799551,38cyclictest3297791-21ssh23:15:030
3231805995446,5cyclictest3240395-21kworker/2:023:17:082
3231805995445,6cyclictest3210759-21kworker/2:319:50:002
3231805995436,13cyclictest275-21dbus-daemon23:40:002
3231801995445,6cyclictest23-21ksoftirqd/100:27:231
3231801995410,10cyclictest131rcu_preempt20:35:021
3231801995410,10cyclictest131rcu_preempt20:35:021
323179799548,6cyclictest131rcu_preempt20:35:020
323179799548,6cyclictest131rcu_preempt20:35:020
323179799543,35cyclictest131rcu_preempt23:28:590
3231797995424,15cyclictest3265672-21ssh22:08:010
3231797995410,5cyclictest131rcu_preempt00:13:270
3231807995333,15cyclictest1-21systemd22:45:013
3231797995327,17cyclictest3234285-21cron20:00:010
3231797995325,12cyclictest3335787-21ssh00:35:030
323179799532,36cyclictest3320428-21ssh00:02:400
3231805995244,5cyclictest3326145-21kworker/2:300:33:022
3231805995243,5cyclictest3240395-21kworker/2:023:00:022
323179799526,42cyclictest3232966-21sshd19:30:020
3231797995225,12cyclictest3314869-21ssh23:50:540
323179799522,12cyclictest12-21ksoftirqd/020:45:020
3231797995214,12cyclictest12-21ksoftirqd/023:40:000
3231797995211,5cyclictest131rcu_preempt22:27:350
3231797995210,12cyclictest12-21ksoftirqd/020:05:010
3231805995143,5cyclictest3326145-21kworker/2:300:25:322
3231805995143,5cyclictest3240395-21kworker/2:023:11:012
3231805995143,5cyclictest3240395-21kworker/2:022:26:222
3231805995143,5cyclictest3240395-21kworker/2:021:50:072
3231805995142,6cyclictest3240395-21kworker/2:023:45:322
3231805995142,5cyclictest3319573-21kworker/2:200:09:592
3231805995140,7cyclictest3240395-21kworker/2:023:27:042
323179799517,23cyclictest0-21swapper/020:45:030
3231797995125,11cyclictest3328972-21ssh00:20:320
323180799504,3cyclictest131rcu_preempt23:51:553
3231805995042,5cyclictest3319573-21kworker/2:200:11:142
3231805995042,5cyclictest3240395-21kworker/2:023:06:112
3231805995042,5cyclictest3240395-21kworker/2:022:46:052
3231805995042,5cyclictest3240395-21kworker/2:022:08:172
3231805995041,6cyclictest3240395-21kworker/2:023:21:262
3231805995041,6cyclictest3240395-21kworker/2:023:21:262
3231805995041,5cyclictest3240395-21kworker/2:000:04:322
3231805995040,6cyclictest3240395-21kworker/2:023:02:242
3231805995033,10cyclictest3252304-21cron21:40:002
3231805995032,13cyclictest1-21systemd20:30:002
3231797995022,12cyclictest3258417-21ssh21:52:340
3231805994941,5cyclictest3240395-21kworker/2:022:40:012
3231805994941,5cyclictest3240395-21kworker/2:022:31:362
3231805994941,5cyclictest3240395-21kworker/2:022:21:422
3231805994940,6cyclictest3210759-21kworker/2:321:26:062
3231805994940,5cyclictest3240395-21kworker/2:021:35:012
323180199492,5cyclictest131rcu_preempt23:05:021
323179799492,40cyclictest3234928-21sshd20:10:020
3231797994915,13cyclictest12-21ksoftirqd/019:15:000
323180799487,31cyclictest3232997-21kworker/u8:319:40:013
3231807994832,11cyclictest3319215-21sed00:00:023
3231807994831,13cyclictest37-21ksoftirqd/319:10:013
3231807994831,13cyclictest37-21ksoftirqd/319:10:003
3231805994840,5cyclictest3240395-21kworker/2:022:54:482
3231805994840,5cyclictest3210759-21kworker/2:321:19:422
3231805994840,5cyclictest3210759-21kworker/2:321:10:422
3231805994840,5cyclictest3210759-21kworker/2:321:00:022
3231805994840,5cyclictest3210759-21kworker/2:320:15:012
3231805994840,5cyclictest3210759-21kworker/2:319:35:012
3231805994828,16cyclictest275-21dbus-daemon20:50:012
3231797994824,17cyclictest0-21swapper/019:40:030
3231797994823,18cyclictest0-21swapper/019:25:010
3231797994814,12cyclictest12-21ksoftirqd/019:51:070
3231807994734,10cyclictest37-21ksoftirqd/322:47:273
3231807994728,13cyclictest275-21dbus-daemon21:40:013
3231805994739,5cyclictest3240395-21kworker/2:022:42:032
3231805994739,5cyclictest3240395-21kworker/2:021:45:412
3231805994739,5cyclictest3210759-21kworker/2:321:06:582
3231805994739,5cyclictest3210759-21kworker/2:321:06:582
3231805994739,5cyclictest3210759-21kworker/2:320:20:532
3231805994728,14cyclictest3467602-21systemd-journal21:04:592
3231801994732,12cyclictest3467602-21systemd-journal23:45:021
3231801994730,13cyclictest275-21dbus-daemon23:00:021
323179799476,9cyclictest12-21ksoftirqd/020:20:000
3231797994714,4cyclictest131rcu_preempt21:00:010
3231805994638,5cyclictest3326145-21kworker/2:300:35:432
3231805994638,5cyclictest3326145-21kworker/2:300:35:432
3231805994634,6cyclictest3233540-21munin-plugin-st19:45:012
323180199463,8cyclictest23-21ksoftirqd/123:55:021
3231801994634,8cyclictest3261826-21tr22:00:011
3231801994634,8cyclictest1-21systemd20:40:011
3231801994628,13cyclictest275-21dbus-daemon00:00:021
3231807994534,6cyclictest37-21ksoftirqd/323:10:003
3231805994539,3cyclictest3240395-21kworker/2:023:57:022
3231801994530,10cyclictest309-21in:imuxsock00:25:011
323179799458,5cyclictest131rcu_preempt20:15:010
3231797994512,5cyclictest131rcu_preempt19:49:590
3231807994434,5cyclictest3254723-21sh21:44:593
3231807994425,11cyclictest309-21in:imuxsock22:00:003
3231807994410,18cyclictest131rcu_preempt21:00:023
3231805994436,5cyclictest3240395-21kworker/2:023:34:012
3231805994436,5cyclictest3240395-21kworker/2:023:34:002
3231805994436,5cyclictest3240395-21kworker/2:022:14:142
3231805994436,5cyclictest3240395-21kworker/2:021:41:172
3231805994435,5cyclictest3210759-21kworker/2:320:00:012
3231805994431,9cyclictest3234930-21cat20:10:022
3231801994431,9cyclictest275-21dbus-daemon19:25:481
3231801994428,12cyclictest275-21dbus-daemon20:00:011
3231797994427,13cyclictest3237671-21sshd21:02:020
323180799437,3cyclictest131rcu_preempt19:50:023
3231807994328,11cyclictest275-21dbus-daemon20:00:013
3231807994323,14cyclictest3236391-21systemctl20:40:003
323180799431,3cyclictest0-21swapper/320:10:023
3231805994332,7cyclictest309-21in:imuxsock19:20:032
3231801994332,6cyclictest309-21in:imuxsock23:10:011
3231801994321,9cyclictest23-21ksoftirqd/100:20:001
3231801994319,18cyclictest275-21dbus-daemon00:35:011
3231801994310,6cyclictest131rcu_preempt22:17:031
323179799433,8cyclictest12-21ksoftirqd/019:45:010
323180799422,3cyclictest131rcu_preempt22:34:513
3231805994234,5cyclictest3240395-21kworker/2:023:40:242
3231801994230,9cyclictest1-21systemd22:10:021
3231801994228,10cyclictest3324004-21sshd00:10:021
3231801994225,13cyclictest3467602-21systemd-journal20:12:031
3231801994218,7cyclictest131rcu_preempt00:40:011
3231801994218,7cyclictest131rcu_preempt00:40:011
323180199421,7cyclictest23-21ksoftirqd/100:04:581
3231797994225,12cyclictest12-21ksoftirqd/020:39:590
323180799413,3cyclictest131rcu_preempt20:20:013
323180799412,3cyclictest131rcu_preempt21:30:463
323180799411,3cyclictest131rcu_preempt22:14:513
323180799411,3cyclictest131rcu_preempt20:50:013
3231805994133,5cyclictest3210759-21kworker/2:321:22:112
3231805994129,9cyclictest3467602-21systemd-journal20:02:032
323180199414,6cyclictest131rcu_preempt23:15:021
3231801994132,6cyclictest3323784-21latency_hist00:10:021
323180199413,18cyclictest0-21swapper/123:37:261
3231801994128,9cyclictest23-21ksoftirqd/123:25:021
3231801994128,9cyclictest23-21ksoftirqd/123:25:011
3231801994123,12cyclictest0-21swapper/120:55:001
3231801994122,15cyclictest3467718-21lldpd23:26:351
3231801994111,7cyclictest131rcu_preempt22:50:001
323179799415,10cyclictest12-21ksoftirqd/019:29:030
3231797994127,11cyclictest3467602-21systemd-journal20:27:310
3231807994031,5cyclictest3319500-21sh00:00:323
323180799402,8cyclictest284-21systemd-logind20:15:013
3231807994027,10cyclictest37-21ksoftirqd/321:30:013
3231807994025,12cyclictest37-21ksoftirqd/300:15:003
3231807994024,12cyclictest37-21ksoftirqd/300:28:003
323180799402,3cyclictest131rcu_preempt20:30:023
323180799401,4cyclictest131rcu_preempt23:35:023
323180799401,4cyclictest131rcu_preempt23:35:023
3231801994027,10cyclictest23-21ksoftirqd/119:45:011
3231801994025,12cyclictest23-21ksoftirqd/122:52:301
3231801994025,12cyclictest23-21ksoftirqd/121:36:501
3231801994025,11cyclictest3467602-21systemd-journal21:20:001
3231801994025,11cyclictest23-21ksoftirqd/121:50:171
3231801994025,11cyclictest23-21ksoftirqd/121:47:231
3231801994024,11cyclictest0-21swapper/120:30:011
3231801994022,5cyclictest23-21ksoftirqd/121:42:071
323179799402,8cyclictest12-21ksoftirqd/020:24:140
323180799394,4cyclictest131rcu_preempt23:45:023
323180799393,8cyclictest3328511-21modprobe00:20:003
3231807993930,6cyclictest3288205-21sed22:55:023
323180799392,9cyclictest37-21ksoftirqd/322:10:023
3231807993929,8cyclictest37-21ksoftirqd/323:24:473
3231807993929,8cyclictest37-21ksoftirqd/323:24:473
3231807993925,10cyclictest275-21dbus-daemon19:35:023
3231807993923,13cyclictest37-21ksoftirqd/322:55:153
3231807993923,13cyclictest275-21dbus-daemon23:15:013
3231807993919,14cyclictest3467718-21lldpd20:55:003
323180799391,12cyclictest3336588-21ssh00:36:403
323180799391,12cyclictest3336588-21ssh00:36:403
3231805993930,6cyclictest3210759-21kworker/2:320:30:592
3231805993930,6cyclictest3210759-21kworker/2:320:30:592
3231805993923,12cyclictest30-21ksoftirqd/219:40:002
3231801993929,8cyclictest23-21ksoftirqd/123:34:311
3231801993929,8cyclictest23-21ksoftirqd/123:34:301
3231801993927,8cyclictest3268910-21cron22:15:001
3231801993926,10cyclictest23-21ksoftirqd/123:45:001
323180199392,3cyclictest131rcu_preempt20:45:011
3231801993922,4cyclictest23-21ksoftirqd/119:40:011
3231801993921,5cyclictest23-21ksoftirqd/122:32:421
323180199391,6cyclictest131rcu_preempt22:25:011
3231801993912,19cyclictest131rcu_preempt21:31:041
323179799392,9cyclictest12-21ksoftirqd/019:10:010
323179799392,9cyclictest12-21ksoftirqd/019:10:010
3231797993929,7cyclictest3237176-21systemctl20:55:000
3231807993831,4cyclictest3234042-21latency_hist19:55:003
3231807993830,5cyclictest3298241-21sh23:16:033
3231807993829,7cyclictest37-21ksoftirqd/300:25:023
3231807993829,7cyclictest3271312-21cron22:20:013
323180799382,3cyclictest131rcu_preempt22:29:003
323180799381,10cyclictest3264156-21ssh22:04:563
323180599384,4cyclictest131rcu_preempt19:15:002
3231805993831,5cyclictest3210759-21kworker/2:320:20:022
3231801993827,8cyclictest3231944-21sshd19:13:011
3231801993824,11cyclictest23-21ksoftirqd/122:29:021
3231801993824,10cyclictest275-21dbus-daemon19:55:001
3231801993824,10cyclictest23-21ksoftirqd/121:25:401
3231801993810,7cyclictest131rcu_preempt22:45:001
3231807993726,8cyclictest3467602-21systemd-journal00:35:013
3231807993725,9cyclictest37-21ksoftirqd/320:01:073
3231807993721,12cyclictest3232827-21cut19:30:023
323180799371,4cyclictest0-21swapper/323:03:453
3231805993728,5cyclictest3210759-21kworker/2:320:45:032
3231805993728,5cyclictest3210759-21kworker/2:320:40:022
3231805993725,8cyclictest3237133-21sshd20:50:592
3231801993727,8cyclictest23-21ksoftirqd/122:39:081
3231801993727,7cyclictest23-21ksoftirqd/123:16:441
3231801993727,6cyclictest3235049-21kworker/u8:121:00:001
3231801993724,10cyclictest23-21ksoftirqd/122:00:051
323180799368,6cyclictest131rcu_preempt23:35:153
3231807993626,8cyclictest3237973-21sh21:10:013
3231807993626,8cyclictest3237973-21sh21:10:003
3231807993626,7cyclictest37-21ksoftirqd/320:35:013
3231807993626,7cyclictest37-21ksoftirqd/320:35:003
3231807993622,11cyclictest37-21ksoftirqd/321:47:293
3231807993621,11cyclictest275-21dbus-daemon21:20:003
3231807993621,11cyclictest1-21systemd21:25:013
323180799361,6cyclictest0-21swapper/321:13:113
3231801993623,10cyclictest23-21ksoftirqd/121:12:331
3231801993622,9cyclictest309-21in:imuxsock19:45:031
3231801993621,11cyclictest3467602-21systemd-journal20:50:011
3231801993618,14cyclictest3467602-21systemd-journal19:24:461
3231801993617,13cyclictest275-21dbus-daemon20:25:001
323180799358,5cyclictest131rcu_preempt00:09:393
323180799352,3cyclictest131rcu_preempt20:25:013
3231807993518,12cyclictest309-21in:imuxsock19:45:013
3231807993517,12cyclictest3467602-21systemd-journal19:24:593
323180599351,4cyclictest131rcu_preempt19:20:012
3231801993528,4cyclictest3233008-21latency_hist19:35:021
3231801993527,5cyclictest3232241-21sendmail19:20:011
3231801993525,6cyclictest23-21ksoftirqd/121:10:001
3231801993525,6cyclictest23-21ksoftirqd/121:10:001
3231801993517,11cyclictest0-21swapper/121:05:001
3231807993427,5cyclictest37-21ksoftirqd/321:05:023
3231807993421,9cyclictest37-21ksoftirqd/319:20:013
3231801993419,12cyclictest3467602-21systemd-journal20:03:321
3231801993419,11cyclictest23-21ksoftirqd/120:06:261
3231807993320,9cyclictest3467602-21systemd-journal20:45:003
3231805993324,6cyclictest3231709-21latency_hist19:10:022
3231805993324,6cyclictest3231709-21latency_hist19:10:012
3231801993326,4cyclictest3235523-21sshd20:20:031
323180199321,3cyclictest131rcu_preempt19:10:021
323180199321,3cyclictest131rcu_preempt19:10:021
3231807993019,9cyclictest37-21ksoftirqd/319:15:013
3231805993019,9cyclictest30-21ksoftirqd/219:30:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional