You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-04 - 06:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Thu Jun 04, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
301741099845,63cyclictest3055314-21ssh22:20:022
301740599801,62cyclictest3068788-21ssh22:50:021
3017405997425,12cyclictest275-21dbus-daemon22:45:021
301741099709,42cyclictest3023634-21diskmemload22:35:032
3017405997055,6cyclictest3032825-21ssh21:30:041
3017405997055,6cyclictest3032825-21ssh21:30:031
3017402996821,28cyclictest12-21ksoftirqd/021:40:010
3017417996640,18cyclictest3467602-21systemd-journal20:50:003
3017402996526,34cyclictest16850irq/134-800000021:50:020
3017402996517,3cyclictest131rcu_preempt21:15:270
3017405996430,7cyclictest3109088-21sendmail00:20:001
3017405996330,9cyclictest3467602-21systemd-journal22:36:321
3017405996327,11cyclictest23-21ksoftirqd/100:09:031
3017417996136,20cyclictest275-21dbus-daemon21:35:013
3017417996136,20cyclictest275-21dbus-daemon21:35:013
3017405996024,12cyclictest23-21ksoftirqd/100:25:121
301740299603,23cyclictest131rcu_preempt22:21:480
3017402995929,14cyclictest3085905-21ssh23:28:090
3017402995929,14cyclictest3085905-21ssh23:28:090
3017402995928,14cyclictest3052373-21ssh22:13:460
3017402995928,14cyclictest3052373-21ssh22:13:460
3017410995827,11cyclictest3017869-21cat19:20:002
3017405995840,14cyclictest1-21systemd20:10:011
3017402995831,23cyclictest3467602-21systemd-journal00:00:000
3017417995735,12cyclictest3046132-21sh22:00:003
3017402995728,13cyclictest3023903-21ssh21:10:050
3017402995728,13cyclictest3023903-21ssh21:10:050
3017402995726,15cyclictest3075423-21ssh23:05:020
301740299572,28cyclictest131rcu_preempt00:29:220
301741799564,7cyclictest131rcu_preempt22:50:013
3017402995626,14cyclictest3070377-21ssh22:53:380
3017410995546,6cyclictest3021822-21kworker/2:021:20:012
3017402995533,17cyclictest275-21dbus-daemon21:55:020
3017402995527,12cyclictest3117849-21ssh00:39:220
3017402995527,12cyclictest3049034-21ssh22:06:060
301740299552,24cyclictest131rcu_preempt22:17:130
3017402995519,32cyclictest16850irq/134-800000022:30:440
3017417995436,14cyclictest3467602-21systemd-journal20:30:013
3017402995422,3cyclictest131rcu_preempt00:17:320
3017402995418,18cyclictest16850irq/134-800000023:35:500
3017410995315,5cyclictest131rcu_preempt23:00:002
301740299538,29cyclictest12-21ksoftirqd/022:30:000
301740299533,3cyclictest131rcu_preempt22:36:530
3017402995326,12cyclictest3083212-21ssh23:22:140
3017402995318,5cyclictest16850irq/134-800000021:32:350
3017402995318,5cyclictest16850irq/134-800000021:32:350
3017402995318,18cyclictest16850irq/134-800000022:47:110
3017402995318,17cyclictest16850irq/134-800000023:54:440
3017402995316,6cyclictest16850irq/134-800000022:56:200
3017417995224,19cyclictest275-21dbus-daemon20:55:003
3017417995222,18cyclictest275-21dbus-daemon19:40:013
3017417995222,18cyclictest275-21dbus-daemon19:40:013
3017410995245,4cyclictest3077662-21kworker/2:023:24:142
3017410995244,5cyclictest3010825-21kworker/2:122:22:162
3017410995243,5cyclictest3010825-21kworker/2:122:42:572
301740299529,5cyclictest131rcu_preempt23:48:500
301740299529,5cyclictest131rcu_preempt23:48:490
3017402995227,10cyclictest3093174-21ssh23:44:390
3017402995217,6cyclictest16850irq/134-800000000:23:150
3017402995217,6cyclictest16850irq/134-800000000:23:150
3017402995216,6cyclictest16850irq/134-800000000:01:260
3017402995216,18cyclictest16850irq/134-800000000:07:390
3017417995126,17cyclictest3467602-21systemd-journal23:15:013
3017410995143,5cyclictest3010825-21kworker/2:122:05:022
3017410995142,5cyclictest3021822-21kworker/2:021:10:212
3017410995142,5cyclictest3021822-21kworker/2:021:10:202
3017402995116,17cyclictest16850irq/134-800000021:42:460
3017402995116,17cyclictest16850irq/134-800000021:42:460
3017402995115,18cyclictest16850irq/134-800000022:44:270
3017402995114,6cyclictest16850irq/134-800000023:05:500
3017417995034,11cyclictest1-21systemd23:45:023
3017417995033,11cyclictest309-21in:imuxsock21:41:513
3017417995033,11cyclictest309-21in:imuxsock21:41:503
3017417995026,5cyclictest37-21ksoftirqd/321:16:273
301741099508,8cyclictest30-21ksoftirqd/200:35:002
3017410995042,5cyclictest3109145-21kworker/2:200:29:382
3017410995042,5cyclictest3055202-21kworker/2:323:32:552
3017410995042,5cyclictest3010825-21kworker/2:122:52:092
3017410995041,6cyclictest3010825-21kworker/2:122:14:362
3017410995041,6cyclictest3010825-21kworker/2:122:14:352
3017402995016,5cyclictest16850irq/134-800000021:57:270
3017402995015,5cyclictest16850irq/134-800000023:30:310
3017402995015,18cyclictest16850irq/134-800000022:00:050
3017402995015,17cyclictest16850irq/134-800000021:26:170
301741799498,28cyclictest3023091-21modprobe21:00:013
3017417994918,7cyclictest3023695-21latency_hist21:10:013
3017410994941,5cyclictest3010825-21kworker/2:122:49:282
3017410994940,5cyclictest3021822-21kworker/2:021:24:282
3017410994940,5cyclictest3010825-21kworker/2:122:07:172
3017410994936,8cyclictest1-21systemd23:20:002
3017410994936,8cyclictest1-21systemd23:20:002
301741099492,4cyclictest131rcu_preempt00:20:012
3017410994919,17cyclictest30-21ksoftirqd/223:47:182
3017410994919,17cyclictest30-21ksoftirqd/223:47:172
301740299493,16cyclictest3023125-21grep21:00:000
3017402994914,16cyclictest16850irq/134-800000000:14:510
3017417994832,13cyclictest3084411-21systemd-run23:25:003
3017410994840,5cyclictest3109145-21kworker/2:200:36:572
3017410994840,5cyclictest3091709-21kworker/2:123:50:132
3017410994840,5cyclictest3055202-21kworker/2:323:38:532
3017410994840,5cyclictest3010825-21kworker/2:122:25:222
3017410994839,6cyclictest3091709-21kworker/2:123:44:492
3017410994839,5cyclictest3055202-21kworker/2:300:08:542
3017410994839,5cyclictest3021822-21kworker/2:021:35:322
3017410994831,12cyclictest275-21dbus-daemon22:40:002
3017410994812,6cyclictest131rcu_preempt23:25:382
3017410994812,6cyclictest131rcu_preempt23:25:382
3017405994835,10cyclictest3061836-21/usr/sbin/munin22:35:011
301741799476,6cyclictest131rcu_preempt22:45:013
301741799474,5cyclictest131rcu_preempt23:56:043
301741799473,34cyclictest3467602-21systemd-journal23:20:013
301741799473,34cyclictest3467602-21systemd-journal23:20:003
301741099477,4cyclictest131rcu_preempt23:00:022
3017410994741,3cyclictest3077662-21kworker/2:023:14:102
3017410994741,3cyclictest3021822-21kworker/2:021:26:032
3017410994740,4cyclictest3055202-21kworker/2:323:05:532
3017410994740,4cyclictest3055202-21kworker/2:300:21:022
3017410994740,4cyclictest3055202-21kworker/2:300:21:022
3017410994739,5cyclictest3010825-21kworker/2:121:44:072
3017410994739,5cyclictest3010825-21kworker/2:121:44:072
3017405994727,14cyclictest3467602-21systemd-journal22:05:001
3017405994711,6cyclictest131rcu_preempt00:38:491
301740299473,3cyclictest131rcu_preempt21:24:560
3017402994727,15cyclictest1-21systemd20:45:000
3017402994719,12cyclictest3079544-21ssh23:14:140
3017402994713,6cyclictest16850irq/134-800000023:20:000
3017402994713,6cyclictest16850irq/134-800000023:19:590
3017402994712,17cyclictest16850irq/134-800000000:30:320
301741799463,3cyclictest131rcu_preempt00:02:483
3017417994633,9cyclictest37-21ksoftirqd/300:11:103
3017417994633,8cyclictest3019734-21grep19:55:003
3017417994630,8cyclictest309-21in:imuxsock22:15:023
3017417994630,11cyclictest3467602-21systemd-journal21:55:013
3017417994623,5cyclictest37-21ksoftirqd/300:30:003
3017410994640,3cyclictest3010825-21kworker/2:121:45:392
3017410994639,4cyclictest3010825-21kworker/2:121:55:592
3017410994638,5cyclictest3091709-21kworker/2:123:56:452
3017410994638,5cyclictest3055202-21kworker/2:300:13:062
3017410994638,5cyclictest30-21ksoftirqd/220:35:012
3017410994632,10cyclictest3467602-21systemd-journal21:00:362
3017405994633,10cyclictest3073108-21cron23:00:001
301740599462,9cyclictest1-21systemd20:55:001
3017405994625,14cyclictest3467602-21systemd-journal19:50:001
301741799453,30cyclictest3021307-21munin-plugin-st20:24:593
3017410994537,5cyclictest3021822-21kworker/2:021:34:162
3017410994537,5cyclictest3021822-21kworker/2:021:34:162
3017410994537,5cyclictest3010825-21kworker/2:120:40:032
3017410994531,10cyclictest30-21ksoftirqd/200:01:562
301740599454,4cyclictest131rcu_preempt00:00:001
3017405994532,10cyclictest23-21ksoftirqd/121:50:011
3017405994528,12cyclictest275-21dbus-daemon22:15:011
3017405994528,12cyclictest275-21dbus-daemon22:15:011
301740599451,3cyclictest131rcu_preempt00:15:031
3017405994510,5cyclictest131rcu_preempt22:30:011
3017402994528,12cyclictest275-21dbus-daemon20:35:010
301740299452,5cyclictest3020759-21sh20:15:000
3017417994433,7cyclictest3115883-21cron00:35:013
3017417994429,11cyclictest37-21ksoftirqd/323:05:033
3017405994416,7cyclictest3109574-21sh00:20:421
3017405994416,7cyclictest3109574-21sh00:20:411
301740299443,7cyclictest3021019-21modprobe20:20:000
301740299442,8cyclictest3018957-21munin-run19:40:000
301740299442,8cyclictest3018957-21munin-run19:40:000
301741799435,8cyclictest131rcu_preempt23:40:003
3017417994327,13cyclictest275-21dbus-daemon20:05:003
3017417994327,12cyclictest275-21dbus-daemon22:10:003
3017417994327,11cyclictest37-21ksoftirqd/321:50:013
3017417994327,11cyclictest3467602-21systemd-journal21:39:593
3017417994325,13cyclictest275-21dbus-daemon21:15:003
3017417994325,13cyclictest275-21dbus-daemon21:15:003
3017417994324,13cyclictest1-21systemd00:25:023
3017417994324,13cyclictest1-21systemd00:25:013
301741099433,4cyclictest131rcu_preempt19:40:022
301741099433,4cyclictest131rcu_preempt19:40:022
3017410994331,8cyclictest3467602-21systemd-journal19:25:002
3017405994328,11cyclictest23-21ksoftirqd/121:10:361
3017405994328,11cyclictest23-21ksoftirqd/121:10:351
3017405994324,14cyclictest3467602-21systemd-journal21:09:591
3017405994316,10cyclictest0-21swapper/121:37:231
3017402994336,4cyclictest16950irq/224-800000020:00:010
301740299432,4cyclictest131rcu_preempt19:45:010
301741799429,6cyclictest131rcu_preempt22:25:013
3017417994232,7cyclictest37-21ksoftirqd/323:55:013
3017417994228,11cyclictest3467718-21lldpd21:20:153
3017417994226,11cyclictest3020767-21munin-plugin-st20:15:003
3017417994221,16cyclictest275-21dbus-daemon23:30:013
3017417994221,16cyclictest275-21dbus-daemon23:30:003
3017410994234,5cyclictest3010825-21kworker/2:119:26:592
3017405994228,11cyclictest23-21ksoftirqd/121:44:031
3017405994228,11cyclictest23-21ksoftirqd/121:44:031
3017405994226,12cyclictest23-21ksoftirqd/123:45:001
3017405994215,6cyclictest131rcu_preempt23:47:151
3017405994215,6cyclictest131rcu_preempt23:47:151
301740299422,35cyclictest3023636-21latency_hist21:10:020
301741799417,6cyclictest131rcu_preempt23:00:003
301741799414,33cyclictest3048406-21ssh22:05:003
3017417994130,8cyclictest37-21ksoftirqd/319:45:013
3017417994128,9cyclictest37-21ksoftirqd/300:09:573
3017417994127,11cyclictest37-21ksoftirqd/322:32:583
3017417994127,11cyclictest37-21ksoftirqd/322:28:073
3017410994133,5cyclictest3010825-21kworker/2:121:50:192
3017410994133,5cyclictest3010825-21kworker/2:121:08:522
3017410994132,6cyclictest1-21systemd19:55:012
3017410994129,9cyclictest275-21dbus-daemon19:35:002
301740599416,4cyclictest131rcu_preempt19:35:011
3017405994126,11cyclictest275-21dbus-daemon20:00:001
3017405994126,11cyclictest23-21ksoftirqd/123:28:481
3017405994126,11cyclictest23-21ksoftirqd/123:28:481
3017405994126,11cyclictest23-21ksoftirqd/119:20:001
3017405994125,13cyclictest3019879-21sed19:55:021
301740599411,20cyclictest0-21swapper/100:34:521
3017402994134,4cyclictest16950irq/224-800000021:05:000
3017402994134,4cyclictest16950irq/224-800000019:15:000
301741799408,23cyclictest3022336-21modprobe20:45:013
3017417994031,5cyclictest3077627-21cron23:10:013
3017417994030,8cyclictest37-21ksoftirqd/320:20:023
3017417994026,8cyclictest3064090-21cron22:39:593
3017417994024,8cyclictest309-21in:imuxsock00:20:003
301741799401,35cyclictest3467602-21systemd-journal19:25:353
3017410994032,5cyclictest3010825-21kworker/2:119:45:022
3017410994017,7cyclictest131rcu_preempt19:15:002
301740599406,5cyclictest131rcu_preempt23:50:451
301740599404,4cyclictest131rcu_preempt22:10:021
301740599403,3cyclictest131rcu_preempt19:40:031
301740599403,3cyclictest131rcu_preempt19:40:021
3017405994028,8cyclictest3023140-21systemd-run21:00:001
3017405994027,10cyclictest23-21ksoftirqd/121:58:201
3017405994024,12cyclictest23-21ksoftirqd/123:04:001
301740599402,15cyclictest3031821-21ssh21:28:001
301740599401,5cyclictest0-21swapper/122:22:501
301740599401,3cyclictest131rcu_preempt22:55:031
301740599401,13cyclictest131rcu_preempt23:10:021
301741799393,8cyclictest0-21swapper/321:28:053
3017417993927,9cyclictest37-21ksoftirqd/320:35:013
3017417993925,10cyclictest37-21ksoftirqd/319:25:023
3017417993921,13cyclictest275-21dbus-daemon23:50:013
3017417993921,13cyclictest275-21dbus-daemon23:50:013
301740599393,3cyclictest131rcu_preempt21:25:011
3017405993930,7cyclictest23-21ksoftirqd/123:16:201
3017405993930,7cyclictest23-21ksoftirqd/123:16:191
3017405993929,8cyclictest23-21ksoftirqd/120:15:011
3017405993929,5cyclictest309-21in:imuxsock19:45:001
3017405993927,9cyclictest309-21in:imuxsock20:16:391
3017405993925,11cyclictest23-21ksoftirqd/120:50:001
3017405993924,12cyclictest23-21ksoftirqd/122:15:591
301740599392,13cyclictest3043551-21ssh21:54:151
3017402993929,6cyclictest19350irq/34-mmc120:23:040
3017402993929,5cyclictest3022846-21munin-plugin-st20:55:000
3017402993928,7cyclictest19350irq/34-mmc120:30:000
3017402993926,9cyclictest3022050-21cron20:40:000
3017402993917,18cyclictest2784600-1kworker/0:1H19:49:500
301741799387,5cyclictest131rcu_preempt19:10:023
301741799387,5cyclictest131rcu_preempt19:10:023
301741799387,23cyclictest3467602-21systemd-journal20:10:003
3017417993826,9cyclictest37-21ksoftirqd/323:31:323
3017417993826,10cyclictest37-21ksoftirqd/319:15:013
3017417993824,11cyclictest37-21ksoftirqd/321:05:013
3017417993820,12cyclictest311-21rs:main3
3017410993830,5cyclictest3010825-21kworker/2:120:24:142
3017410993825,10cyclictest30-21ksoftirqd/219:50:002
301740599384,3cyclictest131rcu_preempt23:36:151
301740599384,3cyclictest131rcu_preempt23:35:001
3017405993825,9cyclictest23-21ksoftirqd/120:01:021
3017405993824,11cyclictest23-21ksoftirqd/120:37:341
3017405993821,12cyclictest275-21dbus-daemon21:05:011
3017405993811,6cyclictest131rcu_preempt21:16:261
3017402993826,8cyclictest3467602-21systemd-journal19:35:000
3017402993819,4cyclictest3019701-21sshd19:51:340
3017417993727,7cyclictest37-21ksoftirqd/319:35:003
3017417993726,9cyclictest37-21ksoftirqd/319:20:003
3017417993724,10cyclictest37-21ksoftirqd/322:53:083
3017417993724,10cyclictest37-21ksoftirqd/322:14:373
3017417993724,10cyclictest37-21ksoftirqd/322:14:363
3017410993730,4cyclictest3023102-21sendmail21:00:002
3017410993721,12cyclictest3467602-21systemd-journal20:55:002
3017405993730,4cyclictest3467562-21sshd20:26:401
3017405993727,7cyclictest23-21ksoftirqd/100:00:181
3017405993726,8cyclictest3021284-21sshd20:23:351
3017405993726,8cyclictest23-21ksoftirqd/123:13:481
301740599372,3cyclictest131rcu_preempt19:15:011
3017402993726,7cyclictest12-21ksoftirqd/020:10:020
3017402993726,6cyclictest19350irq/34-mmc119:25:010
3017417993624,9cyclictest37-21ksoftirqd/319:50:023
3017417993621,11cyclictest37-21ksoftirqd/320:00:013
3017410993629,5cyclictest30-21ksoftirqd/220:30:012
3017410993622,10cyclictest275-21dbus-daemon20:45:002
301740599362,3cyclictest131rcu_preempt19:30:031
301740599361,3cyclictest131rcu_preempt23:23:551
301740299366,23cyclictest3017412-21grep19:10:010
301740299366,23cyclictest3017412-21grep19:10:010
3017402993627,6cyclictest19350irq/34-mmc120:47:570
301740299361,3cyclictest3017823-21modprobe19:16:270
301740299361,31cyclictest0-21swapper/020:03:250
301741799354,25cyclictest1-21systemd20:40:013
301741099353,3cyclictest131rcu_preempt20:50:012
3017410993524,8cyclictest3021261-21sed20:20:022
301741099351,3cyclictest131rcu_preempt20:10:002
3017405993524,6cyclictest309-21in:imuxsock20:35:001
301740599342,4cyclictest131rcu_preempt19:25:011
301740299341,29cyclictest0-21swapper/019:28:030
3017410993325,5cyclictest3467562-21sshd20:11:372
301740599332,3cyclictest131rcu_preempt20:45:011
3017405993321,9cyclictest3017468-21cut19:10:021
3017405993321,9cyclictest3017468-21cut19:10:021
3017410993220,9cyclictest30-21ksoftirqd/220:00:002
3017410993124,4cyclictest3020446-21latency_hist20:05:022
3017410993119,8cyclictest30-21ksoftirqd/219:10:022
3017410993119,8cyclictest30-21ksoftirqd/219:10:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional