You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 07:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sat May 02, 2026 00:45:25)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
638810997225,11cyclictest275-21dbus-daemon22:45:003
638810997020,13cyclictest1-21systemd23:20:013
638810996726,8cyclictest658250-21latency_hist21:40:023
638810996628,5cyclictest661448-21sh21:47:383
638810996625,7cyclictest692644-21sh23:00:003
638801996513,30cyclictest12-21ksoftirqd/021:02:570
638807996454,6cyclictest703479-21kworker/2:100:30:022
63881099634,9cyclictest0-21swapper/320:25:023
638801996315,28cyclictest12-21ksoftirqd/021:30:010
638810996227,11cyclictest37-21ksoftirqd/323:02:233
638810996227,11cyclictest37-21ksoftirqd/323:02:223
63880599614,11cyclictest23-21ksoftirqd/123:35:021
638805996126,11cyclictest23-21ksoftirqd/121:59:591
638810996026,9cyclictest37-21ksoftirqd/322:22:373
63880599606,11cyclictest23-21ksoftirqd/100:30:011
638801996010,5cyclictest12-21ksoftirqd/000:32:500
638810995932,5cyclictest684032-21latency_hist22:40:003
638805995920,7cyclictest23-21ksoftirqd/123:14:501
638801995951,4cyclictest16950irq/224-800000022:15:000
638810995825,8cyclictest37-21ksoftirqd/300:15:183
63880599584,9cyclictest23-21ksoftirqd/122:15:011
63880599584,11cyclictest23-21ksoftirqd/123:30:011
63880199585,5cyclictest131rcu_preempt22:58:360
638810995731,9cyclictest1-21systemd20:50:013
638810995727,7cyclictest37-21ksoftirqd/322:02:313
638810995727,7cyclictest37-21ksoftirqd/322:02:303
638801995737,17cyclictest12-21ksoftirqd/023:55:270
638801995728,14cyclictest729310-21ssh00:24:430
638801995722,31cyclictest12-21ksoftirqd/022:47:380
638801995719,7cyclictest16850irq/134-800000023:37:380
63880199571,31cyclictest680807-21kworker/u8:223:02:360
63880199571,31cyclictest680807-21kworker/u8:223:02:350
63880599567,5cyclictest131rcu_preempt21:50:021
63880599564,10cyclictest23-21ksoftirqd/121:35:001
638805995637,14cyclictest1-21systemd00:25:011
63880599563,11cyclictest23-21ksoftirqd/100:35:031
638805995613,11cyclictest23-21ksoftirqd/122:35:021
638805995613,11cyclictest23-21ksoftirqd/122:35:011
63880199568,29cyclictest16850irq/134-800000022:50:030
63880199568,29cyclictest16850irq/134-800000022:50:020
638801995628,12cyclictest714919-21ssh23:51:070
638801995619,7cyclictest16850irq/134-800000023:48:100
638801995619,7cyclictest16850irq/134-800000023:48:090
638807995539,8cyclictest703479-21kworker/2:100:10:342
638807995537,14cyclictest1-21systemd00:05:012
63880599556,11cyclictest23-21ksoftirqd/121:19:591
638805995534,13cyclictest3467602-21systemd-journal20:39:591
638805995511,5cyclictest131rcu_preempt00:20:021
638805995510,6cyclictest131rcu_preempt19:55:001
638801995526,13cyclictest696139-21ssh23:07:540
63880199552,37cyclictest702235-21ssh23:21:570
638801995519,17cyclictest16850irq/134-800000023:40:370
638805995436,14cyclictest1-21systemd22:40:011
63880199543,16cyclictest666661-21cron22:00:000
638801995428,22cyclictest12-21ksoftirqd/023:33:500
638801995428,11cyclictest658355-21ssh21:40:090
638801995425,14cyclictest698549-21ssh23:13:320
638801995420,6cyclictest16850irq/134-800000022:37:570
638801995419,17cyclictest16850irq/134-800000021:11:010
638801995418,18cyclictest16850irq/134-800000022:43:590
638810995324,9cyclictest37-21ksoftirqd/323:42:293
63881099531,35cyclictest361rcuc/300:10:033
638807995343,7cyclictest666719-21kworker/2:323:20:012
638807995343,6cyclictest644826-21kworker/2:223:48:062
638807995343,6cyclictest644826-21kworker/2:223:48:062
638805995334,14cyclictest23-21ksoftirqd/123:23:031
63880599532,13cyclictest23-21ksoftirqd/123:05:011
63880599532,13cyclictest23-21ksoftirqd/123:05:001
63880599531,16cyclictest0-21swapper/122:57:581
638801995326,12cyclictest681661-21ssh22:34:310
638801995326,12cyclictest681661-21ssh22:34:300
638801995319,30cyclictest654093-21sh21:30:240
638801995319,17cyclictest16850irq/134-800000021:52:490
63880199531,36cyclictest700270-21ssh23:17:250
638807995244,4cyclictest644826-21kworker/2:222:49:002
638807995243,6cyclictest638317-21kworker/2:121:23:182
638807995243,5cyclictest644826-21kworker/2:222:45:012
63880599529,9cyclictest23-21ksoftirqd/123:40:001
63880599529,5cyclictest131rcu_preempt22:40:351
63880599528,8cyclictest23-21ksoftirqd/100:06:151
63880599528,5cyclictest131rcu_preempt21:25:011
63880599525,10cyclictest23-21ksoftirqd/121:40:021
63880199524,3cyclictest131rcu_preempt00:14:500
638801995229,8cyclictest705116-21ssh23:28:470
638801995228,8cyclictest719502-21ssh00:01:490
638801995224,13cyclictest678769-21ssh22:27:440
638801995218,17cyclictest16850irq/134-800000021:45:560
63880199521,4cyclictest131rcu_preempt22:05:010
63880199521,4cyclictest131rcu_preempt22:05:000
63881099519,6cyclictest131rcu_preempt22:50:003
638807995143,5cyclictest291rcuc/200:00:022
638807995142,6cyclictest703479-21kworker/2:100:06:312
638807995142,5cyclictest644826-21kworker/2:223:51:562
638807995141,6cyclictest703479-21kworker/2:100:30:182
63880599516,10cyclictest23-21ksoftirqd/122:06:251
63880599513,9cyclictest23-21ksoftirqd/119:35:011
63880599512,6cyclictest23-21ksoftirqd/122:50:001
63880599512,12cyclictest23-21ksoftirqd/123:10:021
638805995118,10cyclictest23-21ksoftirqd/100:03:151
638805995117,12cyclictest23-21ksoftirqd/121:15:011
638805995113,6cyclictest131rcu_preempt20:00:001
638801995119,18cyclictest12-21ksoftirqd/021:40:000
638810995034,12cyclictest3467602-21systemd-journal23:40:003
638807995043,5cyclictest638317-21kworker/2:121:18:092
638807995042,5cyclictest666719-21kworker/2:322:18:392
638807995042,5cyclictest644826-21kworker/2:222:55:582
638807995033,11cyclictest275-21dbus-daemon19:40:012
638807995033,11cyclictest275-21dbus-daemon19:40:012
63880599509,5cyclictest131rcu_preempt00:38:201
63880599507,11cyclictest23-21ksoftirqd/121:29:591
63880599507,10cyclictest23-21ksoftirqd/123:20:001
63880599507,10cyclictest23-21ksoftirqd/122:25:011
63880599506,8cyclictest23-21ksoftirqd/123:45:091
63880599506,8cyclictest23-21ksoftirqd/123:45:081
63880599505,8cyclictest23-21ksoftirqd/100:12:071
63880599504,9cyclictest23-21ksoftirqd/122:02:571
63880599504,9cyclictest23-21ksoftirqd/122:02:571
63880599503,12cyclictest23-21ksoftirqd/122:30:021
63880599501,16cyclictest0-21swapper/121:55:011
638801995027,17cyclictest1-21systemd21:20:010
638801995023,12cyclictest675809-21ssh22:20:400
638801995022,13cyclictest650858-21ssh21:23:140
638807994941,5cyclictest666719-21kworker/2:322:20:132
638807994941,5cyclictest644826-21kworker/2:223:42:362
638807994941,5cyclictest644826-21kworker/2:222:09:422
638807994941,5cyclictest644826-21kworker/2:221:53:192
638807994940,6cyclictest666719-21kworker/2:323:10:432
63880799492,3cyclictest131rcu_preempt23:35:002
638805994933,11cyclictest3467602-21systemd-journal20:24:591
63880599491,8cyclictest275-21dbus-daemon20:55:011
638805994918,21cyclictest221rcuc/100:00:011
638805994913,8cyclictest131rcu_preempt19:25:011
638801994929,14cyclictest275-21dbus-daemon19:25:010
638801994922,3cyclictest131rcu_preempt00:05:390
638810994836,9cyclictest716409-21cron23:55:003
638807994840,5cyclictest666719-21kworker/2:322:27:562
638807994840,5cyclictest666719-21kworker/2:322:14:402
638807994840,5cyclictest644826-21kworker/2:221:57:402
638807994840,5cyclictest638317-21kworker/2:121:38:512
638807994840,5cyclictest638317-21kworker/2:121:33:432
63880199485,30cyclictest638803-21latency_hist19:10:020
638801994821,12cyclictest725530-21ssh00:15:430
63880199481,3cyclictest131rcu_preempt00:27:350
638807994741,3cyclictest638317-21kworker/2:121:43:372
638807994739,5cyclictest666719-21kworker/2:323:20:332
638807994739,5cyclictest644826-21kworker/2:222:54:142
638807994739,5cyclictest644826-21kworker/2:222:54:142
638807994739,5cyclictest644826-21kworker/2:221:45:472
638807994739,5cyclictest638317-21kworker/2:121:09:192
638807994739,5cyclictest638317-21kworker/2:120:46:562
638807994739,5cyclictest638317-21kworker/2:120:16:572
638807994739,5cyclictest638317-21kworker/2:120:16:562
638807994738,6cyclictest703479-21kworker/2:100:15:552
638807994738,6cyclictest666719-21kworker/2:323:08:372
638807994738,5cyclictest644826-21kworker/2:223:40:002
63880599477,12cyclictest23-21ksoftirqd/122:16:591
63880599475,9cyclictest23-21ksoftirqd/119:45:011
63880599475,6cyclictest131rcu_preempt23:44:071
63880599474,6cyclictest131rcu_preempt23:51:441
638805994715,6cyclictest131rcu_preempt19:50:031
63880599471,30cyclictest23-21ksoftirqd/119:30:031
638801994740,4cyclictest16850irq/134-800000022:10:000
638810994631,11cyclictest3467602-21systemd-journal20:45:003
638810994615,22cyclictest0-21swapper/321:41:323
63880599467,8cyclictest23-21ksoftirqd/121:40:561
63880599464,13cyclictest23-21ksoftirqd/121:10:011
63880599462,3cyclictest0-21swapper/121:00:001
63880599461,8cyclictest643032-21latency_hist20:30:021
638805994617,9cyclictest23-21ksoftirqd/122:52:101
638805994617,9cyclictest23-21ksoftirqd/122:52:091
638805994617,6cyclictest131rcu_preempt19:15:011
638805994611,4cyclictest131rcu_preempt20:20:031
638805994611,4cyclictest131rcu_preempt20:20:021
638801994633,9cyclictest16950irq/224-800000022:19:590
63880199462,38cyclictest644526-21modprobe21:00:000
63880199461,3cyclictest131rcu_preempt19:30:010
638810994531,11cyclictest275-21dbus-daemon00:00:003
638810994530,11cyclictest3467602-21systemd-journal19:31:443
638807994539,3cyclictest644826-21kworker/2:222:35:092
638807994537,5cyclictest644826-21kworker/2:222:03:082
638807994537,5cyclictest644826-21kworker/2:222:03:072
638807994537,5cyclictest644826-21kworker/2:200:36:402
638807994529,12cyclictest640885-21sshd19:48:552
63880599454,7cyclictest23-21ksoftirqd/120:35:021
638805994533,10cyclictest23-21ksoftirqd/120:15:001
638805994513,6cyclictest131rcu_preempt20:10:011
638801994541,2cyclictest0-21swapper/020:00:000
638801994529,13cyclictest643732-21sshd20:40:560
638801994528,14cyclictest734891-21ssh00:37:180
63880199452,7cyclictest1-21systemd21:10:000
638810994429,11cyclictest681786-21sshd22:34:583
638810994429,11cyclictest681786-21sshd22:34:573
63881099441,6cyclictest666926-21sed22:00:023
638807994439,3cyclictest638317-21kworker/2:121:29:432
638807994435,5cyclictest638317-21kworker/2:120:05:012
63880199447,33cyclictest642457-21sendmail20:20:000
63880199447,33cyclictest642457-21sendmail20:20:000
638801994429,12cyclictest642902-21sed20:25:020
63881099434,3cyclictest131rcu_preempt19:55:023
638810994328,10cyclictest275-21dbus-daemon20:15:013
638810994325,13cyclictest275-21dbus-daemon19:40:013
638810994325,13cyclictest275-21dbus-daemon19:40:003
63880799437,11cyclictest30-21ksoftirqd/220:52:562
638807994335,5cyclictest703479-21kworker/2:100:21:282
638807994334,6cyclictest644826-21kworker/2:223:02:052
638807994334,6cyclictest644826-21kworker/2:223:02:052
63880599431,10cyclictest23-21ksoftirqd/119:10:021
638801994328,12cyclictest642158-21sshd20:10:570
638801994328,11cyclictest1-21systemd20:55:000
638810994228,11cyclictest37-21ksoftirqd/322:14:483
63881099422,12cyclictest644258-21cron20:55:003
638807994234,5cyclictest638317-21kworker/2:121:14:352
63880799422,20cyclictest0-21swapper/220:00:022
63880799421,3cyclictest131rcu_preempt22:32:072
63880799421,3cyclictest131rcu_preempt22:32:072
638805994226,11cyclictest1-21systemd19:40:021
638805994226,11cyclictest1-21systemd19:40:011
638801994227,11cyclictest12-21ksoftirqd/020:05:000
638801994221,4cyclictest12-21ksoftirqd/020:30:000
63880199421,4cyclictest640338-21modprobe19:40:000
63880199421,4cyclictest640338-21modprobe19:40:000
638810994131,8cyclictest37-21ksoftirqd/323:48:573
638810994131,8cyclictest37-21ksoftirqd/323:48:563
638810994131,7cyclictest3467602-21systemd-journal19:47:563
638810994113,21cyclictest0-21swapper/321:23:233
638810994112,21cyclictest0-21swapper/323:31:273
638810994110,9cyclictest719945-21ssh00:03:043
638807994133,5cyclictest638992-21kworker/2:020:29:022
638807994132,5cyclictest638317-21kworker/2:120:59:222
638807994129,10cyclictest30-21ksoftirqd/223:30:012
638805994131,7cyclictest23-21ksoftirqd/120:05:021
638805994127,11cyclictest23-21ksoftirqd/119:20:011
63880199414,3cyclictest131rcu_preempt20:50:030
638810994027,9cyclictest3467602-21systemd-journal19:57:573
638810994025,12cyclictest1-21systemd22:10:013
638807994032,5cyclictest638992-21kworker/2:020:31:482
638807994032,5cyclictest638317-21kworker/2:120:13:432
638807994022,13cyclictest3467602-21systemd-journal19:50:562
638805994031,6cyclictest23-21ksoftirqd/120:47:551
638801994032,5cyclictest19350irq/34-mmc119:40:570
638810993930,7cyclictest37-21ksoftirqd/320:05:023
638810993930,5cyclictest309-21in:imuxsock22:30:013
638810993927,9cyclictest37-21ksoftirqd/300:26:153
638810993927,10cyclictest674106-21sshd22:16:573
638810993925,10cyclictest642464-21systemctl20:20:013
638810993925,10cyclictest642464-21systemctl20:20:003
638810993924,11cyclictest37-21ksoftirqd/321:32:213
638810993918,15cyclictest3467602-21systemd-journal20:40:003
638810993915,16cyclictest37-21ksoftirqd/321:05:003
638807993927,9cyclictest3467602-21systemd-journal20:41:562
638807993927,8cyclictest30-21ksoftirqd/219:15:012
63880599392,10cyclictest23-21ksoftirqd/121:02:561
63880599391,7cyclictest0-21swapper/120:41:541
63880199398,8cyclictest12-21ksoftirqd/019:46:310
63881099386,7cyclictest131rcu_preempt21:10:003
63881099386,5cyclictest131rcu_preempt23:27:403
638810993825,10cyclictest736012-21cat00:40:013
638810993822,11cyclictest275-21dbus-daemon19:30:023
638807993830,5cyclictest638317-21kworker/2:119:20:102
63880199386,5cyclictest131rcu_preempt19:55:000
638801993827,8cyclictest12-21ksoftirqd/019:35:020
638801993827,8cyclictest12-21ksoftirqd/019:20:000
63881099379,6cyclictest131rcu_preempt21:15:523
638810993724,10cyclictest37-21ksoftirqd/323:07:453
638810993722,11cyclictest37-21ksoftirqd/321:30:003
63881099371,13cyclictest698100-21ssh23:12:263
63881099371,11cyclictest131rcu_preempt21:50:543
638807993726,9cyclictest30-21ksoftirqd/220:23:562
638801993730,4cyclictest16950irq/224-800000020:06:530
638801993726,8cyclictest643709-21sed20:40:020
638810993629,4cyclictest689250-21sh22:51:503
638810993629,4cyclictest689250-21sh22:51:493
638810993626,8cyclictest37-21ksoftirqd/321:11:403
638810993625,9cyclictest37-21ksoftirqd/300:10:433
638807993628,5cyclictest638317-21kworker/2:119:10:022
638807993620,11cyclictest275-21dbus-daemon19:35:012
638807993612,8cyclictest30-21ksoftirqd/221:05:022
638801993624,9cyclictest12-21ksoftirqd/019:15:010
63881099358,4cyclictest131rcu_preempt19:12:373
63881099352,7cyclictest733586-21ssh00:34:383
638810993525,8cyclictest37-21ksoftirqd/300:22:563
63881099351,3cyclictest131rcu_preempt23:25:013
63880799351,10cyclictest640639-21sshd19:44:562
638810993427,4cyclictest1-21systemd19:25:013
638810993426,5cyclictest641938-21latency_hist20:10:013
638810993422,9cyclictest3467602-21systemd-journal20:30:013
638807993422,8cyclictest30-21ksoftirqd/219:18:312
638807993421,10cyclictest639931-21sed19:30:022
63880799341,12cyclictest642045-21ls20:10:022
63880199342,29cyclictest48-21kcompactd020:33:250
638810993215,11cyclictest644541-21sendmail21:00:003
63881099321,3cyclictest131rcu_preempt19:20:013
63881099321,3cyclictest131rcu_preempt19:10:033
63881099313,7cyclictest131rcu_preempt19:44:593
638810993125,4cyclictest643238-21/usr/sbin/munin20:35:003
638807993123,5cyclictest638992-21kworker/2:020:40:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional