You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 04:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Mar 10, 2026 00:45:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1135940991371,3cyclictest0-21swapper/000:10:020
113594099132123,6cyclictest12-21ksoftirqd/020:00:020
11359569910214,8cyclictest131rcu_preempt22:20:033
113594799991,22cyclictest23-21ksoftirqd/120:45:021
1135947998048,9cyclictest23-21ksoftirqd/122:08:121
113594799697,13cyclictest23-21ksoftirqd/121:25:021
113594799697,13cyclictest23-21ksoftirqd/121:25:021
1135951996640,8cyclictest1229370-21kworker/2:300:34:082
113594099667,6cyclictest131rcu_preempt22:25:010
113594799654,8cyclictest23-21ksoftirqd/120:00:001
1135947996522,8cyclictest131rcu_preempt23:37:321
1135947996522,8cyclictest131rcu_preempt23:37:321
1135951996354,7cyclictest30-21ksoftirqd/223:10:022
1135951996339,13cyclictest1180124-21kworker/2:123:22:482
1135947996325,13cyclictest23-21ksoftirqd/121:20:001
1135947996314,7cyclictest1141617-21systemd-run21:00:001
113594099629,9cyclictest12-21ksoftirqd/022:41:220
113595199617,6cyclictest131rcu_preempt22:55:022
113595199614,9cyclictest30-21ksoftirqd/200:25:002
1135951996137,16cyclictest1224867-21kworker/2:000:17:442
1135951996137,16cyclictest1224867-21kworker/2:000:17:442
1135951996114,6cyclictest131rcu_preempt21:46:432
1135947996118,13cyclictest23-21ksoftirqd/123:10:011
1135947996114,5cyclictest131rcu_preempt00:16:551
1135947996114,5cyclictest131rcu_preempt00:16:551
113594099618,34cyclictest12-21ksoftirqd/023:17:120
1135940996138,14cyclictest0-21swapper/022:05:010
1135951996012,6cyclictest131rcu_preempt23:38:382
1135951996012,6cyclictest131rcu_preempt23:38:382
1135947996014,12cyclictest23-21ksoftirqd/121:50:011
113594099601,11cyclictest12-21ksoftirqd/022:28:070
113594099601,11cyclictest12-21ksoftirqd/022:28:070
1135951995936,13cyclictest1224867-21kworker/2:000:37:062
1135951995915,6cyclictest131rcu_preempt00:05:062
1135951995914,6cyclictest131rcu_preempt23:18:592
1135951995914,6cyclictest131rcu_preempt19:24:592
1135951995914,6cyclictest131rcu_preempt00:26:522
1135951995911,7cyclictest131rcu_preempt20:10:022
113594799591,16cyclictest23-21ksoftirqd/123:45:121
1135951995832,10cyclictest30-21ksoftirqd/223:50:562
113594799587,10cyclictest23-21ksoftirqd/123:20:591
113594799584,13cyclictest23-21ksoftirqd/123:15:021
113594799584,13cyclictest23-21ksoftirqd/123:15:021
113594799582,8cyclictest23-21ksoftirqd/122:45:021
113594799582,11cyclictest23-21ksoftirqd/121:30:011
1135947995815,12cyclictest23-21ksoftirqd/121:55:021
1135947995814,9cyclictest23-21ksoftirqd/100:05:001
1135947995814,10cyclictest23-21ksoftirqd/100:28:181
1135947995813,12cyclictest23-21ksoftirqd/100:37:241
113594099589,6cyclictest131rcu_preempt23:55:010
1135940995842,10cyclictest1135944-21cat19:10:010
113594099583,4cyclictest131rcu_preempt22:15:010
1135940995830,24cyclictest11426751ssh21:11:050
1135940995829,13cyclictest1150991-21ssh21:29:540
113594099581,51cyclictest274-21cron00:00:000
113594099581,51cyclictest274-21cron00:00:000
113595699578,5cyclictest131rcu_preempt00:10:013
113595199578,6cyclictest131rcu_preempt23:30:012
1135951995712,6cyclictest131rcu_preempt22:58:592
1135951995711,5cyclictest131rcu_preempt22:19:042
113594799575,7cyclictest23-21ksoftirqd/100:00:021
113594799575,7cyclictest23-21ksoftirqd/100:00:021
113594799575,11cyclictest23-21ksoftirqd/121:15:011
113594799575,11cyclictest23-21ksoftirqd/100:20:291
1135947995739,14cyclictest1-21systemd22:25:011
1135947995716,9cyclictest23-21ksoftirqd/123:45:001
1135947995716,12cyclictest0-21swapper/123:35:011
1135947995713,9cyclictest23-21ksoftirqd/122:14:201
1135940995730,13cyclictest12-21ksoftirqd/020:50:020
1135940995730,13cyclictest12-21ksoftirqd/020:50:020
1135940995719,33cyclictest16850irq/134-800000022:51:500
1135940995713,10cyclictest12-21ksoftirqd/022:09:170
113595199567,4cyclictest131rcu_preempt21:05:012
113595199567,4cyclictest131rcu_preempt21:05:012
113595199566,12cyclictest30-21ksoftirqd/223:35:012
113595199565,9cyclictest30-21ksoftirqd/222:00:012
1135951995642,5cyclictest1137974-21kworker/2:321:11:002
1135951995618,21cyclictest30-21ksoftirqd/221:35:072
1135951995616,6cyclictest131rcu_preempt00:00:002
1135951995616,6cyclictest131rcu_preempt00:00:002
1135951995616,5cyclictest131rcu_preempt22:27:512
1135951995616,5cyclictest131rcu_preempt22:27:512
1135951995615,5cyclictest131rcu_preempt22:35:252
1135951995614,6cyclictest131rcu_preempt23:47:002
1135951995614,5cyclictest131rcu_preempt22:42:172
1135951995613,5cyclictest131rcu_preempt21:17:142
1135951995611,6cyclictest131rcu_preempt23:14:322
1135951995611,6cyclictest131rcu_preempt23:14:322
1135951995611,6cyclictest131rcu_preempt22:25:012
1135951995610,6cyclictest131rcu_preempt21:25:292
1135951995610,6cyclictest131rcu_preempt20:30:022
113594799569,8cyclictest0-21swapper/121:45:031
113594799567,8cyclictest23-21ksoftirqd/123:30:021
1135947995611,11cyclictest23-21ksoftirqd/122:28:331
1135947995611,11cyclictest23-21ksoftirqd/122:28:331
113594099565,4cyclictest131rcu_preempt20:45:020
1135940995627,22cyclictest0-21swapper/000:21:410
1135940995620,31cyclictest16850irq/134-800000022:33:500
1135940995619,32cyclictest16850irq/134-800000021:41:550
113595699553,12cyclictest37-21ksoftirqd/300:05:003
1135951995537,13cyclictest275-21dbus-daemon19:40:012
113595199552,10cyclictest30-21ksoftirqd/222:48:522
1135951995513,6cyclictest131rcu_preempt21:05:032
1135951995513,6cyclictest131rcu_preempt20:50:012
1135951995513,6cyclictest131rcu_preempt20:50:012
113594799557,9cyclictest23-21ksoftirqd/119:25:011
113594799557,11cyclictest23-21ksoftirqd/123:20:001
113594799556,15cyclictest23-21ksoftirqd/123:05:021
113594799555,13cyclictest23-21ksoftirqd/121:36:021
113594799552,13cyclictest23-21ksoftirqd/122:50:231
113594799551,14cyclictest23-21ksoftirqd/122:17:171
1135947995511,11cyclictest23-21ksoftirqd/123:50:211
1135940995529,10cyclictest1202743-21ssh23:25:050
1135940995520,31cyclictest16850irq/134-800000023:43:370
1135940995520,30cyclictest12-21ksoftirqd/021:17:170
1135940995519,32cyclictest16850irq/134-800000022:18:500
1135940995519,32cyclictest16850irq/134-800000000:03:280
1135940995519,21cyclictest16850irq/134-800000000:19:020
1135940995519,21cyclictest16850irq/134-800000000:19:020
1135940995518,32cyclictest16850irq/134-800000023:46:300
1135956995431,11cyclictest37-21ksoftirqd/322:50:013
1135956995417,7cyclictest37-21ksoftirqd/321:30:003
1135951995446,5cyclictest1137974-21kworker/2:322:14:182
113595199542,9cyclictest30-21ksoftirqd/223:45:002
1135951995416,5cyclictest131rcu_preempt20:00:002
1135951995413,6cyclictest131rcu_preempt19:43:142
1135951995412,5cyclictest131rcu_preempt22:30:102
1135951995411,6cyclictest131rcu_preempt22:06:482
113594799548,12cyclictest23-21ksoftirqd/122:35:021
113594799547,10cyclictest23-21ksoftirqd/100:31:181
113594799546,8cyclictest23-21ksoftirqd/122:39:311
113594799546,15cyclictest23-21ksoftirqd/122:05:021
113594799545,16cyclictest23-21ksoftirqd/122:55:211
113594799545,14cyclictest23-21ksoftirqd/120:10:011
113594799543,16cyclictest23-21ksoftirqd/122:50:011
113594799543,10cyclictest23-21ksoftirqd/122:00:011
113594099548,24cyclictest131rcu_preempt21:52:240
1135940995443,6cyclictest16950irq/224-800000021:10:020
1135940995425,14cyclictest1151790-21ssh21:31:160
1135940995425,14cyclictest1151790-21ssh21:31:160
1135940995419,30cyclictest16850irq/134-800000023:10:580
1135940995419,30cyclictest16850irq/134-800000023:10:580
1135940995419,30cyclictest16850irq/134-800000000:35:490
1135940995418,32cyclictest16850irq/134-800000023:36:550
1135940995418,32cyclictest16850irq/134-800000023:36:550
1135940995418,32cyclictest16850irq/134-800000023:30:140
1135940995418,32cyclictest16850irq/134-800000023:03:200
113594099541,4cyclictest131rcu_preempt00:25:050
1135940995413,37cyclictest16850irq/134-800000021:40:000
1135940995411,7cyclictest131rcu_preempt20:40:010
113595699539,5cyclictest131rcu_preempt19:20:013
1135956995338,12cyclictest1230694-21sshd00:27:563
1135956995335,14cyclictest1-21systemd21:00:003
1135956995325,18cyclictest37-21ksoftirqd/322:42:063
113595199539,6cyclictest131rcu_preempt00:10:242
113595199533,8cyclictest30-21ksoftirqd/200:01:192
1135951995314,6cyclictest131rcu_preempt22:03:422
1135951995312,5cyclictest131rcu_preempt23:00:352
113594799536,8cyclictest23-21ksoftirqd/120:30:011
113594799536,7cyclictest23-21ksoftirqd/120:05:021
113594799535,8cyclictest23-21ksoftirqd/100:10:431
113594799535,13cyclictest23-21ksoftirqd/100:07:591
113594799534,8cyclictest23-21ksoftirqd/119:40:011
113594799534,13cyclictest23-21ksoftirqd/120:35:021
113594799534,13cyclictest23-21ksoftirqd/120:35:021
1135947995310,10cyclictest23-21ksoftirqd/119:55:021
1135940995318,31cyclictest16850irq/134-800000022:48:160
1135940995318,30cyclictest16850irq/134-800000000:10:150
1135940995317,32cyclictest16850irq/134-800000021:45:160
1135940995317,31cyclictest16850irq/134-800000021:20:110
1135940995317,31cyclictest16850irq/134-800000021:20:100
1135940995316,32cyclictest16850irq/134-800000022:56:410
113595699527,5cyclictest131rcu_preempt21:51:063
1135956995229,9cyclictest37-21ksoftirqd/300:25:013
1135956995217,24cyclictest37-21ksoftirqd/300:14:073
1135956995211,6cyclictest131rcu_preempt23:40:483
113595199524,11cyclictest30-21ksoftirqd/221:21:012
113595199524,11cyclictest30-21ksoftirqd/221:21:012
113595199523,10cyclictest30-21ksoftirqd/221:42:572
1135951995211,5cyclictest131rcu_preempt20:45:002
113594799527,13cyclictest23-21ksoftirqd/120:15:021
113594799527,10cyclictest23-21ksoftirqd/119:20:021
113594799525,12cyclictest23-21ksoftirqd/119:10:011
113594799522,15cyclictest1140575-21sendmail-msp20:40:001
1135947995212,10cyclictest23-21ksoftirqd/120:25:021
1135940995219,29cyclictest16850irq/134-800000023:05:220
1135940995218,30cyclictest16850irq/134-800000022:35:470
1135940995217,30cyclictest16850irq/134-800000023:24:060
1135940995217,30cyclictest16850irq/134-800000021:59:550
113595699517,5cyclictest131rcu_preempt20:40:003
113595699515,5cyclictest131rcu_preempt23:35:183
113595699515,5cyclictest131rcu_preempt23:35:183
1135951995110,8cyclictest131rcu_preempt21:50:492
113594799519,8cyclictest23-21ksoftirqd/120:20:021
113594799516,7cyclictest23-21ksoftirqd/119:45:011
113594799515,8cyclictest23-21ksoftirqd/121:35:021
113594799515,8cyclictest23-21ksoftirqd/121:35:021
113594799515,13cyclictest23-21ksoftirqd/120:50:021
113594799515,13cyclictest23-21ksoftirqd/120:50:021
1135947995113,9cyclictest131rcu_preempt21:05:021
1135947995113,9cyclictest131rcu_preempt21:05:021
113595699509,5cyclictest131rcu_preempt23:17:333
113595699508,5cyclictest131rcu_preempt22:11:283
113595699507,5cyclictest131rcu_preempt23:21:513
113595699507,5cyclictest131rcu_preempt22:16:543
113595699505,8cyclictest37-21ksoftirqd/321:56:093
113595699505,8cyclictest37-21ksoftirqd/321:32:033
113595699505,8cyclictest37-21ksoftirqd/321:32:033
113595699505,5cyclictest131rcu_preempt23:57:233
113595699505,5cyclictest131rcu_preempt23:57:233
113595699504,5cyclictest131rcu_preempt23:53:183
113595699504,5cyclictest131rcu_preempt23:47:283
1135956995026,10cyclictest37-21ksoftirqd/321:50:013
1135956995018,4cyclictest1142142-21munin-run21:10:013
1135956995015,8cyclictest131rcu_preempt00:30:293
1135956995013,5cyclictest131rcu_preempt20:05:013
1135951995041,5cyclictest1135070-21kworker/2:219:54:232
113595199503,6cyclictest131rcu_preempt20:15:022
1135951995014,5cyclictest131rcu_preempt21:33:162
1135951995014,5cyclictest131rcu_preempt21:33:162
1135951995010,6cyclictest131rcu_preempt19:15:022
113594099509,6cyclictest131rcu_preempt19:14:590
1135940995017,30cyclictest16850irq/134-800000000:33:280
113595699499,5cyclictest131rcu_preempt22:54:523
113595699499,19cyclictest131rcu_preempt21:15:013
113595699497,5cyclictest131rcu_preempt22:03:103
113595699496,5cyclictest131rcu_preempt23:25:173
113595699496,5cyclictest131rcu_preempt22:07:223
1135956994928,11cyclictest37-21ksoftirqd/323:05:003
1135956994910,5cyclictest131rcu_preempt19:10:023
1135951994942,4cyclictest1135070-21kworker/2:220:34:182
1135951994942,4cyclictest1135070-21kworker/2:220:34:172
113595199491,9cyclictest30-21ksoftirqd/220:20:012
113594799492,9cyclictest23-21ksoftirqd/119:35:021
113595699488,4cyclictest131rcu_preempt22:32:303
113595699484,30cyclictest37-21ksoftirqd/320:45:023
113595699482,6cyclictest131rcu_preempt22:55:093
1135956994823,11cyclictest37-21ksoftirqd/323:35:013
1135956994818,18cyclictest131rcu_preempt21:17:393
1135956994811,5cyclictest131rcu_preempt22:25:023
1135956994811,5cyclictest131rcu_preempt22:25:023
1135951994840,5cyclictest1137974-21kworker/2:320:23:242
1135951994840,5cyclictest1135070-21kworker/2:220:52:582
1135951994840,5cyclictest1135070-21kworker/2:220:01:412
1135951994840,5cyclictest1124739-21kworker/2:119:33:352
113595199483,5cyclictest131rcu_preempt19:10:012
1135951994822,16cyclictest30-21ksoftirqd/219:20:012
113594099482,40cyclictest1138245-21latency_hist19:55:020
113595699478,5cyclictest131rcu_preempt00:35:173
113595699475,5cyclictest131rcu_preempt23:05:333
1135956994732,10cyclictest275-21dbus-daemon21:25:013
1135956994732,10cyclictest275-21dbus-daemon21:25:013
1135956994721,16cyclictest37-21ksoftirqd/321:36:153
1135956994710,18cyclictest37-21ksoftirqd/321:43:463
113595199477,11cyclictest30-21ksoftirqd/219:45:112
1135951994740,5cyclictest1135070-21kworker/2:220:55:132
1135951994739,5cyclictest1135070-21kworker/2:220:35:272
1135951994738,5cyclictest1135070-21kworker/2:219:26:082
113594799477,10cyclictest23-21ksoftirqd/121:10:011
113594799471,13cyclictest23-21ksoftirqd/120:55:001
1135940994738,5cyclictest16950irq/224-800000019:20:010
1135940994710,5cyclictest131rcu_preempt20:05:010
113595699464,6cyclictest131rcu_preempt00:20:003
113595699464,6cyclictest131rcu_preempt00:20:003
113595699462,11cyclictest37-21ksoftirqd/320:22:173
113594799468,11cyclictest23-21ksoftirqd/119:15:021
1135947994635,7cyclictest23-21ksoftirqd/119:25:081
113594099462,5cyclictest131rcu_preempt21:05:020
113594099462,5cyclictest131rcu_preempt21:05:020
1135940994620,18cyclictest12-21ksoftirqd/020:20:020
113594099461,5cyclictest131rcu_preempt19:35:000
113595699456,12cyclictest37-21ksoftirqd/320:12:153
113595699454,5cyclictest131rcu_preempt22:35:343
113594799453,8cyclictest23-21ksoftirqd/119:50:021
113594099453,36cyclictest1140059-21unin-run20:30:010
1135940994524,15cyclictest3467602-21systemd-journal19:50:000
113595699449,5cyclictest131rcu_preempt23:13:293
113595699449,5cyclictest131rcu_preempt23:13:293
113595699443,6cyclictest131rcu_preempt19:25:023
1135956994432,9cyclictest1-21systemd19:45:003
1135956994413,20cyclictest131rcu_preempt19:15:023
1135940994436,5cyclictest12-21ksoftirqd/020:08:110
1135940994423,14cyclictest275-21dbus-daemon21:00:010
113594099441,6cyclictest12-21ksoftirqd/019:25:020
1135940994411,8cyclictest12-21ksoftirqd/020:25:000
1135940994411,6cyclictest131rcu_preempt20:35:030
113595699435,6cyclictest131rcu_preempt20:20:003
113595699431,6cyclictest1138442-21sshd19:57:123
1135940994319,16cyclictest12-21ksoftirqd/020:54:110
113595699428,5cyclictest131rcu_preempt21:05:003
113595699428,5cyclictest131rcu_preempt21:05:003
1135956994213,4cyclictest131rcu_preempt20:52:173
113594099429,6cyclictest131rcu_preempt20:35:030
1135940994217,6cyclictest131rcu_preempt19:30:030
1135940994217,21cyclictest12-21ksoftirqd/020:15:010
113595699416,8cyclictest37-21ksoftirqd/319:37:133
1135956994134,4cyclictest1141055-21sshd20:47:213
1135956994134,4cyclictest1141055-21sshd20:47:213
1135956994118,16cyclictest37-21ksoftirqd/319:35:023
1135956994116,17cyclictest37-21ksoftirqd/319:55:013
1135956994115,16cyclictest37-21ksoftirqd/320:10:033
113595699405,30cyclictest1140310-21unin-run20:35:013
113595699405,30cyclictest1140310-21unin-run20:35:003
1135956994018,9cyclictest37-21ksoftirqd/319:30:013
1135956994018,10cyclictest37-21ksoftirqd/319:50:023
1135940994027,9cyclictest12-21ksoftirqd/019:45:020
1135956993910,9cyclictest131rcu_preempt20:30:013
1135940993927,9cyclictest12-21ksoftirqd/019:40:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional