You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-27 - 09:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Wed May 27, 2026 00:45:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4857499910798,6cyclictest37-21ksoftirqd/320:40:013
48574999891,61cyclictest3467602-21systemd-journal23:00:033
48574999891,61cyclictest3467602-21systemd-journal23:00:023
485736998881,5cyclictest568340-40mandb00:00:040
485741998353,15cyclictest0-21swapper/100:35:011
485749997656,9cyclictest37-21ksoftirqd/323:20:023
485741997328,8cyclictest309-21in:imuxsock00:05:021
48574199714,41cyclictest484890-21kworker/u8:319:35:021
485741997130,12cyclictest3467602-21systemd-journal23:40:001
48573699695,7cyclictest121ksoftirqd/000:12:450
48573699695,7cyclictest121ksoftirqd/000:12:440
48574199681,20cyclictest131rcu_preempt23:20:011
485736996724,38cyclictest1-21systemd22:40:000
485736996716,41cyclictest16850irq/134-800000021:22:490
485741996627,8cyclictest0-21swapper/123:15:021
485741996526,12cyclictest23-21ksoftirqd/100:40:041
485736996515,8cyclictest12-21ksoftirqd/021:20:010
48573699626,6cyclictest12-21ksoftirqd/023:35:510
485736996214,10cyclictest12-21ksoftirqd/020:40:000
485736996119,20cyclictest16850irq/134-800000022:20:060
485741996022,13cyclictest514528-21latency_hist22:00:011
485741996016,17cyclictest0-21swapper/122:45:001
48573699608,5cyclictest131rcu_preempt22:54:200
485745995916,9cyclictest30-21ksoftirqd/222:15:012
485736995954,3cyclictest0-21swapper/023:55:020
485736995920,30cyclictest16850irq/134-800000021:49:040
48574999584,37cyclictest491965-21diskmemload22:50:023
48574599586,12cyclictest30-21ksoftirqd/221:00:002
485736995819,31cyclictest16850irq/134-800000023:28:460
485736995819,31cyclictest16850irq/134-800000021:51:390
485736995819,30cyclictest16850irq/134-800000023:41:010
485736995818,31cyclictest16850irq/134-800000022:28:100
485736995818,31cyclictest16850irq/134-800000000:31:520
485736995817,32cyclictest16850irq/134-800000000:23:020
485736995817,32cyclictest16850irq/134-800000000:23:020
485736995817,31cyclictest16850irq/134-800000022:05:250
48574999578,7cyclictest37-21ksoftirqd/321:45:013
48574999578,7cyclictest37-21ksoftirqd/321:45:003
48574999572,12cyclictest37-21ksoftirqd/300:00:003
485745995715,6cyclictest570717-21cat00:05:012
48573699573,5cyclictest121ksoftirqd/023:16:550
485736995719,6cyclictest16850irq/134-800000023:07:020
485736995719,5cyclictest16850irq/134-800000023:20:170
485736995717,31cyclictest16850irq/134-800000023:33:170
485736995717,31cyclictest16850irq/134-800000022:15:040
485749995635,15cyclictest275-21dbus-daemon19:30:003
485741995637,14cyclictest3467602-21systemd-journal22:10:001
485741995636,14cyclictest3467602-21systemd-journal22:15:001
48573699569,13cyclictest541483-21ssh23:00:000
48573699569,13cyclictest541483-21ssh23:00:000
485736995628,13cyclictest546382-21ssh23:10:240
485736995626,9cyclictest12-21ksoftirqd/021:58:430
485736995618,6cyclictest16850irq/134-800000000:26:350
485736995618,6cyclictest16850irq/134-800000000:26:340
485736995615,11cyclictest12-21ksoftirqd/022:45:220
485749995511,8cyclictest555408-1kworker/u9:100:05:023
48574599555,7cyclictest30-21ksoftirqd/221:17:372
48574599555,6cyclictest30-21ksoftirqd/222:55:022
48574199552,6cyclictest131rcu_preempt21:35:031
48573699557,30cyclictest16850irq/134-800000000:05:000
485736995527,25cyclictest529491-21ssh22:33:210
485736995527,25cyclictest529491-21ssh22:33:210
485736995518,8cyclictest12-21ksoftirqd/019:20:000
485736995517,6cyclictest16850irq/134-800000022:03:590
485736995517,6cyclictest16850irq/134-800000022:03:580
485736995517,5cyclictest16850irq/134-800000021:31:230
485736995515,6cyclictest16850irq/134-800000022:40:200
485745995441,9cyclictest488276-21systemctl20:00:012
485741995430,16cyclictest275-21dbus-daemon21:50:001
48573699546,8cyclictest12-21ksoftirqd/019:55:010
485736995444,6cyclictest491968-21munin-plugin-st21:10:000
485736995430,9cyclictest563959-21ssh23:50:010
485736995429,10cyclictest575912-21ssh00:16:250
485736995424,26cyclictest12-21ksoftirqd/023:01:190
485736995421,30cyclictest12-21ksoftirqd/021:41:440
485736995421,30cyclictest12-21ksoftirqd/021:41:440
48573699541,10cyclictest12-21ksoftirqd/019:15:010
48574999535,7cyclictest37-21ksoftirqd/319:13:193
48574599539,8cyclictest30-21ksoftirqd/222:09:302
48574599537,8cyclictest30-21ksoftirqd/223:15:022
48573699538,7cyclictest12-21ksoftirqd/020:45:010
48573699537,8cyclictest12-21ksoftirqd/019:40:000
48573699535,8cyclictest12-21ksoftirqd/021:35:480
485736995317,6cyclictest16850irq/134-800000021:28:170
485736995316,32cyclictest16850irq/134-800000021:11:350
48574599524,9cyclictest30-21ksoftirqd/222:00:012
48574599524,9cyclictest30-21ksoftirqd/200:19:272
485745995241,8cyclictest291rcuc/222:40:292
485745995219,9cyclictest30-21ksoftirqd/221:40:012
485745995218,10cyclictest30-21ksoftirqd/223:00:022
485745995218,10cyclictest30-21ksoftirqd/223:00:012
485745995211,10cyclictest30-21ksoftirqd/221:54:432
48573699528,7cyclictest12-21ksoftirqd/000:37:560
48573699528,7cyclictest12-21ksoftirqd/000:37:550
48574599518,8cyclictest30-21ksoftirqd/220:25:022
48574599516,8cyclictest30-21ksoftirqd/200:32:352
48574599514,7cyclictest30-21ksoftirqd/222:34:302
48574599514,7cyclictest30-21ksoftirqd/222:34:302
48574599513,12cyclictest30-21ksoftirqd/221:12:272
485741995133,14cyclictest275-21dbus-daemon22:50:001
48573699512,6cyclictest12-21ksoftirqd/000:06:140
48573699511,3cyclictest131rcu_preempt21:00:000
485749995033,12cyclictest275-21dbus-daemon21:40:003
485749995031,9cyclictest37-21ksoftirqd/323:15:013
48574599507,10cyclictest30-21ksoftirqd/220:20:022
48574599507,10cyclictest30-21ksoftirqd/220:20:022
48574599506,11cyclictest30-21ksoftirqd/223:37:422
48574599505,8cyclictest30-21ksoftirqd/221:26:412
48574599505,4cyclictest131rcu_preempt20:35:012
48574599505,4cyclictest131rcu_preempt20:35:012
485745995041,6cyclictest291rcuc/222:39:542
485745995017,5cyclictest131rcu_preempt23:21:462
485741995033,12cyclictest275-21dbus-daemon21:30:001
485736995017,29cyclictest16850irq/134-800000022:11:010
485749994930,14cyclictest3467602-21systemd-journal00:17:203
48574599496,7cyclictest30-21ksoftirqd/221:34:332
48574599495,9cyclictest30-21ksoftirqd/200:07:142
48574599495,5cyclictest30-21ksoftirqd/223:57:512
48574599494,5cyclictest30-21ksoftirqd/221:49:012
485745994941,5cyclictest507785-21kworker/2:322:15:152
48574599492,16cyclictest30-21ksoftirqd/223:01:072
48574599491,12cyclictest30-21ksoftirqd/223:45:522
485741994931,16cyclictest23-21ksoftirqd/121:15:031
485736994913,6cyclictest12-21ksoftirqd/020:25:000
48574999489,5cyclictest131rcu_preempt21:55:123
48574999481,7cyclictest546011-21unin-run23:10:013
48574599489,4cyclictest131rcu_preempt00:22:122
48574599489,4cyclictest131rcu_preempt00:22:112
48574599487,6cyclictest545359-1kworker/u9:223:42:052
48574599486,7cyclictest30-21ksoftirqd/222:21:202
48574599485,8cyclictest30-21ksoftirqd/219:25:012
48574599485,8cyclictest30-21ksoftirqd/219:25:012
48574599485,8cyclictest30-21ksoftirqd/200:11:002
48574599485,8cyclictest30-21ksoftirqd/200:11:002
48574599485,6cyclictest30-21ksoftirqd/200:27:172
48574599485,6cyclictest30-21ksoftirqd/200:27:172
485745994840,5cyclictest291rcuc/223:31:442
48574599483,7cyclictest30-21ksoftirqd/222:47:462
485745994822,9cyclictest30-21ksoftirqd/220:06:422
48574599481,11cyclictest30-21ksoftirqd/223:53:032
485741994832,8cyclictest23-21ksoftirqd/120:43:161
485741994829,15cyclictest275-21dbus-daemon23:10:011
485749994729,13cyclictest3467602-21systemd-journal22:35:203
48574599475,5cyclictest30-21ksoftirqd/223:27:342
485745994740,5cyclictest489370-21kworker/2:222:02:052
485745994740,5cyclictest489370-21kworker/2:222:02:052
485745994738,6cyclictest291rcuc/222:27:022
485745994737,7cyclictest30-21ksoftirqd/223:18:032
48574599471,8cyclictest545841-21ssh23:09:402
485741994726,16cyclictest3467602-21systemd-journal22:25:001
485741994725,15cyclictest275-21dbus-daemon20:50:011
485741994725,15cyclictest275-21dbus-daemon20:50:011
48573699476,4cyclictest131rcu_preempt19:31:190
48573699474,8cyclictest12-21ksoftirqd/019:45:510
48573699474,8cyclictest12-21ksoftirqd/019:45:510
48573699474,7cyclictest12-21ksoftirqd/020:00:020
48573699472,7cyclictest121ksoftirqd/020:06:420
48573699471,10cyclictest12-21ksoftirqd/020:20:020
48573699471,10cyclictest12-21ksoftirqd/020:20:020
48574999463,5cyclictest131rcu_preempt00:40:023
48574999463,5cyclictest131rcu_preempt00:40:013
48574999462,9cyclictest37-21ksoftirqd/319:50:013
48574999462,9cyclictest37-21ksoftirqd/319:50:013
485749994629,12cyclictest37-21ksoftirqd/322:40:063
485749994622,20cyclictest37-21ksoftirqd/323:38:353
485749994621,6cyclictest361rcuc/300:15:013
485749994621,6cyclictest361rcuc/300:15:013
485749994620,17cyclictest37-21ksoftirqd/322:10:003
485749994615,7cyclictest131rcu_preempt22:12:053
485749994614,6cyclictest131rcu_preempt19:25:003
485749994614,6cyclictest131rcu_preempt19:25:003
485749994611,7cyclictest131rcu_preempt22:30:013
485749994610,6cyclictest131rcu_preempt21:23:243
48574599465,4cyclictest131rcu_preempt20:40:012
48574599464,7cyclictest30-21ksoftirqd/221:42:512
48574599464,7cyclictest30-21ksoftirqd/221:42:512
48574599462,8cyclictest30-21ksoftirqd/220:43:142
48574599461,10cyclictest30-21ksoftirqd/221:00:202
48574599461,10cyclictest30-21ksoftirqd/221:00:202
485741994634,8cyclictest3467602-21systemd-journal20:30:191
485741994634,8cyclictest3467602-21systemd-journal20:30:191
485741994630,12cyclictest275-21dbus-daemon22:55:021
485741994628,11cyclictest275-21dbus-daemon20:15:001
48573699464,7cyclictest1-21systemd20:50:010
48573699464,7cyclictest1-21systemd20:50:010
48574999458,5cyclictest131rcu_preempt19:40:023
48574999454,37cyclictest37-21ksoftirqd/320:05:023
48574999454,37cyclictest37-21ksoftirqd/320:05:023
485749994532,9cyclictest275-21dbus-daemon23:55:013
48574999451,8cyclictest503216-21/usr/sbin/munin21:35:003
485749994510,12cyclictest37-21ksoftirqd/319:30:193
48574599456,7cyclictest30-21ksoftirqd/219:30:012
48574599455,6cyclictest30-21ksoftirqd/220:55:012
485745994530,12cyclictest30-21ksoftirqd/219:15:012
485745994529,13cyclictest30-21ksoftirqd/221:20:262
48574599451,12cyclictest30-21ksoftirqd/200:38:022
48574599451,12cyclictest30-21ksoftirqd/200:38:022
485741994535,6cyclictest487779-21latency_hist19:50:011
485741994535,6cyclictest487779-21latency_hist19:50:011
485741994532,7cyclictest309-21in:imuxsock22:40:001
48573699454,9cyclictest12-21ksoftirqd/020:15:010
48573699452,7cyclictest491710-21munin-run21:05:000
48573699452,7cyclictest491710-21munin-run21:05:000
48574999447,10cyclictest37-21ksoftirqd/322:20:203
48574999444,6cyclictest131rcu_preempt00:10:003
485749994417,18cyclictest37-21ksoftirqd/300:25:023
485749994417,18cyclictest37-21ksoftirqd/300:25:013
485749994412,11cyclictest37-21ksoftirqd/320:00:003
48574599441,11cyclictest487384-21latency_hist19:40:022
485741994424,14cyclictest275-21dbus-daemon20:25:011
48574199441,38cyclictest275-21dbus-daemon23:50:001
48573699446,6cyclictest12-21ksoftirqd/020:05:020
48573699446,6cyclictest12-21ksoftirqd/020:05:020
48573699445,6cyclictest131rcu_preempt19:45:010
48574999439,5cyclictest131rcu_preempt22:53:053
485749994334,7cyclictest37-21ksoftirqd/320:27:023
485749994334,6cyclictest37-21ksoftirqd/323:41:203
485749994328,11cyclictest37-21ksoftirqd/320:25:003
48574999432,6cyclictest131rcu_preempt23:50:013
485749994324,15cyclictest275-21dbus-daemon23:35:023
485749994324,13cyclictest3467602-21systemd-journal20:15:003
48574999432,27cyclictest484890-21kworker/u8:320:55:023
485749994319,9cyclictest131rcu_preempt21:10:323
48574599437,8cyclictest30-21ksoftirqd/220:05:022
48574599437,8cyclictest30-21ksoftirqd/220:05:022
48574599437,8cyclictest30-21ksoftirqd/219:55:022
485745994329,10cyclictest30-21ksoftirqd/219:35:012
48574199438,7cyclictest131rcu_preempt21:40:001
485741994312,9cyclictest131rcu_preempt21:25:001
48573699431,7cyclictest12-21ksoftirqd/020:51:040
485749994232,7cyclictest37-21ksoftirqd/300:31:003
485749994229,8cyclictest37-21ksoftirqd/320:10:003
485749994223,4cyclictest37-21ksoftirqd/323:21:533
48574999421,5cyclictest131rcu_preempt21:49:483
48574999421,10cyclictest37-21ksoftirqd/323:27:003
485749994210,6cyclictest131rcu_preempt22:30:053
485749994210,6cyclictest131rcu_preempt22:30:053
48574599429,4cyclictest131rcu_preempt20:50:012
48574599429,4cyclictest131rcu_preempt20:50:002
48574599425,8cyclictest30-21ksoftirqd/220:30:002
48574599423,9cyclictest30-21ksoftirqd/220:15:012
48574199423,7cyclictest23-21ksoftirqd/121:53:291
485741994230,8cyclictest275-21dbus-daemon00:00:011
485741994229,10cyclictest584881-21sshd00:36:201
485741994225,14cyclictest275-21dbus-daemon22:20:011
48573699423,7cyclictest12-21ksoftirqd/019:29:190
48574999418,6cyclictest131rcu_preempt22:05:023
48574999418,6cyclictest131rcu_preempt22:05:023
48574999415,6cyclictest37-21ksoftirqd/321:51:233
485749994131,7cyclictest37-21ksoftirqd/322:18:013
485749994131,7cyclictest37-21ksoftirqd/320:45:013
485749994128,7cyclictest309-21in:imuxsock20:35:023
485749994128,7cyclictest309-21in:imuxsock20:35:013
485749994116,21cyclictest37-21ksoftirqd/300:27:193
485749994116,21cyclictest37-21ksoftirqd/300:27:183
48574199413,4cyclictest131rcu_preempt00:06:441
485741994129,10cyclictest23-21ksoftirqd/122:02:171
485741994129,10cyclictest23-21ksoftirqd/122:02:161
485741994124,13cyclictest275-21dbus-daemon20:00:011
48573699415,8cyclictest12-21ksoftirqd/019:25:010
48573699415,8cyclictest12-21ksoftirqd/019:25:010
48573699413,33cyclictest48-21kcompactd019:10:020
48574999408,6cyclictest131rcu_preempt19:20:023
48574999405,5cyclictest131rcu_preempt21:00:013
485749994027,10cyclictest37-21ksoftirqd/321:20:003
48574999402,5cyclictest131rcu_preempt21:25:543
485749994025,11cyclictest275-21dbus-daemon19:55:003
485749994013,6cyclictest131rcu_preempt23:02:353
48574199403,4cyclictest131rcu_preempt00:20:001
48574199403,3cyclictest150-21pr/ttyS200:25:241
48574199403,3cyclictest150-21pr/ttyS200:25:231
485741994030,7cyclictest488808-21systemctl20:10:001
485741994028,9cyclictest23-21ksoftirqd/122:29:531
485741994025,11cyclictest275-21dbus-daemon00:25:011
485741994025,11cyclictest275-21dbus-daemon00:25:011
48574199402,12cyclictest528723-21ssh22:31:191
48574199402,12cyclictest528723-21ssh22:31:191
48574199401,3cyclictest131rcu_preempt23:28:041
485749993926,10cyclictest3467602-21systemd-journal20:20:003
485749993926,10cyclictest3467602-21systemd-journal20:20:003
48574599396,8cyclictest30-21ksoftirqd/221:10:002
48574599391,9cyclictest30-21ksoftirqd/219:20:012
485745993915,15cyclictest291rcuc/219:40:502
48574199398,15cyclictest0-21swapper/120:55:011
48574199397,3cyclictest131rcu_preempt20:20:021
48574199397,3cyclictest131rcu_preempt20:20:021
485741993927,10cyclictest23-21ksoftirqd/121:41:411
485741993927,10cyclictest23-21ksoftirqd/121:41:411
485741993926,9cyclictest3467602-21systemd-journal21:09:201
485741993925,11cyclictest486944-21sshd19:31:191
485741993922,13cyclictest1-21systemd23:05:011
485741993910,6cyclictest131rcu_preempt19:55:011
48573699392,8cyclictest12-21ksoftirqd/020:35:000
48573699392,8cyclictest12-21ksoftirqd/020:35:000
485749993827,9cyclictest37-21ksoftirqd/320:49:203
485749993827,9cyclictest37-21ksoftirqd/320:49:203
485749993825,10cyclictest491751-21date21:05:033
485749993825,10cyclictest491751-21date21:05:023
48574199386,6cyclictest23-21ksoftirqd/123:40:471
485741993827,8cyclictest23-21ksoftirqd/121:11:561
48574999378,6cyclictest131rcu_preempt21:10:023
485749993726,8cyclictest361rcuc/319:44:193
485745993730,4cyclictest485746-21latency_hist19:10:022
48574199377,4cyclictest131rcu_preempt23:55:021
485741993728,5cyclictest491432-21sh21:00:001
485741993724,11cyclictest23-21ksoftirqd/123:00:001
485741993724,11cyclictest23-21ksoftirqd/123:00:001
48574599362,7cyclictest30-21ksoftirqd/219:50:012
48574599362,7cyclictest30-21ksoftirqd/219:50:012
485741993627,6cyclictest486175-21sshd19:19:191
485741993624,9cyclictest3467602-21systemd-journal19:21:351
485741993624,9cyclictest3467602-21systemd-journal19:21:351
485741993624,9cyclictest23-21ksoftirqd/120:05:001
485741993624,9cyclictest23-21ksoftirqd/120:05:001
48574199362,10cyclictest555898-21ssh23:31:461
48574199355,18cyclictest0-21swapper/121:00:021
48574199355,18cyclictest0-21swapper/121:00:021
48574199354,3cyclictest131rcu_preempt19:30:011
485741993522,11cyclictest23-21ksoftirqd/123:21:281
48573699355,5cyclictest12-21ksoftirqd/020:30:010
48574199343,3cyclictest131rcu_preempt19:15:021
485741993424,6cyclictest489996-21latency_hist20:30:021
485741993410,5cyclictest131rcu_preempt00:11:221
485741993410,5cyclictest131rcu_preempt00:11:211
485741993325,5cyclictest490352-21sendmail20:40:001
485741993225,4cyclictest487644-21latency_hist19:45:031
485741993221,8cyclictest23-21ksoftirqd/119:10:021
48574999305,5cyclictest131rcu_preempt19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional