You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 13:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Fri Mar 27, 2026 00:45:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2225040997717,29cyclictest16950irq/224-800000023:35:020
222504499733,3cyclictest131rcu_preempt21:25:041
2225044996931,4cyclictest2301851-21latency_hist00:00:041
2225044996931,4cyclictest2301851-21latency_hist00:00:031
2225044996732,6cyclictest2316295-21/usr/sbin/munin00:35:011
2225044996627,11cyclictest3467602-21systemd-journal22:00:001
2225044996526,10cyclictest23-21ksoftirqd/123:16:531
2225044996526,10cyclictest23-21ksoftirqd/123:16:521
2225055996417,14cyclictest37-21ksoftirqd/321:20:003
2225044996424,10cyclictest23-21ksoftirqd/122:06:571
2225044996424,10cyclictest23-21ksoftirqd/122:06:571
222504099641,23cyclictest131rcu_preempt22:45:150
222504099634,38cyclictest131rcu_preempt00:27:080
2225055996241,16cyclictest2316879-21sshd00:36:033
2225055996241,16cyclictest2316879-21sshd00:36:023
2225040996227,30cyclictest12-21ksoftirqd/023:30:020
2225044996128,4cyclictest2296279-21sh23:46:411
222505599609,13cyclictest37-21ksoftirqd/322:25:023
2225055996031,8cyclictest37-21ksoftirqd/319:15:013
2225040996031,13cyclictest2279230-21ssh23:05:330
2225044995925,7cyclictest23-21ksoftirqd/120:10:031
2225040995949,2cyclictest0-21swapper/000:35:020
2225044995829,4cyclictest2287415-21sh23:25:161
2225044995827,4cyclictest2269522-21sh22:42:111
2225040995821,32cyclictest16850irq/134-800000021:34:380
2225040995729,12cyclictest2252612-21ssh22:01:410
2225040995728,13cyclictest2308324-21ssh00:15:350
2225040995728,13cyclictest2308324-21ssh00:15:350
2225040995728,12cyclictest2233441-21ssh21:15:150
222504099571,45cyclictest131rcu_preempt21:52:150
222504099571,3cyclictest131rcu_preempt22:20:500
2225051995638,13cyclictest1-21systemd23:35:022
222504099563,24cyclictest131rcu_preempt21:36:260
2225040995629,10cyclictest12-21ksoftirqd/021:45:000
2225040995628,12cyclictest2297305-21ssh23:49:220
2225040995620,3cyclictest131rcu_preempt22:37:050
2225040995618,34cyclictest16850irq/134-800000022:28:440
2225040995618,34cyclictest16850irq/134-800000021:27:300
2225040995618,34cyclictest16850irq/134-800000021:22:350
2225040995618,34cyclictest16850irq/134-800000000:13:140
2225040995618,33cyclictest16850irq/134-800000022:42:510
2225040995616,19cyclictest16850irq/134-800000023:15:030
2225051995547,6cyclictest30-21ksoftirqd/200:10:012
2225040995524,28cyclictest12-21ksoftirqd/021:14:070
2225040995524,28cyclictest12-21ksoftirqd/021:14:070
2225040995523,6cyclictest131rcu_preempt23:22:150
2225040995518,33cyclictest16850irq/134-800000022:15:060
2225040995511,3cyclictest0-21swapper/023:50:200
222505599548,6cyclictest131rcu_preempt21:40:013
222505599542,6cyclictest131rcu_preempt23:15:023
2225055995418,13cyclictest37-21ksoftirqd/319:50:003
2225051995435,15cyclictest1-21systemd23:10:002
2225040995423,6cyclictest12-21ksoftirqd/000:07:030
2225040995419,7cyclictest12-21ksoftirqd/023:40:410
2225040995419,7cyclictest12-21ksoftirqd/023:40:400
2225040995418,6cyclictest16850irq/134-800000022:33:250
2225051995344,5cyclictest2217190-21kworker/2:322:54:192
2225055995238,8cyclictest37-21ksoftirqd/323:00:023
2225055995231,17cyclictest1-21systemd20:15:013
2225051995244,5cyclictest2249683-21kworker/2:123:35:562
2225051995244,5cyclictest2217190-21kworker/2:321:33:512
2225051995244,5cyclictest2217190-21kworker/2:300:35:032
2225040995231,4cyclictest12-21ksoftirqd/000:00:150
2225040995224,12cyclictest2246581-21ssh21:47:240
2225040995216,18cyclictest16850irq/134-800000023:18:100
2225040995216,18cyclictest16850irq/134-800000023:18:100
2225051995143,5cyclictest2217190-21kworker/2:322:48:042
2225051995143,5cyclictest2217190-21kworker/2:320:57:152
2225051995141,5cyclictest2249683-21kworker/2:100:20:012
2225051995141,5cyclictest2249683-21kworker/2:100:20:002
2225051995123,7cyclictest2249683-21kworker/2:100:26:392
222504099512,26cyclictest131rcu_preempt00:23:170
2225040995119,29cyclictest12-21ksoftirqd/022:09:110
2225040995119,29cyclictest12-21ksoftirqd/022:09:110
2225040995116,18cyclictest16850irq/134-800000021:59:170
2225040995116,17cyclictest16850irq/134-800000000:37:450
2225040995116,17cyclictest16850irq/134-800000000:37:440
2225040995115,6cyclictest16850irq/134-800000022:52:090
2225040995114,7cyclictest16850irq/134-800000022:56:160
222505599504,5cyclictest131rcu_preempt23:20:033
222505599504,5cyclictest131rcu_preempt23:20:023
2225055995014,5cyclictest131rcu_preempt00:00:023
2225055995014,5cyclictest131rcu_preempt00:00:023
2225055995013,5cyclictest131rcu_preempt00:15:013
2225051995044,3cyclictest2249683-21kworker/2:123:27:472
2225051995042,5cyclictest2249683-21kworker/2:100:14:142
2225051995042,5cyclictest2217190-21kworker/2:323:18:542
2225051995042,5cyclictest2217190-21kworker/2:323:18:532
2225051995042,5cyclictest2217190-21kworker/2:322:35:362
2225051995042,5cyclictest2217190-21kworker/2:322:02:552
2225051995042,5cyclictest2217190-21kworker/2:321:56:242
2225051995042,5cyclictest2217190-21kworker/2:321:29:352
2225051995041,6cyclictest2217190-21kworker/2:322:11:222
2225051995041,6cyclictest2217190-21kworker/2:322:06:252
2225051995041,6cyclictest2217190-21kworker/2:322:06:242
2225051995041,5cyclictest2217190-21kworker/2:321:35:542
2225040995016,3cyclictest131rcu_preempt23:59:510
2225040995016,3cyclictest131rcu_preempt23:59:500
2225040995014,17cyclictest16850irq/134-800000023:37:100
2225040995013,17cyclictest16850irq/134-800000022:13:470
222504099501,25cyclictest131rcu_preempt23:04:400
222505599498,34cyclictest37-21ksoftirqd/323:30:023
222505599493,6cyclictest37-21ksoftirqd/321:24:033
2225055994911,6cyclictest131rcu_preempt20:40:003
2225051994941,5cyclictest2249683-21kworker/2:123:25:012
2225051994941,5cyclictest2249683-21kworker/2:100:21:522
2225051994941,5cyclictest2217190-21kworker/2:322:56:582
2225051994941,5cyclictest2217190-21kworker/2:322:21:372
2225051994941,5cyclictest2217190-21kworker/2:321:45:042
2225051994941,5cyclictest2217190-21kworker/2:321:12:252
2225051994941,5cyclictest2217190-21kworker/2:321:12:242
2225051994939,5cyclictest2249683-21kworker/2:100:35:492
2225051994939,5cyclictest2249683-21kworker/2:100:35:492
2225051994932,13cyclictest3467602-21systemd-journal20:32:152
2225040994934,12cyclictest12-21ksoftirqd/020:25:000
222504099493,40cyclictest2227317-21cron19:55:000
222505599488,6cyclictest131rcu_preempt23:02:033
2225055994833,10cyclictest275-21dbus-daemon22:30:013
222505599482,42cyclictest2258051-21wc22:15:013
2225055994817,10cyclictest37-21ksoftirqd/323:49:593
2225051994842,3cyclictest2217190-21kworker/2:322:17:132
222505199481,12cyclictest2228624-21modprobe20:20:002
2225044994832,12cyclictest3467602-21systemd-journal21:30:021
2225040994831,9cyclictest12-21ksoftirqd/021:00:000
222505599475,7cyclictest131rcu_preempt20:55:003
2225055994735,9cyclictest37-21ksoftirqd/323:51:193
2225055994712,8cyclictest131rcu_preempt21:52:013
2225051994741,3cyclictest2217190-21kworker/2:322:28:562
2225051994739,5cyclictest2249683-21kworker/2:123:58:052
2225051994739,5cyclictest2249683-21kworker/2:123:58:042
2225051994739,5cyclictest2217190-21kworker/2:321:42:522
2225051994730,13cyclictest1-21systemd19:45:002
222505599469,6cyclictest131rcu_preempt23:25:013
222505599469,6cyclictest131rcu_preempt20:20:003
222505599464,6cyclictest131rcu_preempt21:57:413
222505599462,6cyclictest131rcu_preempt00:10:003
2225055994621,20cyclictest361rcuc/323:45:023
2225055994621,20cyclictest361rcuc/323:45:013
2225055994616,6cyclictest131rcu_preempt22:05:003
2225055994613,10cyclictest37-21ksoftirqd/300:00:203
2225051994638,5cyclictest2249683-21kworker/2:123:53:272
2225051994638,5cyclictest2217190-21kworker/2:322:40:562
2225051994622,8cyclictest2234861-21ssh21:19:092
2225044994629,13cyclictest3467602-21systemd-journal19:25:011
2225044994629,13cyclictest3467602-21systemd-journal19:25:011
2225044994612,7cyclictest131rcu_preempt21:00:001
2225055994529,12cyclictest3467602-21systemd-journal21:00:023
222505599452,6cyclictest131rcu_preempt22:45:153
2225055994519,17cyclictest275-21dbus-daemon21:50:013
2225055994511,6cyclictest131rcu_preempt21:33:023
2225055994510,21cyclictest2230689-21modprobe21:00:013
2225051994537,5cyclictest2249683-21kworker/2:123:02:552
2225051994537,5cyclictest2217190-21kworker/2:323:11:562
2225044994529,12cyclictest275-21dbus-daemon00:40:011
2225044994529,12cyclictest275-21dbus-daemon00:40:001
2225044994528,13cyclictest3467602-21systemd-journal19:45:001
2225044994524,16cyclictest275-21dbus-daemon20:25:001
222504099455,36cyclictest2225488-21cron19:20:000
2225040994538,4cyclictest16850irq/134-800000021:10:010
2225040994532,9cyclictest3467602-21systemd-journal20:35:030
222504099452,3cyclictest131rcu_preempt19:40:020
222504099452,3cyclictest131rcu_preempt19:40:020
222504099451,15cyclictest2230171-21sendmail-mta20:49:040
222504099451,15cyclictest2230171-21sendmail-mta20:49:040
222505599446,6cyclictest131rcu_preempt22:18:093
2225055994435,6cyclictest37-21ksoftirqd/323:06:293
2225055994432,7cyclictest37-21ksoftirqd/300:20:003
2225055994432,7cyclictest37-21ksoftirqd/300:20:003
222505599441,9cyclictest0-21swapper/322:06:023
222505599441,9cyclictest0-21swapper/322:06:013
2225051994438,3cyclictest2217190-21kworker/2:300:01:452
2225051994436,5cyclictest2249683-21kworker/2:123:47:492
2225051994436,5cyclictest2249683-21kworker/2:123:43:382
2225051994436,5cyclictest2249683-21kworker/2:123:43:382
2225044994431,9cyclictest3467602-21systemd-journal20:52:571
222504099441,39cyclictest2228577-21sshd20:16:150
2225055994331,7cyclictest37-21ksoftirqd/322:40:013
2225055994329,11cyclictest37-21ksoftirqd/320:35:013
2225055994328,12cyclictest37-21ksoftirqd/323:25:153
2225055994328,11cyclictest37-21ksoftirqd/322:40:433
222505199436,4cyclictest131rcu_preempt20:40:002
2225044994333,6cyclictest2262159-21munin-plugin-st22:25:001
2225044994329,11cyclictest1-21systemd22:40:021
2225044994328,10cyclictest2225048-21cat19:10:021
2225040994333,5cyclictest309-21in:imuxsock20:15:010
222505599425,10cyclictest37-21ksoftirqd/321:15:013
222505599425,10cyclictest37-21ksoftirqd/321:15:013
2225055994229,10cyclictest37-21ksoftirqd/322:33:153
222505599421,6cyclictest0-21swapper/323:37:533
2225055994210,8cyclictest131rcu_preempt21:30:013
2225051994229,10cyclictest30-21ksoftirqd/222:30:072
2225051994222,14cyclictest275-21dbus-daemon19:50:012
222505199421,3cyclictest131rcu_preempt21:25:012
222504499427,5cyclictest131rcu_preempt23:40:001
2225044994235,4cyclictest2227850-21latency_hist20:05:001
2225044994227,12cyclictest2227296-21sshd19:52:111
222504499421,3cyclictest131rcu_preempt22:30:001
2225040994225,13cyclictest2228086-21sshd20:08:110
2225040994223,14cyclictest275-21dbus-daemon20:00:000
222505599418,5cyclictest131rcu_preempt00:30:233
2225055994130,8cyclictest37-21ksoftirqd/322:52:523
2225055994116,6cyclictest37-21ksoftirqd/320:10:013
222505599411,5cyclictest0-21swapper/321:40:303
2225055994115,20cyclictest131rcu_preempt19:35:003
2225055994112,5cyclictest131rcu_preempt20:45:013
2225051994133,5cyclictest2217190-21kworker/2:320:06:162
2225051994128,10cyclictest30-21ksoftirqd/219:40:012
2225051994128,10cyclictest30-21ksoftirqd/219:40:012
222504499416,18cyclictest2246099-21kworker/u8:122:05:031
222504499413,4cyclictest131rcu_preempt23:35:011
2225044994129,8cyclictest2227541-21sshd19:56:101
2225044994127,9cyclictest23-21ksoftirqd/121:40:001
222504499412,6cyclictest2229164-21munin-run20:30:011
2225044994126,10cyclictest309-21in:imuxsock00:15:001
2225044994125,11cyclictest23-21ksoftirqd/122:15:011
222505599409,5cyclictest131rcu_preempt19:25:023
222505599409,5cyclictest131rcu_preempt19:25:023
2225055994025,11cyclictest37-21ksoftirqd/300:22:373
2225055994011,5cyclictest131rcu_preempt19:30:003
2225051994032,5cyclictest2229412-21kworker/2:020:43:142
222505199401,3cyclictest131rcu_preempt19:22:102
222505199401,3cyclictest131rcu_preempt19:22:102
2225044994029,8cyclictest23-21ksoftirqd/123:10:001
2225044994027,9cyclictest23-21ksoftirqd/122:18:151
2225044994026,10cyclictest3467602-21systemd-journal19:50:021
2225044994021,14cyclictest275-21dbus-daemon00:25:001
222504099403,5cyclictest0-21swapper/019:45:020
2225040994026,11cyclictest12-21ksoftirqd/020:00:180
2225040994024,11cyclictest3467602-21systemd-journal19:11:020
222505599395,7cyclictest37-21ksoftirqd/300:25:273
222505599394,8cyclictest37-21ksoftirqd/319:40:013
222505599394,8cyclictest37-21ksoftirqd/319:40:013
222505599391,6cyclictest131rcu_preempt19:59:593
2225055993912,6cyclictest131rcu_preempt21:10:013
2225055993910,5cyclictest131rcu_preempt19:52:113
222505199394,3cyclictest131rcu_preempt21:52:272
222504499396,7cyclictest23-21ksoftirqd/123:14:051
2225044993927,9cyclictest23-21ksoftirqd/123:00:011
2225044993923,12cyclictest23-21ksoftirqd/122:52:281
222504499391,3cyclictest131rcu_preempt23:55:011
222504099393,3cyclictest131rcu_preempt19:35:010
2225040993932,5cyclictest2230018-21latency_hist20:45:010
2225040993926,10cyclictest2226017-21sshd19:29:110
2225040993924,11cyclictest3467718-21lldpd20:52:320
222504099391,34cyclictest2229381-21sshd20:33:160
2225055993819,14cyclictest3467602-21systemd-journal19:45:003
2225055993814,5cyclictest131rcu_preempt20:50:023
2225055993814,5cyclictest131rcu_preempt20:50:023
222505199384,11cyclictest131rcu_preempt20:15:022
2225051993830,5cyclictest2218766-21kworker/2:119:59:112
2225051993823,11cyclictest275-21dbus-daemon20:55:012
2225051993810,5cyclictest131rcu_preempt20:25:152
222504499385,5cyclictest131rcu_preempt00:15:081
222504499385,5cyclictest131rcu_preempt00:15:071
222504499385,3cyclictest131rcu_preempt21:50:021
222504499384,4cyclictest131rcu_preempt20:45:001
222504499384,4cyclictest131rcu_preempt00:05:011
222504499383,8cyclictest23-21ksoftirqd/122:35:001
222504499382,8cyclictest0-21swapper/121:20:011
2225044993827,9cyclictest23-21ksoftirqd/122:50:011
2225044993827,9cyclictest23-21ksoftirqd/121:25:241
2225040993825,10cyclictest12-21ksoftirqd/020:25:460
2225040993824,10cyclictest12-21ksoftirqd/019:42:270
222505599375,10cyclictest37-21ksoftirqd/320:05:013
222505199374,4cyclictest131rcu_preempt20:25:002
2225051993726,8cyclictest30-21ksoftirqd/219:30:022
2225051993723,10cyclictest30-21ksoftirqd/219:15:322
222505199371,3cyclictest131rcu_preempt21:05:012
222504499377,7cyclictest23-21ksoftirqd/123:03:221
222504499377,7cyclictest23-21ksoftirqd/121:05:151
222504499373,3cyclictest131rcu_preempt21:55:011
2225044993728,6cyclictest2230364-21latency_hist20:50:031
2225044993728,6cyclictest2230364-21latency_hist20:50:021
2225044993725,10cyclictest23-21ksoftirqd/121:45:021
2225044993722,11cyclictest23-21ksoftirqd/100:25:361
222504499371,9cyclictest2295500-21latency_hist23:45:021
222504499371,9cyclictest2295500-21latency_hist23:45:011
2225040993731,3cyclictest0-21swapper/021:01:150
2225040993719,4cyclictest2225758-21sshd19:22:360
2225040993719,4cyclictest2225758-21sshd19:22:360
222505599367,5cyclictest131rcu_preempt19:20:003
222505599362,9cyclictest2229112-21sshd20:25:153
222505199362,3cyclictest131rcu_preempt21:10:002
2225051993622,10cyclictest275-21dbus-daemon19:15:012
2225044993623,10cyclictest23-21ksoftirqd/119:30:001
2225044993623,10cyclictest23-21ksoftirqd/100:06:451
222505599353,6cyclictest131rcu_preempt19:10:023
2225055993511,4cyclictest131rcu_preempt20:25:023
2225044993528,4cyclictest2229622-21sshd20:36:161
222504499352,5cyclictest131rcu_preempt19:20:001
2225044993514,9cyclictest23-21ksoftirqd/123:24:401
2225040993524,9cyclictest12-21ksoftirqd/019:10:020
2225051993427,4cyclictest2218766-21kworker/2:119:52:052
222505199341,5cyclictest131rcu_preempt20:01:112
222504499345,4cyclictest131rcu_preempt19:15:021
2225044993419,10cyclictest23-21ksoftirqd/119:35:001
2225051993324,7cyclictest30-21ksoftirqd/220:50:012
2225051993324,7cyclictest30-21ksoftirqd/220:50:012
2225044993323,8cyclictest23-21ksoftirqd/121:15:031
2225044993323,8cyclictest23-21ksoftirqd/121:15:021
2225044993323,7cyclictest23-21ksoftirqd/120:15:011
2225044993322,7cyclictest23-21ksoftirqd/119:40:001
2225044993322,7cyclictest23-21ksoftirqd/119:40:001
2225051993225,4cyclictest2225071-21latency_hist19:10:022
2225044993224,6cyclictest23-21ksoftirqd/120:35:021
2225044993119,8cyclictest2228609-21modprobe20:20:001
2225044993118,10cyclictest2231036-21wc21:05:021
2225051993014,11cyclictest275-21dbus-daemon19:30:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional