You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-17 - 10:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Jun 17, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2910729997970,6cyclictest12-21ksoftirqd/020:15:020
2910729997130,27cyclictest12-21ksoftirqd/023:10:020
291073999692,17cyclictest2955123-21munin-run22:35:002
2910739996921,41cyclictest309-21in:imuxsock22:30:012
291072999684,9cyclictest121ksoftirqd/023:49:310
2910739996629,32cyclictest309-21in:imuxsock23:20:002
2910739996625,35cyclictest2932702-21latency_hist21:45:012
2910739996620,40cyclictest3467602-21systemd-journal00:30:002
291073999653,19cyclictest364-21snmpd00:31:592
2910739996524,36cyclictest2943890-21latency_hist22:10:012
2910739996521,40cyclictest2980845-21sh23:32:142
291073999652,13cyclictest2937246-21latency_hist21:55:012
2910739996520,39cyclictest309-21in:imuxsock23:30:002
291072999656,7cyclictest12-21ksoftirqd/021:25:290
2910739996432,28cyclictest3467602-21systemd-journal21:38:222
291074399630,54cyclictest0-21swapper/320:55:013
2910739996333,26cyclictest2912227-21sendmail19:40:002
291073999632,17cyclictest2957328-21cron22:39:592
291073999614,18cyclictest2977514-21munin-plugin-st23:25:002
2910739996124,33cyclictest2968684-21latency_hist23:05:022
291072999618,49cyclictest2959781-21ssh22:45:030
291072999618,49cyclictest2959781-21ssh22:45:020
291073999603,17cyclictest2941574-21ssh22:05:012
2910739996022,34cyclictest2923642-21latency_hist21:25:022
2910739996022,33cyclictest2962121-21sh22:50:142
2910739996021,35cyclictest2991136-21latency_hist23:55:032
291072999606,9cyclictest121ksoftirqd/022:47:310
291072999604,7cyclictest12-21ksoftirqd/021:11:270
2910729996015,9cyclictest12-21ksoftirqd/023:55:010
291073999593,19cyclictest2948527-21grep22:20:022
291073999593,19cyclictest2948527-21grep22:20:022
2910739995921,34cyclictest2987932-21sh23:48:072
2910739995917,5cyclictest3002753-21rm00:20:552
291073999591,16cyclictest2986429-21ssh23:44:502
291073999591,15cyclictest2915631-21unin-run20:45:002
2910734995914,8cyclictest23-21ksoftirqd/123:45:021
291072999596,8cyclictest12-21ksoftirqd/022:05:020
2910729995931,12cyclictest2944243-21ssh22:10:350
291073999582,14cyclictest364-21snmpd21:16:202
291073499586,9cyclictest23-21ksoftirqd/123:35:521
291072999588,9cyclictest12-21ksoftirqd/000:09:530
291073999574,15cyclictest2912780-21wc19:50:012
2910739995725,27cyclictest3001421-21sh00:18:102
2910739995725,27cyclictest3001421-21sh00:18:092
2910739995724,29cyclictest30-21ksoftirqd/221:26:112
2910739995722,30cyclictest2913786-21latency_hist20:10:022
2910739995720,33cyclictest2983264-21sh23:37:392
2910739995719,4cyclictest2916909-21diskmemload22:55:382
2910739995718,4cyclictest2916909-21diskmemload22:43:412
2910739995718,4cyclictest2916909-21diskmemload22:43:402
2910739995714,9cyclictest2916618-21date21:00:022
291073999571,18cyclictest2973021-21sh23:15:002
291073999571,18cyclictest2973021-21sh23:15:002
2910729995728,14cyclictest3001558-21ssh00:18:280
2910729995728,14cyclictest3001558-21ssh00:18:270
2910743995616,24cyclictest2955239-21ssh22:35:023
291073999564,12cyclictest2950788-21latency_hist22:25:042
291073999564,12cyclictest2950788-21latency_hist22:25:032
291073999564,12cyclictest2914684-21grep20:25:012
291073499567,11cyclictest23-21ksoftirqd/122:29:271
291073499565,10cyclictest23-21ksoftirqd/122:12:411
2910734995615,12cyclictest23-21ksoftirqd/121:50:011
2910734995615,12cyclictest23-21ksoftirqd/121:50:011
291072999565,10cyclictest12-21ksoftirqd/000:14:200
2910729995644,2cyclictest0-21swapper/021:55:010
2910729995626,14cyclictest2973294-21ssh23:15:060
2910729995617,33cyclictest16850irq/134-800000023:25:060
2910729995610,11cyclictest12-21ksoftirqd/020:40:010
2910729995610,11cyclictest12-21ksoftirqd/020:40:000
291073999553,11cyclictest2993399-21latency_hist00:00:042
291073999553,11cyclictest2993399-21latency_hist00:00:042
291073999552,17cyclictest2914821-21cron20:30:002
291073999552,16cyclictest2934548-21ssh21:49:172
291073999552,16cyclictest2934548-21ssh21:49:172
2910739995518,7cyclictest30-21ksoftirqd/221:56:372
2910739995515,4cyclictest2916909-21diskmemload22:45:432
2910739995514,7cyclictest275-21dbus-daemon20:00:012
291073499551,7cyclictest23-21ksoftirqd/123:26:361
291072999555,7cyclictest2913271-21munin-run20:00:020
2910729995527,13cyclictest2954761-21ssh22:34:140
2910729995526,14cyclictest2955522-21ssh22:35:340
2910729995519,6cyclictest16850irq/134-800000022:58:300
2910729995519,32cyclictest16850irq/134-800000000:27:130
2910729995518,6cyclictest16850irq/134-800000022:54:400
2910729995510,13cyclictest12-21ksoftirqd/000:21:140
291073999543,16cyclictest3011180-21sendmail-msp00:40:012
2910739995429,17cyclictest275-21dbus-daemon00:05:012
2910739995427,24cyclictest30-21ksoftirqd/219:33:212
2910739995417,5cyclictest2916909-21diskmemload23:07:182
291073499545,7cyclictest23-21ksoftirqd/121:13:141
291073499544,6cyclictest23-21ksoftirqd/100:10:021
2910734995436,11cyclictest275-21dbus-daemon22:20:001
2910734995436,11cyclictest275-21dbus-daemon22:20:001
2910734995430,9cyclictest23-21ksoftirqd/122:05:011
2910734995411,5cyclictest131rcu_preempt00:10:511
291073499541,14cyclictest2995421-21modprobe00:04:591
291072999548,6cyclictest19350irq/34-mmc123:57:530
291072999548,6cyclictest19350irq/34-mmc123:57:520
291072999546,10cyclictest12-21ksoftirqd/023:37:120
2910729995418,7cyclictest16850irq/134-800000022:19:070
2910729995418,7cyclictest16850irq/134-800000022:19:070
2910729995418,6cyclictest16850irq/134-800000021:43:570
2910729995416,9cyclictest131rcu_preempt19:20:000
2910739995324,25cyclictest30-21ksoftirqd/200:08:592
291073999532,17cyclictest1-21systemd20:50:022
291073999532,17cyclictest1-21systemd20:50:012
2910739995316,12cyclictest3467602-21systemd-journal00:11:032
291073499539,6cyclictest131rcu_preempt00:30:001
291072999539,4cyclictest131rcu_preempt22:26:220
291072999538,11cyclictest12-21ksoftirqd/022:10:020
2910729995346,4cyclictest16950irq/224-800000021:10:010
2910729995345,5cyclictest16950irq/224-800000021:50:010
2910729995345,5cyclictest16950irq/224-800000021:50:000
2910729995327,11cyclictest2920257-21ssh21:17:200
291072999531,9cyclictest0-21swapper/021:56:080
2910729995318,6cyclictest16850irq/134-800000021:33:420
2910729995318,31cyclictest16850irq/134-800000023:01:190
2910729995317,6cyclictest16850irq/134-800000000:38:130
2910729995317,31cyclictest16850irq/134-800000021:38:310
2910729995316,6cyclictest131rcu_preempt22:25:010
2910729995316,6cyclictest131rcu_preempt22:25:000
2910743995234,13cyclictest0-21swapper/322:55:013
291074399522,6cyclictest131rcu_preempt23:59:593
291074399522,6cyclictest131rcu_preempt00:00:003
2910739995243,6cyclictest2909945-21kworker/2:021:13:372
2910739995220,5cyclictest30-21ksoftirqd/222:13:512
2910739995214,7cyclictest2911344-21cat19:20:012
291073499526,9cyclictest23-21ksoftirqd/123:10:001
291073499526,10cyclictest23-21ksoftirqd/121:54:301
2910734995244,6cyclictest23-21ksoftirqd/100:38:071
291073499524,10cyclictest23-21ksoftirqd/100:21:521
2910734995236,12cyclictest2975248-21cron23:19:591
291073499523,14cyclictest23-21ksoftirqd/123:00:331
291073499522,8cyclictest23-21ksoftirqd/122:32:551
291072999527,9cyclictest12-21ksoftirqd/000:33:330
291072999526,10cyclictest12-21ksoftirqd/023:42:220
2910729995234,13cyclictest3467602-21systemd-journal20:45:000
2910729995218,31cyclictest16850irq/134-800000021:22:260
2910729995217,6cyclictest16850irq/134-800000023:31:320
2910729995217,6cyclictest16850irq/134-800000000:04:170
291073999516,12cyclictest2910778-21grep19:10:012
2910739995142,5cyclictest2909945-21kworker/2:020:20:012
2910739995117,6cyclictest131rcu_preempt19:45:012
2910739995117,6cyclictest131rcu_preempt19:45:012
291073499517,7cyclictest23-21ksoftirqd/123:23:531
291073499516,9cyclictest23-21ksoftirqd/121:21:301
291073499515,8cyclictest23-21ksoftirqd/121:41:571
291073499512,9cyclictest23-21ksoftirqd/100:19:241
291073499512,9cyclictest23-21ksoftirqd/100:19:231
291073499511,10cyclictest2993406-21wc00:00:051
291073499511,10cyclictest2993406-21wc00:00:041
2910729995119,28cyclictest16850irq/134-800000023:20:500
2910729995117,6cyclictest16850irq/134-800000023:14:160
2910729995117,6cyclictest16850irq/134-800000023:14:160
2910729995112,7cyclictest131rcu_preempt20:00:210
291072999511,12cyclictest12-21ksoftirqd/019:55:010
2910743995029,16cyclictest275-21dbus-daemon22:00:013
291073499508,9cyclictest23-21ksoftirqd/122:10:031
291073499508,6cyclictest23-21ksoftirqd/122:00:021
291073499507,8cyclictest23-21ksoftirqd/120:00:001
291073499507,7cyclictest23-21ksoftirqd/121:30:221
291073499505,7cyclictest23-21ksoftirqd/123:30:531
291073499503,9cyclictest23-21ksoftirqd/121:39:231
291073499501,5cyclictest0-21swapper/123:10:451
291073499501,5cyclictest0-21swapper/123:10:441
2910743994932,13cyclictest275-21dbus-daemon21:35:003
2910743994932,12cyclictest275-21dbus-daemon23:00:013
291073999494,11cyclictest2911483-21unin-run19:25:012
2910739994916,11cyclictest30-21ksoftirqd/221:32:402
291073499496,8cyclictest23-21ksoftirqd/100:30:071
291073499494,9cyclictest23-21ksoftirqd/122:44:071
291073499494,9cyclictest23-21ksoftirqd/122:44:071
291073499494,11cyclictest23-21ksoftirqd/123:50:191
2910734994936,9cyclictest23-21ksoftirqd/122:25:021
2910734994936,9cyclictest23-21ksoftirqd/122:25:011
2910743994835,10cyclictest3467602-21systemd-journal00:07:273
2910739994813,7cyclictest30-21ksoftirqd/220:57:252
2910739994813,7cyclictest30-21ksoftirqd/220:57:252
291073499487,9cyclictest23-21ksoftirqd/120:30:001
291073499485,11cyclictest23-21ksoftirqd/121:19:111
2910734994832,12cyclictest1-21systemd20:25:001
2910734994819,20cyclictest221rcuc/122:45:191
2910734994813,5cyclictest131rcu_preempt22:54:011
2910734994811,5cyclictest131rcu_preempt23:49:331
291073499481,13cyclictest23-21ksoftirqd/120:45:011
291072999488,5cyclictest131rcu_preempt19:45:030
2910729994830,14cyclictest2912073-21latency_hist19:35:010
291072999482,39cyclictest2914293-21cron20:20:000
291074399478,7cyclictest131rcu_preempt22:20:013
291074399478,7cyclictest131rcu_preempt22:20:003
291074399474,4cyclictest131rcu_preempt23:46:183
2910743994732,12cyclictest3467602-21systemd-journal22:24:243
2910743994732,12cyclictest3467602-21systemd-journal22:24:233
2910743994730,13cyclictest3467602-21systemd-journal20:40:083
2910743994729,13cyclictest275-21dbus-daemon00:25:003
291073999474,11cyclictest0-21swapper/221:10:002
2910739994739,5cyclictest2902120-21kworker/2:220:01:032
2910739994738,6cyclictest2909945-21kworker/2:019:55:022
2910739994715,6cyclictest30-21ksoftirqd/219:30:022
2910739994715,6cyclictest30-21ksoftirqd/219:30:022
291073499478,7cyclictest23-21ksoftirqd/121:29:251
291073499477,6cyclictest131rcu_preempt22:39:031
2910734994738,6cyclictest23-21ksoftirqd/119:25:011
2910734994716,5cyclictest131rcu_preempt22:56:551
291072999478,5cyclictest131rcu_preempt19:25:010
2910743994632,10cyclictest3467602-21systemd-journal23:26:503
2910734994631,12cyclictest1-21systemd19:35:001
2910734994631,11cyclictest275-21dbus-daemon19:50:001
2910743994530,11cyclictest3467602-21systemd-journal00:35:023
291074399451,3cyclictest131rcu_preempt22:04:573
291073499458,7cyclictest23-21ksoftirqd/119:43:211
291073499458,7cyclictest23-21ksoftirqd/119:43:211
291073499458,5cyclictest131rcu_preempt19:40:001
291073499456,9cyclictest23-21ksoftirqd/119:29:211
291073499456,9cyclictest23-21ksoftirqd/119:29:211
291072999457,7cyclictest12-21ksoftirqd/019:15:020
291072999455,8cyclictest12-21ksoftirqd/019:10:020
2910729994534,8cyclictest12-21ksoftirqd/019:45:020
2910729994534,8cyclictest12-21ksoftirqd/019:45:020
2910729994521,18cyclictest275-21dbus-daemon20:10:010
291074399446,25cyclictest3467602-21systemd-journal21:10:003
2910743994434,7cyclictest37-21ksoftirqd/323:15:373
2910743994427,11cyclictest275-21dbus-daemon20:05:013
291073999441,10cyclictest2915379-21unin-run20:40:002
291073999441,10cyclictest2915379-21unin-run20:40:002
2910734994430,10cyclictest3467602-21systemd-journal19:55:001
291072999446,10cyclictest12-21ksoftirqd/020:35:010
291072999441,8cyclictest0-21swapper/019:30:020
291072999441,8cyclictest0-21swapper/019:30:010
291074399433,4cyclictest131rcu_preempt21:10:043
2910743994331,8cyclictest2952904-21cat22:30:013
2910743994329,12cyclictest37-21ksoftirqd/300:12:083
2910743994320,18cyclictest361rcuc/323:10:013
291073499439,5cyclictest131rcu_preempt20:16:211
291073499434,7cyclictest23-21ksoftirqd/121:00:011
291073499434,7cyclictest23-21ksoftirqd/121:00:001
291073499432,9cyclictest23-21ksoftirqd/120:11:201
2910734994320,5cyclictest221rcuc/121:10:011
291072999436,7cyclictest12-21ksoftirqd/020:55:010
291072999435,8cyclictest12-21ksoftirqd/020:25:000
2910743994230,8cyclictest3467602-21systemd-journal20:12:213
2910743994227,12cyclictest37-21ksoftirqd/321:05:013
291074399421,3cyclictest131rcu_preempt21:26:013
2910729994210,5cyclictest131rcu_preempt20:50:030
2910729994210,5cyclictest131rcu_preempt20:50:020
2910743994129,6cyclictest309-21in:imuxsock20:00:003
2910743994123,14cyclictest37-21ksoftirqd/321:45:213
2910743994123,14cyclictest37-21ksoftirqd/321:45:213
2910743994122,15cyclictest3467602-21systemd-journal19:30:013
2910743994122,15cyclictest3467602-21systemd-journal19:30:003
291074399411,3cyclictest131rcu_preempt21:24:463
291074399411,3cyclictest131rcu_preempt00:38:013
291073999411,16cyclictest2914048-21systemd-run20:15:012
291073499415,6cyclictest23-21ksoftirqd/119:10:011
291073499413,8cyclictest23-21ksoftirqd/119:20:021
291073499413,7cyclictest23-21ksoftirqd/120:40:001
291073499413,7cyclictest23-21ksoftirqd/120:40:001
2910734994129,9cyclictest23-21ksoftirqd/120:05:001
2910734994126,11cyclictest3467602-21systemd-journal20:45:051
2910734994126,11cyclictest3467602-21systemd-journal20:45:051
2910729994132,6cyclictest12-21ksoftirqd/021:05:000
291072999411,36cyclictest2912208-21sshd19:39:220
291074399404,9cyclictest131rcu_preempt21:40:003
291074399404,32cyclictest3002202-21munin-plugin-st00:20:013
291074399404,32cyclictest3002202-21munin-plugin-st00:20:013
2910743994026,11cyclictest2959731-21sed22:45:033
2910743994026,11cyclictest2959731-21sed22:45:023
2910743994026,10cyclictest3467602-21systemd-journal19:20:023
2910743994019,17cyclictest3467602-21systemd-journal19:15:003
2910743994019,16cyclictest37-21ksoftirqd/300:05:003
2910739994016,8cyclictest30-21ksoftirqd/219:15:012
2910734994030,7cyclictest23-21ksoftirqd/120:10:011
2910743993931,6cyclictest37-21ksoftirqd/323:02:463
2910743993930,5cyclictest3006744-21latency_hist00:30:023
2910743993924,10cyclictest309-21in:imuxsock23:40:003
2910743993922,13cyclictest275-21dbus-daemon23:45:023
291074399391,3cyclictest131rcu_preempt22:45:373
291074399391,3cyclictest131rcu_preempt21:40:143
291073499394,6cyclictest2915066-21sshd20:33:201
2910729993927,8cyclictest2914829-21sh20:30:010
2910743993829,7cyclictest37-21ksoftirqd/322:40:023
2910743993828,8cyclictest37-21ksoftirqd/320:31:113
2910743993827,7cyclictest37-21ksoftirqd/319:30:213
2910743993827,6cyclictest309-21in:imuxsock19:20:013
2910743993817,18cyclictest37-21ksoftirqd/322:05:033
2910734993825,10cyclictest2910978-21cat19:15:021
291073499381,7cyclictest0-21swapper/121:01:241
2910743993728,5cyclictest2915859-21cron20:50:003
2910743993728,5cyclictest2915859-21cron20:50:003
2910743993725,10cyclictest37-21ksoftirqd/323:32:543
2910743993724,10cyclictest37-21ksoftirqd/322:10:313
2910743993724,10cyclictest2936074-21sshd21:52:223
2910743993722,11cyclictest37-21ksoftirqd/323:23:173
2910743993719,14cyclictest275-21dbus-daemon19:50:013
291073499371,8cyclictest0-21swapper/120:54:321
291072999378,5cyclictest131rcu_preempt20:55:180
291072999378,5cyclictest131rcu_preempt20:55:170
2910743993628,5cyclictest2916391-21sendmail21:00:013
2910743993628,5cyclictest2916391-21sendmail21:00:003
2910743993628,5cyclictest2913006-21latency_hist19:55:013
2910743993626,7cyclictest2914893-21latency_hist20:30:013
2910743993624,9cyclictest37-21ksoftirqd/323:50:153
2910743993624,9cyclictest37-21ksoftirqd/323:12:463
2910743993624,9cyclictest37-21ksoftirqd/323:12:463
2910743993623,10cyclictest2912279-21sort19:40:013
2910743993615,4cyclictest37-21ksoftirqd/321:16:273
2910743993523,10cyclictest37-21ksoftirqd/319:10:013
2910739993526,6cyclictest2915147-21cat20:35:022
291073999352,3cyclictest131rcu_preempt20:55:012
291074399332,6cyclictest131rcu_preempt20:10:003
2910743993322,5cyclictest37-21ksoftirqd/320:40:013
2910743993322,5cyclictest37-21ksoftirqd/320:40:013
291074399321,4cyclictest131rcu_preempt19:45:013
291074399321,4cyclictest131rcu_preempt19:45:013
291074399302,7cyclictest0-21swapper/320:20:023
291074399284,18cyclictest3467602-21systemd-journal20:25:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional