You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 11:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Apr 23, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2070671998273,6cyclictest12-21ksoftirqd/021:25:020
2070671996811,6cyclictest131rcu_preempt21:49:390
207067199676,11cyclictest12-21ksoftirqd/000:17:080
2070677996625,11cyclictest30-21ksoftirqd/222:27:492
207067199659,6cyclictest131rcu_preempt23:39:550
207067199659,6cyclictest131rcu_preempt23:39:550
2070674996311,14cyclictest2130621-21sh23:14:341
207067199637,5cyclictest131rcu_preempt23:00:000
207068199625,9cyclictest37-21ksoftirqd/322:09:593
207067199625,13cyclictest12-21ksoftirqd/019:25:000
2070671996211,7cyclictest12-21ksoftirqd/023:03:490
207067799615,13cyclictest30-21ksoftirqd/220:40:022
207067799606,7cyclictest30-21ksoftirqd/219:10:022
207067199606,13cyclictest12-21ksoftirqd/019:40:020
207067199596,11cyclictest12-21ksoftirqd/021:38:130
2070671995916,11cyclictest2163221-21cron00:30:000
2070671995916,11cyclictest2163221-21cron00:29:590
207067199589,5cyclictest131rcu_preempt22:35:340
207067199586,14cyclictest12-21ksoftirqd/000:24:410
2070671995830,13cyclictest2086885-21ssh21:33:110
2070671995825,30cyclictest2120077-21cat22:50:020
2070671995820,34cyclictest16850irq/134-800000000:32:310
2070671995818,19cyclictest131rcu_preempt23:55:510
2070681995732,15cyclictest37-21ksoftirqd/323:17:283
2070681995721,7cyclictest37-21ksoftirqd/322:01:463
2070681995711,10cyclictest37-21ksoftirqd/321:28:153
2070671995728,12cyclictest2131194-21ssh23:15:240
2070671995723,30cyclictest12-21ksoftirqd/000:06:480
2070671995723,30cyclictest12-21ksoftirqd/000:06:470
2070671995713,15cyclictest12-21ksoftirqd/023:53:020
2070671995711,17cyclictest131rcu_preempt21:57:320
2070681995616,7cyclictest37-21ksoftirqd/321:47:263
2070674995635,16cyclictest2089753-21systemctl21:40:011
207067499562,18cyclictest2165068-21ssh00:34:081
207067199566,8cyclictest12-21ksoftirqd/000:12:210
207067199565,11cyclictest12-21ksoftirqd/000:38:590
207067199565,11cyclictest12-21ksoftirqd/000:38:590
207068199556,6cyclictest37-21ksoftirqd/323:10:143
207067799557,9cyclictest30-21ksoftirqd/223:44:352
207067799553,6cyclictest30-21ksoftirqd/221:10:162
2070674995517,10cyclictest0-21swapper/122:37:171
207067499551,15cyclictest2157476-21ssh00:16:241
207067499551,15cyclictest2086699-21ssh21:32:431
207067499551,14cyclictest2147077-21ssh23:52:241
207067199559,4cyclictest131rcu_preempt20:45:000
207067199557,13cyclictest12-21ksoftirqd/023:33:070
2070671995515,31cyclictest16850irq/134-800000022:25:020
2070681995411,11cyclictest37-21ksoftirqd/300:27:483
2070681995411,11cyclictest37-21ksoftirqd/300:27:483
207067799549,5cyclictest131rcu_preempt00:16:182
2070677995435,8cyclictest30-21ksoftirqd/221:58:542
207067499542,21cyclictest2096852-21rm21:56:081
207067499541,17cyclictest2110176-21ssh22:26:531
207067499541,17cyclictest2073419-21sed20:00:031
207067199549,5cyclictest131rcu_preempt23:48:150
207067199546,6cyclictest121ksoftirqd/021:16:170
207068199536,7cyclictest37-21ksoftirqd/300:15:563
207068199535,11cyclictest37-21ksoftirqd/323:55:333
2070677995336,13cyclictest275-21dbus-daemon23:00:012
2070677995314,9cyclictest30-21ksoftirqd/223:40:032
2070677995314,9cyclictest30-21ksoftirqd/223:40:022
207067499531,15cyclictest2090407-21ssh21:41:081
207067199538,5cyclictest131rcu_preempt21:40:120
207067199537,11cyclictest12-21ksoftirqd/022:25:430
207067199536,11cyclictest12-21ksoftirqd/022:03:480
2070671995323,14cyclictest2121792-21ssh22:54:050
2070671995317,6cyclictest16850irq/134-800000023:11:080
2070671995316,32cyclictest16850irq/134-800000022:07:240
2070671995315,34cyclictest16850irq/134-800000021:10:010
2070671995315,34cyclictest16850irq/134-800000021:10:010
2070671995315,10cyclictest12-21ksoftirqd/000:05:000
207068199527,7cyclictest37-21ksoftirqd/323:44:293
207068199527,11cyclictest37-21ksoftirqd/323:08:493
2070681995216,6cyclictest37-21ksoftirqd/323:33:093
207067799527,7cyclictest30-21ksoftirqd/200:37:062
207067799527,7cyclictest30-21ksoftirqd/200:37:052
207067799526,8cyclictest30-21ksoftirqd/223:49:522
207067799526,7cyclictest30-21ksoftirqd/221:30:132
207067799523,10cyclictest30-21ksoftirqd/220:45:002
2070677995213,8cyclictest30-21ksoftirqd/222:17:552
207067499523,13cyclictest2152131-21sh00:04:111
207067499521,18cyclictest2076888-21cat21:10:011
207067499521,18cyclictest2076888-21cat21:10:011
207067199527,5cyclictest131rcu_preempt22:11:410
207067199527,5cyclictest131rcu_preempt22:11:400
207067199527,13cyclictest12-21ksoftirqd/021:00:020
207067199527,13cyclictest12-21ksoftirqd/021:00:010
207067199524,11cyclictest12-21ksoftirqd/022:18:330
207067199522,7cyclictest12-21ksoftirqd/023:43:110
2070671995211,11cyclictest12-21ksoftirqd/022:21:310
2070671995210,5cyclictest131rcu_preempt23:24:150
207068199518,4cyclictest131rcu_preempt22:23:343
207068199518,4cyclictest131rcu_preempt22:23:343
207068199517,8cyclictest37-21ksoftirqd/321:17:323
207068199517,6cyclictest37-21ksoftirqd/321:05:023
207068199516,7cyclictest37-21ksoftirqd/321:38:093
207068199515,9cyclictest37-21ksoftirqd/323:52:353
207068199515,7cyclictest37-21ksoftirqd/322:17:083
207068199514,7cyclictest37-21ksoftirqd/300:02:383
2070681995133,13cyclictest275-21dbus-daemon00:40:013
2070681995133,13cyclictest275-21dbus-daemon00:40:003
2070681995133,13cyclictest1-21systemd22:15:013
2070681995133,13cyclictest1-21systemd22:15:013
207068199512,7cyclictest37-21ksoftirqd/300:06:393
207068199512,7cyclictest37-21ksoftirqd/300:06:383
2070681995114,10cyclictest37-21ksoftirqd/323:27:343
2070681995113,9cyclictest37-21ksoftirqd/323:00:523
207067799516,9cyclictest30-21ksoftirqd/200:13:332
207067799516,7cyclictest30-21ksoftirqd/222:08:412
2070677995142,7cyclictest291rcuc/221:25:242
207067499519,28cyclictest2073966-21modprobe20:15:001
2070674995124,19cyclictest3467602-21systemd-journal22:15:011
2070674995124,19cyclictest3467602-21systemd-journal22:15:001
207067499512,14cyclictest2133218-21date23:20:031
207067499511,19cyclictest2116501-21ssh22:41:291
207067499511,16cyclictest2139830-21ssh23:35:321
207067499511,16cyclictest2139830-21ssh23:35:311
207067499511,12cyclictest2121863-21ssh22:54:151
207067199516,11cyclictest12-21ksoftirqd/022:32:070
207067199516,11cyclictest12-21ksoftirqd/022:32:060
207067199515,7cyclictest12-21ksoftirqd/021:13:080
207067199514,9cyclictest12-21ksoftirqd/021:25:200
207068199507,7cyclictest37-21ksoftirqd/321:13:523
207068199505,7cyclictest37-21ksoftirqd/322:57:593
207068199505,7cyclictest37-21ksoftirqd/322:43:043
207068199505,7cyclictest37-21ksoftirqd/321:52:353
207068199505,6cyclictest37-21ksoftirqd/321:41:193
207068199504,7cyclictest37-21ksoftirqd/322:29:483
2070681995037,10cyclictest2119731-21sshd22:49:143
207067799509,5cyclictest131rcu_preempt21:47:232
207067799506,5cyclictest30-21ksoftirqd/223:35:022
207067799505,9cyclictest30-21ksoftirqd/220:15:012
207067799505,8cyclictest30-21ksoftirqd/221:50:272
207067799504,8cyclictest30-21ksoftirqd/223:23:422
207067799504,7cyclictest30-21ksoftirqd/223:16:282
2070677995042,5cyclictest2070868-21kworker/2:021:00:542
2070677995039,8cyclictest291rcuc/222:35:092
207067799503,8cyclictest30-21ksoftirqd/222:03:072
2070677995035,11cyclictest30-21ksoftirqd/220:25:002
2070677995015,7cyclictest30-21ksoftirqd/223:54:442
207067499502,15cyclictest2161091-21latency_hist00:25:021
207067499501,17cyclictest2162177-21ssh00:27:191
207067499501,17cyclictest2162177-21ssh00:27:191
207067499501,15cyclictest2125540-21ssh23:02:281
207067199507,5cyclictest131rcu_preempt23:08:570
207067199505,10cyclictest12-21ksoftirqd/019:55:010
207067199503,11cyclictest12-21ksoftirqd/022:41:120
207068199496,8cyclictest37-21ksoftirqd/319:30:023
207068199495,10cyclictest37-21ksoftirqd/323:21:073
207067799498,9cyclictest30-21ksoftirqd/200:32:002
207067799496,7cyclictest30-21ksoftirqd/223:27:062
207067799496,7cyclictest30-21ksoftirqd/222:53:472
207067799495,5cyclictest30-21ksoftirqd/223:08:472
207067799495,11cyclictest30-21ksoftirqd/200:25:122
207067799495,11cyclictest30-21ksoftirqd/200:25:122
207067799494,6cyclictest30-21ksoftirqd/222:42:062
207067799491,13cyclictest30-21ksoftirqd/219:11:062
207067499492,15cyclictest2128475-21ssh23:09:321
207067499491,15cyclictest2153508-21ssh00:07:141
207067499491,15cyclictest2153508-21ssh00:07:141
207067199494,9cyclictest12-21ksoftirqd/019:30:010
207068199486,7cyclictest37-21ksoftirqd/323:39:433
207068199486,7cyclictest37-21ksoftirqd/323:39:433
207068199486,6cyclictest37-21ksoftirqd/321:20:043
207068199485,6cyclictest37-21ksoftirqd/321:57:553
207068199484,7cyclictest37-21ksoftirqd/321:32:013
207068199484,10cyclictest37-21ksoftirqd/322:35:293
207067799486,7cyclictest30-21ksoftirqd/222:47:202
207067799485,7cyclictest30-21ksoftirqd/223:03:002
207067799485,7cyclictest30-21ksoftirqd/222:32:352
207067799485,7cyclictest30-21ksoftirqd/222:32:342
207067799485,7cyclictest30-21ksoftirqd/222:11:392
207067799485,7cyclictest30-21ksoftirqd/222:11:382
207067799485,7cyclictest30-21ksoftirqd/220:00:002
207067799485,6cyclictest30-21ksoftirqd/200:21:552
207067799485,5cyclictest30-21ksoftirqd/221:16:582
2070677994840,5cyclictest2128712-21kworker/2:123:55:452
2070677994838,7cyclictest291rcuc/219:53:072
2070677994837,7cyclictest2070868-21kworker/2:020:30:022
2070677994814,4cyclictest131rcu_preempt00:10:012
2070677994814,4cyclictest131rcu_preempt00:10:002
2070674994836,9cyclictest3467602-21systemd-journal23:41:021
2070674994829,14cyclictest2156808-21diskmemload00:15:021
2070674994829,14cyclictest2132944-21cron23:20:001
2070674994827,17cyclictest3467602-21systemd-journal20:30:001
2070674994826,16cyclictest275-21dbus-daemon19:20:011
207067499481,14cyclictest2080851-21ssh21:19:211
207067199488,35cyclictest131rcu_preempt19:55:370
207067199484,10cyclictest12-21ksoftirqd/021:52:190
2070671994813,31cyclictest16850irq/134-800000023:25:320
207068199475,9cyclictest37-21ksoftirqd/300:11:523
207068199475,7cyclictest37-21ksoftirqd/300:31:103
207068199474,6cyclictest37-21ksoftirqd/320:00:013
2070681994728,14cyclictest1-21systemd19:15:013
207067799478,4cyclictest131rcu_preempt21:42:052
207067799477,6cyclictest30-21ksoftirqd/222:23:562
207067799477,6cyclictest30-21ksoftirqd/222:23:552
207067799476,8cyclictest30-21ksoftirqd/221:21:152
207067799475,7cyclictest30-21ksoftirqd/200:01:482
207067799475,11cyclictest30-21ksoftirqd/223:10:402
207067799474,10cyclictest30-21ksoftirqd/221:40:002
2070677994710,4cyclictest131rcu_preempt19:35:022
2070674994734,10cyclictest0-21swapper/122:05:031
2070674994733,10cyclictest2113560-21cron22:35:011
2070674994733,10cyclictest2113560-21cron22:35:011
207067499472,15cyclictest2108784-21ssh22:23:571
207067499472,15cyclictest2108784-21ssh22:23:561
207067499471,7cyclictest2078038-21ssh21:12:331
2070671994735,8cyclictest12-21ksoftirqd/020:40:010
207068199466,10cyclictest361rcuc/323:48:143
207068199465,8cyclictest37-21ksoftirqd/322:33:463
207068199465,8cyclictest37-21ksoftirqd/322:33:453
207068199465,10cyclictest37-21ksoftirqd/322:50:473
2070677994639,4cyclictest2074782-21kworker/2:121:05:182
2070677994639,4cyclictest2074782-21kworker/2:121:05:172
2070677994639,4cyclictest2074782-21kworker/2:120:58:262
2070677994639,4cyclictest2074782-21kworker/2:120:58:262
2070677994639,4cyclictest2070868-21kworker/2:020:00:062
2070677994639,4cyclictest2070868-21kworker/2:020:00:052
2070677994639,4cyclictest2070248-21kworker/2:319:41:502
2070677994639,4cyclictest2070248-21kworker/2:319:22:292
2070677994638,5cyclictest2070868-21kworker/2:020:18:172
2070677994638,5cyclictest2070868-21kworker/2:020:07:232
2070677994638,5cyclictest2070868-21kworker/2:019:27:472
2070677994635,7cyclictest291rcuc/220:30:132
2070674994633,10cyclictest2072170-21tr19:39:591
2070674994626,16cyclictest275-21dbus-daemon20:05:011
2070674994626,16cyclictest275-21dbus-daemon20:05:001
207067499462,5cyclictest131rcu_preempt00:00:011
207067499461,14cyclictest2105927-21ssh22:17:071
207068199455,8cyclictest37-21ksoftirqd/319:45:013
2070677994538,4cyclictest2074782-21kworker/2:120:50:282
2070677994536,7cyclictest291rcuc/219:35:592
207067499456,27cyclictest2075281-21munin-plugin-st20:39:591
2070674994528,13cyclictest275-21dbus-daemon21:30:011
2070674994517,20cyclictest275-21dbus-daemon20:45:011
2070671994527,14cyclictest2074484-21sshd20:21:150
207068199446,6cyclictest37-21ksoftirqd/300:20:543
207068199444,8cyclictest37-21ksoftirqd/320:30:013
207068199442,8cyclictest2075803-21munin-run20:50:003
207068199442,8cyclictest2075803-21munin-run20:50:003
207068199442,6cyclictest37-21ksoftirqd/320:05:023
207068199442,6cyclictest37-21ksoftirqd/320:05:013
2070681994424,9cyclictest37-21ksoftirqd/321:10:023
2070681994424,9cyclictest37-21ksoftirqd/321:10:013
207068199441,10cyclictest2071184-21sed19:20:013
2070677994437,4cyclictest2070868-21kworker/2:019:18:582
207067499442,15cyclictest2124004-21sh22:59:151
207067499442,10cyclictest2076842-21diskmemload22:49:211
207067499441,14cyclictest2136768-21ssh23:28:361
207067499441,14cyclictest2094006-21ssh21:49:481
207067199444,8cyclictest12-21ksoftirqd/020:30:010
207068199435,8cyclictest37-21ksoftirqd/319:55:003
207067799436,8cyclictest30-21ksoftirqd/220:45:512
207067799436,8cyclictest30-21ksoftirqd/220:45:512
2070674994332,7cyclictest23-21ksoftirqd/121:00:021
2070674994332,7cyclictest23-21ksoftirqd/121:00:021
207067499433,10cyclictest2076842-21diskmemload00:36:571
207067499433,10cyclictest2076842-21diskmemload00:36:571
2070674994328,9cyclictest309-21in:imuxsock19:35:011
2070674994326,12cyclictest0-21swapper/119:15:011
2070674994318,18cyclictest275-21dbus-daemon23:50:001
2070674994313,6cyclictest131rcu_preempt22:10:011
207067199437,4cyclictest131rcu_preempt19:35:010
207067199436,9cyclictest12-21ksoftirqd/019:50:010
207067199435,9cyclictest12-21ksoftirqd/020:20:000
2070671994327,12cyclictest2071075-21sshd19:15:350
2070671994327,11cyclictest275-21dbus-daemon21:05:000
207068199427,6cyclictest37-21ksoftirqd/319:35:023
207068199425,9cyclictest37-21ksoftirqd/320:25:003
2070681994232,8cyclictest37-21ksoftirqd/320:45:023
2070674994215,19cyclictest275-21dbus-daemon20:35:021
207067199424,7cyclictest12-21ksoftirqd/020:50:010
207067199424,7cyclictest12-21ksoftirqd/020:50:000
2070671994212,6cyclictest131rcu_preempt20:06:030
207068199414,8cyclictest37-21ksoftirqd/321:00:023
207068199414,8cyclictest37-21ksoftirqd/321:00:013
2070674994112,6cyclictest131rcu_preempt23:31:411
207067499411,13cyclictest2076842-21diskmemload21:51:301
207067199414,7cyclictest12-21ksoftirqd/019:40:140
207067199414,6cyclictest12-21ksoftirqd/020:15:010
2070671994132,6cyclictest151rcuc/020:04:040
2070671994132,6cyclictest151rcuc/020:04:040
2070671994128,9cyclictest2075047-21sed20:35:010
2070671994123,8cyclictest12-21ksoftirqd/020:50:270
207068199409,4cyclictest131rcu_preempt20:16:053
207068199404,9cyclictest37-21ksoftirqd/320:09:593
2070681994025,12cyclictest37-21ksoftirqd/319:40:013
207068199394,7cyclictest37-21ksoftirqd/319:20:543
2070681993912,6cyclictest131rcu_preempt20:54:013
207067799394,9cyclictest30-21ksoftirqd/219:47:422
207067499394,4cyclictest131rcu_preempt21:05:001
207067499393,27cyclictest2075805-21cut20:50:011
207067499393,27cyclictest2075805-21cut20:50:001
2070674993926,9cyclictest2083510-21latency_hist21:25:021
207067499383,24cyclictest284-21systemd-logind19:55:001
2070674993827,6cyclictest2076042-21cron20:55:001
207067499381,7cyclictest2074514-21munin-run20:25:001
2070674993815,15cyclictest275-21dbus-daemon19:45:011
207068199373,25cyclictest0-21swapper/320:15:023
2070681993725,9cyclictest2070766-21sed19:10:013
207067499374,5cyclictest131rcu_preempt19:25:001
207067499366,19cyclictest2074244-21cut20:20:011
207067499365,21cyclictest2073713-21modprobe20:10:001
207067499365,21cyclictest2072704-21latency_hist19:50:011
207067499363,27cyclictest2071665-21latency_hist19:30:021
2070681993522,9cyclictest1-21systemd20:40:003
207067199351,3cyclictest2070823-21sshd19:10:340
207068199344,6cyclictest37-21ksoftirqd/319:45:043
207068199341,6cyclictest37-21ksoftirqd/320:35:003
2070671993423,8cyclictest2070773-21sed19:10:020
2070674993322,9cyclictest23-21ksoftirqd/119:10:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional