You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-29 - 18:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Wed Jul 29, 2026 12:45:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
41802029911020,14cyclictest131rcu_preempt10:05:031
418021199817,9cyclictest37-21ksoftirqd/308:20:013
418019799714,6cyclictest131rcu_preempt11:20:030
4180197997132,18cyclictest51200-21ssh11:14:020
4180197997132,18cyclictest51200-21ssh11:14:020
418019799694,6cyclictest131rcu_preempt12:34:130
418021199685,10cyclictest37-21ksoftirqd/307:25:023
418021199674,9cyclictest37-21ksoftirqd/312:15:113
4180206996711,6cyclictest131rcu_preempt11:39:592
418019799679,47cyclictest12-21ksoftirqd/009:11:350
418021199651,13cyclictest37-21ksoftirqd/309:35:003
418019799646,5cyclictest131rcu_preempt12:14:110
418019799642,40cyclictest131rcu_preempt09:20:020
418019799642,40cyclictest131rcu_preempt09:20:020
418019799637,5cyclictest131rcu_preempt11:59:500
4180197996315,6cyclictest131rcu_preempt10:26:510
4180197996314,43cyclictest131rcu_preempt10:11:070
418021199628,10cyclictest37-21ksoftirqd/309:25:003
418021199625,11cyclictest37-21ksoftirqd/311:50:023
4180211996218,11cyclictest37-21ksoftirqd/310:00:003
4180211996214,5cyclictest131rcu_preempt11:38:583
4180197996232,21cyclictest12-21ksoftirqd/010:00:010
4180211996119,12cyclictest37-21ksoftirqd/310:15:033
4180211996117,6cyclictest131rcu_preempt12:40:003
4180211996117,6cyclictest131rcu_preempt12:40:003
4180211996117,6cyclictest131rcu_preempt12:10:003
4180211996114,6cyclictest131rcu_preempt10:10:003
4180211996114,13cyclictest131rcu_preempt09:29:243
4180211996114,13cyclictest131rcu_preempt09:29:233
4180211996112,5cyclictest131rcu_preempt12:00:003
4180211996111,6cyclictest131rcu_preempt10:51:163
4180211996111,6cyclictest131rcu_preempt10:51:153
4180211996110,6cyclictest131rcu_preempt10:43:323
4180211996110,6cyclictest131rcu_preempt10:43:323
4180206996132,12cyclictest30-21ksoftirqd/212:20:002
4180206996122,8cyclictest30-21ksoftirqd/207:22:112
4180197996135,21cyclictest66462-21ssh11:45:290
418021199607,6cyclictest131rcu_preempt11:10:013
418021199605,10cyclictest37-21ksoftirqd/312:15:013
4180211996040,17cyclictest37-21ksoftirqd/309:05:013
418020699604,6cyclictest131rcu_preempt12:05:012
4180206996029,14cyclictest30-21ksoftirqd/210:05:242
4180206996011,7cyclictest131rcu_preempt11:25:032
418019799604,7cyclictest12-21ksoftirqd/009:10:000
4180197996043,14cyclictest16950irq/224-800000007:15:010
4180197996032,15cyclictest12-21ksoftirqd/009:24:130
4180197996024,16cyclictest12-21ksoftirqd/012:10:010
418021199598,8cyclictest37-21ksoftirqd/311:35:013
4180211995914,6cyclictest131rcu_preempt11:23:373
4180211995910,5cyclictest131rcu_preempt12:01:533
4180206995945,6cyclictest80321-21kworker/2:312:38:402
4180206995945,6cyclictest80321-21kworker/2:312:38:392
418021199585,10cyclictest37-21ksoftirqd/308:00:003
418021199585,10cyclictest37-21ksoftirqd/308:00:003
4180211995824,11cyclictest37-21ksoftirqd/310:17:383
4180211995824,11cyclictest37-21ksoftirqd/310:17:383
4180211995816,6cyclictest131rcu_preempt12:26:393
4180211995811,6cyclictest131rcu_preempt09:45:113
418021199581,11cyclictest37-21ksoftirqd/310:40:013
4180206995812,6cyclictest131rcu_preempt09:38:522
4180206995810,6cyclictest131rcu_preempt11:35:012
4180206995810,6cyclictest131rcu_preempt09:55:012
4180206995810,6cyclictest131rcu_preempt09:55:012
418019799588,6cyclictest131rcu_preempt12:01:110
4180197995848,6cyclictest4301-21cron09:35:000
4180197995830,19cyclictest4186383-21diskmemload09:47:400
418021199578,9cyclictest37-21ksoftirqd/311:00:283
418021199578,9cyclictest37-21ksoftirqd/311:00:283
418021199574,10cyclictest37-21ksoftirqd/310:50:003
418021199572,11cyclictest37-21ksoftirqd/310:26:233
4180211995717,8cyclictest131rcu_preempt11:42:103
4180211995715,5cyclictest131rcu_preempt11:26:423
4180211995714,5cyclictest131rcu_preempt12:34:433
4180211995713,5cyclictest131rcu_preempt11:10:223
4180211995713,5cyclictest131rcu_preempt11:10:213
4180211995712,5cyclictest131rcu_preempt11:17:033
4180211995711,7cyclictest131rcu_preempt09:17:023
4180211995711,7cyclictest131rcu_preempt09:17:023
418019799576,8cyclictest121ksoftirqd/011:08:270
418019799575,5cyclictest131rcu_preempt10:20:440
418019799571,40cyclictest4691-21ssh09:35:200
4180197995713,11cyclictest12-21ksoftirqd/011:45:010
418021199567,9cyclictest37-21ksoftirqd/310:58:103
418021199567,8cyclictest37-21ksoftirqd/309:35:133
4180211995613,5cyclictest131rcu_preempt12:20:273
4180211995611,5cyclictest131rcu_preempt10:22:023
4180211995610,6cyclictest131rcu_preempt10:01:303
4180211995610,6cyclictest131rcu_preempt09:40:293
418019799563,4cyclictest131rcu_preempt10:05:290
4180197995631,9cyclictest12-21ksoftirqd/009:44:470
4180197995628,13cyclictest82042-21ssh12:18:310
4180197995615,35cyclictest131rcu_preempt11:01:110
4180197995615,35cyclictest131rcu_preempt11:01:100
418021199555,10cyclictest37-21ksoftirqd/307:55:003
4180211995514,6cyclictest131rcu_preempt09:14:153
418020699554,6cyclictest131rcu_preempt10:17:112
418020699554,6cyclictest131rcu_preempt10:17:102
4180206995546,6cyclictest51854-21kworker/2:011:18:422
4180206995511,6cyclictest131rcu_preempt10:00:002
4180206995510,5cyclictest131rcu_preempt09:10:012
418020299558,43cyclictest11666-21ssh09:50:021
418019799558,32cyclictest131rcu_preempt09:52:240
418019799558,32cyclictest131rcu_preempt09:52:240
4180197995527,13cyclictest60114-21ssh11:32:350
4180197995519,32cyclictest16850irq/134-800000010:42:270
4180197995519,32cyclictest16850irq/134-800000010:42:260
4180197995519,19cyclictest16850irq/134-800000010:51:120
4180197995519,19cyclictest16850irq/134-800000010:51:110
4180197995518,32cyclictest16850irq/134-800000010:03:030
4180197995515,35cyclictest16850irq/134-800000010:59:590
418019799551,38cyclictest799-21ssh09:27:260
418019799551,38cyclictest799-21ssh09:27:260
4180197995512,5cyclictest131rcu_preempt08:15:010
418021199549,5cyclictest131rcu_preempt08:50:023
4180211995413,8cyclictest37-21ksoftirqd/309:52:243
4180211995413,8cyclictest37-21ksoftirqd/309:52:233
4180211995411,5cyclictest131rcu_preempt08:25:023
4180211995411,5cyclictest131rcu_preempt08:25:023
4180211995411,5cyclictest131rcu_preempt07:50:013
4180211995411,4cyclictest131rcu_preempt11:50:233
4180206995436,12cyclictest275-21dbus-daemon11:05:012
4180206995436,12cyclictest275-21dbus-daemon11:05:012
418020699541,8cyclictest30-21ksoftirqd/209:30:002
418020699541,8cyclictest30-21ksoftirqd/209:30:002
4180206995410,6cyclictest131rcu_preempt07:20:012
4180197995425,14cyclictest30784-21ssh10:30:050
4180197995421,29cyclictest54653-21sh11:20:470
4180197995410,6cyclictest131rcu_preempt11:39:590
418021199538,5cyclictest131rcu_preempt07:35:023
418021199538,5cyclictest131rcu_preempt07:15:023
418021199537,6cyclictest131rcu_preempt08:55:013
418021199534,10cyclictest37-21ksoftirqd/308:40:033
418021199532,7cyclictest3467602-21systemd-journal09:10:003
4180211995311,5cyclictest131rcu_preempt08:15:023
418021199531,14cyclictest37-21ksoftirqd/310:30:143
4180211995310,6cyclictest131rcu_preempt07:45:023
4180211995310,6cyclictest131rcu_preempt07:45:023
4180211995310,5cyclictest131rcu_preempt08:30:013
418020699539,6cyclictest131rcu_preempt11:44:152
418020699538,5cyclictest131rcu_preempt11:52:402
4180206995344,6cyclictest4184588-21kworker/2:010:26:172
4180206995339,9cyclictest30-21ksoftirqd/208:35:012
4180206995339,6cyclictest4184022-21kworker/2:211:06:382
4180206995338,6cyclictest4184022-21kworker/2:211:23:222
418020699533,6cyclictest131rcu_preempt10:42:352
418020699533,6cyclictest131rcu_preempt10:42:342
4180206995323,10cyclictest30-21ksoftirqd/210:35:022
4180206995315,9cyclictest30-21ksoftirqd/212:10:092
4180206995313,6cyclictest131rcu_preempt12:00:012
4180206995313,6cyclictest131rcu_preempt10:40:002
4180206995311,5cyclictest131rcu_preempt10:50:012
4180206995310,10cyclictest30-21ksoftirqd/210:52:112
4180206995310,10cyclictest30-21ksoftirqd/210:52:102
4180197995324,14cyclictest58139-21ssh11:28:250
4180197995319,19cyclictest16850irq/134-800000010:36:000
4180197995318,20cyclictest16850irq/134-800000012:36:040
4180197995318,20cyclictest16850irq/134-800000012:36:030
4180197995317,32cyclictest16850irq/134-800000010:15:320
4180197995317,32cyclictest16850irq/134-800000010:15:310
4180197995312,32cyclictest16850irq/134-800000011:51:430
4180197995310,6cyclictest131rcu_preempt07:55:010
418021199525,8cyclictest37-21ksoftirqd/307:25:113
418021199525,7cyclictest37-21ksoftirqd/307:17:013
4180211995211,5cyclictest131rcu_preempt08:35:023
4180206995244,5cyclictest70848-21kworker/2:012:09:162
4180206995244,5cyclictest4184588-21kworker/2:009:49:062
4180206995244,5cyclictest4184022-21kworker/2:211:47:422
4180206995244,5cyclictest4184022-21kworker/2:211:10:242
4180206995244,5cyclictest4184022-21kworker/2:211:10:242
4180206995244,5cyclictest4184022-21kworker/2:209:11:052
4180206995242,6cyclictest80321-21kworker/2:312:24:512
4180206995227,10cyclictest30-21ksoftirqd/212:30:072
4180206995221,24cyclictest291rcuc/207:10:022
4180206995221,24cyclictest291rcuc/207:10:022
4180206995215,7cyclictest131rcu_preempt10:20:032
418019799527,9cyclictest12-21ksoftirqd/008:00:000
418019799527,9cyclictest12-21ksoftirqd/008:00:000
418019799524,5cyclictest131rcu_preempt07:20:010
4180197995217,31cyclictest16850irq/134-800000012:20:150
4180197995216,7cyclictest12-21ksoftirqd/007:30:030
418019799521,47cyclictest399731ssh10:49:540
4180197995210,6cyclictest131rcu_preempt09:00:030
4180197995210,6cyclictest131rcu_preempt09:00:020
418021199518,10cyclictest37-21ksoftirqd/308:45:003
4180211995110,6cyclictest131rcu_preempt07:40:003
4180211995110,5cyclictest131rcu_preempt08:55:103
4180211995110,5cyclictest131rcu_preempt08:55:103
418020699518,6cyclictest131rcu_preempt09:35:002
418020699518,6cyclictest131rcu_preempt08:20:002
418020699515,6cyclictest131rcu_preempt09:41:142
4180206995139,8cyclictest4184588-21kworker/2:010:00:242
4180206995114,8cyclictest131rcu_preempt10:10:522
4180202995132,14cyclictest68403-21cron11:50:001
418019799511,5cyclictest131rcu_preempt12:30:010
4180197995111,6cyclictest131rcu_preempt08:55:000
4180206995042,5cyclictest4184022-21kworker/2:209:18:042
4180206995042,5cyclictest4184022-21kworker/2:209:18:042
4180206995042,5cyclictest40062-21kworker/2:110:56:352
4180206995041,6cyclictest4167828-21kworker/2:107:49:292
418020699503,10cyclictest30-21ksoftirqd/208:40:012
4180206995011,6cyclictest131rcu_preempt08:05:002
4180206995010,6cyclictest131rcu_preempt09:20:242
4180202995034,12cyclictest1-21systemd11:15:011
4180202995034,12cyclictest1-21systemd11:15:001
4180202995020,8cyclictest221rcuc/108:55:031
4180211994920,19cyclictest361rcuc/308:05:033
4180206994941,5cyclictest70848-21kworker/2:012:25:522
4180206994941,5cyclictest4184022-21kworker/2:208:55:102
4180206994941,5cyclictest4184022-21kworker/2:208:55:102
4180206994941,5cyclictest4167828-21kworker/2:107:28:242
4180206994940,6cyclictest4167828-21kworker/2:107:59:552
4180206994940,6cyclictest4167828-21kworker/2:107:59:542
418020699491,5cyclictest131rcu_preempt07:15:012
4180202994933,12cyclictest4186409-21systemd-run09:10:001
4180197994921,11cyclictest12-21ksoftirqd/007:40:000
4180206994841,4cyclictest4184022-21kworker/2:208:45:242
4180206994840,5cyclictest4167828-21kworker/2:108:26:122
4180206994840,5cyclictest4167828-21kworker/2:108:21:112
4180206994840,5cyclictest4167828-21kworker/2:108:21:112
4180206994840,5cyclictest4167828-21kworker/2:108:13:352
4180206994839,6cyclictest4167828-21kworker/2:107:42:552
4180206994839,6cyclictest4167828-21kworker/2:107:42:552
4180206994839,6cyclictest4159881-21kworker/2:008:07:332
4180206994832,9cyclictest30-21ksoftirqd/208:55:012
418019799486,5cyclictest131rcu_preempt09:05:010
4180197994810,6cyclictest131rcu_preempt08:45:010
4180197994810,5cyclictest131rcu_preempt08:10:020
418021199476,8cyclictest37-21ksoftirqd/308:10:003
4180206994739,5cyclictest4184022-21kworker/2:208:40:102
4180206994739,5cyclictest4167828-21kworker/2:107:31:352
4180206994728,9cyclictest30-21ksoftirqd/207:40:022
4180202994736,7cyclictest63631-21sh11:40:001
418020299473,3cyclictest131rcu_preempt10:00:011
418020299473,27cyclictest221rcuc/110:20:021
418020299473,27cyclictest221rcuc/110:20:021
4180202994722,16cyclictest275-21dbus-daemon07:30:011
418019799478,6cyclictest131rcu_preempt08:50:020
4180211994612,10cyclictest37-21ksoftirqd/307:10:023
4180211994612,10cyclictest37-21ksoftirqd/307:10:023
4180206994638,5cyclictest4184022-21kworker/2:209:00:482
4180206994638,5cyclictest4167828-21kworker/2:107:51:152
418020299464,3cyclictest131rcu_preempt12:03:201
418020299464,3cyclictest131rcu_preempt11:58:161
4180202994633,10cyclictest23-21ksoftirqd/111:44:591
4180202994629,14cyclictest21280-21sshd10:10:101
418019799469,5cyclictest131rcu_preempt08:25:000
418019799469,5cyclictest131rcu_preempt08:25:000
4180197994626,15cyclictest275-21dbus-daemon08:30:000
418020299456,7cyclictest23-21ksoftirqd/112:19:061
4180202994534,6cyclictest49559-21cut11:09:591
418020299453,3cyclictest131rcu_preempt12:34:471
4180202994531,10cyclictest275-21dbus-daemon12:40:011
4180202994531,10cyclictest275-21dbus-daemon12:40:001
4180202994527,13cyclictest3467602-21systemd-journal09:28:401
4180202994527,13cyclictest3467602-21systemd-journal09:28:391
418020299451,4cyclictest131rcu_preempt11:30:011
418019799459,5cyclictest131rcu_preempt07:25:030
418019799453,6cyclictest131rcu_preempt08:18:280
418020299443,3cyclictest131rcu_preempt12:12:121
418020299443,3cyclictest131rcu_preempt09:53:511
418020299443,3cyclictest131rcu_preempt09:53:511
418019799447,7cyclictest12-21ksoftirqd/008:05:010
4180197994421,19cyclictest4184818-21latency_hist08:40:020
4180197994412,5cyclictest131rcu_preempt07:35:010
418019799441,11cyclictest12-21ksoftirqd/008:35:010
418020299433,3cyclictest131rcu_preempt10:41:091
418020299433,3cyclictest131rcu_preempt10:41:091
418019799436,6cyclictest131rcu_preempt07:45:010
418019799436,6cyclictest131rcu_preempt07:45:010
418019799434,5cyclictest131rcu_preempt07:50:000
4180202994231,8cyclictest23-21ksoftirqd/109:19:011
4180202994231,8cyclictest23-21ksoftirqd/109:19:001
4180202994228,11cyclictest23-21ksoftirqd/110:51:201
4180202994228,11cyclictest23-21ksoftirqd/110:51:201
4180202994223,13cyclictest275-21dbus-daemon08:15:011
418020299421,3cyclictest131rcu_preempt11:35:021
418020299419,7cyclictest131rcu_preempt09:31:481
418020299413,3cyclictest131rcu_preempt09:10:291
4180202994129,7cyclictest23-21ksoftirqd/110:21:371
4180202994126,10cyclictest23-21ksoftirqd/111:04:111
4180202994126,10cyclictest23-21ksoftirqd/111:04:111
4180202994123,14cyclictest275-21dbus-daemon09:40:021
418020299411,3cyclictest131rcu_preempt08:40:011
4180202994110,6cyclictest131rcu_preempt07:30:391
4180202994029,8cyclictest3467602-21systemd-journal10:38:111
4180202994027,10cyclictest23-21ksoftirqd/112:07:441
4180202994027,10cyclictest23-21ksoftirqd/109:44:531
4180202994024,12cyclictest275-21dbus-daemon11:20:001
418020299402,3cyclictest131rcu_preempt11:51:071
4180202994022,12cyclictest3467602-21systemd-journal08:29:591
418020299401,4cyclictest131rcu_preempt11:20:301
418020299401,3cyclictest131rcu_preempt10:59:141
418020299401,3cyclictest131rcu_preempt10:45:331
418019799406,5cyclictest131rcu_preempt07:10:020
4180202993922,12cyclictest275-21dbus-daemon07:20:011
418020299391,3cyclictest131rcu_preempt12:25:021
418020299391,3cyclictest131rcu_preempt10:09:481
4180202993827,8cyclictest23-21ksoftirqd/110:30:591
4180202993826,9cyclictest23-21ksoftirqd/108:00:011
4180202993826,9cyclictest23-21ksoftirqd/108:00:001
4180202993824,10cyclictest23-21ksoftirqd/108:45:001
4180202993823,9cyclictest275-21dbus-daemon08:05:001
418020299381,5cyclictest131rcu_preempt07:15:011
418020299381,3cyclictest131rcu_preempt12:24:321
418020299381,3cyclictest131rcu_preempt08:35:011
418019799385,28cyclictest4180321-21grep07:10:020
418020299375,3cyclictest131rcu_preempt09:00:001
418020299375,3cyclictest131rcu_preempt09:00:001
4180202993724,7cyclictest309-21in:imuxsock09:24:591
418020299372,3cyclictest131rcu_preempt07:25:011
4180202993627,6cyclictest30032-21sshd10:28:491
418020299362,4cyclictest150-21pr/ttyS207:55:001
418020299358,5cyclictest131rcu_preempt08:20:491
418020299358,5cyclictest131rcu_preempt08:20:491
418020299355,3cyclictest131rcu_preempt07:10:031
418020299354,5cyclictest131rcu_preempt07:40:001
418020299353,6cyclictest131rcu_preempt08:10:001
418020299353,3cyclictest131rcu_preempt08:50:011
4180202993528,4cyclictest4182386-21latency_hist07:50:011
4180202993524,9cyclictest23-21ksoftirqd/109:05:011
4180202993426,5cyclictest4181971-21latency_hist07:45:031
4180202993426,5cyclictest4181971-21latency_hist07:45:021
418020299337,5cyclictest131rcu_preempt08:15:151
4180202992921,5cyclictest4180292-21latency_hist07:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional