You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 14:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sat Mar 21, 2026 00:45:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3916029912852,73cyclictest12-21ksoftirqd/019:50:020
391607996927,9cyclictest394398-21cron20:05:001
391607996927,9cyclictest394398-21cron20:05:001
391607996918,9cyclictest309-21in:imuxsock22:20:011
391616996819,9cyclictest37-21ksoftirqd/300:00:013
391611996827,33cyclictest311-21rs:main2
391616996717,10cyclictest37-21ksoftirqd/323:50:023
391607996723,11cyclictest23-21ksoftirqd/119:12:351
391611996541,8cyclictest398348-21kworker/2:123:07:132
391607996525,4cyclictest397843-21sh21:10:021
391607996518,15cyclictest1-21systemd00:25:001
391616996422,6cyclictest408485-21latency_hist21:35:023
391616996416,9cyclictest37-21ksoftirqd/322:20:003
391607996427,11cyclictest23-21ksoftirqd/123:36:551
391616996319,6cyclictest485042-21sh00:33:053
391616996315,11cyclictest37-21ksoftirqd/321:00:003
391616996313,12cyclictest37-21ksoftirqd/300:36:233
391607996341,16cyclictest3467602-21systemd-journal20:30:011
391607996323,5cyclictest397248-21sh20:59:591
391602996312,44cyclictest0-21swapper/023:20:000
391616996210,13cyclictest37-21ksoftirqd/323:05:193
391607996223,6cyclictest409217-21sh21:36:281
391616996116,7cyclictest37-21ksoftirqd/323:10:573
391616996114,9cyclictest37-21ksoftirqd/321:57:593
391607996122,9cyclictest23-21ksoftirqd/100:07:471
39161699605,14cyclictest393692-21cat19:50:023
391611996040,7cyclictest447435-21kworker/2:023:26:542
391607996028,10cyclictest23-21ksoftirqd/100:29:591
391616995913,9cyclictest37-21ksoftirqd/300:23:383
39160799594,14cyclictest453604-21modprobe23:19:591
391607995936,16cyclictest3467602-21systemd-journal23:55:001
391602995940,14cyclictest16950irq/224-800000000:40:000
391602995929,13cyclictest418723-21ssh21:58:490
39161699583,19cyclictest414956-21sed21:50:013
391616995812,11cyclictest37-21ksoftirqd/322:11:303
391611995814,6cyclictest131rcu_preempt22:14:212
391611995813,7cyclictest131rcu_preempt20:25:002
391607995827,5cyclictest439905-21sh22:48:031
391607995826,4cyclictest415382-21sh21:50:381
391607995826,4cyclictest415382-21sh21:50:381
391607995823,10cyclictest23-21ksoftirqd/100:38:101
391607995817,12cyclictest23-21ksoftirqd/120:20:021
39160299587,34cyclictest12-21ksoftirqd/023:59:340
391616995716,9cyclictest37-21ksoftirqd/300:17:003
391616995713,9cyclictest37-21ksoftirqd/321:21:163
391611995736,8cyclictest398348-21kworker/2:122:21:362
391607995728,11cyclictest23-21ksoftirqd/122:52:011
391607995726,10cyclictest23-21ksoftirqd/123:58:001
391607995724,8cyclictest23-21ksoftirqd/100:32:341
391602995728,14cyclictest444309-21ssh22:58:240
391602995716,36cyclictest16850irq/134-800000023:15:020
391616995611,9cyclictest37-21ksoftirqd/300:11:273
391616995611,9cyclictest37-21ksoftirqd/300:11:273
391611995646,7cyclictest477222-21kworker/2:000:37:152
391607995625,9cyclictest23-21ksoftirqd/122:04:201
391607995625,9cyclictest23-21ksoftirqd/122:04:201
391607995616,13cyclictest1-21systemd21:05:011
391602995628,13cyclictest441113-21ssh22:50:350
391616995513,7cyclictest37-21ksoftirqd/322:38:553
39161199557,10cyclictest30-21ksoftirqd/221:40:002
391607995521,9cyclictest23-21ksoftirqd/122:24:371
391602995529,22cyclictest423461-21latency_hist22:10:020
391602995525,14cyclictest406398-21ssh21:29:580
39161699547,9cyclictest37-21ksoftirqd/321:16:433
391616995412,8cyclictest0-21swapper/323:54:523
391611995434,15cyclictest275-21dbus-daemon00:35:012
391611995417,6cyclictest131rcu_preempt22:31:412
391611995414,5cyclictest131rcu_preempt00:11:042
391611995414,5cyclictest131rcu_preempt00:11:042
391611995411,6cyclictest131rcu_preempt21:00:022
391607995422,9cyclictest23-21ksoftirqd/119:45:011
391602995442,7cyclictest391610-21latency_hist19:10:010
391602995439,11cyclictest16950irq/224-800000022:20:020
391602995426,13cyclictest414003-21ssh21:47:410
39161699533,14cyclictest394558-21cat20:05:023
39161699533,14cyclictest394558-21cat20:05:023
391616995318,10cyclictest37-21ksoftirqd/323:43:423
391616995310,7cyclictest455795-21systemd-run23:25:013
391611995345,5cyclictest447435-21kworker/2:023:15:362
391611995344,6cyclictest462195-21kworker/2:123:52:532
391611995344,6cyclictest30-21ksoftirqd/221:18:202
391611995313,10cyclictest131rcu_preempt00:09:492
391611995312,6cyclictest131rcu_preempt22:40:002
391607995323,9cyclictest23-21ksoftirqd/123:12:261
391602995344,5cyclictest16950irq/224-800000022:30:000
39161699526,8cyclictest37-21ksoftirqd/323:32:443
391616995235,12cyclictest3467602-21systemd-journal22:25:023
391616995214,10cyclictest37-21ksoftirqd/322:55:043
391616995212,8cyclictest37-21ksoftirqd/321:53:443
391616995212,8cyclictest37-21ksoftirqd/321:53:443
39161199525,6cyclictest131rcu_preempt23:20:392
391611995245,4cyclictest477222-21kworker/2:000:20:492
391611995244,5cyclictest447435-21kworker/2:023:12:172
391611995244,5cyclictest396768-21kworker/2:022:48:222
391611995244,5cyclictest396768-21kworker/2:021:52:582
391611995244,5cyclictest396768-21kworker/2:021:52:582
39161199521,8cyclictest275-21dbus-daemon21:30:012
391607995220,9cyclictest413041-21kworker/u8:322:37:241
391602995241,7cyclictest16950irq/224-800000020:00:000
391602995225,23cyclictest400243-21ssh21:15:200
391602995225,12cyclictest448171-21ssh23:07:070
391602995221,4cyclictest4153011ssh21:50:260
391602995221,4cyclictest4153011ssh21:50:260
391602995220,29cyclictest12-21ksoftirqd/022:10:370
391602995214,33cyclictest16850irq/134-800000000:15:000
391602995214,33cyclictest16850irq/134-800000000:15:000
391602995211,20cyclictest131rcu_preempt23:25:100
39161699516,11cyclictest37-21ksoftirqd/322:40:203
39161699516,11cyclictest37-21ksoftirqd/322:40:203
39161699515,14cyclictest456498-21ssh23:26:253
391616995115,7cyclictest37-21ksoftirqd/323:37:593
391616995112,8cyclictest37-21ksoftirqd/322:33:053
391616995110,8cyclictest37-21ksoftirqd/323:00:033
391616995110,8cyclictest37-21ksoftirqd/323:00:033
391611995143,5cyclictest447435-21kworker/2:023:43:582
391611995143,5cyclictest447435-21kworker/2:023:38:592
391611995143,5cyclictest447435-21kworker/2:023:30:402
391611995143,5cyclictest398348-21kworker/2:122:19:542
391611995143,5cyclictest396768-21kworker/2:021:11:532
391611995143,5cyclictest396768-21kworker/2:021:09:102
391611995143,5cyclictest396768-21kworker/2:021:01:122
391611995142,6cyclictest462195-21kworker/2:100:00:472
391611995142,6cyclictest398348-21kworker/2:121:55:412
391611995142,5cyclictest396768-21kworker/2:021:42:562
391611995141,6cyclictest447435-21kworker/2:023:47:422
391611995113,5cyclictest131rcu_preempt22:43:192
391611995113,5cyclictest131rcu_preempt22:43:192
39160299513,9cyclictest12-21ksoftirqd/023:52:240
391602995132,5cyclictest16850irq/134-800000022:30:030
391602995126,21cyclictest484194-21ssh00:30:540
391602995122,13cyclictest403777-21ssh21:23:420
39161699504,11cyclictest37-21ksoftirqd/320:20:013
39161699502,14cyclictest394230-21cut20:00:023
391616995013,9cyclictest37-21ksoftirqd/322:49:503
391616995010,9cyclictest37-21ksoftirqd/300:06:323
391611995043,4cyclictest398348-21kworker/2:121:20:042
391611995043,4cyclictest391372-21kworker/2:219:59:502
391611995042,5cyclictest462195-21kworker/2:100:16:332
391611995042,5cyclictest398348-21kworker/2:123:00:332
391611995042,5cyclictest398348-21kworker/2:123:00:332
391611995042,5cyclictest398348-21kworker/2:122:58:122
391611995042,5cyclictest398348-21kworker/2:122:01:252
391611995042,5cyclictest398348-21kworker/2:122:01:252
391611995042,5cyclictest396768-21kworker/2:022:08:392
391611995041,5cyclictest398348-21kworker/2:122:27:342
391611995041,5cyclictest396768-21kworker/2:021:45:392
391611995032,13cyclictest275-21dbus-daemon22:55:012
391602995032,14cyclictest473618-21ssh00:06:280
391602995032,12cyclictest275-21dbus-daemon23:25:010
39160299502,41cyclictest479313-21munin-run00:20:000
391602995014,32cyclictest16850irq/134-800000000:25:010
39161699497,9cyclictest37-21ksoftirqd/322:28:283
39161699495,14cyclictest391915-21latency_hist19:15:013
391616994916,7cyclictest37-21ksoftirqd/321:13:553
39161699490,18cyclictest396075-21latency_hist20:35:023
391611994942,5cyclictest398348-21kworker/2:121:34:212
391611994941,5cyclictest477222-21kworker/2:000:28:502
391602994924,10cyclictest472501-21ssh00:03:590
39160299491,8cyclictest12-21ksoftirqd/022:37:440
39160299491,4cyclictest131rcu_preempt21:30:290
391611994841,4cyclictest394169-21kworker/2:320:36:052
391611994840,5cyclictest394169-21kworker/2:320:50:182
391611994840,5cyclictest394169-21kworker/2:320:26:392
391611994839,5cyclictest391372-21kworker/2:219:16:562
391611994833,9cyclictest30-21ksoftirqd/219:55:002
391611994814,5cyclictest131rcu_preempt19:10:012
391611994812,6cyclictest131rcu_preempt20:20:022
391607994838,8cyclictest23-21ksoftirqd/119:25:021
391607994830,13cyclictest402080-21cron21:20:001
391607994818,7cyclictest131rcu_preempt23:30:001
39160299488,20cyclictest131rcu_preempt00:27:420
391602994830,13cyclictest447149-21ssh23:05:020
391602994830,13cyclictest447149-21ssh23:05:010
391602994820,24cyclictest12-21ksoftirqd/021:10:090
39160299481,9cyclictest12-21ksoftirqd/023:46:500
39161699475,8cyclictest37-21ksoftirqd/321:35:413
39161699475,12cyclictest37-21ksoftirqd/319:45:023
391616994713,5cyclictest37-21ksoftirqd/322:02:553
391616994713,5cyclictest37-21ksoftirqd/322:02:553
391611994739,5cyclictest394169-21kworker/2:320:45:112
391611994739,5cyclictest394169-21kworker/2:320:34:232
391611994739,5cyclictest391372-21kworker/2:219:45:182
391611994739,5cyclictest391372-21kworker/2:219:42:362
391611994739,5cyclictest391372-21kworker/2:219:30:432
391607994731,10cyclictest23-21ksoftirqd/120:45:021
391607994731,10cyclictest23-21ksoftirqd/120:45:021
39160299472,41cyclictest411448-21ssh21:41:400
391602994721,11cyclictest460357-21ssh23:35:200
39160299471,41cyclictest4384291ssh22:44:470
39160299471,41cyclictest4384291ssh22:44:460
39161699469,7cyclictest394786-21cat20:10:013
39161699466,8cyclictest37-21ksoftirqd/320:40:023
39161699461,9cyclictest423342-21ssh22:09:433
391616994611,9cyclictest0-21swapper/300:01:163
391611994639,4cyclictest391372-21kworker/2:220:00:212
391611994639,4cyclictest391372-21kworker/2:220:00:212
391611994639,4cyclictest391372-21kworker/2:219:24:162
391611994639,4cyclictest391372-21kworker/2:219:10:062
391611994638,5cyclictest394169-21kworker/2:320:40:242
391611994638,5cyclictest394169-21kworker/2:320:40:242
391611994638,5cyclictest394169-21kworker/2:320:12:042
391611994638,5cyclictest391372-21kworker/2:220:06:012
391611994638,5cyclictest391372-21kworker/2:219:25:102
391607994633,10cyclictest479122-21sshd00:19:351
39160299469,6cyclictest131rcu_preempt22:22:430
391602994627,15cyclictest458193-21ssh23:30:130
391602994625,17cyclictest410561-21ssh21:40:000
39160299461,3cyclictest131rcu_preempt22:04:350
39160299461,3cyclictest131rcu_preempt22:04:350
39161699455,5cyclictest0-21swapper/321:05:023
391616994512,7cyclictest0-21swapper/300:25:463
391611994538,4cyclictest391372-21kworker/2:219:36:302
391602994531,11cyclictest463482-21ssh23:42:530
391602994515,27cyclictest12-21ksoftirqd/022:46:450
391616994430,11cyclictest309-21in:imuxsock19:28:343
391616994423,15cyclictest275-21dbus-daemon21:10:013
391616994410,7cyclictest0-21swapper/323:15:553
39160799441,3cyclictest131rcu_preempt21:35:021
391607994410,6cyclictest131rcu_preempt20:35:001
391602994436,5cyclictest0-21swapper/019:30:010
39161699436,7cyclictest37-21ksoftirqd/322:54:363
39161699434,11cyclictest37-21ksoftirqd/319:35:033
391616994327,10cyclictest275-21dbus-daemon19:40:013
391616994311,5cyclictest0-21swapper/321:28:053
391602994334,7cyclictest12-21ksoftirqd/020:30:020
391616994231,9cyclictest37-21ksoftirqd/320:15:023
391616994227,10cyclictest275-21dbus-daemon19:25:003
39161699422,6cyclictest37-21ksoftirqd/321:40:053
391616994211,5cyclictest0-21swapper/319:20:013
391607994224,12cyclictest3467602-21systemd-journal23:05:011
391607994224,12cyclictest3467602-21systemd-journal23:05:011
391607994221,15cyclictest1-21systemd23:25:021
391602994235,4cyclictest16950irq/224-800000021:10:010
391602994225,13cyclictest395938-21sshd20:33:350
391602994225,13cyclictest395429-21sshd20:22:350
39160299421,37cyclictest0-21swapper/020:10:000
391607994130,9cyclictest23-21ksoftirqd/123:09:561
39160299419,7cyclictest131rcu_preempt20:50:000
391602994134,4cyclictest16950irq/224-800000019:45:000
391602994127,9cyclictest0-21swapper/021:05:020
391602994127,11cyclictest3467602-21systemd-journal20:57:320
391607994029,8cyclictest414889-21cron21:50:001
391607994029,8cyclictest275-21dbus-daemon19:20:011
391607994026,11cyclictest23-21ksoftirqd/100:02:371
391607994023,11cyclictest309-21in:imuxsock19:45:021
391616993927,8cyclictest37-21ksoftirqd/320:55:023
391616993925,10cyclictest1-21systemd19:55:003
39160799393,6cyclictest131rcu_preempt23:50:021
391607993928,5cyclictest274-21cron20:40:001
391607993926,10cyclictest23-21ksoftirqd/100:12:101
391607993926,10cyclictest23-21ksoftirqd/100:12:101
391607993925,10cyclictest3467602-21systemd-journal23:00:021
391607993925,10cyclictest23-21ksoftirqd/122:33:541
391602993931,5cyclictest12-21ksoftirqd/019:17:230
39160299392,5cyclictest395217-21munin-run20:20:010
391602993924,12cyclictest393596-21sshd19:46:350
391616993824,10cyclictest37-21ksoftirqd/320:45:033
391616993824,10cyclictest37-21ksoftirqd/320:45:033
391616993823,11cyclictest37-21ksoftirqd/320:50:023
39160799387,6cyclictest131rcu_preempt21:23:221
39160799385,22cyclictest412819-21kworker/1:022:05:031
391607993828,8cyclictest23-21ksoftirqd/122:40:471
391607993828,8cyclictest23-21ksoftirqd/122:40:471
391607993826,9cyclictest23-21ksoftirqd/120:55:001
391607993821,13cyclictest275-21dbus-daemon23:45:011
391602993829,6cyclictest19350irq/34-mmc120:45:020
391602993829,6cyclictest19350irq/34-mmc120:45:020
391602993825,10cyclictest12-21ksoftirqd/020:12:180
391602993824,8cyclictest3467602-21systemd-journal19:39:590
39161699371,32cyclictest37-21ksoftirqd/320:30:023
39160799374,4cyclictest131rcu_preempt19:55:001
391607993725,9cyclictest23-21ksoftirqd/122:26:061
391607993725,9cyclictest23-21ksoftirqd/121:59:441
391607993722,10cyclictest309-21in:imuxsock21:40:021
391602993724,8cyclictest12-21ksoftirqd/019:25:010
39160799367,4cyclictest131rcu_preempt19:40:001
391607993626,8cyclictest23-21ksoftirqd/121:26:351
39160299362,30cyclictest396955-21modprobe20:50:030
39160799354,3cyclictest131rcu_preempt20:50:021
39160799354,21cyclictest394123-21modprobe20:00:001
391607993524,9cyclictest23-21ksoftirqd/122:10:061
39160299352,5cyclictest3467602-21systemd-journal19:15:010
391602993525,7cyclictest394617-21cut20:05:030
391602993525,5cyclictest309-21in:imuxsock19:35:010
391602993513,14cyclictest131rcu_preempt20:38:250
39160799348,5cyclictest131rcu_preempt21:10:551
391607993426,5cyclictest394835-21latency_hist20:10:011
391607993425,7cyclictest23-21ksoftirqd/123:34:301
391602993428,4cyclictest0-21swapper/020:03:390
391616993326,4cyclictest395451-21latency_hist20:25:023
391607993325,5cyclictest391543-21latency_hist19:10:021
391607993320,10cyclictest23-21ksoftirqd/119:35:021
391607993318,11cyclictest275-21dbus-daemon20:15:001
391607993221,7cyclictest309-21in:imuxsock19:30:011
39161699318,19cyclictest391724-21latency_hist19:10:023
391607993022,5cyclictest395553-21latency_hist20:25:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional