You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-09 - 08:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Thu Jul 09, 2026 00:45:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1658838999527,34cyclictest1733186-21sh23:32:362
1658833998661,17cyclictest23-21ksoftirqd/121:40:021
165884399812,59cyclictest1689087-21ssh22:00:023
165883399802,74cyclictest1665139-21ssh21:10:021
165883899741,14cyclictest168155350ssh21:44:412
1658838996829,33cyclictest1755984-21sed00:20:022
1658838996826,36cyclictest1694590-21sh22:11:392
1658838996820,43cyclictest1759770-21sh00:28:072
165883899672,59cyclictest1727063-21cron23:20:012
165883899672,59cyclictest1727063-21cron23:20:002
165883399674,47cyclictest1724849-21ssh23:15:021
165883399671,3cyclictest131rcu_preempt23:28:541
165883399671,3cyclictest131rcu_preempt23:28:541
1658838996621,40cyclictest1736343-21sh23:39:222
165883899661,18cyclictest1745162-21sh23:57:342
165883899661,15cyclictest1756186-21ssh00:20:222
165883899661,15cyclictest1756186-21ssh00:20:212
1658838996526,35cyclictest1702772-21sh22:29:032
1658838996526,35cyclictest1702772-21sh22:29:032
165883899651,13cyclictest1668903-21sshd21:18:162
165883899651,10cyclictest1728030-21sshd23:21:322
1658831996521,39cyclictest16850irq/134-800000023:05:000
165883899632,19cyclictest1722026-21ssh23:09:242
165883899632,19cyclictest1722026-21ssh23:09:242
165883899632,15cyclictest1761823-21ssh00:32:292
165883899632,15cyclictest1761823-21ssh00:32:282
165883899631,9cyclictest1704502-21ssh22:32:302
165883899631,4cyclictest1683247-21ssh21:48:022
165883899631,17cyclictest1697832-21ssh22:18:372
165883899631,17cyclictest1697832-21ssh22:18:372
1658838996250,6cyclictest1665036-21kworker/2:222:00:002
165883899623,53cyclictest1-21systemd19:45:012
1658838996223,34cyclictest30-21ksoftirqd/222:06:552
1658838996222,36cyclictest1689466-21sh22:00:452
165883899622,18cyclictest1708879-21ssh22:41:382
165883899622,18cyclictest1-21systemd20:50:012
165883899622,18cyclictest1-21systemd20:50:012
165883899621,9cyclictest1712135-21ssh22:48:352
165883899621,9cyclictest1712135-21ssh22:48:342
165883899621,4cyclictest1707058-21ssh22:37:552
165883899621,4cyclictest1707058-21ssh22:37:552
165883899621,17cyclictest1673030-21ssh21:26:402
165883899621,16cyclictest1750788-21ssh00:09:302
165883199628,7cyclictest121ksoftirqd/000:13:310
1658831996249,9cyclictest1707950-21tr22:40:000
1658831996249,9cyclictest1707950-21tr22:40:000
1658831996233,9cyclictest12-21ksoftirqd/022:32:530
165884399612,11cyclictest37-21ksoftirqd/323:50:013
165883899612,16cyclictest1739291-21ssh23:45:052
165883899611,9cyclictest1699731-21ssh22:22:282
165883899611,9cyclictest1678513-21ssh21:38:202
165883899611,8cyclictest1713121-21ssh22:50:292
165883899611,8cyclictest1685058-21ssh21:51:402
165883899611,7cyclictest1742901-21sshd23:52:592
165883899611,16cyclictest1665710-21ssh21:11:142
165883899611,16cyclictest1665710-21ssh21:11:142
165883899611,10cyclictest1737846-21sshd23:42:142
1658831996110,5cyclictest131rcu_preempt23:25:500
1658831996110,5cyclictest131rcu_preempt23:25:490
1658843996015,12cyclictest37-21ksoftirqd/322:20:003
1658843996015,12cyclictest37-21ksoftirqd/322:20:003
165883899602,5cyclictest1660833-21sshd19:46:152
165883899602,15cyclictest1746544-21ssh00:00:112
165883899601,8cyclictest1719535-21ssh23:04:062
165883899601,7cyclictest1763159-21ssh00:35:032
165883899601,7cyclictest1670340-21ssh21:20:562
165883899601,5cyclictest1751843-21ssh00:11:302
165883899601,14cyclictest1661128-21munin-run19:55:012
1658831996021,34cyclictest16850irq/134-800000023:45:000
1658831996010,44cyclictest131rcu_preempt23:57:040
165884399592,10cyclictest37-21ksoftirqd/322:40:003
165884399592,10cyclictest37-21ksoftirqd/322:40:003
165883899592,5cyclictest1662151-21munin-run20:15:012
165883899592,5cyclictest1662151-21munin-run20:15:012
165883899592,12cyclictest1662383-21sshd20:17:152
165883899591,8cyclictest1660063-21sshd19:34:152
1658838995918,36cyclictest1663452-21/usr/sbin/munin20:40:012
165883899591,6cyclictest1722566-21ssh23:10:042
165883899591,6cyclictest1715568-21ssh22:55:332
165883899591,6cyclictest1674882-21ssh21:30:332
165883899591,5cyclictest1729763-21ssh23:25:092
165883899591,5cyclictest1729763-21ssh23:25:092
165883199593,5cyclictest131rcu_preempt23:20:260
165884399587,9cyclictest37-21ksoftirqd/320:30:013
165884399585,7cyclictest37-21ksoftirqd/323:55:003
1658843995810,6cyclictest131rcu_preempt21:35:003
165883899588,46cyclictest1664399-21cut20:55:022
165883899582,5cyclictest1660296-21sshd19:35:162
165883899581,11cyclictest364-21snmpd20:31:482
1658831995828,26cyclictest1748667-21ssh00:05:000
1658831995813,39cyclictest131rcu_preempt23:55:010
1658843995711,6cyclictest131rcu_preempt21:40:003
165883899573,49cyclictest1665000-21cron21:10:002
165883899572,6cyclictest1-21systemd20:00:012
165883899572,6cyclictest1-21systemd20:00:012
165883899572,4cyclictest1659541-21sshd19:20:452
1658831995747,6cyclictest12-21ksoftirqd/021:10:000
165883199572,40cyclictest1679313-21ssh21:40:010
1658831995717,35cyclictest16850irq/134-800000022:15:000
1658831995715,38cyclictest12-21ksoftirqd/023:10:000
1658831995715,38cyclictest12-21ksoftirqd/023:10:000
1658831995714,6cyclictest121ksoftirqd/022:55:370
1658831995711,27cyclictest131rcu_preempt00:15:180
1658831995710,8cyclictest12-21ksoftirqd/023:37:400
165884399567,6cyclictest131rcu_preempt00:02:353
1658843995618,7cyclictest37-21ksoftirqd/320:05:013
1658843995614,5cyclictest131rcu_preempt19:35:013
1658843995610,6cyclictest131rcu_preempt20:20:003
1658838995624,27cyclictest309-21in:imuxsock20:04:152
165883899561,4cyclictest1659252-21sshd19:15:422
165883899561,4cyclictest1659252-21sshd19:15:422
165883899561,15cyclictest364-21snmpd20:05:572
165883199567,5cyclictest131rcu_preempt22:05:020
1658831995645,7cyclictest12-21ksoftirqd/022:20:020
1658831995645,7cyclictest12-21ksoftirqd/022:20:020
1658831995637,14cyclictest275-21dbus-daemon22:55:000
1658831995628,13cyclictest1760266-21ssh00:29:220
1658831995628,13cyclictest1710241-21ssh22:44:490
1658831995621,4cyclictest131rcu_preempt21:22:180
1658831995620,31cyclictest16850irq/134-800000023:12:510
1658831995611,8cyclictest131rcu_preempt20:05:010
165884399555,11cyclictest37-21ksoftirqd/322:22:573
165884399553,9cyclictest37-21ksoftirqd/321:10:003
1658843995516,5cyclictest131rcu_preempt21:13:023
1658843995516,5cyclictest131rcu_preempt21:13:023
165883899552,5cyclictest364-21snmpd20:56:052
165883899552,5cyclictest364-21snmpd20:29:562
165883899552,5cyclictest364-21snmpd20:20:582
165883899552,15cyclictest1663847-21cut20:45:022
165883899551,3cyclictest364-21snmpd19:10:442
165883199558,4cyclictest131rcu_preempt23:30:500
1658831995528,12cyclictest1764210-21ssh00:37:230
1658831995518,32cyclictest16850irq/134-800000021:40:300
1658831995518,32cyclictest16850irq/134-800000021:14:320
1658831995518,32cyclictest16850irq/134-800000021:14:320
1658831995517,34cyclictest16850irq/134-800000022:20:560
165883199551,38cyclictest1672980-21ssh21:26:330
165884399549,6cyclictest131rcu_preempt23:04:553
165884399545,5cyclictest131rcu_preempt23:30:243
1658843995436,13cyclictest1-21systemd23:10:023
1658843995436,13cyclictest1-21systemd23:10:013
1658843995416,6cyclictest131rcu_preempt23:15:543
1658843995416,6cyclictest131rcu_preempt23:15:533
1658843995412,6cyclictest131rcu_preempt00:10:033
1658843995410,5cyclictest131rcu_preempt22:05:003
165883899542,47cyclictest0-21swapper/221:01:062
165883899542,47cyclictest0-21swapper/221:01:062
165883899541,7cyclictest364-21snmpd19:26:222
1658833995435,14cyclictest275-21dbus-daemon21:20:011
1658831995425,14cyclictest1758176-21ssh00:24:570
1658831995425,14cyclictest1758176-21ssh00:24:570
1658831995425,14cyclictest1684766-21ssh21:51:040
1658831995425,13cyclictest1712569-21ssh22:49:380
1658831995425,13cyclictest1712569-21ssh22:49:380
1658831995419,31cyclictest16850irq/134-800000022:08:330
1658831995419,31cyclictest16850irq/134-800000000:09:310
1658831995418,32cyclictest16850irq/134-800000021:49:190
1658831995417,33cyclictest12-21ksoftirqd/019:40:490
165883199541,5cyclictest131rcu_preempt23:48:010
165884399537,8cyclictest37-21ksoftirqd/323:14:293
165884399537,6cyclictest131rcu_preempt22:06:533
165884399536,5cyclictest131rcu_preempt21:30:013
165884399535,8cyclictest37-21ksoftirqd/323:25:133
165884399535,8cyclictest37-21ksoftirqd/323:25:123
1658843995311,5cyclictest131rcu_preempt00:30:153
1658843995311,5cyclictest131rcu_preempt00:30:153
165883199532,9cyclictest0-21swapper/000:35:000
165883199532,9cyclictest0-21swapper/000:35:000
1658831995319,19cyclictest16850irq/134-800000021:33:210
1658831995312,6cyclictest131rcu_preempt20:45:000
165884399528,6cyclictest131rcu_preempt22:32:023
165884399524,9cyclictest37-21ksoftirqd/320:50:033
1658843995215,7cyclictest131rcu_preempt23:23:333
1658843995212,5cyclictest131rcu_preempt22:10:563
1658843995211,5cyclictest131rcu_preempt22:49:293
1658843995211,5cyclictest131rcu_preempt22:49:293
1658843995211,5cyclictest131rcu_preempt00:11:493
1658833995231,14cyclictest275-21dbus-daemon19:20:011
1658833995231,14cyclictest275-21dbus-daemon19:20:011
1658831995217,31cyclictest16850irq/134-800000021:16:140
165884399518,6cyclictest131rcu_preempt22:53:473
165884399518,5cyclictest131rcu_preempt22:26:263
165884399518,5cyclictest131rcu_preempt22:26:263
165884399516,8cyclictest37-21ksoftirqd/300:27:533
165884399516,10cyclictest37-21ksoftirqd/300:25:003
165884399516,10cyclictest37-21ksoftirqd/300:25:003
165884399512,8cyclictest37-21ksoftirqd/322:40:413
1658843995113,5cyclictest131rcu_preempt23:57:123
1658843995111,5cyclictest131rcu_preempt21:22:003
1658843995110,6cyclictest131rcu_preempt21:17:043
1658831995116,31cyclictest16850irq/134-800000022:29:280
1658831995116,31cyclictest16850irq/134-800000022:29:280
165884399508,5cyclictest131rcu_preempt21:46:323
165884399504,6cyclictest131rcu_preempt21:53:043
165884399504,11cyclictest37-21ksoftirqd/323:37:133
1658843995014,6cyclictest131rcu_preempt22:55:143
1658843995014,6cyclictest131rcu_preempt19:35:483
1658843995013,11cyclictest37-21ksoftirqd/300:17:363
165883899502,41cyclictest1658923-21ls19:10:012
1658833995030,15cyclictest275-21dbus-daemon22:20:021
1658833995030,15cyclictest275-21dbus-daemon22:20:011
1658831995015,31cyclictest16850irq/134-800000023:15:450
1658831995015,31cyclictest16850irq/134-800000023:15:450
165884399498,7cyclictest37-21ksoftirqd/320:25:013
165884399495,6cyclictest37-21ksoftirqd/323:41:023
1658843994912,6cyclictest131rcu_preempt21:44:333
1658831994914,31cyclictest16850irq/134-800000021:56:290
1658831994914,27cyclictest131rcu_preempt20:51:040
165884399486,5cyclictest131rcu_preempt00:37:403
165884399484,9cyclictest37-21ksoftirqd/320:40:003
165883199485,3cyclictest131rcu_preempt19:35:000
1658831994828,14cyclictest275-21dbus-daemon19:25:000
1658831994821,6cyclictest151rcuc/020:35:020
1658831994812,7cyclictest131rcu_preempt20:40:010
1658831994812,6cyclictest131rcu_preempt20:20:020
165884399479,6cyclictest131rcu_preempt19:20:003
165884399479,6cyclictest131rcu_preempt19:20:003
165884399478,8cyclictest131rcu_preempt19:30:023
165884399478,5cyclictest131rcu_preempt19:15:003
165884399475,10cyclictest37-21ksoftirqd/321:00:003
165884399471,8cyclictest37-21ksoftirqd/319:25:003
1658833994735,6cyclictest309-21in:imuxsock23:50:011
1658833994730,12cyclictest1746201-21mandb00:00:011
1658833994729,13cyclictest1-21systemd21:45:011
165883199469,5cyclictest131rcu_preempt19:20:020
165883199469,5cyclictest131rcu_preempt19:20:010
165883199463,8cyclictest12-21ksoftirqd/020:10:010
1658831994624,9cyclictest12-21ksoftirqd/020:30:010
165883199462,39cyclictest3467602-21systemd-journal19:50:360
165884399457,5cyclictest131rcu_preempt20:50:033
165884399453,8cyclictest37-21ksoftirqd/320:45:013
165883399453,4cyclictest131rcu_preempt22:00:451
165883399453,3cyclictest131rcu_preempt21:12:491
165883399453,3cyclictest131rcu_preempt21:12:491
165883199454,8cyclictest131rcu_preempt21:05:020
165883199454,8cyclictest131rcu_preempt21:05:020
165884399449,5cyclictest131rcu_preempt20:35:003
165884399445,6cyclictest37-21ksoftirqd/319:49:153
165884399442,5cyclictest131rcu_preempt20:15:023
165884399442,5cyclictest131rcu_preempt20:15:023
1658833994431,9cyclictest23-21ksoftirqd/122:45:011
1658833994425,13cyclictest3467602-21systemd-journal20:15:001
1658833994425,13cyclictest3467602-21systemd-journal20:14:591
165883199447,7cyclictest131rcu_preempt19:10:010
1658831994436,5cyclictest12-21ksoftirqd/021:00:010
1658831994430,11cyclictest3467602-21systemd-journal20:47:150
1658831994430,11cyclictest3467602-21systemd-journal20:47:150
165884399438,5cyclictest131rcu_preempt19:45:023
165884399433,6cyclictest131rcu_preempt21:05:003
165884399433,6cyclictest131rcu_preempt21:05:003
165884399433,6cyclictest131rcu_preempt20:00:003
165884399433,6cyclictest131rcu_preempt20:00:003
1658833994328,11cyclictest1-21systemd23:45:001
165883399432,3cyclictest131rcu_preempt21:51:411
1658833994318,7cyclictest0-21swapper/123:36:541
165883399431,4cyclictest131rcu_preempt22:57:011
1658831994336,4cyclictest16950irq/224-800000020:00:010
1658831994336,4cyclictest16950irq/224-800000020:00:010
165883399425,3cyclictest150-21pr/ttyS221:56:291
1658833994224,12cyclictest309-21in:imuxsock21:00:011
165883399422,3cyclictest131rcu_preempt21:47:341
165883399421,3cyclictest131rcu_preempt23:08:161
165883399421,3cyclictest131rcu_preempt23:08:161
165883199426,5cyclictest131rcu_preempt19:40:000
1658831994230,7cyclictest16950irq/224-800000020:10:560
1658831994230,7cyclictest16950irq/224-800000020:10:560
1658843994110,5cyclictest131rcu_preempt20:47:153
165883399413,3cyclictest131rcu_preempt19:25:001
1658833994128,9cyclictest3467602-21systemd-journal20:05:001
1658833994125,11cyclictest275-21dbus-daemon20:20:001
1658833994123,13cyclictest275-21dbus-daemon23:05:011
165883399411,5cyclictest0-21swapper/122:21:111
165883399411,3cyclictest131rcu_preempt21:28:061
165883399411,3cyclictest131rcu_preempt00:34:341
165883399411,3cyclictest131rcu_preempt00:34:331
165883399403,3cyclictest131rcu_preempt00:01:391
1658833994030,7cyclictest1743076-21ssh23:53:201
1658833994029,5cyclictest309-21in:imuxsock20:30:011
1658833994027,7cyclictest1729692-21sh23:25:021
1658833994025,11cyclictest0-21swapper/100:30:001
165883399402,3cyclictest131rcu_preempt23:17:411
165883399402,3cyclictest131rcu_preempt23:17:411
165883199408,5cyclictest131rcu_preempt20:25:010
1658831994033,4cyclictest16950irq/224-800000019:15:010
165884399391,9cyclictest37-21ksoftirqd/319:55:023
165883399394,3cyclictest131rcu_preempt00:37:421
1658833993926,9cyclictest23-21ksoftirqd/122:49:111
1658833993926,9cyclictest23-21ksoftirqd/122:49:101
165883399391,3cyclictest131rcu_preempt22:33:031
165883399391,3cyclictest131rcu_preempt22:11:251
165883399391,3cyclictest131rcu_preempt00:24:341
165883399391,3cyclictest131rcu_preempt00:24:331
1658831993915,6cyclictest131rcu_preempt19:30:020
1658833993829,7cyclictest23-21ksoftirqd/100:17:321
1658833993828,7cyclictest1-21systemd00:10:011
165883399382,3cyclictest131rcu_preempt00:13:531
165883399381,3cyclictest131rcu_preempt22:26:511
165883399381,3cyclictest131rcu_preempt22:26:511
165883399373,3cyclictest131rcu_preempt20:50:011
165883399373,3cyclictest131rcu_preempt20:50:011
1658833993724,9cyclictest23-21ksoftirqd/123:34:151
1658833993724,10cyclictest3467602-21systemd-journal19:35:481
1658833993719,12cyclictest275-21dbus-daemon19:50:011
165883399371,3cyclictest131rcu_preempt22:38:161
165883399371,3cyclictest131rcu_preempt22:38:161
165884399363,6cyclictest131rcu_preempt19:10:013
165883399363,6cyclictest131rcu_preempt20:00:001
165883399363,6cyclictest131rcu_preempt20:00:001
1658833993625,8cyclictest23-21ksoftirqd/122:54:441
1658833993625,8cyclictest1659005-21sshd19:11:151
1658833993622,10cyclictest23-21ksoftirqd/122:09:441
165884399356,6cyclictest131rcu_preempt20:10:003
165883399354,3cyclictest131rcu_preempt21:20:151
1658833993525,7cyclictest23-21ksoftirqd/119:30:481
165883399352,3cyclictest131rcu_preempt20:40:001
165883399351,10cyclictest1664755-21cron21:05:001
165883399351,10cyclictest1664755-21cron21:05:001
165883399344,8cyclictest23-21ksoftirqd/120:51:041
1658833993427,4cyclictest1675516-21sh21:31:551
1658833993423,8cyclictest1661250-21wc19:55:011
1658833993420,9cyclictest23-21ksoftirqd/120:25:021
1658831993422,9cyclictest3467602-21systemd-journal19:45:510
165883399325,3cyclictest131rcu_preempt20:35:001
1658833993224,5cyclictest1658986-21latency_hist19:10:021
165883399315,3cyclictest131rcu_preempt19:28:151
1658833993116,11cyclictest1660638-21cat19:45:021
1658833993114,5cyclictest131rcu_preempt20:45:001
165883399302,3cyclictest131rcu_preempt20:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional