You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-07 - 09:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Sep 07, 2026 00:45:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1695382998827,16cyclictest1797022-21ssh00:35:230
169539099871,80cyclictest1757194-21ssh23:10:021
1695397998624,24cyclictest1742531-21ssh22:38:253
1695397998624,24cyclictest1742531-21ssh22:38:253
169538299862,42cyclictest1732392-21ssh22:16:210
169538299862,42cyclictest1732392-21ssh22:16:200
1695382998527,13cyclictest1743790-21ssh22:40:590
1695382998429,14cyclictest1788532-21ssh00:17:440
1695382998422,12cyclictest1742111-21ssh22:37:340
1695382998422,12cyclictest1742111-21ssh22:37:330
1695382998418,56cyclictest16850irq/134-800000021:34:280
1695382998226,11cyclictest1721019-21ssh21:51:560
1695382998128,11cyclictest1718715-21ssh21:46:480
1695382998028,16cyclictest1705530-21ssh21:18:240
1695382997929,15cyclictest1753844-21ssh23:02:420
1695382997917,18cyclictest16850irq/134-800000000:03:440
1695382997826,4cyclictest0-21swapper/023:29:120
1695382997817,52cyclictest16850irq/134-800000021:13:460
1695382997817,52cyclictest16850irq/134-800000021:13:460
1695382997726,16cyclictest1739407-21ssh22:31:320
1695382997724,3cyclictest0-21swapper/023:55:020
1695397997525,12cyclictest1791485-21ssh00:24:043
169538299754,5cyclictest1787293-21latency_hist00:15:020
1695382997524,13cyclictest1772519-21ssh23:43:410
169538299743,27cyclictest172526750ssh22:00:520
1695382997428,13cyclictest1773777-21ssh23:46:020
1695382997426,13cyclictest1707056-21ssh21:21:260
169538299741,35cyclictest1735855-21ssh22:24:040
1695397997325,15cyclictest1733816-21ssh22:19:453
1695397997325,15cyclictest1733816-21ssh22:19:453
1695397997317,24cyclictest1794667-21ssh00:30:273
1695382997326,10cyclictest1723110-21ssh21:56:230
1695382997326,10cyclictest1723110-21ssh21:56:220
1695397997227,20cyclictest1734327-21sh22:20:283
1695397997215,36cyclictest37-21ksoftirqd/320:45:013
1695382997222,12cyclictest1758448-21ssh23:12:480
1695382997222,12cyclictest1758448-21ssh23:12:480
1695382997218,16cyclictest16850irq/134-800000022:29:460
1695382997217,11cyclictest1778588-21ssh23:56:320
1695382997216,18cyclictest16850irq/134-800000021:40:470
1695382997211,54cyclictest0-21swapper/023:10:030
1695397997129,14cyclictest1704160-21ssh21:15:083
1695382997127,9cyclictest1789982-21ssh00:20:400
1695397997024,15cyclictest1718804-21ssh21:46:593
1695382997024,11cyclictest1746404-21ssh22:46:280
169538299701,33cyclictest1751647-21ssh22:58:060
1695397996959,7cyclictest37-21ksoftirqd/300:05:023
1695397996928,11cyclictest150-21pr/ttyS223:52:263
169538299693,56cyclictest179642850ssh00:34:360
1695382996916,18cyclictest16850irq/134-800000022:06:250
1695382996916,18cyclictest16850irq/134-800000022:06:240
1695382996916,18cyclictest16850irq/134-800000000:05:470
1695397996715,16cyclictest1771239-21ssh23:40:373
169538299676,50cyclictest176646650ssh23:30:050
1695382996716,17cyclictest16850irq/134-800000000:26:530
1695397996625,11cyclictest150-21pr/ttyS221:52:293
1695390996526,35cyclictest1717272-21sh21:43:411
1695382996527,7cyclictest12-21ksoftirqd/021:28:000
1695397996421,15cyclictest1744945-21ssh22:43:433
1695397996413,18cyclictest1759200-21ssh23:14:383
1695397996413,18cyclictest1759200-21ssh23:14:373
169539099641,15cyclictest1720513-21ssh21:50:371
169538299645,49cyclictest174873750ssh22:51:470
169538299642,36cyclictest1730191-21ssh22:11:340
1695382996416,17cyclictest16850irq/134-800000023:37:040
1695397996323,12cyclictest1765077-21rm23:27:203
169539099632,7cyclictest1739412-21ssh22:31:321
1695390996325,34cyclictest1734917-21sh22:21:571
169538299636,52cyclictest176145550ssh23:19:410
1695382996313,45cyclictest16850irq/134-800000021:36:580
1695382996313,45cyclictest16850irq/134-800000021:36:580
1695397996225,12cyclictest150-21pr/ttyS219:24:153
1695397996214,15cyclictest1716020-21ssh21:40:373
169539099622,16cyclictest1780111-21latency_hist00:00:031
169539099621,9cyclictest1797335-21ssh00:36:141
169539099621,9cyclictest1770073-21ssh23:38:181
169539099621,9cyclictest1744185-21ssh22:41:501
169539099621,7cyclictest1764433-21ssh23:25:451
169539099621,7cyclictest1708396-21ssh21:24:361
169538299623,28cyclictest176365150ssh23:24:210
1695382996222,12cyclictest3467602-21systemd-journal19:30:000
1695397996125,6cyclictest1737352-21ssh22:27:073
1695397996123,13cyclictest150-21pr/ttyS219:39:153
1695390996131,26cyclictest3467602-21systemd-journal19:50:001
169539099612,5cyclictest1729356-21munin-run22:10:011
169539099612,5cyclictest1729356-21munin-run22:10:011
169539099611,9cyclictest1783600-21ssh00:07:191
169539099611,8cyclictest1788083-21ssh00:16:411
169539099611,8cyclictest1771628-21ssh23:41:271
169539099611,8cyclictest1753845-21ssh23:02:431
169539099611,8cyclictest1741501-21ssh22:36:021
169539099611,8cyclictest1741501-21ssh22:36:021
169539099611,8cyclictest1711592-21ssh21:31:121
169539099611,7cyclictest1795087-21ssh00:31:291
169539099611,7cyclictest1751596-21ssh22:58:001
1695382996116,13cyclictest1701590-21ssh21:10:010
169539099602,6cyclictest1695580-21munin-run19:15:001
1695390996023,33cyclictest1761183-21sh23:19:061
169539099602,18cyclictest1794244-21systemd-run00:30:021
169539099601,9cyclictest1786846-21ssh00:14:261
169539099601,8cyclictest1776198-21ssh23:51:271
169539099601,8cyclictest1729616-21ssh22:10:101
169539099601,8cyclictest1724981-21ssh22:00:101
169539099601,8cyclictest1702525-21ssh21:11:451
169539099601,8cyclictest1702525-21ssh21:11:451
169539099601,7cyclictest1704364-21ssh21:15:341
1695390996016,39cyclictest23-21ksoftirqd/123:32:311
1695397995928,23cyclictest1760818-21ssh23:18:053
1695397995922,11cyclictest1798417-21ssh00:38:393
1695390995922,32cyclictest1-21systemd19:30:011
169539099591,7cyclictest1738454-21ssh22:29:511
1695397995834,19cyclictest1762583-21ssh23:21:423
169539099583,5cyclictest1759472-21sed23:15:031
169539099583,5cyclictest1759472-21sed23:15:021
1695390995831,24cyclictest23-21ksoftirqd/122:50:431
169539099582,5cyclictest1698455-21munin-run20:10:011
169539099581,7cyclictest1714391-21ssh21:37:191
169539099581,7cyclictest1714391-21ssh21:37:191
169539099581,7cyclictest1709404-21ssh21:26:351
1695382995816,13cyclictest275-21dbus-daemon19:55:010
1695397995724,11cyclictest150-21pr/ttyS220:58:023
169539099572,7cyclictest1697168-21munin-run19:45:011
169539099572,7cyclictest1697168-21munin-run19:45:011
169539099572,14cyclictest1696637-21munin-run19:35:011
169539099571,9cyclictest1781562-21ssh00:03:081
169539099571,9cyclictest1746584-21ssh22:47:041
169539099571,7cyclictest364-21snmpd23:49:231
1695390995716,37cyclictest1790823-1kworker/u9:200:22:541
169539099571,15cyclictest1722772-21ssh21:55:281
169539099571,15cyclictest1722772-21ssh21:55:281
1695382995721,8cyclictest12-21ksoftirqd/020:45:020
1695397995630,21cyclictest1784695-21rm00:09:553
1695397995616,10cyclictest1767515-21ssh23:32:433
169539099567,44cyclictest0-21swapper/123:22:521
169539099563,6cyclictest1699499-21munin-run20:30:001
1695397995546,5cyclictest1710015-21apt-get21:28:013
1695397995531,19cyclictest3467602-21systemd-journal20:30:023
1695397995526,8cyclictest1789577-21munin-run00:20:013
1695394995546,5cyclictest1697673-21kworker/2:322:55:312
1695394995545,6cyclictest1694972-21kworker/2:200:20:022
169539099551,8cyclictest1718994-21ssh21:47:231
169539099551,8cyclictest1698157-21sshd20:01:081
169539099551,7cyclictest1700522-21munin-run20:50:001
1695390995513,36cyclictest275-21dbus-daemon20:15:011
169539099551,15cyclictest1732988-21ssh22:17:481
169539099551,15cyclictest1732988-21ssh22:17:481
1695397995437,13cyclictest1723589-21ssh21:57:373
1695397995437,13cyclictest1723589-21ssh21:57:373
1695397995433,17cyclictest1779030-21sh23:57:433
1695397995433,17cyclictest1770503-21sh23:39:223
1695397995430,19cyclictest1699878-21grep20:35:023
1695394995445,6cyclictest1694972-21kworker/2:223:23:472
169539099542,6cyclictest1697672-21munin-run19:55:011
1695390995420,30cyclictest23-21ksoftirqd/119:19:431
169538299544,22cyclictest275-21dbus-daemon20:20:010
1695397995340,10cyclictest1739199-21ssh22:30:573
1695397995338,12cyclictest1753282-21ssh23:01:223
1695397995338,12cyclictest1750315-21ssh22:55:073
1695397995338,12cyclictest1707609-21ssh21:22:453
1695397995338,11cyclictest1715014-21ssh21:38:533
1695397995338,11cyclictest1715014-21ssh21:38:523
1695397995338,10cyclictest1699248-21munin-run20:25:013
1695397995337,13cyclictest1702286-21ssh21:11:013
1695397995337,13cyclictest1702286-21ssh21:11:013
1695397995336,13cyclictest1729067-21ssh22:09:293
1695397995336,13cyclictest1729067-21ssh22:09:293
1695397995335,14cyclictest1700482-21modprobe20:45:023
169539499535,3cyclictest131rcu_preempt19:55:002
1695394995335,13cyclictest3467602-21systemd-journal23:40:012
169539099532,17cyclictest1698965-21munin-run20:20:001
169539099531,6cyclictest1700789-21munin-run20:55:011
169539099531,12cyclictest1700988-21modprobe20:55:031
1695382995323,7cyclictest12-21ksoftirqd/019:15:010
1695397995242,7cyclictest48-21kcompactd000:29:553
1695397995239,10cyclictest1773316-21ssh23:45:043
1695397995237,12cyclictest1747325-21ssh22:48:493
1695397995236,13cyclictest1755040-21ssh23:05:053
1695397995236,12cyclictest1785506-21ssh00:11:193
1695394995244,5cyclictest1697673-21kworker/2:321:20:142
1695394995244,5cyclictest1694972-21kworker/2:222:30:312
1695394995244,5cyclictest1694972-21kworker/2:200:21:242
1695394995244,5cyclictest1694972-21kworker/2:200:01:122
169539099522,4cyclictest0-21swapper/119:23:381
1695397995138,10cyclictest1729722-21ssh22:10:223
1695397995137,12cyclictest1712400-21ssh21:33:043
1695397995131,15cyclictest1698761-21latency_hist20:15:023
1695397995129,18cyclictest1701723-21ssh21:10:033
1695397995125,23cyclictest1726473-21ssh22:03:383
1695397995122,24cyclictest1697953-21unin-run20:00:013
1695394995143,5cyclictest1768527-21kworker/2:323:45:552
1695394995143,5cyclictest1697673-21kworker/2:323:02:442
169539099512,6cyclictest1699699-21modprobe20:30:021
169539099511,11cyclictest1695413-21latency_hist19:10:011
1695382995125,9cyclictest12-21ksoftirqd/019:22:070
1695397995035,12cyclictest1748198-21ssh22:50:253
1695397995033,14cyclictest1695558-21sshd19:13:073
1695397995027,19cyclictest1700169-21cat20:40:023
1695394995042,5cyclictest1694972-21kworker/2:221:40:262
1695394995041,6cyclictest1694972-21kworker/2:222:17:002
1695394995041,6cyclictest1694972-21kworker/2:222:17:002
1695394995030,15cyclictest275-21dbus-daemon00:00:022
169539499502,12cyclictest1761544-21ssh23:19:522
169539099502,8cyclictest1701311-21grep21:05:001
169539099501,10cyclictest1699966-21modprobe20:37:261
1695397994924,20cyclictest1696468-21latency_hist19:30:023
1695397994919,24cyclictest1-21systemd19:20:013
1695394994941,5cyclictest1694972-21kworker/2:222:48:042
1695394994941,5cyclictest1694972-21kworker/2:221:12:132
1695394994941,5cyclictest1694972-21kworker/2:221:12:122
1695394994940,6cyclictest1768527-21kworker/2:323:41:222
1695394994940,6cyclictest1694972-21kworker/2:200:30:372
1695394994940,5cyclictest1694972-21kworker/2:221:15:092
1695394994935,10cyclictest1-21systemd20:45:002
169539499492,13cyclictest1768191-21ssh23:34:212
169539099492,5cyclictest364-21snmpd19:37:051
169539099491,14cyclictest364-21snmpd19:57:091
1695394994840,6cyclictest1694972-21kworker/2:200:10:592
1695394994840,5cyclictest1768527-21kworker/2:323:51:542
1695394994840,5cyclictest1694972-21kworker/2:222:41:392
1695394994840,5cyclictest1694972-21kworker/2:222:27:532
1695394994840,5cyclictest1694972-21kworker/2:221:38:302
1695394994840,5cyclictest1694972-21kworker/2:221:38:292
1695394994840,5cyclictest1694972-21kworker/2:221:34:042
1695394994839,5cyclictest1697673-21kworker/2:320:03:072
1695394994839,5cyclictest1694972-21kworker/2:221:48:592
1695394994837,8cyclictest3467602-21systemd-journal23:10:072
1695394994837,8cyclictest3467602-21systemd-journal23:10:072
1695394994837,6cyclictest1694972-21kworker/2:223:29:072
1695394994828,10cyclictest309-21in:imuxsock19:45:012
1695394994828,10cyclictest309-21in:imuxsock19:45:012
169539099482,6cyclictest364-21snmpd20:41:481
169539099482,6cyclictest364-21snmpd20:20:491
169539099482,11cyclictest3467602-21systemd-journal21:10:001
1695397994735,9cyclictest1698170-21sshd20:03:073
1695397994729,14cyclictest364-21snmpd20:07:453
1695397994726,17cyclictest1696627-21sh19:35:013
1695397994721,22cyclictest1697478-21latency_hist19:50:013
1695397994721,20cyclictest1701448-21latency_hist21:05:023
1695394994739,5cyclictest1697673-21kworker/2:322:21:142
1695394994739,5cyclictest1697673-21kworker/2:322:05:152
1695394994739,5cyclictest1697673-21kworker/2:322:05:142
1695394994739,5cyclictest1697673-21kworker/2:322:04:532
1695394994739,5cyclictest1694972-21kworker/2:223:08:032
1695394994739,5cyclictest1694972-21kworker/2:200:07:032
1695394994738,5cyclictest1787169-21kworker/2:100:27:422
1695394994734,9cyclictest1724697-21sendmail-msp22:00:002
1695394994734,9cyclictest1724697-21sendmail-msp22:00:002
1695394994711,7cyclictest131rcu_preempt19:40:002
1695397994633,10cyclictest48-21kcompactd019:41:193
1695397994633,10cyclictest48-21kcompactd019:41:193
1695397994628,14cyclictest1697836-21latency_hist19:55:023
1695394994638,5cyclictest1694972-21kworker/2:222:35:072
1695394994638,5cyclictest1694972-21kworker/2:222:35:072
1695394994638,5cyclictest1694972-21kworker/2:221:50:212
1695394994626,15cyclictest1-21systemd19:25:002
1695382994627,15cyclictest275-21dbus-daemon19:45:010
1695382994627,15cyclictest275-21dbus-daemon19:45:010
1695397994533,9cyclictest1700736-21sshd20:50:083
1695394994537,5cyclictest1697673-21kworker/2:321:28:282
1695394994537,5cyclictest1697673-21kworker/2:320:15:002
1695394994531,6cyclictest1699230-21sh20:25:012
1695382994538,4cyclictest16950irq/224-800000020:00:020
1695382994538,4cyclictest16950irq/224-800000019:40:010
1695397994426,14cyclictest1698932-21sshd20:18:073
1695394994436,5cyclictest1697673-21kworker/2:322:51:212
1695394994335,5cyclictest1694972-21kworker/2:200:39:352
1695394994325,13cyclictest275-21dbus-daemon19:30:012
1695394994322,12cyclictest3467602-21systemd-journal20:00:002
169538299423,34cyclictest0-21swapper/019:10:020
1695382994227,3cyclictest16950irq/224-800000020:12:070
1695382994227,11cyclictest3467602-21systemd-journal20:07:220
1695394994131,6cyclictest1694972-21kworker/2:220:31:452
1695382994126,11cyclictest1700488-21sshd20:46:070
169538299411,4cyclictest131rcu_preempt20:05:010
1695397994019,16cyclictest1695465-21latency_hist19:10:013
169539499407,23cyclictest1698950-21modprobe20:20:002
1695394994026,10cyclictest3467602-21systemd-journal20:35:022
1695382994029,8cyclictest12-21ksoftirqd/020:40:020
1695382994027,10cyclictest1699453-21sshd20:26:070
1695394993929,7cyclictest30-21ksoftirqd/222:13:392
1695394993928,8cyclictest1696586-21sshd19:30:072
1695394993925,11cyclictest1700834-21wc20:55:022
1695394993923,11cyclictest1701547-21cron21:10:002
1695382993929,6cyclictest12-21ksoftirqd/020:30:120
1695382993925,11cyclictest12-21ksoftirqd/019:35:030
169538299392,33cyclictest1701289-21sshd21:03:070
169538299391,3cyclictest131rcu_preempt20:55:020
1695394993826,8cyclictest3467602-21systemd-journal20:50:012
1695394993822,11cyclictest309-21in:imuxsock20:30:002
1695382993825,9cyclictest1695820-21sshd19:18:070
169539499375,24cyclictest1695833-21modprobe19:20:002
169539499374,26cyclictest309-21in:imuxsock21:05:012
1695394993726,8cyclictest30-21ksoftirqd/219:50:002
169539499372,12cyclictest3467602-21systemd-journal19:11:072
1695382993724,10cyclictest12-21ksoftirqd/020:25:000
1695382993721,12cyclictest3467602-21systemd-journal19:45:030
169539499366,25cyclictest1701039-21munin-run21:00:002
1695382993528,4cyclictest16950irq/224-800000021:00:000
1695394993423,9cyclictest30-21ksoftirqd/220:10:022
169539499305,17cyclictest1695428-21wc19:10:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional