You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-08 - 10:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 08, 2026 00:45:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1706129991041,3cyclictest131rcu_preempt22:35:022
1706127999634,17cyclictest3467602-21systemd-journal22:20:011
1706134997430,38cyclictest1798111-21sshd00:28:453
170612799748,25cyclictest1711253-21munin-plugin-st20:50:011
1706127996932,12cyclictest23-21ksoftirqd/121:14:591
1706124996925,30cyclictest12-21ksoftirqd/000:25:130
1706134996627,34cyclictest1801029-21sh00:35:063
1706134996627,34cyclictest1801029-21sh00:35:053
1706134996528,33cyclictest1723070-21sh21:34:573
1706134996527,34cyclictest1743483-21sh22:21:593
170613499642,17cyclictest1764066-21munin-run23:10:003
1706124996429,8cyclictest1760340-21rm23:00:550
1706124996429,8cyclictest1760340-21rm23:00:540
170613499633,54cyclictest1755434-21true22:50:013
1706134996327,32cyclictest1795502-21sh00:22:383
1706129996315,16cyclictest1770433-21ssh23:24:452
170613499622,17cyclictest1728190-21ssh21:46:263
170613499621,18cyclictest1733517-21ssh21:59:103
170612499627,5cyclictest131rcu_preempt23:32:440
1706134996125,32cyclictest1707881-21latency_hist19:45:023
1706134996123,34cyclictest275-21dbus-daemon23:05:013
1706134996123,34cyclictest275-21dbus-daemon23:05:003
170613499611,9cyclictest1793040-21ssh00:16:563
170613499611,9cyclictest1793040-21ssh00:16:553
170613499611,8cyclictest1758063-21ssh22:55:463
170613499611,8cyclictest1750064-21ssh22:37:213
170613499611,7cyclictest1765454-21ssh23:13:093
170613499611,7cyclictest1765454-21ssh23:13:093
170613499611,7cyclictest1723720-21ssh21:35:583
170613499611,7cyclictest1723720-21ssh21:35:573
170613499611,6cyclictest1718144-21ssh21:23:193
1706129996115,16cyclictest1709206-21sed20:10:022
1706129996115,16cyclictest1709206-21sed20:10:012
170613499602,15cyclictest1799036-21ssh00:30:353
170613499601,8cyclictest1744937-21ssh22:25:053
170613499601,8cyclictest1719436-21ssh21:26:003
170613499601,7cyclictest1790725-21ssh00:11:293
170613499601,7cyclictest1788508-21ssh00:06:163
170613499601,7cyclictest1766895-21ssh23:16:133
170613499601,6cyclictest1734340-21ssh22:00:343
170613499601,11cyclictest1757534-21ssh22:54:553
170613499601,11cyclictest1757534-21ssh22:54:543
1706124996020,30cyclictest16850irq/134-800000022:31:010
170613499593,51cyclictest1785615-40mandb00:00:023
170613499591,8cyclictest1730271-21ssh21:51:113
170613499591,8cyclictest1725858-21ssh21:40:573
170613499591,7cyclictest1786247-21ssh00:00:543
170613499591,7cyclictest1786247-21ssh00:00:533
170613499591,5cyclictest1753063-21ssh22:44:283
170613499591,5cyclictest1753063-21ssh22:44:273
170613499591,20cyclictest1740342-21latency_hist22:15:023
1706127995939,15cyclictest3467602-21systemd-journal00:00:021
1706124995919,31cyclictest16850irq/134-800000021:40:480
170613499582,49cyclictest1711764-21cut20:59:593
170613499582,18cyclictest1772565-21ssh23:29:453
170613499581,5cyclictest1777846-21ssh23:41:373
170613499581,4cyclictest1781818-21ssh23:50:443
170613499581,4cyclictest1769936-21ssh23:23:283
1706127995838,15cyclictest275-21dbus-daemon23:15:011
1706127995838,15cyclictest275-21dbus-daemon23:15:001
170612499589,15cyclictest1734906-21ssh22:02:040
1706124995819,8cyclictest12-21ksoftirqd/000:31:320
170613499572,5cyclictest1708609-21sshd19:56:223
1706134995725,26cyclictest309-21in:imuxsock20:48:423
170613499572,46cyclictest1706559-21cron19:20:003
170613499571,7cyclictest1740574-21ssh22:15:023
170613499571,4cyclictest1712880-21ssh21:10:523
1706124995720,32cyclictest16850irq/134-800000023:19:390
1706124995720,32cyclictest16850irq/134-800000022:09:230
1706124995720,32cyclictest16850irq/134-800000022:09:230
1706124995718,6cyclictest16850irq/134-800000022:30:010
170613499562,5cyclictest1706355-21cat19:15:023
170613499562,16cyclictest1779606-21ssh23:45:353
170613499561,5cyclictest1747146-21ssh22:30:173
170613499561,4cyclictest1775759-21ssh23:36:483
170613499561,19cyclictest1715136-21ssh21:16:083
1706124995628,13cyclictest1714389-21ssh21:14:450
1706124995621,30cyclictest16850irq/134-800000023:13:540
1706124995621,30cyclictest16850irq/134-800000023:13:530
1706124995618,8cyclictest12-21ksoftirqd/021:30:020
1706124995618,33cyclictest16850irq/134-800000000:00:010
1706124995617,10cyclictest12-21ksoftirqd/000:13:360
170613499553,5cyclictest1710999-21munin-run20:45:013
170613499553,47cyclictest1709149-21sshd20:08:423
170613499553,47cyclictest1709149-21sshd20:08:423
170613499551,7cyclictest1774683-21ssh23:34:353
170613499551,7cyclictest1736565-21ssh22:05:483
170613499551,7cyclictest1736565-21ssh22:05:473
170613499551,6cyclictest1712268-21sshd21:09:413
1706129995533,16cyclictest3467602-21systemd-journal19:14:592
1706124995525,15cyclictest1793301-21ssh00:17:330
1706124995525,15cyclictest1793301-21ssh00:17:320
1706124995524,5cyclictest0-21swapper/023:40:370
1706124995519,31cyclictest16850irq/134-800000022:15:260
1706124995517,10cyclictest12-21ksoftirqd/021:20:000
1706124995514,9cyclictest12-21ksoftirqd/021:20:150
170613499542,6cyclictest1710215-21munin-run20:30:003
1706134995426,25cyclictest37-21ksoftirqd/320:20:033
1706134995426,25cyclictest37-21ksoftirqd/320:20:023
170613499542,5cyclictest1710727-21munin-run20:40:003
170613499542,5cyclictest1710727-21munin-run20:40:003
170613499542,4cyclictest1708111-21sshd19:46:423
1706134995424,27cyclictest37-21ksoftirqd/319:30:013
170613499542,17cyclictest1707630-21grep19:40:013
170613499541,9cyclictest1710438-21sshd20:31:433
170613499541,5cyclictest1708894-21sshd20:02:433
1706124995424,14cyclictest1757915-21ssh22:55:170
1706124995416,10cyclictest12-21ksoftirqd/000:10:000
170613499532,5cyclictest364-21snmpd19:30:393
170613499531,4cyclictest1712005-21sshd21:01:433
170613499531,4cyclictest1709400-21sshd20:13:433
170613499531,16cyclictest1711468-21sshd20:50:433
170613499531,16cyclictest1711468-21sshd20:50:423
170612499532,7cyclictest1709694-21grep20:20:000
1706124995327,11cyclictest1776610-21ssh23:39:070
1706124995325,13cyclictest1770882-21ssh23:25:180
170612499532,47cyclictest1732544-21ssh21:56:330
1706124995317,32cyclictest16850irq/134-800000022:37:440
170612499531,37cyclictest1728741-21ssh21:47:570
1706127995235,11cyclictest1779305-21cut23:45:021
170612499527,9cyclictest12-21ksoftirqd/023:20:400
170612499526,9cyclictest12-21ksoftirqd/023:06:230
1706124995215,6cyclictest151rcuc/000:40:020
1706124995215,6cyclictest151rcuc/000:40:010
1706124995213,9cyclictest12-21ksoftirqd/021:35:190
1706124995213,9cyclictest12-21ksoftirqd/021:35:180
170613499511,5cyclictest1706831-21sshd19:23:423
170613499511,4cyclictest0-21swapper/320:20:013
1706127995135,11cyclictest23-21ksoftirqd/122:03:421
170612499516,9cyclictest121ksoftirqd/022:49:310
1706124995116,31cyclictest151rcuc/022:21:140
1706124995110,37cyclictest1721916-21ssh21:31:520
1706129995034,12cyclictest1-21systemd23:35:012
1706129995031,14cyclictest1765632-21sh23:13:362
1706129995031,14cyclictest1765632-21sh23:13:352
1706129995022,24cyclictest1711247-21latency_hist20:50:022
1706127995026,17cyclictest275-21dbus-daemon22:00:021
1706124995034,12cyclictest3467602-21systemd-journal00:00:040
1706124995034,12cyclictest3467602-21systemd-journal00:00:040
1706124995023,12cyclictest1739918-21ssh22:14:000
1706124995016,30cyclictest16850irq/134-800000000:23:490
1706124995013,7cyclictest12-21ksoftirqd/020:10:020
1706124995013,7cyclictest12-21ksoftirqd/020:10:010
170613499492,5cyclictest364-21snmpd19:50:443
170613499492,5cyclictest364-21snmpd19:50:433
1706129994925,21cyclictest1741174-21ssh22:16:392
1706129994925,20cyclictest1738138-21ssh22:09:482
1706129994925,20cyclictest1738138-21ssh22:09:472
1706129994925,19cyclictest1788627-21ssh00:06:332
1706129994920,26cyclictest1802686-21sh00:39:292
1706129994920,26cyclictest1802686-21sh00:39:292
170612799494,29cyclictest1706563-21modprobe19:20:011
1706124994913,32cyclictest16850irq/134-800000023:52:420
1706129994840,5cyclictest1764096-21kworker/2:223:40:442
1706129994840,5cyclictest1723124-21kworker/2:000:01:562
1706129994840,5cyclictest1723124-21kworker/2:000:01:552
1706129994840,5cyclictest1711527-21kworker/2:321:24:362
1706129994836,8cyclictest1-21systemd20:05:002
1706129994824,20cyclictest1710602-21wc20:35:012
1706129994824,19cyclictest1713293-21ssh21:11:592
1706129994818,25cyclictest1743828-21ssh22:22:502
1706129994810,7cyclictest30-21ksoftirqd/200:14:032
1706127994828,15cyclictest3466785-21systemd-udevd21:45:591
1706127994828,13cyclictest3467602-21systemd-journal20:10:011
1706127994828,13cyclictest3467602-21systemd-journal20:10:001
1706127994824,17cyclictest3467602-21systemd-journal19:59:591
1706124994817,25cyclictest16850irq/134-800000022:43:560
1706124994817,25cyclictest16850irq/134-800000022:43:550
1706129994739,5cyclictest1764096-21kworker/2:223:57:192
1706129994739,5cyclictest1711527-21kworker/2:321:35:172
1706129994739,5cyclictest1711527-21kworker/2:321:35:162
1706129994725,18cyclictest1722389-21ssh21:33:112
1706129994712,5cyclictest30-21ksoftirqd/222:56:292
170612799474,32cyclictest3467602-21systemd-journal19:55:011
170612799474,32cyclictest3467602-21systemd-journal19:55:001
1706127994730,13cyclictest275-21dbus-daemon23:25:491
170612799472,31cyclictest1751163-21systemctl22:39:591
1706129994640,3cyclictest1711527-21kworker/2:321:54:112
1706129994637,6cyclictest1712286-21diskmemload23:05:012
1706129994637,6cyclictest1712286-21diskmemload23:05:002
1706129994624,18cyclictest1770821-21ssh23:25:082
1706129994623,19cyclictest1727055-21sh21:43:582
1706129994623,18cyclictest1732631-21ssh21:56:452
1706129994622,21cyclictest1782924-21sh23:53:352
1706129994621,22cyclictest1777118-21latency_hist23:40:012
1706129994619,23cyclictest1768568-21latency_hist23:20:022
170612799465,33cyclictest1740458-21cat22:15:011
1706127994632,10cyclictest1712272-21cron21:10:011
1706127994631,12cyclictest23-21ksoftirqd/123:35:001
1706124994628,14cyclictest1755878-21ssh22:50:350
1706124994628,14cyclictest1755878-21ssh22:50:340
170612499462,7cyclictest1706582-21modprobe19:20:000
170612499461,41cyclictest0-21swapper/019:15:010
1706129994538,5cyclictest1764096-21kworker/2:200:30:012
1706129994533,10cyclictest30-21ksoftirqd/200:35:022
1706129994528,14cyclictest1757142-21ssh22:54:002
1706129994528,14cyclictest1757142-21ssh22:53:592
1706129994528,12cyclictest1721028-21latency_hist21:30:012
1706129994527,12cyclictest1-21systemd20:25:012
1706129994527,12cyclictest1-21systemd20:25:012
1706129994526,13cyclictest1792506-21ssh00:15:302
1706129994526,13cyclictest1792506-21ssh00:15:292
1706129994524,17cyclictest1762133-21sh23:05:082
1706129994523,17cyclictest1749407-21ssh22:35:382
1706129994523,17cyclictest1745678-21rm22:27:012
1706129994519,21cyclictest1751569-21ssh22:40:332
1706129994519,21cyclictest1751569-21ssh22:40:322
1706129994518,22cyclictest1754616-21ssh22:47:582
1706129994516,25cyclictest1780555-21ssh23:47:582
1706127994535,8cyclictest23-21ksoftirqd/119:45:021
1706127994529,13cyclictest23-21ksoftirqd/100:19:141
1706127994529,13cyclictest23-21ksoftirqd/100:19:141
1706127994527,14cyclictest1-21systemd00:10:011
1706124994538,4cyclictest16950irq/224-800000021:10:020
1706124994538,4cyclictest16850irq/134-800000021:52:440
1706124994513,6cyclictest0-21swapper/023:47:440
1706129994436,5cyclictest1699207-21kworker/2:119:25:052
1706129994430,10cyclictest3467602-21systemd-journal19:35:022
1706129994429,11cyclictest30-21ksoftirqd/219:35:012
1706129994417,22cyclictest1734220-21ssh22:00:162
1706129994416,23cyclictest1738539-21ssh22:10:242
170612799447,7cyclictest131rcu_preempt00:05:031
170612799447,7cyclictest131rcu_preempt00:05:021
1706127994429,11cyclictest1783465-21cron23:55:001
170612799442,4cyclictest131rcu_preempt22:35:021
1706127994423,16cyclictest3467602-21systemd-journal19:30:001
1706124994437,4cyclictest16950irq/224-800000020:30:010
1706124994431,9cyclictest0-21swapper/019:40:030
1706124994429,11cyclictest12-21ksoftirqd/019:55:020
1706124994429,11cyclictest12-21ksoftirqd/019:55:010
170613499433,35cyclictest0-21swapper/319:10:033
170613499433,35cyclictest0-21swapper/319:10:023
1706129994334,6cyclictest1697453-21kworker/2:220:38:582
1706129994334,6cyclictest1697453-21kworker/2:220:38:572
1706129994329,10cyclictest1-21systemd19:50:022
1706129994326,13cyclictest1-21systemd19:45:002
1706129994322,18cyclictest1716344-21sh21:19:102
1706129994322,17cyclictest1709466-21sed20:15:012
1706129994322,16cyclictest1708571-21ls19:55:022
1706129994322,16cyclictest1708571-21ls19:55:022
1706129994321,18cyclictest1795418-21ssh00:22:262
170612799436,8cyclictest23-21ksoftirqd/123:07:471
170612799434,29cyclictest1706851-21munin-run19:24:591
1706127994333,5cyclictest1710721-21sendmail20:40:011
1706127994333,5cyclictest1710721-21sendmail20:40:001
1706127994327,12cyclictest23-21ksoftirqd/121:16:051
1706127994320,14cyclictest309-21in:imuxsock20:45:001
1706127994313,7cyclictest131rcu_preempt23:03:311
1706127994313,7cyclictest131rcu_preempt23:03:301
1706127994312,6cyclictest131rcu_preempt22:25:001
170612499438,30cyclictest0-21swapper/019:44:590
1706124994327,11cyclictest3467602-21systemd-journal21:00:000
170612499431,37cyclictest3467602-21systemd-journal20:00:000
1706124994310,7cyclictest0-21swapper/020:50:560
1706124994310,7cyclictest0-21swapper/020:50:550
1706129994229,10cyclictest30-21ksoftirqd/219:25:002
1706129994214,24cyclictest1711991-21cat21:00:032
170612999421,11cyclictest1711585-21cut20:55:032
170612999421,11cyclictest1711585-21cut20:55:022
1706127994230,10cyclictest23-21ksoftirqd/120:20:001
1706127994226,12cyclictest275-21dbus-daemon21:30:011
170612799422,35cyclictest23-21ksoftirqd/121:36:051
170612799422,35cyclictest23-21ksoftirqd/121:36:051
1706127994222,14cyclictest0-21swapper/122:10:011
1706127994222,14cyclictest0-21swapper/122:10:001
170612499422,9cyclictest1712045-21unin-run21:05:010
1706129994125,12cyclictest1727916-21ssh21:45:382
1706129994123,13cyclictest1706557-21cron19:20:002
1706127994130,8cyclictest3467602-21systemd-journal00:30:321
1706127994129,10cyclictest23-21ksoftirqd/123:46:441
1706127994128,10cyclictest23-21ksoftirqd/100:35:331
1706127994128,10cyclictest23-21ksoftirqd/100:35:321
1706127994127,11cyclictest23-21ksoftirqd/100:20:331
1706127994125,13cyclictest23-21ksoftirqd/121:43:411
1706127994112,8cyclictest131rcu_preempt22:51:171
1706127994112,8cyclictest131rcu_preempt22:51:161
1706124994136,3cyclictest16950irq/224-800000019:33:420
1706129994026,11cyclictest30-21ksoftirqd/221:10:012
1706129994024,11cyclictest1708677-21true20:00:012
170612799403,5cyclictest0-21swapper/123:22:441
1706127994028,9cyclictest1745550-21sshd22:26:441
1706127994028,10cyclictest23-21ksoftirqd/122:56:531
1706127994027,10cyclictest23-21ksoftirqd/119:40:001
1706124994033,4cyclictest16950irq/224-800000020:05:010
170612499401,9cyclictest1710955-21latency_hist20:40:030
1706129993914,21cyclictest1711011-21munin-run20:45:002
170612799393,28cyclictest3467602-21systemd-journal21:05:001
1706127993925,8cyclictest309-21in:imuxsock21:22:411
1706127993925,10cyclictest23-21ksoftirqd/122:47:541
170612799392,4cyclictest131rcu_preempt20:29:591
1706127993924,11cyclictest275-21dbus-daemon00:15:011
170612799392,34cyclictest1768438-21rm23:20:011
1706127993923,11cyclictest275-21dbus-daemon21:00:001
1706127993918,15cyclictest275-21dbus-daemon20:25:021
1706127993918,15cyclictest275-21dbus-daemon20:25:011
1706124993927,7cyclictest309-21in:imuxsock20:35:010
1706124993924,10cyclictest12-21ksoftirqd/020:45:010
170612499391,4cyclictest0-21swapper/020:25:010
170612499391,4cyclictest0-21swapper/020:25:000
1706129993822,12cyclictest1706132-21latency_hist19:10:032
1706129993822,12cyclictest1706132-21latency_hist19:10:022
170612799387,19cyclictest3467602-21systemd-journal19:50:001
170612799384,4cyclictest131rcu_preempt22:40:041
170612799384,4cyclictest131rcu_preempt22:40:031
1706127993830,5cyclictest1776867-21sshd23:39:441
170612799382,14cyclictest1731187-21ssh21:53:421
170612499381,6cyclictest3467602-21systemd-journal20:39:170
1706129993724,10cyclictest1-21systemd21:05:002
170612799377,25cyclictest1710469-21cut20:35:011
170612799372,8cyclictest23-21ksoftirqd/121:34:111
1706127993725,10cyclictest23-21ksoftirqd/100:25:481
1706127993721,11cyclictest3467602-21systemd-journal20:55:011
1706127993721,11cyclictest3467602-21systemd-journal20:55:001
1706124993726,7cyclictest12-21ksoftirqd/019:50:010
1706129993617,16cyclictest364-21snmpd20:28:412
170612999351,30cyclictest1709687-21cat20:20:012
1706124993525,7cyclictest16950irq/224-800000019:30:010
1706124993522,9cyclictest3467602-21systemd-journal19:25:000
1706124993518,13cyclictest1711221-21sshd20:46:410
170612799343,4cyclictest131rcu_preempt19:35:011
1706127993423,8cyclictest23-21ksoftirqd/119:15:011
1706127993419,11cyclictest3467602-21systemd-journal20:14:031
170612499341,30cyclictest0-21swapper/020:11:040
170612799337,19cyclictest1708943-21wc20:05:011
170612499311,26cyclictest1706260-21latency_hist19:10:020
170612499311,26cyclictest1706260-21latency_hist19:10:020
1706127992617,6cyclictest1706052-21latency_hist19:10:031
1706127992617,6cyclictest1706052-21latency_hist19:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional