You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-23 - 03:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Dec 23, 2025 00:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17397532920,6chrt4497-21PK-Backend20:45:112
17397532920,6chrt4497-21PK-Backend20:45:112
170534128665,13sleep20-21swapper/219:09:182
17465772830,6sleep10-21swapper/121:05:021
17620362800,12sleep01762033-21cron21:50:000
17760162780,6sleep10-21swapper/122:30:011
170538127639,14sleep10-21swapper/119:09:501
170527126839,23sleep00-21swapper/019:08:230
17712562670,7sleep24497-21PK-Backend22:15:132
170525926646,14sleep30-21swapper/319:08:143
197399600,9rtkit-daemon1972-21rtkit-daemon23:57:191
197399600,9rtkit-daemon1972-21rtkit-daemon22:41:330
197399600,9rtkit-daemon1972-21rtkit-daemon22:41:330
197399600,9rtkit-daemon1972-21rtkit-daemon00:35:050
197399590,9rtkit-daemon1972-21rtkit-daemon19:26:550
197399590,9rtkit-daemon1972-21rtkit-daemon00:16:200
197399590,9rtkit-daemon1972-21rtkit-daemon00:16:200
197399580,9rtkit-daemon1972-21rtkit-daemon00:34:460
197399570,8rtkit-daemon1972-21rtkit-daemon00:06:570
197399530,8rtkit-daemon1972-21rtkit-daemon19:16:510
17903382530,5sleep01790340-21dump-pmu-power23:10:010
170553599520,48cyclictest1972-21rtkit-daemon19:15:051
1705535995112,18cyclictest17136642if_enp2s019:30:221
197399460,38rtkit-daemon2078-21gmain23:15:140
17505302460,7chrt0-21swapper/121:15:161
170553599460,31cyclictest1765136-21vmstat21:55:301
17203652450,9sleep234-21ksoftirqd/219:50:102
170553599450,21cyclictest1751873-21find21:20:101
1705532994527,13cyclictest1735728-21cron20:35:000
17123482440,8sleep34497-21PK-Backend19:25:283
197399430,24rtkit-daemon931-21irqbalance00:37:571
17984742430,4sleep10-21swapper/123:30:301
1705542994328,10cyclictest1798980-21apt-get23:35:003
1705535994320,18cyclictest1777240-21users22:30:311
170553599430,15cyclictest1766313-21latency22:00:221
1705542994213,6cyclictest2478-21colord21:47:583
1705535994226,12cyclictest1802740-21cron23:45:001
170553599420,18cyclictest1813834-21hddtemp_smartct00:15:201
170553599420,18cyclictest1813834-21hddtemp_smartct00:15:191
170553599420,17cyclictest1722140-21/usr/sbin/munin19:55:261
197399410,29rtkit-daemon4497-21PK-Backend22:24:482
17731092410,7sleep0161rcuc/022:20:210
1705539994128,7cyclictest34-21ksoftirqd/221:47:592
1705539994110,15cyclictest4497-21PK-Backend23:10:012
170553299412,34cyclictest1797907-21diskstats23:30:170
197399400,28rtkit-daemon4497-21PK-Backend19:48:133
197399400,27rtkit-daemon4497-21PK-Backend20:00:172
197399400,27rtkit-daemon4497-21PK-Backend20:00:172
17429982400,7chrt4497-21PK-Backend20:55:012
170553599400,34cyclictest1754203-21memory21:25:221
170553599400,23cyclictest4497-21PK-Backend21:46:221
1705532994021,6cyclictest13-21ksoftirqd/000:00:000
1705532994021,14cyclictest438-20systemd-journal20:15:010
197399390,27rtkit-daemon4497-21PK-Backend20:26:372
197399390,27rtkit-daemon4497-21PK-Backend20:05:062
197399390,26rtkit-daemon4497-21PK-Backend22:29:502
18045692390,6sleep10-21swapper/123:50:021
1705535993924,11cyclictest1712925-21cron19:30:001
170553599391,33cyclictest1810420-21if_err_enp2s000:05:201
1705532993924,11cyclictest1716388-21cron19:40:000
1705532993924,11cyclictest1716388-21cron19:40:000
197399380,27rtkit-daemon4497-21PK-Backend20:19:503
197399380,27rtkit-daemon4497-21PK-Backend19:37:042
197399380,27rtkit-daemon4497-21PK-Backend19:37:042
197399380,26rtkit-daemon4497-21PK-Backend21:43:593
197399380,25rtkit-daemon4497-21PK-Backend19:53:093
197399380,25rtkit-daemon0-21swapper/220:21:202
170554299382,14cyclictest17903412sh23:10:013
1705539993828,6cyclictest34-21ksoftirqd/200:00:012
1705539993828,6cyclictest1054-21snmpd19:45:572
1705539993827,8cyclictest34-21ksoftirqd/221:55:002
1705539993821,13cyclictest1779980-21df22:40:152
1705539993821,13cyclictest1779980-21df22:40:152
1705539993821,12cyclictest1804907-21cut23:50:152
1705535993822,11cyclictest438-20systemd-journal00:21:101
1705535993821,13cyclictest1800799-40dpkg23:37:531
170553599381,17cyclictest1736882-21python320:35:271
170553599381,16cyclictest1763398-21users21:50:321
170553599380,16cyclictest0-21swapper/122:45:291
1705532993824,10cyclictest1755066-21cron21:30:000
197399370,26rtkit-daemon0-21swapper/322:15:403
197399370,24rtkit-daemon0-21swapper/222:03:532
197399370,23rtkit-daemon0-21swapper/220:10:042
1705542993711,9cyclictest974-21in:imuxsock20:35:003
1705535993721,11cyclictest1724646-21seq20:00:261
1705535993721,11cyclictest1724646-21seq20:00:261
1705535993720,12cyclictest438-20systemd-journal21:44:551
1705535993719,14cyclictest1706870-21systemd-udevd19:10:531
1705535993719,14cyclictest1706870-21systemd-udevd19:10:531
1705535993718,14cyclictest1808365-21tr00:00:161
1705532993721,12cyclictest1749915-21pluginstate21:15:010
1705532993721,12cyclictest1749915-21pluginstate21:15:010
1705532993720,13cyclictest1721684-40lsb_release19:52:580
1705532993720,12cyclictest1706867-21systemd-udevd19:10:540
1705532993720,12cyclictest1706867-21systemd-udevd19:10:530
170553299371,32cyclictest0-21swapper/020:45:060
170553299371,32cyclictest0-21swapper/020:45:060
197399360,25rtkit-daemon4497-21PK-Backend19:41:502
197399360,25rtkit-daemon0-21swapper/220:56:532
197399360,24rtkit-daemon4497-21PK-Backend20:58:043
1705542993614,6cyclictest141rcu_preempt21:25:143
1705539993626,7cyclictest28750irq/122-ahci[0020:00:012
1705535993625,7cyclictest1769598-21df22:10:151
1705535993622,10cyclictest26-21ksoftirqd/119:35:141
1705535993622,10cyclictest26-21ksoftirqd/119:35:141
1705535993620,12cyclictest1742231-21ntpq20:50:251
1705535993620,12cyclictest1731077-21expr20:20:141
1705535993620,12cyclictest1730101-21timerwakeupswit20:15:291
1705535993620,11cyclictest1728233-21swap20:10:301
1705535993619,12cyclictest1720197-21grep19:50:021
1705535993618,14cyclictest1743366-21grep20:55:151
1705535993618,13cyclictest438-20systemd-journal00:35:001
1705535993618,13cyclictest1784399-21ntpq22:50:261
170553599360,33cyclictest0-21swapper/122:38:241
1705532993623,9cyclictest13-21ksoftirqd/022:05:010
1705532993622,10cyclictest1748143-21cron21:10:000
1705532993622,10cyclictest13-21ksoftirqd/022:47:020
1705532993622,10cyclictest13-21ksoftirqd/021:30:240
1705532993621,11cyclictest1736733-21ntpq20:35:230
1705532993620,12cyclictest1792068-21/usr/sbin/munin23:15:010
1705532993620,12cyclictest1792068-21/usr/sbin/munin23:15:010
1705532993619,12cyclictest1774418-21awk22:25:130
1705532993617,14cyclictest1713575-21df19:30:170
1705532993617,14cyclictest1710335-21memory19:20:230
197399350,24rtkit-daemon4497-21PK-Backend21:27:502
1705542993523,7cyclictest1748150-21ls21:10:003
1705542993522,9cyclictest1768950-21cron22:14:553
1705542993521,10cyclictest42-21ksoftirqd/323:00:153
1705542993520,11cyclictest1781083-21ls22:45:003
1705542993520,11cyclictest1781083-21ls22:45:003
1705542993519,12cyclictest1802165-21python323:40:303
1705542993518,12cyclictest466-21systemd-udevd21:02:413
1705539993525,6cyclictest34-21ksoftirqd/200:10:142
1705539993524,6cyclictest34-21ksoftirqd/223:25:002
1705539993524,6cyclictest34-21ksoftirqd/220:35:292
1705539993520,11cyclictest1752609-21tr21:20:222
1705539993518,13cyclictest1808986-21ntpq00:00:272
1705535993520,11cyclictest1812476-21awk00:10:261
1705535993520,11cyclictest1716915-21expr19:40:121
1705535993520,10cyclictest1773113-21basename22:20:221
1705535993518,13cyclictest1795590-21cut23:25:021
1705535993517,14cyclictest1756843-21unin-run21:35:011
1705535993513,18cyclictest1739949-21tr20:45:141
1705535993513,18cyclictest1739949-21tr20:45:141
1705532993525,6cyclictest1739128-40apt-check20:40:540
1705532993524,7cyclictest1758544-21turbostat.cron21:40:000
1705532993524,7cyclictest1758544-21turbostat.cron21:40:000
1705532993521,10cyclictest1785135-21cron22:55:000
1705532993521,10cyclictest1729271-21cut20:15:130
1705532993520,11cyclictest1742216-21ntp_kernel_err20:50:250
1705532993519,12cyclictest1779344-21sh22:40:000
1705532993519,12cyclictest1771716-21ntpq22:15:240
197399340,24rtkit-daemon0-21swapper/319:34:043
170554299347,5cyclictest141rcu_preempt22:35:003
170554299346,24cyclictest0-21swapper/319:20:313
170554299343,12cyclictest1706865-21systemd-udevd19:10:543
170554299343,12cyclictest1706865-21systemd-udevd19:10:543
1705542993422,7cyclictest42-21ksoftirqd/321:51:223
1705542993422,6cyclictest42-21ksoftirqd/322:55:013
1705542993421,9cyclictest1813148-21cron00:15:003
1705542993420,10cyclictest1796440-21cat23:25:213
1705542993420,10cyclictest1775890-21munin-plugin-st22:30:003
170554299341,9cyclictest4497-21PK-Backend23:10:153
170554299341,9cyclictest4497-21PK-Backend23:10:153
1705542993419,11cyclictest1817065-21tr00:25:143
1705542993419,11cyclictest1810527-21cat00:05:213
1705542993418,12cyclictest1768995-21systemd22:10:013
1705542993418,12cyclictest1738586-21memory20:40:223
170554299341,28cyclictest438-20systemd-journal00:40:013
1705542993412,19cyclictest4497-21PK-Backend22:39:013
1705539993424,6cyclictest34-21ksoftirqd/222:30:182
1705539993424,6cyclictest34-21ksoftirqd/221:30:282
1705539993423,6cyclictest34-21ksoftirqd/223:03:232
1705539993423,6cyclictest34-21ksoftirqd/222:52:092
1705539993419,11cyclictest1802761-21munin-run23:45:002
1705539993419,11cyclictest1791648-21sed23:10:312
1705539993419,11cyclictest1791648-21sed23:10:312
170553999341,30cyclictest4497-21PK-Backend21:05:542
1705535993422,8cyclictest2069-21gnome-shell23:10:091
1705535993422,8cyclictest2069-21gnome-shell23:10:091
1705535993420,10cyclictest1721100-21ntp_kernel_err19:50:241
1705535993420,10cyclictest1710475-21ntp_kernel_pll_19:20:251
1705535993420,10cyclictest1137-21runrttasks22:15:231
1705535993419,11cyclictest1749647-21sshd21:12:091
1705535993419,11cyclictest1749647-21sshd21:12:091
1705535993418,12cyclictest1-21systemd20:35:001
1705535993418,11cyclictest438-20systemd-journal22:57:091
1705535993418,11cyclictest1795951-21tr23:25:121
1705535993417,12cyclictest1732953-21idleruntime20:25:181
1705535993416,14cyclictest1817052-21grep00:25:151
170553599340,30cyclictest1757629-21latency21:35:201
170553599340,30cyclictest1757629-21latency21:35:201
1705532993423,7cyclictest1811888-21cpuspeed_turbos00:10:140
1705532993420,10cyclictest13-21ksoftirqd/023:45:300
1705532993419,11cyclictest1731388-21cat20:20:210
1705532993417,13cyclictest1723605-21pool-udisksd19:59:560
1705532993416,12cyclictest438-20systemd-journal21:50:090
17503722330,6sleep3411rcuc/321:15:143
1705542993325,5cyclictest4497-21PK-Backend22:22:223
1705542993322,6cyclictest42-21ksoftirqd/322:45:183
1705542993321,8cyclictest1808989-21ntp_states00:00:273
170554299331,28cyclictest4497-21PK-Backend19:15:223
170554299331,28cyclictest4497-21PK-Backend00:34:013
170554299331,28cyclictest4497-21PK-Backend00:24:193
170554299331,28cyclictest4497-21PK-Backend00:17:263
170554299331,28cyclictest4497-21PK-Backend00:17:263
1705539993326,5cyclictest0-21swapper/222:38:462
1705539993323,6cyclictest34-21ksoftirqd/221:41:202
1705539993323,6cyclictest34-21ksoftirqd/219:10:272
1705539993323,6cyclictest34-21ksoftirqd/219:10:272
1705539993323,5cyclictest34-21ksoftirqd/222:10:002
1705539993323,5cyclictest34-21ksoftirqd/220:41:062
1705539993323,5cyclictest34-21ksoftirqd/200:25:002
1705539993322,7cyclictest1798205-21grep23:30:232
1705539993322,6cyclictest34-21ksoftirqd/222:57:082
1705539993322,6cyclictest34-21ksoftirqd/219:25:232
1705539993321,9cyclictest1800797-40apt-check23:38:032
1705539993319,10cyclictest1783380-21sed22:50:012
1705539993318,11cyclictest1749273-21cut21:10:262
1705539993318,11cyclictest1749273-21cut21:10:262
1705539993316,13cyclictest1819306-21netstat00:30:242
1705535993319,10cyclictest1747589-21expr21:05:281
1705535993318,11cyclictest1788118-21sed23:00:301
1705535993318,10cyclictest1792461-21grep23:15:161
1705535993317,12cyclictest1738943-21cat20:40:291
1705532993326,4cyclictest0-21swapper/021:03:440
1705532993325,5cyclictest0-21swapper/021:44:230
1705532993322,7cyclictest1815582-21sort00:20:200
1705532993319,10cyclictest1777331-21anacron22:31:310
1705532993318,12cyclictest1787541-21fw_packets23:00:190
1705532993318,11cyclictest976-21rs:main0
1705532993318,11cyclictest1795807-21grep23:25:120
1705532993318,11cyclictest1768005-21cut22:05:210
1705532993318,11cyclictest1725345-21cron20:05:000
1705532993318,11cyclictest1725345-21cron20:05:000
1705532993318,11cyclictest1719545-21sshd19:46:090
1705532993318,11cyclictest1033-21gdbus23:38:070
1705532993317,12cyclictest1817622-21ntpq00:25:250
170553299331,29cyclictest0-21swapper/023:54:290
170554299322,6cyclictest17927102sleep323:15:193
170554299322,26cyclictest1742251-21ntp_states20:50:263
170554299322,26cyclictest1715339-21hddtemp_smartct19:35:203
170554299322,26cyclictest1715339-21hddtemp_smartct19:35:203
1705542993221,6cyclictest42-21ksoftirqd/320:25:103
1705542993220,8cyclictest1736903-21grep20:35:283
1705542993219,9cyclictest1724535-21sed20:00:233
1705542993219,9cyclictest1724535-21sed20:00:233
1705542993217,11cyclictest923-21dbus-daemon22:05:003
1705542993216,11cyclictest1728316-21grep20:10:313
170554299321,4cyclictest4497-21PK-Backend21:39:123
170554299321,4cyclictest4497-21PK-Backend21:39:123
170554299321,28cyclictest4497-21PK-Backend20:24:473
1705539993224,5cyclictest4497-21PK-Backend23:47:332
1705539993222,7cyclictest1137-21runrttasks22:10:122
1705539993222,6cyclictest34-21ksoftirqd/220:30:222
1705539993222,5cyclictest34-21ksoftirqd/223:27:492
1705539993222,5cyclictest34-21ksoftirqd/223:15:372
1705539993221,6cyclictest34-21ksoftirqd/219:15:262
1705539993220,6cyclictest34-21ksoftirqd/200:35:222
1705539993216,12cyclictest1765503-21sh22:00:012
1705539993215,12cyclictest438-20systemd-journal21:01:092
170553999321,11cyclictest924-21NetworkManager21:15:282
170553999321,10cyclictest4497-21PK-Backend00:25:172
1705535993217,11cyclictest1804951-21cstates23:50:151
1705535993216,12cyclictest1767696-21tr22:05:141
170553599321,29cyclictest0-21swapper/122:42:461
170553599321,29cyclictest0-21swapper/122:42:461
170553599321,28cyclictest0-21swapper/123:07:291
170553599321,27cyclictest0-21swapper/120:09:201
1705532993225,4cyclictest4497-21PK-Backend22:59:270
1705532993223,6cyclictest1794225-21tr23:20:160
1705532993218,10cyclictest1764908-21sensors_temp21:55:270
1705532993217,11cyclictest1752452-21cat21:20:200
1705532993216,12cyclictest1751142-21grep21:15:290
1705532993216,12cyclictest1717068-21smartctl19:40:170
1705532993216,11cyclictest1725679-21awk20:05:140
1705532993215,12cyclictest438-20systemd-journal20:59:550
170554299312,7cyclictest1794466-21hddtemp_smartct23:20:203
1705542993124,4cyclictest4497-21PK-Backend23:55:143
1705542993123,5cyclictest0-21swapper/322:59:533
1705542993119,8cyclictest1054-21snmpd20:06:023
1705542993116,11cyclictest1803521-21sed23:45:203
1705542993116,11cyclictest1723632-21ls20:00:003
1705542993116,10cyclictest1752569-21cut21:20:223
1705539993121,5cyclictest34-21ksoftirqd/220:20:002
170553299315,22cyclictest1-21systemd23:45:010
170554299303,24cyclictest1739930-21tr20:45:133
170554299303,24cyclictest1739930-21tr20:45:133
170554299302,7cyclictest1806016-21python323:52:413
170554299302,7cyclictest1764786-21ntp_states21:55:243
1705539993019,7cyclictest1709737-21sort19:20:132
170553999301,7cyclictest4497-21PK-Backend19:34:072
1705539993015,11cyclictest1813838-21idleruntime00:15:202
1705539993015,11cyclictest1813838-21idleruntime00:15:202
1705532993016,10cyclictest1769375-21grep22:10:130
1705532993014,12cyclictest1733170-21cat20:25:210
170554299293,6cyclictest141rcu_preempt23:40:003
1705542992922,4cyclictest4497-21PK-Backend21:33:533
1705542992919,7cyclictest42-21ksoftirqd/319:43:023
1705542992912,12cyclictest466-21systemd-udevd21:10:413
1705542992912,12cyclictest466-21systemd-udevd21:10:413
1705539992918,7cyclictest4497-21PK-Backend21:36:502
1705539992918,7cyclictest4497-21PK-Backend21:36:502
1705539992916,9cyclictest1810261-21expr00:05:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional