You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-23 - 01:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Tue Sep 22, 2026 12:42:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1698224994836,10cyclictest1717700-21cron08:15:010
1698224994833,12cyclictest1748323-21unixbench_singl09:31:310
1698225994738,6cyclictest421-21dbus-daemon09:55:011
1698224994438,4cyclictest1735232-21tr09:11:120
169822499441,13cyclictest14-21ksoftirqd/010:36:180
1698224994336,4cyclictest1862167-21latency_hist12:40:010
1698224994235,4cyclictest1698235-21systemd-run07:10:010
1698224994234,5cyclictest420-21avahi-daemon11:54:210
1698224994234,5cyclictest420-21avahi-daemon11:54:210
1698224994234,5cyclictest1792214-21systemd-run10:45:020
1698226994134,4cyclictest0-21swapper/211:00:012
1698225994133,5cyclictest1764723-21latency_hist10:00:011
1698225994032,5cyclictest1819704-21latency_hist11:30:021
1698225994032,5cyclictest1723579-21sed08:31:281
1698224994033,4cyclictest1734213-21grep09:10:010
1698226993929,7cyclictest917984-21cups-browsed12:26:072
1698226993929,7cyclictest917984-21cups-browsed12:26:072
1698224993932,4cyclictest1846892-21ssh12:15:010
1698224993930,6cyclictest917984-21cups-browsed12:31:250
1698224993930,6cyclictest917984-21cups-browsed12:31:250
1698224993930,6cyclictest1810439-21latency_hist11:15:020
1698224993930,6cyclictest1003-21mta-sts-daemon12:06:190
1698224993923,13cyclictest0-21swapper/011:06:190
1698226993831,4cyclictest1849937-21munin-run12:20:012
1698226993831,4cyclictest1818324-21grep11:26:282
1698226993830,5cyclictest1708829-21latency_hist07:45:022
1698226993829,6cyclictest917984-21cups-browsed10:49:342
1698226993822,8cyclictest1769081-21ntpq10:06:172
1698225993831,4cyclictest1704228-21kworker/u8:4-rpciod08:40:011
1698225993830,5cyclictest421-21dbus-daemon09:35:011
1698225993829,6cyclictest1811672-21ntp_kernel_err11:16:131
1698225993812,6cyclictest171rcu_preempt11:41:201
1698224993832,4cyclictest1831728-21ssh11:50:020
1698224993832,4cyclictest1831728-21ssh11:50:020
1698224993832,4cyclictest1825621-21cron11:40:010
1698224993829,6cyclictest917984-21cups-browsed09:58:460
1698224993823,12cyclictest0-21swapper/007:20:010
1698224993818,7cyclictest14-21ksoftirqd/008:36:170
1698224993813,18cyclictest1817812-21ssh11:26:180
1698226993728,3cyclictest0-21swapper/210:26:192
169822599372,30cyclictest1790374-21ssh10:41:141
1698224993732,3cyclictest1002-21mta-sts-daemon10:31:240
1698224993731,4cyclictest1822787-21latency_hist11:35:030
169822499372,32cyclictest1848166-21ssh12:16:160
1698226993629,4cyclictest1811310-21fschecks_count11:16:082
1698226993629,4cyclictest1810446-21munin-plugin-st11:15:012
1698226993628,5cyclictest1825616-21ssh11:40:012
1698226993628,5cyclictest1734221-21ssh09:10:012
1698226993626,7cyclictest1731229-21sh09:00:022
1698225993630,3cyclictest0-21swapper/109:25:011
1698225993628,5cyclictest1801310-21munin-plugin-st11:00:011
1698225993627,6cyclictest265-21systemd-journal07:30:001
1698225993625,8cyclictest1744060-21aten2.4-expect09:26:051
1698225993621,12cyclictest271ktimers/109:46:211
1698224993627,5cyclictest1002-21mta-sts-daemon11:22:480
1698224993627,5cyclictest1002-21mta-sts-daemon11:22:480
169822499362,30cyclictest420-21avahi-daemon07:26:200
1698224993618,5cyclictest171rcu_preempt10:51:250
1698224993616,3cyclictest171rcu_preempt10:56:150
1698224993610,10cyclictest14-21ksoftirqd/009:21:200
1698226993528,4cyclictest1816520-21munin-plugin-st11:25:012
1698226993528,4cyclictest1816520-21munin-plugin-st11:25:012
1698226993527,5cyclictest1837786-21ssh12:00:002
1698226993527,5cyclictest1804571-21latency_hist11:05:042
1698226993527,5cyclictest1705731-21systemd-run07:35:012
1698226993526,6cyclictest1737350-21latency_hist09:15:022
1698225993527,5cyclictest1699722-21systemd-run07:15:011
1698225993524,8cyclictest694-21cron08:00:011
169822499353,24cyclictest1781295-21sh10:26:180
1698224993527,6cyclictest1728741-21fschecks_time08:51:100
1698224993527,5cyclictest1837842-21systemd-run12:00:010
1698224993526,6cyclictest1751141-21switchtime09:36:290
1698224993525,4cyclictest0-21swapper/011:16:130
1698224993523,9cyclictest1709968-21timerandwakeup07:46:300
1698226993429,2cyclictest0-21swapper/209:45:012
1698226993427,4cyclictest1821316-21ssh11:31:262
1698226993427,4cyclictest1740318-21idleruntime-cro09:20:002
1698226993427,4cyclictest1740318-21idleruntime-cro09:20:002
1698226993427,4cyclictest1002-21mta-sts-daemon09:59:552
1698226993427,3cyclictest0-21swapper/210:21:182
1698226993423,8cyclictest1771211-21sshd10:10:302
1698225993427,4cyclictest1749519-21kworker/u8:3-xprtiod10:01:281
1698225993426,5cyclictest1740406-21wc09:20:021
1698225993426,5cyclictest1740406-21wc09:20:021
1698225993422,10cyclictest1846621-21sh12:14:101
1698224993428,4cyclictest1702872-21latency_hist07:25:020
1698224993427,4cyclictest1734210-21diskmemload09:45:010
1698224993425,7cyclictest1761685-21sh09:55:010
1698224993425,6cyclictest1827321-21timerandwakeup11:41:270
1698224993424,5cyclictest151ktimers/007:51:240
1698224993423,5cyclictest151ktimers/007:41:150
169822499341,14cyclictest14-21ksoftirqd/009:46:190
1698226993327,3cyclictest0-21swapper/210:36:142
1698226993326,5cyclictest1743460-21latency_hist09:25:022
1698226993326,4cyclictest1001-21mta-sts-daemon11:05:582
1698226993326,4cyclictest0-21swapper/207:30:012
1698226993325,5cyclictest1834749-21ssh11:55:002
1698226993325,5cyclictest1834749-21ssh11:55:002
1698226993325,5cyclictest1719207-21grep08:20:012
1698226993325,5cyclictest1701209-21munin-plugin-st07:20:012
1698225993327,3cyclictest0-21swapper/109:00:011
1698225993326,4cyclictest1001-21mta-sts-daemon10:21:051
1698225993326,3cyclictest0-21swapper/110:36:191
1698225993325,5cyclictest1830134-21proc_pri11:46:241
1698225993325,5cyclictest1830134-21proc_pri11:46:241
1698225993325,5cyclictest1816591-21latency_hist11:25:021
1698225993325,5cyclictest1816591-21latency_hist11:25:021
169822499338,10cyclictest14-21ksoftirqd/010:06:170
169822499332,8cyclictest14-21ksoftirqd/010:11:220
1698224993327,4cyclictest1804359-21cron11:05:010
1698224993327,3cyclictest0-21swapper/009:26:240
1698224993327,3cyclictest0-21swapper/008:10:010
1698224993323,5cyclictest151ktimers/007:56:180
169822699323,22cyclictest1766062-21ssh10:01:182
1698226993226,3cyclictest0-21swapper/209:36:192
169822699322,5cyclictest171rcu_preempt09:31:182
1698226993224,5cyclictest665-21snmpd08:15:022
1698226993222,7cyclictest694-21cron09:50:012
1698225993226,4cyclictest1698223-21cyclictest11:01:101
1698225993225,4cyclictest1805259-21fschecks_time11:06:101
1698225993224,5cyclictest1850957-21kernelversion12:21:111
1698225993224,5cyclictest1701214-21idleruntime-cro07:20:011
1698225993223,5cyclictest1706235-21fschecks_count07:36:101
169822599321,23cyclictest1808736-21sh11:11:181
1698224993219,10cyclictest0-21swapper/007:36:130
1698224993217,3cyclictest171rcu_preempt10:46:190
1698224993216,10cyclictest12650irq/32-eth010:16:230
1698224993215,4cyclictest171rcu_preempt08:01:170
1698226993125,3cyclictest0-21swapper/209:30:012
1698226993125,3cyclictest0-21swapper/208:45:012
1698226993125,3cyclictest0-21swapper/208:45:012
1698226993124,5cyclictest1776129-21sshd10:17:312
1698226993124,4cyclictest1003-21mta-sts-daemon07:46:112
1698226993124,3cyclictest0-21swapper/207:11:222
1698226993123,5cyclictest0-21swapper/210:51:122
1698226993121,5cyclictest1843888-21sh12:10:012
1698225993123,5cyclictest1775772-21head10:16:291
1698225993123,5cyclictest1001-21mta-sts-daemon08:16:071
1698225993122,6cyclictest0-21swapper/112:01:191
169822599312,24cyclictest1793440-21ssh10:46:151
1698225993121,5cyclictest1737430-21latency_hist09:15:031
1698225993110,6cyclictest171rcu_preempt10:11:211
1698224993126,2cyclictest0-21swapper/012:01:250
1698224993125,4cyclictest1-21systemd09:00:020
1698224993124,4cyclictest1727369-21kernelversion08:46:110
1698224993122,3cyclictest171rcu_preempt10:21:160
1698224993116,2cyclictest171rcu_preempt08:16:200
169822499311,13cyclictest14-21ksoftirqd/007:11:190
1698224993110,6cyclictest151ktimers/008:41:170
1698224993110,6cyclictest151ktimers/008:41:170
1698224993110,3cyclictest171rcu_preempt12:26:140
1698224993110,3cyclictest171rcu_preempt12:26:140
1698224993110,2cyclictest171rcu_preempt09:16:220
1698224993110,2cyclictest171rcu_preempt09:16:220
1698226993023,5cyclictest1839632-21unixbench-2d12:01:302
1698226993023,4cyclictest1844773-21gltestperf12:11:112
1698226993023,4cyclictest0-21swapper/208:35:012
1698226993023,3cyclictest0-21swapper/211:41:202
1698226993022,5cyclictest1713202-21sendmail08:00:012
1698226993020,7cyclictest1724236-21fschecks_time08:36:092
1698225993024,3cyclictest0-21swapper/112:00:011
1698225993024,3cyclictest0-21swapper/109:45:011
1698225993024,3cyclictest0-21swapper/109:10:011
1698225993023,4cyclictest1781797-21unixbench_multi10:26:281
1698225993023,4cyclictest1001-21mta-sts-daemon11:53:161
1698225993023,4cyclictest1001-21mta-sts-daemon11:53:161
1698225993022,5cyclictest1768426-21aten_r2power_en10:06:051
1698225993022,5cyclictest1751251-21timerwakeupswit09:36:291
169822599301,25cyclictest1820924-21ssh11:31:201
169822599301,20cyclictest1842220-21ssh12:06:241
1698225993010,4cyclictest171rcu_preempt10:31:251
169822499308,7cyclictest1720184-21ntpq08:21:180
169822499308,7cyclictest14-21ksoftirqd/007:31:160
169822499304,23cyclictest1766174-21ssh10:01:230
1698224993016,11cyclictest0-21swapper/009:01:270
169822499301,12cyclictest14-21ksoftirqd/012:22:510
1698226992922,5cyclictest0-21swapper/210:41:022
1698226992922,4cyclictest1722216-21munin-plugin-st08:30:012
169822699292,20cyclictest1784270-21ssh10:31:192
1698226992921,5cyclictest1-21systemd09:05:022
169822599294,7cyclictest171rcu_preempt08:51:191
1698225992922,4cyclictest1002-21mta-sts-daemon12:26:161
1698225992922,4cyclictest1002-21mta-sts-daemon12:26:161
1698225992920,5cyclictest1726704-21cron08:45:001
1698225992920,5cyclictest1726704-21cron08:45:001
1698225992919,8cyclictest1795756-21sh10:51:011
1698225992919,7cyclictest1702039-21needreboot07:21:121
1698226992822,3cyclictest0-21swapper/212:31:172
1698226992822,3cyclictest0-21swapper/212:31:172
1698226992821,4cyclictest1729626-21unixbench_singl08:51:282
1698226992820,5cyclictest1707197-21cron07:40:002
1698225992821,5cyclictest0-21swapper/109:01:281
1698225992821,3cyclictest0-21swapper/108:06:281
1698225992820,5cyclictest1859935-21sort12:36:111
1698225992820,5cyclictest1857505-21mailstats12:31:221
1698225992820,5cyclictest1857505-21mailstats12:31:221
169822499289,3cyclictest171rcu_preempt08:31:150
169822499287,5cyclictest151ktimers/008:26:160
1698226992721,4cyclictest1698223-21cyclictest08:25:012
1698226992721,4cyclictest1698223-21cyclictest08:01:202
1698226992721,4cyclictest1000-21mta-sts-daemon12:37:132
1698226992721,3cyclictest0-21swapper/212:21:152
1698226992721,3cyclictest0-21swapper/209:51:222
1698226992720,4cyclictest1711489-21timerwakeupswit07:51:292
1698226992720,4cyclictest1000-21mta-sts-daemon11:49:092
1698226992720,4cyclictest1000-21mta-sts-daemon11:49:092
1698225992720,4cyclictest1823184-21aten2.4-expect11:36:051
1698225992719,5cyclictest1711721-21systemd-run07:55:011
1698225992718,5cyclictest1848077-21needreboot12:16:131
1698226992620,4cyclictest1715247-21fschecks_time08:06:102
1698225992617,6cyclictest1714253-21proc_pri08:01:231
1698225992610,3cyclictest171rcu_preempt07:46:241
1698225992610,3cyclictest171rcu_preempt07:41:121
1698225992610,3cyclictest171rcu_preempt07:31:161
1698226992516,6cyclictest1728101-21sed08:46:272
1698225992519,3cyclictest0-21swapper/108:15:011
1698225992417,4cyclictest0-21swapper/108:30:011
1698225992411,3cyclictest171rcu_preempt08:21:191
169822799223,10cyclictest0-21swapper/309:56:233
1698226992217,2cyclictest0-21swapper/207:10:012
1698226992215,5cyclictest1702710-21cron07:25:012
169822799212,8cyclictest0-21swapper/310:56:153
1698225992115,3cyclictest0-21swapper/108:46:051
169822799201,12cyclictest0-21swapper/311:16:133
169822799201,11cyclictest0-21swapper/311:31:193
1698225992014,4cyclictest0-21swapper/107:10:021
169822799193,9cyclictest0-21swapper/310:36:283
169822799193,9cyclictest0-21swapper/310:26:163
169822799192,8cyclictest0-21swapper/311:21:223
169822799192,8cyclictest0-21swapper/311:21:223
169822799192,7cyclictest0-21swapper/307:45:023
169822799192,10cyclictest0-21swapper/310:51:223
169822799192,10cyclictest0-21swapper/308:25:023
169822799191,9cyclictest0-21swapper/310:16:183
169822799191,9cyclictest0-21swapper/309:51:213
169822799191,11cyclictest0-21swapper/311:51:183
169822799191,11cyclictest0-21swapper/311:51:183
169822799191,10cyclictest0-21swapper/312:06:253
169822799191,10cyclictest0-21swapper/309:21:183
169822799183,7cyclictest0-21swapper/309:31:253
169822799182,9cyclictest0-21swapper/312:26:143
169822799182,9cyclictest0-21swapper/312:26:143
169822799182,9cyclictest0-21swapper/310:41:133
169822799182,8cyclictest0-21swapper/312:21:043
169822799182,8cyclictest0-21swapper/307:26:203
169822799181,9cyclictest0-21swapper/311:01:223
169822799181,8cyclictest0-21swapper/311:26:173
169822799181,10cyclictest0-21swapper/312:11:213
169822799181,10cyclictest0-21swapper/310:06:153
169822799173,8cyclictest0-21swapper/310:21:113
169822799172,9cyclictest0-21swapper/309:41:133
169822799172,8cyclictest0-21swapper/310:11:253
169822799171,9cyclictest0-21swapper/311:11:203
169822799171,9cyclictest0-21swapper/310:05:023
169822799171,9cyclictest0-21swapper/308:55:023
169822799171,8cyclictest0-21swapper/308:56:203
169822799171,8cyclictest0-21swapper/308:26:183
169822799171,8cyclictest0-21swapper/307:36:213
169822799171,10cyclictest0-21swapper/311:46:203
169822799171,10cyclictest0-21swapper/311:46:203
169822799171,10cyclictest0-21swapper/309:46:183
169822799171,10cyclictest0-21swapper/309:26:203
169822799164,8cyclictest0-21swapper/311:41:143
169822799163,6cyclictest0-21swapper/309:36:153
169822799162,8cyclictest0-21swapper/312:31:143
169822799162,8cyclictest0-21swapper/312:31:143
169822799162,8cyclictest0-21swapper/311:56:243
169822799162,8cyclictest0-21swapper/311:06:233
169822799162,7cyclictest0-21swapper/309:16:223
169822799162,7cyclictest0-21swapper/309:16:223
169822799162,10cyclictest0-21swapper/312:16:163
169822799161,9cyclictest0-21swapper/309:11:163
169822799161,7cyclictest0-21swapper/312:36:173
169822799161,7cyclictest0-21swapper/307:51:243
169822799161,6cyclictest0-21swapper/309:06:133
169822799161,10cyclictest0-21swapper/310:31:223
169822799153,8cyclictest0-21swapper/310:46:153
169822799153,6cyclictest0-21swapper/309:01:203
169822799152,8cyclictest0-21swapper/312:01:233
169822799152,6cyclictest0-21swapper/307:11:183
169822799151,7cyclictest0-21swapper/308:01:223
169822799151,7cyclictest0-21swapper/307:56:303
169822799151,6cyclictest0-21swapper/308:41:163
169822799151,6cyclictest0-21swapper/308:41:163
169822799151,6cyclictest0-21swapper/308:16:163
169822799151,6cyclictest0-21swapper/307:31:183
169822799142,6cyclictest0-21swapper/308:31:193
169822799141,7cyclictest0-21swapper/308:06:193
169822799135,5cyclictest0-21swapper/307:21:173
169822799132,7cyclictest0-21swapper/311:36:213
169822799132,6cyclictest0-21swapper/307:50:023
169822799131,7cyclictest0-21swapper/308:36:173
169822799131,6cyclictest0-21swapper/308:46:143
169822799131,6cyclictest0-21swapper/307:16:163
169822799121,6cyclictest0-21swapper/308:11:173
16982279991,5cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional