You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 08:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Fri Nov 28, 2025 00:39:42)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1819671995344,6cyclictest1851200-21systemd-run20:55:012
1819670995142,6cyclictest422-21dbus-daemon22:20:011
1819669995039,7cyclictest1954914-21awk23:20:010
1819669994935,10cyclictest0-21swapper/023:10:010
1819671994839,5cyclictest1966422-21latency_hist23:35:022
1819669994727,13cyclictest12450irq/32-eth023:34:210
1819670994638,5cyclictest1939700-21sh23:00:011
1819669994637,7cyclictest1976462-21sh23:49:120
1819669994637,7cyclictest1976462-21sh23:49:120
1819671994432,7cyclictest0-21swapper/200:39:222
1819670994435,3cyclictest0-21swapper/121:39:231
1819669994436,3cyclictest0-21swapper/023:40:020
1819669994435,2cyclictest0-21swapper/022:15:010
1819669994434,6cyclictest1859753-21cron21:15:010
1819671994336,4cyclictest1935905-21grep22:55:012
1819670994334,6cyclictest1943511-21latency_hist23:05:021
1819669994330,5cyclictest453-21systemd-logind19:35:030
1819671994235,5cyclictest351ktimers/200:20:092
1819671994235,5cyclictest351ktimers/200:20:092
1819670994234,5cyclictest4168385-21cups-browsed21:10:121
1819670994234,5cyclictest2009180-21sh00:34:371
1819670994234,5cyclictest2009180-21sh00:34:371
1819670994234,5cyclictest1928398-21ssh22:45:021
1819669994234,5cyclictest2012614-21latency_hist00:40:020
181966999422,32cyclictest151ktimers/021:44:150
181967199414,27cyclictest1885573-21ssh21:49:222
1819671994132,6cyclictest1997655-21aten_r2power_cu00:19:062
1819670994135,3cyclictest0-21swapper/123:10:011
1819670994133,5cyclictest2002747-21latency_hist00:25:011
1819670994133,5cyclictest2002747-21latency_hist00:25:011
1819670994133,5cyclictest1984972-21sh00:00:011
1819670994133,5cyclictest1855692-21diskmemload22:49:241
1819670994133,5cyclictest1846680-21cron20:40:011
1819670994133,5cyclictest1839756-21fschecks_time20:19:091
1819670994133,5cyclictest1830194-21latency_hist19:45:021
1819670994126,10cyclictest1969022-21latency23:39:101
1819670994033,4cyclictest1828640-21cron19:40:011
1819669994033,4cyclictest1984975-21ssh00:00:010
1819671993931,5cyclictest1950710-21sed23:14:282
1819671993929,8cyclictest1821165-21sh19:15:012
1819671993929,6cyclictest1923953-21mailstats22:39:292
1819671993925,10cyclictest1994621-21latency00:14:122
181967099393,32cyclictest1917013-21sh22:30:011
181967199383,29cyclictest4168385-21cups-browsed19:54:162
1819671993829,6cyclictest4168385-21cups-browsed22:24:322
1819671993829,5cyclictest1930800-21fschecks_time22:49:102
1819671993829,4cyclictest0-21swapper/223:04:212
181967199382,30cyclictest1900774-21ssh22:09:202
181967199382,27cyclictest1976733-21ssh23:49:172
181967199382,27cyclictest1976733-21ssh23:49:172
181967099384,28cyclictest1957935-21ssh23:24:211
1819670993831,4cyclictest1890294-21latency_hist21:55:031
1819670993830,5cyclictest1935628-21head22:54:281
181967099380,11cyclictest28-21ksoftirqd/119:29:161
1819669993830,5cyclictest422-21dbus-daemon00:31:030
1819669993830,5cyclictest422-21dbus-daemon00:31:030
1819669993830,5cyclictest1939083-21mailstats22:59:260
1819669993830,4cyclictest0-21swapper/022:46:570
1819669993829,4cyclictest0-21swapper/000:24:180
1819669993829,4cyclictest0-21swapper/000:24:180
1819671993729,5cyclictest1984982-21munin-run00:00:012
1819671993729,5cyclictest1845226-21latency_hist20:35:022
1819671993726,7cyclictest1961354-21/usr/sbin/munin23:29:122
1819670993730,4cyclictest1924593-21ssh22:40:011
1819670993730,4cyclictest1869008-21kworker/u8:3-rpciod21:47:271
1819670993729,5cyclictest1849695-21munin-plugin-st20:50:011
1819670993729,4cyclictest0-21swapper/121:19:151
1819670993728,6cyclictest4168358-21cupsd23:33:471
1819670993727,6cyclictest4168358-21cupsd23:12:501
1819670993726,8cyclictest1977065-21switchtime23:49:271
1819670993726,8cyclictest1977065-21switchtime23:49:271
181967099370,13cyclictest1852121-21ntpq20:59:171
1819669993730,4cyclictest151ktimers/019:49:110
1819669993729,5cyclictest1985041-21cups-browsed00:29:250
1819669993729,5cyclictest1985041-21cups-browsed00:29:250
1819669993729,5cyclictest1-21systemd00:20:020
1819669993728,5cyclictest4168385-21cups-browsed22:17:460
1819669993722,11cyclictest0-21swapper/023:25:020
1819671993628,5cyclictest421-21avahi-daemon21:35:322
1819671993628,5cyclictest1834611-21head19:59:262
1819671993628,4cyclictest0-21swapper/200:29:062
1819671993628,4cyclictest0-21swapper/200:29:062
181967099364,11cyclictest28-21ksoftirqd/123:29:171
1819670993628,5cyclictest1893991-21munin-plugin-st22:00:011
1819670993628,5cyclictest1893991-21munin-plugin-st22:00:011
1819670993627,6cyclictest1833164-21true19:55:011
1819670993626,7cyclictest1988047-21mailstats00:04:281
1819670993625,8cyclictest1881451-21missed_timers21:44:121
1819670993614,5cyclictest28-21ksoftirqd/122:24:211
181966999363,24cyclictest1995032-21ssh00:14:240
1819669993628,5cyclictest266-21systemd-journal19:45:010
1819669993625,8cyclictest0-21swapper/023:54:110
1819671993528,4cyclictest0-21swapper/221:45:012
1819671993526,6cyclictest1907979-21/usr/sbin/munin22:19:082
1819671993526,4cyclictest0-21swapper/221:24:142
1819671993525,7cyclictest698-21cron20:50:012
1819669993526,5cyclictest151ktimers/023:44:200
181966999352,28cyclictest1927437-21ssh22:44:160
1819671993426,5cyclictest1893992-21sendmail22:00:022
1819671993426,5cyclictest1893992-21sendmail22:00:022
1819671993425,7cyclictest453-21systemd-logind19:10:012
1819671993424,7cyclictest1903933-21aten_r2power_po22:14:052
181967199342,26cyclictest1870224-21ssh21:29:212
1819671993421,4cyclictest4168385-21cups-browsed20:37:322
181967099348,8cyclictest28-21ksoftirqd/122:34:191
181967099347,15cyclictest28-21ksoftirqd/121:29:101
181967099344,23cyclictest1972973-21ssh23:44:151
1819670993427,4cyclictest957-21mta-sts-daemon21:09:171
1819670993427,4cyclictest1851184-21sh20:55:011
1819670993422,8cyclictest1953327-21basename23:19:071
1819670993419,4cyclictest57-21kcompactd022:12:221
1819670993414,7cyclictest171rcu_preempt00:27:401
1819670993414,7cyclictest171rcu_preempt00:27:401
1819669993427,5cyclictest1849601-21tail20:49:290
1819669993427,4cyclictest1825810-21latency_hist19:30:030
1819669993426,5cyclictest0-21swapper/022:40:010
181966999342,25cyclictest1987850-21ssh00:04:220
181966999342,20cyclictest1897023-21ssh22:04:200
1819669993417,15cyclictest1892301-21/usr/sbin/munin21:59:080
1819669993417,15cyclictest1892301-21/usr/sbin/munin21:59:080
1819669993416,13cyclictest151ktimers/020:04:140
1819671993326,3cyclictest0-21swapper/222:34:152
1819671993324,6cyclictest957-21mta-sts-daemon23:50:452
1819671993323,7cyclictest0-21swapper/223:40:022
1819670993326,4cyclictest956-21mta-sts-daemon19:34:251
1819670993325,5cyclictest1831677-21systemd-run19:50:021
1819670993322,8cyclictest1981309-21grep23:55:001
1819670993320,6cyclictest171rcu_preempt21:33:451
1819669993328,3cyclictest1942511-21needreboot23:04:200
1819669993325,5cyclictest4168385-21cups-browsed22:34:050
1819669993321,6cyclictest0-21swapper/021:35:320
1819669993321,10cyclictest1847872-21switchtime20:44:260
1819669993319,11cyclictest421-21avahi-daemon21:29:280
1819671993227,3cyclictest0-21swapper/222:05:012
1819671993227,2cyclictest0-21swapper/219:40:012
1819671993224,5cyclictest956-21mta-sts-daemon20:29:162
1819671993224,5cyclictest956-21mta-sts-daemon20:29:162
1819671993224,5cyclictest1927589-21ssh22:44:222
1819671993224,5cyclictest1848205-21idleruntime-cro20:45:012
1819671993221,8cyclictest1972568-21sh23:44:082
181967199321,5cyclictest171rcu_preempt00:34:262
181967199321,5cyclictest171rcu_preempt00:34:262
181967099329,8cyclictest28-21ksoftirqd/100:40:011
1819670993222,7cyclictest1866216-21missed_timers21:24:121
1819670993215,3cyclictest171rcu_preempt22:01:581
1819669993226,3cyclictest0-21swapper/021:20:290
1819669993225,3cyclictest0-21swapper/022:29:180
1819669993224,5cyclictest151ktimers/020:29:200
1819669993224,5cyclictest151ktimers/020:29:200
1819669993224,5cyclictest1-21systemd19:10:010
181966999322,22cyclictest1912263-21ssh22:24:170
1819669993221,8cyclictest0-21swapper/021:34:280
1819669993219,10cyclictest1845118-21tail20:34:310
1819671993125,3cyclictest1828905-21kworker/u8:0-xprtiod21:19:192
1819671993123,5cyclictest1840692-21idleruntime-cro20:20:012
1819671993122,6cyclictest1939927-21latency_hist23:00:032
1819671993120,8cyclictest1952718-21sh23:18:022
1819671993120,4cyclictest0-21swapper/223:09:222
181967199311,23cyclictest1957745-21ssh23:24:142
181967099317,5cyclictest28-21ksoftirqd/100:14:241
181967099313,23cyclictest1998466-21ssh00:19:241
1819670993123,6cyclictest271ktimers/120:29:191
1819670993123,6cyclictest271ktimers/120:29:191
1819670993123,5cyclictest1854117-21unixbench_multi21:04:311
1819670993115,7cyclictest28-21ksoftirqd/122:09:201
1819670993113,6cyclictest1819783-21latency_hist19:10:021
181966999312,24cyclictest1885467-21ssh21:49:170
1819669993119,9cyclictest0-21swapper/020:15:010
1819669993117,11cyclictest491-21gdbus21:19:240
181967199303,20cyclictest1987810-21ssh00:04:212
1819671993025,3cyclictest57-21kcompactd022:29:212
1819671993023,4cyclictest1991847-21unixbench_singl00:09:282
1819671993023,4cyclictest1824158-21cron19:25:012
1819671993022,5cyclictest1825288-21sendmail_mailtr19:29:232
181967099309,4cyclictest271ktimers/119:59:251
181967099307,7cyclictest271ktimers/120:09:271
1819670993022,5cyclictest1985041-21cups-browsed00:08:261
1819670993022,5cyclictest1-21systemd20:15:021
181967099301,9cyclictest28-21ksoftirqd/119:19:191
1819670993015,3cyclictest171rcu_preempt19:14:171
1819670993012,3cyclictest171rcu_preempt20:25:021
1819670993012,3cyclictest171rcu_preempt20:25:021
1819669993023,4cyclictest0-21swapper/022:54:250
1819669993023,3cyclictest0-21swapper/022:10:010
1819669993021,6cyclictest0-21swapper/019:40:020
1819671992924,4cyclictest1819668-21cyclictest21:12:512
1819671992923,3cyclictest0-21swapper/221:09:292
1819671992922,4cyclictest0-21swapper/220:10:012
1819670992920,5cyclictest1845216-21systemd-run20:35:021
1819670992916,3cyclictest171rcu_preempt19:25:021
1819670992915,3cyclictest171rcu_preempt20:44:191
1819669992922,4cyclictest1824280-21latency_hist19:25:030
1819671992823,2cyclictest0-21swapper/220:05:012
1819671992822,4cyclictest1819668-21cyclictest21:55:022
1819671992821,4cyclictest1830705-21uniq19:49:062
1819671992821,3cyclictest0-21swapper/221:34:172
1819671992821,3cyclictest0-21swapper/219:34:152
1819671992820,5cyclictest1822657-21systemctl19:20:012
1819671992819,6cyclictest1854109-21unixbench_multi21:04:312
181967099284,8cyclictest28-21ksoftirqd/120:04:161
1819669992822,4cyclictest1990685-21head00:09:050
1819669992821,3cyclictest1817012-21kworker/dying19:14:190
1819669992818,3cyclictest0-21swapper/023:14:280
1819671992719,5cyclictest266-21systemd-journal20:15:012
181966999274,15cyclictest437-21mta-sts-daemon19:59:160
1819669992722,2cyclictest0-21swapper/021:51:400
1819669992721,4cyclictest1842110-21tail20:24:280
1819669992721,4cyclictest1842110-21tail20:24:280
1819669992721,3cyclictest0-21swapper/021:04:120
1819669992720,5cyclictest1961737-21ntpq23:29:210
1819669992720,5cyclictest1-21systemd20:40:020
1819669992717,5cyclictest151ktimers/020:54:170
1819669992617,7cyclictest1-21systemd20:19:090
1819671992520,3cyclictest1732494-21kworker/u8:10-nfsiod19:44:102
1819671992515,8cyclictest351ktimers/220:24:172
1819671992515,8cyclictest351ktimers/220:24:172
1819669992519,3cyclictest0-21swapper/019:54:280
181967199242,5cyclictest171rcu_preempt20:59:212
1819669992419,2cyclictest0-21swapper/019:20:010
1819669992417,5cyclictest1845418-21kworker/u8:3+nfsiod20:55:420
1819669992313,2cyclictest0-21swapper/021:10:010
1819669992114,5cyclictest1836795-21if_wlan020:09:100
181967299183,8cyclictest0-21swapper/300:29:243
181967299183,8cyclictest0-21swapper/300:29:243
181967299182,8cyclictest0-21swapper/323:59:163
181967299181,9cyclictest0-21swapper/321:59:203
181967299181,9cyclictest0-21swapper/321:59:203
181967299172,8cyclictest0-21swapper/321:54:183
181967299171,7cyclictest0-21swapper/323:24:143
181967299163,7cyclictest0-21swapper/322:04:183
181967299163,7cyclictest0-21swapper/321:39:233
181967299163,7cyclictest0-21swapper/300:34:313
181967299163,7cyclictest0-21swapper/300:34:313
181967299162,6cyclictest0-21swapper/300:04:133
181967299161,8cyclictest0-21swapper/322:49:163
181967299153,7cyclictest0-21swapper/322:59:203
181967299152,8cyclictest0-21swapper/323:04:223
181967299152,7cyclictest0-21swapper/323:49:193
181967299152,7cyclictest0-21swapper/323:49:193
181967299152,7cyclictest0-21swapper/321:44:203
181967299152,6cyclictest0-21swapper/319:39:153
181967299152,6cyclictest0-21swapper/300:19:183
181967299151,8cyclictest0-21swapper/323:54:253
181967299151,8cyclictest0-21swapper/323:09:193
181967299142,7cyclictest0-21swapper/321:24:143
181967299142,6cyclictest0-21swapper/322:34:143
181967299142,6cyclictest0-21swapper/300:39:173
181967299141,7cyclictest0-21swapper/323:39:153
181967299141,6cyclictest0-21swapper/320:44:193
181967299141,6cyclictest0-21swapper/320:24:153
181967299141,6cyclictest0-21swapper/320:24:153
181967299141,6cyclictest0-21swapper/319:49:143
181967299132,8cyclictest0-21swapper/322:14:193
181967299132,7cyclictest0-21swapper/322:24:143
181967299132,6cyclictest0-21swapper/322:39:203
181967299131,7cyclictest0-21swapper/322:29:183
181967299131,6cyclictest0-21swapper/322:19:153
181967299131,6cyclictest0-21swapper/321:04:213
181967299131,6cyclictest0-21swapper/319:59:133
181967299131,6cyclictest0-21swapper/319:44:143
181967299131,5cyclictest0-21swapper/320:49:163
181967299122,7cyclictest0-21swapper/300:24:163
181967299122,7cyclictest0-21swapper/300:24:163
181967299122,6cyclictest0-21swapper/322:55:033
181967299121,8cyclictest0-21swapper/322:44:163
181967299121,7cyclictest0-21swapper/323:44:173
181967299121,7cyclictest0-21swapper/323:34:183
181967299121,7cyclictest0-21swapper/323:19:193
181967299121,7cyclictest0-21swapper/322:09:203
181967299121,7cyclictest0-21swapper/321:34:163
181967299121,7cyclictest0-21swapper/300:14:183
181967299121,6cyclictest0-21swapper/321:19:083
181967299121,6cyclictest0-21swapper/321:09:163
181967299121,6cyclictest0-21swapper/320:54:173
181967299121,5cyclictest0-21swapper/320:34:193
181967299121,5cyclictest0-21swapper/320:13:023
181967299121,5cyclictest0-21swapper/319:54:163
181967299120,6cyclictest0-21swapper/319:29:153
181967299113,6cyclictest0-21swapper/321:29:183
181967299113,5cyclictest0-21swapper/319:34:173
181967299112,5cyclictest0-21swapper/323:14:153
181967299111,6cyclictest0-21swapper/321:47:273
181967299111,6cyclictest0-21swapper/320:40:013
181967299111,6cyclictest0-21swapper/319:14:183
181967299111,5cyclictest0-21swapper/320:04:123
181967299102,5cyclictest0-21swapper/321:10:123
181967299101,6cyclictest0-21swapper/320:29:173
181967299101,6cyclictest0-21swapper/320:29:173
181967299101,5cyclictest0-21swapper/323:25:123
181967299101,5cyclictest0-21swapper/320:09:153
181967299101,5cyclictest0-21swapper/319:24:173
181967299101,5cyclictest0-21swapper/319:19:133
18196729991,6cyclictest0-21swapper/300:09:153
18196729991,5cyclictest0-21swapper/320:56:123
18196729991,5cyclictest0-21swapper/320:16:093
18196729981,4cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional