You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-13 - 13:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Apr 13, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
180959321270,3sleep00-21swapper/021:35:020
176847721120,3sleep03056-21PK-Backend19:35:190
178069621020,3sleep00-21swapper/020:10:200
139099850,71rtkit-daemon0-21swapper/319:05:043
175788428140,13sleep10-21swapper/119:06:051
175812426544,14sleep00-21swapper/019:09:090
175798126444,14sleep20-21swapper/219:07:182
18511102500,1sleep20-21swapper/223:30:192
18524412480,3sleep30-21swapper/323:35:133
18721872440,4sleep20-21swapper/200:30:182
18077732440,3sleep1251rcuc/121:30:011
17616352380,3sleep30-21swapper/319:15:213
175835699302,6cyclictest0-21swapper/219:55:112
1758356993010,5cyclictest1793843-21sed20:50:002
175835699298,7cyclictest436-20systemd-journal22:50:002
175835699298,6cyclictest1836677-21cat22:50:142
175835699292,6cyclictest0-21swapper/200:35:122
175835699292,5cyclictest0-21swapper/223:50:132
1758359992813,13cyclictest1820365-21unin-run22:05:023
175835699288,5cyclictest1787829-21cat20:30:242
175835699286,6cyclictest1848585-21find23:25:102
175835699281,9cyclictest1862502-21perf00:05:002
1758359992712,13cyclictest1832765-21unin-run22:40:003
175835699279,6cyclictest1803425-21sshd21:15:192
175835699278,5cyclictest1848415-21wc23:25:002
175835699278,5cyclictest1801345-21expr21:10:122
175835699277,5cyclictest1804653-21expr21:20:132
175835699277,5cyclictest1804653-21expr21:20:132
175835699273,9cyclictest1769439-21apt19:40:102
175835699272,10cyclictest0-21swapper/223:55:112
175835699271,9cyclictest1772660-21idleruntime-cro19:50:002
1758356992711,4cyclictest1839043-21processes22:55:192
175835699267,5cyclictest1794458-21ls20:50:172
175835699262,8cyclictest82332-21update-manager21:55:062
175835699261,8cyclictest1758694-21cpuspeed_turbos19:10:142
175835699261,8cyclictest1-21systemd20:40:002
175835699261,7cyclictest1778115-21apt20:05:112
1758356992613,3cyclictest1867090-21python300:15:192
175835699260,9cyclictest1842332-21tail23:05:182
175835999253,12cyclictest0-21swapper/322:20:543
175835699258,3cyclictest0-21swapper/223:00:212
175835699252,6cyclictest1825525-21apt-get22:20:032
175835699251,9cyclictest1832134-21python322:35:212
175835699251,8cyclictest1857268-21sh23:50:002
175835699251,8cyclictest1769157-21cron19:40:012
175835699251,7cyclictest1834469-21sh22:45:002
175835699251,7cyclictest1785057-21apt20:25:122
175835699251,6cyclictest1809418-21sleep21:35:002
175835699251,10cyclictest1780992-21python320:11:052
175835699250,4cyclictest978-21snmpd20:42:542
175835699250,10cyclictest1819002-21cut22:00:122
18565282240,8sleep126-21ksoftirqd/123:45:181
1758359992412,10cyclictest1808304-21df_inode21:30:133
175835699247,4cyclictest0-21swapper/220:20:172
175835699245,5cyclictest1765092-21/usr/sbin/munin19:25:202
175835699241,7cyclictest1766807-21/usr/sbin/munin19:30:202
175835699241,5cyclictest18681912chrt00:20:032
1758356992410,3cyclictest1795760-21/usr/sbin/munin20:55:182
1758354992415,7cyclictest1836197-21cron22:50:001
18431822230,2sleep00-21swapper/023:10:000
175835999235,16cyclictest1009-21lldpd22:00:003
175835999230,12cyclictest0-21swapper/320:10:013
175835699238,4cyclictest1799737-21kernelversion21:05:162
175835699236,3cyclictest0-21swapper/221:50:012
175835699232,8cyclictest1846005-21python323:15:242
175835699231,8cyclictest1811630-21apt-get21:40:052
1758356992317,5cyclictest0-21swapper/219:15:272
175835699231,10cyclictest1828661-21python322:25:202
18031172220,3sleep0161rcuc/021:15:160
175835999226,14cyclictest1867302-21users00:15:213
175835999226,14cyclictest1843617-21ls23:10:123
175835999226,14cyclictest1838781-21memory22:55:173
1758359992220,1cyclictest0-21swapper/320:45:113
1758359992220,1cyclictest0-21swapper/319:59:083
1758359992220,1cyclictest0-21swapper/319:41:563
1758359992220,1cyclictest0-21swapper/300:35:333
1758359992220,1cyclictest0-21swapper/300:01:403
175835999221,6cyclictest1846460-21sshd23:18:193
1758359992212,7cyclictest436-20systemd-journal21:08:193
175835999220,21cyclictest0-21swapper/321:18:573
175835699221,7cyclictest1798391-21python321:00:212
175835699221,6cyclictest1821467-21python322:05:202
1758356992215,5cyclictest883-21dbus-daemon22:35:002
1758354992219,2cyclictest3056-21PK-Backend22:53:361
175835499220,20cyclictest3056-21PK-Backend19:45:051
1758350992220,1cyclictest0-21swapper/023:43:090
1758350992220,1cyclictest0-21swapper/023:33:360
1758350992220,1cyclictest0-21swapper/022:40:430
1758350992220,1cyclictest0-21swapper/020:27:000
1758350992220,1cyclictest0-21swapper/019:49:140
1758350992220,1cyclictest0-21swapper/000:31:150
1758350992212,8cyclictest2254-21gdbus20:40:040
175835099220,21cyclictest0-21swapper/023:23:020
175835999217,4cyclictest1850836-21uname23:30:163
175835999217,3cyclictest1764281-21cpuspeed19:25:123
175835999215,14cyclictest1858413-21python323:50:223
1758359992119,1cyclictest0-21swapper/322:51:303
1758359992119,1cyclictest0-21swapper/321:54:253
1758359992119,1cyclictest0-21swapper/319:51:503
1758359992119,1cyclictest0-21swapper/300:13:503
175835999210,20cyclictest42-21ksoftirqd/320:17:183
175835999210,20cyclictest0-21swapper/323:05:573
175835999210,20cyclictest0-21swapper/321:04:103
175835699212,6cyclictest1854681-21python323:40:212
1758356992119,1cyclictest0-21swapper/221:53:582
1758356992114,1cyclictest0-21swapper/220:19:082
1758354992119,1cyclictest3056-21PK-Backend21:57:191
1758354992119,1cyclictest3056-21PK-Backend19:59:581
1758354992119,1cyclictest3056-21PK-Backend19:50:181
1758354992119,1cyclictest3056-21PK-Backend00:36:521
1758354992118,2cyclictest3056-21PK-Backend23:37:231
1758354992118,2cyclictest3056-21PK-Backend20:52:301
1758354992115,4cyclictest1800723-21turbostat.cron21:10:011
175835499211,1cyclictest141rcu_preempt22:20:011
175835499210,19cyclictest3056-21PK-Backend22:37:001
175835499210,19cyclictest3056-21PK-Backend21:36:301
175835499210,19cyclictest3056-21PK-Backend19:41:401
175835499210,19cyclictest3056-21PK-Backend00:23:061
175835499210,19cyclictest3056-21PK-Backend00:14:001
1758350992119,1cyclictest3056-21PK-Backend20:15:310
1758350992119,1cyclictest0-21swapper/023:37:280
1758350992119,1cyclictest0-21swapper/021:53:250
1758350992119,1cyclictest0-21swapper/021:10:410
1758350992119,1cyclictest0-21swapper/021:07:490
1758350992119,1cyclictest0-21swapper/021:00:320
1758350992119,1cyclictest0-21swapper/020:46:000
1758350992119,1cyclictest0-21swapper/020:36:530
1758350992119,1cyclictest0-21swapper/020:33:150
1758350992119,1cyclictest0-21swapper/019:42:590
1758350992119,1cyclictest0-21swapper/000:19:530
175835099210,20cyclictest0-21swapper/023:53:300
175835099210,20cyclictest0-21swapper/023:19:100
175835099210,20cyclictest0-21swapper/022:09:130
175835099210,20cyclictest0-21swapper/021:25:270
175835099210,20cyclictest0-21swapper/019:59:090
175835099210,20cyclictest0-21swapper/019:26:310
175835099210,20cyclictest0-21swapper/000:22:440
18162432200,3sleep13056-21PK-Backend21:50:221
175835999208,10cyclictest1777853-21latency_hist20:05:013
175835999206,12cyclictest1829905-21df_inode22:30:143
175835999205,5cyclictest1863871-21taskset00:05:373
175835999203,15cyclictest1809785-21cat21:35:143
175835999203,15cyclictest0-21swapper/300:20:013
1758359992018,1cyclictest0-21swapper/322:25:283
1758359992018,1cyclictest0-21swapper/322:19:333
1758359992016,2cyclictest1870013-21cpu00:25:133
1758359992012,6cyclictest1-21systemd19:35:003
175835999200,1cyclictest3056-21PK-Backend21:45:063
175835999200,1cyclictest0-21swapper/322:46:373
175835999200,1cyclictest0-21swapper/321:14:173
1758356992018,1cyclictest0-21swapper/221:38:302
1758356992018,1cyclictest0-21swapper/200:25:332
1758356992018,1cyclictest0-21swapper/200:08:162
175835699201,6cyclictest1807064-21python321:25:202
175835699201,13cyclictest1852078-21apt23:35:132
175835699200,19cyclictest0-21swapper/200:10:532
175835499209,9cyclictest3056-21PK-Backend21:20:001
1758354992018,1cyclictest3056-21PK-Backend23:59:271
1758354992018,1cyclictest3056-21PK-Backend22:30:311
1758354992018,1cyclictest3056-21PK-Backend21:47:251
1758354992018,1cyclictest3056-21PK-Backend21:32:041
1758354992018,1cyclictest3056-21PK-Backend19:39:021
1758354992018,1cyclictest3056-21PK-Backend00:31:241
1758354992018,1cyclictest3056-21PK-Backend00:04:541
1758354992018,1cyclictest0-21swapper/120:18:301
175835499200,1cyclictest3056-21PK-Backend21:12:031
175835499200,1cyclictest0-21swapper/123:07:361
175835499200,19cyclictest3056-21PK-Backend22:42:281
175835499200,19cyclictest3056-21PK-Backend22:09:021
175835499200,19cyclictest3056-21PK-Backend20:35:471
175835499200,18cyclictest3056-21PK-Backend21:42:461
175835499200,18cyclictest3056-21PK-Backend20:21:071
175835499200,18cyclictest3056-21PK-Backend00:17:391
1758350992018,1cyclictest0-21swapper/021:45:550
1758350992018,1cyclictest0-21swapper/019:32:310
1758350992018,1cyclictest0-21swapper/019:14:540
175835099200,1cyclictest3056-21PK-Backend22:26:170
175835099200,1cyclictest0-21swapper/023:01:520
175835099200,1cyclictest0-21swapper/022:19:340
175835099200,19cyclictest0-21swapper/022:51:190
175835099200,19cyclictest0-21swapper/022:34:490
175835099200,19cyclictest0-21swapper/022:13:530
175835099200,19cyclictest0-21swapper/022:00:550
175835099200,19cyclictest0-21swapper/021:58:050
175835099200,19cyclictest0-21swapper/020:02:460
175835099200,19cyclictest0-21swapper/000:12:230
175835099200,19cyclictest0-21swapper/000:05:540
175835099200,19cyclictest0-21swapper/000:03:030
139099200,14rtkit-daemon0-21swapper/223:12:412
175835999195,12cyclictest1833751-21ntp_states22:40:203
175835999194,14cyclictest1854533-21grep23:40:193
175835999194,13cyclictest1791094-21latency20:40:163
1758359991917,1cyclictest42-21ksoftirqd/320:32:263
1758359991917,1cyclictest0-21swapper/319:48:123
1758359991912,5cyclictest1859275-21/usr/sbin/munin23:55:123
1758359991912,5cyclictest1822885-21cat22:10:163
175835999191,16cyclictest1812032-21taskset21:40:133
1758359991911,6cyclictest1779840-21awk20:10:113
1758359991910,7cyclictest1805051-21memory21:20:183
1758359991910,7cyclictest1805051-21memory21:20:183
175835999190,5cyclictest42-21ksoftirqd/323:45:193
175835999190,1cyclictest0-21swapper/323:24:153
175835999190,1cyclictest0-21swapper/323:01:043
175835999190,18cyclictest0-21swapper/323:28:083
175835999190,18cyclictest0-21swapper/320:58:423
175835999190,18cyclictest0-21swapper/320:38:413
175835999190,18cyclictest0-21swapper/300:34:323
175835999190,17cyclictest1762457-21taskset19:20:033
175835999190,17cyclictest1762457-21taskset19:20:033
1758356991914,4cyclictest0-21swapper/219:23:112
1758356991914,4cyclictest0-21swapper/219:23:112
1758354991917,1cyclictest3056-21PK-Backend23:32:431
1758354991917,1cyclictest3056-21PK-Backend22:56:271
1758354991917,1cyclictest3056-21PK-Backend21:01:231
1758354991917,1cyclictest3056-21PK-Backend20:33:101
1758354991917,1cyclictest0-21swapper/121:20:221
1758354991917,1cyclictest0-21swapper/121:20:221
1758354991916,2cyclictest3056-21PK-Backend20:59:331
1758354991916,2cyclictest3056-21PK-Backend19:13:071
175835499190,1cyclictest3056-21PK-Backend23:42:511
175835499190,1cyclictest3056-21PK-Backend23:10:271
175835499190,1cyclictest3056-21PK-Backend23:04:451
175835499190,1cyclictest3056-21PK-Backend22:03:341
175835499190,1cyclictest3056-21PK-Backend20:11:281
175835499190,1cyclictest3056-21PK-Backend19:15:031
175835499190,1cyclictest0-21swapper/123:51:091
175835499190,1cyclictest0-21swapper/120:40:141
175835499190,1cyclictest0-21swapper/119:21:391
175835499190,1cyclictest0-21swapper/119:21:381
175835499190,18cyclictest3056-21PK-Backend20:48:041
175835499190,18cyclictest0-21swapper/122:20:581
1758350991917,1cyclictest0-21swapper/023:10:380
1758350991917,1cyclictest0-21swapper/022:56:110
1758350991917,1cyclictest0-21swapper/022:36:510
1758350991917,1cyclictest0-21swapper/000:38:460
1758350991914,3cyclictest1095-21runrttasks19:15:020
1758350991913,5cyclictest1871280-21sshd00:28:200
1758350991913,4cyclictest1813164-21pluginstate21:45:000
1758350991912,5cyclictest883-21dbus-daemon20:55:000
1758350991911,6cyclictest1811129-21apt-get21:40:000
1758350991911,6cyclictest1778555-21awk20:05:150
175835099190,1cyclictest3056-21PK-Backend19:51:050
175835099190,1cyclictest0-21swapper/021:22:360
175835099190,1cyclictest0-21swapper/021:22:360
175835099190,1cyclictest0-21swapper/020:59:310
139099190,14rtkit-daemon0-21swapper/222:11:472
175835999186,9cyclictest0-21swapper/320:25:123
175835999186,10cyclictest1758529-21/usr/sbin/munin19:10:113
1758359991816,1cyclictest0-21swapper/319:37:303
1758359991810,6cyclictest1806723-21ls21:25:163
175835699182,12cyclictest1824041-21/usr/sbin/munin22:15:222
1758356991811,5cyclictest1777876-21systemd-run20:05:012
1758354991816,1cyclictest3056-21PK-Backend20:26:211
1758354991816,1cyclictest3056-21PK-Backend20:08:501
1758354991815,2cyclictest3056-21PK-Backend23:29:531
175835099186,3cyclictest141rcu_preempt23:30:000
1758350991816,1cyclictest3056-21PK-Backend22:23:260
1758350991810,6cyclictest1855814-21apt23:45:120
139099180,2rtkit-daemon1389-21rtkit-daemon00:06:371
1758359991715,1cyclictest0-21swapper/320:20:583
1758359991712,3cyclictest1794592-21latency20:50:183
1758359991710,5cyclictest1820698-21wc22:05:123
1758356991715,1cyclictest0-21swapper/219:54:072
1758354991715,1cyclictest3056-21PK-Backend23:17:571
1758354991715,1cyclictest3056-21PK-Backend00:25:571
1758354991712,3cyclictest1776900-21wc20:00:171
1758350991715,1cyclictest0-21swapper/019:20:220
1758350991715,1cyclictest0-21swapper/019:20:210
1758350991713,3cyclictest13-21ksoftirqd/000:00:010
1758354991614,1cyclictest3056-21PK-Backend22:29:301
1758354991614,1cyclictest3056-21PK-Backend22:10:031
1758350991610,5cyclictest1784120-21grep20:20:210
175835099160,14cyclictest1835405-21grep22:45:170
1758354991510,2cyclictest26-21ksoftirqd/119:30:201
175835499150,13cyclictest1765706-21grep19:30:001
1758354991412,1cyclictest3056-21PK-Backend23:20:471
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional