You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 21:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Nov 21, 2025 12:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
178348223040,7sleep30-21swapper/307:05:153
18693092950,17sleep342-21ksoftirqd/311:05:323
162399910,27rtkit-daemon0-21swapper/208:50:352
162399910,27rtkit-daemon0-21swapper/208:50:342
18030242900,4sleep10-21swapper/108:00:031
18264682890,5sleep30-21swapper/309:05:163
178442628661,17sleep20-21swapper/207:06:402
18673762830,5sleep21753-21gnome-shell11:00:262
178460927559,10sleep10-21swapper/107:08:591
178451227438,9sleep00-21swapper/007:07:450
162399740,5rtkit-daemon0-21swapper/310:04:243
18385042710,21sleep126-21ksoftirqd/109:40:121
162399710,34rtkit-daemon1753-21gnome-shell07:50:313
18599802700,6sleep00-21swapper/010:40:170
1784824996049,8cyclictest13-21ksoftirqd/008:10:020
162399600,9rtkit-daemon1622-21rtkit-daemon08:49:243
18919272590,8chrt1753-21gnome-shell12:10:152
162399580,9rtkit-daemon1622-21rtkit-daemon08:03:383
162399580,8rtkit-daemon1622-21rtkit-daemon11:56:483
18230672570,17sleep342-21ksoftirqd/308:55:153
18230672570,17sleep342-21ksoftirqd/308:55:153
162399570,8rtkit-daemon1622-21rtkit-daemon12:34:023
162399570,8rtkit-daemon1622-21rtkit-daemon12:16:192
18484352560,6sleep0161rcuc/010:06:510
178483599560,52cyclictest1622-21rtkit-daemon07:34:063
162399560,8rtkit-daemon1622-21rtkit-daemon11:18:463
162399560,8rtkit-daemon1622-21rtkit-daemon08:44:333
162399550,9rtkit-daemon1622-21rtkit-daemon08:15:223
18852632540,5chrt0-21swapper/011:50:240
1784835995422,27cyclictest42-21ksoftirqd/311:27:163
178483599540,50cyclictest1622-21rtkit-daemon08:34:173
162399540,8rtkit-daemon1622-21rtkit-daemon07:28:173
18890792520,6sleep32197014-21PK-Backend12:00:263
178483599522,34cyclictest1847713-21irqstats10:05:223
178483599520,48cyclictest1622-21rtkit-daemon08:09:053
162399520,8rtkit-daemon1622-21rtkit-daemon09:28:113
18934522510,24sleep342-21ksoftirqd/312:15:023
1784835995127,20cyclictest1753-21gnome-shell12:24:483
178483599510,47cyclictest1622-21rtkit-daemon08:20:143
178483599510,47cyclictest1622-21rtkit-daemon07:37:333
178483599510,47cyclictest1622-21rtkit-daemon07:37:323
1784824995140,8cyclictest13-21ksoftirqd/010:00:470
18570882500,4sleep30-21swapper/310:30:293
18570882500,4sleep30-21swapper/310:30:293
1784835995030,16cyclictest1753-21gnome-shell11:30:373
1784824995016,7cyclictest141rcu_preempt09:15:400
162399500,8rtkit-daemon1622-21rtkit-daemon11:35:412
178482999492,16cyclictest17531gnome-shell10:20:021
162399490,35rtkit-daemon2197014-21PK-Backend09:41:232
18337422480,6chrt0-21swapper/109:25:191
178483599482,17cyclictest0-21swapper/308:35:313
1784835994813,10cyclictest42-21ksoftirqd/309:45:363
1784835994813,10cyclictest42-21ksoftirqd/309:45:363
1784829994825,19cyclictest1825008-21df09:00:141
1784824994820,9cyclictest13-21ksoftirqd/009:11:480
1784824994820,15cyclictest13-21ksoftirqd/012:16:500
1784835994721,5cyclictest42-21ksoftirqd/307:18:093
1784829994721,22cyclictest1877722-40lsb_release11:30:201
1784824994720,14cyclictest141rcu_preempt10:23:450
18375052460,13sleep0161rcuc/009:35:250
18024732460,11sleep31802472-21timerandwakeup07:55:293
1784835994625,17cyclictest860997-21snmpd10:19:163
1784835994625,10cyclictest42-21ksoftirqd/310:53:063
1784835994623,6cyclictest42-21ksoftirqd/309:15:003
1784835994620,10cyclictest42-21ksoftirqd/311:00:273
1784829994624,18cyclictest1796267-21/usr/sbin/munin07:40:311
178482499469,9cyclictest13-21ksoftirqd/011:39:120
178482499469,8cyclictest13-21ksoftirqd/008:50:180
178482499469,8cyclictest13-21ksoftirqd/008:50:170
1784824994618,8cyclictest13-21ksoftirqd/008:41:500
1784824994616,10cyclictest13-21ksoftirqd/010:47:350
18134052450,9sleep13242253-21nautilus08:30:011
178483599451,7cyclictest141rcu_preempt07:15:023
1784829994519,22cyclictest1835406-21cat09:30:141
1784824994511,6cyclictest141rcu_preempt11:02:300
1784824994511,17cyclictest1839772-21taskset09:42:320
1784835994432,8cyclictest1753-21gnome-shell12:28:513
1784835994425,7cyclictest42-21ksoftirqd/312:15:123
1784829994420,20cyclictest1846961-21latency_hist10:05:011
1784829994418,21cyclictest1815884-21cat08:35:131
1784829994417,9cyclictest1880947-21cut11:40:011
1784824994418,5cyclictest141rcu_preempt08:38:430
1784824994414,6cyclictest141rcu_preempt08:25:110
1784824994414,10cyclictest13-21ksoftirqd/008:00:590
1784824994413,5cyclictest141rcu_preempt10:59:550
1784824994412,7cyclictest141rcu_preempt09:30:460
1784824994412,7cyclictest141rcu_preempt09:06:300
1784824994412,6cyclictest141rcu_preempt09:26:340
162399440,31rtkit-daemon1753-21gnome-shell10:47:493
162399440,10rtkit-daemon1753-21gnome-shell09:40:553
1784835994334,6cyclictest42-21ksoftirqd/309:20:203
1784835994326,6cyclictest42-21ksoftirqd/312:07:083
1784835994322,6cyclictest42-21ksoftirqd/307:45:273
1784835994322,14cyclictest42-21ksoftirqd/310:22:313
1784835994318,10cyclictest42-21ksoftirqd/309:39:053
1784835994317,10cyclictest42-21ksoftirqd/311:38:313
178482999437,7cyclictest26-21ksoftirqd/109:38:001
178482999436,6cyclictest141rcu_preempt10:41:441
1784829994323,6cyclictest1873757-21tr11:20:151
1784829994312,13cyclictest141rcu_preempt08:45:321
1784829994311,17cyclictest26-21ksoftirqd/108:08:351
178482499437,19cyclictest13-21ksoftirqd/011:30:370
178482499436,15cyclictest2197014-21PK-Backend09:20:400
178482499434,9cyclictest13-21ksoftirqd/007:45:260
1784824994331,7cyclictest13-21ksoftirqd/011:30:030
1784824994327,11cyclictest1855231-21ps10:25:260
1784824994319,6cyclictest141rcu_preempt10:31:390
1784824994319,6cyclictest141rcu_preempt10:31:390
1784824994318,6cyclictest141rcu_preempt07:17:430
1784824994315,9cyclictest13-21ksoftirqd/009:04:380
1784824994315,6cyclictest141rcu_preempt07:54:300
1784824994314,5cyclictest141rcu_preempt07:55:560
1784824994312,8cyclictest13-21ksoftirqd/008:16:150
1784824994312,13cyclictest141rcu_preempt08:58:250
1784824994312,13cyclictest141rcu_preempt08:58:250
178482499431,16cyclictest1891820-21apt12:10:130
1784824994311,6cyclictest141rcu_preempt12:25:290
1784824994311,13cyclictest141rcu_preempt11:14:250
162399430,7rtkit-daemon1753-21gnome-shell08:35:402
18755332420,7sleep22197014-21PK-Backend11:25:102
1784835994230,8cyclictest1753-21gnome-shell09:01:203
1784835994224,5cyclictest42-21ksoftirqd/310:00:023
1784835994224,13cyclictest42-21ksoftirqd/311:50:273
1784832994231,7cyclictest1794238-21munin-run07:35:012
178482999428,5cyclictest141rcu_preempt11:15:371
1784829994227,11cyclictest1855908-21sshd10:29:371
1784829994214,23cyclictest3242255-21gdbus07:30:021
1784829994214,14cyclictest26-21ksoftirqd/111:51:101
1784829994213,17cyclictest26-21ksoftirqd/110:53:431
178482499422,17cyclictest13-21ksoftirqd/009:52:020
1784824994218,7cyclictest141rcu_preempt07:11:310
1784824994217,7cyclictest141rcu_preempt12:07:370
1784824994213,5cyclictest141rcu_preempt09:49:570
1784824994213,5cyclictest141rcu_preempt09:49:570
1784824994213,5cyclictest141rcu_preempt07:36:350
1784824994213,5cyclictest141rcu_preempt07:36:350
178482499421,17cyclictest1813901-40apt-check08:30:180
1784824994210,16cyclictest13-21ksoftirqd/011:59:330
162399420,7rtkit-daemon856-21gmain11:13:073
162399420,5rtkit-daemon0-21swapper/310:14:443
162399420,34rtkit-daemon2197014-21PK-Backend09:11:022
18913472411,7sleep234-21ksoftirqd/212:07:382
178483599419,17cyclictest42-21ksoftirqd/310:27:303
1784835994121,12cyclictest42-21ksoftirqd/309:53:373
1784835994116,9cyclictest42-21ksoftirqd/311:40:233
1784832994123,13cyclictest438-20systemd-journal09:39:272
1784832994119,11cyclictest1817544-21taskset08:40:032
178482999412,36cyclictest231irq_work/109:23:251
1784829994120,17cyclictest26-21ksoftirqd/111:27:181
1784829994111,15cyclictest26-21ksoftirqd/108:30:241
178482499419,6cyclictest141rcu_preempt11:21:050
178482499413,15cyclictest0-21swapper/011:41:120
178482499413,14cyclictest0-21swapper/012:20:210
1784824994114,7cyclictest141rcu_preempt09:57:390
1784824994114,6cyclictest141rcu_preempt10:10:420
1784824994114,15cyclictest141rcu_preempt12:00:040
1784824994110,6cyclictest141rcu_preempt08:11:250
1784824994110,5cyclictest141rcu_preempt07:20:380
162399410,34rtkit-daemon2197014-21PK-Backend08:26:472
178483599408,9cyclictest42-21ksoftirqd/311:20:123
1784835994023,6cyclictest42-21ksoftirqd/311:47:453
1784835994023,6cyclictest42-21ksoftirqd/310:58:583
178483599402,34cyclictest1753-21gnome-shell10:40:353
1784835994023,13cyclictest42-21ksoftirqd/309:30:193
1784835994023,13cyclictest42-21ksoftirqd/307:20:113
1784835994014,7cyclictest1829886-21cpu09:15:133
1784832994028,8cyclictest1824819-21tr09:00:142
178483299401,8cyclictest2197014-21PK-Backend10:33:322
178483299401,8cyclictest2197014-21PK-Backend10:33:312
1784832994012,7cyclictest141rcu_preempt07:50:172
178482999409,5cyclictest141rcu_preempt09:15:021
1784829994025,11cyclictest1841431-21sshd09:46:331
1784829994025,11cyclictest1841431-21sshd09:46:321
1784829994023,12cyclictest1895095-21lsb_release12:17:061
1784829994015,11cyclictest141rcu_preempt07:14:271
1784829994012,14cyclictest26-21ksoftirqd/111:15:011
1784829994012,12cyclictest1788970-21acpi07:20:101
178482499409,6cyclictest141rcu_preempt12:35:020
178482499403,15cyclictest0-21swapper/011:45:390
1784824994029,8cyclictest13-21ksoftirqd/007:30:130
178482499402,15cyclictest0-21swapper/010:38:040
178482499402,14cyclictest0-21swapper/010:52:590
1784824994020,14cyclictest438-20systemd-journal12:40:010
1784824994020,14cyclictest438-20systemd-journal12:40:000
1784824994016,5cyclictest141rcu_preempt11:06:230
178482499401,15cyclictest2197014-21PK-Backend07:25:210
178483599395,30cyclictest1753-21gnome-shell08:10:343
1784835993925,10cyclictest42-21ksoftirqd/308:26:273
1784835993923,12cyclictest1-21systemd07:45:013
1784835993922,6cyclictest42-21ksoftirqd/308:53:253
1784835993922,6cyclictest42-21ksoftirqd/308:53:253
1784835993915,9cyclictest42-21ksoftirqd/310:38:343
178482999399,13cyclictest1874720-21kworker/1:111:40:241
178482999392,31cyclictest854-21avahi-daemon07:15:131
1784829993923,11cyclictest1827165-21pluginstate09:05:301
1784829993917,5cyclictest141rcu_preempt07:33:071
1784829993916,19cyclictest26-21ksoftirqd/110:40:011
1784829993916,11cyclictest141rcu_preempt08:50:041
1784829993916,11cyclictest141rcu_preempt08:50:041
1784829993915,7cyclictest26-21ksoftirqd/110:23:401
1784829993912,25cyclictest1818960-40apt-check08:42:291
178482499399,6cyclictest141rcu_preempt08:22:420
1784824993928,7cyclictest1753-21gnome-shell07:42:300
162399390,32rtkit-daemon854-21avahi-daemon07:35:042
162399390,32rtkit-daemon854-21avahi-daemon07:35:042
18301562380,6chrt2197014-21PK-Backend09:15:142
1784835993820,6cyclictest42-21ksoftirqd/312:35:213
1784835993820,6cyclictest42-21ksoftirqd/312:35:213
1784832993827,7cyclictest1753-21gnome-shell08:16:562
178483299382,12cyclictest1753-21gnome-shell10:13:572
1784832993821,13cyclictest1798905-21awk07:45:272
1784832993821,12cyclictest438-20systemd-journal11:30:322
1784832993819,14cyclictest1788152-21switchtime07:15:292
1784832993813,6cyclictest141rcu_preempt09:26:252
178482999387,5cyclictest141rcu_preempt07:50:091
178482999386,9cyclictest26-21ksoftirqd/109:55:271
1784829993824,10cyclictest1831102-21sshd09:17:321
1784829993822,12cyclictest1843319-21sshd09:53:341
1784829993812,12cyclictest141rcu_preempt12:08:411
1784829993812,11cyclictest1898111-21ntp_kernel_pll_12:25:231
178482499381,15cyclictest2197014-21PK-Backend11:18:300
178483299372,31cyclictest1753-21gnome-shell12:27:272
1784832993722,10cyclictest438-20systemd-journal10:37:332
178483299372,12cyclictest1753-21gnome-shell09:33:272
1784832993714,19cyclictest1852196-21pool-org.gnome.10:20:012
1784832993711,6cyclictest141rcu_preempt10:45:012
178482999379,6cyclictest141rcu_preempt10:33:411
178482999379,6cyclictest141rcu_preempt10:33:411
178482999378,14cyclictest26-21ksoftirqd/112:15:011
178482999372,9cyclictest26-21ksoftirqd/108:03:091
178482999372,7cyclictest1865481-21memory10:55:221
1784829993725,8cyclictest26-21ksoftirqd/111:45:231
1784829993712,12cyclictest141rcu_preempt10:05:221
1784829993712,11cyclictest141rcu_preempt10:45:041
1784829993710,7cyclictest26-21ksoftirqd/112:34:461
178482499379,7cyclictest141rcu_preempt10:19:450
178482499371,13cyclictest2197014-21PK-Backend08:45:320
178483299362,7cyclictest1855681-40lsb_release10:27:222
178483299362,13cyclictest1753-21gnome-shell11:58:052
178483299362,12cyclictest1753-21gnome-shell11:41:052
178483299362,12cyclictest1753-21gnome-shell08:01:182
1784832993619,13cyclictest1753-21gnome-shell10:02:032
1784832993611,5cyclictest141rcu_preempt12:22:342
1784832993610,6cyclictest141rcu_preempt09:52:072
1784829993626,7cyclictest1867596-21grep11:00:341
1784829993625,8cyclictest26-21ksoftirqd/112:01:101
1784829993621,11cyclictest1896371-21awk12:20:251
1784829993621,11cyclictest1822994-21cat08:55:131
1784829993621,11cyclictest1822994-21cat08:55:131
1784829993620,12cyclictest1886994-21idleruntime11:55:241
1784829993612,8cyclictest26-21ksoftirqd/111:08:201
1784829993612,11cyclictest1798451-21cat07:45:171
178482999361,16cyclictest1808405-21munin-node08:15:121
1784829993611,21cyclictest26-21ksoftirqd/108:10:101
162399360,29rtkit-daemon34-21ksoftirqd/211:51:362
178483299358,9cyclictest1358-1kworker/u9:107:20:252
1784832993525,7cyclictest1862879-21cron10:50:012
1784832993520,11cyclictest1902196-21sshd12:38:342
1784832993520,11cyclictest1902196-21sshd12:38:332
1784832993518,13cyclictest1889688-21sshd12:03:332
1784832993518,13cyclictest1820228-21awk08:45:282
1784832993517,14cyclictest1813901-40apt-check08:30:132
178483299351,30cyclictest1791433-21ntpq07:25:222
1784832993510,5cyclictest141rcu_preempt12:30:282
178482999359,5cyclictest141rcu_preempt10:10:211
1784829993521,10cyclictest1901259-21/usr/sbin/munin12:35:181
1784829993521,10cyclictest1901259-21/usr/sbin/munin12:35:181
1784829993520,11cyclictest1794747-21expr07:35:151
1784829993520,11cyclictest1794747-21expr07:35:151
162399350,28rtkit-daemon0-21swapper/211:13:292
18276322346,5sleep2141rcu_preempt09:07:342
178483299342,14cyclictest1753-21gnome-shell11:50:022
1784832993420,10cyclictest34-21ksoftirqd/210:21:402
1784832993415,15cyclictest1753-21gnome-shell10:57:572
178483299341,30cyclictest854-21avahi-daemon09:57:382
178483299341,14cyclictest1873720-21grep11:20:142
1784829993422,8cyclictest1753-21gnome-shell08:21:311
178483299332,12cyclictest1753-21gnome-shell08:08:372
1784832993320,10cyclictest1848094-21gzip10:05:312
1784832993319,10cyclictest1868634-21expr11:05:152
1784832993318,11cyclictest1872441-21sed11:15:222
1784832993318,11cyclictest1863362-21cat10:50:162
1784832993318,11cyclictest1841731-21basename09:50:012
1784832993318,11cyclictest1841731-21basename09:50:012
1784832993317,12cyclictest2181-21gdbus08:25:022
1784832993316,14cyclictest1753-21gnome-shell09:20:522
178483299331,5cyclictest2197014-21PK-Backend07:40:232
162399330,6rtkit-daemon1753-21gnome-shell07:11:342
1784832993220,8cyclictest1069-21runrttasks07:58:332
178483299321,27cyclictest1753-21gnome-shell08:10:032
1784832993210,18cyclictest1753-21gnome-shell08:57:122
1784832993210,18cyclictest1753-21gnome-shell08:57:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional