You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 19:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Thu Nov 20, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
121319222840,9sleep30-21swapper/307:05:183
121312622560,4sleep20-21swapper/207:05:172
12748662930,5sleep30-21swapper/310:00:013
12698172930,6sleep10-21swapper/109:45:181
162399920,9rtkit-daemon1622-21rtkit-daemon11:51:401
162399920,9rtkit-daemon1622-21rtkit-daemon11:51:391
162399910,8rtkit-daemon1622-21rtkit-daemon11:45:281
162399910,8rtkit-daemon1622-21rtkit-daemon11:20:231
12244182910,5sleep20-21swapper/207:35:192
162399880,9rtkit-daemon1622-21rtkit-daemon11:40:561
162399850,9rtkit-daemon1622-21rtkit-daemon10:36:021
13034452780,6sleep11753-21gnome-shell11:20:011
121390027739,9sleep10-21swapper/107:06:431
162399750,9rtkit-daemon1753-21gnome-shell11:16:000
162399740,64rtkit-daemon0-21swapper/007:06:450
162399650,8rtkit-daemon1622-21rtkit-daemon12:17:271
13126242650,4sleep3411rcuc/311:45:153
121433699641,37cyclictest1312990-21df11:45:192
1214336996323,35cyclictest1272346-21cat09:50:362
121433699622,32cyclictest1753-21gnome-shell07:58:522
1214336996022,34cyclictest855-21dbus-daemon09:50:022
1214336996022,33cyclictest1284151-21mktemp10:25:032
121433499600,55cyclictest1622-21rtkit-daemon11:06:031
121433699592,34cyclictest1753-21gnome-shell07:23:002
121433699592,34cyclictest1753-21gnome-shell07:23:002
1214336995917,37cyclictest438-20systemd-journal11:31:282
121433699591,35cyclictest860997-21snmpd07:49:342
162399580,44rtkit-daemon1753-21gnome-shell07:48:060
1214340995824,30cyclictest1276886-21cstates10:05:143
1214340995816,12cyclictest438-20systemd-journal07:40:003
162399570,9rtkit-daemon1622-21rtkit-daemon07:21:401
162399570,9rtkit-daemon1622-21rtkit-daemon07:21:391
12225972570,7sleep013-21ksoftirqd/007:30:150
1214340995721,13cyclictest855-21dbus-daemon12:25:003
1214340995716,13cyclictest1322753-21apt-get12:15:003
1214334995720,32cyclictest1230982-21sh07:55:001
162399560,9rtkit-daemon1622-21rtkit-daemon10:48:051
12867272560,13sleep1251rcuc/110:30:261
12760192560,3chrt1276020-21/usr/sbin/munin10:00:303
12613932560,17sleep342-21ksoftirqd/309:20:213
1214340995618,12cyclictest1268115-21which09:40:223
1214340995614,20cyclictest42-21ksoftirqd/310:30:033
121433699562,31cyclictest1753-21gnome-shell11:58:332
1214336995616,35cyclictest1293140-21cat10:50:142
121433699561,34cyclictest1280291-21dpkg10:15:082
121433699561,32cyclictest1287678-21/usr/sbin/munin10:35:312
1214334995626,25cyclictest1253944-21tr09:00:141
162399550,8rtkit-daemon1622-21rtkit-daemon10:52:151
162399550,8rtkit-daemon1622-21rtkit-daemon07:17:171
162399550,7rtkit-daemon1622-21rtkit-daemon12:39:581
12786012550,11chrt1278606-21cpuspeed_turbos10:10:173
1214340995521,12cyclictest1296131-21apt-get11:00:023
1214340995517,6cyclictest391irq_work/308:48:243
1214336995523,28cyclictest1069-21runrttasks12:09:482
121433699552,30cyclictest1753-21gnome-shell07:44:522
1214336995519,32cyclictest1278172-21pool-org.gnome.10:10:012
1214336995518,32cyclictest1254504-21ntp_kernel_err09:00:232
121434099544,23cyclictest1292061-21uname10:45:273
1214340995416,13cyclictest1236633-21cut08:10:133
1214340995415,14cyclictest1632-21dbus-daemon10:15:073
1214340995413,20cyclictest42-21ksoftirqd/309:28:583
121433699542,27cyclictest1253161-21users08:55:412
1214336995415,34cyclictest3242253-21nautilus07:25:042
121433699541,32cyclictest1215310-21processes07:10:282
1214334995417,10cyclictest1247098-21cat08:40:121
121433499541,34cyclictest1279931-21taskset10:15:011
162399530,9rtkit-daemon1622-21rtkit-daemon11:37:270
1214340995314,11cyclictest1227614-21sed07:45:033
1214336995316,32cyclictest1291692-21uname10:45:202
1214336995315,34cyclictest1265134-21wc09:30:292
121433699531,48cyclictest860997-21snmpd11:37:022
121433699531,46cyclictest1300435-21expr11:10:122
121433499532,46cyclictest1753-21gnome-shell07:30:191
1214334995316,13cyclictest875-21polkitd09:13:171
1214334995314,14cyclictest1632-21dbus-daemon09:45:001
12912082520,7chrt2197014-21PK-Backend10:45:100
1214340995222,9cyclictest1753-21gnome-shell11:37:553
1214340995220,7cyclictest12675892chrt09:40:023
1214340995219,6cyclictest1283658-21sh10:25:003
1214340995215,11cyclictest1219024-21cat07:20:213
1214340995215,11cyclictest1219024-21cat07:20:203
1214340995214,12cyclictest1328540-21grep12:30:143
1214340995213,13cyclictest1247771-21cat08:40:273
1214340995213,11cyclictest1216892-21grep07:15:163
1214336995220,28cyclictest34-21ksoftirqd/208:36:182
1214336995218,29cyclictest1299155-21ntp_kernel_err11:05:232
1214336995217,30cyclictest1329113-21memory12:30:222
1214336995216,32cyclictest1275695-21sh10:00:252
1214336995215,32cyclictest438-20systemd-journal08:05:002
1214336995214,33cyclictest1247192-21expr08:40:122
1214336995213,34cyclictest1234831-21cat08:05:132
121433499522,11cyclictest1296967-21ls11:00:161
12357462510,6chrt0-21swapper/108:05:291
1214340995122,8cyclictest1753-21gnome-shell10:41:003
1214340995119,9cyclictest1232712-21cron08:00:013
1214340995118,10cyclictest42-21ksoftirqd/308:15:253
1214340995118,10cyclictest42-21ksoftirqd/308:15:243
1214336995121,26cyclictest1753-21gnome-shell09:12:522
1214336995117,29cyclictest1242027-21sh08:25:132
1214336995116,31cyclictest1290326-21switchtime10:40:312
1214336995115,31cyclictest1305351-21sed11:25:022
121433699511,47cyclictest0-21swapper/211:43:422
121433699511,45cyclictest1250903-21latency08:50:232
121433699511,26cyclictest0-21swapper/209:27:012
1214336995110,37cyclictest1753-21gnome-shell12:21:462
121434099502,9cyclictest1753-21gnome-shell10:50:183
1214340995022,8cyclictest1753-21gnome-shell09:45:173
1214340995020,9cyclictest1296648-31apt-check11:00:123
1214340995016,12cyclictest2962348-21rs:main3
1214340995016,11cyclictest1215949-21cron07:15:003
1214340995015,10cyclictest1221201-21sh07:25:243
1214340995013,14cyclictest1306012-21df11:25:183
1214340995013,10cyclictest1243285-21wc08:30:023
1214340995012,13cyclictest1245725-21missed_timers08:35:253
1214336995014,32cyclictest1279165-21sh10:10:292
1214336995013,33cyclictest1314592-21expr11:50:162
1214336995013,33cyclictest1314592-21expr11:50:162
1214336995013,33cyclictest1240159-21expr08:20:162
1214336995013,33cyclictest1240159-21expr08:20:152
121433699501,26cyclictest0-21swapper/210:30:462
1214334995021,25cyclictest1225730-21/usr/sbin/munin07:40:291
121434099492,7cyclictest1311251-21df11:40:153
121434099492,25cyclictest1286341-21df_inode10:30:173
121434099492,25cyclictest1235601-21python308:05:283
1214340994919,8cyclictest1256308-21proc_pri09:05:253
1214340994919,10cyclictest42-21ksoftirqd/307:40:293
1214340994918,9cyclictest1753-21gnome-shell12:29:163
1214340994917,7cyclictest1299504-21timerwakeupswit11:05:293
1214340994915,11cyclictest42-21ksoftirqd/311:24:003
121434099491,44cyclictest1622-21rtkit-daemon08:00:423
1214340994912,11cyclictest1316723-21cat11:55:213
121434099491,12cyclictest1308510-21users11:30:343
1214340994910,16cyclictest1753-21gnome-shell12:15:193
121433699497,37cyclictest1753-21gnome-shell12:38:012
121433699492,42cyclictest1297054-21diskstats11:00:202
1214336994917,27cyclictest1318653-21processes12:00:272
1214336994916,29cyclictest1249663-21sed08:45:292
1214336994915,30cyclictest1237133-21strings08:10:232
121433699491,44cyclictest881143-21gmain12:11:212
1214336994912,33cyclictest1222859-21cat07:30:222
121433499492,9cyclictest1248966-21df_inode08:45:141
1214334994922,22cyclictest1753-21gnome-shell09:56:071
1214334994917,10cyclictest1289574-21tr10:40:151
121434099482,25cyclictest1753-21gnome-shell10:35:573
1214340994819,8cyclictest1753-21gnome-shell09:52:483
121434099481,11cyclictest1230857-21taskset07:53:293
1214340994810,18cyclictest1753-21gnome-shell09:03:353
1214336994816,28cyclictest1216700-21/usr/sbin/munin07:15:292
1214336994816,27cyclictest1260981-21cstates09:20:142
121433699481,43cyclictest1753-21gnome-shell08:17:182
121433699481,43cyclictest1753-21gnome-shell08:17:172
1214336994811,32cyclictest1327215-21cat12:25:172
1214334994820,10cyclictest26-21ksoftirqd/108:29:281
1214334994819,7cyclictest1753-21gnome-shell09:50:261
121433099482,8cyclictest1753-21gnome-shell10:13:020
12586032470,8sleep31753-21gnome-shell09:13:183
1214340994719,7cyclictest1753-21gnome-shell09:17:493
1214340994718,6cyclictest1241399-21latency_hist08:25:033
1214340994718,6cyclictest1241399-21latency_hist08:25:033
1214340994715,9cyclictest1318828-21sed12:00:313
1214340994711,13cyclictest1250695-21df08:50:163
1214340994710,17cyclictest1753-21gnome-shell12:07:423
121433699477,36cyclictest1753-21gnome-shell09:09:592
1214336994715,28cyclictest1307236-21latency_hist11:30:022
1214336994715,28cyclictest1266086-21cpuspeed_turbos09:35:112
1214336994715,27cyclictest1303381-21latency_hist11:20:012
121433699471,41cyclictest1753-21gnome-shell10:20:202
1214336994712,30cyclictest1267966-21expr09:40:172
1214336994710,33cyclictest1753-21gnome-shell09:15:312
1214334994724,19cyclictest1236595-21tr08:10:141
1214334994720,23cyclictest1753-21gnome-shell09:23:101
1214334994719,6cyclictest1224141-21cpuspeed_turbos07:35:141
121433499471,11cyclictest1263842-21chrt09:28:331
12706202460,8sleep02197014-21PK-Backend09:46:300
121434099462,24cyclictest1753-21gnome-shell12:37:303
121434099461,8cyclictest1302016-21grep11:15:143
121433699469,32cyclictest2962348-21rs:main2
1214336994614,27cyclictest1230190-21needreboot07:50:232
121433499468,11cyclictest1241233-21seq08:23:071
121433499468,11cyclictest1241233-21seq08:23:071
1214334994621,21cyclictest1753-21gnome-shell07:11:581
1214334994620,7cyclictest1753-21gnome-shell12:27:181
1214334994619,8cyclictest1753-21gnome-shell11:33:011
1214334994615,8cyclictest1231229-21taskset07:55:081
121433499461,22cyclictest1329268-21/usr/sbin/munin12:30:241
13000572450,7sleep02197014-21PK-Backend11:10:010
1214340994516,8cyclictest1753-21gnome-shell09:30:033
121433699452,38cyclictest1324074-40lsb_release12:15:282
1214336994515,27cyclictest855-21dbus-daemon11:00:012
121433499459,10cyclictest1260616-21date09:20:011
1214334994523,18cyclictest1753-21gnome-shell07:29:171
1214334994519,7cyclictest1753-21gnome-shell08:18:401
1214334994519,7cyclictest1753-21gnome-shell08:18:401
1214334994517,6cyclictest1310626-21latency_hist11:40:011
1214334994516,7cyclictest1244723-21taskset08:33:231
121433499451,22cyclictest1753-21gnome-shell09:39:431
1214334994511,11cyclictest1320312-21open_files12:05:251
12736302440,8sleep02197014-21PK-Backend09:55:200
121434099441,22cyclictest1300151-21/usr/sbin/munin11:10:193
121433699447,32cyclictest1273412-21wc09:55:152
121433499449,15cyclictest1753-21gnome-shell10:03:431
121433499443,17cyclictest1753-21gnome-shell09:31:141
1214334994419,21cyclictest1282396-21cstates10:20:151
1214334994418,7cyclictest1753-21gnome-shell11:57:021
1214334994417,8cyclictest1753-21gnome-shell12:14:061
1214334994416,10cyclictest26-21ksoftirqd/108:50:121
1214334994414,5cyclictest0-21swapper/111:10:111
1214334994413,9cyclictest1229321-21sed07:50:011
1214330994428,12cyclictest1248273-21sshd08:44:290
162399430,35rtkit-daemon2197014-21PK-Backend11:56:140
121433499439,15cyclictest1753-21gnome-shell11:27:381
121433499439,15cyclictest1753-21gnome-shell08:58:121
1214334994314,8cyclictest1753-21gnome-shell10:56:381
1214334994311,8cyclictest0-21swapper/110:26:191
1214330994320,18cyclictest1313711-21users11:45:340
121433499429,15cyclictest1753-21gnome-shell12:24:071
121433499429,15cyclictest1753-21gnome-shell10:05:421
121433499422,16cyclictest1753-21gnome-shell12:05:001
121433499422,16cyclictest1753-21gnome-shell08:04:181
1214334994220,19cyclictest1280406-21apt10:15:191
1214334994212,7cyclictest0-21swapper/109:05:221
121433099422,16cyclictest1312385-21unin-run11:45:010
162399410,27rtkit-daemon2197014-21PK-Backend08:06:020
1214340994129,8cyclictest1222886-21tail07:30:223
121433099418,13cyclictest141rcu_preempt09:05:110
121433099411,9cyclictest0-21swapper/010:07:010
162399400,27rtkit-daemon2197014-21PK-Backend08:16:490
162399400,27rtkit-daemon2197014-21PK-Backend08:16:480
12217262400,8chrt2197014-21PK-Backend07:27:250
1214340994017,7cyclictest1315224-21sensors_temp11:50:293
1214340994017,7cyclictest1315224-21sensors_temp11:50:293
121433499401,20cyclictest1246536-21runrttasks08:39:261
121433099402,18cyclictest1753-21gnome-shell11:31:070
121433099401,16cyclictest2197014-21PK-Backend10:57:370
162399390,31rtkit-daemon2197014-21PK-Backend07:36:360
1214330993928,6cyclictest1218180-21turbostat07:20:000
1214330993924,11cyclictest1280560-21cut10:15:210
121433099392,32cyclictest1753-21gnome-shell11:29:580
1214330993921,13cyclictest1243137-21turbostat08:29:590
1214330993919,14cyclictest438-20systemd-journal11:00:440
162399380,26rtkit-daemon2197014-21PK-Backend08:13:230
1214330993829,5cyclictest2197014-21PK-Backend10:50:300
121433099382,8cyclictest1753-21gnome-shell10:01:340
1214330993827,7cyclictest1753-21gnome-shell08:50:160
1214330993826,8cyclictest1753-21gnome-shell11:51:380
1214330993826,8cyclictest1753-21gnome-shell11:51:380
121433099382,32cyclictest1753-21gnome-shell11:21:560
121433099382,32cyclictest1753-21gnome-shell10:35:280
121433099381,8cyclictest12587162sleep009:14:310
121433099381,4cyclictest2197014-21PK-Backend12:15:140
121433099381,4cyclictest2197014-21PK-Backend07:51:250
1214330993727,6cyclictest2197014-21PK-Backend09:25:230
121433099371,31cyclictest2197014-21PK-Backend09:35:290
121433099371,15cyclictest2197014-21PK-Backend12:38:550
1214330993627,5cyclictest2197014-21PK-Backend12:29:040
1214330993626,6cyclictest2197014-21PK-Backend09:04:300
1214330993619,12cyclictest1322337-21grep12:10:300
121433099361,30cyclictest2197014-21PK-Backend08:20:360
121433099361,30cyclictest2197014-21PK-Backend08:20:360
121433099361,13cyclictest2197014-21PK-Backend12:34:070
162399350,27rtkit-daemon2197014-21PK-Backend07:59:250
1214340993519,12cyclictest1244220-21processes08:30:263
1214330993527,4cyclictest2197014-21PK-Backend09:20:350
1214330993526,5cyclictest2197014-21PK-Backend08:55:330
121433099351,9cyclictest2197014-21PK-Backend10:26:330
121433099351,9cyclictest2197014-21PK-Backend09:17:410
12271552341,10sleep02197014-21PK-Backend07:44:250
121433099346,24cyclictest1234455-21ls08:05:010
1214330993423,7cyclictest1753-21gnome-shell10:40:390
1214330993416,13cyclictest1283697-21pool-org.gnome.10:25:010
121433099341,28cyclictest2197014-21PK-Backend09:33:370
121433099341,28cyclictest2197014-21PK-Backend08:35:080
121433099341,15cyclictest2197014-21PK-Backend12:07:390
1214330993325,4cyclictest2197014-21PK-Backend08:30:160
121433099331,6cyclictest1753-21gnome-shell09:40:090
121433099331,29cyclictest0-21swapper/008:46:310
121433099331,28cyclictest2197014-21PK-Backend12:04:340
121433099331,28cyclictest1215972-21turbostat07:15:000
1214330993217,12cyclictest1272246-21grep09:50:320
121433099321,4cyclictest2197014-21PK-Backend12:24:470
121433099321,4cyclictest2197014-21PK-Backend11:11:230
121433099321,4cyclictest0-21swapper/010:32:320
1214330993123,5cyclictest2197014-21PK-Backend07:21:280
1214330993123,5cyclictest2197014-21PK-Backend07:21:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional