You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 18:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Mon Dec 01, 2025 12:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
329050924140,8sleep32197014-21PK-Backend07:10:013
330976021040,4chrt3309759-21users08:00:301
33806352990,6sleep22197014-21PK-Backend11:20:232
162399940,8rtkit-daemon1622-21rtkit-daemon12:11:323
33911902900,5sleep00-21swapper/011:50:310
162399900,9rtkit-daemon1622-21rtkit-daemon11:51:443
33869762880,7sleep20-21swapper/211:40:152
3290556997818,38cyclictest438-20systemd-journal11:54:111
33505212750,7chrt2197014-21PK-Backend09:55:302
329017527540,26sleep00-21swapper/007:06:290
162399720,11rtkit-daemon13-21ksoftirqd/011:37:111
329033626847,13sleep10-21swapper/107:08:351
329030726838,24sleep20-21swapper/207:08:122
33696692660,7sleep21753-21gnome-shell10:50:202
162399640,8rtkit-daemon1622-21rtkit-daemon11:16:270
162399640,8rtkit-daemon1622-21rtkit-daemon11:16:270
162399630,9rtkit-daemon1622-21rtkit-daemon10:38:370
329055699622,36cyclictest1753-21gnome-shell11:55:341
329055699606,48cyclictest3353013-21cpu10:05:161
329055699606,48cyclictest3353013-21cpu10:05:161
33399042590,8sleep126-21ksoftirqd/109:25:271
162399590,9rtkit-daemon1622-21rtkit-daemon08:20:281
329056299583,17cyclictest855-21dbus-daemon08:40:012
3290556995825,29cyclictest3344468-21tr09:40:141
3290556995818,35cyclictest3355889-21expr10:10:291
3290556995818,35cyclictest3324064-21pool-org.gnome.08:45:031
3290556995813,40cyclictest3387253-21entropy11:40:191
162399580,8rtkit-daemon1622-21rtkit-daemon11:26:402
329055699573,48cyclictest3341911-21grep09:30:311
329055699572,50cyclictest3296485-21apt07:25:131
329055699572,14cyclictest3326430-21/usr/sbin/munin08:50:141
162399570,9rtkit-daemon1622-21rtkit-daemon09:45:521
329055699569,42cyclictest3299682-21latency_hist07:35:031
3290556995612,40cyclictest3372188-21cat10:55:311
329055699552,48cyclictest3377332-21python311:10:291
3290556995519,31cyclictest3406500-21wc12:35:221
3290556995511,9cyclictest3305710-21sed07:50:221
162399550,4rtkit-daemon2197014-21PK-Backend11:04:560
329055699542,47cyclictest3319461-21df08:30:151
329055699542,47cyclictest3313139-21sed08:10:291
329055699542,46cyclictest3337716-21df09:20:161
329055699542,13cyclictest438-20systemd-journal11:46:171
329055699541,48cyclictest3299828-21latency_hist07:35:021
33131612530,6sleep2859-21NetworkManager08:10:292
329055699535,45cyclictest3367662-21tr10:45:151
329055699534,44cyclictest3349656-21tr09:55:141
329055699532,8cyclictest3330206-21df09:00:151
329055699532,8cyclictest3330206-21df09:00:151
329055699532,8cyclictest3242253-21nautilus10:40:011
329055699532,46cyclictest3373472-21irqstats11:00:221
3290556995320,28cyclictest3322540-21/usr/sbin/munin08:40:321
3290556995318,30cyclictest3379902-21/usr/sbin/munin11:20:171
3290556995315,34cyclictest1753-21gnome-shell12:02:351
329055299530,48cyclictest1622-21rtkit-daemon10:47:090
162399530,44rtkit-daemon3307086-21expr07:55:161
162399530,44rtkit-daemon3307086-21expr07:55:161
329055699527,8cyclictest438-20systemd-journal10:40:021
329055699522,8cyclictest3333942-21df_inode09:10:151
329055699522,8cyclictest1753-21gnome-shell12:30:151
329055699522,8cyclictest1753-21gnome-shell07:48:561
3290556995219,28cyclictest3320811-21/usr/sbin/munin08:35:251
3290556995213,35cyclictest3362941-21grep10:30:251
3290556995213,35cyclictest3362941-21grep10:30:251
3290556995211,36cyclictest3296045-21seq07:23:021
329055699521,12cyclictest3384476-21cat11:30:331
329055299520,48cyclictest1622-21rtkit-daemon08:01:210
3290564995126,20cyclictest3375471-21ntp_states11:05:243
3290564995118,17cyclictest1753-21gnome-shell08:25:113
3290564995110,24cyclictest42-21ksoftirqd/310:05:333
3290564995110,24cyclictest42-21ksoftirqd/310:05:333
329055699517,41cyclictest3370888-21latency_hist10:55:021
329055699513,7cyclictest3357638-21python310:15:291
329055699512,9cyclictest1753-21gnome-shell07:35:111
329055699512,7cyclictest3293496-21irqstats07:15:201
3290556995121,27cyclictest1069-21runrttasks12:29:591
3290556995120,28cyclictest1753-21gnome-shell10:22:181
3290556995119,28cyclictest1069-21runrttasks08:28:031
3290556995117,30cyclictest1069-21runrttasks07:42:591
329055699511,11cyclictest3399808-21which12:15:291
329055699511,11cyclictest1069-21runrttasks09:08:151
3290552995139,7cyclictest13-21ksoftirqd/009:10:110
162399510,36rtkit-daemon2197014-21PK-Backend07:41:312
162399510,25rtkit-daemon1622-21rtkit-daemon09:44:270
3290562995024,14cyclictest34-21ksoftirqd/210:02:512
3290562995024,14cyclictest34-21ksoftirqd/209:21:162
329056299501,5cyclictest141rcu_preempt11:10:282
329055699507,15cyclictest1753-21gnome-shell09:50:521
329055699501,44cyclictest3313905-21grep08:15:141
3290556995012,34cyclictest3310719-21expr08:05:191
3290556995011,34cyclictest1617-10pipewire-media-09:20:001
329055699501,11cyclictest3401596-21ps12:20:271
329055699501,11cyclictest3401596-21ps12:20:271
329056499497,38cyclictest42-21ksoftirqd/310:13:023
3290562994922,15cyclictest34-21ksoftirqd/210:05:282
3290562994922,15cyclictest34-21ksoftirqd/210:05:282
329055699499,12cyclictest3396352-21processes12:05:271
329055699496,39cyclictest3351355-21cpuspeed_turbos10:00:141
3290556994915,31cyclictest1069-21runrttasks10:26:521
3290556994915,29cyclictest3378558-21df11:15:161
3290556994915,29cyclictest3378558-21df11:15:151
3290556994915,29cyclictest1800-21upowerd08:55:251
3290556994915,29cyclictest1800-21upowerd08:55:241
3290552994913,21cyclictest1753-21gnome-shell11:06:030
162399490,9rtkit-daemon1753-21gnome-shell11:45:340
329055699487,11cyclictest3343356-21open_files09:35:261
3290556994814,29cyclictest0-21swapper/111:25:151
3290556994810,10cyclictest3375377-21needreboot11:05:241
162399480,9rtkit-daemon1753-21gnome-shell11:44:330
329055699472,40cyclictest3398146-21python312:10:281
162399470,34rtkit-daemon1753-21gnome-shell10:25:202
33856192460,6sleep20-21swapper/211:35:202
33666122460,10sleep0131ksoftirqd/010:40:320
33661012460,7sleep32197014-21PK-Backend10:40:183
329056499463,30cyclictest1753-21gnome-shell07:40:563
3290562994634,9cyclictest34-21ksoftirqd/208:30:012
3290562994623,14cyclictest34-21ksoftirqd/210:15:322
3290562994621,14cyclictest34-21ksoftirqd/212:38:422
3290562994621,14cyclictest34-21ksoftirqd/210:40:202
329055699461,7cyclictest3292157-21taskset07:13:561
329055299461,23cyclictest3338853-21sh09:25:000
32908012450,5sleep0161rcuc/007:10:140
3290564994525,16cyclictest42-21ksoftirqd/309:55:023
3290564994519,16cyclictest42-21ksoftirqd/309:35:263
3290564994516,13cyclictest1753-21gnome-shell10:25:313
329056499451,27cyclictest42-21ksoftirqd/307:25:233
3290562994522,14cyclictest34-21ksoftirqd/212:21:392
3290562994522,14cyclictest34-21ksoftirqd/212:21:382
329056299452,19cyclictest3391264-21switchtime11:50:342
329055299452,18cyclictest3325071-21irqstats08:45:250
3290552994521,15cyclictest13-21ksoftirqd/009:55:010
329055299451,23cyclictest3353480-21/usr/sbin/munin10:05:230
329055299451,23cyclictest3353480-21/usr/sbin/munin10:05:230
162399450,33rtkit-daemon1753-21gnome-shell10:56:562
3290564994426,6cyclictest42-21ksoftirqd/309:55:023
3290564994425,7cyclictest42-21ksoftirqd/308:45:203
3290564994423,13cyclictest42-21ksoftirqd/311:17:053
3290564994423,13cyclictest42-21ksoftirqd/311:17:053
3290562994423,13cyclictest34-21ksoftirqd/209:50:572
3290562994420,13cyclictest34-21ksoftirqd/209:10:562
3290562994420,13cyclictest34-21ksoftirqd/207:17:042
3290552994429,11cyclictest3299484-21sshd07:32:370
3290552994429,11cyclictest3299484-21sshd07:32:370
33650022430,14sleep2341ksoftirqd/210:35:312
3290564994325,6cyclictest42-21ksoftirqd/311:37:393
3290564994325,6cyclictest42-21ksoftirqd/308:24:403
3290564994325,6cyclictest42-21ksoftirqd/307:18:233
3290564994324,6cyclictest42-21ksoftirqd/312:15:283
3290564994322,6cyclictest42-21ksoftirqd/312:35:223
3290562994332,7cyclictest1753-21gnome-shell12:07:512
3290562994331,9cyclictest34-21ksoftirqd/208:05:002
3290562994322,6cyclictest34-21ksoftirqd/209:30:242
3290562994322,14cyclictest34-21ksoftirqd/208:20:112
3290562994322,13cyclictest34-21ksoftirqd/209:48:502
3290562994321,14cyclictest34-21ksoftirqd/208:09:532
3290562994320,14cyclictest34-21ksoftirqd/209:00:012
3290562994320,14cyclictest34-21ksoftirqd/209:00:012
3290562994320,13cyclictest34-21ksoftirqd/211:46:462
3290562994320,13cyclictest34-21ksoftirqd/207:26:502
3290562994313,26cyclictest3339905-21/usr/sbin/munin09:25:272
3290552994313,10cyclictest1753-21gnome-shell11:59:470
3290564994224,6cyclictest42-21ksoftirqd/311:57:323
3290564994224,6cyclictest42-21ksoftirqd/311:41:213
3290564994224,6cyclictest42-21ksoftirqd/311:23:053
3290564994224,6cyclictest42-21ksoftirqd/309:44:403
3290564994224,6cyclictest42-21ksoftirqd/308:31:143
3290564994224,6cyclictest42-21ksoftirqd/308:04:513
3290564994224,6cyclictest42-21ksoftirqd/307:56:373
3290564994224,6cyclictest42-21ksoftirqd/307:56:373
3290564994224,6cyclictest42-21ksoftirqd/307:50:173
3290564994224,14cyclictest42-21ksoftirqd/311:14:423
3290564994224,14cyclictest42-21ksoftirqd/311:03:113
3290564994223,6cyclictest42-21ksoftirqd/312:20:143
3290564994223,6cyclictest42-21ksoftirqd/312:20:133
3290564994223,6cyclictest42-21ksoftirqd/312:05:353
3290564994223,6cyclictest42-21ksoftirqd/310:35:163
3290564994223,6cyclictest42-21ksoftirqd/308:11:193
3290564994223,6cyclictest42-21ksoftirqd/308:07:573
3290564994221,16cyclictest42-21ksoftirqd/309:25:263
3290564994221,13cyclictest42-21ksoftirqd/308:41:563
3290562994225,13cyclictest3344071-21awk09:40:002
3290562994221,13cyclictest34-21ksoftirqd/211:05:572
3290562994221,13cyclictest34-21ksoftirqd/210:32:072
3290562994221,13cyclictest34-21ksoftirqd/210:32:072
3290562994221,13cyclictest34-21ksoftirqd/209:03:482
3290562994221,13cyclictest34-21ksoftirqd/209:03:482
3290562994220,13cyclictest34-21ksoftirqd/210:21:312
3290562994220,13cyclictest34-21ksoftirqd/207:49:402
3290552994226,6cyclictest13-21ksoftirqd/008:35:340
3290552994220,7cyclictest1753-21gnome-shell12:29:540
3290552994219,14cyclictest13-21ksoftirqd/009:50:010
329055299421,16cyclictest0-21swapper/007:19:270
3290552994211,12cyclictest141rcu_preempt10:20:110
162399420,33rtkit-daemon3290550-21cyclictest08:27:510
3290564994124,6cyclictest42-21ksoftirqd/308:38:483
3290564994123,6cyclictest42-21ksoftirqd/312:30:153
3290564994123,6cyclictest42-21ksoftirqd/310:57:033
3290564994123,6cyclictest42-21ksoftirqd/310:20:033
3290564994123,6cyclictest42-21ksoftirqd/309:17:263
3290564994123,6cyclictest42-21ksoftirqd/307:39:393
3290564994123,6cyclictest42-21ksoftirqd/307:23:493
3290564994123,6cyclictest42-21ksoftirqd/307:10:113
3290564994122,7cyclictest42-21ksoftirqd/310:15:263
3290564994122,7cyclictest42-21ksoftirqd/309:00:273
3290564994122,7cyclictest42-21ksoftirqd/309:00:273
3290564994122,6cyclictest42-21ksoftirqd/311:32:183
3290564994122,6cyclictest42-21ksoftirqd/309:20:163
3290564994122,6cyclictest42-21ksoftirqd/308:55:173
3290564994122,6cyclictest42-21ksoftirqd/308:55:173
3290564994121,5cyclictest42-21ksoftirqd/310:45:243
3290564994114,8cyclictest3402796-21expr12:25:153
3290562994122,6cyclictest34-21ksoftirqd/211:01:262
3290562994122,6cyclictest34-21ksoftirqd/209:19:162
3290562994121,6cyclictest34-21ksoftirqd/207:35:262
3290562994113,16cyclictest34-21ksoftirqd/210:12:282
3290562994111,11cyclictest34-21ksoftirqd/208:45:282
329055299418,8cyclictest1358-1kworker/u9:107:52:490
3290552994124,6cyclictest13-21ksoftirqd/010:15:010
329055299412,34cyclictest3314646-21ntp_states08:15:270
3290552994114,6cyclictest3311933-21kworker/0:008:44:390
329055299411,36cyclictest3333016-21chrt09:09:520
329055299411,10cyclictest2197014-21PK-Backend10:01:450
329056499409,12cyclictest0-21swapper/308:50:263
3290564994022,6cyclictest42-21ksoftirqd/311:45:063
3290564994021,6cyclictest42-21ksoftirqd/309:10:023
3290564994016,5cyclictest141rcu_preempt09:09:573
3290564994016,15cyclictest42-21ksoftirqd/310:30:153
3290564994016,15cyclictest42-21ksoftirqd/310:30:153
3290562994023,14cyclictest34-21ksoftirqd/212:17:432
3290562994023,13cyclictest34-21ksoftirqd/212:29:472
3290562994020,6cyclictest34-21ksoftirqd/211:34:002
3290562994020,13cyclictest34-21ksoftirqd/207:31:462
3290562994020,13cyclictest34-21ksoftirqd/207:31:462
3290562994019,13cyclictest34-21ksoftirqd/211:55:202
329055299408,11cyclictest13-21ksoftirqd/012:15:010
3290552994025,12cyclictest3311903-21awk08:10:000
3290552994025,11cyclictest3317047-21sshd08:24:370
3290552994023,13cyclictest13-21ksoftirqd/010:25:080
329055299402,17cyclictest3330219-21diskstats09:00:160
329055299402,17cyclictest3330219-21diskstats09:00:150
329055299401,35cyclictest3363356-21cat10:30:330
329055299401,35cyclictest3363356-21cat10:30:330
3290552994012,8cyclictest1753-21gnome-shell10:17:070
33522602390,8sleep30-21swapper/310:00:303
33191562390,8chrt0-21swapper/208:30:142
3290564993921,6cyclictest42-21ksoftirqd/312:03:433
3290564993921,6cyclictest42-21ksoftirqd/310:51:003
329056499391,6cyclictest2197014-21PK-Backend09:30:313
3290564993915,7cyclictest1753-21gnome-shell09:46:243
3290562993928,7cyclictest1753-21gnome-shell07:14:242
3290562993920,6cyclictest34-21ksoftirqd/208:50:232
3290562993920,13cyclictest34-21ksoftirqd/211:16:122
3290562993920,13cyclictest34-21ksoftirqd/211:16:122
329056299391,8cyclictest1358-1kworker/u9:108:40:212
329056299391,33cyclictest1753-21gnome-shell07:22:242
3290552993925,10cyclictest13-21ksoftirqd/007:35:320
3290552993924,11cyclictest3344053-21sh09:40:000
3290552993923,12cyclictest3302494-21ntpq07:40:250
3290552993917,13cyclictest13-21ksoftirqd/011:30:340
3290552993912,12cyclictest141rcu_preempt10:50:100
3290552993911,8cyclictest1753-21gnome-shell08:51:460
162399390,27rtkit-daemon2197014-21PK-Backend12:31:050
162399390,27rtkit-daemon2197014-21PK-Backend12:21:590
162399390,27rtkit-daemon2197014-21PK-Backend12:21:590
329056499389,16cyclictest42-21ksoftirqd/308:17:313
3290564993819,6cyclictest42-21ksoftirqd/307:45:183
3290562993827,7cyclictest1753-21gnome-shell12:11:362
3290562993827,7cyclictest1753-21gnome-shell12:01:302
3290562993827,7cyclictest1753-21gnome-shell09:08:562
3290562993826,8cyclictest1753-21gnome-shell07:51:352
3290562993823,11cyclictest3307078-21tr07:55:162
3290562993823,11cyclictest3307078-21tr07:55:162
3290562993821,12cyclictest3404942-21ntpq12:30:252
3290562993821,12cyclictest3367963-21awk10:45:212
329055299387,9cyclictest1753-21gnome-shell09:55:260
329055299384,18cyclictest141rcu_preempt11:20:220
329055299383,20cyclictest1753-21gnome-shell11:29:360
3290552993823,10cyclictest3371092-21apt-config10:55:030
329055299382,19cyclictest3311933-21kworker/0:009:30:290
329055299382,11cyclictest1753-21gnome-shell08:31:400
3290552993819,6cyclictest141rcu_preempt07:22:180
329055299381,33cyclictest3400506-20unin-run12:20:000
3290552993813,15cyclictest13-21ksoftirqd/011:10:130
3290564993719,6cyclictest42-21ksoftirqd/311:25:373
3290564993711,13cyclictest42-21ksoftirqd/307:34:173
3290564993711,13cyclictest42-21ksoftirqd/307:34:173
3290562993726,7cyclictest1753-21gnome-shell09:40:232
329055299379,8cyclictest1753-21gnome-shell08:10:050
329055299372,19cyclictest3385614-21irqstats11:35:200
329055299372,12cyclictest1753-21gnome-shell12:35:570
3290552993720,12cyclictest3328789-21systemd-udevd08:55:240
3290552993720,12cyclictest3328789-21systemd-udevd08:55:240
329055299371,22cyclictest3395904-21cstates12:05:150
329055299371,21cyclictest3394201-21/usr/sbin/munin12:00:180
3290552993711,11cyclictest141rcu_preempt09:25:130
3290552993710,7cyclictest0-21swapper/007:29:170
3290562993618,6cyclictest34-21ksoftirqd/208:15:162
329055299362,19cyclictest0-21swapper/007:55:140
329055299362,19cyclictest0-21swapper/007:55:140
329055299361,19cyclictest3303481-21apt07:45:130
329055299358,7cyclictest141rcu_preempt09:15:310
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional