You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-13 - 02:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Wed Nov 12, 2025 12:44:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
84127623180,9sleep32197014-21PK-Backend07:05:273
84182822400,6sleep2841827-21systemctl07:10:012
162399960,9rtkit-daemon1753-21gnome-shell09:31:392
9298152880,9sleep2929541-40apt-check11:15:162
84163828671,9sleep00-21swapper/007:08:220
162399780,6rtkit-daemon2197014-21PK-Backend09:50:272
162399700,10rtkit-daemon1622-21rtkit-daemon10:22:473
162399690,8rtkit-daemon1622-21rtkit-daemon12:09:563
84172726545,14sleep10-21swapper/107:09:321
84192999614,53cyclictest1622-21rtkit-daemon07:15:331
162399610,9rtkit-daemon1622-21rtkit-daemon12:17:483
9172902580,16sleep2331rcuc/210:40:312
8491272580,7chrt2197014-21PK-Backend07:28:292
841929995833,10cyclictest26-21ksoftirqd/110:50:221
8535812570,5chrt0-21swapper/007:40:220
841925995717,20cyclictest13-21ksoftirqd/011:15:010
162399550,8rtkit-daemon1622-21rtkit-daemon11:00:293
8555112540,7chrt2197014-21PK-Backend07:45:240
84193499540,18cyclictest945302-21ls12:00:023
84193499540,18cyclictest945302-21ls12:00:023
162399540,9rtkit-daemon1622-21rtkit-daemon11:17:243
162399520,8rtkit-daemon1622-21rtkit-daemon12:19:422
9395312510,8sleep0131ksoftirqd/011:40:300
84193499519,10cyclictest863853-21diskstats08:10:213
841934995112,13cyclictest854-21avahi-daemon08:05:013
84192599514,19cyclictest13-21ksoftirqd/009:13:080
841925995113,21cyclictest13-21ksoftirqd/012:05:020
162399510,36rtkit-daemon2197014-21PK-Backend09:29:592
84193499508,14cyclictest934416-21sort11:25:313
84193499506,21cyclictest940934-21sed11:45:273
841925995014,5cyclictest141rcu_preempt11:15:150
162399500,43rtkit-daemon2284-21update-notifier10:12:123
84193499494,19cyclictest923120-21cat10:55:313
841934994911,19cyclictest1753-21gnome-shell09:05:233
8546242480,13sleep1251rcuc/107:45:011
84193499488,12cyclictest1753-21gnome-shell09:50:293
84193499484,26cyclictest1753-21gnome-shell09:47:023
84193499484,26cyclictest1753-21gnome-shell09:47:023
841934994810,12cyclictest1753-21gnome-shell10:51:283
841929994822,21cyclictest874290-21df08:40:151
841925994818,8cyclictest13-21ksoftirqd/010:46:460
841925994818,8cyclictest13-21ksoftirqd/010:46:460
162399480,33rtkit-daemon1753-21gnome-shell09:17:403
84193499479,13cyclictest1753-21gnome-shell07:31:093
84193499478,12cyclictest1753-21gnome-shell12:10:183
84193499478,12cyclictest1753-21gnome-shell11:31:403
84193499477,10cyclictest855321-21latency07:45:233
84193499475,22cyclictest1753-21gnome-shell12:00:553
841934994720,22cyclictest918910-40lsb_release10:45:243
841934994720,22cyclictest918910-40lsb_release10:45:243
841934994710,19cyclictest1753-21gnome-shell09:25:173
84193499470,28cyclictest1753-21gnome-shell07:41:243
84193499470,23cyclictest952806-21expr12:20:143
841929994717,9cyclictest26-21ksoftirqd/111:15:161
841925994719,9cyclictest13-21ksoftirqd/010:55:020
162399470,39rtkit-daemon1753-21gnome-shell10:30:382
162399470,39rtkit-daemon1753-21gnome-shell10:30:382
84193499469,14cyclictest874174-21expr08:40:133
84193499467,13cyclictest931883-21expr11:20:163
841934994610,18cyclictest1753-21gnome-shell10:40:583
841934994610,12cyclictest1753-21gnome-shell08:05:143
841934994610,11cyclictest901128-21irqstats09:55:213
841934994610,11cyclictest1753-21gnome-shell08:35:303
841925994635,7cyclictest13-21ksoftirqd/010:10:180
84193499458,14cyclictest849329-21cat07:30:023
84193499458,14cyclictest846045-21/usr/sbin/munin07:20:283
84193499457,15cyclictest904544-21basename10:05:203
84193499456,14cyclictest0-21swapper/309:22:043
84193499454,23cyclictest1753-21gnome-shell07:10:373
84193499454,23cyclictest1753-21gnome-shell07:10:373
841934994510,18cyclictest1753-21gnome-shell11:53:443
841934994510,18cyclictest1753-21gnome-shell11:53:443
84193499450,19cyclictest3242253-21nautilus07:40:013
84193499450,19cyclictest3242253-21nautilus07:40:013
841929994522,19cyclictest953336-21python312:20:281
841929994514,14cyclictest26-21ksoftirqd/108:50:231
841925994513,17cyclictest13-21ksoftirqd/009:05:000
9011292440,4chrt0-21swapper/009:55:200
8674232440,9sleep342-21ksoftirqd/308:20:213
84193499449,17cyclictest1753-21gnome-shell11:10:023
84193499449,16cyclictest957410-21munin-run12:35:003
84193499449,12cyclictest911800-21ntp_states10:25:273
84193499447,14cyclictest881771-21sed09:00:243
84193499447,11cyclictest954943-21ntp_states12:25:263
84193499446,15cyclictest871366-21timerandwakeup08:30:323
84193499445,10cyclictest854-21avahi-daemon07:55:043
84193499444,15cyclictest894301-21/usr/sbin/munin09:35:223
84193499443,24cyclictest1753-21gnome-shell08:54:053
841934994429,11cyclictest1753-21gnome-shell10:03:183
84193499441,17cyclictest8664592chrt08:17:313
84193499441,17cyclictest8664592chrt08:17:313
84192999445,12cyclictest141rcu_preempt08:45:201
841929994415,24cyclictest26-21ksoftirqd/110:35:031
841929994415,24cyclictest26-21ksoftirqd/110:35:021
841925994432,9cyclictest13-21ksoftirqd/008:20:210
84192599442,20cyclictest1753-21gnome-shell08:45:240
84192599441,21cyclictest917757-21apt-key10:45:010
841925994412,10cyclictest141rcu_preempt10:03:340
841925994410,14cyclictest141rcu_preempt11:47:570
841925994410,13cyclictest141rcu_preempt08:17:070
841925994410,13cyclictest141rcu_preempt08:17:070
162399440,32rtkit-daemon1753-21gnome-shell08:39:362
84193499438,11cyclictest854-21avahi-daemon07:55:003
84193499438,11cyclictest1753-21gnome-shell10:30:213
84193499438,11cyclictest1753-21gnome-shell10:30:213
84193499434,22cyclictest1753-21gnome-shell10:15:263
841934994325,13cyclictest929036-21apt-get11:15:003
841934994311,10cyclictest885377-21diskstats09:10:173
841934994310,10cyclictest1753-21gnome-shell08:49:103
84192999438,5cyclictest141rcu_preempt09:13:091
841929994321,10cyclictest26-21ksoftirqd/111:54:341
841929994321,10cyclictest26-21ksoftirqd/111:54:341
841929994316,22cyclictest26-21ksoftirqd/110:15:151
841929994313,11cyclictest26-21ksoftirqd/112:04:591
84192599439,6cyclictest141rcu_preempt09:50:010
84192599439,6cyclictest141rcu_preempt09:50:000
84192599436,16cyclictest13-21ksoftirqd/009:05:320
841925994326,12cyclictest13-21ksoftirqd/009:18:520
841925994319,9cyclictest13-21ksoftirqd/010:35:120
9009962420,7chrt2197014-21PK-Backend09:55:152
8913322420,4sleep11753-21gnome-shell09:25:321
84193499424,11cyclictest2197014-21PK-Backend11:40:313
841934994210,9cyclictest958038-21df_inode12:35:163
841932994225,12cyclictest929036-21apt-get11:15:002
841932994216,13cyclictest2181-21gdbus09:05:022
84192999421,15cyclictest2197014-21PK-Backend11:15:031
841929994210,6cyclictest141rcu_preempt08:00:591
841925994222,12cyclictest13-21ksoftirqd/010:17:210
841925994215,8cyclictest13-21ksoftirqd/011:50:390
841925994215,8cyclictest13-21ksoftirqd/011:50:390
84192599421,21cyclictest13-21ksoftirqd/011:10:010
84192599421,19cyclictest1753-21gnome-shell08:45:000
162399420,32rtkit-daemon1753-21gnome-shell08:00:122
8463152410,8chrt2197014-21PK-Backend07:20:142
84193499419,10cyclictest1753-21gnome-shell08:57:513
84193499418,10cyclictest2197014-21PK-Backend07:15:093
84193499414,14cyclictest1753-21gnome-shell10:35:263
84193499410,17cyclictest1753-21gnome-shell08:29:063
841932994110,14cyclictest141rcu_preempt08:24:392
84192999415,4cyclictest141rcu_preempt09:10:011
841929994126,11cyclictest952208-21/usr/sbin/munin12:20:021
841929994119,8cyclictest26-21ksoftirqd/110:05:021
841929994114,8cyclictest931193-21kworker/1:111:45:151
841929994112,6cyclictest141rcu_preempt10:50:031
841929994112,6cyclictest141rcu_preempt10:50:021
84192599416,17cyclictest2197014-21PK-Backend07:16:570
841925994127,10cyclictest13-21ksoftirqd/012:20:260
841925994125,12cyclictest13-21ksoftirqd/012:29:260
841925994122,12cyclictest13-21ksoftirqd/007:25:350
841925994120,12cyclictest13-21ksoftirqd/012:18:450
841925994116,16cyclictest13-21ksoftirqd/007:34:520
84192599411,19cyclictest1753-21gnome-shell09:34:340
84192599411,18cyclictest958040-21chrt12:35:160
84192599411,17cyclictest922438-21diskstats10:55:170
162399410,29rtkit-daemon2197014-21PK-Backend09:14:172
84193499408,9cyclictest896518-21sed09:40:313
841932994023,13cyclictest875930-40apt-check08:45:172
841932994022,13cyclictest438-20systemd-journal12:00:482
841932994015,6cyclictest141rcu_preempt10:45:192
841932994015,6cyclictest141rcu_preempt10:45:182
841929994023,13cyclictest957021-21vmstat12:30:321
841929994023,12cyclictest438-20systemd-journal08:00:011
841929994022,14cyclictest958470-21wc12:35:271
841929994014,9cyclictest26-21ksoftirqd/109:30:211
841925994026,10cyclictest13-21ksoftirqd/008:55:010
841925994024,12cyclictest13-21ksoftirqd/011:24:240
841925994023,13cyclictest13-21ksoftirqd/007:59:160
841925994022,7cyclictest13-21ksoftirqd/008:57:320
841925994022,13cyclictest2962348-21rs:main0
841925994013,6cyclictest141rcu_preempt07:38:360
841925994013,6cyclictest141rcu_preempt07:38:350
84192599401,18cyclictest945074-21seq11:57:490
84192599401,18cyclictest945074-21seq11:57:480
84192599401,18cyclictest1753-21gnome-shell12:31:030
84192599401,17cyclictest863035-21chrt08:08:070
162399400,28rtkit-daemon2197014-21PK-Backend08:54:032
162399400,28rtkit-daemon2197014-21PK-Backend07:34:462
162399400,27rtkit-daemon2197014-21PK-Backend09:22:412
8563432390,5chrt0-21swapper/207:50:022
84193499394,11cyclictest1753-21gnome-shell09:32:183
84192999399,17cyclictest931193-21kworker/1:112:13:021
84192999399,16cyclictest26-21ksoftirqd/111:06:081
84192999394,14cyclictest141rcu_preempt10:26:251
84192999392,33cyclictest1753-21gnome-shell09:59:321
841929993921,6cyclictest26-21ksoftirqd/110:15:011
841929993921,6cyclictest26-21ksoftirqd/108:30:011
841929993919,16cyclictest910131-21processes10:20:301
841929993917,10cyclictest26-21ksoftirqd/111:01:481
84192999391,21cyclictest26-21ksoftirqd/108:55:151
841929993911,15cyclictest26-21ksoftirqd/109:15:221
841929993911,14cyclictest26-21ksoftirqd/108:19:341
841929993911,14cyclictest26-21ksoftirqd/108:19:331
84192599399,13cyclictest13-21ksoftirqd/011:33:210
84192599395,31cyclictest0-21swapper/008:30:450
841925993929,7cyclictest13-21ksoftirqd/009:35:590
841925993922,13cyclictest13-21ksoftirqd/009:29:040
841925993922,12cyclictest896657-21pool-udisksd09:41:560
841925993921,7cyclictest13-21ksoftirqd/009:51:510
841925993919,12cyclictest13-21ksoftirqd/009:20:040
841925993919,12cyclictest13-21ksoftirqd/008:27:390
841925993912,6cyclictest141rcu_preempt08:35:150
84192599391,14cyclictest950490-21unin-run12:15:010
162399390,29rtkit-daemon0-21swapper/209:48:192
162399390,29rtkit-daemon0-21swapper/209:48:192
162399390,27rtkit-daemon2197014-21PK-Backend09:05:402
84193499386,10cyclictest937612-21ntp_states11:35:263
84193299382,31cyclictest935871-21ntp_states11:30:272
841932993821,13cyclictest959143-21/usr/sbin/munin12:40:012
841932993820,13cyclictest954200-21apt12:25:132
84193299381,9cyclictest1753-21gnome-shell07:39:392
84193299381,9cyclictest1753-21gnome-shell07:39:392
841932993818,16cyclictest3242253-21nautilus10:55:022
841932993813,5cyclictest141rcu_preempt08:30:012
84192999384,5cyclictest141rcu_preempt09:50:021
84192999384,5cyclictest141rcu_preempt09:50:011
841929993829,6cyclictest915978-21munin-plugin-st10:39:591
841929993827,7cyclictest935839-21ntp_kernel_err11:30:241
841929993826,8cyclictest882799-21apt-config09:05:011
84192999382,21cyclictest1753-21gnome-shell11:23:491
841929993821,6cyclictest26-21ksoftirqd/107:24:531
841929993821,5cyclictest26-21ksoftirqd/111:40:301
841929993821,13cyclictest855-21dbus-daemon07:30:011
841929993821,13cyclictest26-21ksoftirqd/108:05:161
841929993818,8cyclictest26-21ksoftirqd/109:55:011
84192599384,19cyclictest13-21ksoftirqd/007:50:200
841925993821,7cyclictest13-21ksoftirqd/010:07:000
841925993821,13cyclictest13-21ksoftirqd/011:26:430
841925993820,14cyclictest13-21ksoftirqd/010:23:250
841925993814,9cyclictest13-21ksoftirqd/010:31:090
841925993814,9cyclictest13-21ksoftirqd/010:31:080
84192599381,15cyclictest860384-21chrt08:00:180
841925993810,13cyclictest141rcu_preempt11:39:590
162399380,27rtkit-daemon2197014-21PK-Backend09:17:352
162399380,27rtkit-daemon2197014-21PK-Backend07:13:342
162399380,27rtkit-daemon2197014-21PK-Backend07:13:332
84193299372,18cyclictest1753-21gnome-shell08:41:122
84192999377,9cyclictest141rcu_preempt11:40:021
84192999377,15cyclictest26-21ksoftirqd/112:05:301
841929993721,12cyclictest944922-21sshd11:55:521
841929993721,12cyclictest944922-21sshd11:55:511
841929993721,11cyclictest922423-21expr10:55:141
841929993720,13cyclictest26-21ksoftirqd/107:10:251
841929993720,13cyclictest26-21ksoftirqd/107:10:251
841929993719,6cyclictest26-21ksoftirqd/110:05:151
841929993719,13cyclictest954253-21cpuspeed12:25:131
841929993717,8cyclictest141rcu_preempt08:32:041
841925993721,6cyclictest13-21ksoftirqd/007:11:380
841925993721,6cyclictest13-21ksoftirqd/007:11:370
841925993721,12cyclictest13-21ksoftirqd/012:09:040
841925993721,12cyclictest13-21ksoftirqd/010:28:410
841925993721,12cyclictest13-21ksoftirqd/008:11:430
841925993721,12cyclictest13-21ksoftirqd/007:22:350
162399370,26rtkit-daemon0-21swapper/207:56:362
162399370,25rtkit-daemon0-21swapper/208:57:042
84193299369,16cyclictest1753-21gnome-shell11:07:152
841932993621,10cyclictest34-21ksoftirqd/209:35:202
841932993619,12cyclictest922627-21cat10:55:212
841932993618,15cyclictest1753-21gnome-shell10:10:232
841929993619,13cyclictest26-21ksoftirqd/107:30:131
841929993617,7cyclictest141rcu_preempt09:41:091
841929993616,8cyclictest141rcu_preempt09:20:551
841929993616,8cyclictest141rcu_preempt08:35:201
841929993616,7cyclictest141rcu_preempt08:20:321
841929993614,9cyclictest843620-21kworker/1:207:50:131
841929993611,6cyclictest141rcu_preempt07:35:591
841929993611,6cyclictest141rcu_preempt07:35:591
162399360,25rtkit-daemon0-21swapper/208:13:182
84193299358,6cyclictest141rcu_preempt08:20:022
84193299358,6cyclictest141rcu_preempt08:20:012
84193299352,12cyclictest141rcu_preempt12:10:232
841932993520,11cyclictest911586-21cat10:25:232
841932993519,12cyclictest855-21dbus-daemon07:41:562
841932993518,13cyclictest1753-21gnome-shell07:50:592
841932993518,13cyclictest1632-21dbus-daemon11:55:022
841932993518,13cyclictest1632-21dbus-daemon11:55:012
841929993519,12cyclictest854947-21grep07:45:141
841929993519,11cyclictest916960-21ntp_kernel_err10:40:231
162399350,24rtkit-daemon0-21swapper/207:17:212
841932993422,8cyclictest938570-21/usr/sbin/munin11:40:312
84193299341,9cyclictest2197014-21PK-Backend09:42:072
841932993419,11cyclictest937254-21forks11:35:182
841932993418,12cyclictest941587-21seq11:47:342
841932993418,11cyclictest905302-21sshd10:06:392
84193299341,30cyclictest2197014-21PK-Backend08:31:332
84193299341,12cyclictest1753-21gnome-shell10:15:422
841929993423,7cyclictest863643-21cpuspeed_turbos08:10:171
162399340,27rtkit-daemon0-21swapper/210:21:192
8621842330,5sleep2331rcuc/208:05:222
841932993321,8cyclictest28750irq/122-ahci[0010:00:252
84193299331,9cyclictest2197014-21PK-Backend11:20:592
841932993317,12cyclictest944234-21if_enp2s011:55:232
841932993317,12cyclictest944234-21if_enp2s011:55:232
84193299331,4cyclictest2197014-21PK-Backend12:06:432
841929993319,10cyclictest895066-21seq09:38:461
841929993315,14cyclictest933790-21which11:25:181
841932993219,10cyclictest34-21ksoftirqd/211:28:362
84193299321,4cyclictest2197014-21PK-Backend12:31:072
84193299321,13cyclictest1069-21runrttasks12:24:172
84193299312,12cyclictest141rcu_preempt10:35:162
84193299309,17cyclictest1753-21gnome-shell11:03:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional