You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-10 - 01:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Nov 09, 2025 12:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
33597622810,7sleep32197014-21PK-Backend08:50:143
33236902790,5sleep00-21swapper/007:10:010
33987322780,5sleep02197014-21PK-Backend10:40:150
33723702780,9sleep3411rcuc/309:25:203
162399780,5rtkit-daemon0-21swapper/109:32:081
332320327753,16sleep20-21swapper/207:06:152
162399770,5rtkit-daemon0-21swapper/110:52:481
332319927546,17sleep10-21swapper/107:06:131
332201827548,17sleep30-21swapper/307:05:103
33316832730,4sleep00-21swapper/007:30:130
162399710,8rtkit-daemon1622-21rtkit-daemon09:52:443
162399670,8rtkit-daemon1622-21rtkit-daemon11:19:272
162399660,9rtkit-daemon1622-21rtkit-daemon09:55:062
162399660,9rtkit-daemon1622-21rtkit-daemon09:54:072
162399650,9rtkit-daemon1622-21rtkit-daemon12:36:502
162399650,9rtkit-daemon1622-21rtkit-daemon12:36:492
33840502630,18sleep126-21ksoftirqd/109:59:551
162399630,8rtkit-daemon1622-21rtkit-daemon07:24:272
162399580,9rtkit-daemon1622-21rtkit-daemon08:15:133
33984042570,8sleep32197014-21PK-Backend10:40:023
162399570,8rtkit-daemon1622-21rtkit-daemon12:09:243
162399570,8rtkit-daemon1622-21rtkit-daemon09:02:033
332368099560,51cyclictest1622-21rtkit-daemon10:14:262
33846612540,7chrt0-21swapper/110:00:151
33325002540,8chrt1753-21gnome-shell07:30:311
162399540,8rtkit-daemon1622-21rtkit-daemon11:08:323
162399530,6rtkit-daemon0-21swapper/111:04:081
33515462520,5sleep00-21swapper/008:25:280
3323685995218,19cyclictest42-21ksoftirqd/310:25:013
162399520,5rtkit-daemon860997-21snmpd11:27:332
34286852510,5sleep00-21swapper/012:05:160
33930722510,7chrt0-21swapper/010:25:030
34015792500,16sleep342-21ksoftirqd/310:45:343
332368599506,9cyclictest42-21ksoftirqd/311:55:013
332368599504,16cyclictest42-21ksoftirqd/311:20:013
3323685995017,19cyclictest42-21ksoftirqd/307:40:013
332368599501,16cyclictest42-21ksoftirqd/310:00:023
162399500,43rtkit-daemon3323666-21cyclictest10:16:463
162399500,43rtkit-daemon3323666-21cyclictest10:16:463
33441732490,4sleep10-21swapper/108:05:171
332368599497,15cyclictest42-21ksoftirqd/311:45:153
332368599496,9cyclictest42-21ksoftirqd/307:20:013
332368599485,7cyclictest42-21ksoftirqd/309:50:003
332368599485,12cyclictest42-21ksoftirqd/312:40:003
332368599485,12cyclictest42-21ksoftirqd/312:40:003
3323680994825,19cyclictest1753-21gnome-shell07:55:302
332368599479,13cyclictest42-21ksoftirqd/310:40:203
3323685994715,18cyclictest42-21ksoftirqd/307:55:023
3323685994712,19cyclictest42-21ksoftirqd/308:10:023
3323671994736,8cyclictest1753-21gnome-shell09:09:350
162399470,38rtkit-daemon3323666-21cyclictest11:54:262
3323685994612,20cyclictest42-21ksoftirqd/310:05:013
3323680994618,8cyclictest3391026-21basename10:20:002
3323680994618,8cyclictest3391026-21basename10:20:002
332367199464,6cyclictest141rcu_preempt11:15:010
162399460,39rtkit-daemon1903-21packagekitd07:14:503
332368599459,14cyclictest42-21ksoftirqd/312:05:033
332368599455,8cyclictest42-21ksoftirqd/308:04:173
332368599451,19cyclictest42-21ksoftirqd/307:40:093
332368599451,19cyclictest42-21ksoftirqd/307:40:093
332368599451,11cyclictest42-21ksoftirqd/312:25:013
3323685994511,18cyclictest42-21ksoftirqd/308:50:013
3323685994511,18cyclictest42-21ksoftirqd/308:50:013
3323680994511,5cyclictest141rcu_preempt10:45:262
3323675994515,13cyclictest3416330-21grep11:30:171
332368599449,17cyclictest42-21ksoftirqd/311:55:153
332368599447,16cyclictest42-21ksoftirqd/309:25:013
332368599446,9cyclictest42-21ksoftirqd/308:40:013
332368599444,13cyclictest42-21ksoftirqd/308:15:003
3323685994415,10cyclictest42-21ksoftirqd/311:15:003
3323685994411,19cyclictest42-21ksoftirqd/312:25:153
3323685994411,18cyclictest42-21ksoftirqd/310:27:253
3323685994410,19cyclictest42-21ksoftirqd/311:00:023
332368099443,12cyclictest34-21ksoftirqd/211:00:252
3323680994427,13cyclictest3362683-40apt-check08:57:162
3323675994423,17cyclictest3350874-21/usr/sbin/munin08:25:151
3323671994426,14cyclictest3436741-40apt-check12:27:190
162399440,37rtkit-daemon1753-21gnome-shell08:45:522
162399440,37rtkit-daemon1753-21gnome-shell08:45:522
162399440,37rtkit-daemon1753-21gnome-shell07:16:452
332368599436,12cyclictest42-21ksoftirqd/308:20:143
332368599435,16cyclictest42-21ksoftirqd/312:10:213
332368599434,8cyclictest42-21ksoftirqd/312:20:023
3323685994334,5cyclictest411rcuc/311:35:143
3323685994334,5cyclictest411rcuc/309:15:003
3323685994323,16cyclictest3395172-21sort10:30:143
332368599432,15cyclictest42-21ksoftirqd/307:31:573
3323685994318,11cyclictest860997-21snmpd12:31:383
332368599431,17cyclictest42-21ksoftirqd/311:40:253
3323685994310,12cyclictest42-21ksoftirqd/307:45:253
3323680994325,14cyclictest855-21dbus-daemon09:05:002
3323680994310,12cyclictest3432305-21ls12:15:222
3323671994331,8cyclictest1069-21runrttasks09:27:150
3323671994328,11cyclictest3336585-21cron07:45:000
3323671994328,11cyclictest3336585-21cron07:45:000
3323671994317,8cyclictest13-21ksoftirqd/010:10:270
3323671994312,12cyclictest141rcu_preempt08:20:020
162399430,35rtkit-daemon2197014-21PK-Backend09:43:163
332368599425,8cyclictest42-21ksoftirqd/309:35:033
332368599424,10cyclictest42-21ksoftirqd/311:35:013
3323685994234,5cyclictest411rcuc/309:17:583
3323685994221,8cyclictest42-21ksoftirqd/308:28:013
3323685994210,17cyclictest42-21ksoftirqd/308:40:183
3323680994229,9cyclictest1753-21gnome-shell10:51:062
332368099422,18cyclictest1753-21gnome-shell11:35:362
3323680994217,12cyclictest141rcu_preempt09:30:142
3323680994214,12cyclictest141rcu_preempt08:05:032
332368099421,12cyclictest860997-21snmpd10:28:432
3323675994222,15cyclictest3242253-21nautilus07:30:011
3323675994214,11cyclictest3386963-21switchtime10:05:301
332367199428,20cyclictest1753-21gnome-shell07:45:220
3323671994213,25cyclictest3398406-21latency_hist10:40:020
332368599417,18cyclictest42-21ksoftirqd/309:30:153
332368599416,20cyclictest42-21ksoftirqd/311:30:003
332368599416,16cyclictest2197014-21PK-Backend11:20:303
332368599416,16cyclictest2197014-21PK-Backend11:20:303
332368099419,16cyclictest141rcu_preempt08:10:012
3323680994122,14cyclictest3335501-21df07:40:182
3323680994122,14cyclictest3335501-21df07:40:172
3323680994119,11cyclictest3379778-21cat09:45:252
332367599411,8cyclictest0-21swapper/109:07:301
3323675994113,24cyclictest3343614-21ls08:05:021
3323671994125,12cyclictest3375102-21apt-get09:35:010
3323671994122,7cyclictest1753-21gnome-shell12:14:520
33336592400,5chrt0-21swapper/207:35:152
332368599407,18cyclictest42-21ksoftirqd/308:55:193
3323685994024,12cyclictest860997-21snmpd08:33:373
332368599402,13cyclictest2197014-21PK-Backend10:50:353
3323685994017,8cyclictest42-21ksoftirqd/310:05:283
332368099407,17cyclictest141rcu_preempt07:50:142
3323680994017,19cyclictest3332487-31apt-check07:30:302
3323675994024,12cyclictest3327209-21sshd07:15:381
3323675994020,16cyclictest3388920-20journal-offline10:10:421
3323675994017,18cyclictest3361227-21gpgconf08:55:031
3323675994014,8cyclictest3335320-21grep07:40:161
3323675994014,8cyclictest3335320-21grep07:40:151
3323671994024,12cyclictest3377123-21cron09:39:590
332367199402,21cyclictest3326479-21diskstats07:15:150
3323671994021,15cyclictest3402773-21irqstats10:50:200
3323671994015,7cyclictest141rcu_preempt09:52:060
162399400,32rtkit-daemon34-21ksoftirqd/211:09:362
34099132390,7chrt2197014-21PK-Backend11:10:242
34006352390,12chrt0-21swapper/110:45:151
332368599396,15cyclictest42-21ksoftirqd/311:00:323
332368599395,13cyclictest17531gnome-shell07:55:043
332368599395,11cyclictest42-21ksoftirqd/307:20:303
3323685993914,7cyclictest42-21ksoftirqd/309:07:543
3323685993910,14cyclictest42-21ksoftirqd/307:25:473
332368099399,12cyclictest141rcu_preempt07:10:172
332368099398,12cyclictest3357625-21cat08:45:022
332368099392,6cyclictest141rcu_preempt12:10:012
332368099392,33cyclictest1753-21gnome-shell09:18:322
3323680993922,12cyclictest3399983-21apt-get10:45:022
3323680993922,12cyclictest3347804-21cat08:15:222
3323680993913,13cyclictest141rcu_preempt12:27:472
3323680993912,12cyclictest141rcu_preempt11:00:022
332368099391,20cyclictest3371796-21latency_hist09:25:022
332368099391,12cyclictest3337133-21expr07:45:162
3323680993910,9cyclictest1069-21runrttasks08:12:562
332367599397,9cyclictest26-21ksoftirqd/108:25:001
3323675993922,6cyclictest26-21ksoftirqd/111:20:011
3323675993919,16cyclictest3431752-21/usr/sbin/munin12:15:261
332367599391,7cyclictest3338335-21apt07:49:591
3323675993915,10cyclictest1069-21runrttasks11:40:241
3323675993912,23cyclictest3376788-40apt-check09:36:171
332367199392,20cyclictest3362716-40lsb_release08:57:230
332367199392,19cyclictest1753-21gnome-shell10:05:210
332367199392,18cyclictest1753-21gnome-shell07:10:160
332367199392,17cyclictest3350142-21users08:20:320
3323671993920,15cyclictest3406884-20journal-offline11:00:500
332367199391,19cyclictest3413593-21cat11:20:300
332367199391,19cyclictest3413593-21cat11:20:300
332368599386,9cyclictest2197014-21PK-Backend10:10:223
332368099389,11cyclictest141rcu_preempt10:05:202
332368099384,18cyclictest141rcu_preempt10:30:242
3323680993825,9cyclictest1753-21gnome-shell08:36:572
3323680993824,10cyclictest3331154-21sort07:30:012
332368099382,32cyclictest1753-21gnome-shell08:22:152
332368099382,21cyclictest175350gnome-shell08:50:252
3323680993818,16cyclictest3376788-40apt-check09:36:192
332368099381,15cyclictest1753-21gnome-shell11:45:302
3323680993810,11cyclictest141rcu_preempt11:40:232
332367599383,13cyclictest26-21ksoftirqd/111:20:111
332367599383,13cyclictest26-21ksoftirqd/111:20:111
3323675993826,8cyclictest3393860-21memory10:25:231
3323675993819,14cyclictest918-21gdbus11:11:561
332367199384,16cyclictest1753-21gnome-shell09:59:300
332367199383,8cyclictest1753-21gnome-shell10:55:410
332367199383,17cyclictest1753-21gnome-shell11:32:070
332367199383,16cyclictest1753-21gnome-shell08:53:550
332367199382,9cyclictest1753-21gnome-shell08:15:020
3323671993823,11cyclictest3353793-21seq08:32:400
3323671993822,12cyclictest3422822-21cron11:50:000
332367199382,19cyclictest3432143-21df12:15:160
3323671993820,13cyclictest3401266-21ntpq10:45:270
332367199381,6cyclictest2197014-21PK-Backend08:05:150
162399380,31rtkit-daemon2197014-21PK-Backend11:23:012
162399380,31rtkit-daemon2197014-21PK-Backend11:23:002
162399380,26rtkit-daemon0-21swapper/110:20:441
34359652370,11chrt251rcuc/112:25:211
332368099379,12cyclictest141rcu_preempt12:25:032
332368099377,12cyclictest141rcu_preempt12:10:212
332368099372,9cyclictest3611406-21snapd11:30:172
3323680993723,10cyclictest34-21ksoftirqd/209:15:012
3323680993722,11cyclictest3392728-21cron10:24:592
3323680993712,5cyclictest141rcu_preempt10:00:142
332368099371,21cyclictest175350gnome-shell08:25:192
3323675993726,7cyclictest3382336-21sh09:55:011
3323675993723,10cyclictest3419364-21perf11:40:001
3323675993720,13cyclictest26-21ksoftirqd/109:15:011
3323675993718,14cyclictest438-20systemd-journal09:29:551
3323675993715,6cyclictest33576212latency_hist08:45:021
3323675993712,21cyclictest3341867-21cut08:00:011
332367199373,19cyclictest3426957-21diskstats12:00:180
332367199373,16cyclictest3333933-21memory07:35:220
332367199372,7cyclictest3368169-21turbostat09:14:590
332367199372,19cyclictest3371249-21users09:20:320
332367199372,18cyclictest3415517-21users11:25:320
332367199372,18cyclictest3393485-21/usr/sbin/munin10:25:150
332367199372,18cyclictest3385093-21ntp_states10:00:270
332367199372,18cyclictest3356743-21ntp_states08:40:290
332367199372,18cyclictest3329950-21diskstats07:25:180
332367199372,18cyclictest1753-21gnome-shell07:20:340
332367199372,17cyclictest3342436-21df_inode08:00:160
332367199372,17cyclictest1753-21gnome-shell09:40:080
332367199372,17cyclictest0-21swapper/007:50:350
162399370,25rtkit-daemon0-21swapper/107:54:581
332368099365,12cyclictest141rcu_preempt09:40:252
3323680993626,7cyclictest1753-21gnome-shell08:34:272
332368099362,12cyclictest3437666-21diskstats12:30:192
332368099362,11cyclictest1753-21gnome-shell09:09:392
3323680993618,13cyclictest3372644-21ntpq09:25:252
3323680993610,10cyclictest141rcu_preempt10:35:332
332367599369,11cyclictest26-21ksoftirqd/112:03:061
3323675993625,7cyclictest26-21ksoftirqd/107:13:321
3323675993625,6cyclictest26-21ksoftirqd/111:00:021
3323675993624,9cyclictest0-21swapper/107:39:011
3323675993623,10cyclictest26-21ksoftirqd/111:25:131
3323675993622,9cyclictest3242253-21nautilus09:05:001
3323675993622,10cyclictest3438763-21/usr/sbin/munin12:35:001
3323675993622,10cyclictest26-21ksoftirqd/108:55:261
3323675993620,12cyclictest3408760-21sshd11:07:401
3323675993618,13cyclictest438-20systemd-journal11:50:001
3323675993615,17cyclictest26-21ksoftirqd/108:30:221
3323675993614,6cyclictest141rcu_preempt11:55:281
332367199365,12cyclictest0-21swapper/009:45:240
332367199362,17cyclictest3340739-21hddtemp_smartct07:55:190
3323671993621,11cyclictest3364704-21sh09:05:010
332367199362,10cyclictest141rcu_preempt11:37:240
3323671993619,12cyclictest3354969-21ntpq08:35:240
3323671993617,14cyclictest3242253-21nautilus08:50:020
3323671993617,14cyclictest3242253-21nautilus08:50:020
332367199361,19cyclictest3368951-21latency09:15:220
332368099351,19cyclictest3426558-21find12:00:122
3323675993521,10cyclictest3434988-21ls12:25:001
3323675993521,10cyclictest3377575-21cpuspeed_turbos09:40:141
3323675993521,10cyclictest3345990-21sshd08:10:221
3323675993520,11cyclictest3358341-21wc08:45:231
3323675993520,11cyclictest3358341-21wc08:45:231
3323675993520,11cyclictest26-21ksoftirqd/112:09:591
3323675993514,16cyclictest26-21ksoftirqd/112:35:161
3323675993514,16cyclictest26-21ksoftirqd/112:35:161
3323671993520,11cyclictest3439667-21cat12:35:250
3323671993520,11cyclictest3439667-21cat12:35:250
3323671993519,12cyclictest3437683-21fw_packets12:30:200
3323671993519,12cyclictest3411364-21sh11:15:200
332367199351,30cyclictest2197014-21PK-Backend11:55:010
332367199351,13cyclictest3407469-21chrt11:05:110
3323680993420,10cyclictest34-21ksoftirqd/211:55:142
3323675993425,6cyclictest3379268-21tr09:45:171
3323675993423,7cyclictest3398496-21/usr/sbin/munin10:40:321
3323675993419,11cyclictest3371117-21find09:20:321
3323675993419,11cyclictest3369924-21systemctl09:20:001
3323675993419,10cyclictest1753-21gnome-shell10:33:181
3323675993417,13cyclictest3328052-21cstates07:20:171
3323675993411,6cyclictest3430893-21ps12:10:261
332367199348,7cyclictest1753-21gnome-shell10:30:040
3323671993419,11cyclictest3425870-21cat11:55:330
3323675993322,7cyclictest26-21ksoftirqd/108:35:141
3323675993321,7cyclictest26-21ksoftirqd/110:35:291
332367599332,10cyclictest1753-21gnome-shell10:15:151
332367599332,10cyclictest1753-21gnome-shell10:15:141
3323675993318,11cyclictest3348398-21gdbus08:15:501
3323675993314,15cyclictest26-21ksoftirqd/111:50:231
332367199331,8cyclictest3434412-21sort12:20:300
332367199331,11cyclictest3420777-21sshd11:41:030
332367199321,27cyclictest2197014-21PK-Backend10:16:300
332367199321,27cyclictest2197014-21PK-Backend10:16:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional