You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-21 - 16:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Aug 21, 2025 12:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991060,20rtkit-daemon0-21swapper/312:20:013
1623991040,17rtkit-daemon0-21swapper/308:35:313
162399970,21rtkit-daemon2197014-21PK-Backend11:30:093
162399890,10rtkit-daemon1753-21gnome-shell09:56:452
384387928340,36sleep30-21swapper/307:08:553
39314402820,4chrt3931439-21hddtemp_smartct11:10:192
162399750,5rtkit-daemon0-21swapper/207:22:552
38869012720,8sleep33242253-21nautilus09:10:023
384367027039,25sleep00-21swapper/007:06:150
384274326446,10sleep10-21swapper/107:05:161
162399630,9rtkit-daemon1622-21rtkit-daemon11:37:163
384288026247,9sleep20-21swapper/207:05:172
162399610,9rtkit-daemon1622-21rtkit-daemon10:58:272
162399610,9rtkit-daemon1622-21rtkit-daemon10:58:262
162399600,9rtkit-daemon1622-21rtkit-daemon12:34:553
162399600,9rtkit-daemon1622-21rtkit-daemon12:31:482
162399600,9rtkit-daemon1622-21rtkit-daemon11:18:463
162399590,8rtkit-daemon1622-21rtkit-daemon10:47:493
162399590,8rtkit-daemon1622-21rtkit-daemon10:20:263
162399580,9rtkit-daemon1622-21rtkit-daemon10:11:503
162399570,9rtkit-daemon1622-21rtkit-daemon10:16:343
162399570,8rtkit-daemon1622-21rtkit-daemon11:55:063
162399560,8rtkit-daemon1622-21rtkit-daemon12:07:463
162399540,9rtkit-daemon1622-21rtkit-daemon10:02:273
38929302530,13sleep234-21ksoftirqd/209:25:142
162399530,8rtkit-daemon1622-21rtkit-daemon11:52:332
162399530,8rtkit-daemon1622-21rtkit-daemon11:15:052
38704032520,6sleep32197014-21PK-Backend08:20:313
162399520,8rtkit-daemon1622-21rtkit-daemon12:04:583
162399520,8rtkit-daemon1622-21rtkit-daemon11:22:152
384412799491,19cyclictest438-20systemd-journal07:25:003
162399480,34rtkit-daemon1753-21gnome-shell07:38:252
3844127994731,12cyclictest855-21dbus-daemon09:18:523
3844124994716,18cyclictest1753-21gnome-shell10:35:142
3844121994713,6cyclictest39594012latency_hist12:30:031
162399470,33rtkit-daemon1753-21gnome-shell08:31:272
38934222460,5sleep10-21swapper/109:25:251
38790262460,17sleep126-21ksoftirqd/108:45:241
3844127994628,6cyclictest42-21ksoftirqd/312:35:143
384412199462,6cyclictest141rcu_preempt11:38:381
3844121994528,13cyclictest855-21dbus-daemon09:51:281
3844121994525,16cyclictest3872549-21sh08:30:001
3844121994525,16cyclictest3872549-21sh08:30:001
384412199452,39cyclictest854-21avahi-daemon10:02:511
162399450,36rtkit-daemon869-21irqbalance10:40:053
38995452440,12chrt251rcuc/109:45:011
38911202440,5sleep20-21swapper/209:20:142
3844127994427,12cyclictest438-20systemd-journal11:02:043
3844127994426,13cyclictest3905274-21/usr/sbin/munin10:00:023
3844127994424,15cyclictest3899948-40apt-check09:45:173
3844124994410,20cyclictest141rcu_preempt12:40:022
3844116994420,19cyclictest3848892-21memory07:20:220
39441602430,5sleep22197014-21PK-Backend11:45:212
384412799439,30cyclictest1753-21gnome-shell12:26:523
384412799439,29cyclictest854-21avahi-daemon09:35:353
3844127994325,13cyclictest438-20systemd-journal10:05:583
3844127994324,6cyclictest42-21ksoftirqd/311:50:273
3844127994320,6cyclictest42-21ksoftirqd/307:35:023
3844121994325,14cyclictest3939117-21package-system-11:30:291
384411699435,11cyclictest2197014-21PK-Backend07:33:370
3844116994317,7cyclictest38934232ntpq09:25:250
3844116994311,6cyclictest141rcu_preempt11:55:000
162399430,32rtkit-daemon1753-21gnome-shell07:51:032
39362052420,4chrt0-21swapper/011:25:030
38572572420,5chrt0-21swapper/207:45:162
3844127994221,16cyclictest855-21dbus-daemon11:10:013
3844124994225,13cyclictest855-21dbus-daemon09:35:012
3844124994212,26cyclictest3917475-40lsb_release10:30:282
384412199424,10cyclictest26-21ksoftirqd/111:40:481
384412199424,10cyclictest26-21ksoftirqd/111:40:481
3844121994228,10cyclictest26-21ksoftirqd/110:45:121
3844121994225,12cyclictest438-20systemd-journal09:20:011
3844121994222,16cyclictest3916547-21apt-config10:30:031
3844116994226,12cyclictest855-21dbus-daemon12:20:010
3844116994225,13cyclictest3848053-21systemctl07:20:000
162399420,29rtkit-daemon854-21avahi-daemon09:40:382
3844127994130,8cyclictest42-21ksoftirqd/308:35:023
3844127994125,12cyclictest3945106-21cron11:50:003
3844127994124,6cyclictest42-21ksoftirqd/309:20:123
3844127994123,6cyclictest42-21ksoftirqd/309:35:003
3844127994123,14cyclictest42-21ksoftirqd/310:35:163
3844127994123,14cyclictest42-21ksoftirqd/308:00:173
3844127994123,13cyclictest42-21ksoftirqd/311:20:233
384412799411,9cyclictest2197014-21PK-Backend08:54:233
3844127994117,20cyclictest3855323-21/usr/sbin/munin07:40:263
3844124994123,13cyclictest3958355-21awk12:25:262
3844121994130,7cyclictest3865974-21tr08:10:131
3844121994125,12cyclictest3865208-21users08:05:481
3844121994124,13cyclictest3875477-21cat08:35:231
3844121994122,14cyclictest855-21dbus-daemon12:10:001
3844121994117,19cyclictest3897033-21rt-features09:35:311
3844121994117,13cyclictest3845769-21cron07:15:001
3844121994112,14cyclictest38704042cat08:20:311
3844116994131,6cyclictest3916690-40apt-check10:30:160
3844116994126,11cyclictest3943074-21sshd11:41:240
3844116994126,11cyclictest3943074-21sshd11:41:240
3844116994120,12cyclictest13-21ksoftirqd/007:39:280
162399410,34rtkit-daemon1753-21gnome-shell10:51:523
162399410,33rtkit-daemon2197014-21PK-Backend11:55:472
162399410,32rtkit-daemon3844108-21cyclictest12:19:342
162399410,28rtkit-daemon2197014-21PK-Backend08:59:142
39107962400,6sleep234-21ksoftirqd/210:15:132
3844127994026,10cyclictest42-21ksoftirqd/311:28:263
3844127994023,6cyclictest42-21ksoftirqd/310:30:143
3844127994023,5cyclictest42-21ksoftirqd/312:10:273
3844127994022,6cyclictest42-21ksoftirqd/309:25:113
3844127994022,5cyclictest42-21ksoftirqd/307:40:003
3844127994018,19cyclictest860997-21snmpd11:13:303
384412799401,17cyclictest38574412expr07:45:173
384412499406,15cyclictest141rcu_preempt08:40:302
3844124994022,14cyclictest34-21ksoftirqd/209:09:202
3844124994022,13cyclictest3930036-21ntpq11:05:272
3844124994020,15cyclictest34-21ksoftirqd/208:25:052
3844124994020,15cyclictest34-21ksoftirqd/208:25:052
3844124994018,13cyclictest34-21ksoftirqd/208:23:282
384412199403,33cyclictest3853284-21munin-plugin-st07:35:001
3844121994026,10cyclictest26-21ksoftirqd/112:30:231
3844121994026,10cyclictest26-21ksoftirqd/109:00:171
3844121994025,11cyclictest3860022-21sshd07:51:211
3844121994023,13cyclictest3956822-21cat12:20:321
3844121994011,24cyclictest2179-21gvfsd-metadata09:45:031
384411699406,5cyclictest141rcu_preempt10:14:310
384411699404,9cyclictest13-21ksoftirqd/012:21:290
384411699403,10cyclictest13-21ksoftirqd/011:14:430
3844116994029,8cyclictest13-21ksoftirqd/010:20:160
3844116994026,10cyclictest3872560-21idleruntime-cro08:30:000
3844116994026,10cyclictest3872560-21idleruntime-cro08:30:000
3844116994025,11cyclictest3904067-21expr09:55:150
384411699402,34cyclictest1753-21gnome-shell08:46:450
3844116994023,12cyclictest438-20systemd-journal08:00:250
3844116994022,13cyclictest3962594-21sshd12:36:190
3844116994021,14cyclictest3926168-21df_inode10:55:170
3844116994021,14cyclictest3926168-21df_inode10:55:170
162399400,34rtkit-daemon127-1kworker/2:1H10:07:282
162399400,32rtkit-daemon3844108-21cyclictest09:53:463
162399400,32rtkit-daemon3844108-21cyclictest07:12:203
162399400,29rtkit-daemon2197014-21PK-Backend08:47:572
39217192390,7chrt3921715-21cron10:45:002
3844127993922,5cyclictest42-21ksoftirqd/311:40:253
3844127993922,5cyclictest42-21ksoftirqd/311:40:253
3844127993922,13cyclictest42-21ksoftirqd/310:55:313
3844127993922,13cyclictest42-21ksoftirqd/310:55:313
3844127993920,6cyclictest42-21ksoftirqd/309:00:323
3844124993922,13cyclictest34-21ksoftirqd/207:14:132
3844124993920,14cyclictest34-21ksoftirqd/210:30:022
384412199392,8cyclictest3892094-40lsb_release09:21:251
3844121993928,7cyclictest1753-21gnome-shell11:22:461
3844121993923,12cyclictest3925773-21/usr/sbin/munin10:55:121
3844121993923,12cyclictest3925773-21/usr/sbin/munin10:55:121
3844121993922,13cyclictest2181-21gdbus10:16:241
3844121993922,12cyclictest438-20systemd-journal08:17:201
3844121993917,18cyclictest1617-10pipewire-media-07:45:351
384411699396,5cyclictest141rcu_preempt07:40:200
384411699396,11cyclictest13-21ksoftirqd/011:07:280
384411699393,13cyclictest3242253-21nautilus08:25:010
3844116993926,8cyclictest904-21systemd-logind11:56:490
3844116993922,13cyclictest13-21ksoftirqd/009:33:450
384411699392,10cyclictest13-21ksoftirqd/009:45:230
3844116993917,17cyclictest13-21ksoftirqd/007:29:120
162399390,28rtkit-daemon2197014-21PK-Backend09:37:392
162399390,28rtkit-daemon2197014-21PK-Backend09:10:192
162399390,27rtkit-daemon2197014-21PK-Backend09:02:502
162399390,27rtkit-daemon2197014-21PK-Backend08:08:422
162399390,26rtkit-daemon2197014-21PK-Backend08:16:312
39366892380,5sleep0161rcuc/011:25:230
3844127993824,10cyclictest42-21ksoftirqd/307:52:513
3844127993821,13cyclictest855-21dbus-daemon10:30:013
384412499385,10cyclictest34-21ksoftirqd/211:43:242
384412499385,10cyclictest34-21ksoftirqd/211:43:232
3844124993827,7cyclictest1753-21gnome-shell10:47:072
3844124993826,8cyclictest3902178-21tr09:50:182
3844124993822,12cyclictest3939085-21ntp_states11:30:272
3844124993820,14cyclictest34-21ksoftirqd/208:36:112
3844124993819,6cyclictest34-21ksoftirqd/207:44:102
3844124993817,16cyclictest3242253-21nautilus10:02:552
3844121993823,11cyclictest3863307-21threads08:00:331
3844121993822,12cyclictest3937789-21apt-get11:30:001
3844121993821,13cyclictest3885340-21/usr/sbin/munin09:05:261
3844121993821,12cyclictest438-20systemd-journal08:55:001
3844121993817,17cyclictest3853518-21/usr/sbin/munin07:35:291
3844121993813,14cyclictest3846371-21cpuspeed07:15:051
3844121993811,13cyclictest1069-21runrttasks07:44:271
384412199381,10cyclictest0-21swapper/107:56:381
384411699384,9cyclictest13-21ksoftirqd/011:32:060
384411699382,9cyclictest1358-1kworker/u9:112:13:250
3844116993827,7cyclictest1753-21gnome-shell09:14:080
3844116993825,9cyclictest1753-21gnome-shell07:14:290
3844116993824,10cyclictest3880998-21python308:50:300
384411699382,32cyclictest3857853-21ntp_states07:45:270
384411699382,32cyclictest1753-21gnome-shell11:03:430
384411699382,32cyclictest1753-21gnome-shell08:08:170
3844116993821,5cyclictest13-21ksoftirqd/008:40:090
3844116993821,13cyclictest13-21ksoftirqd/010:46:140
3844116993821,12cyclictest3907296-21apt10:05:140
3844116993819,15cyclictest3958832-21seq12:26:490
3844116993819,14cyclictest13-21ksoftirqd/011:46:360
3844116993819,14cyclictest13-21ksoftirqd/008:58:150
384411699381,33cyclictest3949838-21sed12:00:420
384411699381,11cyclictest2197014-21PK-Backend08:36:220
162399380,26rtkit-daemon2197014-21PK-Backend07:29:392
3844127993725,7cyclictest2197014-21PK-Backend12:15:033
3844127993723,10cyclictest855-21dbus-daemon07:30:013
3844127993723,10cyclictest42-21ksoftirqd/308:27:493
3844127993723,10cyclictest42-21ksoftirqd/308:27:483
384412799372,12cyclictest3242253-21nautilus09:45:003
3844124993727,6cyclictest3949542-21tail12:00:272
3844124993724,8cyclictest1800-21upowerd10:20:172
3844124993720,12cyclictest3953027-21needreboot12:10:242
3844124993719,13cyclictest438-20systemd-journal10:52:032
3844124993711,22cyclictest3853301-21pool-org.gnome.07:35:002
3844121993727,7cyclictest3882396-21grep08:55:231
3844121993723,10cyclictest3928701-21sshd11:01:071
3844121993723,10cyclictest3877172-21cat08:40:231
3844121993723,10cyclictest26-21ksoftirqd/111:15:021
3844121993723,10cyclictest26-21ksoftirqd/110:38:131
3844121993721,12cyclictest3930464-21sshd11:06:191
3844121993719,13cyclictest3931407-21df_inode11:10:161
384411699378,5cyclictest141rcu_preempt09:18:420
3844116993728,5cyclictest2197014-21PK-Backend08:34:530
3844116993721,12cyclictest918-21gdbus08:11:560
3844116993720,13cyclictest855-21dbus-daemon10:45:000
3844116993720,12cyclictest3897917-21find09:40:110
384411699371,8cyclictest3960028-21forks12:30:200
384411699371,8cyclictest0-21swapper/009:51:030
39371652360,9sleep234-21ksoftirqd/211:25:332
3844127993619,5cyclictest42-21ksoftirqd/308:05:443
3844127993618,14cyclictest1753-21gnome-shell08:14:583
384412499363,7cyclictest17531gnome-shell07:56:142
3844124993618,13cyclictest34-21ksoftirqd/210:14:482
3844124993617,14cyclictest854-21avahi-daemon09:18:512
384412499361,31cyclictest854-21avahi-daemon08:12:572
384412199369,23cyclictest3909540-21kernelversion10:10:231
3844121993624,8cyclictest1826-21JS14:31:441
3844121993624,8cyclictest1753-21gnome-shell07:21:411
3844121993622,11cyclictest26-21ksoftirqd/109:55:211
3844121993621,11cyclictest3925220-21seq10:51:261
3844121993620,11cyclictest3943719-21cat11:45:121
3844121993619,13cyclictest1738-21gdbus11:51:561
3844121993619,13cyclictest1-21systemd12:05:011
384411699363,6cyclictest141rcu_preempt10:25:240
3844116993620,12cyclictest3891773-21awk09:20:310
3844116993620,12cyclictest3885257-21ls09:05:020
3844116993615,16cyclictest3925231-21gdbus10:51:270
384411699361,10cyclictest0-21swapper/007:56:360
3844127993522,9cyclictest1753-21gnome-shell08:55:303
3844127993520,10cyclictest42-21ksoftirqd/307:15:233
3844124993524,8cyclictest3863161-21grep08:00:312
3844124993520,11cyclictest3951020-21awk12:05:152
3844124993520,11cyclictest3847721-21sshd07:16:202
3844124993519,12cyclictest3900218-21cat09:45:152
3844124993518,12cyclictest3881632-21pool-org.gnome.08:55:012
3844124993516,6cyclictest34-21ksoftirqd/212:24:012
3844121993520,11cyclictest3952831-21cat12:10:211
3844121993518,6cyclictest26-21ksoftirqd/108:33:131
3844121993518,12cyclictest3962612-21sshd12:36:291
3844121993518,12cyclictest3907486-21grep10:05:151
384411699353,11cyclictest13-21ksoftirqd/011:35:170
384411699352,30cyclictest0-21swapper/012:08:080
3844116993522,9cyclictest1753-21gnome-shell07:51:410
3844116993522,8cyclictest3933502-21tr11:15:250
3844116993520,11cyclictest3919453-21cat10:35:310
3844116993518,13cyclictest3867568-21basename08:15:130
384411699351,30cyclictest3906282-21cat10:00:290
384411699351,30cyclictest2179-21gvfsd-metadata10:16:250
3844127993424,6cyclictest42-21ksoftirqd/308:44:303
3844127993423,7cyclictest42-21ksoftirqd/309:10:293
3844127993422,6cyclictest42-21ksoftirqd/308:49:143
3844127993421,9cyclictest1753-21gnome-shell08:16:253
384412799341,30cyclictest2197014-21PK-Backend07:55:483
3844124993416,13cyclictest34-21ksoftirqd/211:35:512
3844121993423,7cyclictest3955167-21timerwakeupswit12:15:311
3844121993422,8cyclictest3947940-21wc11:55:271
3844121993421,9cyclictest64850irq/126-i91507:25:191
3844121993419,11cyclictest3894624-21seq09:30:141
3844121993416,13cyclictest3913260-21systemd-udevd10:20:171
384412199341,28cyclictest3921386-21sshd10:40:531
3844116993426,4cyclictest2197014-21PK-Backend09:35:560
384411699342,29cyclictest0-21swapper/009:07:230
3844124993318,11cyclictest3928455-21switchtime11:00:312
3844121993318,11cyclictest3915284-21load10:25:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional