You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 05:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri May 09, 2025 00:44:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
294402722240,8sleep20-21swapper/219:05:242
295699521000,7sleep00-21swapper/019:40:270
30000642970,9sleep11753-21gnome-shell21:45:181
162399900,78rtkit-daemon0-21swapper/319:05:483
30416472870,4sleep30-21swapper/323:40:303
162399820,10rtkit-daemon1753-21gnome-shell22:37:043
162399810,10rtkit-daemon1622-21rtkit-daemon19:50:432
162399810,10rtkit-daemon1622-21rtkit-daemon19:50:432
30411762800,6sleep10-21swapper/123:40:211
29792742800,5sleep2331rcuc/220:45:172
162399790,6rtkit-daemon2197014-21PK-Backend22:05:073
162399790,6rtkit-daemon2197014-21PK-Backend22:05:073
162399770,7rtkit-daemon2197014-21PK-Backend21:46:493
162399770,6rtkit-daemon0-21swapper/320:21:193
162399770,5rtkit-daemon0-21swapper/322:20:143
162399760,6rtkit-daemon2197014-21PK-Backend23:15:543
162399740,6rtkit-daemon0-21swapper/320:09:573
294452826948,14sleep00-21swapper/019:09:000
294395426844,15sleep10-21swapper/119:05:241
30572362660,5chrt0-21swapper/000:25:310
162399610,9rtkit-daemon1622-21rtkit-daemon19:43:293
162399580,9rtkit-daemon1622-21rtkit-daemon19:46:222
162399580,8rtkit-daemon1622-21rtkit-daemon19:17:313
29952212570,5sleep22197014-21PK-Backend21:30:252
2944771995721,11cyclictest42-21ksoftirqd/319:20:293
2944771995716,11cyclictest1-21systemd20:05:013
2944771995716,10cyclictest2977774-21wc20:40:223
2944771995716,10cyclictest2977774-21wc20:40:213
162399570,9rtkit-daemon1622-21rtkit-daemon23:10:240
2944771995611,40cyclictest3038743-21cron23:35:013
2944771995611,40cyclictest3038743-21cron23:35:013
294477199552,27cyclictest1753-21gnome-shell23:06:363
294477199542,25cyclictest1753-21gnome-shell21:27:493
294477199542,25cyclictest0-21swapper/300:36:233
294477199532,26cyclictest1753-21gnome-shell00:07:313
294477199532,25cyclictest1753-21gnome-shell23:03:153
294477199532,25cyclictest1753-21gnome-shell20:16:203
2944771995319,8cyclictest1-21systemd19:40:013
2944771995319,13cyclictest2972068-21df20:25:173
2944771995317,7cyclictest2197014-21PK-Backend20:11:373
2944771995316,12cyclictest994-21lldpd23:14:043
29487812520,5sleep0161rcuc/019:20:030
294477199529,16cyclictest1753-21gnome-shell22:15:163
294477199529,16cyclictest1753-21gnome-shell22:12:423
294477199522,7cyclictest1753-21gnome-shell21:20:173
294477199522,7cyclictest1753-21gnome-shell21:20:173
294477199522,25cyclictest1753-21gnome-shell20:55:223
2944771995220,8cyclictest1753-21gnome-shell00:22:073
2944771995219,8cyclictest1753-21gnome-shell00:00:133
2944771995219,7cyclictest3005233-21cpu22:00:163
2944771995218,7cyclictest1069-21runrttasks22:25:533
2944771995218,7cyclictest1069-21runrttasks22:25:523
2944771995210,16cyclictest1753-21gnome-shell19:55:443
294476899522,10cyclictest34-21ksoftirqd/200:00:022
29530702510,16sleep342-21ksoftirqd/319:30:193
294477199519,17cyclictest1753-21gnome-shell20:47:323
294477199512,8cyclictest1753-21gnome-shell19:25:293
2944771995122,8cyclictest1753-21gnome-shell19:50:283
2944771995122,8cyclictest1753-21gnome-shell19:50:283
2944771995117,7cyclictest1069-21runrttasks21:17:473
294477199511,22cyclictest1753-21gnome-shell00:10:233
2944771995110,16cyclictest1753-21gnome-shell23:25:273
2944771995110,16cyclictest1753-21gnome-shell22:42:363
2944771995110,16cyclictest1753-21gnome-shell22:42:363
2944768995111,22cyclictest34-21ksoftirqd/223:55:022
2944761995115,10cyclictest13-21ksoftirqd/022:05:400
2944761995115,10cyclictest13-21ksoftirqd/022:05:400
162399510,37rtkit-daemon2980662-21cpu20:50:123
30335432500,11sleep03033568-21munin-run23:19:590
294477199509,16cyclictest1753-21gnome-shell22:46:163
294477199509,16cyclictest1753-21gnome-shell00:29:443
294477199502,22cyclictest1753-21gnome-shell00:32:533
294477199502,22cyclictest1753-21gnome-shell00:32:523
2944771995018,8cyclictest1753-21gnome-shell20:39:123
2944771995015,10cyclictest42-21ksoftirqd/321:03:163
294477199501,23cyclictest1753-21gnome-shell23:59:333
2944768995030,8cyclictest34-21ksoftirqd/220:39:382
294477199498,16cyclictest1753-21gnome-shell19:48:113
294477199491,7cyclictest30259112sleep322:58:443
2944771994917,7cyclictest2197014-21PK-Backend21:07:403
294477199491,22cyclictest1753-21gnome-shell23:37:483
2944768994927,8cyclictest34-21ksoftirqd/220:20:002
294477199489,16cyclictest1753-21gnome-shell00:18:183
294477199482,24cyclictest3043066-21memory23:45:243
294477199481,22cyclictest1753-21gnome-shell21:55:543
294476899486,9cyclictest34-21ksoftirqd/221:20:012
2944768994821,8cyclictest34-21ksoftirqd/223:00:002
294476899476,19cyclictest141rcu_preempt20:33:502
2944768994713,19cyclictest34-21ksoftirqd/200:35:002
2944768994713,19cyclictest34-21ksoftirqd/200:35:002
2944768994711,20cyclictest34-21ksoftirqd/223:15:032
162399470,36rtkit-daemon0-21swapper/200:29:132
294477199469,16cyclictest1753-21gnome-shell21:41:103
2944768994618,10cyclictest34-21ksoftirqd/223:15:232
294476199461,41cyclictest1622-21rtkit-daemon00:05:260
2944771994514,9cyclictest3017178-21sed22:35:003
294476899457,20cyclictest34-21ksoftirqd/222:50:012
294476899456,18cyclictest34-21ksoftirqd/219:20:282
2944768994534,8cyclictest34-21ksoftirqd/200:20:332
2944768994513,12cyclictest1753-21gnome-shell23:45:252
294477199441,6cyclictest2946453-21nin-node19:15:023
294477199441,6cyclictest2946453-21nin-node19:15:023
294476899446,4cyclictest141rcu_preempt23:35:012
294476899446,4cyclictest141rcu_preempt23:35:002
2944768994425,8cyclictest34-21ksoftirqd/222:30:202
2944768994422,18cyclictest2953069-21hddtemp_smartct19:30:202
2944768994418,8cyclictest34-21ksoftirqd/219:25:142
2944768994416,10cyclictest34-21ksoftirqd/220:50:392
294476899437,20cyclictest34-21ksoftirqd/221:05:012
2944768994332,7cyclictest1069-21runrttasks22:26:222
2944768994332,7cyclictest1069-21runrttasks22:26:222
2944768994322,10cyclictest141rcu_preempt22:22:192
2944768994320,9cyclictest34-21ksoftirqd/221:28:112
2944768994320,8cyclictest34-21ksoftirqd/220:05:142
294476899431,13cyclictest34-21ksoftirqd/200:35:302
2944768994310,18cyclictest34-21ksoftirqd/219:45:022
2944761994322,18cyclictest2992526-21pluginstate21:25:020
2944761994322,18cyclictest2992526-21pluginstate21:25:010
294476199431,17cyclictest29628372cut20:00:020
162399430,30rtkit-daemon1753-21gnome-shell23:43:512
162399430,30rtkit-daemon0-21swapper/222:08:582
162399430,30rtkit-daemon0-21swapper/222:08:582
30454302420,16sleep126-21ksoftirqd/123:52:071
2944771994224,14cyclictest1632-21dbus-daemon22:55:013
294476899427,17cyclictest34-21ksoftirqd/220:55:172
294476899425,20cyclictest34-21ksoftirqd/220:14:412
2944768994221,8cyclictest34-21ksoftirqd/219:58:182
2944768994219,8cyclictest34-21ksoftirqd/220:25:012
2944768994212,14cyclictest34-21ksoftirqd/222:15:462
2944766994211,26cyclictest3032060-31apt-check23:15:131
162399420,29rtkit-daemon0-21swapper/220:40:052
162399420,29rtkit-daemon0-21swapper/220:40:052
294477199411,29cyclictest2197014-21PK-Backend20:32:263
294476899419,18cyclictest34-21ksoftirqd/200:05:162
294476899417,18cyclictest34-21ksoftirqd/223:01:242
294476899416,20cyclictest34-21ksoftirqd/221:47:012
2944768994131,7cyclictest34-21ksoftirqd/221:55:192
2944766994131,7cyclictest3046149-21cstates23:55:151
2944766994124,12cyclictest438-20systemd-journal22:38:551
294476899408,18cyclictest34-21ksoftirqd/223:25:002
294476899405,18cyclictest34-21ksoftirqd/223:26:242
2944768994028,9cyclictest34-21ksoftirqd/221:20:412
2944768994028,9cyclictest34-21ksoftirqd/221:20:412
2944768994017,8cyclictest34-21ksoftirqd/220:26:592
2944768994017,10cyclictest34-21ksoftirqd/200:15:162
294476899401,19cyclictest1753-21gnome-shell21:40:202
294476699402,7cyclictest141rcu_preempt00:20:011
2944766994022,13cyclictest438-20systemd-journal21:10:501
2944766994019,17cyclictest2982499-21cpuspeed_turbos20:55:161
2944766994018,14cyclictest26-21ksoftirqd/100:35:021
2944766994018,14cyclictest26-21ksoftirqd/100:35:021
294476699401,11cyclictest3061216-40apt-check00:36:131
294476199402,34cyclictest1753-21gnome-shell23:09:170
294476199402,33cyclictest1753-21gnome-shell00:23:230
2944761994020,15cyclictest855-21dbus-daemon00:00:010
2944761994010,13cyclictest141rcu_preempt20:30:010
162399400,6rtkit-daemon2197014-21PK-Backend23:53:093
30056392390,6sleep10-21swapper/122:00:221
2944768993928,8cyclictest34-21ksoftirqd/223:07:062
294476899392,33cyclictest1753-21gnome-shell21:11:412
2944768993915,8cyclictest34-21ksoftirqd/222:10:542
294476899391,23cyclictest1826-21JS14:31:442
2944768993911,10cyclictest34-21ksoftirqd/221:35:302
294476699392,33cyclictest1753-21gnome-shell21:39:301
294476699392,33cyclictest1753-21gnome-shell21:30:451
294476699392,33cyclictest1753-21gnome-shell00:26:121
2944766993923,12cyclictest2962348-21rs:main1
2944766993922,6cyclictest26-21ksoftirqd/123:35:001
2944766993922,6cyclictest26-21ksoftirqd/123:35:001
2944766993916,18cyclictest438-20systemd-journal19:20:001
294476199392,7cyclictest1753-21gnome-shell20:31:400
2944761993924,11cyclictest2995791-21anacron21:33:060
2944761993924,10cyclictest2997956-21apt21:40:130
294476199392,33cyclictest1753-21gnome-shell20:46:410
162399390,28rtkit-daemon2197014-21PK-Backend21:32:063
162399390,27rtkit-daemon2197014-21PK-Backend21:08:572
29684122380,7chrt2968417-21cpuspeed_turbos20:15:141
294476899388,8cyclictest34-21ksoftirqd/222:35:282
294476899385,10cyclictest34-21ksoftirqd/221:50:142
294476899384,11cyclictest34-21ksoftirqd/200:15:002
294476899382,7cyclictest1753-21gnome-shell19:39:062
294476899382,19cyclictest34-21ksoftirqd/219:17:102
2944766993828,6cyclictest3034920-21users23:20:371
294476699382,7cyclictest1753-21gnome-shell20:28:121
2944766993825,10cyclictest26-21ksoftirqd/100:00:051
2944766993822,12cyclictest3022902-21df_inode22:50:171
2944766993821,13cyclictest3052279-21sshd00:10:551
2944766993821,12cyclictest438-20systemd-journal22:57:281
294476199385,9cyclictest141rcu_preempt22:57:280
2944761993826,8cyclictest1753-21gnome-shell22:33:180
2944761993825,9cyclictest1753-21gnome-shell19:27:110
2944761993824,10cyclictest3000807-21sshd21:45:500
294476199382,32cyclictest1753-21gnome-shell19:33:120
2944761993822,12cyclictest855-21dbus-daemon21:15:010
2944761993822,12cyclictest2950315-40apt-check19:21:140
2944761993820,14cyclictest2997309-21users21:35:320
2944761993820,13cyclictest2983872-21pool-org.gnome.21:00:000
2944761993818,15cyclictest2962348-21rs:main0
2944761993813,7cyclictest141rcu_preempt21:05:220
2944761993810,17cyclictest1753-21gnome-shell19:13:000
2944761993810,17cyclictest1753-21gnome-shell19:13:000
29638452370,14sleep013-21ksoftirqd/020:00:280
294476899375,15cyclictest34-21ksoftirqd/200:00:092
294476899375,13cyclictest34-21ksoftirqd/223:37:202
294476899371,16cyclictest2197014-21PK-Backend20:04:562
294476899371,15cyclictest2197014-21PK-Backend22:01:282
2944766993725,8cyclictest3002271-21ps21:50:321
2944766993725,8cyclictest1753-21gnome-shell19:30:171
2944766993722,11cyclictest2999283-21seq21:43:321
2944766993722,11cyclictest2959413-21cut19:50:021
294476699372,20cyclictest30028682apt-get21:55:061
2944766993721,12cyclictest1738-21gdbus20:11:561
2944766993721,11cyclictest438-20systemd-journal19:20:501
2944766993719,14cyclictest2977736-21memory20:40:221
2944766993719,14cyclictest2977736-21memory20:40:221
294476699371,8cyclictest1753-21gnome-shell20:54:471
2944766993716,17cyclictest3008403-21unin-run22:10:011
2944766993716,17cyclictest3008403-21unin-run22:10:011
294476199374,30cyclictest1069-21runrttasks22:50:560
2944761993726,7cyclictest1753-21gnome-shell21:25:090
2944761993726,7cyclictest1753-21gnome-shell00:14:390
294476199372,31cyclictest1753-21gnome-shell23:27:070
2944761993723,10cyclictest3049182-21systemctl00:05:000
2944761993722,11cyclictest2985233-21sshd21:00:500
2944761993721,12cyclictest918-21gdbus23:31:570
2944761993721,12cyclictest918-21gdbus23:31:560
2944761993721,12cyclictest13-21ksoftirqd/021:50:330
2944761993720,13cyclictest2959977-21fw_packets19:50:170
2944761993720,13cyclictest2959977-21fw_packets19:50:160
2944761993719,14cyclictest3019444-21df_inode22:40:170
2944761993719,14cyclictest3019444-21df_inode22:40:170
294476199371,7cyclictest3059542-21turbostat00:35:010
294476199371,7cyclictest3059542-21turbostat00:35:000
162399370,25rtkit-daemon2197014-21PK-Backend21:37:073
2944771993618,15cyclictest2197014-21PK-Backend23:24:593
294476899361,18cyclictest2946061-21seq19:10:442
294476899361,18cyclictest2946061-21seq19:10:442
294476699365,5cyclictest141rcu_preempt23:13:041
2944766993625,6cyclictest26-21ksoftirqd/119:40:001
2944766993623,10cyclictest3029792-21awk23:10:001
2944766993622,10cyclictest3056065-21cron00:24:591
2944766993622,10cyclictest26-21ksoftirqd/119:25:051
2944766993621,11cyclictest26-21ksoftirqd/123:49:461
2944766993620,12cyclictest3019835-21awk22:40:281
2944766993620,12cyclictest3019835-21awk22:40:271
2944766993619,13cyclictest2979376-21cat20:45:221
2944766993618,15cyclictest1753-21gnome-shell20:30:251
294476199364,6cyclictest141rcu_preempt20:44:560
294476199364,6cyclictest141rcu_preempt20:44:560
294476199363,6cyclictest141rcu_preempt19:50:010
2944761993626,7cyclictest1069-21runrttasks19:37:250
2944761993622,11cyclictest3015369-21awk22:30:000
2944761993622,11cyclictest3015369-21awk22:30:000
294476199362,12cyclictest1753-21gnome-shell00:35:170
2944761993620,12cyclictest3039771-21ntp_states23:35:270
2944761993620,12cyclictest3017494-21cpuspeed_turbos22:35:160
2944761993620,12cyclictest2990385-21awk21:15:300
2944761993620,12cyclictest1-21systemd20:25:020
2944761993619,13cyclictest3027748-21apt-get23:05:010
2944761993618,14cyclictest3043578-21awk23:45:340
294476899356,15cyclictest34-21ksoftirqd/222:43:222
294476899356,15cyclictest34-21ksoftirqd/222:43:212
294476699359,21cyclictest2945011-21cpuspeed19:10:131
294476699359,21cyclictest2945011-21cpuspeed19:10:131
294476699357,24cyclictest855-21dbus-daemon21:20:001
294476699357,23cyclictest26-21ksoftirqd/123:25:231
294476699354,27cyclictest2975565-21smartctl20:35:181
2944766993526,6cyclictest2991184-21tr21:20:161
2944766993526,6cyclictest2991184-21tr21:20:161
2944766993525,7cyclictest1069-21runrttasks19:50:461
2944766993525,7cyclictest1069-21runrttasks19:50:461
2944766993524,7cyclictest2970090-21cpuspeed_turbos20:20:111
2944766993524,7cyclictest1753-21gnome-shell20:08:451
2944766993520,11cyclictest3049686-21expr00:05:151
2944766993520,11cyclictest3014238-21expr22:25:161
2944766993520,11cyclictest3014238-21expr22:25:151
2944766993519,12cyclictest855-21dbus-daemon23:05:011
2944766993517,13cyclictest438-20systemd-journal21:05:031
294476699351,15cyclictest29569942smart_sda19:40:261
294476199358,11cyclictest1069-21runrttasks20:53:280
2944761993520,11cyclictest3053450-21cat00:15:250
2944761993520,11cyclictest3009064-21basename22:10:200
2944761993520,11cyclictest3006182-21cat22:00:330
2944761993514,6cyclictest141rcu_preempt23:44:100
294476199351,30cyclictest3021576-21open_inodes22:45:300
162399350,24rtkit-daemon0-21swapper/321:12:013
294476699343,4cyclictest141rcu_preempt22:15:291
2944766993423,7cyclictest3012887-21ntpq22:20:231
294476199348,6cyclictest141rcu_preempt23:21:040
2944761993419,11cyclictest2968332-21wc20:15:140
2944761993419,11cyclictest2967580-21cat20:10:310
294476199341,6cyclictest141rcu_preempt22:20:240
294476199341,5cyclictest141rcu_preempt23:50:160
294476199341,11cyclictest2966274-21cat20:10:000
294477199331,7cyclictest2197014-21PK-Backend21:53:283
2944766993324,6cyclictest2984298-21tr21:00:161
2944766993322,7cyclictest3009366-21ntp_kernel_pll_22:10:251
2944766993319,10cyclictest3015930-21expr22:30:151
2944766993319,10cyclictest26-21ksoftirqd/120:00:191
2944761993317,12cyclictest2975211-21awk20:35:110
2944766993220,8cyclictest3038995-21/usr/sbin/munin23:35:121
2944766993214,13cyclictest438-20systemd-journal20:00:011
2944761993210,7cyclictest141rcu_preempt22:15:180
294476699306,9cyclictest26-21ksoftirqd/122:47:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional