You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 12:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Sat Jul 12, 2025 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4112699995140,7cyclictest4155647-21grep21:10:250
4112710994936,9cyclictest391ktimers/320:55:133
4112699994941,5cyclictest23068-21latency_hist00:10:000
4112703994635,7cyclictest4124747-21grep19:40:291
4112699994637,5cyclictest299-21systemd-logind23:57:450
4112699994637,5cyclictest299-21systemd-logind23:57:450
4112699994433,7cyclictest4158003-21apt-key21:20:010
4112710994333,6cyclictest4117934-21unixbench_singl19:20:243
4112710994313,9cyclictest20598-21awk00:00:253
4112706994328,11cyclictest311ktimers/222:00:152
4112706994328,11cyclictest311ktimers/222:00:152
4112710994233,5cyclictest4125182-21munin-plugin-st19:45:013
4112706994233,6cyclictest5585-21sh23:20:002
4112706994032,6cyclictest4138673-21kworker/u8:2+events_unbound20:37:152
4112703994029,7cyclictest4116658-21sendmail19:20:011
4112710993910,10cyclictest4148538-21expr20:50:243
4112710993910,10cyclictest4148538-21expr20:50:243
4112706993930,5cyclictest0-21swapper/222:45:012
4112703993932,4cyclictest4178908-21sendmail22:20:001
4112703993932,4cyclictest4178908-21sendmail22:20:001
4112703993928,6cyclictest0-21swapper/120:22:121
4112710993831,4cyclictest4132211-21cron20:05:013
4112706993831,4cyclictest4139042-21munin-plugin-st20:25:012
4112703993828,6cyclictest0-21swapper/122:25:241
4112699993831,4cyclictest4163327-21idleruntime-cro21:35:010
4112710993730,3cyclictest4189361-21apt-get22:50:153
4112699993731,3cyclictest0-21swapper/023:10:000
4112699993728,6cyclictest10692-21cron23:35:010
4112710993626,7cyclictest0-21swapper/319:15:013
4112706993630,4cyclictest33508-21cron00:40:002
4112706993630,4cyclictest33508-21cron00:40:002
4112706993625,8cyclictest311ktimers/219:25:142
4112703993625,7cyclictest24361-21grep00:10:261
4112703993625,7cyclictest0-21swapper/120:35:261
4112699993629,4cyclictest4131740-21unixbench_multi20:00:270
4112699993628,4cyclictest0-21swapper/020:05:180
4112710993527,4cyclictest0-21swapper/321:00:183
4112710993524,8cyclictest4154605-21apt-get21:10:133
4112710993523,9cyclictest391ktimers/321:40:123
4112710993522,10cyclictest9233-21apt-get23:30:123
4112706993528,4cyclictest4114494-21fschecks_count19:15:012
4112706993527,4cyclictest0-21swapper/219:20:242
4112703993528,4cyclictest4157783-21cron21:20:011
4112703993524,7cyclictest4150526-21grep20:55:251
4112699993524,8cyclictest27758-21grep00:20:250
4112710993426,5cyclictest4144127-21cron20:40:003
4112710993421,9cyclictest4145391-21grep20:40:273
4112710993421,10cyclictest4160493-21/usr/sbin/munin21:25:203
4112710993421,10cyclictest4138687-21cron20:21:013
4112706993429,3cyclictest4116488-21kworker/u8:0+events_unbound19:32:042
4112706993427,5cyclictest4162951-21kworker/u8:1+events_unbound21:32:232
4112706993427,5cyclictest4126756-21kworker/u8:2+events_unbound19:47:092
4112706993427,5cyclictest4126756-21kworker/u8:2+events_unbound19:47:092
4112706993426,4cyclictest0-21swapper/221:45:252
4112703993427,4cyclictest0-21swapper/123:40:011
4112703993424,7cyclictest5588-21munin-run23:20:001
4112703993424,6cyclictest0-21swapper/123:10:291
4112703993424,6cyclictest0-21swapper/123:10:291
4112703993424,6cyclictest0-21swapper/119:50:271
4112703993424,6cyclictest0-21swapper/119:35:011
411271099338,8cyclictest0-21swapper/300:06:513
4112710993326,4cyclictest5579-21cron23:20:003
4112710993323,7cyclictest4116364-21sshd19:16:593
4112710993318,11cyclictest4177538-21/usr/sbin/munin22:15:193
4112710993318,11cyclictest4177538-21/usr/sbin/munin22:15:193
4112706993326,4cyclictest0-21swapper/200:25:262
4112706993325,5cyclictest4171852-21munin-run22:00:002
4112706993325,4cyclictest0-21swapper/222:35:272
4112703993325,4cyclictest0-21swapper/123:25:241
4112699993326,4cyclictest0-21swapper/022:20:260
4112699993326,4cyclictest0-21swapper/019:15:270
4112699993324,5cyclictest0-21swapper/000:34:000
4112699993322,6cyclictest0-21swapper/023:05:000
411271099325,9cyclictest4134227-21awk20:10:143
4112710993224,4cyclictest0-21swapper/319:55:203
4112710993222,7cyclictest4139260-21/usr/sbin/munin20:25:123
4112710993220,9cyclictest4169910-21sshd21:52:213
4112710993220,9cyclictest12090-21sshd23:36:223
4112710993219,9cyclictest4163998-21/usr/sbin/munin21:35:213
4112710993219,10cyclictest4176084-21/usr/sbin/munin22:10:193
4112706993224,4cyclictest0-21swapper/223:55:262
4112706993224,4cyclictest0-21swapper/223:55:262
4112706993224,4cyclictest0-21swapper/200:20:152
4112703993225,4cyclictest4189269-21latency_hist22:50:011
4112703993225,4cyclictest33510-21sh00:40:001
4112703993225,4cyclictest33510-21sh00:40:001
4112703993225,4cyclictest0-21swapper/122:40:281
4112703993224,4cyclictest0-21swapper/122:20:161
4112703993224,4cyclictest0-21swapper/100:30:271
4112703993221,6cyclictest0-21swapper/120:15:011
4112703993220,7cyclictest0-21swapper/121:45:251
4112699993222,6cyclictest0-21swapper/019:20:240
4112710993123,4cyclictest0-21swapper/321:20:013
4112710993121,7cyclictest4191722-21/usr/sbin/munin22:55:203
4112710993121,7cyclictest4185533-21cron22:38:013
4112710993121,7cyclictest391ktimers/300:15:133
4112710993121,7cyclictest2750-21/usr/sbin/munin23:10:213
4112710993121,7cyclictest2750-21/usr/sbin/munin23:10:213
4112710993120,7cyclictest0-21swapper/320:10:013
4112710993118,10cyclictest4147376-21cron20:48:003
4112710993118,10cyclictest28763-21/usr/sbin/munin00:25:193
4112710993115,12cyclictest4141413-21/usr/sbin/munin20:30:203
4112706993124,5cyclictest17419-21kworker/u8:1+events_unbound00:12:322
4112706993124,3cyclictest0-21swapper/220:55:252
4112706993123,4cyclictest0-21swapper/223:40:242
4112706993121,6cyclictest4189118-21cron22:50:002
4112703993125,4cyclictest7068-21kworker/u8:0+events_unbound23:22:411
4112703993125,3cyclictest4149259-21munin-plugin-st20:55:001
4112703993125,3cyclictest4149259-21munin-plugin-st20:55:001
4112703993121,6cyclictest0-21swapper/120:00:271
4112703993121,6cyclictest0-21swapper/100:25:001
4112703993118,9cyclictest4152660-21cron21:05:001
4112699993124,4cyclictest4185733-21idleruntime-cro22:40:010
4112699993124,4cyclictest0-21swapper/023:45:240
4112699993123,4cyclictest0-21swapper/021:40:260
4112699993116,12cyclictest4114521-21tune2fs19:15:010
411271099307,13cyclictest0-21swapper/321:46:413
4112710993023,4cyclictest26720-21apt-get00:20:113
4112710993020,7cyclictest4186408-21/usr/sbin/munin22:40:223
4112710993020,7cyclictest4158581-21/usr/sbin/munin21:20:223
4112710993020,6cyclictest4172098-21/usr/sbin/munin22:00:143
4112710993020,6cyclictest4172098-21/usr/sbin/munin22:00:143
4112710993020,6cyclictest14695-21/usr/sbin/munin23:45:173
4112710993019,8cyclictest4189067-21sshd22:49:223
4112710993019,8cyclictest4182164-21sshd22:28:233
4112710993019,8cyclictest32640-21/usr/sbin/munin00:35:233
4112710993019,8cyclictest32640-21/usr/sbin/munin00:35:233
4112710993018,9cyclictest4163006-21sshd21:31:213
4112710993018,8cyclictest31498-21sshd00:34:003
4112710993017,10cyclictest23394-21/usr/sbin/munin00:10:153
4112710993013,13cyclictest16584-21/usr/sbin/munin23:50:223
4112706993020,6cyclictest12402-21sh23:40:012
4112703993022,5cyclictest0-21swapper/119:15:011
4112703993022,4cyclictest0-21swapper/120:05:271
4112703993021,4cyclictest0-21swapper/100:15:171
4112703993020,6cyclictest0-21swapper/121:50:291
4112703993020,6cyclictest0-21swapper/100:25:261
4112699993024,3cyclictest0-21swapper/023:35:300
4112699993023,4cyclictest0-21swapper/022:05:000
4112699993023,4cyclictest0-21swapper/022:05:000
4112699993010,17cyclictest121ktimers/021:20:220
4112710992923,4cyclictest4127316-21apt-get19:50:143
4112710992919,7cyclictest4179582-21meminfo22:20:193
4112710992919,7cyclictest4123139-21sshd19:36:223
4112710992918,8cyclictest4194246-21sshd23:02:363
4112710992918,8cyclictest4127024-21sshd19:49:223
4112710992918,8cyclictest4127024-21sshd19:49:223
4112710992918,8cyclictest4118926-21/usr/sbin/munin19:25:193
4112710992918,8cyclictest1832-21cron23:06:003
4112710992918,8cyclictest13498-21/usr/sbin/munin23:40:233
4112710992916,9cyclictest4175384-21date22:10:003
4112710992916,9cyclictest4175384-21date22:10:003
4112706992922,5cyclictest311ktimers/223:20:132
4112706992921,6cyclictest4175373-21apt-config22:10:002
4112706992921,6cyclictest4175373-21apt-config22:10:002
4112706992918,8cyclictest311ktimers/223:05:122
4112703992922,3cyclictest0-21swapper/100:05:241
4112703992921,4cyclictest0-21swapper/121:35:251
4112703992920,6cyclictest2167-21cron23:10:001
4112703992920,5cyclictest0-21swapper/123:05:001
4112703992919,6cyclictest0-21swapper/123:40:151
4112703992918,7cyclictest4142004-21grep20:30:251
4112699992922,4cyclictest4189130-21kthreadd22:50:000
4112699992922,4cyclictest4135615-21idleruntime-cro20:15:010
4112699992922,4cyclictest13650irq/42-30be0000.ethernet23:10:280
4112699992922,4cyclictest13650irq/42-30be0000.ethernet23:10:280
4112699992922,3cyclictest0-21swapper/023:20:260
411269999291,26cyclictest0-21swapper/020:20:150
4112710992823,2cyclictest5833-21apt-get23:20:133
4112710992819,6cyclictest4183260-21/usr/sbin/munin22:30:243
4112710992818,7cyclictest4170936-21/usr/sbin/munin21:55:253
4112710992818,7cyclictest4153172-21/usr/sbin/munin21:05:173
4112710992818,7cyclictest4137250-21sshd20:19:213
4112706992821,5cyclictest4132016-21kworker/u8:1+events_unbound20:27:132
4112706992821,3cyclictest0-21swapper/220:15:162
4112706992819,5cyclictest0-21swapper/220:40:272
411270699281,4cyclictest131rcu_preempt19:55:162
4112703992822,3cyclictest0-21swapper/120:50:001
4112703992819,6cyclictest19432-21cron00:00:001
4112703992819,6cyclictest19432-21cron00:00:001
4112703992819,5cyclictest0-21swapper/122:35:261
4112703992819,5cyclictest0-21swapper/121:05:251
4112703992818,7cyclictest0-21swapper/121:25:011
4112699992821,5cyclictest4142212-21kworker/u8:0+events_unbound20:47:160
4112699992818,6cyclictest0-21swapper/020:30:010
4112710992715,8cyclictest4120280-21munin-node19:30:123
4112706992721,4cyclictest4154391-21kworker/u8:2+events_unbound21:12:202
4112706992720,5cyclictest4170018-21kworker/u8:2+events_unbound22:28:142
4112706992720,5cyclictest4162951-21kworker/u8:1+events_unbound21:35:112
4112706992720,4cyclictest0-21swapper/221:20:302
4112706992719,6cyclictest4147540-21kworker/u8:2+events_unbound20:51:012
4112706992719,6cyclictest4147540-21kworker/u8:2+events_unbound20:51:012
4112706992717,6cyclictest0-21swapper/200:00:272
4112703992719,4cyclictest0-21swapper/121:30:251
411269999272,4cyclictest20951-21sed00:00:270
4112699992723,2cyclictest13650irq/42-30be0000.ethernet21:25:130
4112699992720,4cyclictest0-21swapper/022:30:250
4112699992719,5cyclictest0-21swapper/019:30:250
4112699992718,7cyclictest4181947-21kworker/u8:1+events_unbound22:55:060
411269999271,6cyclictest4129594-21ntp_states19:55:250
4112706992620,4cyclictest8782-21kworker/u8:2+events_unbound23:27:432
4112706992620,3cyclictest0-21swapper/223:00:232
4112706992619,5cyclictest4164688-21kworker/u8:0+events_unbound21:41:472
4112706992619,4cyclictest4135180-21head20:10:262
4112706992619,4cyclictest0-21swapper/220:00:272
4112706992617,6cyclictest4127453-21cut19:50:172
4112706992616,7cyclictest4158087-21sed21:20:022
4112703992620,3cyclictest0-21swapper/119:35:261
4112703992619,5cyclictest4181947-21kworker/u8:1+events_unbound22:51:411
4112703992618,5cyclictest20950-21unixbench_singl00:00:271
4112703992617,6cyclictest0-21swapper/122:10:001
4112703992617,6cyclictest0-21swapper/122:10:001
4112699992620,3cyclictest0-21swapper/019:40:010
4112699992618,5cyclictest0-21swapper/021:35:270
4112699992617,6cyclictest0-21swapper/000:20:000
4112699992616,7cyclictest4191051-21sort22:55:010
4112699992615,8cyclictest4178897-21ls22:20:000
4112699992615,8cyclictest4178897-21ls22:20:000
411270699254,7cyclictest4175216-21kworker/u8:1+events_unbound22:15:002
4112706992519,4cyclictest4114147-21kworker/u8:1+events_unbound19:16:592
4112706992519,3cyclictest0-21swapper/220:35:012
4112706992518,4cyclictest0-21swapper/223:10:212
4112706992518,4cyclictest0-21swapper/223:10:212
4112706992511,5cyclictest7068-21kworker/u8:0+events_unbound23:30:542
4112703992518,4cyclictest17519-21apt-get23:55:001
4112703992517,5cyclictest0-21swapper/121:40:261
4112703992517,4cyclictest0-21swapper/119:25:261
4112703992516,5cyclictest0-21swapper/119:45:291
4112703992516,5cyclictest0-21swapper/119:45:291
4112703992514,8cyclictest0-21swapper/122:00:001
4112699992520,2cyclictest0-21swapper/023:55:010
4112699992519,4cyclictest0-21swapper/022:40:400
4112699992519,4cyclictest0-21swapper/019:25:270
4112699992517,5cyclictest0-21swapper/023:40:240
4112699992516,6cyclictest4125180-21basename19:45:010
4112699992511,10cyclictest215-21dbus-daemon23:17:390
4112706992417,5cyclictest4142212-21kworker/u8:0+events_unbound20:48:232
4112706992415,6cyclictest311ktimers/221:25:062
4112706992415,4cyclictest23062-21sh00:10:002
4112703992414,7cyclictest4140723-21cron20:30:011
4112703992414,6cyclictest0-21swapper/120:45:001
4112703992413,7cyclictest10253-21grep23:30:251
4112699992418,3cyclictest0-21swapper/022:05:130
4112699992418,3cyclictest0-21swapper/022:05:130
4112699992417,3cyclictest0-21swapper/020:32:400
411269999241,19cyclictest0-21swapper/021:07:100
411269999240,20cyclictest0-21swapper/020:59:370
4112706992317,4cyclictest4181947-21kworker/u8:1+events_unbound22:32:342
4112706992317,3cyclictest0-21swapper/222:50:162
4112706992316,5cyclictest4170018-21kworker/u8:2+events_unbound22:18:102
4112706992316,5cyclictest4170018-21kworker/u8:2+events_unbound22:18:102
4112706992316,5cyclictest4124803-21kworker/u8:0+events_unbound19:42:082
4112706992316,3cyclictest0-21swapper/223:50:222
4112706992313,7cyclictest0-21swapper/222:55:212
411270399238,7cyclictest4137268-21seq20:19:341
4112703992316,3cyclictest0-21swapper/122:00:281
4112703992316,3cyclictest0-21swapper/122:00:281
4112703992314,6cyclictest0-21swapper/123:45:191
4112703992313,6cyclictest0-21swapper/119:55:221
4112699992316,3cyclictest0-21swapper/022:28:420
4112699992316,3cyclictest0-21swapper/021:45:230
4112699992316,3cyclictest0-21swapper/019:53:350
411269999231,5cyclictest28937-21missed_timers00:25:210
4112699992315,5cyclictest0-21swapper/000:10:260
4112699992315,4cyclictest0-21swapper/020:35:260
4112699992314,6cyclictest4172644-1kworker/0:1H+mmc_complete22:15:010
4112699992314,5cyclictest0-21swapper/020:42:500
411269999231,18cyclictest0-21swapper/021:59:460
411269999231,18cyclictest0-21swapper/019:49:380
411269999231,18cyclictest0-21swapper/019:49:380
411269999230,19cyclictest0-21swapper/020:16:420
4112706992217,3cyclictest0-21swapper/200:15:172
4112706992215,4cyclictest311ktimers/200:30:202
411270699221,4cyclictest0-21swapper/223:45:172
4112706992214,4cyclictest4122195-21fschecks_time19:35:172
4112706992214,4cyclictest0-21swapper/221:50:292
4112703992215,3cyclictest0-21swapper/121:10:231
4112703992214,5cyclictest0-21swapper/121:25:271
411270399221,11cyclictest0-21swapper/122:32:511
4112699992215,4cyclictest4168998-21mii-tool21:50:210
4112699992215,3cyclictest0-21swapper/023:28:120
411269999220,18cyclictest0-21swapper/021:01:540
4112706992113,4cyclictest4180138-21unixbench_multi22:20:252
4112706992113,4cyclictest0-21swapper/220:05:272
4112703992115,3cyclictest0-21swapper/119:20:221
4112703992111,7cyclictest4175690-21gdbus22:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional