You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 06:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Thu Nov 27, 2025 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4050051995428,22cyclictest4107521-21cat22:00:013
4050051995226,22cyclictest391ktimers/323:50:153
4050036995132,15cyclictest0-21swapper/022:20:010
4050046995043,4cyclictest4089620-21munin-plugin-st21:10:012
4050046994923,22cyclictest0-21swapper/220:55:232
4050036994925,20cyclictest0-21swapper/019:10:260
4050051994817,6cyclictest4071711-21fschecks_time20:10:183
4050039994835,10cyclictest231ktimers/123:35:121
4050036994839,6cyclictest4105333-21grep21:50:250
4050036994839,6cyclictest4105333-21grep21:50:250
4050051994719,7cyclictest4130140-21grep23:00:253
4050039994732,11cyclictest231ktimers/100:10:111
4050051994522,7cyclictest391ktimers/319:25:113
4050051994417,4cyclictest0-21swapper/322:05:253
4050046994427,12cyclictest0-21swapper/219:25:252
4050036994415,4cyclictest4158089-21latency_hist00:25:000
4050036994415,4cyclictest4158089-21latency_hist00:24:590
4050051994332,7cyclictest4150826-21grep00:00:253
4050051994331,8cyclictest4114979-21grep22:20:163
405004699432,4cyclictest4051956-21cut19:15:012
405004699431,9cyclictest4068421-21/usr/sbin/munin20:00:222
405003699438,6cyclictest0-21swapper/023:00:250
4050051994234,4cyclictest0-21swapper/322:20:013
405004699423,8cyclictest4120772-21/usr/sbin/munin22:35:192
4050046994230,7cyclictest0-21swapper/221:30:242
405004699421,9cyclictest4099327-21df_inode21:35:142
4050039994233,6cyclictest4053568-21unixbench_singl19:15:241
4050036994230,9cyclictest4112626-21cron22:15:010
4050036994218,20cyclictest0-21swapper/022:09:590
4050036994217,3cyclictest13950irq/36-mmc122:50:010
4050036994214,7cyclictest4073904-21which20:15:260
4050036994213,3cyclictest0-21swapper/019:10:000
405004699412,6cyclictest4070363-21ntp_states20:05:212
405004699411,8cyclictest4151138-21cron00:04:002
4050036994117,20cyclictest0-21swapper/022:55:150
4050036994114,3cyclictest0-21swapper/000:30:170
4050051994033,4cyclictest4123529-21idleruntime-cro22:45:013
405004699402,5cyclictest4146796-21memory23:50:232
405004699402,4cyclictest4106637-21/usr/sbin/munin21:55:222
405004699401,9cyclictest4158869-21ntp_states00:25:222
405004699401,9cyclictest4083358-21irqstats20:50:172
405004699401,8cyclictest4061833-21/usr/sbin/munin19:40:232
405004699401,7cyclictest4123115-21/usr/sbin/munin22:40:232
405004699401,6cyclictest4094290-21/usr/sbin/munin21:20:202
405004699401,6cyclictest4092094-21/usr/sbin/munin21:15:162
4050046994012,24cyclictest0-21swapper/222:25:272
405004699401,10cyclictest4157167-21ntp_states00:20:212
405004699401,10cyclictest4157167-21ntp_states00:20:212
405004699401,10cyclictest4134991-21open_inodes23:15:222
405003699401,9cyclictest4142896-21sed23:40:020
4050036994016,3cyclictest0-21swapper/023:30:250
4050036994015,4cyclictest0-21swapper/022:40:000
4050036994013,3cyclictest0-21swapper/023:55:250
4050036994011,5cyclictest0-21swapper/023:45:230
405004699393,3cyclictest4138387-21ntp_states23:25:242
4050046993928,8cyclictest311ktimers/222:00:152
405004699392,6cyclictest4143958-21tail23:40:232
405004699392,4cyclictest4151979-21/usr/sbin/munin00:05:192
405004699392,34cyclictest4080062-21memory20:40:162
405004699391,9cyclictest4125662-21apt-get22:50:142
405004699391,8cyclictest53-21kcompactd019:17:182
405004699391,8cyclictest4155390-21/usr/sbin/munin00:15:212
405004699391,8cyclictest4144946-21/usr/sbin/munin23:45:172
405004699391,8cyclictest4081658-21irqstats20:45:182
405004699391,7cyclictest4103090-21meminfo21:45:192
405004699391,6cyclictest4075442-21/usr/sbin/munin20:20:192
405004699391,5cyclictest4111166-21apt-get22:10:122
405004699391,5cyclictest4087427-21/usr/sbin/munin21:00:252
405004699391,4cyclictest4136115-21apt-get23:20:122
405004699391,3cyclictest4090755-21/usr/sbin/munin21:10:232
405004699391,11cyclictest4113313-21memory22:15:182
4050046993910,25cyclictest0-21swapper/219:30:142
405004699390,8cyclictest4139516-21/usr/sbin/munin23:30:112
4050036993929,6cyclictest0-21swapper/020:05:000
4050036993917,3cyclictest0-21swapper/022:25:150
4050036993916,3cyclictest0-21swapper/023:55:000
4050036993910,5cyclictest4092812-21grep21:15:240
405004699383,3cyclictest4073655-21memory20:15:222
405004699382,5cyclictest4142296-21users23:35:242
405004699382,3cyclictest4118626-21apt-get22:30:162
405004699381,9cyclictest4054693-21meminfo19:20:222
405004699381,5cyclictest4124203-21memory22:45:202
405004699381,5cyclictest4114883-21cpuspeed22:20:132
405004699381,4cyclictest4148602-21ntp_states23:55:212
405004699381,4cyclictest4127809-21/usr/sbin/munin22:55:192
405004699381,4cyclictest4109600-21/usr/sbin/munin22:05:162
405004699381,4cyclictest4101457-21/usr/sbin/munin21:40:202
405004699381,4cyclictest4066272-21df19:55:172
4050046993811,24cyclictest0-21swapper/219:35:172
4050039993828,6cyclictest0-21swapper/121:36:471
4050039993827,8cyclictest231ktimers/123:10:121
4050039993827,8cyclictest231ktimers/123:10:121
4050051993730,3cyclictest0-21swapper/322:40:003
405004699372,4cyclictest4153600-21/usr/sbin/munin00:10:182
405004699371,8cyclictest4104349-21egrep21:50:122
405004699371,8cyclictest4104349-21egrep21:50:122
405004699371,8cyclictest4095766-21/usr/sbin/munin21:25:172
405004699371,6cyclictest4161714-21/usr/sbin/munin00:35:132
405004699371,6cyclictest4078098-21/usr/sbin/munin20:30:192
405004699371,4cyclictest4131332-21irqstats23:05:162
405004699371,4cyclictest4063212-21memory19:45:182
4050046993714,19cyclictest0-21swapper/223:10:212
4050046993714,19cyclictest0-21swapper/223:10:212
405004699371,3cyclictest4071609-21/usr/sbin/munin20:10:152
405004699371,11cyclictest4065038-21ntp_states19:50:222
405004699370,4cyclictest4159997-21apt-get00:30:132
4050039993728,4cyclictest0-21swapper/121:16:451
405003699374,12cyclictest13650irq/42-30be0000.ethernet23:10:110
405003699374,12cyclictest13650irq/42-30be0000.ethernet23:10:110
4050051993629,4cyclictest4079376-21munin-plugin-st20:40:013
4050051993629,4cyclictest0-21swapper/323:15:153
4050051993628,4cyclictest0-21swapper/300:29:073
405004699362,3cyclictest4129163-21/usr/sbin/munin23:00:142
405004699361,11cyclictest4079373-21sh20:40:002
4050036993624,7cyclictest4138833-21grep23:25:270
4050036993615,4cyclictest121ktimers/000:28:040
4050051993528,4cyclictest0-21swapper/323:30:253
4050051993525,7cyclictest4092807-21grep21:15:243
405004699351,8cyclictest630-21snmpd20:25:272
4050039993512,12cyclictest4081458-21grep20:45:151
4050036993528,3cyclictest0-21swapper/023:24:590
4050036993513,3cyclictest13650irq/42-30be0000.ethernet19:25:140
4050051993424,6cyclictest4055255-21grep19:20:283
4050039993427,3cyclictest0-21swapper/122:55:001
4050039993426,5cyclictest40900632sleep121:10:141
4050051993326,3cyclictest0-21swapper/320:45:233
4050051993325,4cyclictest0-21swapper/322:04:593
405003999338,6cyclictest24-21ksoftirqd/122:00:001
4050039993326,4cyclictest4114333-21cron22:20:001
4050039993323,6cyclictest231ktimers/121:40:121
4050039993323,6cyclictest0-21swapper/121:30:241
4050036993325,4cyclictest0-21swapper/022:40:240
4050036993313,16cyclictest0-21swapper/022:00:000
4050051993225,4cyclictest0-21swapper/320:05:003
4050051993224,4cyclictest0-21swapper/323:55:153
4050051993224,4cyclictest0-21swapper/323:00:003
405003999329,13cyclictest24-21ksoftirqd/121:25:211
4050039993224,4cyclictest0-21swapper/122:06:541
4050036993224,5cyclictest0-21swapper/019:45:010
4050036993224,4cyclictest0-21swapper/020:45:250
4050051993122,6cyclictest4157655-21unixbench_singl00:20:263
4050051993122,6cyclictest4157655-21unixbench_singl00:20:253
4050051993121,6cyclictest0-21swapper/320:55:003
405003999317,6cyclictest24-21ksoftirqd/121:50:181
405003999317,6cyclictest24-21ksoftirqd/121:50:171
4050039993123,4cyclictest0-21swapper/121:00:161
4050039993123,4cyclictest0-21swapper/100:25:171
4050039993122,6cyclictest4085784-21unixbench_singl20:55:251
4050036993113,15cyclictest0-21swapper/021:10:010
4050051993023,4cyclictest0-21swapper/319:55:273
4050051993020,5cyclictest0-21swapper/319:50:263
4050051993018,7cyclictest0-21swapper/321:45:003
405003999309,5cyclictest24-21ksoftirqd/123:00:221
4050039993021,6cyclictest4117785-21unixbench_singl22:25:261
4050039993021,5cyclictest0-21swapper/123:05:241
4050039993011,12cyclictest131rcu_preempt22:45:141
4050036993022,4cyclictest0-21swapper/022:50:270
4050051992923,3cyclictest0-21swapper/300:15:243
4050051992922,3cyclictest0-21swapper/321:30:023
4050051992921,5cyclictest4112628-21cron22:15:003
4050051992921,4cyclictest0-21swapper/321:45:253
4050051992921,4cyclictest0-21swapper/319:20:013
4050051992920,5cyclictest0-21swapper/319:35:263
4050051992919,6cyclictest4143940-21grep23:40:243
405003999299,13cyclictest4050034-21cyclictest19:50:171
405003999297,15cyclictest24-21ksoftirqd/119:46:021
4050039992922,3cyclictest0-21swapper/122:55:151
4050039992921,5cyclictest0-21swapper/120:40:001
4050039992920,5cyclictest0-21swapper/123:25:191
405005199287,3cyclictest40983402unixbench_singl21:30:253
4050051992820,4cyclictest0-21swapper/300:05:163
4050051992818,6cyclictest4117715-21grep22:25:263
4050051992817,7cyclictest4138841-21grep23:25:273
4050051992817,7cyclictest4078323-21grep20:30:203
405003999287,5cyclictest0-21swapper/119:30:221
4050039992821,4cyclictest0-21swapper/119:55:271
4050039992821,3cyclictest0-21swapper/119:40:011
4050039992819,6cyclictest4116264-21cron22:25:001
4050036992822,4cyclictest0-21swapper/019:45:120
4050036992820,4cyclictest0-21swapper/023:40:240
4050036992820,4cyclictest0-21swapper/023:20:000
4050036992820,4cyclictest0-21swapper/021:15:000
4050036992820,4cyclictest0-21swapper/019:35:260
4050036992819,6cyclictest0-21swapper/021:35:020
4050036992818,6cyclictest4055247-21grep19:20:270
4050036992812,13cyclictest0-21swapper/023:05:230
4050051992720,4cyclictest0-21swapper/320:05:253
4050051992714,9cyclictest4120092-21apt22:35:003
4050039992721,3cyclictest0-21swapper/122:40:011
4050039992720,4cyclictest4152922-21cron00:10:001
4050039992720,4cyclictest0-21swapper/120:05:261
4050039992716,6cyclictest0-21swapper/121:10:001
4050039992710,8cyclictest131rcu_preempt22:04:281
4050039992710,4cyclictest24-21ksoftirqd/123:55:151
4050039992710,10cyclictest131rcu_preempt00:35:151
4050036992719,4cyclictest0-21swapper/000:00:260
4050036992719,3cyclictest0-21swapper/020:55:150
4050036992718,5cyclictest0-21swapper/020:50:240
4050036992710,13cyclictest215-21dbus-daemon22:21:570
4050051992620,3cyclictest0-21swapper/323:20:153
4050051992619,4cyclictest0-21swapper/321:35:253
4050051992618,4cyclictest0-21swapper/323:35:163
4050051992618,3cyclictest0-21swapper/319:10:263
405003999269,5cyclictest24-21ksoftirqd/123:20:131
405003999264,5cyclictest24-21ksoftirqd/120:30:011
4050039992616,3cyclictest131rcu_preempt00:04:041
4050039992611,4cyclictest131rcu_preempt22:30:151
4050036992616,6cyclictest4071267-21latency_hist20:10:010
4050036992616,6cyclictest0-21swapper/021:43:520
4050036992616,6cyclictest0-21swapper/019:55:270
4050051992518,3cyclictest0-21swapper/321:10:023
4050051992518,3cyclictest0-21swapper/321:00:263
405003999259,9cyclictest131rcu_preempt23:16:401
405003999258,4cyclictest24-21ksoftirqd/120:00:241
405003999256,5cyclictest0-21swapper/120:50:251
4050039992514,3cyclictest131rcu_preempt21:23:081
4050039992514,2cyclictest131rcu_preempt23:48:161
4050039992513,3cyclictest131rcu_preempt20:21:111
4050039992511,7cyclictest131rcu_preempt00:15:571
4050036992519,3cyclictest0-21swapper/000:35:260
4050036992517,4cyclictest0-21swapper/020:15:020
4050036992514,8cyclictest4156453-21date00:20:000
405003699251,21cyclictest637-21runrttasks19:30:150
405003699251,20cyclictest0-21swapper/020:32:060
405005199243,11cyclictest299-21systemd-logind20:26:373
4050051992418,3cyclictest0-21swapper/321:20:143
4050051992417,4cyclictest4062049-21grep19:40:253
4050051992416,4cyclictest0-21swapper/323:45:243
4050051992416,4cyclictest0-21swapper/319:30:243
4050051992413,9cyclictest4163185-21cron00:40:003
405005199240,5cyclictest4090744-21/usr/sbin/munin21:10:223
405003999248,8cyclictest131rcu_preempt21:46:231
405003999248,6cyclictest24-21ksoftirqd/100:20:131
405003999248,6cyclictest24-21ksoftirqd/100:20:131
405003999248,4cyclictest131rcu_preempt19:13:221
405003999246,5cyclictest24-21ksoftirqd/122:40:521
405003999246,4cyclictest24-21ksoftirqd/120:45:001
405003999243,6cyclictest24-21ksoftirqd/123:40:111
4050039992413,7cyclictest1-21systemd00:31:471
4050039992412,8cyclictest0-21swapper/120:15:271
4050036992418,3cyclictest0-21swapper/000:10:150
4050036992417,5cyclictest0-21swapper/019:55:000
4050036992415,6cyclictest121ktimers/021:45:220
4050036992413,8cyclictest215-21dbus-daemon20:21:360
4050051992318,3cyclictest4160907-21kworker/u8:2+events_unbound00:31:493
4050051992317,3cyclictest0-21swapper/321:50:213
4050051992317,3cyclictest0-21swapper/321:50:203
4050051992315,5cyclictest4074704-21sendmail-msp20:20:013
4050051992314,6cyclictest0-21swapper/322:50:013
405003999233,11cyclictest24-21ksoftirqd/120:30:211
405003999232,13cyclictest0-21swapper/122:10:481
405003999231,7cyclictest41465192cut23:50:181
4050039992317,3cyclictest0-21swapper/119:20:141
4050039992316,4cyclictest0-21swapper/119:25:151
4050039992312,8cyclictest4072075-21ntp_states20:10:231
4050036992317,3cyclictest13950irq/36-mmc100:10:010
4050036992317,3cyclictest0-21swapper/022:30:280
4050036992317,3cyclictest0-21swapper/021:20:200
4050036992313,7cyclictest4080434-21cat20:40:200
405003699231,18cyclictest0-21swapper/020:39:020
405005199222,14cyclictest4127129-21debian-sa122:55:013
4050051992214,5cyclictest4134325-21seq23:15:013
4050051992214,5cyclictest4134325-21seq23:15:013
4050051992213,4cyclictest4076383-21cron20:25:013
405005199221,15cyclictest4086193-21apt21:00:013
405003999227,8cyclictest131rcu_preempt19:40:171
4050036992215,3cyclictest0-21swapper/021:03:490
405003699220,18cyclictest0-21swapper/021:25:530
405003699220,18cyclictest0-21swapper/020:26:490
405005199214,11cyclictest4080024-21sort20:40:153
4050051992115,3cyclictest0-21swapper/323:05:203
405003699218,10cyclictest121ktimers/022:00:170
4050051992014,4cyclictest4062951-21fschecks_time19:45:163
4050051992010,7cyclictest4153382-21cut00:10:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional