You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-09 - 09:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackfslot7s.osadl.org (updated Fri Jan 09, 2026 00:44:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9290652152104,28sleep30-21swapper/319:09:243
9288452142107,30sleep20-21swapper/219:06:342
9288172141110,26sleep10-21swapper/119:06:131
9289902140106,29sleep00-21swapper/019:08:240
92936399304,7cyclictest1000065-21smartctl22:45:191
929358993023,4cyclictest0-21swapper/019:20:010
929367992823,3cyclictest645-21dbus-daemon23:20:012
92936399289,5cyclictest1029016-21ntpq00:15:181
92936399284,5cyclictest968391-21perf-exec21:10:011
92936399271,8cyclictest977876-21awk21:40:011
92936399261,9cyclictest932781-21perf19:20:011
929358992621,3cyclictest681-21gdbus23:40:010
929358992618,6cyclictest1023167-21awk00:00:000
92936399258,7cyclictest1002327-21sshd22:54:521
929372992418,3cyclictest0-21swapper/319:20:003
92936399249,3cyclictest331ktimers/123:20:151
929372992320,2cyclictest656-21systemd-logind21:40:023
929367992311,7cyclictest0-21swapper/223:15:012
929372992218,3cyclictest661-21rs:main3
929372992213,6cyclictest1010523-21run-parts23:20:003
929372992211,7cyclictest941009-21basename19:45:003
92936399221,9cyclictest1013947-21apt-get23:30:121
929363992212,8cyclictest1004204-21awk23:00:001
92936399221,10cyclictest954631-21df_inode20:25:131
92936799213,10cyclictest43-21ksoftirqd/221:25:012
929363992110,2cyclictest759-21mta-sts-daemon21:50:181
92935899219,1cyclictest103-21hwrng21:06:540
929358992115,4cyclictest0-21swapper/021:40:000
929372992016,3cyclictest991458-21latency_hist22:20:013
92936799203,6cyclictest43-21ksoftirqd/222:59:592
92936799203,4cyclictest962062-21munin-run20:50:002
92936799201,5cyclictest43-21ksoftirqd/200:25:002
929367992013,2cyclictest191rcu_preempt21:25:172
929363992016,2cyclictest642-21cron00:35:001
929363992010,4cyclictest794-21snmpd20:54:151
929358992017,2cyclictest1030631-21ntp_kernel_pll_00:20:150
929358992012,7cyclictest355-21systemd-journal19:45:000
929358992011,6cyclictest1013689-21/usr/sbin/munin23:30:010
92935899201,10cyclictest221rcuc/023:20:160
92937299199,6cyclictest681-21gdbus20:40:003
92937299191,12cyclictest1011389-21grep23:20:153
92936799199,4cyclictest43-21ksoftirqd/223:40:012
92936799197,1cyclictest191rcu_preempt00:35:192
929367991916,2cyclictest645-21dbus-daemon22:14:592
929367991912,5cyclictest990005-21irqstats22:15:152
929367991912,5cyclictest795-21runrttasks00:28:512
929367991912,5cyclictest1-21systemd19:55:002
929367991912,5cyclictest1000613-21runrttasks22:48:042
929367991912,2cyclictest191rcu_preempt23:45:112
92936799191,10cyclictest1023158-21munin-run00:00:012
929367991910,6cyclictest191rcu_preempt23:25:182
929367991910,6cyclictest1021566-21cron23:55:012
929363991915,3cyclictest945901-21apt-key20:00:011
92936399191,16cyclictest946270-21apt-get20:00:101
929358991915,1cyclictest0-21swapper/022:40:130
92935899191,11cyclictest1010524-21systemctl23:20:010
92937299189,7cyclictest0-21swapper/322:39:593
92937299189,4cyclictest1002463-10gpg-connect-age22:55:013
929372991812,6cyclictest0-21swapper/322:05:163
929372991811,5cyclictest991766-21df22:20:133
92937299180,5cyclictest977787-21seq21:39:143
92936799187,4cyclictest43-21ksoftirqd/220:19:592
929367991814,2cyclictest977873-21sendmail21:40:012
929367991813,1cyclictest191rcu_preempt20:42:522
929367991812,2cyclictest191rcu_preempt20:05:142
929367991810,7cyclictest967594-21ntpq21:05:172
92936799180,7cyclictest945770-21true19:59:592
92936799180,5cyclictest43-21ksoftirqd/220:15:002
92936799180,16cyclictest985192-21/usr/sbin/munin22:00:132
92936799180,16cyclictest938020-21/usr/sbin/munin19:35:162
92936799180,16cyclictest794-21snmpd21:11:102
92936799180,10cyclictest984451-21apt-get22:00:002
92936399188,4cyclictest992551-21seq22:20:561
929363991814,2cyclictest659-21in:imuxsock21:30:001
929363991811,1cyclictest0-21swapper/123:05:171
92936399180,16cyclictest958743-21sendmail-mta20:38:171
929358991812,5cyclictest0-21swapper/019:55:150
929358991812,4cyclictest1015155-21sshd23:33:510
929358991811,5cyclictest984457-21systemctl22:00:000
929358991810,6cyclictest1006536-21ntpq23:05:170
92935899180,16cyclictest990931-21taskset22:18:580
92937299177,6cyclictest962074-21basename20:49:593
929372991713,2cyclictest945744-21perf-exec19:59:593
929372991711,5cyclictest975695-21/usr/sbin/munin21:30:193
929372991711,4cyclictest965062-21taskset20:58:193
929372991711,4cyclictest1026361-21taskset00:06:103
929372991711,4cyclictest1004572-21cstates23:00:113
929372991710,5cyclictest1-21systemd22:35:013
929372991710,5cyclictest1004200-21apt23:00:003
929372991710,4cyclictest952627-21dpkg20:20:003
92937299170,15cyclictest955640-21taskset20:28:543
92937299170,15cyclictest953141-21/usr/sbin/munin20:20:133
92936799172,5cyclictest43-21ksoftirqd/200:05:002
929367991711,4cyclictest932470-21chrt19:16:222
929367991711,4cyclictest1018073-21taskset23:40:372
929367991710,2cyclictest191rcu_preempt00:10:162
92936799170,4cyclictest935943-21sshd19:26:522
92936799170,3cyclictest992890-21perf22:25:002
92936799170,15cyclictest971192-21runrttasks21:15:552
92936799170,15cyclictest966480-21taskset21:01:012
92936799170,11cyclictest0-21swapper/222:39:592
92936399178,6cyclictest1-21systemd22:00:001
929363991713,2cyclictest965386-21latency_hist21:00:001
929363991711,4cyclictest940960-21taskset19:44:421
92936399170,4cyclictest972926-21seq21:22:391
92936399170,16cyclictest1031668-21seq00:25:091
92936399170,15cyclictest1004732-21/usr/sbin/munin23:00:141
92935899177,6cyclictest960504-21basename20:45:000
92935899176,6cyclictest954179-21perf-exec20:25:000
92935899172,1cyclictest759-21mta-sts-daemon23:10:180
929358991714,2cyclictest13649-21gmain21:29:080
92935899171,3cyclictest191rcu_preempt22:35:000
92935899171,3cyclictest13236-21gnome-shell22:01:090
929358991711,5cyclictest13154-21gmain19:40:010
929358991710,5cyclictest936694-21smartctl19:30:130
929358991710,2cyclictest191rcu_preempt20:05:130
92935899170,4cyclictest996722-21/usr/sbin/munin22:35:160
92935899170,4cyclictest795-21runrttasks21:30:090
92935899170,15cyclictest943233-21/usr/sbin/munin19:50:160
92937299169,5cyclictest1031983-21irqstats00:25:153
92937299167,6cyclictest938269-21hddtemp_smartct19:35:133
92937299167,5cyclictest661-21rs:main3
92937299165,5cyclictest1020799-10ld23:50:163
929372991612,2cyclictest13236-21gnome-shell23:05:593
929372991611,4cyclictest936433-21taskset19:30:013
92937299161,12cyclictest960488-21munin-run20:44:593
929372991610,5cyclictest1014883-21chrt23:30:253
929372991610,4cyclictest950568-21taskset20:13:093
929372991610,4cyclictest1034458-21taskset00:33:383
929372991610,4cyclictest1016759-21taskset23:39:143
92937299160,5cyclictest929674-21df_inode19:10:113
92937299160,4cyclictest974434-21taskset21:26:493
92937299160,4cyclictest1028474-21runrttasks00:15:013
92937299160,14cyclictest984391-21chrt21:59:313
92937299160,14cyclictest971242-21runrttasks21:16:353
92937299160,14cyclictest948983-21taskset20:08:003
92936799169,5cyclictest980380-21memory21:45:162
92936799169,5cyclictest1002342-21cron22:55:002
92936799167,6cyclictest1015284-21systemd-run23:35:012
92936799164,7cyclictest43-21ksoftirqd/219:25:012
929367991613,2cyclictest759-21mta-sts-daemon20:55:162
929367991613,2cyclictest355-21systemd-journal22:30:202
929367991612,2cyclictest0-21swapper/220:35:002
92936799160,5cyclictest1034600-21true00:34:592
92936799160,15cyclictest994222-21taskset22:27:102
92936399169,5cyclictest999215-21awk22:45:001
92936399169,4cyclictest1016315-21find23:35:201
92936399168,5cyclictest355-21systemd-journal21:16:441
92936399167,6cyclictest1025365-21apt-get00:05:111
92936399162,13cyclictest794-21snmpd20:13:041
92936399161,8cyclictest938640-21smartctl19:35:181
929363991614,1cyclictest1023330-21savelog00:00:011
929363991613,2cyclictest645-21dbus-daemon23:45:011
92936399161,2cyclictest996301-21apt-get22:35:121
92936399161,13cyclictest0-21swapper/122:25:121
929363991610,4cyclictest984992-21apt-get22:00:121
92936399160,6cyclictest1018886-21df23:45:121
92936399160,4cyclictest962019-21taskset20:49:401
92936399160,4cyclictest942286-21seq19:46:291
92936399160,4cyclictest1024590-21/usr/sbin/munin00:00:201
92936399160,15cyclictest951555-21/usr/sbin/munin20:15:181
92936399160,15cyclictest1009550-21/usr/sbin/munin23:15:151
92935899168,5cyclictest994465-21ls22:30:000
92935899168,5cyclictest355-21systemd-journal00:00:020
92935899168,2cyclictest191rcu_preempt00:15:000
929358991612,2cyclictest0-21swapper/023:45:010
929358991612,2cyclictest0-21swapper/020:54:590
929358991611,4cyclictest795-21runrttasks20:31:190
929358991610,4cyclictest981755-21sh21:50:140
929358991610,4cyclictest979203-21find21:40:180
92935899160,4cyclictest1028710-21/usr/sbin/munin00:15:140
92935899160,15cyclictest1003091-21/usr/sbin/munin22:55:120
92935899160,14cyclictest935412-21ntpq19:25:160
94898150,1rtkit-daemon0-21swapper/100:35:541
92937299158,5cyclictest969145-21ldconfig21:10:153
929372991512,2cyclictest13246-21gmain23:12:533
929372991512,1cyclictest758-21mta-sts-daemon20:00:093
929372991511,2cyclictest982491-21/usr/sbin/munin21:50:213
929372991511,2cyclictest760-21mta-sts-daemon22:25:143
92937299150,2cyclictest1000756-21taskset22:49:533
92936799159,5cyclictest982080-21ntp_kernel_pll_21:50:172
92936799159,4cyclictest940308-21/usr/sbin/munin19:40:182
92936799158,5cyclictest1006440-21meminfo23:05:162
92936799157,5cyclictest355-21systemd-journal20:39:592
92936799156,2cyclictest191rcu_preempt20:20:012
92936799151,3cyclictest43-21ksoftirqd/223:25:002
929367991513,2cyclictest932877-21kworker/2:2-events20:25:152
929367991510,1cyclictest191rcu_preempt22:40:172
92936399159,4cyclictest1013557-21taskset23:28:371
92936399159,4cyclictest1008854-21seq23:14:121
92936399158,4cyclictest1027224-21/usr/sbin/munin00:10:161
92936399151,2cyclictest1020268-21apt-get23:50:121
92936399151,11cyclictest934373-21munin-run19:25:011
92936399150,7cyclictest978698-21irqstats21:40:131
92936399150,2cyclictest13236-21gnome-shell20:41:101
92936399150,11cyclictest965784-21irqstats21:00:141
92935899159,4cyclictest992702-21sshd22:22:520
92935899159,4cyclictest966501-21taskset21:01:190
92935899159,4cyclictest795-21runrttasks23:46:040
92935899156,6cyclictest1002712-21sed22:55:010
92935899156,4cyclictest1034835-10awk00:35:110
92935899154,2cyclictest191rcu_preempt20:14:590
929358991511,2cyclictest970933-21python321:15:190
92935899150,3cyclictest979963-21/usr/sbin/munin21:45:160
92937299148,4cyclictest935839-21taskset19:25:333
92937299146,4cyclictest998287-21/usr/sbin/munin22:40:143
92937299145,6cyclictest1012599-21smartctl23:25:143
92937299145,5cyclictest1035523-21wc00:35:183
92937299141,9cyclictest1018441-21idleruntime-cro23:45:013
929372991412,1cyclictest942372-21chrt19:47:373
929372991411,2cyclictest981110-21chrt21:47:503
929372991411,2cyclictest1031096-21taskset00:20:593
929372991410,3cyclictest956542-21ntp_kernel_pll_20:30:203
92937299140,2cyclictest13236-21gnome-shell00:02:093
92937299140,12cyclictest973017-21taskset21:23:503
92936799149,1cyclictest191rcu_preempt22:05:162
92936799148,4cyclictest946486-21/usr/sbin/munin20:00:122
92936799147,5cyclictest1004867-21meminfo23:00:152
92936799141,9cyclictest1029844-21idleruntime-cro00:20:002
92936799141,3cyclictest976283-21munin-run21:35:002
929367991412,1cyclictest760-21mta-sts-daemon19:30:102
929367991411,2cyclictest1025561-21/usr/sbin/munin00:05:112
92936799140,13cyclictest930086-21/usr/sbin/munin19:10:162
92936399149,3cyclictest988116-21/usr/sbin/munin22:10:141
92936399148,1cyclictest994709-21/usr/sbin/munin22:30:111
92936399146,5cyclictest355-21systemd-journal22:09:511
929363991412,1cyclictest1030688-21/usr/sbin/munin00:20:171
929363991411,2cyclictest952830-21/usr/sbin/munin20:20:181
929363991411,2cyclictest331ktimers/120:05:121
929363991411,1cyclictest942842-21apt-get19:50:131
929363991410,2cyclictest13236-21gnome-shell22:15:591
92936399140,3cyclictest13236-21gnome-shell21:45:581
92935899149,3cyclictest0-21swapper/021:15:010
92935899148,4cyclictest934586-21dpkg19:25:000
92935899147,5cyclictest951385-21meminfo20:15:170
92935899145,5cyclictest964399-21awk20:55:150
929358991412,1cyclictest1004452-21/usr/sbin/munin23:00:150
92935899141,11cyclictest13236-21gnome-shell00:34:590
92935899140,5cyclictest987834-21taskset22:09:400
92935899140,1cyclictest0-21swapper/022:47:380
92937299137,4cyclictest985518-21ld22:00:173
92937299133,2cyclictest66450NetworkManager21:00:173
92937299131,9cyclictest934363-21apt-get19:25:003
92937299130,4cyclictest1027149-21taskset00:10:143
92937299130,12cyclictest0-21swapper/322:10:153
92936799138,1cyclictest191rcu_preempt19:45:122
929367991311,1cyclictest977862-21apt-get21:40:022
929367991311,1cyclictest0-21swapper/220:50:132
92936399134,7cyclictest975171-21diskstats21:30:141
929363991311,1cyclictest929866-21/usr/sbin/munin19:10:141
92936399130,1cyclictest0-21swapper/120:30:221
92936399130,12cyclictest934868-21/usr/sbin/munin19:25:121
92935899139,3cyclictest934424-21kworker/0:0+events19:45:200
92935899135,6cyclictest13214-21gvfs-afc-volume22:10:160
92935899135,5cyclictest958132-21ld20:35:180
92935899134,7cyclictest1020468-21df_inode23:50:130
92935899131,9cyclictest1032376-21wc00:25:200
929358991310,2cyclictest961297-21ntp_kernel_pll_20:45:190
929358991310,2cyclictest946245-21/usr/sbin/munin20:00:140
929358991310,2cyclictest171ktimers/000:05:170
92935899130,12cyclictest0-21swapper/019:10:440
92937299126,4cyclictest968036-21taskset21:05:503
92937299121,8cyclictest963644-21basename20:55:003
92937299120,1cyclictest944161-21idleruntime-cro19:55:013
92936399121,8cyclictest968847-21diskstats21:10:131
92936399121,8cyclictest937774-21munin-run19:35:001
929358991210,1cyclictest954404-21/usr/sbin/munin20:25:110
92935899120,11cyclictest972288-21strings21:20:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional