You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-12 - 08:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #f, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackfslot7s.osadl.org (updated Mon Jan 12, 2026 00:44:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23683802164135,23sleep00-21swapper/019:05:200
23686002146119,22sleep20-21swapper/219:08:122
23685492141109,27sleep30-21swapper/319:07:313
23684062139101,32sleep10-21swapper/119:05:421
2369003993224,5cyclictest759-21mta-sts-daemon19:45:000
2369003993121,7cyclictest2423824-21sh22:00:010
2369008992623,2cyclictest2372396-21sh19:20:001
2369003992617,6cyclictest2404606-21sendmail21:00:000
2369013992317,4cyclictest2377388-21/usr/sbin/munin19:35:193
2369013992215,4cyclictest0-21swapper/323:00:003
2369011992218,3cyclictest2368998-21cyclictest23:10:012
236900899229,6cyclictest2427038-10dpkg22:10:011
2369008992217,3cyclictest2433602-21idleruntime-cro22:30:001
2369003992218,2cyclictest2462336-21idleruntime-cro00:00:010
236901399218,11cyclictest2385359-21awk20:00:013
236901399210,18cyclictest2400666-21ldconfig20:45:183
2369011992117,3cyclictest13548-21gmain21:30:392
2369011992115,4cyclictest0-21swapper/219:35:002
2369008992118,2cyclictest2386932-21dump-pmu-power20:05:001
2369003992117,2cyclictest2381929-21systemd-run19:50:000
2369003992116,3cyclictest2372414-21grep19:20:000
2369013992017,2cyclictest2449694-21sendmail23:20:003
2369013992017,2cyclictest2406174-21dump-pmu-power21:05:003
2369013992015,3cyclictest2469899-21pmu-power00:20:193
2369013992012,6cyclictest661-21rs:main3
2369011992017,2cyclictest2414538-21gdbus21:30:012
2369008992015,3cyclictest2399566-21idleruntime-cro20:45:001
236900899200,17cyclictest2403814-21ntpq20:55:191
2369003992017,2cyclictest355-21systemd-journal22:25:010
2369003992012,6cyclictest2377156-21awk19:34:590
2369013991911,5cyclictest2446485-21dpkg23:10:003
236901199193,15cyclictest661-21rs:main2
236901199193,14cyclictest2462325-21awk00:00:012
2369011991914,2cyclictest0-21swapper/220:10:002
2369011991910,6cyclictest681-21gdbus20:59:592
236900899196,9cyclictest2444673-21basename23:04:591
2369008991915,2cyclictest0-21swapper/100:35:001
2369008991910,7cyclictest750-21gmain21:40:001
2369003991916,2cyclictest2433624-21systemd-run22:30:000
2369003991916,2cyclictest103-21hwrng21:50:080
2369003991912,5cyclictest2448088-21cron23:15:000
2369013991815,2cyclictest2408565-21ntpq21:10:173
2369013991812,3cyclictest0-21swapper/300:20:003
2369013991811,5cyclictest2412335-21chrt21:23:103
236901399180,16cyclictest13236-21gnome-shell22:11:593
236901199189,4cyclictest2442784-10sshd22:56:312
2369011991812,4cyclictest2394200-21/usr/sbin/munin20:25:202
2369011991811,5cyclictest2460397-21sshd23:50:452
2369011991811,1cyclictest2377680-21/usr/sbin/munin19:35:152
236901199180,5cyclictest2399339-21taskset20:42:122
2369008991812,4cyclictest2452019-21needreboot23:25:181
2369008991811,5cyclictest2465445-21seq00:05:391
236900899181,15cyclictest2382381-21appstreamcli19:50:011
2369008991810,6cyclictest2420944-21apt-get21:50:091
2369008991810,6cyclictest2397986-21/usr/sbin/munin20:40:011
236900899180,16cyclictest2378613-21sshd19:38:451
236900399189,7cyclictest2414084-21awk21:29:590
236900399187,4cyclictest2401059-10dpkg20:45:330
236900399181,15cyclictest13236-21gnome-shell20:53:590
236900399181,14cyclictest13709-21gnome-software00:05:390
236900399180,16cyclictest2458858-21taskset23:46:120
236901399174,7cyclictest2381256-21proc_pri19:45:183
2369013991715,1cyclictest659-21in:imuxsock19:34:593
2369013991714,2cyclictest2426158-21needreboot22:05:163
236901399170,4cyclictest2439832-21seq22:48:593
236901399170,16cyclictest2461518-21/usr/sbin/munin23:55:183
236901399170,15cyclictest2444518-21taskset23:03:163
236901199179,5cyclictest2402781-21runrttasks20:52:182
236901199172,14cyclictest2378985-21/usr/sbin/munin19:40:182
2369011991714,2cyclictest421ktimers/223:30:132
2369011991713,3cyclictest13649-21gmain21:46:322
2369011991710,5cyclictest355-21systemd-journal23:20:002
236901199170,16cyclictest2380583-21apt-get19:45:102
236901199170,15cyclictest2457213-21/usr/sbin/munin23:40:212
236901199170,15cyclictest2454667-21/usr/sbin/munin23:35:222
236900899179,5cyclictest2414378-21chmod21:30:011
236900899177,6cyclictest2392944-21sshd20:21:141
236900899177,5cyclictest2463752-10sshd00:02:461
236900899176,6cyclictest2419844-21strings21:45:171
2369008991714,2cyclictest639-21avahi-daemon00:29:591
2369008991713,2cyclictest0-21swapper/122:56:441
2369008991711,4cyclictest795-21runrttasks20:06:171
236900899170,5cyclictest2401119-21runrttasks20:46:141
236900899170,15cyclictest2470480-21sshd00:24:121
236900899170,15cyclictest2467272-21runrttasks00:13:531
236900399178,6cyclictest2466064-21apt-get00:10:110
236900399177,7cyclictest661-21rs:main0
236900399171,3cyclictest2442804-21sshd22:56:440
2369003991712,3cyclictest2426989-21cron22:10:010
2369003991710,5cyclictest2472151-21awk00:30:000
236900399170,4cyclictest794-21snmpd23:17:440
236900399170,4cyclictest2434487-21/usr/sbin/munin22:30:200
236900399170,4cyclictest2399560-21cron20:45:000
236900399170,15cyclictest795-21runrttasks22:54:400
236901399169,5cyclictest2454909-21/usr/sbin/munin23:35:143
236901399169,3cyclictest355-21systemd-journal21:40:003
236901399168,6cyclictest2458439-21ntpq23:45:193
236901399168,5cyclictest2409586-21apt-get21:15:113
2369013991612,3cyclictest13214-21gvfs-afc-volume22:51:113
2369013991612,2cyclictest2464377-21apt-key00:05:003
2369013991612,2cyclictest0-21swapper/321:30:003
2369013991611,4cyclictest2422021-21chrt21:52:113
2369013991610,4cyclictest2434882-21chrt22:31:253
236901399160,4cyclictest2370054-21taskset19:10:363
236901399160,14cyclictest2452736-21taskset23:28:433
236901199168,5cyclictest661-21rs:main2
236901199166,6cyclictest2469463-21hddtemp_smartct00:20:142
236901199163,10cyclictest13471-21gnome-software00:05:372
236901199161,7cyclictest2459185-21idleruntime-cro23:50:002
2369011991612,3cyclictest13065-10wireplumber19:15:302
2369011991611,2cyclictest795-21runrttasks22:50:142
2369011991610,4cyclictest2464147-21sort00:05:002
2369011991610,4cyclictest2399567-21apt-get20:45:032
236901199160,5cyclictest2432509-21chrt22:25:152
236901199160,5cyclictest2383360-21users19:50:222
236901199160,4cyclictest2409804-21runrttasks21:15:142
236901199160,4cyclictest2370402-21cron19:15:012
236901199160,3cyclictest2370652-21munin-node20:15:112
236900899169,1cyclictest2388745-21/usr/sbin/munin20:10:231
236900899168,6cyclictest2429196-21/usr/sbin/munin22:15:131
236900899168,5cyclictest2475333-21sh00:40:001
236900899168,5cyclictest2415965-21systemd21:35:001
236900899166,7cyclictest355-21systemd-journal19:59:591
2369008991614,1cyclictest13246-21gmain19:13:051
2369008991612,2cyclictest2408230-21hddtemp_smartct21:10:121
2369008991611,3cyclictest2372914-21hddtemp_smartct19:20:151
236900899161,12cyclictest2461238-21smartctl23:55:151
236900899161,11cyclictest2423831-21munin-run22:00:001
2369008991610,5cyclictest2449653-21chrt23:19:501
236900899160,5cyclictest2402717-21taskset20:51:291
236900899160,14cyclictest2447996-21taskset23:13:511
236900899160,14cyclictest2438902-21/usr/sbin/munin22:45:171
236900399169,5cyclictest2425233-21taskset22:02:450
236900399168,5cyclictest2389332-21ntpq20:10:190
236900399164,4cyclictest1-21systemd20:35:010
2369003991610,4cyclictest795-21runrttasks20:25:040
236900399160,4cyclictest2473715-21latency_hist00:35:000
236900399160,2cyclictest794-21snmpd23:01:150
236900399160,14cyclictest13236-21gnome-shell00:01:580
236900399160,14cyclictest1-21systemd20:25:000
236901399159,5cyclictest2430355-21chrt22:19:113
236901399159,4cyclictest2402886-21taskset20:53:413
236901399159,4cyclictest2392160-21smartctl20:20:133
236901399158,5cyclictest2404598-21perf-exec20:59:593
236901399157,6cyclictest2414565-21pool-org.gnome.21:30:073
236901399156,6cyclictest2473977-21apt-get00:35:113
236901399156,6cyclictest13573-21gdbus19:50:343
236901399152,10cyclictest13065-10wireplumber19:44:593
2369013991512,2cyclictest355-21systemd-journal20:45:013
2369013991512,1cyclictest2472145-21grep00:30:003
236901399151,12cyclictest645-21dbus-daemon19:30:013
2369013991511,2cyclictest2450682-21/usr/sbin/munin23:20:203
236901399150,2cyclictest2437587-21ntpq22:40:173
236901199159,4cyclictest2396225-21chrt20:32:542
236901199158,5cyclictest2419133-21systemctl21:45:012
236901199158,4cyclictest2451505-21taskset23:25:032
236901199157,5cyclictest2417652-21sort21:40:002
236901199155,7cyclictest2405088-21hddtemp_smartct21:00:132
2369011991512,2cyclictest2421183-21/usr/sbin/munin21:50:122
236901199151,10cyclictest2473728-21idleruntime-cro00:34:592
236901199151,10cyclictest2444680-21idleruntime-cro23:04:592
236901199151,10cyclictest2436765-21awk22:39:592
2369011991510,2cyclictest0-21swapper/223:10:002
236901199150,3cyclictest2385626-21taskset20:00:122
236901199150,14cyclictest2470804-21/usr/sbin/munin00:25:222
236901199150,14cyclictest2373885-21sshd19:23:452
236901199150,13cyclictest2437247-21/usr/sbin/munin22:40:122
236901199150,13cyclictest2411803-21/usr/sbin/munin21:20:202
236900899158,5cyclictest355-21systemd-journal21:05:001
236900899157,6cyclictest0-21swapper/120:28:451
236900899155,7cyclictest2430911-21df22:20:131
236900899151,8cyclictest2440578-21meminfo22:50:161
2369008991511,2cyclictest2410164-21ntpq21:15:191
2369008991511,2cyclictest2379215-21/usr/sbin/munin19:40:111
236900899151,10cyclictest2457603-21munin-run23:45:001
2369008991510,4cyclictest2412483-21sh21:25:001
2369008991510,4cyclictest13109-40gmain22:10:191
2369008991510,1cyclictest2445125-21/usr/sbin/munin23:05:141
236900899150,13cyclictest2453314-21df_inode23:30:141
236900899150,13cyclictest2418017-21runrttasks21:40:141
236900899150,12cyclictest2437016-21/usr/sbin/munin22:40:181
236900399159,4cyclictest2451533-21awk23:25:130
236900399156,5cyclictest2450562-21grep23:20:180
236900399155,6cyclictest645-21dbus-daemon22:45:000
236900399154,7cyclictest2415551-21pool-org.gnome.21:30:380
236900399153,11cyclictest0-21swapper/022:10:370
2369003991511,3cyclictest2419348-21/usr/sbin/munin21:45:180
236900399151,13cyclictest13236-21gnome-shell19:11:580
2369003991511,2cyclictest2390562-21hddtemp_smartct20:15:130
236900399150,4cyclictest2438838-21smartctl22:45:160
236900399150,3cyclictest2454080-21anacron23:30:460
236900399150,2cyclictest13236-21gnome-shell19:58:090
236900399150,14cyclictest2374516-21/usr/sbin/munin19:25:140
236900399150,13cyclictest2455075-21/usr/sbin/munin23:35:150
236900399150,12cyclictest0-21swapper/000:20:110
236901399149,4cyclictest2396121-21chrt20:31:343
236901399149,4cyclictest2390015-21chrt20:14:243
236901399147,5cyclictest2433305-21chrt22:26:243
236901399147,5cyclictest2371634-21awk19:15:323
236901399145,6cyclictest2459659-21hddtemp_smartct23:50:133
236901399145,6cyclictest2423842-21ls22:00:013
236901399144,6cyclictest2424714-21cut22:00:183
236901399141,7cyclictest2473725-21awk00:35:003
2369013991410,2cyclictest2430921-21runrttasks22:20:123
2369013991410,2cyclictest2387396-21df_inode20:05:133
2369013991410,2cyclictest0-21swapper/300:10:153
236901399140,3cyclictest2446748-21/usr/sbin/munin23:10:163
236901399140,1cyclictest0-21swapper/320:15:193
236901399140,13cyclictest2386761-21runrttasks20:03:013
236901399140,12cyclictest0-21swapper/323:41:093
236901199147,5cyclictest661-21rs:main2
236901199146,6cyclictest2406828-21memory21:05:162
236901199146,5cyclictest2384150-21expr19:55:132
236901199145,6cyclictest2374797-21ntpq19:25:172
236901199142,5cyclictest2427122-10apt-config22:10:012
236901199141,8cyclictest645-21dbus-daemon20:40:012
236901199141,2cyclictest0-21swapper/222:30:222
2369011991412,1cyclictest2389161-21/usr/sbin/munin20:10:182
2369011991410,3cyclictest2466039-21/usr/sbin/munin00:10:102
236900899148,4cyclictest0-21swapper/100:15:131
236900899146,5cyclictest2460020-21cat23:50:181
236900899144,5cyclictest2406908-21strings21:05:151
236900899141,9cyclictest2377148-21munin-run19:35:001
236900899141,6cyclictest2390630-21irqstats20:15:141
2369008991411,2cyclictest2424092-21/usr/sbin/munin22:00:211
2369008991411,1cyclictest2457862-21apt-get23:45:121
236900899141,10cyclictest2455082-21meminfo23:35:161
236900899140,2cyclictest2434978-21seq22:32:371
236900899140,13cyclictest2436452-21chrt22:36:191
236900899140,12cyclictest794-21snmpd23:20:441
236900399148,4cyclictest2459634-21sh23:50:130
236900399148,4cyclictest2381898-21apt-get19:50:010
236900399147,5cyclictest2373717-21taskset19:21:350
236900399146,5cyclictest2406184-21basename21:04:590
236900399144,6cyclictest2385852-21smartctl20:00:150
236900399141,9cyclictest2397986-21munin-run20:40:000
2369003991412,1cyclictest2410713-21taskset21:17:500
236900399141,10cyclictest2457609-21awk23:45:010
2369003991410,2cyclictest2411724-21ldconfig21:20:190
236900399140,3cyclictest2377388-21/usr/sbin/munin19:35:130
236900399140,13cyclictest2474194-21/usr/sbin/munin00:35:140
236900399140,13cyclictest2444909-21/usr/sbin/munin23:05:140
236900399140,13cyclictest0-21swapper/020:07:050
236901399135,5cyclictest2419962-21cut21:45:193
236901399134,4cyclictest2465426-10dpkg00:05:323
2369013991311,1cyclictest2394243-21/usr/sbin/munin20:25:213
2369013991311,1cyclictest0-21swapper/320:35:133
236901399130,2cyclictest2374024-21munin-run19:25:003
236901399130,1cyclictest0-21swapper/323:30:123
236901399130,11cyclictest0-21swapper/322:35:153
236901199135,5cyclictest2429649-21awk22:15:182
236901199134,6cyclictest2468203-21gcc00:15:172
2369011991311,1cyclictest2473952-21/usr/sbin/munin00:35:132
2369011991310,2cyclictest2407985-21/usr/sbin/munin21:10:092
2369011991310,2cyclictest2392874-21seq20:20:222
236901199130,12cyclictest2422903-21/usr/sbin/munin21:55:142
236901199130,11cyclictest2430693-21/usr/sbin/munin22:20:192
236900899137,4cyclictest2375503-21seq19:29:191
2369008991311,1cyclictest2381785-21sshd19:48:461
2369008991310,2cyclictest13338-21JS19:58:001
236900399135,6cyclictest2416202-21apt-get21:35:120
2369003991310,2cyclictest2419071-21sshd21:44:460
236900399130,1cyclictest2468778-21taskset00:17:590
236900399130,12cyclictest0-21swapper/022:15:150
236901399128,2cyclictest659-21in:imuxsock21:05:013
236901199123,6cyclictest2424633-21collect222:00:172
236900399128,2cyclictest2408204-21sh21:10:120
236900399124,6cyclictest2407741-21/usr/sbin/munin21:09:590
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional