You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-08 - 20:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Aug 08, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
315652236161,25sleep00-21swapper/007:07:270
316052233163,57sleep30-21swapper/307:07:583
314852202166,24sleep20-21swapper/207:06:242
298052201166,24sleep10-21swapper/107:05:071
2628321430,5sleep23186399cyclictest11:50:012
666921410,4sleep13185799cyclictest11:05:061
1297921390,4sleep33187099cyclictest08:50:193
31863998783,3cyclictest0-21swapper/210:52:542
62672520,1sleep10-21swapper/109:50:131
3185399401,28cyclictest977-21/usr/sbin/munin08:25:160
3185399401,27cyclictest808-21snmpd10:33:210
3185399383,10cyclictest1011450kworker/u8:211:05:160
3185399381,25cyclictest2059-21taskset10:55:010
3185399379,13cyclictest30931-21munin-run08:20:010
3185399379,13cyclictest30931-21munin-run08:20:010
3185399371,6cyclictest3364-21open_files10:55:250
3185399371,22cyclictest30852-21/usr/sbin/munin12:00:190
3185399371,22cyclictest17054-21/usr/sbin/munin10:15:210
3185399362,18cyclictest17024-21/usr/sbin/munin07:45:230
3185399351,4cyclictest23274-21chrt10:28:200
3185399351,20cyclictest827-21runrttasks12:10:240
3186399341,18cyclictest0-21swapper/207:13:392
3185399349,12cyclictest11734-21/usr/sbin/munin12:30:260
3185399344,27cyclictest30021-21awk10:45:010
3185399343,3cyclictest12050irq/123-ahci[0009:00:020
211562340,1sleep10-21swapper/107:55:191
3186399331,13cyclictest0-21swapper/210:45:032
3185399337,22cyclictest23165-21pluginstate11:40:220
3185399337,22cyclictest23165-21pluginstate11:40:220
3185399336,12cyclictest13966-21awk08:55:010
3185399335,3cyclictest87550irq/128-enp1s0-07:42:500
3185399333,3cyclictest87550irq/128-enp1s0-11:10:250
3185399331,5cyclictest12435-21awk10:05:020
3185399331,4cyclictest0-21swapper/012:20:280
3185399331,3cyclictest87350irq/126-enp1s0-09:40:010
3185399331,2cyclictest0-21swapper/011:59:110
3185399329,19cyclictest909-21sssd_nss09:45:010
3185399328,21cyclictest3854-21/usr/sbin/munin09:45:240
3185399326,13cyclictest26765-21/usr/sbin/munin08:10:190
3185399325,3cyclictest25842-21/usr/sbin/munin10:35:200
3185399321,3cyclictest810-21sshd07:36:370
3185399321,3cyclictest609-21/usr/sbin/munin12:05:260
3185399321,28cyclictest17667-21/usr/sbin/munin11:30:240
3185399319,19cyclictest22362-21/usr/sbin/munin08:00:130
3185399316,12cyclictest21836-21munin-run11:40:010
3185399312,6cyclictest20832-21/usr/sbin/munin09:10:190
3185399312,3cyclictest5101-21/usr/sbin/munin12:15:180
3185399311,6cyclictest8597-21diskstats09:55:140
3185399311,3cyclictest23842-21/usr/sbin/munin09:15:180
31853993113,15cyclictest10840-21ntp_offset08:45:210
3185399311,14cyclictest17444-21munin-run11:30:010
31853993110,18cyclictest7861-21expr08:40:120
31853993110,18cyclictest18651-21cpu09:05:130
3187099304,9cyclictest7658-21needreboot11:05:213
3185399309,18cyclictest6981-21acpi07:25:120
3185399309,17cyclictest9772-21grep07:30:170
3185399308,19cyclictest3118-21sort07:15:160
3185399307,20cyclictest4782-21/usr/sbin/munin07:20:190
3185399302,4cyclictest29392-21/usr/sbin/munin09:30:010
3185399302,4cyclictest29392-21/usr/sbin/munin09:30:010
3185399302,3cyclictest14116-21/usr/sbin/munin10:05:260
3185399301,4cyclictest30840-21/usr/sbin/munin09:30:240
3185399301,4cyclictest13990-21/usr/sbin/munin12:35:140
3185399301,3cyclictest25473-21/usr/sbin/munin08:05:190
31853993013,14cyclictest32469-21grep07:10:150
31853993010,17cyclictest27049-21tail11:50:170
31853993010,17cyclictest13545-21cat11:20:140
3187099297,12cyclictest827-21runrttasks12:13:263
3187099296,13cyclictest579-21systemd-journal12:20:213
31870992913,8cyclictest25685-21sed11:45:253
31870992912,3cyclictest5646-21grep11:00:233
3186399291,13cyclictest0-21swapper/209:26:032
3186399291,13cyclictest0-21swapper/209:26:032
3185399297,19cyclictest11196-21expr11:15:140
3185399292,3cyclictest4458-21/usr/sbin/munin08:30:230
3185399291,4cyclictest6963-21memory09:50:190
3185399291,4cyclictest4639-21/usr/sbin/munin11:00:130
3185399291,4cyclictest26686-21/usr/sbin/munin09:20:260
3185399291,4cyclictest18293-21/usr/sbin/munin07:50:120
3185399291,3cyclictest31269-21/usr/sbin/munin10:45:220
3185399291,3cyclictest25478-21/usr/sbin/munin11:45:240
3185399291,3cyclictest20021-21/usr/sbin/munin10:20:170
3185399291,3cyclictest17862-21/usr/sbin/munin09:00:260
3187099281,15cyclictest27417-21/usr/sbin/munin09:25:233
3187099281,15cyclictest27417-21/usr/sbin/munin09:25:233
3187099281,13cyclictest10443-21/usr/sbin/munin10:00:163
3187099280,21cyclictest401ktimersoftd/311:10:193
3186399281,13cyclictest0-21swapper/208:29:232
3185399288,17cyclictest20356-21expr07:55:130
3185399281,4cyclictest9893-21entropy12:25:150
3185399281,4cyclictest15692-21/usr/sbin/munin10:10:190
3185399281,3cyclictest6081-21hwlatdetect08:35:160
3185399281,3cyclictest31322-21/usr/sbin/munin08:20:120
3187099275,14cyclictest0-21swapper/311:25:253
3187099274,5cyclictest0-21swapper/311:30:233
3187099274,14cyclictest24104-21cut11:45:013
3187099274,14cyclictest24104-21cut11:45:003
3187099271,15cyclictest30929-21ls08:20:013
3187099271,15cyclictest30929-21ls08:20:013
3187099271,13cyclictest29619-21/usr/sbin/munin09:30:153
3186399271,13cyclictest0-21swapper/212:29:152
3186399271,12cyclictest20595-21diskmemload11:26:432
3187099268,11cyclictest15184-21df10:10:133
3187099266,13cyclictest1361-21hddtemp_smartct12:05:163
3187099264,9cyclictest21829-21latency_hist11:40:013
3187099264,12cyclictest25617-21munin-run10:35:013
3187099264,11cyclictest11140-21latency_hist07:35:023
3187099263,11cyclictest28642-21grep10:40:163
3187099261,7cyclictest0-21swapper/310:05:133
3187099261,18cyclictest21176-21df_inode09:10:153
3187099261,14cyclictest28895-21cstates11:55:143
3187099260,16cyclictest1417-21perf09:40:003
3186399261,13cyclictest0-21swapper/208:34:552
3186399261,11cyclictest0-21swapper/211:40:192
3186399261,11cyclictest0-21swapper/211:40:192
31863992610,2cyclictest0-21swapper/209:50:132
3186399260,13cyclictest0-21swapper/209:23:432
3187099257,5cyclictest15903-21awk12:40:013
31870992515,6cyclictest1522-40dnf07:13:403
31870992513,7cyclictest16820-40dnf10:15:043
3187099250,17cyclictest0-21swapper/309:23:353
3186399251,14cyclictest7740-21sort12:20:172
31857992512,11cyclictest8615-21kworker/u8:208:16:441
31857992512,11cyclictest8615-21kworker/u8:208:16:441
252462250,4sleep232-21ksoftirqd/210:30:272
3187099245,5cyclictest6716-40dnf12:15:403
3187099243,13cyclictest373-21munin-run12:05:003
3187099242,5cyclictest18493-21cat09:05:013
3187099241,17cyclictest0-21swapper/310:35:203
3187099241,14cyclictest4237-21latency_hist11:00:023
3185799242,4cyclictest21222-21munin-run10:25:011
31857992414,8cyclictest87550irq/128-enp1s0-09:30:251
3185799241,16cyclictest0-21swapper/109:15:351
31857992410,12cyclictest24534-21kworker/u8:007:23:081
3187099236,11cyclictest0-21swapper/310:46:263
3187099233,9cyclictest26458-21/usr/sbin/munin11:50:223
3187099233,7cyclictest17744-21awk07:50:023
3187099233,15cyclictest87350irq/126-enp1s0-09:42:223
3187099231,17cyclictest0-21swapper/312:28:293
3187099231,13cyclictest0-21swapper/308:10:163
3187099230,10cyclictest20590-21crond09:10:013
3186399235,3cyclictest11443-21netstat10:00:192
31863992316,4cyclictest12135-40dnf11:15:222
3186399231,13cyclictest5647-21wc11:00:232
31857992313,7cyclictest87550irq/128-enp1s0-11:40:131
31857992313,7cyclictest87550irq/128-enp1s0-11:40:131
3185799231,15cyclictest0-21swapper/112:31:221
3185799230,18cyclictest0-21swapper/109:37:431
3187099226,13cyclictest14841-21grep11:20:273
3187099225,6cyclictest19599-21entropy10:20:153
3187099224,8cyclictest0-21swapper/309:15:213
3187099221,13cyclictest746-21cat08:25:013
3187099221,12cyclictest21446-21/usr/sbin/munin10:25:163
3187099220,6cyclictest12081-21diskstats12:30:153
3186399228,2cyclictest0-21swapper/207:32:592
3186399227,6cyclictest707-21cpuspeed_turbos12:05:132
3186399223,14cyclictest22759-40dnf09:14:392
31863992217,3cyclictest2938-21tail10:55:182
3185799226,7cyclictest14180-21/usr/sbin/munin08:55:251
3185799222,17cyclictest26179-21df10:35:161
31857992216,3cyclictest30047-21timerandwakeup11:55:271
3185799221,16cyclictest0-21swapper/108:34:281
3185799221,15cyclictest0-21swapper/111:25:251
3185799221,15cyclictest0-21swapper/110:50:041
3185799221,15cyclictest0-21swapper/108:50:061
3185799221,15cyclictest0-21swapper/108:12:431
3185799220,4cyclictest0-21swapper/110:10:251
3185799220,16cyclictest0-21swapper/109:05:211
3185799220,14cyclictest0-21swapper/109:29:361
3185799220,14cyclictest0-21swapper/109:29:361
3187099216,6cyclictest0-21swapper/310:50:053
3187099215,3cyclictest16164-21latency_hist09:00:023
3187099215,2cyclictest87650irq/129-enp1s0-07:15:183
3187099212,11cyclictest13566-21/usr/sbin/munin07:40:163
3187099210,11cyclictest87550irq/128-enp1s0-11:15:233
3186399215,3cyclictest0-21swapper/207:21:572
3186399214,4cyclictest9053-21cstates11:10:122
3186399212,8cyclictest87350irq/126-enp1s0-09:35:152
31863992114,4cyclictest30930-21dump-pmu-power08:20:012
31863992114,4cyclictest30930-21dump-pmu-power08:20:012
3186399211,16cyclictest0-21swapper/209:00:072
3186399210,13cyclictest0-21swapper/207:30:012
3185799216,12cyclictest1283-21systemd-journal08:25:131
3185799211,9cyclictest0-21swapper/109:13:501
3185799211,18cyclictest0-21swapper/107:45:151
3185799211,17cyclictest8505-21memory08:40:191
3185799211,15cyclictest0-21swapper/112:26:301
3185799211,14cyclictest0-21swapper/110:16:131
3185799211,14cyclictest0-21swapper/109:03:241
3185799211,12cyclictest3802-21awk10:55:281
3185799211,11cyclictest6979-21users08:35:261
3185799211,11cyclictest32228-21ntp_states08:20:211
3185799211,11cyclictest10844-21munin-run11:15:021
3185799210,15cyclictest0-21swapper/111:18:141
107462210,1sleep10-21swapper/108:45:201
3187099205,2cyclictest87350irq/126-enp1s0-09:55:223
3187099203,5cyclictest0-21swapper/307:20:163
3187099202,13cyclictest0-21swapper/309:54:013
31870992013,4cyclictest3854-21/usr/sbin/munin09:45:223
31870992011,6cyclictest1327-21diskstats08:25:153
31870992010,4cyclictest87350irq/126-enp1s0-08:37:063
3186399202,4cyclictest763-21dbus-daemon08:14:232
3186399202,15cyclictest9648-21/usr/sbin/munin09:55:252
3186399201,9cyclictest0-21swapper/210:23:242
31863992014,4cyclictest14833-21sed08:55:172
3186399201,3cyclictest0-21swapper/211:21:012
3186399201,16cyclictest0-21swapper/212:00:152
3185799203,11cyclictest87550irq/128-enp1s0-10:06:461
3185799201,9cyclictest0-21swapper/112:39:071
31857992017,2cyclictest0-21swapper/111:49:161
31857992016,3cyclictest0-21swapper/109:44:231
3185799201,2cyclictest0-21swapper/110:48:391
31857992012,3cyclictest0-21swapper/109:57:161
3185799201,17cyclictest0-21swapper/107:52:291
3185799201,15cyclictest0-21swapper/110:27:261
3185799201,14cyclictest0-21swapper/112:17:291
3185799201,14cyclictest0-21swapper/112:12:281
3185799201,14cyclictest0-21swapper/112:07:191
31857992011,3cyclictest0-21swapper/112:20:231
3185799201,11cyclictest3398-21irqstats07:15:191
3185799201,11cyclictest1259-21uptime07:10:261
3185799201,10cyclictest23617-21awk08:00:241
3185799201,10cyclictest13338-21munin-run07:40:011
3185799201,10cyclictest0-21swapper/107:25:301
3185799200,3cyclictest0-21swapper/109:23:391
3185799200,15cyclictest0-21swapper/111:01:381
3185799200,14cyclictest0-21swapper/111:35:141
31870991911,5cyclictest22708-21diskstats08:00:143
3187099191,11cyclictest3198-21/usr/sbin/munin08:30:203
3187099191,11cyclictest0-21swapper/308:40:413
31870991910,2cyclictest12050irq/123-ahci[0007:50:233
3186399198,3cyclictest14744-21irqstats12:35:202
3186399195,8cyclictest16820-40dnf10:15:032
3186399195,3cyclictest89150irq/134-enp2s0-10:40:012
3186399193,2cyclictest87350irq/126-enp1s0-08:37:092
3186399193,2cyclictest0-21swapper/207:45:132
31863991913,3cyclictest10430-21grep08:45:162
3186399191,15cyclictest0-21swapper/211:57:592
3186399191,15cyclictest0-21swapper/207:57:512
3186399191,14cyclictest0-21swapper/212:15:032
3186399191,14cyclictest0-21swapper/210:10:432
3186399191,14cyclictest0-21swapper/209:06:032
3186399191,14cyclictest0-21swapper/208:20:552
3186399191,14cyclictest0-21swapper/207:17:512
3185799192,11cyclictest87550irq/128-enp1s0-12:00:201
3185799191,16cyclictest0-21swapper/111:51:011
3185799191,16cyclictest0-21swapper/110:42:531
31857991911,3cyclictest0-21swapper/110:31:561
3185799190,14cyclictest0-21swapper/111:30:361
3187099189,6cyclictest11735-21/usr/sbin/munin07:35:153
3187099184,11cyclictest10316-21fschecks_time08:45:163
3187099181,3cyclictest8043-21ntp_states07:25:213
31870991811,2cyclictest401ktimersoftd/308:05:183
31870991810,5cyclictest21422-21smart_sda07:55:233
3186399182,7cyclictest0-21swapper/210:40:232
3186399181,6cyclictest0-21swapper/210:05:252
3186399181,4cyclictest26801-21df_inode11:50:152
3186399181,4cyclictest13911-21df_inode07:40:152
31863991813,3cyclictest20201-21df_inode11:35:142
3186399181,13cyclictest0-21swapper/212:14:392
3186399181,10cyclictest0-21swapper/211:06:002
3186399181,10cyclictest0-21swapper/209:31:292
3185799188,3cyclictest0-21swapper/111:20:341
3185799184,6cyclictest3854-21/usr/sbin/munin09:45:181
31857991814,3cyclictest0-21swapper/108:07:501
31857991812,2cyclictest0-21swapper/110:02:551
3185799181,10cyclictest0-21swapper/107:40:261
3186399173,4cyclictest0-21swapper/209:15:112
3186399173,3cyclictest87350irq/126-enp1s0-09:45:212
3186399173,2cyclictest87550irq/128-enp1s0-11:31:252
3186399172,3cyclictest909-21sssd_nss12:30:232
3186399171,3cyclictest0-21swapper/208:40:232
3186399171,3cyclictest0-21swapper/208:08:552
31863991711,3cyclictest827-21runrttasks10:25:112
3186399171,13cyclictest0-21swapper/207:50:032
3186399171,11cyclictest3247-21users09:40:272
3185799171,2cyclictest0-21swapper/107:31:261
3186399165,2cyclictest87450irq/127-enp1s0-08:00:252
3186399161,5cyclictest13047-21ntp_states08:50:212
3186399161,12cyclictest0-21swapper/207:35:112
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional