You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-29 - 22:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jun 29, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
45462238162,25sleep00-21swapper/007:05:440
46652235162,21sleep20-21swapper/207:07:202
46912224193,20sleep30-21swapper/307:07:413
29172203165,25sleep10-21swapper/107:05:101
1871221850,5sleep3497799cyclictest11:20:123
1539021800,5sleep3497799cyclictest08:45:133
4971997876,1cyclictest0-21swapper/208:58:502
4971996260,1cyclictest0-21swapper/211:04:402
173322450,1sleep30-21swapper/310:00:263
496299399,14cyclictest20669-21munin-run11:25:010
4971993532,2cyclictest0-21swapper/209:31:382
496299352,26cyclictest24115-21kworker/0:110:27:330
4977993214,8cyclictest25017-21users07:50:253
497799321,18cyclictest0-21swapper/308:56:523
4971993229,2cyclictest0-21swapper/207:33:162
4971993229,2cyclictest0-21swapper/207:33:162
497799312,11cyclictest0-21swapper/312:18:253
496299301,10cyclictest14477-21kworker/0:112:23:490
496299292,15cyclictest8439-21munin-run08:30:010
497799281,24cyclictest0-21swapper/310:46:413
497799281,23cyclictest26019-21diskmemload09:15:203
496699281,6cyclictest25017-21diskstats10:20:161
496699281,11cyclictest0-21swapper/110:38:351
496299281,9cyclictest17702-21kworker/0:310:10:370
497799275,4cyclictest0-21swapper/309:45:263
497799271,16cyclictest21082-21irqstats10:10:173
496699278,10cyclictest0-21swapper/112:30:191
496699277,12cyclictest0-21swapper/110:27:201
4966992713,6cyclictest32269-21systemd-journal09:20:411
496299278,3cyclictest31023-21kworker/0:210:40:250
496299278,10cyclictest24292-21kworker/0:209:15:130
496299274,3cyclictest6052-21kworker/0:208:44:570
496299274,3cyclictest6052-21kworker/0:208:44:570
496299272,11cyclictest25057-21kworker/0:107:55:170
496299272,10cyclictest14477-21kworker/0:112:30:010
496299271,4cyclictest5642-21kworker/0:112:09:010
496299271,10cyclictest18710-21kworker/0:107:40:250
4962992710,4cyclictest16088-21kworker/0:211:15:170
497199263,8cyclictest12355-40dnf12:18:252
496699263,16cyclictest87550irq/128-enp1s0-11:45:151
496299269,15cyclictest26019-21diskmemload12:40:010
496299261,12cyclictest32361-21kworker/0:208:14:130
496299261,10cyclictest3394-21kworker/0:311:59:170
497799252,20cyclictest0-21swapper/308:00:423
497799251,21cyclictest4264-21/usr/sbin/munin08:20:123
497799251,21cyclictest0-21swapper/310:52:053
497799251,11cyclictest8861-21sshd09:42:413
497799251,11cyclictest31714-21ntp_states09:20:223
497799251,11cyclictest24052-21/usr/sbin/munin09:05:123
497799251,11cyclictest2053-21df11:55:133
497799251,11cyclictest13173-21ntp_states11:05:213
497799251,11cyclictest13173-21ntp_states11:05:213
497799250,7cyclictest4482-21kworker/u8:309:50:213
4966992514,7cyclictest0-21swapper/109:16:441
496699251,18cyclictest0-21swapper/110:49:411
496699251,17cyclictest0-21swapper/112:12:181
496699250,7cyclictest0-21swapper/111:53:301
496299251,2cyclictest16000-21kworker/0:107:17:530
496299251,10cyclictest24836-21kworker/0:111:36:160
497799242,11cyclictest2065-21/usr/sbin/munin08:15:173
497799242,11cyclictest14741-21ntp_states09:55:213
497799242,11cyclictest14741-21ntp_states09:55:213
497799242,11cyclictest10861-21/usr/sbin/munin08:35:273
497799242,10cyclictest32226-21/usr/sbin/munin08:10:153
497799241,20cyclictest26019-21diskmemload11:31:433
497799241,16cyclictest0-21swapper/312:04:233
497799241,14cyclictest0-21swapper/308:44:573
497799241,14cyclictest0-21swapper/308:44:573
497799241,13cyclictest9802-21latency_hist11:00:013
497799241,12cyclictest0-21swapper/307:15:333
497799241,11cyclictest27931-21ntp_states10:25:223
497799241,11cyclictest12722-21/usr/sbin/munin12:20:133
497799241,11cyclictest10250-21/usr/sbin/munin07:20:263
497799240,7cyclictest0-21swapper/310:41:133
497799240,6cyclictest0-21swapper/310:21:393
497199241,20cyclictest0-21swapper/207:10:162
496699245,4cyclictest14579-21sshd12:23:401
496699242,17cyclictest87550irq/128-enp1s0-09:47:371
496699241,16cyclictest6528-21kworker/u8:211:20:451
496699241,12cyclictest0-21swapper/109:41:591
496699240,17cyclictest0-21swapper/108:05:171
496299249,3cyclictest19335-21kworker/0:109:07:330
496299248,3cyclictest30355-21kworker/0:109:20:250
497799239,3cyclictest7787-21sshd12:06:093
497799237,3cyclictest0-21swapper/312:12:213
4977992316,4cyclictest23828-21/usr/sbin/munin09:05:003
497799231,19cyclictest0-21swapper/311:42:343
497799231,13cyclictest0-21swapper/310:05:133
497799231,11cyclictest4882-21/usr/sbin/munin09:35:223
497799231,11cyclictest19039-21/usr/sbin/munin07:40:183
497799231,11cyclictest0-21swapper/312:28:103
497799231,11cyclictest0-21swapper/310:35:583
497799231,11cyclictest0-21swapper/309:13:023
497799231,10cyclictest30032-21/usr/sbin/munin08:05:203
497799231,10cyclictest26019-21diskmemload09:25:143
497799230,6cyclictest0-21swapper/311:38:413
496699238,4cyclictest6622-21cpuspeed_turbos08:25:121
496699231,20cyclictest0-21swapper/107:50:111
4966992311,6cyclictest6528-21kworker/u8:212:35:051
496699231,16cyclictest0-21swapper/111:03:311
496699231,13cyclictest0-21swapper/110:54:291
496699230,6cyclictest0-21swapper/112:29:261
496699230,10cyclictest3677-21kworker/u8:210:08:031
496299231,6cyclictest0-21swapper/009:55:000
496299231,2cyclictest87650irq/129-enp1s0-08:21:410
496299230,2cyclictest87650irq/129-enp1s0-11:40:440
497799228,3cyclictest808-21snmpd11:48:233
497799228,3cyclictest0-21swapper/308:29:403
497799226,6cyclictest0-21swapper/312:39:113
497799222,10cyclictest14644-21df_inode11:10:143
497799221,8cyclictest0-21swapper/310:30:393
497799221,6cyclictest0-21swapper/311:00:183
497799221,4cyclictest0-21swapper/311:54:143
497799221,10cyclictest20890-21/usr/sbin/munin11:25:133
497799221,10cyclictest16841-21/usr/sbin/munin07:35:253
497799221,10cyclictest16499-21/usr/sbin/munin11:15:183
496699221,7cyclictest20624-21uptime07:40:271
496699221,3cyclictest15512-21cat07:30:201
496699221,3cyclictest15512-21cat07:30:201
496699221,18cyclictest9459-21interrupts08:30:161
496699221,16cyclictest0-21swapper/109:59:341
496699221,16cyclictest0-21swapper/109:59:341
496699221,15cyclictest0-21swapper/111:35:131
496699221,15cyclictest0-21swapper/111:13:161
496699221,15cyclictest0-21swapper/109:35:071
496699221,15cyclictest0-21swapper/109:30:181
496699221,14cyclictest0-21swapper/109:51:141
496699221,11cyclictest24202-21cpuspeed_turbos09:05:131
496699220,16cyclictest0-21swapper/110:15:531
496299227,3cyclictest3757-21kworker/0:008:19:010
496299226,3cyclictest1723-21kworker/0:209:27:570
496299226,2cyclictest6052-21kworker/0:208:35:330
496299223,2cyclictest87550irq/128-enp1s0-07:36:570
496299222,11cyclictest6052-21kworker/0:208:46:130
496299221,7cyclictest6052-21kworker/0:208:51:530
496299221,6cyclictest9711-21kworker/0:207:20:530
496299221,6cyclictest0-21swapper/012:17:530
496299221,11cyclictest482-21ntp_states11:50:210
497799219,3cyclictest0-21swapper/309:33:293
497799211,19cyclictest3677-21kworker/u8:210:17:473
497799211,10cyclictest0-21swapper/312:30:203
497199217,3cyclictest25589-21cstates11:35:142
497199216,3cyclictest29976-21cat11:45:142
496699218,2cyclictest29804-21crond08:05:011
496699216,9cyclictest0-21swapper/110:55:291
496699216,4cyclictest26019-21diskmemload10:40:241
496699215,9cyclictest0-21swapper/111:25:031
496699215,8cyclictest18678-21gzip08:50:221
4966992116,2cyclictest0-21swapper/110:30:131
4966992114,2cyclictest6528-21kworker/u8:212:06:171
496699211,3cyclictest0-21swapper/112:15:121
496699211,2cyclictest11556-21wc08:35:171
496699211,2cyclictest11051-21/usr/sbin/munin07:20:191
496699211,15cyclictest0-21swapper/111:57:121
496699211,15cyclictest0-21swapper/111:43:331
496699211,13cyclictest0-21swapper/110:11:081
496699211,11cyclictest20614-21sed08:55:211
496699211,11cyclictest17270-21grep07:35:161
496699211,11cyclictest16841-21users08:45:271
496699211,11cyclictest0-21swapper/111:07:361
496699211,11cyclictest0-21swapper/111:07:361
496699211,10cyclictest10174-21date07:20:011
496699211,10cyclictest0-21swapper/107:55:221
4966992110,3cyclictest2446-21latency_hist09:30:011
496299216,8cyclictest87450irq/127-enp1s0-09:00:050
496299216,4cyclictest28713-21kworker/0:010:31:050
496299212,10cyclictest21620-21munin-run09:00:010
4962992117,3cyclictest0-21swapper/012:30:410
496299211,5cyclictest0-21swapper/010:49:260
4962992111,8cyclictest11608-21sshd11:01:420
496299211,11cyclictest12105-21/usr/sbin/munin11:05:130
496299211,11cyclictest12105-21/usr/sbin/munin11:05:130
497799208,3cyclictest0-21swapper/308:51:523
497799206,3cyclictest12050irq/123-ahci[0007:30:013
4977992010,3cyclictest0-21swapper/308:30:213
497199205,4cyclictest26019-21diskmemload11:26:372
497199204,3cyclictest21740-21sed10:10:262
497199202,6cyclictest15255-21/usr/sbin/munin08:45:242
497199201,7cyclictest8053-21users08:25:262
497199201,7cyclictest6127-21/usr/sbin/munin12:05:212
497199201,4cyclictest1165-21cpuspeed_turbos10:40:132
496699207,2cyclictest5487-21cut08:20:231
4966992017,2cyclictest0-21swapper/108:16:191
496699201,6cyclictest4362-21fschecks_count12:00:141
4966992013,2cyclictest0-21swapper/111:30:021
496699201,17cyclictest0-21swapper/107:12:521
496699201,12cyclictest0-21swapper/111:16:361
496699200,5cyclictest0-21swapper/109:13:161
496299209,2cyclictest4288-21kworker/0:109:35:240
496299208,2cyclictest15655-21awk10:00:010
496299208,2cyclictest15655-21awk10:00:000
496299207,2cyclictest28713-21kworker/0:010:35:290
496299206,3cyclictest87550irq/128-enp1s0-10:55:170
496299205,6cyclictest20887-21kworker/0:207:47:190
496299205,3cyclictest12190-21kworker/0:307:35:010
496299205,3cyclictest12190-21kworker/0:307:35:000
496299201,2cyclictest87550irq/128-enp1s0-07:28:360
496299201,14cyclictest87350irq/126-enp1s0-10:16:420
496299201,14cyclictest87350irq/126-enp1s0-07:50:180
496299201,10cyclictest19145-21ntp_states10:05:230
302812200,4sleep0496299cyclictest08:05:130
497799191,14cyclictest0-21swapper/307:50:013
497199196,3cyclictest87550irq/128-enp1s0-08:54:402
497199194,3cyclictest30702-21sshd10:30:532
497199192,15cyclictest0-21swapper/209:20:312
497199191,6cyclictest25017-21diskstats10:20:162
497199191,6cyclictest14300-21/usr/sbin/munin11:10:262
497199191,6cyclictest0-21swapper/207:51:592
497199191,5cyclictest808-21snmpd10:58:122
497199191,5cyclictest30424-21munin-run09:20:022
497199191,3cyclictest0-21swapper/212:27:102
4971991913,3cyclictest8053-21/usr/sbin/munin07:15:262
497199191,11cyclictest0-21swapper/211:17:402
497199191,11cyclictest0-21swapper/210:52:262
497199191,11cyclictest0-21swapper/210:00:092
496699198,3cyclictest22905-21ntp_states09:00:201
496699194,9cyclictest0-21swapper/108:42:481
496699194,9cyclictest0-21swapper/108:42:481
496699191,6cyclictest17144-21awk10:00:251
496699191,16cyclictest0-21swapper/107:45:351
496699191,12cyclictest0-21swapper/107:27:361
496699190,2cyclictest0-21swapper/108:11:401
496299192,14cyclictest29056-21find08:00:220
496299191,8cyclictest0-21swapper/011:11:250
496299191,7cyclictest2682-21/usr/sbin/munin09:30:190
4962991917,1cyclictest0-21swapper/008:30:230
496299191,6cyclictest9279-21/usr/sbin/munin09:45:110
496299191,6cyclictest827-21runrttasks12:02:010
496299191,6cyclictest0-21swapper/010:53:310
496299191,4cyclictest0-21swapper/009:41:320
4962991911,5cyclictest25329-21sed10:20:180
496299191,12cyclictest0-21swapper/010:03:560
496299191,10cyclictest29682-21/usr/sbin/munin11:45:230
497799181,8cyclictest16689-21latency_hist07:35:013
497799181,8cyclictest16689-21latency_hist07:35:013
497799181,5cyclictest25626-21/usr/sbin/munin07:55:253
497199184,3cyclictest87350irq/126-enp1s0-12:30:152
497199183,4cyclictest529-21taskset10:37:012
497199183,3cyclictest4886-21users10:45:262
497199182,4cyclictest28891-21sa108:00:212
497199182,4cyclictest1336-21latency09:25:182
497199182,4cyclictest10623-21crond08:35:012
497199182,13cyclictest2906-21cut08:15:182
497199181,9cyclictest0-21swapper/210:07:402
497199181,7cyclictest32-21ksoftirqd/211:20:162
497199181,7cyclictest21279-21latency_hist12:40:022
497199181,5cyclictest26250-21/usr/sbin/munin09:10:132
497199181,5cyclictest13255-21irqstats07:25:162
497199181,5cyclictest11318-21ntp_states07:20:232
497199181,4cyclictest9625-21diskstats09:45:132
497199181,4cyclictest13674-21/usr/sbin/munin09:55:242
497199181,4cyclictest13674-21/usr/sbin/munin09:55:242
4971991812,3cyclictest3185-21grep11:55:252
4971991812,3cyclictest22369-21grep09:00:132
497199181,15cyclictest0-21swapper/209:42:242
497199181,15cyclictest0-21swapper/207:58:262
497199181,14cyclictest26019-21diskmemload11:30:312
496299185,3cyclictest87550irq/128-enp1s0-11:30:190
496299181,9cyclictest5268-21grep07:10:120
496299181,3cyclictest16088-21kworker/0:211:25:210
4962991813,2cyclictest26250-21/usr/sbin/munin09:10:150
496299181,10cyclictest0-21swapper/012:14:510
497199173,3cyclictest87350irq/126-enp1s0-12:00:122
497199172,12cyclictest12819-21tr12:20:122
497199171,9cyclictest0-21swapper/212:12:172
497199171,7cyclictest0-21swapper/209:06:282
497199171,6cyclictest11478-21/usr/sbin/munin09:50:142
497199171,5cyclictest22492-21cpu10:15:132
497199171,4cyclictest27480-21/usr/sbin/munin11:40:222
497199171,4cyclictest1788-21kworker/u8:108:08:162
497199171,4cyclictest12105-21/usr/sbin/munin11:05:172
497199171,4cyclictest12105-21/usr/sbin/munin11:05:172
497199171,4cyclictest0-21swapper/210:25:222
497199171,15cyclictest0-21swapper/211:53:352
497199171,14cyclictest0-21swapper/208:10:242
4971991711,3cyclictest21350-21cpuspeed_turbos07:45:102
497799161,6cyclictest5160-21/usr/sbin/munin07:10:163
497199161,14cyclictest0-21swapper/207:38:192
4971991611,3cyclictest827-21runrttasks08:35:072
4971991611,3cyclictest5226-21cat08:20:192
497199161,12cyclictest5540-21sed09:35:152
497199160,15cyclictest0-21swapper/207:44:142
4971991510,3cyclictest13348-21expr08:40:102
4971991510,3cyclictest13348-21expr08:40:102
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional