You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-20 - 07:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Apr 20, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29812237164,22sleep20-21swapper/219:06:272
32412232162,58sleep00-21swapper/019:09:500
30452203167,24sleep30-21swapper/319:07:163
29162203166,24sleep10-21swapper/119:05:361
2043221440,4sleep0333799cyclictest22:12:380
2043221440,4sleep0333799cyclictest22:12:380
3349991090,107cyclictest0-21swapper/222:22:572
3349998885,2cyclictest0-21swapper/222:31:502
123432560,1sleep30-21swapper/320:40:193
3349995449,3cyclictest0-21swapper/223:10:262
236362510,3sleep121-21rcuc/122:20:161
247322500,2sleep20-21swapper/219:55:192
3349994238,3cyclictest0-21swapper/219:12:462
334999410,27cyclictest22386-21/usr/sbin/munin22:15:262
334999402,27cyclictest20246-21users22:10:272
334999402,27cyclictest20246-21users22:10:272
3349993936,2cyclictest0-21swapper/220:28:242
334999391,28cyclictest8716-21idleruntime-cro00:15:012
334999381,33cyclictest6080-21grep00:05:272
334999381,26cyclictest15840-21chrt20:50:122
334999381,23cyclictest7180-21/usr/sbin/munin19:15:172
3337993811,12cyclictest0-21swapper/022:55:020
334999375,21cyclictest87650irq/129-enp1s0-20:55:152
334999371,27cyclictest0-21swapper/200:25:422
334999371,25cyclictest21810-21df23:30:132
334999371,25cyclictest17819-21cpuspeed_turbos00:35:132
334999371,25cyclictest17079-21/usr/sbin/munin23:20:132
334999371,25cyclictest17079-21/usr/sbin/munin23:20:122
334999371,25cyclictest13752-21taskset21:56:382
333799377,14cyclictest20010-21munin-run21:00:010
334999363,21cyclictest25871-21/usr/sbin/munin23:40:172
334999361,24cyclictest32462-21/usr/sbin/munin23:55:212
334999361,23cyclictest7920-21taskset22:58:252
334999361,22cyclictest3981-21sort22:50:122
334999361,21cyclictest15675-21cpuspeed_turbos00:30:142
334999361,21cyclictest13029-21/usr/sbin/munin19:30:272
334999361,20cyclictest0-21swapper/200:15:272
335499351,1cyclictest0-21swapper/323:27:323
334999353,23cyclictest31695-21timerwakeupswit23:50:252
334999351,24cyclictest27274-21sed22:30:022
334999351,24cyclictest21817-21/usr/sbin/munin19:50:232
334999351,24cyclictest21817-21/usr/sbin/munin19:50:232
334999351,19cyclictest24850-21expr21:10:142
333799358,13cyclictest4627-21munin-run20:25:020
333799357,13cyclictest4311-21awk00:05:010
3337993511,11cyclictest28813-21awk21:20:010
334999341,22cyclictest24485-21irqstats23:35:192
334999341,22cyclictest19619-21/usr/sbin/munin19:45:222
334999341,22cyclictest1682-21/usr/sbin/munin22:45:112
3354993319,3cyclictest0-21swapper/321:01:523
3354993319,3cyclictest0-21swapper/321:01:523
334999336,17cyclictest87350irq/126-enp1s0-23:45:172
334999331,22cyclictest11441-21cstates00:20:142
334999331,21cyclictest9279-21cpu20:35:122
334999331,19cyclictest4621-21latency_hist20:25:012
334999331,18cyclictest29715-21hwlatdetect21:20:182
334999331,15cyclictest30908-21smartctl22:35:232
333799332,24cyclictest8457-21kworker/0:100:16:060
334999322,15cyclictest0-21swapper/222:01:032
334999321,28cyclictest1436-21ls22:45:012
334999321,21cyclictest7663-21/usr/sbin/munin21:45:162
334999321,20cyclictest5453-21runrttasks21:40:042
334999321,18cyclictest21031-21taskset23:27:232
333799322,19cyclictest11585-21kworker/0:121:59:060
334999315,13cyclictest16541-21kworker/u8:023:16:132
334999314,16cyclictest7767-21kworker/u8:223:01:232
334999312,20cyclictest7919-1kworker/u9:021:19:192
334999311,27cyclictest12613-21/usr/sbin/munin20:40:222
334999311,18cyclictest87350irq/126-enp1s0-21:35:142
3349993111,17cyclictest17992-21unixbench_singl22:05:252
3349993110,18cyclictest11927-21open_inodes19:25:232
334299313,15cyclictest3718-40dnf20:20:251
333799312,10cyclictest30679-21kworker/0:021:25:020
333799311,17cyclictest24844-21kworker/0:322:41:550
334999303,19cyclictest87650irq/129-enp1s0-21:51:462
334999302,24cyclictest26555-21diskstats20:00:152
334999302,18cyclictest87650irq/129-enp1s0-20:10:122
334999301,26cyclictest1872-21/usr/sbin/munin21:30:192
334999301,25cyclictest32270-21/usr/sbin/munin21:25:222
334999301,12cyclictest87650irq/129-enp1s0-20:45:182
334999301,12cyclictest87650irq/129-enp1s0-19:35:152
333799302,11cyclictest23397-21kworker/0:123:45:220
334999292,24cyclictest909-21sssd_nss00:00:182
334999292,24cyclictest28764-21diskstats20:05:142
334999291,25cyclictest8144-21/usr/sbin/munin20:30:222
334999291,25cyclictest22785-21diskstats21:05:152
334999291,25cyclictest20232-21/usr/sbin/munin21:00:212
334999291,25cyclictest20232-21/usr/sbin/munin21:00:212
334999291,25cyclictest1790-21/usr/sbin/munin20:15:252
3349992912,14cyclictest11510-21netstat23:05:192
333799299,17cyclictest13409-21idleruntime-cro20:45:010
333799292,11cyclictest12601-21kworker/0:119:32:060
333799291,13cyclictest7441-21munin-run21:45:010
333799291,11cyclictest1750-21kworker/0:023:58:500
335499289,10cyclictest0-21swapper/322:09:253
334999281,24cyclictest909-21sssd_nss19:40:162
334999281,23cyclictest8635-21/usr/sbin/munin19:20:142
333799289,16cyclictest16852-21munin-run23:20:010
333799282,10cyclictest11585-21kworker/0:122:01:150
333799281,15cyclictest21262-21basename23:30:010
333799281,10cyclictest31575-21kworker/0:122:48:350
3337992810,15cyclictest21603-21awk19:50:000
97852270,2sleep30-21swapper/321:50:013
3342992715,4cyclictest4329-21ntp_states21:35:221
334299271,17cyclictest0-21swapper/121:19:041
333799272,10cyclictest22605-21kworker/0:022:26:070
333799272,10cyclictest22118-21kworker/0:121:09:390
333799272,10cyclictest12925-21kworker/0:100:26:390
333799271,11cyclictest31575-21kworker/0:122:58:590
335499263,12cyclictest0-21swapper/322:20:243
333799262,9cyclictest10689-21kworker/0:000:23:070
333799261,10cyclictest9284-21kworker/0:021:46:220
333799260,11cyclictest23397-21kworker/0:123:39:510
335499250,14cyclictest24817-21cstates21:10:133
334299252,19cyclictest1437-21munin-run22:45:011
333799250,19cyclictest14863-21kworker/0:019:39:390
333799250,10cyclictest31575-21kworker/0:123:01:190
56312240,4sleep1334299cyclictest00:05:231
335499245,9cyclictest0-21swapper/323:35:013
3354992411,3cyclictest0-21swapper/320:55:163
335499240,11cyclictest23671-21/usr/sbin/munin23:35:123
334299243,5cyclictest18795-40dnf23:21:571
334299243,5cyclictest18795-40dnf23:21:571
334299242,3cyclictest21594-21munin-run19:50:001
334299242,3cyclictest15613-21munin-run20:50:011
3342992410,3cyclictest23886-21cat21:05:261
333799247,2cyclictest30679-21kworker/0:021:21:500
333799244,8cyclictest12206-21kworker/0:223:10:170
333799242,8cyclictest12206-21kworker/0:223:22:390
333799242,8cyclictest12206-21kworker/0:223:22:390
333799241,9cyclictest29892-21kworker/0:200:11:060
335499238,7cyclictest27795-21diskstats22:30:153
334299231,3cyclictest22070-21ps22:15:221
334299231,19cyclictest29155-21tr21:20:141
334299231,16cyclictest0-21swapper/120:39:301
334299231,16cyclictest0-21swapper/100:20:051
334299231,12cyclictest31803-21cut21:25:171
334299231,10cyclictest23-21ksoftirqd/121:45:021
333799238,2cyclictest30679-21kworker/0:021:31:260
333799238,2cyclictest17529-21kworker/0:221:00:230
333799238,2cyclictest17529-21kworker/0:221:00:230
333799236,9cyclictest13373-21kworker/0:120:45:470
333799236,15cyclictest31622-21munin-run22:40:020
333799232,9cyclictest9284-21kworker/0:021:52:140
333799232,9cyclictest24844-21kworker/0:322:31:590
333799232,15cyclictest3539-21kworker/0:119:20:300
333799231,11cyclictest5105-21kworker/0:221:39:460
333799230,11cyclictest17435-21kworker/0:000:34:470
335499228,3cyclictest0-21swapper/300:24:493
335499227,6cyclictest12050irq/123-ahci[0022:30:023
335499226,9cyclictest0-21swapper/319:21:553
335499226,8cyclictest31566-21df_inode21:25:153
335499226,7cyclictest12050irq/123-ahci[0023:20:003
335499223,10cyclictest401ktimersoftd/323:00:183
335499221,4cyclictest0-21swapper/322:45:293
3354992210,6cyclictest909-21sssd_nss19:34:553
335499220,11cyclictest22237-21sort19:50:163
335499220,11cyclictest22237-21sort19:50:163
334299226,3cyclictest13125-21cpuspeed_turbos19:30:121
334299222,11cyclictest22215-21munin-run21:05:011
334299222,11cyclictest22215-21munin-run21:05:011
334299221,3cyclictest8011-21diskstats21:45:141
334299221,16cyclictest0-21swapper/123:59:141
334299221,16cyclictest0-21swapper/119:18:201
334299221,16cyclictest0-21swapper/100:03:411
334299221,12cyclictest29991-21df_inode22:35:131
334299220,17cyclictest0-21swapper/120:53:041
334299220,16cyclictest0-21swapper/123:17:361
334299220,16cyclictest0-21swapper/121:31:031
334299220,16cyclictest0-21swapper/121:14:211
333799229,2cyclictest22118-21kworker/0:121:10:510
333799227,2cyclictest29892-21kworker/0:200:07:140
333799225,3cyclictest29892-21kworker/0:223:53:140
333799222,11cyclictest3539-21kworker/0:119:11:300
333799222,10cyclictest23508-21kworker/0:220:07:470
333799221,14cyclictest21145-21kworker/0:223:31:110
333799221,12cyclictest3539-21kworker/0:119:17:030
335499218,3cyclictest291312basename20:05:173
335499217,2cyclictest20634-21munin-run22:15:013
335499217,2cyclictest20634-21munin-run22:15:013
335499215,7cyclictest27745-21memory21:15:183
335499211,9cyclictest0-21swapper/323:11:433
334299219,4cyclictest13760-21seq21:56:441
334299213,6cyclictest16784-21smart_sda00:30:241
334299212,2cyclictest104032sleep119:22:321
334299212,10cyclictest22736-21smart_sda23:30:241
334299211,8cyclictest0-21swapper/100:39:231
334299211,15cyclictest0-21swapper/120:32:361
334299211,15cyclictest0-21swapper/119:57:331
334299211,15cyclictest0-21swapper/119:51:061
334299211,15cyclictest0-21swapper/119:51:061
334299211,14cyclictest0-21swapper/123:44:481
334299211,14cyclictest0-21swapper/120:16:131
334299211,11cyclictest31604-21grep23:50:251
334299211,11cyclictest10500-21grep21:50:151
334299211,10cyclictest21252-21munin-run23:30:011
3342992110,3cyclictest19724-21ntp_states22:10:211
3342992110,3cyclictest19724-21ntp_states22:10:211
334299210,16cyclictest0-21swapper/100:28:251
333799219,2cyclictest30317-21kworker/0:020:18:230
333799216,7cyclictest31575-21kworker/0:123:05:510
333799216,4cyclictest20395-21kworker/0:122:16:270
333799216,3cyclictest13373-21kworker/0:120:50:190
333799214,9cyclictest12925-21kworker/0:100:37:030
333799212,9cyclictest22605-21kworker/0:022:22:590
333799211,8cyclictest15915-21kworker/0:022:05:530
333799211,12cyclictest32520-21kworker/0:220:25:150
335499209,3cyclictest0-21swapper/321:35:393
335499205,7cyclictest39-21rcuc/300:35:143
335499205,6cyclictest808-21snmpd23:56:043
335499205,12cyclictest827-21runrttasks00:02:453
335499203,8cyclictest0-21swapper/323:50:123
335499203,5cyclictest808-21snmpd19:35:333
335499203,5cyclictest0-21swapper/323:43:113
335499202,9cyclictest87350irq/126-enp1s0-22:40:023
335499201,8cyclictest0-21swapper/300:05:203
3354992014,3cyclictest16138-21tail00:30:173
335499201,10cyclictest41-21ksoftirqd/323:45:103
334299209,3cyclictest4126-21cstates22:50:121
334299209,3cyclictest19440-21cat19:45:001
334299208,4cyclictest0-21swapper/119:35:121
334299201,8cyclictest0-21swapper/122:58:521
334299201,8cyclictest0-21swapper/122:32:541
334299201,8cyclictest0-21swapper/100:10:061
3342992013,3cyclictest0-21swapper/122:28:331
3342992012,3cyclictest0-21swapper/120:07:321
334299201,17cyclictest0-21swapper/120:02:271
334299201,15cyclictest0-21swapper/119:25:331
334299201,14cyclictest0-21swapper/123:35:041
334299201,14cyclictest0-21swapper/120:55:561
334299201,14cyclictest0-21swapper/120:27:211
334299201,11cyclictest9845-21memory00:15:181
334299201,11cyclictest16462-21/usr/sbin/munin22:05:241
334299201,11cyclictest14601-21diskstats22:00:131
334299200,15cyclictest0-21swapper/122:45:041
334299200,14cyclictest0-21swapper/123:45:021
333799208,2cyclictest14863-21kworker/0:019:50:260
333799208,2cyclictest14863-21kworker/0:019:50:260
333799207,2cyclictest30317-21kworker/0:020:12:390
333799202,9cyclictest25775-21kworker/0:020:02:310
333799202,10cyclictest23508-21kworker/0:219:59:260
333799202,10cyclictest14863-21kworker/0:019:40:060
333799201,11cyclictest8821-21kworker/0:320:35:270
335499199,2cyclictest17479-21sshd20:50:543
335499195,3cyclictest0-21swapper/323:20:443
335499195,3cyclictest0-21swapper/323:20:443
335499195,11cyclictest6825-21awk20:30:023
335499194,3cyclictest0-21swapper/320:03:243
335499192,8cyclictest0-21swapper/322:40:403
335499192,7cyclictest87550irq/128-enp1s0-22:00:193
335499191,9cyclictest41-21ksoftirqd/321:32:193
335499191,14cyclictest22414-21head22:15:263
335499191,12cyclictest0-21swapper/322:51:143
335499191,11cyclictest0-21swapper/323:05:323
335499191,11cyclictest0-21swapper/321:22:503
3354991910,2cyclictest87650irq/129-enp1s0-00:17:543
334299191,17cyclictest0-21swapper/123:05:171
334299191,16cyclictest0-21swapper/119:10:051
334299191,14cyclictest0-21swapper/123:11:561
334299190,14cyclictest0-21swapper/120:12:561
333799196,3cyclictest23397-21kworker/0:123:42:100
152452191,2sleep30-21swapper/320:45:283
335499188,4cyclictest3750-21/usr/sbin/munin20:20:243
335499187,3cyclictest6436-21/usr/sbin/munin19:15:113
335499187,3cyclictest25690-21sshd19:56:273
335499184,6cyclictest41-21ksoftirqd/322:59:453
335499184,4cyclictest13650-21cat00:25:163
335499182,13cyclictest10421-21grep20:35:233
335499181,8cyclictest0-21swapper/321:55:573
335499181,6cyclictest0-21swapper/321:54:233
335499181,5cyclictest0-21swapper/300:14:113
335499181,11cyclictest0-21swapper/321:42:543
3342991812,3cyclictest12121-21hddtemp_smartct20:40:161
3342991810,3cyclictest0-21swapper/123:01:111
333799180,10cyclictest4459-21kworker/0:120:30:230
335499179,2cyclictest87650irq/129-enp1s0-20:15:153
335499178,6cyclictest31292-21hwlatdetect20:10:173
335499172,4cyclictest4654-21ls19:10:213
335499172,4cyclictest20190-21fschecks_time19:45:153
3337991712,3cyclictest11536-21basename19:25:170
335499161,3cyclictest0-21swapper/319:40:143
3354991611,3cyclictest23415-21needreboot21:05:203
3354991611,3cyclictest11522-21hwlatdetect19:25:183
335499161,12cyclictest7051-21/usr/sbin/munin20:30:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional