You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-21 - 08:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Fri Aug 21, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25572235163,24sleep20-21swapper/219:07:282
24112234164,57sleep30-21swapper/319:05:363
24962204167,25sleep10-21swapper/119:06:431
25702203166,25sleep00-21swapper/019:07:380
2068921670,4sleep0282899cyclictest21:00:190
2557321360,4sleep3284899cyclictest21:10:253
540821300,1sleep30-21swapper/321:40:153
2844996157,3cyclictest0-21swapper/222:37:222
2844996157,3cyclictest0-21swapper/222:37:222
216782580,4sleep011-21rcuc/022:15:230
76332550,2sleep20-21swapper/221:45:152
166352530,5sleep1283799cyclictest22:05:171
224132470,3sleep20-21swapper/219:50:222
128272470,3sleep212829-21gltestperf23:10:152
284499416,4cyclictest3465-21sort20:20:242
2844993813,3cyclictest25099-21users19:55:262
36192370,3sleep13623-21latency21:35:191
2848993711,3cyclictest26949-21/usr/sbin/munin22:30:143
282899378,13cyclictest1710-21awk00:00:010
2828993712,10cyclictest1523-21df_inode22:45:150
284899361,33cyclictest0-21swapper/321:47:033
2844993633,2cyclictest0-21swapper/222:53:362
284499362,7cyclictest0-21swapper/221:21:592
284499361,4cyclictest30240-21diskstats23:50:152
284499361,13cyclictest0-21swapper/222:44:312
284499361,12cyclictest0-21swapper/223:03:472
284499351,3cyclictest4718-21sshd21:39:562
284499351,3cyclictest12050irq/123-ahci[0000:30:022
284499351,16cyclictest0-21swapper/221:08:512
284499351,13cyclictest0-21swapper/223:31:232
283799352,2cyclictest23905-21diskmemload23:58:131
282899358,13cyclictest808-21snmpd20:25:020
2848993423,3cyclictest0-21swapper/321:29:233
2848993412,3cyclictest7179-21sendmail_mailst19:15:243
2848993411,3cyclictest9977-21cut23:05:023
2848993411,3cyclictest16017-21hwlatdetect20:50:163
284499349,3cyclictest0-21swapper/219:36:312
284499341,6cyclictest0-21swapper/223:21:512
2844993411,3cyclictest0-21swapper/221:03:432
284499341,13cyclictest0-21swapper/220:19:152
2844993411,3cyclictest0-21swapper/220:14:192
2844993411,3cyclictest0-21swapper/220:14:192
284499341,12cyclictest0-21swapper/220:09:512
2828993415,10cyclictest24305-40dnf22:22:230
284899333,8cyclictest12580-40dnf00:23:253
284899331,6cyclictest0-21swapper/322:24:083
284899331,4cyclictest0-21swapper/300:30:573
284899331,28cyclictest47092sleep319:11:193
284899330,15cyclictest0-21swapper/320:39:393
284499339,3cyclictest0-21swapper/220:29:152
284499338,3cyclictest0-21swapper/221:57:312
284499333,3cyclictest87350irq/126-enp1s0-21:30:352
2844993313,3cyclictest29663-21grep23:50:012
2844993313,3cyclictest29663-21grep23:50:012
284499331,13cyclictest0-21swapper/219:23:152
284499331,12cyclictest0-21swapper/223:27:552
284499331,12cyclictest0-21swapper/223:07:552
2844993310,3cyclictest0-21swapper/220:46:312
282899336,8cyclictest14824-21ntp_states22:00:220
2828993312,6cyclictest5118-21cat19:15:010
284899321,6cyclictest0-21swapper/321:33:303
2848993211,3cyclictest6578-21sort22:55:193
284899321,11cyclictest0-21swapper/300:09:313
284499329,3cyclictest0-21swapper/223:40:272
284499329,3cyclictest0-21swapper/220:31:512
284499329,3cyclictest0-21swapper/200:08:072
284499321,6cyclictest0-21swapper/223:57:522
284499321,6cyclictest0-21swapper/222:31:052
284499321,6cyclictest0-21swapper/222:20:322
284499321,6cyclictest0-21swapper/221:43:112
284499321,5cyclictest1923-21/usr/sbin/munin00:00:222
284499321,5cyclictest0-21swapper/222:49:302
284499321,13cyclictest0-21swapper/223:39:592
284499321,13cyclictest0-21swapper/219:17:352
284499321,11cyclictest0-21swapper/221:27:472
284499321,10cyclictest0-21swapper/219:30:312
2844993210,3cyclictest0-21swapper/200:34:392
282899327,18cyclictest87350irq/126-enp1s0-23:45:170
282899327,18cyclictest87350irq/126-enp1s0-23:45:170
284899313,9cyclictest30247-40dnf21:21:593
284899312,4cyclictest20414-21cpu22:15:113
284899312,13cyclictest495-21jbd2/dm-0-819:40:273
284899311,9cyclictest15093-21turbostat.cron20:49:563
284899311,6cyclictest0-21swapper/319:25:233
284899311,13cyclictest0-21swapper/323:38:473
284899311,12cyclictest0-21swapper/323:19:313
284899311,12cyclictest0-21swapper/322:36:513
284899311,12cyclictest0-21swapper/322:36:513
284899311,11cyclictest0-21swapper/321:59:553
284899311,11cyclictest0-21swapper/300:39:353
284499319,3cyclictest0-21swapper/220:38:432
284499318,3cyclictest0-21swapper/222:09:352
284499317,3cyclictest0-21swapper/219:43:192
284499314,3cyclictest87650irq/129-enp1s0-23:15:232
284499311,5cyclictest0-21swapper/221:14:162
284499311,13cyclictest0-21swapper/200:39:512
282899318,8cyclictest4321-21ls00:05:150
2828993110,11cyclictest23280-21crond19:55:010
284899301,9cyclictest0-21swapper/319:59:103
2848993017,3cyclictest0-21swapper/323:53:433
284899301,26cyclictest12571-21df_inode23:10:133
284899301,15cyclictest0-21swapper/321:50:193
284899301,14cyclictest0-21swapper/320:22:473
284899301,13cyclictest17299-21/usr/sbin/munin23:20:153
284899301,13cyclictest0-21swapper/323:42:553
284899301,13cyclictest0-21swapper/320:01:273
284899301,12cyclictest0-21swapper/322:11:593
284899301,10cyclictest4348-21/usr/sbin/munin20:25:193
2848993010,3cyclictest20170-21ntpq19:45:193
284899300,15cyclictest0-21swapper/300:17:393
284899300,12cyclictest0-21swapper/320:14:033
284899300,12cyclictest0-21swapper/320:14:033
284499307,3cyclictest0-21swapper/219:47:472
2844993014,3cyclictest0-21swapper/222:04:202
284499301,13cyclictest0-21swapper/200:15:522
284499301,13cyclictest0-21swapper/200:13:272
284499301,12cyclictest0-21swapper/221:15:392
282899305,15cyclictest5878-21memory21:40:190
282899304,17cyclictest839-21lldpd00:20:160
284899299,4cyclictest0-21swapper/323:34:073
284899291,11cyclictest2016-21irqstats22:45:193
284899290,12cyclictest0-21swapper/322:50:033
284499297,3cyclictest0-21swapper/219:25:432
284499296,3cyclictest0-21swapper/220:03:512
284499291,13cyclictest0-21swapper/222:11:112
284499291,13cyclictest0-21swapper/221:53:472
284499291,12cyclictest0-21swapper/222:56:152
284499291,12cyclictest0-21swapper/200:24:152
2844992910,3cyclictest0-21swapper/222:28:512
284499290,13cyclictest0-21swapper/222:16:032
282899296,11cyclictest12050irq/123-ahci[0023:45:010
282899296,10cyclictest13813-21sed00:25:190
2828992912,4cyclictest11834-21unixbench_multi19:25:260
284899289,3cyclictest0-21swapper/319:38:393
284899281,11cyclictest0-21swapper/319:51:513
284899281,11cyclictest0-21swapper/300:12:393
2848992810,3cyclictest0-21swapper/321:08:273
284899280,9cyclictest0-21swapper/323:29:153
284899280,12cyclictest0-21swapper/323:09:353
284499281,13cyclictest0-21swapper/220:42:272
284499281,12cyclictest0-21swapper/219:10:512
282899289,16cyclictest3773-21taskset00:03:150
282899288,16cyclictest9574-21expr21:50:130
282899287,8cyclictest23905-21diskmemload22:40:140
282899287,18cyclictest26394-21kworker/u8:022:30:150
282899286,9cyclictest612-21grep21:30:130
282899286,14cyclictest4746-21perf21:40:010
282899286,13cyclictest7163-21/usr/sbin/munin21:45:170
2828992819,5cyclictest9776-40dnf19:20:570
2828992818,6cyclictest30107-21/usr/sbin/munin20:10:270
2828992818,6cyclictest30107-21/usr/sbin/munin20:10:270
284899277,3cyclictest0-21swapper/320:55:273
284899271,12cyclictest0-21swapper/322:29:393
284899270,16cyclictest0-21swapper/320:15:353
284499271,17cyclictest0-21swapper/220:55:312
282899279,5cyclictest8745-21/usr/sbin/munin20:35:140
282899278,5cyclictest29891-21/usr/sbin/munin23:50:190
282899278,15cyclictest12557-21grep21:55:190
282899277,5cyclictest87350irq/126-enp1s0-21:20:140
282899277,11cyclictest19980-21awk23:25:220
282899276,10cyclictest0-21swapper/021:15:200
282899273,15cyclictest12492-21kworker/u8:200:30:130
282899273,15cyclictest0-21swapper/023:15:130
282899272,9cyclictest10068-21unixbench_multi00:15:270
2828992718,5cyclictest18181-21smart_sda19:40:230
2828992713,6cyclictest11619-21users23:05:270
284899269,3cyclictest0-21swapper/322:42:073
284899269,3cyclictest0-21swapper/322:00:073
284899269,3cyclictest0-21swapper/321:04:393
284899268,3cyclictest0-21swapper/300:29:393
284899267,3cyclictest0-21swapper/323:56:393
284899267,3cyclictest0-21swapper/320:32:313
284899261,12cyclictest0-21swapper/300:01:113
2848992610,3cyclictest0-21swapper/319:33:233
284499269,3cyclictest0-21swapper/220:52:552
283799261,17cyclictest13297-21ntp_states23:10:211
282899266,9cyclictest17675-21expr00:35:140
282899266,7cyclictest87350irq/126-enp1s0-22:55:210
2828992617,5cyclictest13489-21diskstats20:45:140
282899261,5cyclictest87350irq/126-enp1s0-23:36:490
2828992612,7cyclictest808-21snmpd22:12:560
2828992612,5cyclictest8172-21diskstats23:00:140
2828992610,13cyclictest4134-21awk20:25:010
284899257,3cyclictest0-21swapper/321:16:553
283799251,12cyclictest0-21swapper/123:36:301
282899259,10cyclictest87650irq/129-enp1s0-23:20:250
282899257,14cyclictest2996-21kworker/u8:000:10:250
282899256,10cyclictest0-21swapper/020:10:010
282899255,9cyclictest23905-21diskmemload22:05:160
282899255,12cyclictest6545-21/usr/sbin/munin20:30:170
282899252,10cyclictest14723-21/usr/sbin/munin19:35:180
2828992517,5cyclictest25207-21ntp_states21:10:210
2828992515,3cyclictest87350irq/126-enp1s0-22:25:170
284899241,12cyclictest28483-21irqstats23:45:193
284899241,12cyclictest28483-21irqstats23:45:193
284899241,12cyclictest0-21swapper/321:36:193
2848992410,3cyclictest0-21swapper/320:06:513
2848992410,3cyclictest0-21swapper/319:20:073
283799242,12cyclictest12306-21munin-run19:30:011
283799241,16cyclictest0-21swapper/119:51:001
282899247,14cyclictest29204-21cpu22:35:110
282899247,14cyclictest29204-21cpu22:35:110
282899245,8cyclictest21323-21cstates23:30:130
282899244,17cyclictest10941-21/usr/sbin/munin20:40:180
2828992415,6cyclictest22476-21fschecks_time21:05:150
2828992415,6cyclictest19117-21/usr/sbin/munin19:45:220
2828992412,5cyclictest819-21lldpd20:18:370
283799235,11cyclictest0-21swapper/122:42:441
283799232,3cyclictest23-21ksoftirqd/122:36:131
283799232,3cyclictest23-21ksoftirqd/122:36:131
283799231,8cyclictest0-21swapper/122:26:191
283799231,8cyclictest0-21swapper/122:21:571
283799231,7cyclictest0-21swapper/122:10:081
2837992315,2cyclictest0-21swapper/120:02:041
283799231,17cyclictest0-21swapper/119:39:591
283799231,15cyclictest0-21swapper/121:53:101
282899237,9cyclictest87350irq/126-enp1s0-20:58:020
282899231,9cyclictest0-21swapper/021:25:120
2828992315,5cyclictest24425-21memory19:55:190
2828992314,6cyclictest12226-21/usr/sbin/munin23:10:120
2828992314,6cyclictest0-21swapper/020:50:230
222572230,10sleep121-21rcuc/121:05:131
284899223,3cyclictest0-21swapper/322:07:153
283799222,10cyclictest18894-21munin-run19:45:001
283799221,7cyclictest0-21swapper/122:51:291
2837992213,2cyclictest23905-21diskmemload21:24:471
283799221,2cyclictest25305-21ps21:10:221
283799221,10cyclictest0-21swapper/100:33:141
283799220,17cyclictest0-21swapper/121:43:581
283799220,16cyclictest0-21swapper/120:08:101
282899227,3cyclictest12528-21/usr/sbin/munin19:30:230
2828992213,6cyclictest5015-21/usr/sbin/munin22:50:260
2828992213,6cyclictest25706-21/usr/sbin/munin20:00:130
2828992212,6cyclictest808-21snmpd19:15:030
301802210,4sleep121-21rcuc/120:10:111
301802210,4sleep121-21rcuc/120:10:111
283799219,3cyclictest18418-40dnf23:22:571
283799215,4cyclictest0-21swapper/121:59:071
283799213,5cyclictest0-21swapper/100:18:331
2837992114,2cyclictest0-21swapper/122:01:171
283799211,3cyclictest23905-21diskmemload21:25:411
2837992113,3cyclictest0-21swapper/123:52:361
2837992113,2cyclictest0-21swapper/119:45:171
283799211,2cyclictest87650irq/129-enp1s0-23:07:351
283799211,2cyclictest0-21swapper/100:07:551
2837992112,2cyclictest23905-21diskmemload22:56:301
283799211,15cyclictest0-21swapper/123:42:231
283799211,15cyclictest0-21swapper/123:33:371
283799211,15cyclictest0-21swapper/121:16:001
283799211,15cyclictest0-21swapper/120:58:281
283799211,11cyclictest10717-21munin-run20:40:011
283799211,10cyclictest0-21swapper/100:22:261
283799210,16cyclictest0-21swapper/123:01:121
283799210,15cyclictest0-21swapper/119:12:241
283799210,10cyclictest0-21swapper/122:47:261
283799209,3cyclictest6907-21grep00:10:151
283799201,3cyclictest25049-21kworker/u8:100:03:081
2837992013,2cyclictest0-21swapper/100:37:391
283799201,15cyclictest0-21swapper/119:32:591
283799201,14cyclictest0-21swapper/123:15:031
283799201,14cyclictest0-21swapper/121:04:571
283799201,14cyclictest0-21swapper/120:51:521
283799201,14cyclictest0-21swapper/120:23:011
283799201,11cyclictest13382-21expr20:45:131
283799201,10cyclictest7461-21memory20:30:191
283799201,10cyclictest29667-21perf23:50:011
283799201,10cyclictest29667-21perf23:50:011
283799200,15cyclictest0-21swapper/123:27:181
283799200,15cyclictest0-21swapper/122:34:521
284899198,3cyclictest0-21swapper/320:40:033
283799199,3cyclictest20355-21/usr/sbin/munin22:15:141
283799191,5cyclictest0-21swapper/100:26:311
2837991911,3cyclictest0-21swapper/121:31:041
283799190,17cyclictest0-21swapper/119:56:481
283799190,14cyclictest0-21swapper/120:25:271
283799190,14cyclictest0-21swapper/120:15:101
283799190,14cyclictest0-21swapper/119:21:311
283799188,2cyclictest5579-21ntp_states19:15:031
283799181,15cyclictest0-21swapper/120:40:061
283799180,14cyclictest0-21swapper/121:47:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional