You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 11:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Nov 16, 2025 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
165832240163,25sleep20-21swapper/219:06:432
167552235160,24sleep30-21swapper/319:08:533
167952225158,54sleep10-21swapper/119:09:241
167242203165,25sleep00-21swapper/019:08:310
1237321590,4sleep11693599cyclictest23:50:201
2602921400,4sleep31694699cyclictest20:40:223
1934521050,3sleep00-21swapper/021:40:160
16940997370,2cyclictest0-21swapper/220:31:002
243442640,3sleep011-21rcuc/020:35:280
16940995854,3cyclictest0-21swapper/223:10:232
16940995753,3cyclictest0-21swapper/221:42:172
16940995753,3cyclictest0-21swapper/221:06:122
16940995450,3cyclictest0-21swapper/220:15:272
120222530,1sleep20-21swapper/223:50:172
325012510,2sleep30-21swapper/322:10:143
277922500,1sleep20-21swapper/220:45:192
1693599421,12cyclictest30211-21kworker/u8:119:16:031
1693099424,14cyclictest28295-21munin-run23:15:000
1694699384,6cyclictest10006-40dnf21:19:343
1694699355,16cyclictest780-21rngd20:39:333
16946993513,3cyclictest19720-21/usr/sbin/munin22:55:163
16946993511,3cyclictest26324-21/usr/sbin/munin23:10:163
1693599351,7cyclictest5667-21diskmemload22:11:221
16946993414,17cyclictest23813-21find00:15:283
16946993411,3cyclictest26977-21/usr/sbin/munin00:25:193
16946993411,3cyclictest26977-21/usr/sbin/munin00:25:193
1693599341,6cyclictest0-21swapper/120:44:081
1693599341,15cyclictest0-21swapper/119:20:161
1694699338,3cyclictest11620-21sadc20:10:013
1694699338,3cyclictest11620-21sadc20:10:013
1694699338,3cyclictest11549-21/usr/sbin/munin23:50:153
16946993313,17cyclictest10624-21df21:20:143
1694699331,10cyclictest5304-21diskstats23:35:163
16946993310,2cyclictest16764-21irqstats00:00:183
1694699329,3cyclictest30063-21date22:05:023
1694699328,3cyclictest0-21swapper/323:05:183
1694699327,1cyclictest0-21swapper/319:45:243
1694699323,5cyclictest0-21swapper/320:00:203
1694699321,7cyclictest12592-21chrt22:36:233
1694699321,5cyclictest27641-21latency_hist22:00:023
1694699321,5cyclictest16406-21taskset21:32:033
1694699321,3cyclictest12050irq/123-ahci[0022:50:013
16946993212,9cyclictest0-21swapper/321:10:533
16946993212,9cyclictest0-21swapper/321:10:533
16946993212,3cyclictest30572-21cut20:50:243
16946993211,4cyclictest32623-21/usr/sbin/munin20:55:253
1694699319,3cyclictest951-21latency_hist00:40:013
1694699319,2cyclictest19484-21fschecks_count21:40:193
1694699318,3cyclictest6526-21/usr/sbin/munin22:25:273
1694699318,3cyclictest2148-21users23:25:303
1694699315,18cyclictest780-21rngd20:14:393
16946993112,3cyclictest7353-21cat23:40:133
16946993112,16cyclictest4272-21sed23:30:263
16946993111,3cyclictest3845-21expr21:05:143
1694699311,12cyclictest30721-21/usr/sbin/munin23:20:133
1694699310,9cyclictest0-21swapper/321:25:253
1693599311,6cyclictest0-21swapper/100:22:551
1693099310,14cyclictest6927-21munin-run23:40:010
1694699307,3cyclictest16346-21cpuspeed_turbos20:20:153
1694699303,5cyclictest10421-21ntp_states23:45:233
1694699303,4cyclictest0-21swapper/320:30:283
1694699301,6cyclictest14905-21sshd22:42:493
1694699301,3cyclictest1459-21/usr/sbin/munin21:00:163
1694699301,12cyclictest28496-21awk20:45:283
1694699301,12cyclictest18456-21/usr/sbin/munin20:25:193
16946993010,3cyclictest836-21runrttasks19:22:303
16946993010,2cyclictest22101-21kernelversion21:45:213
1694699300,9cyclictest0-21swapper/322:05:253
1693099307,11cyclictest6311-21awk22:25:010
1693099304,12cyclictest12272-21perf21:25:010
1694699299,3cyclictest31378-21forks19:40:183
1694699299,3cyclictest24614-21cat19:25:183
1694699299,2cyclictest836-21runrttasks00:12:013
1694699298,2cyclictest836-21runrttasks22:32:413
1694699296,4cyclictest12050irq/123-ahci[0022:55:013
1694699293,2cyclictest86850irq/129-enp1s0-23:15:223
1694699291,4cyclictest5015-21latency_hist19:55:023
1694699291,3cyclictest86550irq/126-enp1s0-22:25:013
16946992912,4cyclictest30792-21perf19:40:013
16946992911,3cyclictest20145-21sh00:10:003
16946992910,3cyclictest24243-21mii-tool21:50:193
1694699290,17cyclictest0-21swapper/300:30:363
1693599291,13cyclictest0-21swapper/121:00:221
1693599291,12cyclictest0-21swapper/123:30:221
1693599291,11cyclictest12196-21df_inode20:10:161
1693099292,12cyclictest31427-21idleruntime-cro20:55:000
1694699281,3cyclictest86550irq/126-enp1s0-21:35:153
16946992811,3cyclictest14917-21cat20:15:203
16946992810,3cyclictest22205-21expr23:00:133
16946992810,3cyclictest17778-21head19:10:183
1694699280,2cyclictest86750irq/128-enp1s0-00:20:483
1693599281,12cyclictest0-21swapper/121:17:001
1693599281,11cyclictest0-21swapper/122:26:501
1693599281,10cyclictest0-21swapper/122:01:581
1693599281,10cyclictest0-21swapper/121:32:081
1693099282,18cyclictest5667-21diskmemload00:20:330
1694699279,3cyclictest7397-21latency_hist20:00:023
16946992712,3cyclictest0-21swapper/322:16:123
16946992712,3cyclictest0-21swapper/322:16:113
1693599270,6cyclictest0-21swapper/100:11:391
1693599270,12cyclictest5667-21diskmemload22:40:481
16930992711,14cyclictest13536-21perf23:55:010
31432260,2sleep00-21swapper/023:30:150
310692261,21sleep031187-21cut23:20:160
1694699261,3cyclictest86550irq/126-enp1s0-23:55:543
1694699261,2cyclictest0-21swapper/319:15:153
1693599261,13cyclictest30085-21/usr/sbin/munin22:05:141
1693599260,5cyclictest0-21swapper/119:52:101
1694699251,4cyclictest26620-21/usr/sbin/munin19:30:193
1693599253,4cyclictest784-21NetworkManager00:26:311
1693599253,4cyclictest784-21NetworkManager00:26:311
1693599252,20cyclictest24552-21munin-run00:20:011
1693599241,20cyclictest22585-21hddtemp_smartct23:00:171
1693599241,11cyclictest0-21swapper/120:45:121
1693599240,14cyclictest0-21swapper/123:25:281
1693099241,11cyclictest7222-21munin-run20:00:000
1693599231,13cyclictest0-21swapper/121:26:181
1693599230,9cyclictest0-21swapper/100:05:371
1693599230,10cyclictest0-21swapper/123:07:121
1693099238,12cyclictest10698-21munin-run22:35:010
16930992317,3cyclictest1987-21head19:45:250
1693599221,19cyclictest0-21swapper/123:18:151
1693599221,18cyclictest18702-21cstates20:25:151
1693599221,14cyclictest0-21swapper/119:37:221
1693599221,13cyclictest0-21swapper/123:38:371
1693599221,11cyclictest26749-21cpuspeed_turbos19:30:131
1693099229,11cyclictest5035-21awk19:55:000
1693099224,15cyclictest11638-21awk20:10:010
1693099224,15cyclictest11638-21awk20:10:010
16930992211,3cyclictest14634-21snmpd20:26:100
1694099216,6cyclictest836-21runrttasks19:33:522
1693599218,4cyclictest17845-21users20:20:281
1693599211,8cyclictest0-21swapper/122:45:521
16935992117,3cyclictest0-21swapper/123:13:381
16935992117,3cyclictest0-21swapper/120:54:181
16935992117,3cyclictest0-21swapper/119:10:181
16935992114,4cyclictest4987-21cut22:20:181
16935992111,3cyclictest32633-21smart_sda00:35:271
1693599211,13cyclictest0-21swapper/123:46:331
1693599211,13cyclictest0-21swapper/119:45:281
1693599211,12cyclictest5598-21diskstats19:55:171
1693599211,11cyclictest7708-21grep23:40:171
1693599211,11cyclictest21728-21ntp_states20:30:271
1693599211,11cyclictest0-21swapper/122:15:521
1693599211,11cyclictest0-21swapper/122:15:521
16935992110,3cyclictest25239-21irqstats19:25:221
1693599210,13cyclictest0-21swapper/121:40:241
1693599210,12cyclictest0-21swapper/120:55:181
1693099214,11cyclictest14634-21snmpd21:19:330
1693099212,11cyclictest13928-21irqstats22:40:180
1693099211,11cyclictest15952-21/usr/sbin/munin00:00:260
1693099211,11cyclictest15314-21/usr/sbin/munin22:45:270
1694099203,3cyclictest113082netstat21:20:202
1694099201,9cyclictest0-21swapper/222:01:232
1694099201,5cyclictest0-21swapper/200:30:012
16940992012,3cyclictest1247-21grep19:45:182
1693599209,3cyclictest32308-21users23:20:291
16935992016,3cyclictest0-21swapper/122:35:181
16935992016,3cyclictest0-21swapper/100:30:481
1693599201,18cyclictest0-21swapper/121:39:251
1693599201,18cyclictest0-21swapper/120:39:391
1693599201,13cyclictest0-21swapper/122:50:241
1693599201,11cyclictest10462-21irqstats20:05:191
1693599201,11cyclictest10462-21irqstats20:05:191
1693599201,10cyclictest16304-21/usr/sbin/munin00:00:161
1693599200,18cyclictest0-21swapper/122:30:251
1693599200,16cyclictest86850irq/129-enp1s0-21:51:051
1693599200,13cyclictest0-21swapper/120:17:461
1693099209,3cyclictest3691-21/usr/sbin/munin21:05:240
1693099209,3cyclictest31414-21cpu00:35:130
1693099202,10cyclictest459-21/usr/sbin/munin23:25:280
1693099202,10cyclictest21282-21/usr/sbin/munin21:45:220
1693099201,7cyclictest26742-21ntp_states21:55:230
16930992015,3cyclictest19112-21awk19:15:020
16930992014,3cyclictest31214-21sed00:35:010
16930992011,7cyclictest6874-21kworker/u8:123:47:200
1693099201,11cyclictest20027-21/usr/sbin/munin19:15:250
1693099201,10cyclictest27248-21/usr/sbin/munin20:45:140
1693099201,10cyclictest25049-21/usr/sbin/munin20:40:190
1693099200,10cyclictest0-21swapper/022:35:390
215942191,3sleep00-21swapper/022:59:010
1694099193,3cyclictest0-21swapper/219:37:472
1694099192,14cyclictest6430-21grep23:35:282
1693599198,3cyclictest14815-21ntp_states23:55:241
1693599193,2cyclictest86850irq/129-enp1s0-21:55:151
1693599191,8cyclictest31366-21seq19:40:171
1693599191,17cyclictest0-21swapper/121:23:431
1693599191,17cyclictest0-21swapper/120:00:371
1693599191,10cyclictest7678-21chrt21:12:471
1693599191,10cyclictest7678-21chrt21:12:471
1693599191,10cyclictest22832-21sed21:45:281
1693099198,3cyclictest23480-21/usr/sbin/munin21:50:170
1693099197,4cyclictest0-21swapper/000:26:360
1693099197,4cyclictest0-21swapper/000:26:360
1693099196,5cyclictest0-21swapper/000:11:130
1693099191,3cyclictest16259-21acpi20:20:130
16930991913,4cyclictest25059-21head19:25:200
1693099191,2cyclictest86550irq/126-enp1s0-22:10:160
16930991912,4cyclictest7785-21df_inode20:00:130
1693099191,11cyclictest0-21swapper/023:41:400
1693099191,10cyclictest5667-21diskmemload22:52:470
1693099191,10cyclictest29025-21fschecks_time23:15:160
1693099191,10cyclictest14685-21/usr/sbin/munin21:30:130
1693099191,10cyclictest0-21swapper/021:10:250
1693099191,10cyclictest0-21swapper/021:10:250
1693099190,9cyclictest0-21swapper/021:37:330
81912181,2sleep20-21swapper/222:26:202
317432181,2sleep20-21swapper/222:06:132
1694099183,6cyclictest28148-21ps00:25:282
1694099183,6cyclictest28148-21ps00:25:272
1694099183,5cyclictest0-21swapper/223:46:482
1694099183,3cyclictest14422-21hddtemp_smartct23:55:182
1694099182,6cyclictest19140-21munin-node19:15:012
1694099182,6cyclictest0-21swapper/221:10:232
1694099182,6cyclictest0-21swapper/221:10:232
1694099181,9cyclictest31219-21fschecks_time23:20:172
1694099181,7cyclictest0-21swapper/222:40:562
1694099181,6cyclictest22789-21unixbench_multi21:45:292
1694099181,6cyclictest20709-21strings22:55:242
1694099181,4cyclictest0-21swapper/223:28:482
16940991813,3cyclictest23726-21grep19:20:272
16940991812,3cyclictest8200-21cpuspeed_turbos21:15:152
16940991812,3cyclictest24737-21sensors_temp21:50:262
1694099181,15cyclictest0-21swapper/220:05:082
1694099181,15cyclictest0-21swapper/220:05:082
1694099181,12cyclictest0-21swapper/223:33:572
1694099181,10cyclictest0-21swapper/200:16:282
1693599187,3cyclictest20631-21load22:55:221
1693599181,2cyclictest0-21swapper/121:09:261
1693099187,3cyclictest31647-21/usr/sbin/munin20:55:300
1693099187,3cyclictest28216-21df22:00:160
1693099186,3cyclictest0-21swapper/022:26:100
1693099181,8cyclictest0-21swapper/019:38:330
16930991813,3cyclictest18168-21/usr/sbin/munin00:05:250
16930991811,4cyclictest27540-21cat19:30:190
1693099181,11cyclictest0-21swapper/023:05:250
1693099181,11cyclictest0-21swapper/022:16:100
1693099181,11cyclictest0-21swapper/022:16:100
1693099180,8cyclictest0-21swapper/000:15:040
118422181,3sleep00-21swapper/020:10:040
209422171,2sleep20-21swapper/219:15:232
1694099173,4cyclictest0-21swapper/200:00:272
1694099172,14cyclictest0-21swapper/200:09:182
1694099171,7cyclictest32182-21irqstats00:35:192
1694099171,7cyclictest0-21swapper/222:35:282
1694099171,6cyclictest14634-21snmpd21:01:182
1694099171,6cyclictest0-21swapper/222:15:162
1694099171,6cyclictest0-21swapper/222:15:162
16940991713,3cyclictest0-21swapper/220:39:002
16940991712,3cyclictest31019-21/usr/sbin/munin19:40:302
16940991712,3cyclictest28713-21expr23:15:152
16940991712,3cyclictest26273-21grep00:20:272
16940991712,3cyclictest20385-21/usr/sbin/munin00:10:202
16940991712,3cyclictest18989-21grep22:50:292
16940991712,3cyclictest13189-21idleruntime21:25:162
1694099171,15cyclictest0-21swapper/221:57:052
1694099171,14cyclictest0-21swapper/219:56:512
1694099171,13cyclictest32415-21tr22:10:142
16940991710,4cyclictest8844-21grep22:30:142
1693099171,9cyclictest0-21swapper/023:57:240
1693099171,9cyclictest0-21swapper/019:20:140
1693099171,3cyclictest0-21swapper/022:05:490
16930991713,2cyclictest22959-21ntp_kernel_pll_23:00:210
16930991712,3cyclictest20659-21/usr/sbin/munin20:30:240
165442171,2sleep20-21swapper/221:33:482
1694099169,4cyclictest3397-21taskset19:50:152
1694099161,9cyclictest0-21swapper/223:40:132
1694099161,8cyclictest0-21swapper/221:35:252
1694099161,7cyclictest0-21swapper/223:00:162
1694099161,7cyclictest0-21swapper/220:40:212
1694099161,3cyclictest0-21swapper/220:50:052
16940991611,3cyclictest7444-21/usr/sbin/munin20:00:232
16940991611,3cyclictest5067-21cat22:20:202
16940991611,3cyclictest24297-21cat23:05:122
16940991611,3cyclictest19367-21load20:25:212
1694099161,13cyclictest16884-21gltestperf20:20:182
16940991610,4cyclictest25273-21/usr/sbin/munin19:25:242
16940991610,3cyclictest15680-21fschecks_count22:45:172
16930991611,3cyclictest12814-21expr21:25:130
16930991610,3cyclictest31116-21cpuspeed_turbos19:40:150
16930991610,3cyclictest14051-21/usr/sbin/munin20:15:180
136212161,2sleep20-21swapper/220:12:362
16940991510,3cyclictest31907-21expr20:55:182
1693099159,3cyclictest2625-21grep21:00:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional