You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Fri May 02, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
111092238164,24sleep30-21swapper/319:09:583
109492231154,25sleep20-21swapper/219:07:562
109222221152,55sleep00-21swapper/019:07:340
107792203165,25sleep10-21swapper/119:05:421
2587921730,3sleep21120999cyclictest22:05:312
306022600,2sleep10-21swapper/123:30:281
11209996057,2cyclictest0-21swapper/222:27:012
228532590,4sleep21120999cyclictest19:35:012
146442460,3sleep121-21rcuc/121:40:291
209962450,2sleep00-21swapper/019:30:140
11215993512,11cyclictest25057-21awk19:40:013
1121599344,22cyclictest0-21swapper/321:19:163
155452330,4sleep121-21rcuc/119:15:251
11215993315,4cyclictest6206-21/usr/sbin/munin22:35:233
11209993321,2cyclictest11116-21kworker/2:123:40:222
1121599328,15cyclictest4883-21timerandwakeup23:45:283
1121599319,10cyclictest8196-21/usr/sbin/munin21:25:303
1121599319,10cyclictest8196-21/usr/sbin/munin21:25:303
11215993111,7cyclictest780-21rngd20:35:233
1121599309,5cyclictest11405-21/usr/sbin/munin19:10:223
1121599309,5cyclictest11405-21/usr/sbin/munin19:10:223
1121599309,11cyclictest32423-21grep22:20:283
1121599305,13cyclictest0-21swapper/323:11:333
1120999302,12cyclictest11116-21kworker/2:121:50:142
1121599298,12cyclictest31249-21latency_hist23:35:013
1121599297,14cyclictest0-21swapper/300:35:213
1121599295,14cyclictest1702-21/usr/sbin/munin20:00:153
11215992919,6cyclictest15815-40dnf22:59:333
1121599291,8cyclictest0-21swapper/321:40:193
11215992912,9cyclictest25246-21diskstats23:20:163
1119999294,12cyclictest32242-21diskmemload23:55:010
1121599289,11cyclictest623-21/usr/sbin/munin22:25:223
1121599288,13cyclictest0-21swapper/321:58:153
1121599286,10cyclictest14435-21taskset20:27:133
1121599284,9cyclictest0-21swapper/323:00:183
1121599283,17cyclictest0-21swapper/300:25:193
11215992818,6cyclictest27707-40dnf20:58:323
11215992818,6cyclictest10100-40dnf00:00:113
11215992815,5cyclictest780-21rngd20:48:033
1121599281,11cyclictest30126-21cat22:15:273
11215992810,14cyclictest24964-21hddtemp_smartct22:05:193
11209992810,3cyclictest11116-21kworker/2:120:07:122
1120599281,17cyclictest0-21swapper/100:35:231
1121599279,7cyclictest11705-21/usr/sbin/munin22:50:143
1121599279,5cyclictest0-21swapper/321:35:253
1121599278,6cyclictest23323-21/usr/sbin/munin00:30:293
1121599276,9cyclictest18305-21/usr/sbin/munin23:05:283
1121599274,15cyclictest9363-21cat22:45:023
11215992719,5cyclictest10843-21diskstats20:20:183
11215992719,5cyclictest10843-21diskstats20:20:183
11215992711,13cyclictest17393-21runrttasks21:49:193
11215992710,5cyclictest23679-21/usr/sbin/munin20:50:233
11215992710,11cyclictest86650irq/127-enp1s0-00:20:293
1121599269,9cyclictest20883-21/usr/sbin/munin19:30:133
1121599262,17cyclictest0-21swapper/323:55:223
11215992618,5cyclictest19749-21ntp_states19:25:223
11215992617,5cyclictest6978-21smart_sda23:50:283
11215992610,5cyclictest16493-21/usr/sbin/munin19:20:243
11215992610,13cyclictest4501-21sa121:20:103
11209992618,2cyclictest11116-21kworker/2:121:28:542
11209992618,2cyclictest11116-21kworker/2:121:28:542
11209992617,3cyclictest11116-21kworker/2:120:57:382
1120599261,22cyclictest10105-40updatedb00:00:101
1120599261,17cyclictest32663-21cstates21:10:161
1121599258,10cyclictest86750irq/128-enp1s0-19:56:323
1121599257,8cyclictest13099-1kworker/3:1H20:40:283
1121599257,7cyclictest8720-21uniq20:15:193
1121599257,11cyclictest1157-21ntp_offset21:10:243
1121599256,6cyclictest22082-21/usr/sbin/munin22:00:223
1121599255,8cyclictest29668-21/usr/sbin/munin19:50:283
1121599253,7cyclictest8400-21kworker/u8:021:31:003
1121599253,11cyclictest0-21swapper/300:11:503
11215992518,4cyclictest10709-21/usr/sbin/munin22:45:243
11215992517,5cyclictest2393-21ntp_states23:40:233
11215992511,5cyclictest14660-21forks19:15:163
1120999253,9cyclictest0-21swapper/223:21:202
11209992510,3cyclictest11116-21kworker/2:100:15:202
1120599252,16cyclictest86750irq/128-enp1s0-23:06:331
1120599251,20cyclictest20650-21turbostat19:30:011
1120599251,17cyclictest0-21swapper/121:30:251
11205992510,13cyclictest10807-21kworker/u8:121:45:371
1119999255,3cyclictest86750irq/128-enp1s0-22:45:230
1119999251,11cyclictest7087-21awk22:40:010
1121599246,14cyclictest4365-21fschecks_count20:05:193
11215992416,5cyclictest18190-21cut21:50:173
11215992415,6cyclictest31189-21memory21:05:213
11215992414,7cyclictest7017-21memory20:10:223
1121599241,10cyclictest0-21swapper/320:32:453
11215992410,5cyclictest3826-21memory22:30:213
11215992410,5cyclictest3826-21memory22:30:213
1120999244,2cyclictest86650irq/127-enp1s0-21:59:362
1120599242,12cyclictest1235-40dnf19:57:341
1120599241,17cyclictest0-21swapper/100:10:261
1120599241,10cyclictest10275-21awk20:20:011
193182230,3sleep10-21swapper/121:50:491
171542231,2sleep00-21swapper/020:35:140
1121599238,5cyclictest0-21swapper/323:35:203
1121599238,5cyclictest0-21swapper/323:35:203
11215992316,4cyclictest28422-21diskstats21:00:193
11215992316,4cyclictest26828-21df_inode22:10:133
11215992315,5cyclictest24286-21users23:15:293
11215992314,6cyclictest848-21sssd_nss19:40:223
11215992314,6cyclictest27090-21/usr/sbin/munin23:25:243
11215992314,6cyclictest12331-21/usr/sbin/munin00:05:163
1121599231,13cyclictest0-21swapper/300:15:033
11205992318,2cyclictest50682sleep121:20:221
11205992312,3cyclictest86750irq/128-enp1s0-21:25:191
11205992312,3cyclictest86750irq/128-enp1s0-21:25:191
1120599231,19cyclictest16444-21df_inode23:00:161
1120599231,17cyclictest0-21swapper/121:07:021
1120599231,16cyclictest0-21swapper/122:57:061
1120599231,16cyclictest0-21swapper/122:44:101
1120599231,15cyclictest0-21swapper/123:46:311
1120599230,17cyclictest0-21swapper/123:59:221
11215992213,6cyclictest848-21sssd_nss19:45:213
1120999228,3cyclictest11116-21kworker/2:120:36:182
1120999228,2cyclictest11116-21kworker/2:120:02:022
1120999223,11cyclictest15823-21chrt22:59:332
1120999221,8cyclictest11116-21kworker/2:121:12:122
11205992214,3cyclictest0-21swapper/121:15:411
11205992214,3cyclictest0-21swapper/119:13:201
11205992214,3cyclictest0-21swapper/119:13:191
1120599221,16cyclictest0-21swapper/123:20:421
1120599221,16cyclictest0-21swapper/122:47:231
1120599221,16cyclictest0-21swapper/122:27:451
1120599221,16cyclictest0-21swapper/122:12:311
1120599221,16cyclictest0-21swapper/121:59:581
1120599221,15cyclictest0-21swapper/123:52:241
1120599221,15cyclictest0-21swapper/120:40:371
11205992211,3cyclictest31637-21munin-run19:55:021
1120599220,16cyclictest0-21swapper/123:12:011
1120599220,16cyclictest0-21swapper/100:28:281
1120599220,15cyclictest32242-21diskmemload21:35:391
1119999221,8cyclictest0-21swapper/023:20:210
246552211,2sleep10-21swapper/120:50:241
1120999214,4cyclictest11116-21kworker/2:121:00:282
1120999212,3cyclictest11116-21kworker/2:121:05:322
1120999212,16cyclictest18423-21cut23:05:132
1120999211,9cyclictest11116-21kworker/2:121:34:142
1120599212,10cyclictest32242-21diskmemload00:15:221
1120599212,10cyclictest13046-21diskstats20:25:171
1120599211,6cyclictest0-21swapper/123:39:231
1120599211,6cyclictest0-21swapper/123:39:231
11205992114,2cyclictest0-21swapper/119:37:121
11205992113,2cyclictest0-21swapper/122:15:131
1120599211,19cyclictest0-21swapper/100:23:221
1120599211,17cyclictest28199-21cpuspeed_turbos21:00:171
1120599211,15cyclictest0-21swapper/123:18:141
1120599211,15cyclictest0-21swapper/122:53:491
1120599211,15cyclictest0-21swapper/122:38:381
1120599211,15cyclictest0-21swapper/122:04:041
1120599211,15cyclictest0-21swapper/120:59:311
1120599211,15cyclictest0-21swapper/120:22:071
1120599211,15cyclictest0-21swapper/120:22:071
1120599211,15cyclictest0-21swapper/119:22:271
1120599211,14cyclictest0-21swapper/119:45:131
1120599210,16cyclictest0-21swapper/122:31:211
1120599210,16cyclictest0-21swapper/122:31:211
1119999215,3cyclictest8545-21kworker/0:121:35:190
1119999211,8cyclictest168582latency_hist20:35:020
1119999211,5cyclictest0-21swapper/000:33:480
324942201,2sleep00-21swapper/022:20:350
245672201,3sleep00-21swapper/023:18:560
1120999208,2cyclictest86650irq/127-enp1s0-19:53:022
1120999208,2cyclictest11116-21kworker/2:120:41:382
1120999202,10cyclictest11116-21kworker/2:120:15:162
1120599207,4cyclictest12830-21fschecks_time00:05:171
11205992017,2cyclictest0-21swapper/119:41:291
11205992014,3cyclictest28012-21cat23:25:201
11205992014,3cyclictest15362-21cut20:30:161
11205992013,2cyclictest0-21swapper/122:08:291
1120599201,15cyclictest0-21swapper/119:30:451
1120599201,15cyclictest0-21swapper/100:30:521
1120599201,14cyclictest0-21swapper/120:46:561
1120599201,14cyclictest0-21swapper/120:07:361
1120599201,11cyclictest3690-21awk20:05:011
11205992010,2cyclictest31561-21/usr/sbin/munin22:20:181
1119999209,3cyclictest6104-21/usr/sbin/munin20:10:260
1119999209,3cyclictest21484-21/usr/sbin/munin20:45:300
1119999201,6cyclictest0-21swapper/022:12:560
1119999201,11cyclictest22082-21/usr/sbin/munin22:00:260
1119999201,11cyclictest18934-21/usr/sbin/munin00:20:300
78102190,4sleep011-21rcuc/021:25:260
78102190,4sleep011-21rcuc/021:25:260
75962191,2sleep20-21swapper/223:54:052
51462191,2sleep00-21swapper/023:45:490
252712191,2sleep00-21swapper/019:40:100
217392191,2sleep00-21swapper/021:58:430
206312191,2sleep20-21swapper/219:29:552
1120999194,3cyclictest86750irq/128-enp1s0-22:54:142
1120999194,3cyclictest11116-21kworker/2:123:55:142
1120999192,6cyclictest581-21systemd-journal23:40:002
1120999192,6cyclictest581-21systemd-journal23:40:002
1120999191,8cyclictest27302-21kernelversion22:10:172
11209991914,3cyclictest5106-21/usr/sbin/munin22:35:142
11209991914,3cyclictest21818-21df20:45:172
1120999191,10cyclictest0-21swapper/221:17:132
11205991916,2cyclictest0-21swapper/123:40:491
1120599191,10cyclictest8062-21latency_hist20:15:011
1119999198,9cyclictest31484-21/usr/sbin/munin23:35:210
1119999198,9cyclictest31484-21/usr/sbin/munin23:35:210
1119999198,3cyclictest11705-21/usr/sbin/munin22:50:200
1119999194,4cyclictest26023-21taskset22:07:120
11199991913,3cyclictest3580-21gltestperf22:30:180
11199991913,3cyclictest3580-21gltestperf22:30:180
1119999191,10cyclictest32481-21/usr/sbin/munin21:10:270
1119999191,10cyclictest16101-21/usr/sbin/munin23:00:230
1119999191,10cyclictest14296-21/usr/sbin/munin19:15:220
1119999191,10cyclictest13302-21/usr/sbin/munin21:40:250
292922180,2sleep00-21swapper/019:48:210
166282181,2sleep20-21swapper/220:32:092
1120999185,4cyclictest25167-21sshd00:33:332
1120999184,12cyclictest21780-21head00:25:162
1120999183,3cyclictest24139-21ntp_states19:35:212
1120999183,2cyclictest86650irq/127-enp1s0-00:10:142
1120999181,8cyclictest7303-21/usr/sbin/munin22:40:172
1120999181,7cyclictest24053-21crond22:05:022
1120999181,3cyclictest86550irq/126-enp1s0-22:15:082
11209991813,3cyclictest32533-21seq22:21:062
1120999180,9cyclictest0-21swapper/223:10:192
1120999180,8cyclictest86750irq/128-enp1s0-22:30:202
1120999180,8cyclictest86750irq/128-enp1s0-22:30:202
11205991811,4cyclictest17741-21tail20:35:161
1119999188,2cyclictest8903-21/usr/sbin/munin21:30:220
1119999187,3cyclictest23077-21/usr/sbin/munin19:35:230
1119999184,3cyclictest1777-21ntpq22:25:240
1119999183,3cyclictest13378-21ntp_kernel_pll_00:05:220
1119999183,3cyclictest0-21swapper/023:55:090
1119999182,10cyclictest31827-1kworker/u9:200:11:510
1119999181,8cyclictest8298-21/usr/sbin/munin20:15:240
1119999181,8cyclictest0-21swapper/021:50:240
1119999181,4cyclictest0-21swapper/023:27:120
1119999181,3cyclictest0-21swapper/023:05:220
11199991813,3cyclictest3294-21awk23:45:010
11199991811,4cyclictest25987-21/usr/sbin/munin00:35:170
11199991810,5cyclictest848-21sssd_nss23:10:190
189152170,2sleep20-21swapper/200:20:022
181922171,2sleep00-21swapper/019:21:440
178052170,2sleep00-21swapper/000:15:240
1120999179,5cyclictest26126-21grep00:35:162
1120999178,2cyclictest86750irq/128-enp1s0-21:20:282
1120999175,2cyclictest30-21rcuc/200:03:402
1120999174,2cyclictest86550irq/126-enp1s0-22:45:222
1120999173,7cyclictest0-21swapper/221:44:152
1120999172,3cyclictest11116-21kworker/2:119:20:222
1120999171,7cyclictest27473-21/usr/sbin/munin19:45:262
1120999171,7cyclictest0-21swapper/221:37:092
1120999171,6cyclictest15834-21df21:45:152
1120999171,6cyclictest0-21swapper/223:00:152
11209991712,3cyclictest27090-21/usr/sbin/munin23:25:242
11209991712,3cyclictest13831-21fschecks_count19:15:022
11209991711,3cyclictest6707-21tail20:10:182
1120999171,13cyclictest581-21systemd-journal20:55:002
1120999171,13cyclictest4659-21pmu-power23:45:242
11209991711,3cyclictest11492-21ls19:10:142
11209991711,3cyclictest11492-21ls19:10:142
11209991711,3cyclictest11116-21kworker/2:120:25:382
1119999174,3cyclictest24026-21diskstats20:50:180
1119999172,6cyclictest0-21swapper/021:15:230
1119999172,5cyclictest30479-21irqstats19:50:200
1119999171,8cyclictest14634-21snmpd21:47:330
1119999171,6cyclictest0-21swapper/023:30:170
1119999171,6cyclictest0-21swapper/000:03:160
1119999171,5cyclictest0-21swapper/022:15:160
1119999171,3cyclictest0-21swapper/000:28:590
11199991712,3cyclictest31922-21cpu19:55:160
11199991712,3cyclictest1767-21ls20:00:150
1120999169,4cyclictest13768-21timerwakeupswit00:05:262
1120999166,3cyclictest86750irq/128-enp1s0-23:30:202
1120999162,11cyclictest26333-21ntpq19:40:272
1120999161,6cyclictest31862-21/usr/sbin/munin19:55:172
1120999161,6cyclictest0-21swapper/223:15:112
11209991612,3cyclictest0-21swapper/220:24:582
11209991612,3cyclictest0-21swapper/220:24:582
1119999161,7cyclictest28074-21/usr/sbin/munin21:00:200
1119999161,6cyclictest4514-21/usr/sbin/munin21:20:210
1119999161,6cyclictest32242-21diskmemload22:40:300
1119999161,6cyclictest30831-21perl21:05:170
1119999161,4cyclictest0-21swapper/022:55:180
11199991611,3cyclictest836-21runrttasks19:10:240
11199991611,3cyclictest836-21runrttasks19:10:230
11199991611,3cyclictest4933-21nfs4_client20:05:240
11199991611,3cyclictest27294-21cat20:55:300
11199991610,3cyclictest10653-21tr20:20:160
11199991610,3cyclictest10653-21tr20:20:160
1119999152,4cyclictest0-21swapper/019:26:080
1119999151,11cyclictest14305-21sshd20:25:320
11199991510,3cyclictest21266-21basename20:45:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional