You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-03 - 11:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 03, 2023 00:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
181902241163,25sleep20-21swapper/219:05:532
184972236163,22sleep10-21swapper/119:09:511
182432208140,55sleep30-21swapper/319:06:353
184132204166,25sleep00-21swapper/019:08:450
18613999290,1cyclictest0-21swapper/222:23:192
225472670,2sleep30-21swapper/300:35:233
133502650,3sleep17300-21diskmemload21:20:401
95472630,3sleep11860799cyclictest23:55:271
250752590,2sleep10-21swapper/119:20:301
12582550,4sleep121-21rcuc/122:10:381
18613995344,8cyclictest0-21swapper/222:12:022
179402520,1sleep30-21swapper/300:20:263
1862099431,27cyclictest19871-21latency_hist20:25:013
1862099413,31cyclictest548-40dnf20:54:333
1862099401,29cyclictest2176-21/usr/sbin/munin22:15:303
1862099401,27cyclictest12186-21gltestperf22:40:283
254722390,1sleep10-21swapper/120:35:321
1862099392,21cyclictest27702-21munin-run22:00:013
1862099391,26cyclictest20353-21smart_sda00:25:423
1862099383,25cyclictest16724-21tr22:55:223
1862099383,21cyclictest10393-21tail21:15:213
1862099381,25cyclictest21609-21/usr/sbin/munin23:10:273
1862099371,26cyclictest7011-21cstates23:50:233
1862099371,25cyclictest22034-21df19:15:193
1862099362,23cyclictest14386-21/usr/sbin/munin22:50:193
1862099362,23cyclictest14386-21/usr/sbin/munin22:50:193
1862099354,18cyclictest780-21rngd20:57:373
1862099354,17cyclictest86550irq/126-enp1s0-20:05:193
1862099353,18cyclictest14141-21/usr/sbin/munin21:25:413
1862099352,27cyclictest10071-21ntpq23:55:333
1862099351,23cyclictest20116-21/usr/sbin/munin20:25:363
1862099351,22cyclictest29644-21timerwakeupswit23:25:363
1862099351,21cyclictest31890-21unixbench_multi23:30:363
1862099351,20cyclictest9109-21/usr/sbin/munin20:00:203
1862099351,17cyclictest22101-21runrttasks21:40:383
1862099344,15cyclictest0-21swapper/321:10:213
1862099343,26cyclictest3496-21ntp_states23:40:283
1862099343,22cyclictest9907-1kworker/3:0H20:44:563
1862099342,23cyclictest18809-21/usr/sbin/munin19:10:253
1862099341,24cyclictest25125-21/usr/sbin/munin21:50:243
1862099340,23cyclictest5044-21sed22:20:293
1861399349,5cyclictest12050irq/123-ahci[0000:30:022
18620993313,16cyclictest1-21systemd22:30:093
1862099331,22cyclictest28899-21/usr/sbin/munin20:45:313
1862099331,20cyclictest784-21NetworkManager23:08:503
1862099331,19cyclictest18286-21latency_hist21:35:013
1862099330,26cyclictest0-21swapper/322:35:263
1862099323,19cyclictest86550irq/126-enp1s0-21:37:123
1862099321,28cyclictest23811-21/usr/sbin/munin23:15:203
1862099321,28cyclictest18788-21/usr/sbin/munin23:00:263
1862099321,27cyclictest32689-21/usr/sbin/munin19:40:333
1862099321,20cyclictest848-21sssd_nss19:58:203
1862099321,13cyclictest0-21swapper/320:36:123
1862099313,20cyclictest0-21swapper/322:01:203
1862099312,25cyclictest2518-21/usr/sbin/munin19:45:313
1862099312,15cyclictest836-21runrttasks00:34:073
1862099311,27cyclictest836-21runrttasks19:50:233
1862099311,27cyclictest23897-21/usr/sbin/munin19:20:263
1862099311,26cyclictest27169-21ntp_states19:25:323
1862099311,20cyclictest86850irq/129-enp1s0-22:05:133
1862099311,18cyclictest22918-21/usr/sbin/munin21:45:163
1862099311,17cyclictest27825-21sshd23:23:203
149782310,4sleep2311ktimersoftd/200:10:342
1862099305,18cyclictest12050irq/123-ahci[0021:05:223
1862099304,23cyclictest848-21sssd_nss19:35:223
1862099304,23cyclictest848-21sssd_nss19:35:223
1862099304,14cyclictest780-21rngd21:00:193
1862099302,24cyclictest16062-21diskstats20:15:213
1862099302,16cyclictest86550irq/126-enp1s0-22:46:093
1862099301,3cyclictest11850-21/usr/sbin/munin00:05:233
1862099301,26cyclictest28293-21/usr/sbin/munin19:30:213
1862099301,26cyclictest13516-21/usr/sbin/munin20:10:333
1862099301,25cyclictest1294-21ntp_states23:35:373
1862099300,1cyclictest0-21swapper/321:20:393
1862099294,11cyclictest780-21rngd00:13:313
1862099292,24cyclictest32692-21df_inode22:10:243
1862099292,15cyclictest86550irq/126-enp1s0-23:45:293
1862099291,25cyclictest22985-21hddtemp_smartct20:30:263
1862099286,9cyclictest86650irq/127-enp1s0-22:29:153
1862099281,13cyclictest0-21swapper/300:03:293
1862099281,13cyclictest0-21swapper/300:03:293
1860399281,19cyclictest0-21swapper/023:55:300
1860399281,19cyclictest0-21swapper/023:40:270
1862099271,23cyclictest16642-21munin-run00:20:013
1860799271,20cyclictest0-21swapper/121:30:471
1860399271,17cyclictest0-21swapper/022:20:350
1860399270,16cyclictest0-21swapper/022:29:550
211542260,2sleep19-21rcu_preempt20:25:331
1860799261,17cyclictest7955-21if_err_enp1s022:30:211
1860799261,13cyclictest0-21swapper/121:03:231
1860399267,2cyclictest11-21rcuc/022:40:010
1860399262,9cyclictest1033-21ntp_states22:10:340
1860399262,17cyclictest0-21swapper/022:43:080
1860399261,22cyclictest836-21runrttasks21:19:370
1860399261,18cyclictest0-21swapper/023:18:170
1860399261,18cyclictest0-21swapper/022:15:360
1860399261,18cyclictest0-21swapper/000:32:370
1860399261,16cyclictest0-21swapper/021:59:220
1860399260,17cyclictest0-21swapper/021:49:470
317722250,4sleep230-21rcuc/223:30:352
198532250,4sleep232-21ksoftirqd/219:10:322
1861399255,3cyclictest0-21swapper/219:33:372
1861399253,6cyclictest0-21swapper/221:19:372
1861399252,13cyclictest0-21swapper/223:55:402
1860799252,14cyclictest0-21swapper/122:35:391
1860799251,12cyclictest0-21swapper/121:05:111
1860799250,19cyclictest0-21swapper/121:40:211
1860799250,19cyclictest0-21swapper/100:34:471
1860399252,11cyclictest28066-21awk19:30:020
18603992519,3cyclictest11-21rcuc/019:20:310
1860399251,18cyclictest0-21swapper/021:43:060
1860399251,18cyclictest0-21swapper/000:05:180
1860399251,17cyclictest0-21swapper/022:55:380
1860399251,17cyclictest0-21swapper/021:25:300
1860399251,17cyclictest0-21swapper/021:20:590
1860399251,17cyclictest0-21swapper/000:39:090
1860399250,17cyclictest0-21swapper/021:12:200
1860399250,15cyclictest0-21swapper/021:38:430
1861399249,3cyclictest12050irq/123-ahci[0019:25:002
1861399241,9cyclictest0-21swapper/222:08:552
1860799247,9cyclictest0-21swapper/123:40:301
1860799240,17cyclictest0-21swapper/122:50:411
1860799240,17cyclictest0-21swapper/122:50:411
18603992415,5cyclictest16704-21awk21:30:270
1860399241,17cyclictest0-21swapper/023:46:100
1860399241,17cyclictest0-21swapper/023:29:490
1860399241,17cyclictest0-21swapper/023:10:490
1860399241,16cyclictest0-21swapper/022:07:090
1860399241,15cyclictest0-21swapper/023:50:320
1860399241,15cyclictest0-21swapper/022:30:130
1860399241,14cyclictest0-21swapper/023:05:440
1860399241,12cyclictest4490-21idleruntime-cro19:50:010
1860399241,11cyclictest16041-21awk00:15:010
18613992314,7cyclictest32210-21kworker/u8:220:00:322
1860799237,14cyclictest4333-21kworker/u8:223:50:501
1860799235,8cyclictest0-21swapper/123:37:151
1860799232,13cyclictest0-21swapper/100:23:411
1860799231,16cyclictest0-21swapper/119:49:021
1860799231,16cyclictest0-21swapper/119:44:571
1860799231,14cyclictest0-21swapper/123:27:461
1860799231,14cyclictest0-21swapper/123:13:191
1860799230,5cyclictest0-21swapper/121:59:591
1860799230,17cyclictest0-21swapper/121:17:151
1860799230,14cyclictest0-21swapper/121:35:341
1860399238,3cyclictest28620-21sed22:00:190
1860399236,10cyclictest0-21swapper/022:47:240
1860399231,16cyclictest0-21swapper/023:38:050
1860399231,16cyclictest0-21swapper/023:01:010
1860399230,15cyclictest0-21swapper/000:04:080
1860399230,15cyclictest0-21swapper/000:04:080
1861399227,3cyclictest12050irq/123-ahci[0021:00:022
1861399227,3cyclictest0-21swapper/222:00:312
1861399225,4cyclictest0-21swapper/220:14:192
1860799225,9cyclictest0-21swapper/122:08:001
1860799221,18cyclictest823-21snmpd23:08:481
1860799221,16cyclictest0-21swapper/120:44:381
1860799221,16cyclictest0-21swapper/119:13:481
1860799221,15cyclictest0-21swapper/120:56:051
1860799221,15cyclictest0-21swapper/120:16:011
1860799221,15cyclictest0-21swapper/119:16:431
18607992211,2cyclictest19430-21diskstats00:25:231
1860799220,15cyclictest30641-21kworker/u8:121:27:441
1860399221,16cyclictest0-21swapper/023:22:490
1860399221,15cyclictest0-21swapper/000:25:350
1861399216,5cyclictest792-21avahi-daemon22:48:432
1861399215,5cyclictest792-21avahi-daemon20:08:172
1861399215,4cyclictest0-21swapper/200:19:502
1861399213,15cyclictest30068-21ps20:45:312
1861399212,9cyclictest86850irq/129-enp1s0-21:50:302
1861399211,7cyclictest19088-21perl21:35:232
1861399211,10cyclictest2850-21smartctl22:15:232
1860799219,3cyclictest7966-21uniq21:10:231
1860799219,3cyclictest31472-21ntp_states23:30:311
1860799211,8cyclictest0-21swapper/121:50:081
1860799211,2cyclictest5680-21cat19:50:301
1860799211,15cyclictest0-21swapper/122:28:081
1860799211,15cyclictest0-21swapper/100:13:361
1860799211,14cyclictest0-21swapper/100:15:461
1860799211,13cyclictest0-21swapper/123:22:061
1860799211,11cyclictest6330-21sh22:25:011
1860799211,10cyclictest28058-21perf19:30:011
1860799210,9cyclictest0-21swapper/119:35:351
1860799210,9cyclictest0-21swapper/119:35:351
1860799210,15cyclictest0-21swapper/122:55:051
1860799210,15cyclictest0-21swapper/119:55:221
1860399216,10cyclictest0-21swapper/021:50:410
1860399215,14cyclictest17686-21perf20:20:010
1860399212,11cyclictest5339-21/usr/sbin/munin21:05:350
1860399212,10cyclictest9109-21/usr/sbin/munin20:00:210
1860399211,9cyclictest0-21swapper/022:50:540
1860399211,9cyclictest0-21swapper/022:50:540
1860399211,15cyclictest86550irq/126-enp1s0-19:10:330
1860399211,14cyclictest15847-21kworker/u8:200:15:590
1860399211,13cyclictest0-21swapper/000:24:100
1860399211,11cyclictest11318-21/usr/sbin/munin20:05:310
39192200,3sleep11860799cyclictest22:17:201
1861399208,3cyclictest12050irq/123-ahci[0021:35:012
1861399207,2cyclictest86550irq/126-enp1s0-00:20:162
1861399205,5cyclictest792-21avahi-daemon23:54:052
1861399202,15cyclictest32138-21ntpq20:50:292
1861399201,8cyclictest311ktimersoftd/221:50:012
1861399201,8cyclictest0-21swapper/222:50:242
1861399201,8cyclictest0-21swapper/222:50:242
1861399201,6cyclictest581-21systemd-journal23:45:012
1861399201,5cyclictest32-21ksoftirqd/221:10:222
18613992015,3cyclictest4078-21grep19:45:332
1861399201,3cyclictest86550irq/126-enp1s0-21:55:022
18613992013,4cyclictest23573-21latency_hist23:15:012
1861399201,11cyclictest0-21swapper/220:40:192
1860799207,2cyclictest86850irq/129-enp1s0-20:05:221
1860799202,8cyclictest0-21swapper/123:04:031
1860799201,18cyclictest0-21swapper/100:08:511
1860799201,17cyclictest0-21swapper/100:35:041
1860799201,15cyclictest0-21swapper/120:45:441
1860799201,15cyclictest0-21swapper/120:23:451
1860799201,14cyclictest0-21swapper/122:43:381
1860799201,10cyclictest31949-21cat20:50:271
1860799200,3cyclictest0-21swapper/121:45:151
1860799200,15cyclictest0-21swapper/122:45:541
1860799200,15cyclictest0-21swapper/119:30:121
1860799200,14cyclictest0-21swapper/120:00:431
1860399202,6cyclictest28899-21/usr/sbin/munin20:45:190
1860399202,13cyclictest86550irq/126-enp1s0-19:35:240
1860399202,13cyclictest86550irq/126-enp1s0-19:35:240
1860399201,6cyclictest611-21perf20:55:010
1860399201,6cyclictest23486-21seq19:17:560
18603992016,3cyclictest0-21swapper/021:03:490
1860399201,4cyclictest0-21swapper/023:30:260
1860399201,11cyclictest4719-21/usr/sbin/munin19:50:270
1860399201,11cyclictest20116-21/usr/sbin/munin20:25:350
1860399201,10cyclictest0-21swapper/020:20:300
1861399199,3cyclictest0-21swapper/220:26:072
1861399198,3cyclictest86550irq/126-enp1s0-20:20:352
1861399196,4cyclictest86850irq/129-enp1s0-23:45:312
1861399196,3cyclictest86550irq/126-enp1s0-22:55:222
1861399195,3cyclictest86750irq/128-enp1s0-21:25:262
1861399195,3cyclictest32-21ksoftirqd/221:42:132
1861399195,3cyclictest254732cat20:35:322
1861399193,5cyclictest792-21avahi-daemon19:51:012
1861399193,5cyclictest0-21swapper/221:20:392
1861399193,3cyclictest86550irq/126-enp1s0-20:35:012
1861399192,14cyclictest16362-21gltestperf20:15:232
1861399192,14cyclictest13101-21unixbench_multi22:40:402
1861399191,7cyclictest0-21swapper/200:07:292
1861399191,6cyclictest26803-21idleruntime19:25:262
1861399191,6cyclictest19943-21awk23:00:282
1861399191,6cyclictest0-21swapper/200:34:012
1861399191,4cyclictest0-21swapper/222:26:262
18613991914,3cyclictest32-21ksoftirqd/221:00:342
18613991913,3cyclictest21130-21load19:15:032
1861399191,17cyclictest0-21swapper/200:35:502
1860799197,4cyclictest28263-21df22:00:151
1860799192,11cyclictest86850irq/129-enp1s0-23:15:311
18607991915,3cyclictest0-21swapper/120:13:361
1860799191,3cyclictest0-21swapper/100:03:511
1860799191,3cyclictest0-21swapper/100:03:511
1860799190,15cyclictest0-21swapper/120:31:401
1860399196,2cyclictest5750irq/9-acpi19:30:240
1860399195,8cyclictest11-21rcuc/019:55:250
322442181,2sleep20-21swapper/219:37:342
322442181,2sleep20-21swapper/219:37:342
1861399184,3cyclictest32-21ksoftirqd/223:25:212
1861399182,7cyclictest14092-21kworker/2:019:40:152
1861399181,7cyclictest0-21swapper/223:18:452
1861399181,6cyclictest0-21swapper/200:01:272
1861399181,6cyclictest0-21swapper/200:01:272
1861399181,5cyclictest9737-21diskstats22:35:232
18613991813,3cyclictest1913-21taskset23:36:342
18613991812,3cyclictest6990-21sort19:55:222
1860799180,17cyclictest0-21swapper/123:45:271
1860399184,2cyclictest27729-21kworker/0:120:14:190
18603991813,3cyclictest823-21snmpd20:55:200
1860399181,15cyclictest0-21swapper/020:35:030
54032170,2sleep20-21swapper/221:05:212
1861399174,3cyclictest0-21swapper/223:08:572
1861399174,2cyclictest86550irq/126-enp1s0-23:20:282
1861399171,4cyclictest8353-21ps22:30:262
1860399175,3cyclictest0-21swapper/020:30:230
18603991713,2cyclictest1120-21irqstats19:40:280
1860399161,5cyclictest0-21swapper/020:40:560
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional