You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-10-04 - 08:57
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot4.osadl.org (updated Wed Oct 04, 2023 00:45:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6189742277196,63sleep00-21swapper/019:09:550
6172422263223,26sleep10-21swapper/119:05:141
61912599230193,6cyclictest627069-21kworker/u8:220:40:340
619129991950,2cyclictest655605-21fwupd20:40:341
159199760,4rtkit-daemon1590-21rtkit-daemon19:59:121
159199730,4rtkit-daemon1590-21rtkit-daemon23:10:410
159199710,4rtkit-daemon1590-21rtkit-daemon23:43:541
159199710,4rtkit-daemon1590-21rtkit-daemon23:29:351
159199710,4rtkit-daemon1590-21rtkit-daemon20:58:540
159199700,4rtkit-daemon1590-21rtkit-daemon20:12:591
159199670,3rtkit-daemon0-21swapper/121:31:210
619125996314,10cyclictest734298-21gpgconf22:45:030
159199620,4rtkit-daemon0-21swapper/120:14:470
159199620,3rtkit-daemon0-21swapper/119:57:220
159199610,3rtkit-daemon0-21swapper/021:04:231
619129996017,6cyclictest709564-21latency_hist22:10:011
619129996014,23cyclictest700016-21latency21:55:311
159199590,3rtkit-daemon1590-21rtkit-daemon22:01:391
7817522580,4sleep10-21swapper/123:50:411
619129995828,6cyclictest687348-21sh21:37:281
6908892570,4sleep10-21swapper/121:42:111
619125995723,4cyclictest801429-21sh00:20:000
159199570,4rtkit-daemon0-21swapper/119:43:220
159199570,3rtkit-daemon0-21swapper/023:25:320
159199570,3rtkit-daemon0-21swapper/023:05:111
159199560,3rtkit-daemon0-21swapper/122:10:070
159199560,3rtkit-daemon0-21swapper/121:28:190
159199560,3rtkit-daemon0-21swapper/020:54:051
6502582550,7sleep0161rcuc/020:25:380
619129995533,6cyclictest762010-21runrttasks23:22:441
159199550,3rtkit-daemon0-21swapper/121:20:490
159199550,3rtkit-daemon0-21swapper/020:57:491
159199540,4rtkit-daemon0-21swapper/023:01:321
159199540,3rtkit-daemon0-21swapper/121:43:060
619125995315,2cyclictest141rcu_preempt23:35:300
619125995313,20cyclictest661120-21dump-pmu-power20:55:010
159199530,3rtkit-daemon0-21swapper/100:21:200
159199530,3rtkit-daemon0-21swapper/022:42:281
619129995217,13cyclictest635878-21gltestperf19:50:301
619125995223,6cyclictest792706-21unixbench_singl00:05:340
159199520,3rtkit-daemon0-21swapper/119:31:040
619129995122,13cyclictest815397-21latency_hist00:40:011
619129995122,13cyclictest815397-21latency_hist00:40:011
159199510,3rtkit-daemon0-21swapper/122:01:120
6269042500,7sleep00-21swapper/019:25:350
619129995027,7cyclictest727630-21fschecks_count22:35:291
619129995027,6cyclictest796288-21sh00:10:431
619129995027,6cyclictest647984-21open_inodes20:20:321
619129995027,5cyclictest745250-21sh22:59:101
619129995024,6cyclictest632952-21sshd19:41:291
61912599508,2cyclictest0-21swapper/000:30:520
61912599508,2cyclictest0-21swapper/000:28:410
61912599506,27cyclictest752503-21munin-run23:10:000
619125995022,5cyclictest692448-21cpuspeed21:45:220
61912599501,3cyclictest141rcu_preempt22:10:000
159199500,5rtkit-daemon669319-21ssh21:11:570
159199500,5rtkit-daemon669319-21ssh21:11:570
159199500,3rtkit-daemon0-21swapper/119:25:021
159199500,3rtkit-daemon0-21swapper/023:17:351
61912999498,2cyclictest141rcu_preempt20:05:391
619129994927,8cyclictest723808-21/usr/sbin/munin22:30:211
619129994926,6cyclictest713414-21cpu22:15:201
619129994924,6cyclictest716934-21/usr/sbin/munin22:20:301
619129994924,6cyclictest716934-21/usr/sbin/munin22:20:291
619129994918,6cyclictest620015-21ntp_kernel_pll_19:10:321
619129994914,17cyclictest624818-21/usr/sbin/munin19:20:321
619125994915,11cyclictest762378-21ssh23:23:420
159199490,5rtkit-daemon771594-21ssh23:35:351
159199490,4rtkit-daemon737648-21ssh22:49:271
619129994825,7cyclictest778312-21timerwakeupswit23:45:351
619129994825,6cyclictest710886-21rt-features22:10:351
619129994818,7cyclictest803665-21sh00:22:051
619129994818,12cyclictest645641-21hwlatdetect20:15:251
619129994817,7cyclictest678095-21runrttasks21:25:081
619129994814,15cyclictest692339-21/usr/sbin/munin21:45:211
61912599484,21cyclictest621758-21/usr/sbin/munin19:15:310
619125994818,5cyclictest665333-21cpuspeed_turbos21:05:190
159199480,3rtkit-daemon0-21swapper/120:49:310
159199480,3rtkit-daemon0-21swapper/022:26:031
619129994725,7cyclictest696869-21/usr/sbin/munin21:50:331
619129994725,6cyclictest627808-21cstates19:30:231
619129994723,6cyclictest673040-21sh21:17:091
619129994722,7cyclictest784547-21/usr/sbin/munin23:55:271
619129994721,10cyclictest767608-21pmu-power23:30:341
619129994719,12cyclictest666521-21unixbench_singl21:05:351
619129994717,7cyclictest752792-21/usr/sbin/munin23:10:311
619129994717,6cyclictest653011-21/usr/sbin/munin20:35:251
619129994712,8cyclictest650401-21unixbench_singl20:25:391
619129994712,11cyclictest682806-21timerwakeupswit21:30:381
619129994712,11cyclictest629634-21cpu19:35:211
61912599478,6cyclictest723849-21cpu22:30:210
61912599476,3cyclictest0-21swapper/023:59:460
619125994730,8cyclictest743233-21strings22:55:340
619125994723,8cyclictest768323-21ssh23:31:070
619125994720,12cyclictest733922-21latency_hist22:45:010
159199470,3rtkit-daemon0-21swapper/120:20:560
159199470,3rtkit-daemon0-21swapper/119:38:400
159199470,3rtkit-daemon0-21swapper/119:12:010
159199470,29rtkit-daemon0-21swapper/100:18:571
159199470,29rtkit-daemon0-21swapper/021:17:350
619129994624,5cyclictest793720-21sh00:08:051
619129994624,5cyclictest651873-21/usr/sbin/munin20:30:331
61912999462,3cyclictest0-21swapper/100:25:371
619129994623,7cyclictest657413-21/usr/sbin/munin20:45:281
619129994619,12cyclictest808999-21fschecks_count00:30:271
619129994618,7cyclictest789258-21unixbench-2d00:00:381
619129994614,11cyclictest640584-21unixbench_multi20:00:371
619129994613,15cyclictest676778-21sshd21:22:211
61912599462,3cyclictest0-21swapper/000:11:590
619125994620,14cyclictest780549-21sh23:50:020
619125994617,12cyclictest815396-21latency_hist00:40:010
619125994617,12cyclictest815396-21latency_hist00:40:010
619129994522,7cyclictest634678-21timerwakeupswit19:45:351
619129994519,6cyclictest667296-21sh21:10:141
619129994519,6cyclictest667296-21sh21:10:131
61912599459,3cyclictest0-21swapper/023:00:080
61912599459,3cyclictest0-21swapper/021:37:280
61912599457,2cyclictest0-21swapper/021:00:270
61912599453,5cyclictest759670-21cut23:20:010
619125994518,11cyclictest699946-21kernelversion21:55:310
619125994513,11cyclictest781079-21apt-get23:50:320
619125994512,5cyclictest729195-21sh22:36:430
619125994511,9cyclictest738825-21fschecks_count22:50:240
619125994511,9cyclictest738825-21fschecks_count22:50:240
159199450,3rtkit-daemon0-21swapper/022:28:320
61912999449,2cyclictest738531-21/usr/sbin/munin22:50:201
61912999449,2cyclictest738531-21/usr/sbin/munin22:50:201
619129994421,6cyclictest621758-21/usr/sbin/munin19:15:201
619125994320,12cyclictest623724-21/usr/sbin/munin19:20:340
619125994313,2cyclictest695893-21cut21:50:180
619125994311,6cyclictest776031-21ssh23:43:140
619125994310,3cyclictest0-21swapper/020:15:330
61912599430,2cyclictest651431-21grep20:30:270
61912599428,4cyclictest0-21swapper/022:15:200
61912599423,5cyclictest633756-21cut19:45:240
61912599422,2cyclictest0-21swapper/000:03:430
61912599420,2cyclictest653232-21df20:35:260
61912599417,4cyclictest0-21swapper/022:20:260
61912599417,4cyclictest0-21swapper/022:20:250
619125994115,9cyclictest635553-21cpuspeed_turbos19:50:240
61912599400,3cyclictest641529-21diskstats20:05:250
61912599398,5cyclictest639574-21sh20:00:230
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional