You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-10-01 - 01:58
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: reduced 
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8.osadl.org (updated Sat Sep 30, 2023 12:44:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
691173993932,5cyclictest858582-21diskstats09:40:232
691175993430,3cyclictest996126-21kworker/u8:2+flush-8:012:15:013
691168993229,2cyclictest863822-21gpgv09:45:011
691164993227,3cyclictest731921-21apt-get07:45:200
691164993227,3cyclictest4170283-21systemd-journal08:40:000
691164993128,2cyclictest880721-21kworker/u8:3+flush-8:010:10:200
691173993024,4cyclictest4170283-21systemd-journal09:40:002
691173992925,3cyclictest979745-21kworker/u8:0+flush-8:011:40:012
691173992925,3cyclictest775849-21kworker/u8:3+flush-8:009:15:012
691175992723,3cyclictest886385-21kworker/u8:2+flush-8:010:20:013
691168992721,4cyclictest4170283-21systemd-journal10:49:591
691168992721,4cyclictest1018560-21apt-get12:05:201
691175992622,3cyclictest775849-21kworker/u8:3+flush-8:008:44:593
691175992518,5cyclictest925045-21apt-get10:40:003
691173992522,3cyclictest709806-21kworker/u8:0+flush-8:008:10:272
691164992521,3cyclictest880721-21kworker/u8:3+flush-8:011:20:270
691175992418,4cyclictest852397-21apt-get09:35:013
691173992418,4cyclictest996358-21apt-get11:45:192
691173992418,4cyclictest743125-21appstreamcli07:55:022
691173992418,4cyclictest696476-21latency_hist07:15:012
691164992421,2cyclictest1001566-21kworker/u8:5+events_unbound12:10:300
691175992317,4cyclictest809261-21diskstats08:55:253
691173992318,3cyclictest891611-21latency_hist10:10:002
691168992321,1cyclictest1444-21dbus-daemon09:00:001
691168992319,3cyclictest775849-21kworker/u8:3+flush-8:008:25:351
691168992319,2cyclictest822896-21kthreadcore09:05:281
691168992316,5cyclictest764619-21latency_hist08:15:021
691164992319,2cyclictest1-21systemd12:10:010
691175992216,4cyclictest808560-21apt-get08:54:593
691173992219,2cyclictest1001566-21kworker/u8:5+flush-8:011:55:352
691173992218,3cyclictest886385-21kworker/u8:2+flush-8:011:15:342
691173992216,5cyclictest875478-21apt-get09:55:202
691173992216,4cyclictest958027-21latency_hist11:10:002
691168992219,2cyclictest1051011-21apt12:35:001
691168992216,4cyclictest4170283-21systemd-journal09:25:001
69116899220,2cyclictest977796-21kthreadcore11:25:311
691164992218,3cyclictest996126-21kworker/u8:2+flush-8:012:00:250
691164992218,3cyclictest880721-21kworker/u8:3+flush-8:011:05:260
691164992218,3cyclictest703904-21kworker/u8:4+flush-8:007:40:010
691175992116,3cyclictest703941-21apt-get07:20:013
691173992118,2cyclictest886385-21kworker/u8:2+events_unbound11:00:392
691173992117,3cyclictest1516-21rs:main2
69117399210,19cyclictest852446-21munin-run09:35:002
691168992119,1cyclictest0-21swapper/111:10:331
691168992118,2cyclictest696483-21grep07:15:011
691168992117,3cyclictest1002104-21/usr/sbin/munin11:50:321
691168992116,3cyclictest897167-21gpgv10:15:011
691164992118,2cyclictest886385-21kworker/u8:2+flush-8:010:25:350
691164992117,3cyclictest775849-21kworker/u8:3+events_unbound09:05:330
691164992117,3cyclictest1001566-21kworker/u8:5+flush-8:012:25:330
691175992017,2cyclictest792254-21systemd-run08:40:003
691175992015,3cyclictest814597-21apt-get09:00:223
691173992014,4cyclictest781551-21apt-get08:30:202
691168992019,1cyclictest0-21swapper/110:20:291
691168992016,3cyclictest979745-21kworker/u8:0+flush-8:011:30:251
691168992015,3cyclictest830637-21latency_hist09:15:011
69116899200,19cyclictest0-21swapper/108:05:261
69116899200,0cyclictest0-21swapper/107:55:281
691164992017,2cyclictest968921-21kworker/u8:1+events_unbound11:30:240
691164992017,1cyclictest952587-21kworker/0:0+kacpid11:10:240
691164992016,3cyclictest709806-21kworker/u8:0+events_unbound08:30:310
691164992016,3cyclictest1034632-21kworker/u8:0+events_unbound12:30:350
691164992015,3cyclictest886385-21kworker/u8:2+events_unbound10:45:010
69116499200,18cyclictest941733-21cron10:55:000
691175991917,1cyclictest748326-21cron08:00:003
691175991916,2cyclictest1034667-21cron12:20:003
691175991914,3cyclictest936545-21apt-get10:50:193
691173991916,2cyclictest936961-21irqcore10:50:252
691173991914,3cyclictest919488-21kworker/u8:1+flush-8:010:35:002
691168991918,1cyclictest0-21swapper/109:45:261
691168991918,1cyclictest0-21swapper/109:00:301
691168991917,1cyclictest790113-21grep08:35:261
691168991917,1cyclictest0-21swapper/111:48:321
691168991917,1cyclictest0-21swapper/111:40:251
691168991917,1cyclictest0-21swapper/110:50:131
691168991917,1cyclictest0-21swapper/107:20:021
691168991916,2cyclictest770216-21gpgv08:20:011
691168991916,2cyclictest703932-21awk07:20:001
691168991916,1cyclictest968577-21switchtime11:15:371
691168991915,3cyclictest968921-21kworker/u8:1+events_unbound12:40:011
691168991913,4cyclictest924977-21apt-get10:39:081
691168991913,4cyclictest4170283-21systemd-journal10:30:011
69116899190,1cyclictest0-21swapper/110:05:131
691164991915,3cyclictest919488-21kworker/u8:1+events_unbound10:35:010
691164991915,3cyclictest880721-21kworker/u8:3+flush-8:010:40:250
691164991915,2cyclictest699171-21irqcore07:15:250
69116499190,1cyclictest1007391-21idleruntime-cro11:55:010
691175991816,1cyclictest0-21swapper/308:50:013
691175991815,2cyclictest996126-21kworker/u8:2+events_unbound11:45:383
691175991815,2cyclictest886385-21kworker/u8:2+events_unbound10:05:263
691175991813,3cyclictest952873-21apt-get11:05:213
691173991814,3cyclictest830622-21kworker/u8:2+flush-8:009:15:322
691173991814,3cyclictest1034632-21kworker/u8:0+flush-8:012:30:252
691173991813,3cyclictest841537-21latency_hist09:25:002
691173991813,3cyclictest715501-21apt-get07:30:222
69117399181,15cyclictest1892-21sshd11:25:382
691168991817,1cyclictest0-21swapper/112:26:341
691168991817,1cyclictest0-21swapper/112:18:401
691168991817,1cyclictest0-21swapper/109:53:121
691168991817,0cyclictest0-21swapper/112:01:351
691168991817,0cyclictest0-21swapper/107:50:231
691168991816,1cyclictest675146-21kworker/u8:2+flush-8:007:25:311
691168991816,1cyclictest0-21swapper/110:33:301
691168991816,1cyclictest0-21swapper/109:55:171
691168991816,1cyclictest0-21swapper/109:25:251
691168991816,1cyclictest0-21swapper/107:48:351
691168991815,2cyclictest1008058-21irqcore11:55:261
69116899180,1cyclictest0-21swapper/111:39:031
69116899180,1cyclictest0-21swapper/107:30:011
69116899180,17cyclictest0-21swapper/111:24:471
69116899180,17cyclictest0-21swapper/110:58:231
69116899180,17cyclictest0-21swapper/108:34:451
69116899180,17cyclictest0-21swapper/107:35:051
69116899180,16cyclictest0-21swapper/108:45:071
691164991817,1cyclictest863875-21kworker/0:0+kacpid10:15:220
691164991817,1cyclictest764588-21kworker/0:0+kacpid09:10:300
691164991816,1cyclictest910293-21kworker/0:1+kacpid10:35:250
691164991814,3cyclictest675146-21kworker/u8:2+events_unbound08:20:240
691164991813,4cyclictest775849-21kworker/u8:3+flush-8:009:05:000
691164991811,5cyclictest720724-21latency_hist07:35:010
69116499180,16cyclictest979771-21turbostat11:29:590
691175991715,1cyclictest1478-21polkitd08:20:023
691175991715,1cyclictest1013972-21irqrtprio12:00:263
691175991712,3cyclictest969253-21diskstats11:20:233
691175991710,5cyclictest830285-21python309:10:323
691173991715,1cyclictest0-21swapper/211:30:382
691173991713,3cyclictest675146-21kworker/u8:2+flush-8:007:45:312
691173991712,3cyclictest803385-21apt-get08:50:212
691173991712,3cyclictest4170283-21systemd-journal12:21:372
691173991711,4cyclictest720983-21apt-get07:35:202
691168991716,1cyclictest0-21swapper/112:11:411
691168991716,1cyclictest0-21swapper/110:18:071
691168991716,1cyclictest0-21swapper/109:35:191
691168991716,1cyclictest0-21swapper/108:02:061
691168991716,0cyclictest0-21swapper/109:31:221
691168991715,1cyclictest0-21swapper/111:06:461
691168991715,1cyclictest0-21swapper/111:00:431
691168991714,2cyclictest1001566-21kworker/u8:5+flush-8:012:20:321
69116899171,14cyclictest880980-21apt-get10:00:231
69116899170,1cyclictest0-21swapper/110:42:561
69116899170,17cyclictest0-21swapper/108:44:461
69116899170,16cyclictest0-21swapper/109:16:321
69116899170,0cyclictest0-21swapper/108:22:521
69116499176,9cyclictest858179-21kworker/u8:1+flush-8:009:50:280
691164991717,0cyclictest748385-21kworker/0:1+kacpid08:10:210
691164991715,1cyclictest863875-21kworker/0:0+kacpid09:45:280
691164991715,1cyclictest844038-21kworker/0:2+kacpid09:40:300
691164991714,2cyclictest830622-21kworker/u8:2+events_unbound09:25:220
691164991714,2cyclictest775849-21kworker/u8:3+events_unbound09:35:230
691164991714,2cyclictest715231-21cron07:30:000
691164991713,3cyclictest880721-21kworker/u8:3+flush-8:010:20:280
691164991713,3cyclictest797709-21kworker/u8:0+events_unbound08:55:230
691164991712,4cyclictest880721-21kworker/u8:3+flush-8:010:55:200
69116499170,15cyclictest709782-21munin-run07:25:010
691175991614,1cyclictest726247-21systemd-run07:40:003
691175991613,2cyclictest948326-21irqrtprio11:00:303
691175991613,2cyclictest926968-21kthreadcore10:40:333
691175991613,2cyclictest759832-21irqcore08:10:263
691175991610,4cyclictest1051010-21latency_hist12:35:003
691173991613,2cyclictest902491-21kworker/u8:0+flush-8:010:39:082
691173991613,2cyclictest709806-21kworker/u8:0+events_unbound08:25:262
691173991612,3cyclictest737070-21kworker/u8:1+events_unbound07:50:022
69116499169,4cyclictest742559-21apt-get07:55:010
691164991615,1cyclictest863875-21kworker/0:0+kacpid09:55:220
691164991614,1cyclictest974361-21kworker/0:3+kacpid12:15:220
691164991614,1cyclictest968921-21kworker/u8:1+events_unbound11:40:340
691164991614,1cyclictest910293-21kworker/0:1+kacpid11:00:240
691164991614,1cyclictest764588-21kworker/0:0+kacpid08:40:330
691164991613,2cyclictest830622-21kworker/u8:2+flush-8:009:15:280
691164991613,2cyclictest775849-21kworker/u8:3+events_unbound09:30:280
691164991613,2cyclictest1012805-21cron12:00:000
691164991612,3cyclictest968921-21kworker/u8:1+events_unbound11:50:010
691164991612,2cyclictest0-21swapper/007:55:020
691175991512,2cyclictest995682-21systemctl11:40:373
691175991512,2cyclictest1052135-21irqrtprio12:35:263
691175991511,2cyclictest0-21swapper/310:15:023
69117599150,13cyclictest841700-21sed09:25:003
691173991513,1cyclictest902844-21cron10:20:002
691173991512,2cyclictest908318-21cut10:25:002
691173991512,2cyclictest737070-21kworker/u8:1+flush-8:008:05:312
691173991512,2cyclictest675146-21kworker/u8:2+events_unbound07:25:332
691173991512,2cyclictest0-21swapper/207:20:002
691168991514,1cyclictest0-21swapper/108:51:301
69116499151,8cyclictest891616-21turbostat10:10:010
691164991511,3cyclictest690939-21kworker/u8:0+flush-8:007:10:200
691164991511,3cyclictest1001566-21kworker/u8:5+events_unbound12:35:230
69117599148,5cyclictest963482-21true11:15:003
691175991412,1cyclictest0-21swapper/309:50:013
691175991411,2cyclictest974864-21gltestperf11:25:263
691175991410,2cyclictest696587-21systemctl07:15:013
691175991410,2cyclictest0-21swapper/310:35:013
69117399144,5cyclictest1056507-21plymouth12:40:002
691173991411,2cyclictest958589-21gltestperf11:10:292
691173991411,2cyclictest786725-21kworker/u8:1+events_unbound08:45:232
691173991411,2cyclictest703904-21kworker/u8:4+flush-8:007:40:262
691173991411,2cyclictest0-21swapper/211:00:002
691173991410,3cyclictest797709-21kworker/u8:0+flush-8:009:00:292
69116899148,4cyclictest4170283-21systemd-journal07:43:391
691164991412,1cyclictest764588-21kworker/0:0+kacpid08:45:200
691164991411,2cyclictest808557-21cron08:55:000
691164991410,3cyclictest979745-21kworker/u8:0+events_unbound11:35:240
69117599138,2cyclictest906838-21kthreadcore10:20:273
691175991311,1cyclictest0-21swapper/312:05:213
691175991311,1cyclictest0-21swapper/309:55:003
691175991310,2cyclictest0-21swapper/307:55:003
691173991311,1cyclictest0-21swapper/211:45:012
691173991311,1cyclictest0-21swapper/210:42:472
691173991310,2cyclictest775849-21kworker/u8:3+events_unbound08:40:292
691173991310,2cyclictest770346-21apt-key08:20:022
691173991310,1cyclictest1029205-21cron12:15:012
69116499139,3cyclictest709806-21kworker/u8:0+flush-8:008:15:290
69117599128,3cyclictest770082-21apt-get08:20:023
69117599127,3cyclictest0-21swapper/310:30:013
69117599121,3cyclictest886158-21latency_hist10:05:013
691175991210,1cyclictest0-21swapper/309:05:283
69117399129,2cyclictest775849-21kworker/u8:3+flush-8:008:24:082
69117399128,3cyclictest931088-21apt-get10:45:232
69117399128,3cyclictest1001566-21kworker/u8:5+events_unbound12:25:242
691173991210,1cyclictest0-21swapper/211:50:262
69116499128,3cyclictest737070-21kworker/u8:1+events_unbound08:05:220
691164991211,1cyclictest974361-21kworker/0:3+kacpid12:20:200
691164991210,1cyclictest0-21swapper/010:00:300
69117599119,1cyclictest0-21swapper/312:20:213
69117599118,2cyclictest715799-21gltestperf07:30:263
69117599118,2cyclictest0-21swapper/311:50:353
69117599118,1cyclictest0-21swapper/312:30:013
69117599117,3cyclictest980215-21grep11:30:253
69117599117,3cyclictest832220-21tr09:15:283
69117599117,2cyclictest0-21swapper/307:40:333
69117599116,4cyclictest786750-21latency_hist08:35:013
69117599116,3cyclictest852397-21apt-get09:35:023
69117599116,3cyclictest732106-21tune2fs07:45:243
69117599110,9cyclictest990688-21systemd-run11:40:003
69117399118,1cyclictest0-21swapper/211:20:232
69117399117,3cyclictest968921-21kworker/u8:1+events_unbound12:15:282
69117399110,10cyclictest0-21swapper/208:01:452
69116499118,2cyclictest731652-21cron07:45:000
69116499117,3cyclictest737070-21kworker/u8:1+events_unbound08:00:250
69116499111,9cyclictest0-21swapper/011:15:370
691164991111,0cyclictest764588-21kworker/0:0+kacpid08:25:320
69117599108,1cyclictest0-21swapper/308:00:273
69117599107,1cyclictest0-21swapper/307:25:353
69117599106,2cyclictest1012831-21apt12:00:013
69117599100,4cyclictest864083-21cat09:45:013
69117599100,3cyclictest969073-21cat11:20:003
69117599100,2cyclictest947188-21latency_hist11:00:013
69117599100,2cyclictest846959-21latency_hist09:30:003
69117599100,2cyclictest709776-21latency_hist07:25:003
69117399108,1cyclictest381rcuc/210:10:282
69117399108,1cyclictest0-21swapper/208:35:222
69117399106,3cyclictest863709-21apt-get09:45:022
69117399106,3cyclictest820056-21apt-get09:05:202
6911759995,2cyclictest690961-21latency_hist07:10:003
6911759994,3cyclictest931194-21/usr/sbin/munin10:45:313
6911739998,1cyclictest0-21swapper/207:23:302
6911739996,1cyclictest0-21swapper/212:05:322
6911739995,3cyclictest809023-21cat08:55:012
6911739994,3cyclictest1444-21dbus-daemon09:25:332
6911689994,3cyclictest690961-21latency_hist07:10:011
6911759985,2cyclictest17250irq/158-ahci[0000:00:17.0]08:25:243
6911759981,4cyclictest875446-21/usr/sbin/munin09:55:253
6911739986,1cyclictest0-21swapper/210:25:342
6911739985,2cyclictest881396-21irqcore10:00:272
6911739984,3cyclictest391ktimers/212:00:202
6911649982,3cyclictest836362-21apt-get09:20:210
6911759971,4cyclictest753770-21latency_hist08:05:013
6911649971,4cyclictest691166-21ls07:10:010
6911739953,1cyclictest690961-21latency_hist07:10:002
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional