You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-22 - 06:32
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: reduced 
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8.osadl.org (updated Fri Sep 22, 2023 00:44:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2624205994744,2cyclictest2938329-21kworker/u8:3+events_unbound00:09:103
2624200994326,4cyclictest2845394-21latency_hist22:30:001
2624196994133,5cyclictest2954865-21latency_hist00:10:010
2624196994033,4cyclictest2894365-21latency_hist23:15:000
2624205993733,3cyclictest2938329-21kworker/u8:3+events_unbound00:15:343
2624196993529,4cyclictest4170283-21systemd-journal20:45:300
2624205993431,2cyclictest2691756-21kworker/u8:0+flush-8:020:30:293
2624196993430,3cyclictest2845418-21kworker/u8:2+flush-8:022:35:300
2624205993330,2cyclictest2807551-21kworker/u8:0+flush-8:022:00:323
2624205993328,3cyclictest2680914-21kworker/u8:1+flush-8:021:20:003
2624200993327,4cyclictest2883503-21latency_hist23:05:011
2624196993327,4cyclictest4170283-21systemd-journal21:00:300
2624205993229,2cyclictest2670000-21kworker/u8:3+flush-8:020:20:293
2624205993228,3cyclictest2735893-21kworker/u8:3+flush-8:021:30:003
2624205993227,3cyclictest2927117-21latency_hist23:45:003
2624196993227,3cyclictest2790515-21apt-get21:40:030
2624196993225,4cyclictest2729951-21latency_hist20:45:010
2624205993124,3cyclictest2807551-21kworker/u8:0+events_unbound22:05:293
2624205993026,3cyclictest2878055-21kworker/u8:1+flush-8:023:35:003
2624200993023,5cyclictest2927685-21appstreamcli23:45:021
2624205992922,5cyclictest2801788-21apt-get21:47:293
2624200992926,2cyclictest2938613-21apt-get23:55:181
2624205992823,4cyclictest2680914-21kworker/u8:1+events_unbound20:40:303
2624200992822,4cyclictest2790515-21apt-get21:40:001
2624205992723,3cyclictest2670000-21kworker/u8:3+events_unbound19:55:313
2624205992719,2cyclictest2680914-21kworker/u8:1+events_unbound20:15:003
2624202992721,4cyclictest2642443-21latency_hist19:25:012
2624196992723,3cyclictest2670000-21kworker/u8:3+events_unbound19:55:300
2624200992622,3cyclictest2911003-21apt-get23:30:191
2624200992620,4cyclictest2643013-21appstreamcli19:25:011
2624202992519,4cyclictest2664799-21apt-get19:45:212
2624196992523,1cyclictest2829055-21kworker/u8:1+flush-8:022:30:350
2624196992520,3cyclictest2922340-21kworker/u8:0+flush-8:000:00:000
2624205992420,3cyclictest2730180-21kworker/u8:0+events_unbound21:20:283
2624205992420,3cyclictest2607870-21kworker/u8:1+events_unbound19:40:243
2624205992419,3cyclictest2718991-21kworker/u8:4+flush-8:020:35:293
2624205992418,4cyclictest4170283-21systemd-journal22:00:003
2624205992418,4cyclictest2680914-21kworker/u8:1+flush-8:021:10:263
2624202992410,12cyclictest2648241-21cron19:30:002
2624200992420,3cyclictest2713839-21apt-get20:30:201
2624196992420,3cyclictest2944085-21kworker/u8:4+flush-8:000:20:230
2624205992319,3cyclictest2960723-21kworker/u8:0+flush-8:000:20:383
2624205992319,3cyclictest2944085-21kworker/u8:4+flush-8:000:30:263
2624205992317,4cyclictest2839632-21python322:20:323
2624202992315,6cyclictest2675404-21latency_hist19:55:012
2624200992319,2cyclictest4170283-21systemd-journal23:55:001
262420099230,21cyclictest0-21swapper/121:45:291
2624196992320,2cyclictest2845385-21cron22:29:590
2624196992319,3cyclictest2691756-21kworker/u8:0+flush-8:020:25:320
2624196992318,3cyclictest4170283-21systemd-journal23:10:000
2624196992318,3cyclictest2746592-21latency_hist21:00:000
2624205992218,3cyclictest2888919-21kworker/u8:2+flush-8:023:35:433
2624205992218,3cyclictest2845418-21kworker/u8:2+flush-8:022:41:343
2624205992218,3cyclictest2680914-21kworker/u8:1+events_unbound21:00:013
2624202992219,2cyclictest1-21systemd00:00:012
2624202992218,3cyclictest4170283-21systemd-journal21:50:012
2624200992218,2cyclictest0-21swapper/119:20:011
2624200992217,3cyclictest2899812-21latency_hist23:20:011
2624196992219,2cyclictest2899848-21systemd-run23:20:010
2624196992219,2cyclictest2607870-21kworker/u8:1+flush-8:019:25:000
2624196992218,3cyclictest2735893-21kworker/u8:3+flush-8:021:20:200
262419699220,15cyclictest2669982-21awk19:50:000
2624205992118,2cyclictest2680914-21kworker/u8:1+events_unbound20:10:013
2624205992118,2cyclictest2629515-21kworker/u8:3+events_unbound19:20:253
2624205992117,3cyclictest2922340-21kworker/u8:0+flush-8:023:45:333
262420299217,4cyclictest2971593-21latency_hist00:25:012
262420299217,2cyclictest2729981-21systemd-run20:45:002
2624202992118,2cyclictest2768741-21sendmail21:20:002
2624200992119,1cyclictest0-21swapper/119:55:241
2624200992114,5cyclictest4170283-21systemd-journal19:35:011
2624200992114,5cyclictest2921635-21latency_hist23:40:001
2624196992117,2cyclictest2814419-21kworker/u8:3+flush-8:022:10:240
2624196992115,4cyclictest2790513-21latency_hist21:40:010
2624205992017,2cyclictest2814419-21kworker/u8:3+flush-8:022:15:313
2624205992017,2cyclictest2814419-21kworker/u8:3+events_unbound22:25:223
2624205992017,2cyclictest2779845-21kworker/u8:4+events_unbound21:35:313
2624205992016,3cyclictest2944085-21kworker/u8:4+flush-8:000:25:293
2624205992016,3cyclictest2691756-21kworker/u8:0+flush-8:020:25:203
2624205992016,3cyclictest2607870-21kworker/u8:1+events_unbound19:25:283
2624205992015,2cyclictest2735893-21kworker/u8:3+events_unbound21:50:303
2624200992018,1cyclictest0-21swapper/120:25:241
2624200992010,7cyclictest2746584-21apt-get21:00:001
262420099200,2cyclictest2861702-21idleruntime-cro22:44:591
262420099200,19cyclictest2866560-21sed22:45:261
262420099200,19cyclictest2722619-21kthreadcore20:35:241
262420099200,19cyclictest2691992-21apt-get20:10:181
262420099200,19cyclictest0-21swapper/121:15:211
2624196992017,2cyclictest2856195-21kworker/u8:0+events_unbound22:55:000
2624196992017,2cyclictest2807351-21systemd-run21:55:000
2624196992016,3cyclictest2888919-21kworker/u8:2+events_unbound23:50:220
2624196992016,3cyclictest2861922-21kworker/u8:3+events_unbound22:45:200
2624196992015,3cyclictest2888919-21kworker/u8:2+flush-8:023:35:250
2624205991916,2cyclictest2691756-21kworker/u8:0+flush-8:020:15:303
2624205991915,3cyclictest2867378-21kworker/u8:5+flush-8:023:00:423
2624205991915,3cyclictest2735893-21kworker/u8:3+flush-8:021:05:323
2624205991915,3cyclictest2670000-21kworker/u8:3+events_unbound20:00:313
2624202991916,2cyclictest1444-21dbus-daemon22:00:002
2624200991917,1cyclictest0-21swapper/123:14:021
2624200991917,1cyclictest0-21swapper/100:20:211
2624200991914,3cyclictest2905546-21apt-get23:25:191
2624200991913,4cyclictest2883761-21apt-get23:05:181
2624200991913,4cyclictest2813485-21ps22:00:251
262420099190,18cyclictest0-21swapper/122:07:311
262420099190,18cyclictest0-21swapper/120:22:221
262420099190,17cyclictest0-21swapper/122:55:151
262420099190,17cyclictest0-21swapper/120:45:131
262420099190,17cyclictest0-21swapper/100:15:101
2624196991917,1cyclictest2680916-21kworker/0:2+kacpid21:15:210
2624196991915,3cyclictest2944085-21kworker/u8:4+flush-8:000:25:370
262419699190,17cyclictest2659092-21munin-run19:40:000
2624205991815,2cyclictest0-21swapper/321:00:003
2624205991814,3cyclictest2960723-21kworker/u8:0+flush-8:000:35:273
2624205991814,3cyclictest2867377-21kworker/u8:4+flush-8:022:55:253
2624200991816,1cyclictest0-21swapper/122:30:131
2624200991816,1cyclictest0-21swapper/122:10:471
2624200991816,1cyclictest0-21swapper/121:55:101
2624200991816,1cyclictest0-21swapper/121:32:501
2624200991816,1cyclictest0-21swapper/121:01:171
2624200991816,1cyclictest0-21swapper/120:18:421
2624200991816,1cyclictest0-21swapper/119:52:261
2624200991816,1cyclictest0-21swapper/119:49:261
2624200991816,1cyclictest0-21swapper/119:38:571
2624200991816,1cyclictest0-21swapper/100:25:251
2624200991816,1cyclictest0-21swapper/100:12:321
2624200991814,2cyclictest0-21swapper/121:25:001
2624200991813,3cyclictest4170283-21systemd-journal22:39:281
262420099180,18cyclictest0-21swapper/122:22:371
262420099180,17cyclictest0-21swapper/100:37:401
2624196991817,1cyclictest2938328-21kworker/0:1+kacpid00:10:220
2624196991816,1cyclictest2629656-21kworker/0:1+kacpid19:25:240
2624196991815,2cyclictest2938328-21kworker/0:1+kacpid00:16:280
2624196991815,2cyclictest2801824-21kworker/u8:2+flush-8:021:55:340
2624196991815,2cyclictest2670000-21kworker/u8:3+events_unbound20:00:330
2624196991814,3cyclictest2888919-21kworker/u8:2+events_unbound23:25:220
2624196991814,3cyclictest2691756-21kworker/u8:0+flush-8:020:20:300
2624205991714,2cyclictest2894370-21kworker/u8:3+events_unbound23:20:213
2624205991713,3cyclictest2878055-21kworker/u8:1+events_unbound23:15:203
2624205991713,3cyclictest2856195-21kworker/u8:0+events_unbound23:10:333
2624202991715,1cyclictest0-21swapper/221:40:012
2624202991714,2cyclictest0-21swapper/222:40:002
2624202991713,3cyclictest2718991-21kworker/u8:4+flush-8:020:35:002
2624200991716,1cyclictest0-21swapper/123:40:151
2624200991716,1cyclictest0-21swapper/122:17:061
2624200991716,1cyclictest0-21swapper/100:05:531
2624200991715,1cyclictest2637277-21apt-get19:20:211
2624200991714,2cyclictest2659783-21irqcore19:40:241
262420099170,17cyclictest0-21swapper/122:51:511
262420099170,17cyclictest0-21swapper/121:51:471
262420099170,17cyclictest0-21swapper/119:12:041
262420099170,16cyclictest0-21swapper/100:02:551
262420099170,0cyclictest0-21swapper/123:21:521
262420099170,0cyclictest0-21swapper/120:54:111
262420099170,0cyclictest0-21swapper/120:40:041
262420099170,0cyclictest0-21swapper/120:08:351
262420099170,0cyclictest0-21swapper/120:00:021
2624196991714,2cyclictest2607870-21kworker/u8:1+flush-8:019:40:220
2624196991713,3cyclictest2779845-21kworker/u8:4+events_unbound21:30:300
2624196991712,3cyclictest2814419-21kworker/u8:3+flush-8:022:20:270
2624205991612,3cyclictest2938329-21kworker/u8:3+flush-8:023:55:203
2624205991612,3cyclictest2938329-21kworker/u8:3+flush-8:000:10:313
2624205991612,3cyclictest2905513-21kworker/u8:0+events_unbound23:28:303
2624205991612,3cyclictest2888919-21kworker/u8:2+events_unbound23:50:273
2624205991612,3cyclictest2607870-21kworker/u8:1+events_unbound19:32:303
2624202991613,2cyclictest2791551-21irqcore21:40:242
2624202991613,2cyclictest2632225-21irqcore19:15:262
2624202991613,1cyclictest0-21swapper/200:39:592
2624202991612,3cyclictest2960723-21kworker/u8:0+flush-8:000:30:002
2624202991610,4cyclictest4170283-21systemd-journal21:53:292
2624202991610,4cyclictest2900086-21apt-get23:20:192
262420099160,16cyclictest0-21swapper/121:26:591
262420099160,16cyclictest0-21swapper/121:09:581
262420099160,0cyclictest0-21swapper/121:42:091
262420099160,0cyclictest0-21swapper/121:10:521
2624196991613,2cyclictest2867378-21kworker/u8:5+flush-8:023:00:420
2624196991612,3cyclictest2680914-21kworker/u8:1+flush-8:020:15:240
2624196991612,3cyclictest2680914-21kworker/u8:1+events_unbound20:50:190
2624196991611,3cyclictest2752670-21apt-get21:05:180
2624205991512,2cyclictest2779845-21kworker/u8:4+events_unbound21:34:563
2624205991512,2cyclictest2734848-21needreboot20:45:293
2624205991512,2cyclictest2613525-21kworker/u8:0+flush-8:019:15:273
2624205991511,2cyclictest0-21swapper/319:10:213
2624202991513,1cyclictest2977743-21irqcore00:30:312
2624202991512,2cyclictest2780305-21irqcore21:30:262
2624202991510,3cyclictest2878068-21latency_hist23:00:002
2624196991511,3cyclictest2927126-21cut23:45:000
2624196991511,3cyclictest2905513-21kworker/u8:0+flush-8:023:30:390
2624196991511,3cyclictest2691756-21kworker/u8:0+events_unbound20:10:250
2624196991510,4cyclictest2607870-21kworker/u8:1+events_unbound19:30:290
2624205991412,2cyclictest2653685-21kworker/u8:2+events_unbound19:50:033
2624205991411,2cyclictest2944085-21kworker/u8:4+events_unbound00:04:313
2624205991411,2cyclictest2856195-21kworker/u8:0+events_unbound22:47:413
2624205991411,2cyclictest2823941-21apt-get22:10:203
2624205991410,3cyclictest2856195-21kworker/u8:0+events_unbound23:05:203
2624202991412,1cyclictest2680861-21cron19:59:592
2624202991411,2cyclictest2846068-21irqcore22:30:262
2624202991411,2cyclictest2839977-21cron22:25:002
2624202991410,3cyclictest2680914-21kworker/u8:1+events_unbound21:10:262
2624202991410,3cyclictest2607870-21kworker/u8:1+flush-8:019:30:322
2624202991410,2cyclictest0-21swapper/222:19:592
262419699147,1cyclictest2675414-21turbostat19:55:000
2624196991413,1cyclictest2829044-21kworker/0:1+kacpid22:40:190
2624196991411,2cyclictest2829759-21irqcore22:15:230
2624196991410,3cyclictest2625193-21ps19:10:220
2624196991410,2cyclictest2867378-21kworker/u8:5+flush-8:023:00:000
262420599139,3cyclictest2845418-21kworker/u8:2+events_unbound22:35:323
2624205991310,2cyclictest0-21swapper/319:40:013
262420299134,4cyclictest2938357-21munin-run23:55:012
2624202991310,2cyclictest0-21swapper/220:25:262
2624202991310,1cyclictest2755458-21tail21:05:262
2624200991313,0cyclictest0-21swapper/100:33:091
2624196991312,1cyclictest2938328-21kworker/0:1+kacpid00:00:420
2624196991312,1cyclictest2916415-21kworker/0:2+kacpid23:45:200
2624196991310,2cyclictest2735893-21kworker/u8:3+events_unbound21:25:350
2624196991310,2cyclictest2720150-21irqrtprio20:35:220
262420599129,2cyclictest2741007-21apt-key20:55:013
262420299129,2cyclictest2861684-21cron22:44:592
262420299129,2cyclictest2686298-21ls20:05:002
262420299129,2cyclictest1532-21gdbus23:30:002
262420299129,1cyclictest0-21swapper/223:45:002
262420299126,4cyclictest1532-21gdbus20:40:012
2624202991210,1cyclictest2917737-21kthreadcore23:35:342
2624202991210,1cyclictest2821276-21tr22:05:262
262419699129,2cyclictest2680914-21kworker/u8:1+flush-8:020:05:290
262419699128,3cyclictest2629515-21kworker/u8:3+events_unbound19:15:010
2624196991210,1cyclictest2796328-21kworker/0:1+kacpid21:45:280
262420599119,2cyclictest2730180-21kworker/u8:0+events_unbound21:42:303
262420299119,1cyclictest2691735-21latency_hist20:10:012
262420299118,2cyclictest2960169-21needreboot00:10:332
262420299118,2cyclictest0-21swapper/220:55:012
262420299117,3cyclictest1532-21gdbus22:15:012
262420299114,5cyclictest1532-21gdbus23:50:012
262419699118,2cyclictest2894370-21kworker/u8:3+events_unbound23:20:280
262419699117,3cyclictest2960723-21kworker/u8:0+events_unbound00:30:270
262419699117,3cyclictest2680914-21kworker/u8:1+events_unbound21:10:200
262420599106,3cyclictest2664301-21kworker/u8:0+events_unbound19:47:133
262420299108,1cyclictest2748084-21irqrtprio21:00:232
262420299108,1cyclictest0-21swapper/222:45:232
262420299107,2cyclictest4179814-21systemd-udevd23:12:322
262420299107,2cyclictest2911418-21irqcore23:30:272
262420299105,3cyclictest4170283-21systemd-journal19:40:002
262420299105,3cyclictest2872610-21sshd22:52:302
262420299105,3cyclictest1532-21gdbus21:00:002
262420299103,5cyclictest1532-21gdbus23:05:002
2624202991010,0cyclictest2834559-21kworker/2:3+tpm_dev_wq00:09:082
262419699109,0cyclictest2801827-21kworker/0:0+kacpid22:00:230
262419699108,1cyclictest2938328-21kworker/0:1+kacpid00:35:230
262419699108,1cyclictest0-21swapper/022:05:280
2624196991010,0cyclictest2680916-21kworker/0:2+kacpid20:30:210
41339890,5rtkit-daemon3074285-21ntpd23:05:192
26242059996,2cyclictest17250irq/158-ahci[0000:00:17.0]22:55:013
26242059992,4cyclictest2845649-21apt-get22:30:213
26242029997,1cyclictest0-21swapper/221:25:232
26242029997,1cyclictest0-21swapper/220:10:322
26242029997,1cyclictest0-21swapper/219:40:302
26242029996,2cyclictest2735130-21grep20:45:322
26242029995,3cyclictest2961173-21fw_packets00:15:232
26242029995,2cyclictest2899705-21tail23:15:332
26242029994,3cyclictest2702619-21sshd20:17:202
26242029993,4cyclictest1532-21gdbus22:30:002
26242009993,4cyclictest2624219-21grep19:10:011
26242029986,1cyclictest0-21swapper/219:10:262
26242029985,2cyclictest391ktimers/200:00:212
26242029984,2cyclictest2769352-21/usr/sbin/munin21:20:212
26242029981,5cyclictest2705682-21kthreadcore20:20:252
41339870,3rtkit-daemon0-21swapper/219:05:122
26242029972,3cyclictest2813062-21apt-get22:00:192
26241969975,1cyclictest0-21swapper/019:10:010
26242059960,4cyclictest2624008-21latency_hist19:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional