You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-05 - 04:52
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack0slot2.osadl.org (updated Mon May 05, 2025 00:43:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
55175426555,7sleep40-21swapper/403:34:254
55181025848,7sleep50-21swapper/503:35:145
55181125442,8sleep60-21swapper/603:35:156
55173724914,31sleep70-21swapper/703:34:127
552236993835,2cyclictest1039424-21kworker/u16:1+flush-8:008:32:225
552232993532,2cyclictest628148-21kworker/u16:2+events_unbound05:02:174
552242993227,3cyclictest864019-21grep06:02:297
552236993230,2cyclictest843118-21kworker/u16:5+events_unbound06:07:355
552237993026,3cyclictest782390-21kworker/u16:0+flush-8:005:32:226
552236993028,2cyclictest858769-21kworker/u16:2+flush-8:007:31:095
552236993028,2cyclictest616819-21kworker/u16:4+flush-8:004:42:335
552236993028,2cyclictest1080400-21kworker/u16:0+events_unbound08:09:095
552232993028,2cyclictest1153086-21kworker/u16:3+flush-8:008:41:594
552236992927,2cyclictest858769-21kworker/u16:2+events_unbound07:46:095
552236992927,2cyclictest700152-21kworker/u16:0+flush-8:004:54:095
552236992927,2cyclictest1026546-21kworker/u16:0+events_unbound07:40:085
552232992927,2cyclictest503099-21kworker/u16:0+flush-8:003:52:024
552232992926,2cyclictest700152-21kworker/u16:0+events_unbound04:57:014
55224299280,26cyclictest1018778-21latency_hist07:22:007
552236992826,2cyclictest843118-21kworker/u16:5+events_unbound06:31:145
552236992826,2cyclictest793379-21kworker/u16:4+events_unbound05:50:105
552236992826,2cyclictest503099-21kworker/u16:0+events_unbound03:59:105
552236992725,2cyclictest937292-21kworker/u16:0+events_unbound06:53:085
552236992725,2cyclictest937292-21kworker/u16:0+events_unbound06:53:085
552236992725,2cyclictest843118-21kworker/u16:5+events_unbound06:13:145
552236992725,2cyclictest751670-21kworker/u16:4+events_unbound05:16:145
552236992725,2cyclictest711051-21kworker/u16:3+flush-8:005:27:205
552236992725,2cyclictest628148-21kworker/u16:2+events_unbound04:58:095
552236992725,2cyclictest1203828-21kworker/u16:0+events_unbound09:02:305
552232992725,2cyclictest858769-21kworker/u16:2+events_unbound07:46:544
552232992724,2cyclictest858769-21kworker/u16:2+events_unbound07:27:594
55224299260,24cyclictest597193-21latency_hist03:57:007
552236992624,2cyclictest858769-21kworker/u16:2+events_unbound07:08:535
552236992624,2cyclictest607119-21kworker/u16:3+events_unbound04:34:095
552236992624,2cyclictest607119-21kworker/u16:3+events_unbound04:10:105
552236992623,2cyclictest916739-21kworker/u16:1+flush-8:006:37:105
552232992624,2cyclictest1069757-21kworker/u16:3+events_unbound07:57:124
552232992624,2cyclictest1039424-21kworker/u16:1+events_unbound07:32:134
552242992523,1cyclictest0-21swapper/705:27:297
55224299250,23cyclictest730856-21latency_hist05:02:007
552237992523,2cyclictest925382-21kworker/u16:4+flush-8:006:46:596
552237992523,2cyclictest628148-21kworker/u16:2+flush-8:005:52:236
552237992521,3cyclictest894314-21kworker/u16:3+events_unbound06:32:216
552236992523,2cyclictest925382-21kworker/u16:4+events_unbound06:48:525
552236992523,2cyclictest858769-21kworker/u16:2+events_unbound07:15:095
552236992523,2cyclictest1177845-21kworker/u16:0+events_unbound08:41:415
552236992522,2cyclictest782390-21kworker/u16:0+events_unbound05:57:245
552232992523,2cyclictest616819-21kworker/u16:4+events_unbound04:12:234
552232992523,2cyclictest1039424-21kworker/u16:1+events_unbound08:32:014
552232992522,2cyclictest1120604-21kworker/u16:5+flush-8:008:12:254
552242992423,0cyclictest0-21swapper/703:47:307
552237992422,2cyclictest1039424-21kworker/u16:1+events_unbound07:42:246
552236992422,2cyclictest711051-21kworker/u16:3+events_unbound05:04:105
552236992422,2cyclictest628148-21kworker/u16:2+flush-8:004:15:145
552236992422,2cyclictest1069757-21kworker/u16:3+events_unbound07:55:105
552236992421,2cyclictest1080400-21kworker/u16:0+events_unbound08:16:255
552237992320,3cyclictest925382-21kworker/u16:4+events_unbound06:47:126
552237992320,2cyclictest858769-21kworker/u16:2+flush-8:007:07:106
552236992321,2cyclictest843118-21kworker/u16:5+events_unbound06:21:105
552236992321,2cyclictest782390-21kworker/u16:0+events_unbound05:38:135
552236992321,2cyclictest503099-21kworker/u16:0+flush-8:004:05:105
552236992321,2cyclictest419995-21kworker/u16:3+events_unbound03:46:225
552236992321,2cyclictest419995-21kworker/u16:3+events_unbound03:46:225
552236992321,2cyclictest1203828-21kworker/u16:0+events_unbound08:54:085
552236992320,2cyclictest858769-21kworker/u16:2+events_unbound06:42:235
552236992320,2cyclictest1069757-21kworker/u16:3+events_unbound07:57:085
55224299220,20cyclictest643-21systemd-journal06:51:597
552237992220,2cyclictest751670-21kworker/u16:4+events_unbound05:12:176
552237992220,2cyclictest1151406-21kworker/u16:2+flush-8:008:42:246
552237992219,2cyclictest782390-21kworker/u16:0+events_unbound06:17:186
552237992219,2cyclictest616819-21kworker/u16:4+events_unbound04:37:186
552236992220,2cyclictest997937-21kworker/u16:5+events_unbound07:25:095
552236992220,2cyclictest688675-21kworker/u16:1+events_unbound04:41:095
552236992219,2cyclictest967138-21kworker/u16:1+flush-8:006:57:125
552236992219,2cyclictest843118-21kworker/u16:5+events_unbound06:24:105
552236992219,2cyclictest782390-21kworker/u16:0+events_unbound05:32:095
552236992219,2cyclictest1120604-21kworker/u16:5+flush-8:008:57:045
552232992220,2cyclictest628148-21kworker/u16:2+flush-8:005:31:544
552232992220,2cyclictest1080400-21kworker/u16:0+events_unbound08:07:024
552232992219,2cyclictest997937-21kworker/u16:5+flush-8:007:37:144
552242992118,2cyclictest720581-21munin-run04:57:007
552237992119,2cyclictest782390-21kworker/u16:0+flush-8:006:11:546
552237992118,2cyclictest711051-21kworker/u16:3+flush-8:005:07:216
552237992117,3cyclictest1120604-21kworker/u16:5+events_unbound08:17:236
552236992119,2cyclictest793379-21kworker/u16:4+events_unbound05:43:235
552236992119,2cyclictest503099-21kworker/u16:0+events_unbound03:53:105
552236992119,2cyclictest1153086-21kworker/u16:3+events_unbound08:51:165
552236992119,2cyclictest1039424-21kworker/u16:1+flush-8:007:47:095
552236992118,2cyclictest858769-21kworker/u16:2+events_unbound07:06:095
552232992119,2cyclictest858769-21kworker/u16:2+events_unbound07:17:024
552232992117,3cyclictest782390-21kworker/u16:0+flush-8:005:32:314
552242992019,0cyclictest0-21swapper/707:27:147
552242992019,0cyclictest0-21swapper/704:22:027
552242992019,0cyclictest0-21swapper/704:07:277
552237992018,2cyclictest730881-21kworker/u16:1+events_unbound05:17:286
552237992018,2cyclictest730881-21kworker/u16:1+events_unbound05:17:286
552237992018,2cyclictest1080400-21kworker/u16:0+flush-8:008:26:546
552237992016,3cyclictest894314-21kworker/u16:3+flush-8:006:37:206
552236992019,1cyclictest974347-21kworker/u16:3+events_unbound07:20:025
552236992018,2cyclictest858769-21kworker/u16:2+flush-8:008:02:055
552236992018,2cyclictest858769-21kworker/u16:2+flush-8:007:35:255
552236992016,3cyclictest688675-21kworker/u16:1+flush-8:004:47:215
552232992018,2cyclictest855078-21kworker/u16:1+flush-8:006:02:144
55224299190,17cyclictest752974-21cat05:12:177
552237991917,2cyclictest637318-21kworker/u16:5+events_unbound04:32:026
552237991917,2cyclictest419995-21kworker/u16:3+events_unbound03:42:296
552237991917,2cyclictest419995-21kworker/u16:3+events_unbound03:42:296
552237991916,2cyclictest1039424-21kworker/u16:1+flush-8:008:32:196
552237991915,3cyclictest628148-21kworker/u16:2+flush-8:004:52:316
552236991917,2cyclictest858769-21kworker/u16:2+events_unbound06:36:585
552236991917,2cyclictest1151406-21kworker/u16:2+flush-8:008:30:335
552236991917,2cyclictest1151406-21kworker/u16:2+flush-8:008:30:335
552232991917,2cyclictest843118-21kworker/u16:5+events_unbound06:27:334
552232991917,2cyclictest793379-21kworker/u16:4+events_unbound05:47:284
552232991916,2cyclictest858769-21kworker/u16:2+flush-8:006:42:214
552232991916,2cyclictest607119-21kworker/u16:3+flush-8:004:32:304
55224299180,16cyclictest648617-21latency_hist04:21:597
552237991816,2cyclictest937292-21kworker/u16:0+flush-8:006:57:156
552237991816,2cyclictest858769-21kworker/u16:2+events_unbound07:12:326
552237991816,2cyclictest782390-21kworker/u16:0+flush-8:006:31:556
552237991816,2cyclictest521453-21kworker/u16:1+flush-8:003:37:276
552237991815,2cyclictest503099-21kworker/u16:0+events_unbound04:02:196
552236991816,2cyclictest782390-21kworker/u16:0+events_unbound06:06:545
552236991816,2cyclictest607119-21kworker/u16:3+events_unbound04:20:065
552236991816,2cyclictest503099-21kworker/u16:0+events_unbound03:38:115
552232991814,3cyclictest1080400-21kworker/u16:0+events_unbound08:27:214
552232991814,3cyclictest1080400-21kworker/u16:0+events_unbound08:27:214
55224299179,7cyclictest27642-21gmain08:17:247
55224299170,1cyclictest0-21swapper/706:42:277
55224299170,16cyclictest1313-21snmpd08:13:027
55224299170,15cyclictest803645-21exim405:37:017
55224299170,15cyclictest1279-21tuned07:42:197
552237991715,2cyclictest855078-21kworker/u16:1+flush-8:006:12:196
552237991715,2cyclictest688675-21kworker/u16:1+flush-8:004:47:276
552237991715,2cyclictest475641-21kworker/u16:2+flush-8:003:56:556
552236991715,2cyclictest793379-21kworker/u16:4+events_unbound05:55:345
552236991715,2cyclictest616819-21kworker/u16:4+flush-8:004:22:205
552232991715,2cyclictest997937-21kworker/u16:5+flush-8:007:11:594
552232991715,2cyclictest974347-21kworker/u16:3+flush-8:006:57:244
552232991715,2cyclictest843118-21kworker/u16:5+events_unbound06:22:174
552232991715,2cyclictest419995-21kworker/u16:3+flush-8:003:42:054
552232991715,2cyclictest419995-21kworker/u16:3+flush-8:003:42:054
552232991715,2cyclictest1039424-21kworker/u16:1+flush-8:008:22:214
552232991714,2cyclictest855078-21kworker/u16:1+flush-8:006:09:204
552232991713,3cyclictest654587-21cut04:22:214
552237991614,2cyclictest700152-21kworker/u16:0+flush-8:004:57:236
552237991613,2cyclictest843118-21kworker/u16:5+flush-8:005:57:176
552236991614,2cyclictest711051-21kworker/u16:3+events_unbound05:07:245
552236991614,2cyclictest1039424-21kworker/u16:1+events_unbound08:20:175
552232991614,2cyclictest858769-21kworker/u16:2+events_unbound07:13:094
552232991614,2cyclictest475641-21kworker/u16:2+events_unbound04:02:024
552232991614,2cyclictest1151406-21kworker/u16:2+flush-8:008:46:544
552232991613,2cyclictest937292-21kworker/u16:0+flush-8:006:52:244
552232991613,2cyclictest937292-21kworker/u16:0+flush-8:006:52:244
55224299159,5cyclictest802916-21turbostat05:36:597
552242991514,0cyclictest0-21swapper/705:07:227
552242991514,0cyclictest0-21swapper/705:02:307
552242991513,1cyclictest854353-21munin-run06:01:597
55224299150,1cyclictest0-21swapper/707:47:277
55224299150,14cyclictest689757-21turbostat04:41:597
55224299150,14cyclictest643-21systemd-journal08:56:597
55223799152,5cyclictest0-21swapper/605:37:156
552237991513,2cyclictest1080400-21kworker/u16:0+events_unbound08:15:106
552237991512,2cyclictest858769-21kworker/u16:2+flush-8:007:17:106
552236991513,2cyclictest628148-21kworker/u16:2+events_unbound05:25:255
552236991513,2cyclictest503099-21kworker/u16:0+events_unbound03:50:345
552236991513,2cyclictest1120604-21kworker/u16:5+events_unbound08:44:005
552232991513,2cyclictest858769-21kworker/u16:2+events_unbound07:47:214
552232991513,2cyclictest637318-21kworker/u16:5+events_unbound04:31:544
552232991512,2cyclictest858769-21kworker/u16:2+flush-8:006:37:194
55223299150,14cyclictest0-21swapper/404:17:244
55224299146,0cyclictest0-21swapper/706:22:237
552242991411,2cyclictest916013-21munin-run06:31:597
55224299140,2cyclictest1090653-21munin-run07:56:597
55224299140,13cyclictest874903-21latency_hist06:11:597
552237991411,2cyclictest858769-21kworker/u16:2+events_unbound08:02:016
552237991411,2cyclictest1080400-21kworker/u16:0+flush-8:008:27:136
552237991411,2cyclictest1080400-21kworker/u16:0+flush-8:008:27:136
552237991410,3cyclictest607119-21kworker/u16:3+flush-8:004:12:306
55223699146,8cyclictest0-21swapper/508:22:165
552236991412,2cyclictest628148-21kworker/u16:2+flush-8:005:19:145
552236991412,2cyclictest628148-21kworker/u16:2+flush-8:005:19:145
552232991413,1cyclictest0-21swapper/403:47:214
552232991412,2cyclictest782390-21kworker/u16:0+flush-8:005:52:284
552232991412,2cyclictest1153086-21kworker/u16:3+flush-8:008:52:144
55223299140,14cyclictest0-21swapper/408:17:244
55224299135,6cyclictest27759-21gdbus04:48:517
552242991311,1cyclictest1100944-21awk08:01:597
55224299130,12cyclictest998978-21exim407:12:007
55224299130,12cyclictest1234646-21latency_hist09:06:597
552237991311,2cyclictest793379-21kworker/u16:4+flush-8:005:42:256
55223799130,1cyclictest0-21swapper/604:22:256
55223299132,11cyclictest0-21swapper/404:42:204
552232991311,2cyclictest711051-21kworker/u16:3+flush-8:005:16:594
552232991311,2cyclictest711051-21kworker/u16:3+flush-8:004:52:214
552232991311,2cyclictest628148-21kworker/u16:2+events_unbound05:22:334
55224299129,2cyclictest833634-21runrttasks05:50:067
55224299129,2cyclictest1183251-21munin-run08:41:597
55224299124,7cyclictest27177-21gmain06:17:257
552242991212,0cyclictest0-21swapper/707:08:087
552242991212,0cyclictest0-21swapper/705:45:327
552242991211,0cyclictest0-21swapper/706:57:257
552242991211,0cyclictest0-21swapper/706:37:157
552242991211,0cyclictest0-21swapper/704:29:527
552242991210,1cyclictest562413-21turbostat03:41:597
552242991210,1cyclictest1152370-21awk08:26:597
55224299120,3cyclictest777114-21kthreadcore05:22:247
55224299120,3cyclictest1202208-21kthreadcore08:47:217
55224299120,2cyclictest987976-21munin-run07:06:597
55224299120,11cyclictest96-21ksoftirqd/704:02:237
55224299120,11cyclictest1313-21snmpd08:57:487
55224299120,11cyclictest1313-21snmpd08:34:177
55224299120,11cyclictest1313-21snmpd06:56:337
55224299120,11cyclictest1313-21snmpd06:56:337
552237991210,2cyclictest793379-21kworker/u16:4+flush-8:005:47:266
552237991210,2cyclictest637318-21kworker/u16:5+flush-8:004:42:106
552237991210,2cyclictest628148-21kworker/u16:2+events_unbound04:17:016
552237991210,2cyclictest1080400-21kworker/u16:0+events_unbound08:08:106
552232991211,1cyclictest0-21swapper/405:58:124
55223299120,1cyclictest0-21swapper/407:54:234
55223299120,1cyclictest0-21swapper/405:44:104
55223299120,12cyclictest0-21swapper/405:07:554
55224299119,1cyclictest1049527-21awk07:36:597
55224299118,2cyclictest607462-21munin-run04:02:007
55224299115,5cyclictest27924-21gmain06:12:297
55224299114,6cyclictest27642-21gmain04:12:217
55224299114,6cyclictest27394-21gmain04:32:337
552242991111,0cyclictest0-21swapper/706:37:007
552242991110,0cyclictest0-21swapper/707:40:097
55224299110,9cyclictest1313-21snmpd05:52:167
55224299110,1cyclictest0-21swapper/708:07:277
55223799119,2cyclictest628148-21kworker/u16:2+flush-8:005:22:186
55223799119,2cyclictest1026546-21kworker/u16:0+flush-8:007:27:256
55223799118,3cyclictest1151406-21kworker/u16:2+flush-8:008:37:206
55223799117,3cyclictest1026546-21kworker/u16:0+flush-8:007:32:196
552237991110,1cyclictest0-21swapper/609:06:186
552237991110,1cyclictest0-21swapper/606:03:456
55223799110,1cyclictest0-21swapper/608:00:436
55223799110,1cyclictest0-21swapper/606:52:086
55223799110,1cyclictest0-21swapper/606:52:086
55223799110,11cyclictest0-21swapper/607:23:506
55223699110,11cyclictest0-21swapper/504:27:285
55223299119,2cyclictest925382-21kworker/u16:4+events_unbound06:47:294
55223299118,3cyclictest1153086-21kworker/u16:3+flush-8:009:01:554
552232991110,1cyclictest0-21swapper/405:18:574
552232991110,1cyclictest0-21swapper/405:18:574
55223299110,11cyclictest0-21swapper/404:38:544
55223299110,11cyclictest0-21swapper/403:58:334
55224299106,3cyclictest27763-21gvfs-afc-volume04:42:207
55224299100,9cyclictest1313-21snmpd08:27:057
55224299100,9cyclictest1313-21snmpd08:27:057
55224299100,8cyclictest563266-21exim403:42:017
55224299100,8cyclictest563266-21exim403:42:017
55224299100,2cyclictest1101661-21/usr/sbin/munin08:02:297
55224299100,1cyclictest0-21swapper/705:17:267
55224299100,1cyclictest0-21swapper/705:17:267
55223799108,2cyclictest1039424-21kworker/u16:1+events_unbound07:53:096
55223799107,2cyclictest711051-21kworker/u16:3+flush-8:005:27:106
55223299109,1cyclictest0-21swapper/407:04:254
55223299109,1cyclictest0-21swapper/404:48:034
55223299100,9cyclictest0-21swapper/406:32:164
5522429990,8cyclictest1019503-21exim407:22:007
5522429990,2cyclictest948-21NetworkManager08:42:317
5522379997,2cyclictest1203828-21kworker/u16:0+events_unbound08:52:146
5522379995,3cyclictest896953-21cat06:22:186
5522379995,3cyclictest858769-21kworker/u16:2+flush-8:007:02:306
5522379995,3cyclictest732325-21tune2fs05:02:156
5522329998,1cyclictest0-21swapper/407:25:324
5522329997,2cyclictest419995-21kworker/u16:3+events_unbound03:37:184
5522329996,2cyclictest885909-21/usr/sbin/munin06:17:154
5522379986,2cyclictest1151406-21kworker/u16:2+flush-8:008:47:146
5522379986,2cyclictest1069757-21kworker/u16:3+events_unbound07:51:106
5522379985,2cyclictest1058878-21ntp_states07:37:266
5522379984,3cyclictest598685-21hwlatdetect03:57:176
5522379983,4cyclictest624345-21cut04:07:206
5522379980,1cyclictest0-21swapper/603:47:186
5522329987,1cyclictest0-21swapper/408:51:464
5522329987,1cyclictest0-21swapper/405:39:074
5522329987,1cyclictest0-21swapper/404:11:384
5522329983,4cyclictest1107925-21kthreadcore08:02:204
5522379975,1cyclictest0-21swapper/608:57:226
5522379972,4cyclictest662179-21kthreadcore04:27:216
5522329976,1cyclictest0-21swapper/406:12:044
5522329962,3cyclictest1225221-21cat09:02:104
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional