Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE#12994]Optimize config operation and capacity management. #13002

Merged
merged 19 commits into from
Mar 5, 2025
Merged

[ISSUE#12994]Optimize config operation and capacity management. #13002

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
4ac2eba
optimize config operation.
KiteSoar Dec 30, 2024
046940e
Update the unit tests.
KiteSoar Jan 1, 2025
745949c
Update Aspect.
KiteSoar Jan 1, 2025
08564eb
Refactor aspect.
KiteSoar Jan 10, 2025
e854f8f
Fix pmd.
KiteSoar Jan 10, 2025
dfd421e
Fix unit test.
KiteSoar Jan 10, 2025
83b5957
Fix capacityManagementAspect.
KiteSoar Jan 10, 2025
789c353
Merge remote-tracking branch 'upstream/develop' into optimize-config-…
KiteSoar Feb 12, 2025
3ad36a1
[ISSUE#12994] Optimize config operation.
KiteSoar Feb 15, 2025
353c23c
[ISSUE#12994] Optimize batch import configuration operations.
KiteSoar Feb 18, 2025
3a57e4d
[ISSUE#12994] Remove the importConfig parameter.
KiteSoar Feb 18, 2025
a1b8222
[ISSUE#12994] Add parameter verification.
KiteSoar Feb 18, 2025
b3fd30b
[ISSUE#12994] Solve conflicts.
KiteSoar Feb 20, 2025
e261fce
[ISSUE#12994] Optimized configuration aspect class.
KiteSoar Feb 20, 2025
1abdaa8
[ISSUE#12994] Modify log format.
KiteSoar Feb 24, 2025
eb7cfcc
[ISSUE#12994] Fix the error code.
KiteSoar Feb 24, 2025
02d71b8
[ISSUE#13129] Fix SQL syntax for capacity management.
KiteSoar Mar 3, 2025
5b9ed8d
[ISSUE#13129] Fix CI.
KiteSoar Mar 4, 2025
d2afe08
[ISSUE#13129] Refactor the capacity management Mapper.
KiteSoar Mar 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
231 changes: 104 additions & 127 deletions config/src/main/java/com/alibaba/nacos/config/server/aspect/CapacityManagementAspect.java
View file
Open in desktop

Large diffs are not rendered by default.

212 changes: 86 additions & 126 deletions config/src/main/java/com/alibaba/nacos/config/server/aspect/ConfigChangeAspect.java
View file
Open in desktop

Large diffs are not rendered by default.

246 changes: 90 additions & 156 deletions config/src/main/java/com/alibaba/nacos/config/server/aspect/RequestLogAspect.java
View file
Open in desktop

Large diffs are not rendered by default.

32 changes: 7 additions & 25 deletions config/src/main/java/com/alibaba/nacos/config/server/controller/ConfigController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,7 @@ public Boolean publishConfig(HttpServletRequest request, HttpServletResponse res

ConfigRequestInfo configRequestInfo = new ConfigRequestInfo();
configRequestInfo.setSrcIp(RequestUtil.getRemoteIp(request));
configRequestInfo.setSrcType("http");
configRequestInfo.setRequestIpApp(RequestUtil.getAppName(request));
configRequestInfo.setBetaIps(request.getHeader("betaIps"));
configRequestInfo.setCasMd5(request.getHeader("casMd5"));
Expand Down Expand Up @@ -299,38 +300,19 @@ public Boolean deleteConfig(HttpServletRequest request, HttpServletResponse resp
String clientIp = RequestUtil.getRemoteIp(request);
String srcUser = RequestUtil.getSrcUserName(request);

return configOperationService.deleteConfig(dataId, group, tenant, tag, clientIp, srcUser);
return configOperationService.deleteConfig(dataId, group, tenant, tag, clientIp, srcUser, "http");
}

/**
* Execute delete config operation.
*
* @return java.lang.Boolean
* @author klw
* @Description: delete configuration based on multiple config ids
* @Date 2019/7/5 10:26
* @Param [request, response, dataId, group, tenant, tag]
* delete configs based on the IDs list.
*/
@DeleteMapping(params = "delType=ids")
@Secured(action = ActionTypes.WRITE, signType = SignType.CONFIG)
public RestResult<Boolean> deleteConfigs(HttpServletRequest request, @RequestParam(value = "ids") List<Long> ids) {
String clientIp = RequestUtil.getRemoteIp(request);
String srcUser = RequestUtil.getSrcUserName(request);
final Timestamp time = TimeUtils.getCurrentTime();
List<ConfigAllInfo> configInfoList = configInfoPersistService.removeConfigInfoByIds(ids, clientIp, srcUser);
if (CollectionUtils.isEmpty(configInfoList)) {
return RestResultUtils.success(true);
}
for (ConfigAllInfo configInfo : configInfoList) {
ConfigChangePublisher.notifyConfigChange(
new ConfigDataChangeEvent(configInfo.getDataId(), configInfo.getGroup(), configInfo.getTenant(),
time.getTime()));

ConfigTraceService.logPersistenceEvent(configInfo.getDataId(), configInfo.getGroup(),
configInfo.getTenant(), null, time.getTime(), clientIp, ConfigTraceService.PERSISTENCE_EVENT,
ConfigTraceService.PERSISTENCE_TYPE_REMOVE, null);
}
return RestResultUtils.success(true);

return RestResultUtils.success(configOperationService.deleteConfigs(ids, clientIp, srcUser));
}

@GetMapping("/catalog")
Expand Down Expand Up @@ -683,7 +665,7 @@ public RestResult<Map<String, Object>> importAndPublishConfig(HttpServletRequest
final String srcIp = RequestUtil.getRemoteIp(request);
String requestIpApp = RequestUtil.getAppName(request);
final Timestamp time = TimeUtils.getCurrentTime();
Map<String, Object> saveResult = configInfoPersistService.batchInsertOrUpdate(configInfoList, srcUser, srcIp,
Map<String, Object> saveResult = configOperationService.batchInsertOrUpdate(configInfoList, srcUser, srcIp,
null, policy);
for (ConfigInfo configInfo : configInfoList) {
ConfigChangePublisher.notifyConfigChange(
Expand Down Expand Up @@ -935,7 +917,7 @@ public RestResult<Map<String, Object>> cloneConfig(HttpServletRequest request,
final String srcIp = RequestUtil.getRemoteIp(request);
String requestIpApp = RequestUtil.getAppName(request);
final Timestamp time = TimeUtils.getCurrentTime();
Map<String, Object> saveResult = configInfoPersistService.batchInsertOrUpdate(configInfoList4Clone, srcUser,
Map<String, Object> saveResult = configOperationService.batchInsertOrUpdate(configInfoList4Clone, srcUser,
srcIp, null, policy);
for (ConfigInfo configInfo : configInfoList4Clone) {
ConfigChangePublisher.notifyConfigChange(
Expand Down
3 changes: 2 additions & 1 deletion config/src/main/java/com/alibaba/nacos/config/server/controller/v2/ConfigControllerV2.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ public Result<Boolean> publishConfig(ConfigForm configForm, HttpServletRequest r

ConfigRequestInfo configRequestInfo = new ConfigRequestInfo();
configRequestInfo.setSrcIp(RequestUtil.getRemoteIp(request));
configRequestInfo.setSrcType("http");
configRequestInfo.setRequestIpApp(RequestUtil.getAppName(request));
configRequestInfo.setBetaIps(request.getHeader("betaIps"));
configRequestInfo.setCasMd5(request.getHeader("casMd5"));
Expand Down Expand Up @@ -169,7 +170,7 @@ public Result<Boolean> deleteConfig(HttpServletRequest request, @RequestParam("d

String clientIp = RequestUtil.getRemoteIp(request);
String srcUser = RequestUtil.getSrcUserName(request);
return Result.success(configOperationService.deleteConfig(dataId, group, namespaceId, tag, clientIp, srcUser));
return Result.success(configOperationService.deleteConfig(dataId, group, namespaceId, tag, clientIp, srcUser, "http"));
}

/**
Expand Down
13 changes: 12 additions & 1 deletion config/src/main/java/com/alibaba/nacos/config/server/model/ConfigRequestInfo.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,17 @@ public class ConfigRequestInfo implements Serializable {

private String srcIp;

private String srcType;

private String requestIpApp;

private String betaIps;

private String casMd5;

public ConfigRequestInfo(String srcIp, String requestIpApp, String betaIps, String casMd5) {
public ConfigRequestInfo(String srcIp, String srcType, String requestIpApp, String betaIps, String casMd5) {
this.srcIp = srcIp;
this.srcType = srcType;
this.requestIpApp = requestIpApp;
this.betaIps = betaIps;
this.casMd5 = casMd5;
Expand All @@ -54,6 +57,14 @@ public void setSrcIp(String srcIp) {
this.srcIp = srcIp;
}

public String getSrcType() {
return srcType;
}

public void setSrcType(String srcType) {
this.srcType = srcType;
}

public String getRequestIpApp() {
return requestIpApp;
}
Expand Down
10 changes: 0 additions & 10 deletions config/src/main/java/com/alibaba/nacos/config/server/monitor/MetricsMonitor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,21 +175,11 @@ public static Timer getReadConfigRtTimer() {
.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "readConfigRt");
}

public static Timer getReadConfigRpcRtTimer() {
return NacosMeterRegistryCenter
.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "readConfigRpcRt");
}

public static Timer getWriteConfigRtTimer() {
return NacosMeterRegistryCenter
.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "writeConfigRt");
}

public static Timer getWriteConfigRpcRtTimer() {
return NacosMeterRegistryCenter
.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "writeConfigRpcRt");
}

public static Timer getNotifyRtTimer() {
return NacosMeterRegistryCenter.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "notifyRt");
}
Expand Down
1 change: 1 addition & 0 deletions config/src/main/java/com/alibaba/nacos/config/server/remote/ConfigPublishRequestHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ public ConfigPublishResponse handle(ConfigPublishRequest request, RequestMeta me

ConfigRequestInfo configRequestInfo = new ConfigRequestInfo();
configRequestInfo.setSrcIp(srcIp);
configRequestInfo.setSrcType("rpc");
configRequestInfo.setRequestIpApp(meta.getLabels().get(Constants.APPNAME));
configRequestInfo.setBetaIps(request.getAdditionParam("betaIps"));
configRequestInfo.setCasMd5(request.getCasMd5());
Expand Down
51 changes: 10 additions & 41 deletions config/src/main/java/com/alibaba/nacos/config/server/remote/ConfigRemoveRequestHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,8 @@
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.remote.request.RequestMeta;
import com.alibaba.nacos.auth.annotation.Secured;
import com.alibaba.nacos.common.utils.StringUtils;
import com.alibaba.nacos.config.server.model.event.ConfigDataChangeEvent;
import com.alibaba.nacos.config.server.model.gray.TagGrayRule;
import com.alibaba.nacos.config.server.service.ConfigChangePublisher;
import com.alibaba.nacos.config.server.service.repository.ConfigInfoGrayPersistService;
import com.alibaba.nacos.config.server.service.repository.ConfigInfoPersistService;
import com.alibaba.nacos.config.server.service.trace.ConfigTraceService;
import com.alibaba.nacos.config.server.service.ConfigOperationService;
import com.alibaba.nacos.config.server.utils.ParamUtils;
import com.alibaba.nacos.config.server.utils.TimeUtils;
import com.alibaba.nacos.core.control.TpsControl;
import com.alibaba.nacos.core.paramcheck.ExtractorManager;
import com.alibaba.nacos.core.paramcheck.impl.ConfigRequestParamExtractor;
Expand All @@ -39,8 +32,6 @@
import com.alibaba.nacos.plugin.auth.constant.SignType;
import org.springframework.stereotype.Component;

import java.sql.Timestamp;

/**
* handler to remove config.
*
Expand All @@ -50,14 +41,10 @@
@Component
public class ConfigRemoveRequestHandler extends RequestHandler<ConfigRemoveRequest, ConfigRemoveResponse> {

private final ConfigInfoPersistService configInfoPersistService;

private final ConfigInfoGrayPersistService configInfoGrayPersistService;
private final ConfigOperationService configOperationService;

public ConfigRemoveRequestHandler(ConfigInfoPersistService configInfoPersistService,
ConfigInfoGrayPersistService configInfoGrayPersistService) {
this.configInfoPersistService = configInfoPersistService;
this.configInfoGrayPersistService = configInfoGrayPersistService;
public ConfigRemoveRequestHandler(ConfigOperationService configOperationService) {
this.configOperationService = configOperationService;
}

@Override
Expand All @@ -66,36 +53,18 @@ public ConfigRemoveRequestHandler(ConfigInfoPersistService configInfoPersistServ
@ExtractorManager.Extractor(rpcExtractor = ConfigRequestParamExtractor.class)
public ConfigRemoveResponse handle(ConfigRemoveRequest configRemoveRequest, RequestMeta meta)
throws NacosException {
// check tenant
String tenant = configRemoveRequest.getTenant();
String dataId = configRemoveRequest.getDataId();
String group = configRemoveRequest.getGroup();
String tag = configRemoveRequest.getTag();
String underLine = "_";
String clientIp = meta.getClientIp();
ParamUtils.checkTenant(tenant);
ParamUtils.checkParam(dataId, group, "datumId", "rm");
ParamUtils.checkParam(tag);

try {
ParamUtils.checkTenant(tenant);
ParamUtils.checkParam(dataId, group, "datumId", "rm");
ParamUtils.checkParam(tag);
String persistEvent = ConfigTraceService.PERSISTENCE_EVENT;

String clientIp = meta.getClientIp();
String grayName = null;
if (StringUtils.isBlank(tag)) {

configInfoPersistService.removeConfigInfo(dataId, group, tenant, clientIp, null);
} else {
persistEvent = ConfigTraceService.PERSISTENCE_EVENT_TAG + underLine + tag;

grayName = TagGrayRule.TYPE_TAG + underLine + tag;
configInfoGrayPersistService.removeConfigInfoGray(dataId, group, tenant, grayName, clientIp, null);
}
final Timestamp time = TimeUtils.getCurrentTime();
ConfigTraceService.logPersistenceEvent(dataId, group, tenant, null, time.getTime(), clientIp, persistEvent,
ConfigTraceService.PERSISTENCE_TYPE_REMOVE, null);
ConfigChangePublisher.notifyConfigChange(
new ConfigDataChangeEvent(dataId, group, tenant, grayName, time.getTime()));
configOperationService.deleteConfig(dataId, group, tenant, tag, clientIp, null, "rpc");
return ConfigRemoveResponse.buildSuccessResponse();

} catch (Exception e) {
Loggers.REMOTE_DIGEST.error("remove config error,error msg is {}", e.getMessage(), e);
return ConfigRemoveResponse.buildFailResponse(e.getMessage());
Expand Down
36 changes: 34 additions & 2 deletions config/src/main/java/com/alibaba/nacos/config/server/service/ConfigOperationService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,11 @@
import com.alibaba.nacos.common.utils.MapUtil;
import com.alibaba.nacos.common.utils.NumberUtils;
import com.alibaba.nacos.common.utils.StringUtils;
import com.alibaba.nacos.config.server.model.ConfigAllInfo;
import com.alibaba.nacos.config.server.model.ConfigInfo;
import com.alibaba.nacos.config.server.model.ConfigOperateResult;
import com.alibaba.nacos.config.server.model.ConfigRequestInfo;
import com.alibaba.nacos.config.server.model.SameConfigPolicy;
import com.alibaba.nacos.config.server.model.event.ConfigDataChangeEvent;
import com.alibaba.nacos.config.server.model.form.ConfigForm;
import com.alibaba.nacos.config.server.model.gray.BetaGrayRule;
Expand Down Expand Up @@ -293,13 +295,13 @@ private int getMaxGrayVersionCount() {
* Synchronously delete all pre-aggregation data under a dataId.
*/
public Boolean deleteConfig(String dataId, String group, String namespaceId, String tag, String clientIp,
String srcUser) {
String srcUser, String srcType) {
String persistEvent = ConfigTraceService.PERSISTENCE_EVENT;
String grayName = "";
if (StringUtils.isBlank(tag)) {
configInfoPersistService.removeConfigInfo(dataId, group, namespaceId, clientIp, srcUser);
} else {
persistEvent = ConfigTraceService.PERSISTENCE_EVENT_TAG + "-" + tag;
persistEvent = ConfigTraceService.PERSISTENCE_EVENT_TAG + "_" + tag;
Copy link
Collaborator

@shiyiyue1102 shiyiyue1102 Feb 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why changing split "-" to "_"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is when the deletion configuration method on the SDK side is changed to call the deletion method of ConfigOperationService. In order to maintain consistency with the original format on the SDK side, as shown in the following figure.
image

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all right , if delete config using "" and publish config using "-", it will be more proper to change "" to "-" in delete config.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done，Has been changed to "-".

grayName = TagGrayRule.TYPE_TAG + "_" + tag;
configInfoGrayPersistService.removeConfigInfoGray(dataId, group, namespaceId, grayName, clientIp, srcUser);
deleteConfigTagv1(dataId, group, namespaceId, tag, clientIp, srcUser);
Expand All @@ -319,6 +321,36 @@ private void deleteConfigTagv1(String dataId, String group, String namespaceId,
configInfoTagPersistService.removeConfigInfoTag(dataId, group, namespaceId, tag, clientIp, srcUser);
}
}

/**
* Deletes configuration information based on the IDs list.
*/
public Boolean deleteConfigs(List<Long> ids, String srcIp, String srcUser) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleteConfigs should check capacity , or can we split deleteConfigs operation to serval single deleteConfig operation in ConfigController?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we have already split batch operations on controllers, method deteConfigs should be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

List<ConfigAllInfo> configInfoList = configInfoPersistService.removeConfigInfoByIds(ids, srcIp, srcUser);
if (configInfoList == null || configInfoList.isEmpty()) {
return true;
}

Timestamp time = TimeUtils.getCurrentTime();
for (ConfigAllInfo configInfo : configInfoList) {
ConfigChangePublisher.notifyConfigChange(
new ConfigDataChangeEvent(configInfo.getDataId(), configInfo.getGroup(), configInfo.getTenant(),
time.getTime()));
ConfigTraceService.logPersistenceEvent(configInfo.getDataId(), configInfo.getGroup(),
configInfo.getTenant(), null, time.getTime(), srcIp, ConfigTraceService.PERSISTENCE_EVENT,
ConfigTraceService.PERSISTENCE_TYPE_REMOVE, null);
}

return true;
}

/**
* Batch insert or update configuration information.
*/
public Map<String, Object> batchInsertOrUpdate(List<ConfigAllInfo> configInfoList, String srcUser, String srcIp,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all batch operation invoked ConfigOperationService single operation interface, all aspects works at single config publish & delete interface of ConfigOperationService

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we have already split batch operations on controllers, method batchInsertOrUpdate should be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Map<String, Object> configAdvanceInfo, SameConfigPolicy policy) throws NacosException {
return configInfoPersistService.batchInsertOrUpdate(configInfoList, srcUser, srcIp, configAdvanceInfo, policy);
}

public Map<String, Object> getConfigAdvanceInfo(ConfigForm configForm) {
Map<String, Object> configAdvanceInfo = new HashMap<>(10);
Expand Down
Loading
Loading