-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature-16396][storage-plugin] Tencent Cloud COS Storage Plugin #16565
base: dev
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we don't have any tencent cos support for integration testing. I will -1 on this until we can find a way to test against this in out UT, IT and E2E.
dolphinscheduler-storage-plugin/dolphinscheduler-storage-cos/src/test/resources/logback.xml
Show resolved
Hide resolved
...eduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java
Outdated
Show resolved
Hide resolved
String cosKey = transformAbsolutePathToCOSKey(srcFilePath); | ||
|
||
File dstFile = new File(dstFilePath); | ||
if (dstFile.isDirectory()) { |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression
|
||
File dstFile = new File(dstFilePath); | ||
if (dstFile.isDirectory()) { | ||
Files.delete(dstFile.toPath()); |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression
ea644bc
to
40e5ff2
Compare
01be125
to
49cc628
Compare
Quality Gate failedFailed conditions |
49cc628
to
8a33b77
Compare
* Fixed SonarCloud Quality Readability B (required >= A) * Added missing license files * YAML-ized Tencent COS configuration from common.properties * Added remote logger support with Tencent COS
8a33b77
to
037a91b
Compare
@Override | ||
public String getStorageBaseDirectory() { | ||
// All directory should end with File.separator | ||
if (resourceBaseAbsolutePath.startsWith("/")) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use File.separator
to instead of /
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The File.separator
works poorly on Windows 11 x64.
- When the user successfully uploads the "test.log" file, there is no way to enter the deeper directory to view the file.
- When the user tries to delete the folder, an exception is thrown stating that the folder does not exist.
So, for now, I think we should continue using the UNIX filesystem separator ("/"), just like the already existing code, until a DSIP for Windows platform compatibility is provided.
...age-cos/src/main/java/org/apache/dolphinscheduler/plugin/storage/cos/CosStorageOperator.java
Outdated
Show resolved
Hide resolved
...age-cos/src/main/java/org/apache/dolphinscheduler/plugin/storage/cos/CosStorageOperator.java
Show resolved
Hide resolved
.../src/main/java/org/apache/dolphinscheduler/plugin/storage/cos/CosStorageOperatorFactory.java
Outdated
Show resolved
Hide resolved
- Removed redundant dependency: `bcprov-jdk15on-1.69.jar` - Relocated Tencent COS Constants, and reduced magic values in COS with constants - Fixed CodeQL: Uncontrolled data used in path expression - Isolated cloud storage configuration in `common.properties` -> `resource-center.yaml`
037a91b
to
a1fdd4c
Compare
public static final String TENCENT_CLOUD_ACCESS_KEY_ID = "resource.tencent.cloud.access.key.id"; | ||
public static final String TENCENT_CLOUD_ACCESS_KEY_SECRET = "resource.tencent.cloud.access.key.secret"; | ||
|
||
public static final String ILLEGAL_DOWNLOAD_PATH_PREFIX_ETC = "/etc"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use vaule of data.basedir.path
Purpose of the pull request
This pull request adds a Storage Plugin to the DS Resource Center to support Tencent Cloud COS and aims to close #16396 .
Brief change log
dolphinscheduler-storage-cos
indolphinscheduler-storage-plugin
Verify this pull request
Pull Request Notice
Pull Request Notice
If your pull request contain incompatible change, you should also add it to
docs/docs/en/guide/upgrede/incompatible.md