Skip to content

[Fix-18177][Task Plugin] Fix AliyunServerlessSpark plugin dependency conflicts and improve exception handling#18180

Open
includetts wants to merge 12 commits intoapache:devfrom
includetts:fix/aliyun-serverless-spark-deps-v2
Open

[Fix-18177][Task Plugin] Fix AliyunServerlessSpark plugin dependency conflicts and improve exception handling#18180
includetts wants to merge 12 commits intoapache:devfrom
includetts:fix/aliyun-serverless-spark-deps-v2

Conversation

@includetts
Copy link
Copy Markdown

Was this PR generated or assisted by AI?

NO

Purpose of the pull request

Fix two critical dependency version bugs that cause NoSuchMethodError and NoSuchFieldError at runtime when using the AliyunServerlessSpark task plugin, and improve exception handling to preserve root cause information for debugging.

Closes #18177 Closes #18178

Brief change log

  • Upgrade emr_serverless_spark20230808 from 1.0.0 to 2.4.1 in datasource plugin to match task plugin version
    • Upgrade credentials-java from 0.3.0 to 1.0.1 in both datasource and task plugins to satisfy tea-openapi:0.3.8 requirement
    • Propagate original exception message and cause in 3 catch blocks (buildClient, startJobRun, cancelJobRun) in AliyunServerlessSparkTask.java

Verify this pull request

Manually verified the change by testing locally:

  • Built both shade jars and confirmed Client.getTemplate() method exists in datasource plugin shade jar
    • Confirmed CredentialModel.providerName field present in both shade jars (credentials-java 1.0.1)
    • Confirmed exception cause chain is preserved via bytecode inspection

Pull Request Notice

Pull Request Notice

@SbloodyS SbloodyS added bug Something isn't working first time contributor First-time contributor labels Apr 19, 2026
@SbloodyS SbloodyS added this to the 3.4.2 milestone Apr 19, 2026
Comment thread dolphinscheduler-task-plugin/dolphinscheduler-task-aliyunserverlessspark/pom.xml Outdated
@SbloodyS
Copy link
Copy Markdown
Member

Please don't create PR repeatedly, which will increase the work of reviewer.

@includetts
Copy link
Copy Markdown
Author

Hi @SbloodyS, thanks for the review! I've addressed your feedback:

  1. Version properties — Done. Both pom.xml files now use to manage emr-serverless-spark.version and credentials-java.version
  2. Exception handling in buildClient (L121) — Reverted. You're right that log.error("...", e) already outputs the full stacktrace to the task instance log
  3. Exception handling in startJobRun (L157) and cancelJobRun (L227) — I kept these two changes because, unlike buildClient, these catch blocks do not have log.error() before the throw. Without propagating the cause, the original API error (e.g., authentication failure, parameter validation error from Alibaba Cloud) would be completely lost and invisible in the task instance log. The user would only see "Failed to start job run!" without any diagnostic information
  4. Regarding multiple PRs — Sorry about that. The first PR (fix: [Task Plugin] Fix AliyunServerlessSpark plugin dependency conflicts and improve exception handling #18179) had file formatting corruption caused by the GitHub web editor, so I had to recreate it. I'll be more careful in the future.
    Please take another look when you have a chance. Thank you!

@SbloodyS
Copy link
Copy Markdown
Member

You should run mvn spotless:apply to fix CI. @includetts

@sonarqubecloud
Copy link
Copy Markdown

Copy link
Copy Markdown
Member

@SbloodyS SbloodyS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically LGTM, Just some NIT.

Comment on lines +33 to +34
<emr-serverless-spark.version>2.4.1</emr-serverless-spark.version>
<credentials-java.version>1.0.1</credentials-java.version>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's bettter to put it in bom for unified management.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend bug Something isn't working first time contributor First-time contributor

Projects

None yet

2 participants