Skip to content

[SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management #52728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+13 −2
Open

[SPARK-54026][BUILD] Use BOM for AWS Java SDK V2 dependency management #52728

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1189,6 +1189,13 @@
<artifactId>jline</artifactId>
<version>2.14.6</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bom</artifactId>
<version>${aws.java.sdk.v2.version}</version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you have used BOM to manage versions, the explicit version definition at the sub-module's pom.xml is not required, right?

hadoop-cloud/pom.xml
resource-managers/kubernetes/integration-tests/pom.xml

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pan3793 Both hadoop-cloud/pom.xml and resource-managers/kubernetes/integration-tests/pom.xml use bundle instead of individual jar dependencies while kinesis-asl relies on transitive dependencies of the producer library that use sts, auth and etc. BOM only defines individual dependencies and bundle still requires version to be specified in the dependency. Otherwise maven build fails:

 ./build/mvn -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests
Using `mvn` from path: /local/home/vrozov/ws/spark-sbt/build/apache-maven-3.9.11/bin/mvn
[INFO] Maveniverse Nisse 0.4.6 loaded
[INFO] Nisse injecting 39 properties into User Properties
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for software.amazon.awssdk:bundle:jar is missing. @ line 82, column 17
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.apache.spark:spark-kubernetes-integration-tests_2.13:4.1.0-SNAPSHOT (/local/home/vrozov/ws/spark-sbt/resource-managers/kubernetes/integration-tests/pom.xml) has 1 error
[ERROR]     'dependencies.dependency.version' for software.amazon.awssdk:bundle:jar is missing. @ line 82, column 17
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException

It may be necessary to see why both explicitly define dependency on bundle instead of relying on transitive dependencies. Though, I think, it is outside of the scope for this PR, WDYT?

<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.binary.version}</artifactId>
Expand Down
8 changes: 6 additions & 2 deletions project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1189,12 +1189,16 @@ object DependencyOverrides {
lazy val guavaVersion = sys.props.get("guava.version").getOrElse("33.4.0-jre")
lazy val jacksonVersion = sys.props.get("fasterxml.jackson.version").getOrElse("2.20.0")
lazy val jacksonDeps = Bom.dependencies("com.fasterxml.jackson" % "jackson-bom" % jacksonVersion)
lazy val settings = jacksonDeps ++ Seq(
lazy val awsJavaSdkV2Version = sys.props.get("aws.java.sdk.v2.version").getOrElse("2.29.52")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the drawback is, we need to duplicate the version definition here again, would be better to investigate if we can extract the version from pom.xml directly

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I agree, it would be better to see if the property can be read from the pom.xml. As it needs to be done not only for AWS Java SDK v2, I'd suggest to do this on a separate PR.

lazy val awsJavaSdkV2Deps =
Bom.dependencies("software.amazon.awssdk" % "bom" % awsJavaSdkV2Version)
lazy val settings = jacksonDeps ++ awsJavaSdkV2Deps ++ Seq(
dependencyOverrides += "com.google.guava" % "guava" % guavaVersion,
dependencyOverrides ++= jacksonDeps.key.value,
dependencyOverrides += "jline" % "jline" % "2.14.6",
dependencyOverrides += "org.apache.avro" % "avro" % "1.12.1",
dependencyOverrides += "org.slf4j" % "slf4j-api" % "2.0.17")
dependencyOverrides += "org.slf4j" % "slf4j-api" % "2.0.17",
dependencyOverrides ++= awsJavaSdkV2Deps.key.value)
}

/**
Expand Down