Skip to content

Commit

Permalink
[SPARK-49569][BUILD][FOLLOWUP] Adds scala-library maven dependency …
Browse files Browse the repository at this point in the history
…to the `spark-connect-shims` module to fix Maven build errors

### What changes were proposed in this pull request?
This PR adds `scala-library` maven dependency to the `spark-connect-shims` module to fix Maven build errors.

### Why are the changes needed?
Maven daily test pipeline build failed:

- https://github.com/apache/spark/actions/runs/11255598249
- https://github.com/apache/spark/actions/runs/11256610976

```
scaladoc error: fatal error: object scala in compiler mirror not found.
Error:  Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.9.1:doc-jar (attach-scaladocs) on project spark-connect-shims_2.13: MavenReportException: Error while creating archive: wrap: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
Error:
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Error:
Error:  After correcting the problems, you can resume the build with the command
Error:    mvn <args> -rf :spark-connect-shims_2.13
Error: Process completed with exit code 1.
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass GitHub Actions
- locally test:

```
build/mvn clean install -DskipTests -Phive
```

**Before**

```
[INFO] --- scala:4.9.1:doc-jar (attach-scaladocs)  spark-connect-shims_2.13 ---
scaladoc error: fatal error: object scala in compiler mirror not found.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.833 s]
[INFO] Spark Project Tags ................................. SUCCESS [  5.292 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  5.675 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 16.762 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  7.735 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 11.389 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  9.159 s]
[INFO] Spark Project Variant .............................. SUCCESS [  3.618 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  9.692 s]
[INFO] Spark Project Connect Shims ........................ FAILURE [  2.478 s]
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project SQL API .............................. SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project Connect Common ....................... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] Spark Protobuf ..................................... SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Connect Server ....................... SKIPPED
[INFO] Spark Project Connect Client ....................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:15 min
[INFO] Finished at: 2024-10-09T23:43:58+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.9.1:doc-jar (attach-scaladocs) on project spark-connect-shims_2.13: MavenReportException: Error while creating archive: wrap: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :spark-connect-shims_2.13

```

**After**

```
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.0.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.766 s]
[INFO] Spark Project Tags ................................. SUCCESS [  5.398 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  6.361 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 16.919 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  8.083 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 11.240 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  9.438 s]
[INFO] Spark Project Variant .............................. SUCCESS [  3.697 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  9.939 s]
[INFO] Spark Project Connect Shims ........................ SUCCESS [  2.938 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  6.502 s]
[INFO] Spark Project Core ................................. SUCCESS [01:33 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 18.220 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 20.923 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 29.949 s]
[INFO] Spark Project SQL API .............................. SUCCESS [ 25.842 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [02:02 min]
[INFO] Spark Project SQL .................................. SUCCESS [02:18 min]
[INFO] Spark Project ML Library ........................... SUCCESS [01:38 min]
[INFO] Spark Project Tools ................................ SUCCESS [  3.365 s]
[INFO] Spark Project Hive ................................. SUCCESS [ 45.357 s]
[INFO] Spark Project Connect Common ....................... SUCCESS [ 33.636 s]
[INFO] Spark Avro ......................................... SUCCESS [ 22.040 s]
[INFO] Spark Protobuf ..................................... SUCCESS [ 24.557 s]
[INFO] Spark Project REPL ................................. SUCCESS [ 13.843 s]
[INFO] Spark Project Connect Server ....................... SUCCESS [ 35.587 s]
[INFO] Spark Project Connect Client ....................... SUCCESS [ 33.929 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  5.121 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 12.623 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 16.908 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 23.664 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 30.777 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  6.997 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15:40 min
[INFO] Finished at: 2024-10-09T23:27:20+08:00
[INFO] ------------------------------------------------------------------------

```

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #48399 from LuciferYang/SPARK-49569-FOLLOWUP.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
  • Loading branch information
LuciferYang committed Oct 9, 2024
1 parent 97a5aa6 commit fed9a8d
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions sql/connect/shims/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,13 @@
<sbt.project.name>connect-shims</sbt.project.name>
</properties>

<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
</dependencies>

<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
Expand Down

0 comments on commit fed9a8d

Please sign in to comment.