Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-49746][BUILD] Upgrade Scala to 2.13.15 #48192

Closed
wants to merge 13 commits into from
Closed

[SPARK-49746][BUILD] Upgrade Scala to 2.13.15 #48192

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
fb04931
[SPARK-49746][BUILD] Upgrade Scala to 2.13.15
panbingkun Sep 20, 2024
dfc4a47
Merge branch 'master' into SPARK-49746
panbingkun Sep 24, 2024
a1be4ca
update
panbingkun Sep 24, 2024
cd8e385
Merge branch 'master' into SPARK-49746
panbingkun Sep 24, 2024
116c08e
fix compile error
panbingkun Sep 24, 2024
c143074
fix
panbingkun Sep 24, 2024
6eb47ad
Merge branch 'master' into SPARK-49746
panbingkun Sep 24, 2024
a1f3dc1
run maven tests
panbingkun Sep 24, 2024
12ebfb1
Rewrite -Wconf:cat=deprecation:wv,any:e for maven
LuciferYang Sep 24, 2024
208c59f
Rewrite -Wconf:cat=deprecation:wv,any:e for sbt
LuciferYang Sep 24, 2024
68ae766
Revert change of CompressionSchemeBenchmark.scala
LuciferYang Sep 24, 2024
f84896e
Merge branch 'master' into SPARK-49746
panbingkun Sep 25, 2024
ad36c12
update
panbingkun Sep 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ jetty-util-ajax/11.0.23//jetty-util-ajax-11.0.23.jar
jetty-util/11.0.23//jetty-util-11.0.23.jar
jjwt-api/0.12.6//jjwt-api-0.12.6.jar
jline/2.14.6//jline-2.14.6.jar
jline/3.25.1//jline-3.25.1.jar
jline/3.26.3//jline-3.26.3.jar
jna/5.14.0//jna-5.14.0.jar
joda-time/2.13.0//joda-time-2.13.0.jar
jodd-core/3.5.2//jodd-core-3.5.2.jar
Expand Down Expand Up @@ -252,11 +252,11 @@ py4j/0.10.9.7//py4j-0.10.9.7.jar
remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar
rocksdbjni/9.5.2//rocksdbjni-9.5.2.jar
scala-collection-compat_2.13/2.7.0//scala-collection-compat_2.13-2.7.0.jar
scala-compiler/2.13.14//scala-compiler-2.13.14.jar
scala-library/2.13.14//scala-library-2.13.14.jar
scala-compiler/2.13.15//scala-compiler-2.13.15.jar
scala-library/2.13.15//scala-library-2.13.15.jar
scala-parallel-collections_2.13/1.0.4//scala-parallel-collections_2.13-1.0.4.jar
scala-parser-combinators_2.13/2.4.0//scala-parser-combinators_2.13-2.4.0.jar
scala-reflect/2.13.14//scala-reflect-2.13.14.jar
scala-reflect/2.13.15//scala-reflect-2.13.15.jar
scala-xml_2.13/2.3.0//scala-xml_2.13-2.3.0.jar
slf4j-api/2.0.16//slf4j-api-2.0.16.jar
snakeyaml-engine/2.7//snakeyaml-engine-2.7.jar
Expand Down
2 changes: 1 addition & 1 deletion docs/_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ include:
SPARK_VERSION: 4.0.0-SNAPSHOT
SPARK_VERSION_SHORT: 4.0.0
SCALA_BINARY_VERSION: "2.13"
SCALA_VERSION: "2.13.14"
SCALA_VERSION: "2.13.15"
SPARK_ISSUE_TRACKER_URL: https://issues.apache.org/jira/browse/SPARK
SPARK_GITHUB_URL: https://github.com/apache/spark
# Before a new release, we should:
Expand Down
9 changes: 7 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@
<!-- managed up from 3.2.1 for SPARK-11652 -->
<commons.collections.version>3.2.2</commons.collections.version>
<commons.collections4.version>4.4</commons.collections4.version>
<scala.version>2.13.14</scala.version>
<scala.version>2.13.15</scala.version>
<scala.binary.version>2.13</scala.binary.version>
<scalatest-maven-plugin.version>2.2.0</scalatest-maven-plugin.version>
<scala-maven-plugin.version>4.9.1</scala-maven-plugin.version>
Expand Down Expand Up @@ -226,7 +226,7 @@
and ./python/packaging/connect/setup.py too.
-->
<arrow.version>17.0.0</arrow.version>
<ammonite.version>3.0.0-M2</ammonite.version>
<ammonite.version>3.0.0</ammonite.version>
<jjwt.version>0.12.6</jjwt.version>

<!-- org.fusesource.leveldbjni will be used except on arm64 platform. -->
Expand Down Expand Up @@ -3073,6 +3073,11 @@
reduce the cost of migration in subsequent versions.
-->
<arg>-Wconf:cat=deprecation&amp;msg=it will become a keyword in Scala 3:e</arg>
<!--
Some deprecated calls is a compilation error in Scala 2.13.15.
-->
<arg>-Wconf:cat=deprecation&amp;msg=^(?=.*?method|value|object|trait|class|constructor)(?=.*?in)(?=.*?deprecated).+$:w</arg>
<arg>-Wconf:cat=deprecation&amp;msg=shadowing a nested class of a parent is deprecated:w</arg>
</args>
<jvmArgs>
<jvmArg>-Xss128m</jvmArg>
Expand Down
5 changes: 4 additions & 1 deletion project/SparkBuild.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,10 @@ object SparkBuild extends PomBuild {
// reduce the cost of migration in subsequent versions.
"-Wconf:cat=deprecation&msg=it will become a keyword in Scala 3:e",
// SPARK-46938 to prevent enum scan on pmml-model, under spark-mllib module.
"-Wconf:cat=other&site=org.dmg.pmml.*:w"
"-Wconf:cat=other&site=org.dmg.pmml.*:w",
// Some deprecated calls is a compilation error in Scala 2.13.15.
"-Wconf:cat=deprecation&msg=^(?=.*?method|value|object|trait|class|constructor)(?=.*?in)(?=.*?deprecated).+$:w",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should make this rule more specific, as its current scope of matching is too broad. Additionally, we should leave a TODO to remind us to fix these deprecated API usages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, let's go through Maven's test first.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spark/project/SparkBuild.scala

Line 237 in 73d6bd7

"-Wconf:cat=deprecation:wv,any:e",

I want to know if this line of configuration has become invalid in Scala 2.13.15？Here we have configured -Wconf:cat=deprecation:wv.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me verify it.

Copy link
Contributor

@LuciferYang LuciferYang Sep 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Scala 2.13.14 with -Wconf:cat=deprecation:wv,any:e, printed a warning message for the use of deprecated API
~/Tools/scala-2.13.14/bin/scala -Wconf:cat=deprecation:wv,any:e
Welcome to Scala 2.13.14 (OpenJDK 64-Bit Server VM, Java 17.0.12).
Type in expressions for evaluation. Or try :help.

scala> class Foo {
     |   def isEmpty(): Boolean = true
     | }
class Foo

scala> val foo = new Foo
val foo: Foo = Foo@706ddbc8

scala> val ret = foo.isEmpty
                     ^
       warning: Auto-application to `()` is deprecated. Supply the empty argument list `()` explicitly to invoke method isEmpty,
       or remove the empty argument list from its definition (Java-defined methods are exempt).
       In Scala 3, an unapplied method like this will be eta-expanded into a function. [quickfixable]
       Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=ret, version=2.13.3
val ret: Boolean = true
  • Scala 2.13.15 with -Wconf:cat=deprecation:wv,any:e, printed an error message for the use of deprecated API
~/Tools/scala-2.13.15/bin/scala -Wconf:cat=deprecation:wv,any:e
Welcome to Scala 2.13.15 (OpenJDK 64-Bit Server VM, Java 17.0.12).
Type in expressions for evaluation. Or try :help.

scala> class Foo {
     |   def isEmpty(): Boolean = true
     | }
class Foo

scala> val foo = new Foo
val foo: Foo = Foo@53f0d09c

scala> val ret = foo.isEmpty
                     ^
       error: Auto-application to `()` is deprecated. Supply the empty argument list `()` explicitly to invoke method isEmpty,
       or remove the empty argument list from its definition (Java-defined methods are exempt).
       In Scala 3, an unapplied method like this will be eta-expanded into a function. [quickfixable]
       Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=ret, version=2.13.3

friendly ping @SethTisue Is this a change that was intentionally introduced in Scala 2.13.15?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, see first bullet at https://github.com/scala/scala-dev/blob/e708445fd47b3637ac1c4fe54152e03999dbc1ec/2.13.15.md

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SethTisue ~

@panbingkun I think we should test

<arg>-Wconf:any:e</arg>
<arg>-Wconf:cat=deprecation:wv</arg>

for maven

and

"-Wconf:any:e",
"-Wconf:cat=deprecation:wv",

for sbt

"-Wconf:cat=deprecation&msg=shadowing a nested class of a parent is deprecated:w"
)
}
)
Expand Down
2 changes: 1 addition & 1 deletion ...cala/org/apache/spark/sql/execution/columnar/compression/CompressionSchemeBenchmark.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ object CompressionSchemeBenchmark extends BenchmarkBase with AllCompressionSchem

schemes.filter(_.supports(tpe)).foreach { scheme =>
val (compressFunc, compressionRatio, buf) = prepareEncodeInternal(count, tpe, scheme, input)
val label = s"${getFormattedClassName(scheme)}(${compressionRatio.formatted("%.3f")})"
val label = s"${getFormattedClassName(scheme)}(${"%.3f".format(compressionRatio)})"

benchmark.addCase(label)({ i: Int =>
for (n <- 0L until iters) {
Expand Down