Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-49746][BUILD] Upgrade Scala to 2.13.15 #48192

Closed
wants to merge 13 commits into from
Closed

[SPARK-49746][BUILD] Upgrade Scala to 2.13.15 #48192

wants to merge 13 commits into from

Conversation

panbingkun
Copy link
Contributor

@panbingkun panbingkun commented Sep 20, 2024 •
edited
Loading

What changes were proposed in this pull request?

The pr aims to upgrade scala from 2.13.14 to 2.13.15.

Why are the changes needed?

https://contributors.scala-lang.org/t/scala-2-13-15-release-planning/6649
image

Note: since 2.13.15, "-Wconf:cat=deprecation:wv,any:e" no longer takes effect and needs to be changed to "-Wconf:any:e", "-Wconf:cat=deprecation:wv", please refer to the details: scala/scala#10708

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No.

@panbingkun
Copy link
Contributor Author

panbingkun commented Sep 20, 2024 •
edited
Loading

For the record, this is blocked by:

@dongjoon-hyun
Copy link
Member

Thank you for tracking this, @panbingkun .

@panbingkun
Copy link
Contributor Author

compile warning deprecation

[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/TestUtils.scala:424:11: method write in class Files is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.TestUtils.createTempScriptWithExpectedOutput, origin=com.google.common.io.Files.write
[error]     Files.write(script, file, StandardCharsets.UTF_8)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:170:30: method getAllStatistics in class FileSystem is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.SparkHadoopUtil.getFSBytesReadOnThreadCallback.f.$anonfun, origin=org.apache.hadoop.fs.FileSystem.getAllStatistics
[error]     val f = () => FileSystem.getAllStatistics.asScala.map(_.getThreadStatistics.getBytesRead).sum
[error]                              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:201:34: method getAllStatistics in class FileSystem is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.SparkHadoopUtil.getFSBytesWrittenOnThreadCallback.threadStats, origin=org.apache.hadoop.fs.FileSystem.getAllStatistics
[error]     val threadStats = FileSystem.getAllStatistics.asScala.map(_.getThreadStatistics)
[error]                                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/history/BasicEventFilterBuilder.scala:160:13: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.BasicEventFilter._acceptFn, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]     case e: SparkListenerExecutorBlacklisted => liveExecutors.contains(e.executorId)
[error]             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/history/BasicEventFilterBuilder.scala:161:13: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.BasicEventFilter._acceptFn, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]     case e: SparkListenerExecutorUnblacklisted => liveExecutors.contains(e.executorId)
[error]             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/history/HistoryAppStatusStore.scala:75:40: value isBlacklisted in class ExecutorSummary is deprecated (since 3.1.0): use isExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.HistoryAppStatusStore.replaceExecutorLogs, origin=org.apache.spark.status.api.v1.ExecutorSummary.isBlacklisted, version=3.1.0
[error]       source.totalShuffleWrite, source.isBlacklisted, source.maxMemory, source.addTime,
[error]                                        ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/history/HistoryAppStatusStore.scala:77:14: value blacklistedInStages in class ExecutorSummary is deprecated (since 3.1.0): use excludedInStages instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.HistoryAppStatusStore.replaceExecutorLogs, origin=org.apache.spark.status.api.v1.ExecutorSummary.blacklistedInStages, version=3.1.0
[error]       source.blacklistedInStages, source.peakMemoryMetrics, source.attributes, source.resources,
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala:219:13: method append in class Files is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.worker.DriverRunner.runDriver.initialize, origin=com.google.common.io.Files.append
[error]       Files.append(header, stderr, StandardCharsets.UTF_8)
[error]             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala:194:13: method write in class Files is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor, origin=com.google.common.io.Files.write
[error]       Files.write(header, stderr, StandardCharsets.UTF_8)
[error]             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/executor/ExecutorSource.scala:36:16: method getAllStatistics in class FileSystem is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.executor.ExecutorSource.fileStats, origin=org.apache.hadoop.fs.FileSystem.getAllStatistics
[error]     FileSystem.getAllStatistics.asScala.find(s => s.getScheme.equals(scheme))
[error]                ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:121:7: method newTaskTempFile in class FileCommitProtocol is deprecated (since 3.3.0): use newTaskTempFile(..., spec: FileNameSpec) instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.internal.io.FileCommitProtocol.newTaskTempFile, origin=org.apache.spark.internal.io.FileCommitProtocol.newTaskTempFile, version=3.3.0
[error]       newTaskTempFile(taskContext, dir, spec.suffix)
[error]       ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:157:7: method newTaskTempFileAbsPath in class FileCommitProtocol is deprecated (since 3.3.0): use newTaskTempFileAbsPath(..., spec: FileNameSpec) instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.internal.io.FileCommitProtocol.newTaskTempFileAbsPath, origin=org.apache.spark.internal.io.FileCommitProtocol.newTaskTempFileAbsPath, version=3.3.0
[error]       newTaskTempFileAbsPath(taskContext, absoluteDir, spec.suffix)
[error]       ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:193:45: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onExecutorBlacklisted.event, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]   override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
[error]                                             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:202:14: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onExecutorBlacklistedForStage.event, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]       event: SparkListenerExecutorBlacklistedForStage): Unit = {
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:211:49: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onNodeBlacklistedForStage.event, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]   override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
[error]                                                 ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:219:47: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onExecutorUnblacklisted.event, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]   override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
[error]                                               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:228:41: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onNodeBlacklisted.event, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]   override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
[error]                                         ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:236:43: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onNodeUnblacklisted.event, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]   override def onNodeUnblacklisted(event: SparkListenerNodeUnblacklisted): Unit = {
[error]                                           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:120:28: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.applyExcludeOnFailureTimeout.$anonfun, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]           listenerBus.post(SparkListenerExecutorUnblacklisted(now, exec))
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:136:28: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.applyExcludeOnFailureTimeout.$anonfun, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]           listenerBus.post(SparkListenerNodeUnblacklisted(now, node))
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:240:28: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.updateExcludedForFetchFailure, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]           listenerBus.post(SparkListenerNodeBlacklisted(now, host, 1))
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:253:26: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.updateExcludedForFetchFailure, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]         listenerBus.post(SparkListenerExecutorBlacklisted(now, exec, 1))
[error]                          ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:290:26: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.updateExcludedForSuccessfulTaskSet.$anonfun, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]         listenerBus.post(SparkListenerExecutorBlacklisted(now, exec, newTotal))
[error]                          ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:309:28: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.updateExcludedForSuccessfulTaskSet.$anonfun, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]           listenerBus.post(SparkListenerNodeBlacklisted(now, node, excludedExecsOnNode.size))
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:545:28: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onExecutorBlacklisted.executorBlacklisted, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]       executorBlacklisted: SparkListenerExecutorBlacklisted): Unit = { }
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:550:36: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onExecutorBlacklistedForStage.executorBlacklistedForStage, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]       executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit = { }
[error]                                    ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:555:32: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onNodeBlacklistedForStage.nodeBlacklistedForStage, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]       nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit = { }
[error]                                ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:560:30: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onExecutorUnblacklisted.executorUnblacklisted, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]       executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit = { }
[error]                              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:565:24: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onNodeBlacklisted.nodeBlacklisted, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]       nodeBlacklisted: SparkListenerNodeBlacklisted): Unit = { }
[error]                        ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:570:26: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onNodeUnblacklisted.nodeUnblacklisted, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]       nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit = { }
[error]                          ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:66:41: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]       case executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage =>
[error]                                         ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:67:18: method onExecutorBlacklistedForStage in trait SparkListenerInterface is deprecated (since 3.1.0): use onExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onExecutorBlacklistedForStage, version=3.1.0
[error]         listener.onExecutorBlacklistedForStage(executorBlacklistedForStage)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:68:37: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]       case nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage =>
[error]                                     ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:69:18: method onNodeBlacklistedForStage in trait SparkListenerInterface is deprecated (since 3.1.0): use onNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onNodeBlacklistedForStage, version=3.1.0
[error]         listener.onNodeBlacklistedForStage(nodeBlacklistedForStage)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:70:33: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]       case executorBlacklisted: SparkListenerExecutorBlacklisted =>
[error]                                 ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:71:18: method onExecutorBlacklisted in trait SparkListenerInterface is deprecated (since 3.1.0): use onExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onExecutorBlacklisted, version=3.1.0
[error]         listener.onExecutorBlacklisted(executorBlacklisted)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:72:35: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]       case executorUnblacklisted: SparkListenerExecutorUnblacklisted =>
[error]                                   ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:73:18: method onExecutorUnblacklisted in trait SparkListenerInterface is deprecated (since 3.1.0): use onExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onExecutorUnblacklisted, version=3.1.0
[error]         listener.onExecutorUnblacklisted(executorUnblacklisted)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:74:29: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]       case nodeBlacklisted: SparkListenerNodeBlacklisted =>
[error]                             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:75:18: method onNodeBlacklisted in trait SparkListenerInterface is deprecated (since 3.1.0): use onNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onNodeBlacklisted, version=3.1.0
[error]         listener.onNodeBlacklisted(nodeBlacklisted)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:76:31: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]       case nodeUnblacklisted: SparkListenerNodeUnblacklisted =>
[error]                               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:77:18: method onNodeUnblacklisted in trait SparkListenerInterface is deprecated (since 3.1.0): use onNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onNodeUnblacklisted, version=3.1.0
[error]         listener.onNodeUnblacklisted(nodeUnblacklisted)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/TaskSetExcludeList.scala:150:11: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.TaskSetExcludelist.updateExcludedForFailedTask, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]           SparkListenerExecutorBlacklistedForStage(now, exec, numFailures, stageId, stageAttemptId))
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/TaskSetExcludeList.scala:161:15: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.TaskSetExcludelist.updateExcludedForFailedTask, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]               SparkListenerNodeBlacklistedForStage(now, host, numFailExec, stageId, stageAttemptId))
[error]               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:79:6: class AccessController in package security is deprecated (since 17)
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.serializer.SerializationDebugger.enableDebugging, origin=java.security.AccessController, version=17
[error]     !AccessController.doPrivileged(action).booleanValue()
[error]      ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:293:45: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onExecutorBlacklisted.event, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]   override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
[error]                                             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:302:14: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onExecutorBlacklistedForStage.event, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]       event: SparkListenerExecutorBlacklistedForStage): Unit = {
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:311:49: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onNodeBlacklistedForStage.event, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]   override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
[error]                                                 ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:334:47: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onExecutorUnblacklisted.event, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]   override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
[error]                                               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:342:41: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onNodeBlacklisted.event, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]   override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
[error]                                         ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:350:43: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onNodeUnblacklisted.event, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]   override def onNodeUnblacklisted(event: SparkListenerNodeUnblacklisted): Unit = {
[error]                                           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:400:35: value BLACKLISTED_EXECUTORS in class AppStatusSource is deprecated (since 3.1.0): use excludedExecutors instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.updateExecExclusionStatus.$anonfun, origin=org.apache.spark.status.AppStatusSource.BLACKLISTED_EXECUTORS, version=3.1.0
[error]         appStatusSource.foreach(_.BLACKLISTED_EXECUTORS.inc())
[error]                                   ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:403:35: value UNBLACKLISTED_EXECUTORS in class AppStatusSource is deprecated (since 3.1.0): use unexcludedExecutors instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.updateExecExclusionStatus.$anonfun, origin=org.apache.spark.status.AppStatusSource.UNBLACKLISTED_EXECUTORS, version=3.1.0
[error]         appStatusSource.foreach(_.UNBLACKLISTED_EXECUTORS.inc())
[error]                                   ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala:121:40: value isBlacklisted in class ExecutorSummary is deprecated (since 3.1.0): use isExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusStore.replaceDriverGcTime, origin=org.apache.spark.status.api.v1.ExecutorSummary.isBlacklisted, version=3.1.0
[error]       source.totalShuffleWrite, source.isBlacklisted, source.maxMemory, source.addTime,
[error]                                        ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala:123:14: value blacklistedInStages in class ExecutorSummary is deprecated (since 3.1.0): use excludedInStages instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusStore.replaceDriverGcTime, origin=org.apache.spark.status.api.v1.ExecutorSummary.blacklistedInStages, version=3.1.0
[error]       source.blacklistedInStages, source.peakMemoryMetrics, source.attributes, source.resources,
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/api/v1/api.scala:529:12: value holdingLocks in class ThreadStackTrace is deprecated (since 4.0.0): using synchronizers and monitors instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.api.v1.ThreadStackTrace.productElement, origin=org.apache.spark.status.api.v1.ThreadStackTrace.holdingLocks, version=4.0.0
[error] case class ThreadStackTrace(
[error]            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/protobuf/ExecutorStageSummarySerializer.scala:41:39: value isBlacklistedForStage in class ExecutorStageSummary is deprecated (since 3.1.0): use isExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.protobuf.ExecutorStageSummarySerializer.serialize.builder, origin=org.apache.spark.status.api.v1.ExecutorStageSummary.isBlacklistedForStage, version=3.1.0
[error]       .setIsBlacklistedForStage(input.isBlacklistedForStage)
[error]                                       ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/protobuf/ExecutorSummaryWrapperSerializer.scala:64:31: value isBlacklisted in class ExecutorSummary is deprecated (since 3.1.0): use isExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.protobuf.ExecutorSummaryWrapperSerializer.serializeExecutorSummary.builder, origin=org.apache.spark.status.api.v1.ExecutorSummary.isBlacklisted, version=3.1.0
[error]       .setIsBlacklisted(input.isBlacklisted)
[error]                               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/protobuf/ExecutorSummaryWrapperSerializer.scala:79:11: value blacklistedInStages in class ExecutorSummary is deprecated (since 3.1.0): use excludedInStages instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.protobuf.ExecutorSummaryWrapperSerializer.serializeExecutorSummary, origin=org.apache.spark.status.api.v1.ExecutorSummary.blacklistedInStages, version=3.1.0
[error]     input.blacklistedInStages.foreach { stage =>
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/util/collection/AppendOnlyMap.scala:211:45: method murmur3_32 in class Hashing is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.util.collection.AppendOnlyMap.rehash, origin=com.google.common.hash.Hashing.murmur3_32
[error]   private def rehash(h: Int): Int = Hashing.murmur3_32().hashInt(h).asInt()
[error]                                             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala:269:47: method murmur3_32 in class Hashing is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.util.collection.OpenHashSet.hashcode, origin=com.google.common.hash.Hashing.murmur3_32
[error]   private def hashcode(h: Int): Int = Hashing.murmur3_32().hashInt(h).asInt()
[error]                                               ^
[error] 61 errors found
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:156:12: method addAllPosArguments in class Builder is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.SparkSession.sql.$anonfun.cmd.$anonfun, origin=org.apache.spark.connect.proto.SqlCommand.Builder.addAllPosArguments
[error]           .addAllPosArguments(args.map(lit(_).expr).toImmutableArraySeq.asJava)))
[error]            ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:155:12: method setSql in class Builder is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.SparkSession.sql.$anonfun.cmd.$anonfun, origin=org.apache.spark.connect.proto.SqlCommand.Builder.setSql
[error]           .setSql(sqlText)
[error]            ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:188:14: method putAllNamedArguments in class Builder is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.SparkSession.sql.$anonfun.cmd.$anonfun, origin=org.apache.spark.connect.proto.SqlCommand.Builder.putAllNamedArguments
[error]             .putAllNamedArguments(args.asScala.map { case (k, v) => (k, lit(v).expr) }.asJava)))
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:187:14: method setSql in class Builder is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.SparkSession.sql.$anonfun.cmd.$anonfun, origin=org.apache.spark.connect.proto.SqlCommand.Builder.setSql
[error]             .setSql(sqlText)
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:101:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, path)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:105:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, path, source)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:112:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, source, options)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:126:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, source, options)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:134:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, source, schema, options)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:158:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, source, schema, options)
[error]           ^
[error] 10 errors found
[error] (core / Compile / compileIncremental) Compilation failed
[error] (connect-client-jvm / Compile / compileIncremental) Compilation failed
[error] Total time: 121 s (02:01), completed Sep 24, 2024, 3:39:40 PM

@panbingkun
Copy link
Contributor Author 

[error] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/compression/CompressionSchemeBenchmark.scala:94:73: method formatted in class StringFormat is deprecated (since 2.12.16): Use `formatString.format(value)` instead of `value.formatted(formatString)`,
[error] or use the `f""` string interpolator. In Java 15 and later, `formatted` resolves to the new method in String which has reversed parameters.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.execution.columnar.compression.CompressionSchemeBenchmark.runEncodeBenchmark.$anonfun.label, origin=scala.Predef.StringFormat.formatted, version=2.12.16
[error]       val label = s"${getFormattedClassName(scheme)}(${compressionRatio.formatted("%.3f")})"
[error]

@github-actions github-actions bot added the SQL label Sep 24, 2024
@panbingkun panbingkun marked this pull request as ready for review September 24, 2024 10:51
@panbingkun
Copy link
Contributor Author

cc @dongjoon-hyun @LuciferYang

@panbingkun
Copy link
Contributor Author

Let me run Maven's compilation again.

LuciferYang
LuciferYang reviewed Sep 24, 2024
View reviewed changes
project/SparkBuild.scala Outdated
"-Wconf:cat=other&site=org.dmg.pmml.*:w"
"-Wconf:cat=other&site=org.dmg.pmml.*:w",
// Some deprecated calls is a compilation error in Scala 2.13.15.
"-Wconf:cat=deprecation&msg=^(?=.*?method|value|object|trait|class|constructor)(?=.*?in)(?=.*?deprecated).+$:w",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should make this rule more specific, as its current scope of matching is too broad. Additionally, we should leave a TODO to remind us to fix these deprecated API usages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, let's go through Maven's test first.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spark/project/SparkBuild.scala

Line 237 in 73d6bd7

"-Wconf:cat=deprecation:wv,any:e",

I want to know if this line of configuration has become invalid in Scala 2.13.15？Here we have configured -Wconf:cat=deprecation:wv.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me verify it.

Copy link
Contributor

@LuciferYang LuciferYang Sep 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Scala 2.13.14 with -Wconf:cat=deprecation:wv,any:e, printed a warning message for the use of deprecated API
~/Tools/scala-2.13.14/bin/scala -Wconf:cat=deprecation:wv,any:e
Welcome to Scala 2.13.14 (OpenJDK 64-Bit Server VM, Java 17.0.12).
Type in expressions for evaluation. Or try :help.

scala> class Foo {
     |   def isEmpty(): Boolean = true
     | }
class Foo

scala> val foo = new Foo
val foo: Foo = Foo@706ddbc8

scala> val ret = foo.isEmpty
                     ^
       warning: Auto-application to `()` is deprecated. Supply the empty argument list `()` explicitly to invoke method isEmpty,
       or remove the empty argument list from its definition (Java-defined methods are exempt).
       In Scala 3, an unapplied method like this will be eta-expanded into a function. [quickfixable]
       Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=ret, version=2.13.3
val ret: Boolean = true
  • Scala 2.13.15 with -Wconf:cat=deprecation:wv,any:e, printed an error message for the use of deprecated API
~/Tools/scala-2.13.15/bin/scala -Wconf:cat=deprecation:wv,any:e
Welcome to Scala 2.13.15 (OpenJDK 64-Bit Server VM, Java 17.0.12).
Type in expressions for evaluation. Or try :help.

scala> class Foo {
     |   def isEmpty(): Boolean = true
     | }
class Foo

scala> val foo = new Foo
val foo: Foo = Foo@53f0d09c

scala> val ret = foo.isEmpty
                     ^
       error: Auto-application to `()` is deprecated. Supply the empty argument list `()` explicitly to invoke method isEmpty,
       or remove the empty argument list from its definition (Java-defined methods are exempt).
       In Scala 3, an unapplied method like this will be eta-expanded into a function. [quickfixable]
       Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=ret, version=2.13.3

friendly ping @SethTisue Is this a change that was intentionally introduced in Scala 2.13.15?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, see first bullet at https://github.com/scala/scala-dev/blob/e708445fd47b3637ac1c4fe54152e03999dbc1ec/2.13.15.md

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SethTisue ~

@panbingkun I think we should test

<arg>-Wconf:any:e</arg>
<arg>-Wconf:cat=deprecation:wv</arg>

for maven

and

"-Wconf:any:e",
"-Wconf:cat=deprecation:wv",

for sbt

@github-actions github-actions bot added the INFRA label Sep 24, 2024
@panbingkun
Copy link
Contributor Author

panbingkun commented Sep 24, 2024 •
edited
Loading

Currently, the testing based on sbt has passed: https://github.com/panbingkun/spark/runs/30569167386

@github-actions github-actions bot removed the SQL label Sep 24, 2024
@panbingkun panbingkun changed the title [WIP][SPARK-49746][BUILD] Upgrade Scala to 2.13.15 [SPARK-49746][BUILD] Upgrade Scala to 2.13.15 Sep 25, 2024
dongjoon-hyun
dongjoon-hyun reviewed Sep 25, 2024
View reviewed changes
.github/workflows/build_and_test.yml Outdated
name: "Run Maven Test"
permissions:
packages: write
uses: ./.github/workflows/maven_test.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you revert this leftover, @panbingkun ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

LuciferYang
LuciferYang reviewed Sep 25, 2024
View reviewed changes
pom.xml
@@ -3051,7 +3051,8 @@
<arg>-explaintypes</arg>
<arg>-release</arg>
<arg>17</arg>
<arg>-Wconf:cat=deprecation:wv,any:e</arg>
<arg>-Wconf:any:e</arg>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we should add some comments and highlight this change in the PR description.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

@github-actions github-actions bot removed the INFRA label Sep 25, 2024
LuciferYang
LuciferYang reviewed Sep 25, 2024
View reviewed changes
Copy link
Contributor

@LuciferYang LuciferYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems fine to me

dongjoon-hyun
dongjoon-hyun approved these changes Sep 25, 2024
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @panbingkun , @SethTisue and @LuciferYang .
Merged to master for Apache Spark 4.0.0.

attilapiros pushed a commit to attilapiros/spark that referenced this pull request Oct 4, 2024
@panbingkun @LuciferYang @attilapiros
[SPARK-49746][BUILD] Upgrade Scala to 2.13.15
2c71608 
### What changes were proposed in this pull request?
The pr aims to upgrade `scala` from `2.13.14` to `2.13.15`.

### Why are the changes needed?
https://contributors.scala-lang.org/t/scala-2-13-15-release-planning/6649
<img width="770" alt="image" src="https://github.com/user-attachments/assets/277cfdb4-8542-42fe-86e5-ad72ca2bba4c">

**Note: since 2.13.15, "-Wconf:cat=deprecation:wv,any:e" no longer takes effect and needs to be changed to "-Wconf:any:e", "-Wconf:cat=deprecation:wv", please refer to the details: scala/scala#10708

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#48192 from panbingkun/SPARK-49746.

Lead-authored-by: panbingkun <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
himadripal pushed a commit to himadripal/spark that referenced this pull request Oct 19, 2024
@panbingkun @LuciferYang
[SPARK-49746][BUILD] Upgrade Scala to 2.13.15
f685b64 
### What changes were proposed in this pull request?
The pr aims to upgrade `scala` from `2.13.14` to `2.13.15`.

### Why are the changes needed?
https://contributors.scala-lang.org/t/scala-2-13-15-release-planning/6649
<img width="770" alt="image" src="https://github.com/user-attachments/assets/277cfdb4-8542-42fe-86e5-ad72ca2bba4c">

**Note: since 2.13.15, "-Wconf:cat=deprecation:wv,any:e" no longer takes effect and needs to be changed to "-Wconf:any:e", "-Wconf:cat=deprecation:wv", please refer to the details: scala/scala#10708

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Pass GA.

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes apache#48192 from panbingkun/SPARK-49746.

Lead-authored-by: panbingkun <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SethTisue SethTisue SethTisue left review comments

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

@LuciferYang LuciferYang LuciferYang approved these changes

Assignees
No one assigned
Labels
BUILD DOCS
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@panbingkun @dongjoon-hyun @SethTisue @LuciferYang