Skip to content

Commit 3ef529b

Browse files
LuciferYangdongjoon-hyun
authored andcommitted
[SPARK-49014][BUILD][DOCS][FOLLOWUP] Synchronize the Avro version in SparkBuild.scala/docs/comments to 1.12.0
### What changes were proposed in this pull request? #47498 has upgraded Avro to version 1.12.0, Synchronize the Avro version in `SparkBuild.scala/docs/comments to` 1.12.0. ### Why are the changes needed? The Avro version within the project should be consistent. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes #49552 from LuciferYang/SPARK-49014-FOLLOWUP. Authored-by: yangjie01 <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 4e47023 commit 3ef529b

File tree

4 files changed

+6
-6
lines changed

4 files changed

+6
-6
lines changed

docs/sql-data-sources-avro.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -477,7 +477,7 @@ Submission Guide for more details.
477477
[adm]: submitting-applications.html#advanced-dependency-management
478478

479479
## Supported types for Avro -> Spark SQL conversion
480-
Currently Spark supports reading all [primitive types](https://avro.apache.org/docs/1.11.3/specification/#primitive-types) and [complex types](https://avro.apache.org/docs/1.11.3/specification/#complex-types) under records of Avro.
480+
Currently Spark supports reading all [primitive types](https://avro.apache.org/docs/1.12.0/specification/#primitive-types) and [complex types](https://avro.apache.org/docs/1.12.0/specification/#complex-types) under records of Avro.
481481
<table>
482482
<thead><tr><th><b>Avro type</b></th><th><b>Spark SQL type</b></th></tr></thead>
483483
<tr>
@@ -541,7 +541,7 @@ In addition to the types listed above, it supports reading `union` types. The fo
541541
3. `union(something, null)`, where something is any supported Avro type. This will be mapped to the same Spark SQL type as that of something, with nullable set to true.
542542
All other union types are considered complex. They will be mapped to StructType where field names are member0, member1, etc., in accordance with members of the union. This is consistent with the behavior when converting between Avro and Parquet.
543543

544-
It also supports reading the following Avro [logical types](https://avro.apache.org/docs/1.11.3/specification/#logical-types):
544+
It also supports reading the following Avro [logical types](https://avro.apache.org/docs/1.12.0/specification/#logical-types):
545545

546546
<table>
547547
<thead><tr><th><b>Avro logical type</b></th><th><b>Avro type</b></th><th><b>Spark SQL type</b></th></tr></thead>

project/SparkBuild.scala

+1-1
Original file line numberDiff line numberDiff line change
@@ -1061,7 +1061,7 @@ object DependencyOverrides {
10611061
lazy val settings = Seq(
10621062
dependencyOverrides += "com.google.guava" % "guava" % guavaVersion,
10631063
dependencyOverrides += "jline" % "jline" % "2.14.6",
1064-
dependencyOverrides += "org.apache.avro" % "avro" % "1.11.3")
1064+
dependencyOverrides += "org.apache.avro" % "avro" % "1.12.0")
10651065
}
10661066

10671067
/**

sql/core/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala

+2-2
Original file line numberDiff line numberDiff line change
@@ -82,14 +82,14 @@ private[sql] class AvroOptions(
8282

8383
/**
8484
* Top level record name in write result, which is required in Avro spec.
85-
* See https://avro.apache.org/docs/1.11.3/specification/#schema-record .
85+
* See https://avro.apache.org/docs/1.12.0/specification/#schema-record .
8686
* Default value is "topLevelRecord"
8787
*/
8888
val recordName: String = parameters.getOrElse(RECORD_NAME, "topLevelRecord")
8989

9090
/**
9191
* Record namespace in write result. Default value is "".
92-
* See Avro spec for details: https://avro.apache.org/docs/1.11.3/specification/#schema-record .
92+
* See Avro spec for details: https://avro.apache.org/docs/1.12.0/specification/#schema-record .
9393
*/
9494
val recordNamespace: String = parameters.getOrElse(RECORD_NAMESPACE, "")
9595

sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala

+1-1
Original file line numberDiff line numberDiff line change
@@ -814,7 +814,7 @@ class HiveClientSuite(version: String) extends HiveVersionSuite(version) {
814814
test("Decimal support of Avro Hive serde") {
815815
val tableName = "tab1"
816816
// TODO: add the other logical types. For details, see the link:
817-
// https://avro.apache.org/docs/1.11.3/specification/#logical-types
817+
// https://avro.apache.org/docs/1.12.0/specification/#logical-types
818818
val avroSchema =
819819
"""{
820820
| "name": "test_record",

0 commit comments

Comments
 (0)