Skip to content

Commit

Permalink
[SPARK-49583][SQL] Define the error sub-condition SECONDS_FRACTION
Browse files Browse the repository at this point in the history
…for invalid seconds fraction pattern

### What changes were proposed in this pull request?
In the PR, I propose new sub-condition `SECONDS_FRACTION` of the error condition `INVALID_DATETIME_PATTERN` in case when datetime pattern doesn't contain proper seconds fraction of variable length.

### Why are the changes needed?
To fix the failure on internal assert. This change should improve user experience with Spark SQL.

Before the changes, Spark fails while parsing the datetime patterns like `\nSSSS\r`:
```
java.lang.AssertionError: assertion failed
	at scala.Predef$.assert(Predef.scala:264)
	at org.apache.spark.ErrorClassesJsonReader.getMessageTemplate(ErrorClassesJSONReader.scala:91)
	at org.apache.spark.ErrorClassesJsonReader.getErrorMessage(ErrorClassesJSONReader.scala:46)
```

### Does this PR introduce _any_ user-facing change?
Should not. Only if user's code depends on the format of error message.

After changes, users get the error:
```
org.apache.spark.SparkIllegalArgumentException: [INVALID_DATETIME_PATTERN.SECONDS_FRACTION] Unrecognized datetime pattern:
. Cannot detect a seconds fraction pattern of variable length. Please make sure the pattern contains 'S', and does not contain illegal characters. SQLSTATE: 22007
```

### How was this patch tested?
By running new test suites:
```
$ build/sbt "test:testOnly *org.apache.spark.sql.catalyst.util.DateTimeFormatterHelperSuite"
```

### Was this patch authored or co-authored using generative AI tooling?
No.

Closes #48058 from MaxGekk/fix-INVALID_DATETIME_PATTERN.

Authored-by: Max Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
  • Loading branch information
MaxGekk committed Sep 10, 2024
1 parent ab7aea1 commit 7355864
Show file tree
Hide file tree
Showing 3 changed files with 16 additions and 1 deletion.
5 changes: 5 additions & 0 deletions common/utils/src/main/resources/error/error-conditions.json
Original file line number Diff line number Diff line change
Expand Up @@ -2092,6 +2092,11 @@
"message" : [
"Too many letters in datetime pattern: <pattern>. Please reduce pattern length."
]
},
"SECONDS_FRACTION" : {
"message" : [
"Cannot detect a seconds fraction pattern of variable length. Please make sure the pattern contains 'S', and does not contain illegal characters."
]
}
},
"sqlState" : "22007"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ private object DateTimeFormatterHelper {
rest = suffix
case _ =>
throw new SparkIllegalArgumentException(
errorClass = "INVALID_DATETIME_PATTERN",
errorClass = "INVALID_DATETIME_PATTERN.SECONDS_FRACTION",
messageParameters = Map("pattern" -> pattern))
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,4 +83,14 @@ class DateTimeFormatterHelperSuite extends SparkFunSuite {
assert(convertIncompatiblePattern("yyyy-MM-dd'e'HH:mm:ss") === "uuuu-MM-dd'e'HH:mm:ss")
assert(convertIncompatiblePattern("yyyy-MM-dd'T'") === "uuuu-MM-dd'T'")
}

test("SPARK-49583: invalid var length second fraction") {
val pattern = "\nSSSS\r"
checkError(
exception = intercept[SparkIllegalArgumentException] {
createBuilderWithVarLengthSecondFraction(pattern)
},
errorClass = "INVALID_DATETIME_PATTERN.SECONDS_FRACTION",
parameters = Map("pattern" -> pattern))
}
}

0 comments on commit 7355864

Please sign in to comment.