[SPARK-51419][SQL] Get hours of TIME datatype #50355

senthh · 2025-03-23T06:10:07Z

What changes were proposed in this pull request?

This PR adds support for extracting the hour component from TIME (TimeType) values in Spark SQL.

scala> spark.sql("SELECT hour(TIME'07:01:09.12312321231232');").show()
+----------------------------+
|hour(TIME '07:01:09.123123')|
+----------------------------+
|                           7|
+----------------------------+

scala> spark.sql("SELECT hour('2009-07-30 12:58:59')").show()
+-------------------------+
|hour(2009-07-30 12:58:59)|
+-------------------------+
|                       12|
+-------------------------+

Why are the changes needed?

Spark previously supported hour() for only TIMESTAMP type values. TIME support was missing, leading to implicit casting attempt to TIMESTAMP, which was incorrect. This PR ensures that hour(TIME'HH:MM:SS.######') behaves correctly without unnecessary type coercion.

Does this PR introduce any user-facing change?

Yes

Before this PR, calling hour(TIME'HH:MM:SS.######') resulted in a type mismatch error or an implicit cast attempt to TIMESTAMP, which was incorrect.
With this PR, hour(TIME'HH:MM:SS.######') now works correctly for TIME values without implicit casting.
Users can now extract the hour component from TIME values natively.

How was this patch tested?

By running new tests:

$ build/sbt "test:testOnly *TimeExpressionsSuite"

Was this patch authored or co-authored using generative AI tooling?

No

senthh · 2025-03-23T06:16:37Z

Hi @MaxGekk

Could you please review this PR?

MaxGekk

Could you fix the output of the example:

scala> spark.sql("SELECT hour(TIME'07:01:09.12312321231232');").show()
+------------------------------+
|minute(TIME '07:01:09.123123')|
+------------------------------+
| 7|
+------------------------------+

should be hour not minute.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala

...catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeExpressionsSuite.scala

senthh · 2025-03-23T11:43:40Z

Could you fix the output of the example:

scala> spark.sql("SELECT hour(TIME'07:01:09.12312321231232');").show()
+------------------------------+
|minute(TIME '07:01:09.123123')|
+------------------------------+
| 7|
+------------------------------+

should be hour not minute.

Yes corrected

senthh · 2025-03-23T12:04:53Z

Modified as per review feedback @MaxGekk , Please check it looks good

senthh · 2025-03-23T17:03:58Z

ExpressionsSchemaSuite failled with below error

 - Check schemas for expression examples *** FAILED *** (444 milliseconds)
[info]   "SELECT hour('20[09-07-30] 12:58:59')" did not equal "SELECT hour('20[18-02-14] 12:58:59')" SQL query did not match (ExpressionsSchemaSuite.scala:190)
[info]   Analysis:
[info]   "SELECT hour('20[09-07-30] 12:58:59')" -> "SELECT hour('20[18-02-14] 12:58:59')"

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala

sql/core/src/test/resources/sql-functions/sql-expression-schema.md

MaxGekk · 2025-03-24T17:46:06Z

...catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeExpressionsSuite.scala

+
+  test("Hour with TIME type") {
+    // A few test times in microseconds since midnight:
+    //   time in microseconds -> expected minute


Suggested change

// time in microseconds -> expected minute

// time in microseconds -> expected hours

...catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeExpressionsSuite.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala

...catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeExpressionsSuite.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala

beliefer

Overall, LGTM except @MaxGekk 's comments.

…ionBuilder

…Builder in TimeExpressionSuite

…rExpressionBuilder" This reverts commit e24c07f.

…d at Hour

MaxGekk

lgtm except of a couple comments.

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala

MaxGekk · 2025-03-26T06:10:53Z

The test failure is not related to the changes, I believe:

[info] - SPARK-51097: Verify snapshot lag metrics are updated correctly with RocksDBStateStoreProvider (with changelog checkpointing) *** FAILED *** (11 seconds, 275 milliseconds)
[info]   Assert on query failed: Execute: The code passed to eventually never returned normally. Attempted 667 times over 10.010695936 seconds. Last failure message: instanceMetrics.forall(((x$2: (String, Long)) => x$2._2.==(2))) was false.

+1, LGTM. Merging to master.
Thank you, @senthh and @HyukjinKwon @beliefer for review.

senthh · 2025-03-26T06:19:57Z

@MaxGekk @HyukjinKwon @beliefer Thank you for patiently reviewing my PR

### What changes were proposed in this pull request? This PR adds support for extracting the hour component from TIME (TimeType) values in Spark SQL. ``` scala> spark.sql("SELECT hour(TIME'07:01:09.12312321231232');").show() +----------------------------+ |hour(TIME '07:01:09.123123')| +----------------------------+ | 7| +----------------------------+ scala> spark.sql("SELECT hour('2009-07-30 12:58:59')").show() +-------------------------+ |hour(2009-07-30 12:58:59)| +-------------------------+ | 12| +-------------------------+ ``` ### Why are the changes needed? Spark previously supported hour() for only TIMESTAMP type values. TIME support was missing, leading to implicit casting attempt to TIMESTAMP, which was incorrect. This PR ensures that `hour(TIME'HH:MM:SS.######')` behaves correctly without unnecessary type coercion. ### Does this PR introduce _any_ user-facing change? Yes - Before this PR, calling hour(TIME'HH:MM:SS.######') resulted in a type mismatch error or an implicit cast attempt to TIMESTAMP, which was incorrect. - With this PR, hour(TIME'HH:MM:SS.######') now works correctly for TIME values without implicit casting. - Users can now extract the hour component from TIME values natively. ### How was this patch tested? By running new tests: ```$ build/sbt "test:testOnly *TimeExpressionsSuite"``` ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#50355 from senthh/getHour. Authored-by: senthh <[email protected]> Signed-off-by: Max Gekk <[email protected]>

MaxGekk · 2025-04-09T12:51:39Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala

+    Seq(child.dataType)
+  )
+
+  override def inputTypes: Seq[AbstractDataType] = Seq(TimeType())


@senthh Could you open a follow up PR and allow any valid precision of the TIME type other it fails now with the error:

spark-sql (default)> select hour(cast('12:30' as time(0))); [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "hour(CAST(12:30 AS TIME(0)))" due to data type mismatch: The first parameter requires the "TIME(6)" type, however "CAST(12:30 AS TIME(0))" has the type "TIME(0)". SQLSTATE: 42K09; line 1 pos 7;

@MaxGekk Sure Sure

…sion of TIME type ### What changes were proposed in this pull request? This is followup PR of [SPARK-51419 ](#50355) ### Why are the changes needed? This Followup PR allows any precision in the range of [0,6] for the hour function ### Does this PR introduce _any_ user-facing change? Yes. This changes allows User to execute below query ``` spark.sql("select hour(cast('12:00:01.123' as time(3)))").show(false) ``` ### How was this patch tested? We tested by running sample query as below ``` spark.sql("select hour(cast('12:00:01.123' as time(3)))").show(false) ``` Output: ``` +-----------------------------------+ |hour(CAST(12:00:01.123 AS TIME(3)))| +-----------------------------------+ |12 | +-----------------------------------+ ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes #50554 from senthh/SPARK-51419_followup. Authored-by: senthh <[email protected]> Signed-off-by: Max Gekk <[email protected]>

github-actions bot added the SQL label Mar 23, 2025

MaxGekk requested changes Mar 23, 2025

View reviewed changes

senthh requested a review from MaxGekk March 23, 2025 11:44

HyukjinKwon reviewed Mar 24, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala Outdated Show resolved Hide resolved

senthh requested a review from HyukjinKwon March 24, 2025 06:22

MaxGekk requested changes Mar 24, 2025

View reviewed changes

senthh added 4 commits March 25, 2025 08:44

[SPARK-51419][SQL] Get hour of TIME datatype

7aebf83

[SPARK-51419][SQL] Get hour of TIME datatype

cffb62c

[SPARK-51419][SQL] Get hour of TIME datatype

49f5e9e

[SPARK-51419][SQL] Generated new GoldenFile sql-expression-schema.md

3db3f5d

senthh force-pushed the getHour branch from e369895 to 3db3f5d Compare March 25, 2025 03:19

[SPARK-51419][SQL] Modified as minute into hour in comment sections

6fdf339

senthh requested a review from MaxGekk March 25, 2025 03:33

MaxGekk requested changes Mar 25, 2025

View reviewed changes

...catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/TimeExpressionsSuite.scala Outdated Show resolved Hide resolved

MaxGekk changed the title ~~[SPARK-51419][SQL] Get hour of TIME datatype~~ [SPARK-51419][SQL] Get hours of TIME datatype Mar 25, 2025

HyukjinKwon reviewed Mar 25, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala Outdated Show resolved Hide resolved

beliefer approved these changes Mar 25, 2025

View reviewed changes

[SPARK-51419][SQL] Removing as ExpressionDescription from HourExpress…

e24c07f

…ionBuilder

senthh requested review from MaxGekk, beliefer and HyukjinKwon March 25, 2025 11:32

senthh added 3 commits March 25, 2025 17:13

[SPARK-51419][SQL] Chenged MinutesExpressionBuilder to HourExpression…

e294345

…Builder in TimeExpressionSuite

Revert "[SPARK-51419][SQL] Removing as ExpressionDescription from Hou…

a48c6e1

…rExpressionBuilder" This reverts commit e24c07f.

[SPARK-51419][SQL] Removing ExpressionDescription from HoursOfTime an…

5d5707b

…d at Hour

MaxGekk reviewed Mar 25, 2025

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala Outdated Show resolved Hide resolved

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala Outdated Show resolved Hide resolved

HyukjinKwon approved these changes Mar 26, 2025

View reviewed changes

[SPARK-51419][SQL] Removed "scalastyle:on line.size.limit" lines

13d113a

senthh requested a review from MaxGekk March 26, 2025 02:01

MaxGekk approved these changes Mar 26, 2025

View reviewed changes

MaxGekk closed this in 8a50b0f Mar 26, 2025

MaxGekk reviewed Apr 9, 2025

View reviewed changes

senthh mentioned this pull request Apr 10, 2025

[SPARK-51419][SQL][FOLLOWUP] Making hour function to accept any precision of TIME type #50554

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-51419][SQL] Get hours of TIME datatype #50355

[SPARK-51419][SQL] Get hours of TIME datatype #50355

senthh commented Mar 23, 2025 •

edited by MaxGekk

Loading

senthh commented Mar 23, 2025

MaxGekk left a comment

senthh commented Mar 23, 2025

senthh commented Mar 23, 2025

senthh commented Mar 23, 2025

MaxGekk Mar 24, 2025

beliefer left a comment

MaxGekk left a comment

MaxGekk commented Mar 26, 2025

senthh commented Mar 26, 2025

MaxGekk Apr 9, 2025

senthh Apr 10, 2025

	// time in microseconds -> expected minute
	// time in microseconds -> expected hours

[SPARK-51419][SQL] Get hours of TIME datatype #50355

[SPARK-51419][SQL] Get hours of TIME datatype #50355

Conversation

senthh commented Mar 23, 2025 • edited by MaxGekk Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

senthh commented Mar 23, 2025

MaxGekk left a comment

Choose a reason for hiding this comment

senthh commented Mar 23, 2025

senthh commented Mar 23, 2025

senthh commented Mar 23, 2025

MaxGekk Mar 24, 2025

Choose a reason for hiding this comment

beliefer left a comment

Choose a reason for hiding this comment

MaxGekk left a comment

Choose a reason for hiding this comment

MaxGekk commented Mar 26, 2025

senthh commented Mar 26, 2025

MaxGekk Apr 9, 2025

Choose a reason for hiding this comment

senthh Apr 10, 2025

Choose a reason for hiding this comment

senthh commented Mar 23, 2025 •

edited by MaxGekk

Loading