New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-49552][PYTHON] Add DataFrame API support for new 'randstr' and 'uniform' SQL functions #48143

Open

dtenedor wants to merge 28 commits into apache:master from dtenedor:dataframes-uniform-randstr

+329 −10

Contributor

dtenedor commented Sep 18, 2024 •

edited

Loading

What changes were proposed in this pull request?

In #48004 we added new SQL functions randstr and uniform. This PR adds DataFrame API support for them.

For example, in Scala:

sql("create table t(col int not null) using csv")
sql("insert into t values (0)")
val df = sql("select col from t")
df.select(randstr(lit(5), lit(0)).alias("x")).select(length(col("x")))
> 5

df.select(uniform(lit(10), lit(20), lit(0)).alias("x")).selectExpr("x > 5")
> true

Why are the changes needed?

This improves DataFrame parity with the SQL API.

Does this PR introduce any user-facing change?

Yes, see above.

How was this patch tested?

This PR adds unit test coverage.

Was this patch authored or co-authored using generative AI tooling?

No.

dtenedor added 20 commits

September 5, 2024 15:40


          commit

d9223e5

commit

uniform expression

commit

commit

commit


          respond to code review comments

c0a2551


          respond to code review comments

f6ffde0


          respond to code review comments

c22fef2


          fix test

c78d8f0


          fix test

5b6194e


          fix test

da390f9


          commit

836ec8e

commit


          commit

4b41b34


          respond to code review comments

54a1a2e


          fix function description

c372075


          fix test

515b046


          fix test


          respond to code review comments

respond to code review comments

respond to code review comments

respond to code review comments


          respond to code review comments

7ec25a8


          fix RandomSuite

1f5e866


          respond to code review comments

8b64f33


          commit

0313fbe


          sync

d54eec7


          sync

48f1dde

github-actions bot added SQL DOCS PYTHON CONNECT labels


          commit

f496e86

dtenedor changed the title ~~[WIP][SPARK-49552][Python] Add DataFrame API support for new 'randstr' and 'uniform' SQL functions~~ [SPARK-49552][Python] Add DataFrame API support for new 'randstr' and 'uniform' SQL functions

dtenedor marked this pull request as ready for review

September 18, 2024 15:59

Contributor Author

dtenedor commented Sep 18, 2024

cc @HyukjinKwon @MaxGekk here is the DataFrame support for the new randstr and uniform functions :)

HyukjinKwon changed the title ~~[SPARK-49552][Python] Add DataFrame API support for new 'randstr' and 'uniform' SQL functions~~ [SPARK-49552][PYTHON] Add DataFrame API support for new 'randstr' and 'uniform' SQL functions

zhengruifeng reviewed

View reviewed changes

python/pyspark/sql/connect/functions/builtin.py Outdated Show resolved Hide resolved

python/pyspark/sql/connect/functions/builtin.py Outdated Show resolved Hide resolved

python/pyspark/sql/functions/builtin.py Outdated Show resolved Hide resolved

python/pyspark/sql/functions/builtin.py Show resolved Hide resolved

python/pyspark/sql/functions/builtin.py Outdated Show resolved Hide resolved

zhengruifeng reviewed

View reviewed changes

python/pyspark/sql/functions/builtin.py Outdated Show resolved Hide resolved


          respond to code review comments

160a5f4

dtenedor commented

View reviewed changes

Contributor Author

dtenedor left a comment

Thanks @zhengruifeng for your review! Responded to your comments, please take another look.

python/pyspark/sql/functions/builtin.py Show resolved Hide resolved

python/pyspark/sql/connect/functions/builtin.py Outdated Show resolved Hide resolved

python/pyspark/sql/connect/functions/builtin.py Outdated Show resolved Hide resolved

python/pyspark/sql/functions/builtin.py Outdated Show resolved Hide resolved

python/pyspark/sql/functions/builtin.py Outdated Show resolved Hide resolved

python/pyspark/sql/functions/builtin.py Outdated Show resolved Hide resolved

dtenedor requested a review from zhengruifeng

September 19, 2024 10:14

dtenedor added 2 commits

September 19, 2024 15:13


          fix test

795f5a6

fix

6bb123f

xinrong-meng reviewed

View reviewed changes

python/pyspark/sql/functions/builtin.py Outdated

+                  +------+
+                  | ceV0P|
+                  +------+

Member

xinrong-meng Sep 20, 2024

nit: we normally don't include an empty line at the end of the docstring

Contributor Author

dtenedor Sep 20, 2024

Sounds good, this is done.

xinrong-meng reviewed

View reviewed changes

python/pyspark/sql/functions/builtin.py Outdated

+                  +------+
+                  |     7|
+                  +------+

Member

xinrong-meng Sep 20, 2024

ditto

Contributor Author

dtenedor Sep 20, 2024

Sounds good, this is done.

dtenedor added 4 commits

September 20, 2024 13:59


          commit

05a674c


          fix test

fd3262f

fix

4929fbd

fix

ef7f9e7

dtenedor requested a review from xinrong-meng

September 20, 2024 15:26

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CONNECT DOCS PYTHON SQL