Skip to content

Conversation

@Shubhambhusate
Copy link

@Shubhambhusate Shubhambhusate commented Dec 19, 2025

What changes were proposed in this pull request?

Changes made:

Added new error condition in error-conditions.json:
UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF - A clear error message for SQL UDFs used in lambda functions.

Why are the changes needed?

Currently, when a SQL UDF is used inside a higher-order function like transform, the error message is confusing:


CREATE FUNCTION lower_udf(s STRING) RETURNS STRING RETURN lower(s);
SELECT transform(array('A', 'B'), x -> lower_udf(x));

Before (confusing error):

[MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT]
Resolved attribute(s) "x" missing from in operator !Project [cast(lambda x#20395 as string) AS s#20397].
SQLSTATE: XX000

Screenshot 2025-12-18 at 6 13 29 PM

This error doesn't explain why the attribute is missing or what the user should do.

After (clear error):

[UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF] The feature is not supported: Lambda function with SQL UDF "spark_catalog.default.lower_udf(lambda x)" in a higher order function. SQLSTATE: 0A000

Screenshot 2025-12-18 at 6 14 11 PM

This is consistent with the existing error message for Python UDFs in the same scenario (UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_PYTHON_UDF).

Does this PR introduce any user-facing change?

Yes. Users will now see a clearer, more actionable error message when attempting to use a SQL UDF inside a higher-order function's lambda expression.

How was this patch tested?

Test 1:

Added a new test case "SQL UDF in higher-order function should fail with clear error message" in SQLFunctionSuite.scala that:
Creates a SQL UDF
Attempts to use it in a transform higher-order function
Verifies the error condition is UNSUPPORTED_FEATURE.LAMBDA_FUNCTION_WITH_SQL_UDF
Verifies the error message contains the function name and lambda x

Test 2:
Manual testing
spark.sql("CREATE OR REPLACE FUNCTION test_lower_udf(s STRING) RETURNS STRING RETURN lower(s)") spark.sql("SELECT transform(array('A', 'B'), x -> test_lower_udf(x))").show()

Was this patch authored or co-authored using generative AI tooling?

No

@github-actions github-actions bot added the SQL label Dec 19, 2025
@Shubhambhusate Shubhambhusate changed the title Improved the logs message regarding lambda function with SQL UDF [SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDFImproved the logs message regarding lambda function with SQL UDF Dec 19, 2025
@Shubhambhusate Shubhambhusate force-pushed the fix/LAMBDA_FUNCTION_WITH_SQL_UDF branch from e87c41c to 6f50c79 Compare December 19, 2025 18:31
@Shubhambhusate Shubhambhusate changed the title [SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDFImproved the logs message regarding lambda function with SQL UDF [SPARK-54776][SQL] Improved the logs message regarding lambda function with SQL UDF Dec 20, 2025
@Shubhambhusate Shubhambhusate force-pushed the fix/LAMBDA_FUNCTION_WITH_SQL_UDF branch from 6f50c79 to 5d61f52 Compare December 21, 2025 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant