feat(isthmus): add dynamic function conversion for Substrait<->Calcite #457

ZorinAnton · 2025-08-05T12:58:14Z

This PR introduces support for dynamic scalar User-Defined Functions (UDFs) for round-trip conversion between Substrait plans and Calcite RelNodes.

It establishes the Substrait YAML extension files as the single source of truth (SSoT) for all function signatures. The primary addition is a new utility, YamlToSqlOperator, which acts as a bridge between the Substrait and Calcite worlds by dynamically generating Calcite SqlOperator objects from the parsed YAML definitions.

The implementation is optimized to only generate these operators for functions that are truly dynamic —that is, functions present in the provided extensions but not already defined in the default SubstraitOperatorTable.

Key Changes:

New YamlToSqlOperator Utility: A type-safe converter that translates SimpleExtension.ExtensionCollection objects into a List.
Enhanced SqlToSubstrait (SQL → Substrait): The SqlToSubstrait constructor is updated to:
- Identify dynamic UDFs by filtering the loaded extensions against the known operators in SubstraitOperatorTable.
- Use YamlToSqlOperator to generate operators for these dynamic UDFs.
- Inject the new operators into Calcite's validator via a сhained SqlOperatorTable.
Enhanced SubstraitToCalcite (Substrait → SQL): The SubstraitRelNodeConverter is similarly enhanced to:
- Also identify dynamic UDFs from the provided SimpleExtension.ExtensionCollection.
- Dynamically create FunctionMappings.Sig objects for these UDFs, enabling the ScalarFunctionConverter to correctly map them back to their SqlOperator during plan conversion.
Architectural Refactoring: The EXTENSION_COLLECTION in SqlConverterBase has been changed from static final to an instance field to allow for the dynamic configuration required by this feature.

vbarua

I started taking a look at this today, but didn't have time to finish.

I'm going to restate what I think your doing with these changes to make sure that we're on the same page.

To start, you're generating SqlOperators dynamically from the function definitions in SimpleExtensions. You then use these to generate a SqlOperatorTable that covers functions that aren't already in the SubstraitOperatorTable.

This let's you do things like parse SQL queries with functions that are defined only in the SimpleExtension, and aren't part of the SubstraitOperatorTable.

With these definitions, you can also add additional mappings when converting from Substrait to Calcite, and back, for functions that aren't explicitly mapped in the static SCALAR_SIGS mapping.

Meta Comment

I find the usage of "custom" for this concept somewhat confusing. These aren't really custom functions because they are part of the core Substrait spec. A better word for this might be "dynamic". In my head, you're effectively generating SqlOperators dynamically based on the Substrait definitions, and then using them as fall-backs when explicit mappings.

isthmus/src/main/java/io/substrait/isthmus/sql/SubstraitSqlValidator.java

isthmus/src/main/java/io/substrait/isthmus/YamlToSqlOperator.java

isthmus/src/main/java/io/substrait/isthmus/SubstraitToSql.java

vbarua · 2025-08-08T00:42:43Z

isthmus/src/main/java/io/substrait/isthmus/SqlToSubstrait.java

+    SimpleExtension.ExtensionCollection customExtensionCollection =
+        ExtensionUtils.getCustomExtensions(extensions);
+    List<SqlOperator> generatedCustomOperators =
+        YamlToSqlOperator.from(customExtensionCollection, this.factory);


Do we actually need to distinguish between the SqlOperators defined explicitly in the SubstraitOperatorTable and those that you are generating here?

I think we can just generate mappings for every function found in the extension collection, and then chain the operator tables like:

SqlOperatorTables.chain( SubstraitOperatorTable.INSTANCE, SqlOperatorTables.of(generatedCustomOperators) );

we automatically give precedence to the non-custom ones.

I'm not sure that precedence should be given the to default (non-custom) operators. What if users want to override them?

If I understood correctly, you are suggesting to generate SqlOperators for all extensions, the default ones and the dynamic:

List<SqlOperator> generatedCustomOperators = SimpleExtensionToSqlOperator.from(dynamicExtensionCollection, this.factory);

I see two issues here:

duplicated work (default operators already exist in SubstraitOperatorTable.INSTANCE

it requires the fully fledged TypeExpressionEvaluator that will probably be needed at some point anyway, but I guess this should be done in a different PR

I'm not sure that precedence should be given the to default (non-custom) operators.

I would argue that it's preferrable to bind to the built-ins when possible, because internally in both the RelBuilder and optimizer rules Calcite looks for specific operator instances when performing optimization (e.g. comparison simplification, aggregate rewrititing, etc). If we use dynamic operators instead of built-in ones, Calcite can no longer perform these optimizations.

Ok, I changed the order so that built-in operators have precedence.

vbarua · 2025-08-08T00:44:14Z

isthmus/src/main/java/io/substrait/isthmus/SqlToSubstrait.java

+    this(SimpleExtension.loadDefaults(), features);
+  }
+
+  public SqlToSubstrait(SimpleExtension.ExtensionCollection extensions, FeatureBoard features) {


I think this should just be

public SqlToSubstrait(SqlOperatorTable operatorTable, FeatureBoard features)

to give users direct control over the operator table they are using.

In this constructor we need to call the constructor of the base class, SqlConverterBase that requires an instance of SimpleExtension.ExtensionCollection as an argument. Otherwise it uses SimpleExtension.loadDefaults() which will cause an inconsistency between the operatorTable and the extensionCollection fields, as the former will contain operators that don't have corresponding extensions in the latter.

ZorinAnton · 2025-08-08T12:31:40Z

I'm going to restate what I think your doing with these changes to make sure that we're on the same page.

Yes, you've correctly summarized the changes.

Meta Comment

I find the usage of "custom" for this concept somewhat confusing. These aren't really custom functions because they are part of the core Substrait spec. A better word for this might be "dynamic". In my head, you're effectively generating SqlOperators dynamically based on the Substrait definitions, and then using them as fall-backs when explicit mappings.

I renamed all "custom" to "dynamic" in the changes, however in my mind User Defined Functions is always something custom :)

vbarua · 2025-08-11T23:23:35Z

isthmus/src/main/java/io/substrait/isthmus/SimpleExtensionToSqlOperator.java

+      TypeConverter typeConverter) {
+    List<SimpleExtension.ValueArgument> requiredArgs =
+        function.args().stream()
+            .filter(SimpleExtension.Argument::required)


I believe this is meaningless because it's always true if you look at the implementations. I'm not even sure what require is used for actually.

The required method is used here:

substrait-java/core/src/main/java/io/substrait/extension/SimpleExtension.java

Line 235 in ec76720

return args().stream().filter(Argument::required).collect(Collectors.toList());

. I would think it is supposed to reflect whether the argument is required or optional. So probably it is not correct to always return true.

vbarua · 2025-08-11T23:30:09Z

isthmus/src/main/java/io/substrait/isthmus/SimpleExtensionToSqlOperator.java

+    List<SimpleExtension.ValueArgument> requiredArgs =
+        function.args().stream()
+            .filter(SimpleExtension.Argument::required)
+            .filter(t -> t instanceof SimpleExtension.ValueArgument)


I'm not sure if it's valid to ignore Enum arguments when generating functions.

For example in extract

impls: - args: - name: component options: [ YEAR, ISO_YEAR, US_YEAR, HOUR, MINUTE, SECOND, MILLISECOND, MICROSECOND, SUBSECOND, PICOSECOND, UNIX_TIME, TIMEZONE_OFFSET ] description: The part of the value to extract. - name: x value: timestamp_tz - name: timezone description: Timezone string from IANA tzdb. value: string return: i64

the enumeration argument for component is part of the function signature. It might make sense to ignore function with enum arguments to start with.

I added the code to handle EnumArgument, however I'm not sure if currently sql with functions using it can be translated to substrait, at least I couldn't manage to do it. Comments in this function indirectly mention that.

isthmus/src/main/java/io/substrait/isthmus/SimpleExtensionToSqlOperator.java

# Conflicts: # isthmus/src/main/java/io/substrait/isthmus/SqlToSubstrait.java # Conflicts: # isthmus/src/main/java/io/substrait/isthmus/SqlToSubstrait.java # isthmus/src/test/java/io/substrait/isthmus/OptimizerIntegrationTest.java # isthmus/src/test/java/io/substrait/isthmus/PlanTestBase.java

…ToSqlOperator

# Conflicts: # isthmus/src/main/java/io/substrait/isthmus/SqlToSubstrait.java

…ToSqlOperator

Signed-off-by: Niels Pardon <[email protected]>

vbarua

Found some time to take a look at this again.

Overall I think it's pretty reasonable feature, but I'm a little wary of enabling it by default everywhere like you have because I can't fully think of the implications of having dynamic mappings in places where they weren't expected before. Examples of things I'm worried about are:

Systems that might actually prefer mappings to fail in order to warn users about missing features.
Dynamic function generation failures with custom user catalogs.

Ideally, I would love to make this capability something that users can opt into. That way, we could release it and users could try it out in their system and report any issues. The way you've structured it now though, it's pretty difficult to do this because the dynamic mappings are created as part of the creating the various converters that are used to to convert between SQL, Substrait and Calcite.

How difficult do you think it would be to make the dynamic mapping behaviour opt-in?

vbarua · 2025-11-26T19:14:05Z

isthmus/src/main/java/io/substrait/isthmus/sql/SubstraitSqlToCalcite.java

+   * @param sqlStatement a SQL statement string
+   * @param catalogReader the {@link Prepare.CatalogReader} for finding tables/views referenced in
+   *     the SQL statement
+   * @param operatorTable the {@link SqlOperatorTable} for dynamic operators


Strictly speaking, you can inject a SqlOperatorTable here that incorporates dynamic operators, but you can also inject a SqlOperatorTable that doesn't. What this allows you to do is have more fine-grained control over valid operators by allowing you to control what SqlOperatorTable you use, instead of just being forced to use the default SubstraitOperatorTable that we provide.

Suggested change

* @param operatorTable the {@link SqlOperatorTable} for dynamic operators

* @param operatorTable the {@link SqlOperatorTable} for controlling valid operators

vbarua · 2025-11-26T19:17:18Z

isthmus/src/main/java/io/substrait/isthmus/sql/SubstraitSqlToCalcite.java

+   * @param sqlStatements a string containing one or more SQL statements
+   * @param catalogReader the {@link Prepare.CatalogReader} for finding tables/views referenced in
+   *     the SQL statements
+   * @param operatorTable the {@link SqlOperatorTable} for dynamic operators


Suggested change

* @param operatorTable the {@link SqlOperatorTable} for dynamic operators

* @param operatorTable the {@link SqlOperatorTable} for controlling valid operators

isthmus/src/main/java/io/substrait/isthmus/SqlExpressionToSubstrait.java

vbarua · 2025-11-26T20:03:40Z

isthmus/src/main/java/io/substrait/isthmus/SimpleExtensionToSqlOperator.java

+
+  public static List<SqlOperator> from(
+      SimpleExtension.ExtensionCollection collection, RelDataTypeFactory typeFactory) {
+    TypeConverter typeConverter = TypeConverter.DEFAULT;


I would suggest making the TypeConverter injectable as well to allow users with custom extension with custom types to use this capability more easily.

Effectively we can have these 2 constructors:

public static List<SqlOperator> from(SimpleExtension.ExtensionCollection collection) {} public static List<SqlOperator> from( SimpleExtension.ExtensionCollection collection, RelDataTypeFactory typeFactory, TypeConverter typeConverter)

The first for default behaviour, and the second for customizable behaviour.

isthmus/src/main/java/io/substrait/isthmus/SimpleExtensionToSqlOperator.java

isthmus/src/main/java/io/substrait/isthmus/ExtensionUtils.java

vbarua · 2025-11-27T01:38:20Z

isthmus/src/main/java/io/substrait/isthmus/ExtensionUtils.java

+      allExtensions = allExtensions.merge(SimpleExtension.load(yamlFunctionFiles));
+    }
+    return allExtensions;
+  }


It doesn't appear that this is used. Do we need this or can we delete this?

vbarua · 2025-11-27T02:00:19Z

isthmus/src/test/java/io/substrait/isthmus/SimpleExtensionToSqlOperatorTest.java

+class SimpleExtensionToSqlOperatorTest {
+
+  @Test
+  void test() throws IOException {


This test does demonstrate that we can produce a SqlOperator for a function declared in an extension, but it doesn't necessarily show that the argument and return type inference align of the SqlOperator are aligned with the function declaration?

How feasible do you think it would be to check that the operators we are producing:

Apply the correct argument restrictions

Have the correct return type inference

Extended to verify number of args, their types and the return type of dynamic functions.

isthmus/src/main/java/io/substrait/isthmus/SubstraitRelNodeConverter.java

…version

ZorinAnton · 2025-12-04T10:30:05Z

hi @vbarua, thank you for reviewing the PR. I used FeatureBoard to make configurable the support of dynamic functions for both directions: Substrait->Sql and Sql->Substrait. Not sure if it makes sense to have different configurations for different directions. Please let me know your thoughts.

vbarua · 2025-12-05T01:08:04Z

isthmus/src/main/java/io/substrait/isthmus/SubstraitRelVisitor.java

+        SimpleExtensionToSqlOperator.from(dynamicExtensionCollection, typeFactory);
+
+    List<FunctionMappings.Sig> additionalSignatures =
+        dynamicOperators.stream()


Dynamic operators are still enabled by default here.

vbarua · 2025-12-05T01:10:34Z

isthmus/src/test/java/io/substrait/isthmus/PlanTestBase.java

+   */
+  protected RelRoot assertSqlSubstraitRelRoundTripLoosePojoComparison(
+      String query, Prepare.CatalogReader catalogReader) throws Exception {
+    return assertSqlSubstraitRelRoundTripLoosePojoComparison(query, catalogReader, null);


Instead of using null here, I would just pass int eh default feature board and avoid the null checking code in the method above.

vbarua

The dynamic operator behavior is not fully disabled by default in both directions.

I'm not 100% confident that this is the correct approach for this in the long-term, but once its fully opt-in I'm comfortable merging this in and iterating from there.

ZorinAnton marked this pull request as draft August 5, 2025 13:05

ZorinAnton force-pushed the zor-udf branch 6 times, most recently from 2f8f79a to ad11957 Compare August 7, 2025 10:08

ZorinAnton marked this pull request as ready for review August 7, 2025 11:15

nielspardon self-requested a review August 7, 2025 13:30

vbarua reviewed Aug 8, 2025

View reviewed changes

ZorinAnton force-pushed the zor-udf branch from ad11957 to 2973628 Compare August 8, 2025 12:18

ZorinAnton force-pushed the zor-udf branch 2 times, most recently from 2356755 to b232a95 Compare August 11, 2025 15:07

vbarua reviewed Aug 11, 2025

View reviewed changes

ZorinAnton force-pushed the zor-udf branch 5 times, most recently from f2527e1 to e7a3d0a Compare August 15, 2025 10:37

ZorinAnton requested a review from vbarua August 18, 2025 08:28

ZorinAnton changed the title ~~Add udf support for Substrait<->Calcite conversion~~ feat(isthmus): add udf support for Substrait<->Calcite conversion Sep 1, 2025

ZorinAnton added 4 commits September 1, 2025 16:50

chore(isthmus): handle nullability and EnumArgument in SimplExtension…

556847b

…ToSqlOperator

feat(isthmus): udf support for substrait<->calcite

857fe2c

# Conflicts: # isthmus/src/main/java/io/substrait/isthmus/SqlToSubstrait.java

chore(isthmus): handle nullability and EnumArgument in SimplExtension…

eeb97cd

…ToSqlOperator

ZorinAnton force-pushed the zor-udf branch from e7a3d0a to eeb97cd Compare September 1, 2025 17:30

ZorinAnton and others added 3 commits October 13, 2025 18:21

Merge branch 'main' into zor-udf

f30b556

chore: resolve pmd

6d9cc2a

chore: sync submodules with main

1a1bf47

chore(isthmus): fix yaml for custom udf

6ba115f

ZorinAnton force-pushed the zor-udf branch from 3c8e6a0 to 6ba115f Compare October 14, 2025 09:36

Merge branch 'main' into zor-udf

e739184

ZorinAnton force-pushed the zor-udf branch from 5b87418 to e739184 Compare November 26, 2025 13:56

ZorinAnton and others added 2 commits November 26, 2025 22:30

chore(isthmus): fix regression

838a0a7

fix: add missing serialVersionUIDs (substrait-io#621)

32ae275

Signed-off-by: Niels Pardon <[email protected]>

ZorinAnton force-pushed the zor-udf branch from 0e3b3dc to 32ae275 Compare November 26, 2025 21:31

ZorinAnton and others added 2 commits November 26, 2025 22:33

Merge branch 'main' into zor-udf

e4f656d

chore(isthmus): fix regression

8d2b4ed

vbarua reviewed Nov 27, 2025

View reviewed changes

ZorinAnton added 8 commits December 3, 2025 19:16

feat(isthmus): enable dynamic UDFs with FeatureBoard

9b0c027

chore(isthmus): refactor SimpleExtensionToSqlOperatorTest

a9e7c60

chore(isthmus): address minor PR comments

2f6cd28

feat(isthmus): make dynamic UDF configurable for substrait to sql con…

71fcdfc

…version

Merge branch 'main' into zor-udf

ca3d967

chore(isthmus): fix format violations

142d372

chore(isthmus): minor refactoring

74fd2e7

chore(isthmus): check arg types in SimpleExtensionToSqlOperatorTest

ec45173

chore(isthmus): minor refactoring of SimpleExtensionToSqlOperator

6eedacd

ZorinAnton force-pushed the zor-udf branch from 6291193 to 6eedacd Compare December 4, 2025 10:32

ZorinAnton requested a review from vbarua December 4, 2025 10:47

vbarua reviewed Dec 5, 2025

View reviewed changes

vbarua requested changes Dec 5, 2025

View reviewed changes

ZorinAnton added 3 commits December 5, 2025 17:32

chore(isthmus): fix opt-in for dynamic operators

b002d9b

Merge branch 'main' into zor-udf

fcf24bb

chore(isthmus): fix regression in test

fc2d576

ZorinAnton requested a review from vbarua December 5, 2025 17:05

Merge branch 'main' into zor-udf

26c15dc

	* @param operatorTable the {@link SqlOperatorTable} for dynamic operators
	* @param operatorTable the {@link SqlOperatorTable} for controlling valid operators

feat(isthmus): add dynamic function conversion for Substrait<->Calcite #457

Are you sure you want to change the base?

feat(isthmus): add dynamic function conversion for Substrait<->Calcite #457

Uh oh!

Conversation

ZorinAnton commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vbarua left a comment

Choose a reason for hiding this comment

Meta Comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZorinAnton Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZorinAnton commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Meta Comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vbarua left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vbarua Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ZorinAnton commented Dec 4, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZorinAnton commented Aug 5, 2025 •

edited

Loading

ZorinAnton Aug 8, 2025 •

edited

Loading

ZorinAnton commented Aug 8, 2025 •

edited

Loading

vbarua left a comment •

edited

Loading

vbarua Nov 27, 2025 •

edited

Loading