Problem
The Presto optimizer/planner is written in Java, but the runtime is increasingly moving to Velox (C++/native). Functions like map_key_exists are native Velox built-in functions in production, but are only available in the Java test harness through the SqlInvokedFunctionsPlugin (which provides a SQL-invoked fallback implementation).
When an optimizer rule generates plan nodes referencing these native functions (e.g. map_key_exists in PrefilterForLimitingAggregation), the Java-based tests fail because the function is not registered in the test server's function registry.
Current State
map_key_exists is a native Velox function (efficient hash probe on maps) in production- In the Java codebase, it exists only as a SQL-invoked function in
presto-sql-invoked-functions-plugin (RETURN CONTAINS(MAP_KEYS(input), k))
- The
TpchQueryRunnerBuilder (used by most distributed query tests) does not install this plugin by default
- This means any optimizer that references native Velox functions will fail in the OSS Java tests
Impact
As more optimizer rules are written to target native Velox functions, this gap between the Java test harness and the production runtime will become a recurring issue. Each new usage requires manually adding the plugin to the test builder, which is fragile and easy to miss.
Proposed Solution
Consider one or more of:
- Load
SqlInvokedFunctionsPlugin by default in TpchQueryRunnerBuilder and other standard test builders so all SQL-invoked function fallbacks are available
- Establish a registry mapping from native Velox function names to their Java fallback implementations, so the test harness can automatically resolve them
- Document the pattern for optimizer developers: when referencing native functions, ensure the Java fallback exists and the plugin is loaded in tests
Context
Discovered while adding map_key_exists usage in PrefilterForLimitingAggregation (PR #27678). The immediate fix was to install SqlInvokedFunctionsPlugin in TpchQueryRunnerBuilder, but the broader issue affects any optimizer that uses native Velox functions.
Problem
The Presto optimizer/planner is written in Java, but the runtime is increasingly moving to Velox (C++/native). Functions like
map_key_existsare native Velox built-in functions in production, but are only available in the Java test harness through theSqlInvokedFunctionsPlugin(which provides a SQL-invoked fallback implementation).When an optimizer rule generates plan nodes referencing these native functions (e.g.
map_key_existsinPrefilterForLimitingAggregation), the Java-based tests fail because the function is not registered in the test server's function registry.Current State
map_key_existsis a native Velox function (efficient hash probe on maps) in productionpresto-sql-invoked-functions-plugin(RETURN CONTAINS(MAP_KEYS(input), k))TpchQueryRunnerBuilder(used by most distributed query tests) does not install this plugin by defaultImpact
As more optimizer rules are written to target native Velox functions, this gap between the Java test harness and the production runtime will become a recurring issue. Each new usage requires manually adding the plugin to the test builder, which is fragile and easy to miss.
Proposed Solution
Consider one or more of:
SqlInvokedFunctionsPluginby default inTpchQueryRunnerBuilderand other standard test builders so all SQL-invoked function fallbacks are availableContext
Discovered while adding
map_key_existsusage inPrefilterForLimitingAggregation(PR #27678). The immediate fix was to installSqlInvokedFunctionsPlugininTpchQueryRunnerBuilder, but the broader issue affects any optimizer that uses native Velox functions.