Skip to content

[FLINK-40497][table] Allow materialized tables to default to ON CONFLICT DO ERROR - #29040

Open
gustavodemorais wants to merge 1 commit into
apache:masterfrom
confluentinc:FLINK-40497
Open

[FLINK-40497][table] Allow materialized tables to default to ON CONFLICT DO ERROR#29040
gustavodemorais wants to merge 1 commit into
apache:masterfrom
confluentinc:FLINK-40497

Conversation

@gustavodemorais

Copy link
Copy Markdown
Contributor

What is the purpose of the change

A materialized table has no syntax for an explicit ON CONFLICT clause, so it always falls back to the implicit DO DEDUPLICATE strategy when its primary key differs from its query's upsert key. This adds an opt-in option to default it to DO ERROR instead, avoiding the state-heavier deduplicating materializer, applied only when every source already declares a watermark so table creation never fails because of it.

Brief change log

  • Add TABLE_EXEC_SINK_MATERIALIZED_TABLE_FORCES_ON_CONFLICT_ERROR, default false.
  • convertMaterializedTableAsToRel picks ON CONFLICT DO ERROR when the option is enabled and every source has a watermark, otherwise keeps the previous behavior.
  • Add allSourcesHaveWatermarks, walking the logical input tree to check each source's watermark spec.
  • Thread getTableConfig through PlannerBase's call to convertMaterializedTableAsToRel.
  • Tailor the require-on-conflict validation message with a materialized-table-specific hint when the new option is enabled but a source is missing a watermark.
  • Document the new option and its interaction with require-on-conflict in the INSERT reference docs.

Verifying this change

  • ExplainTest

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes - new option added to ExecutionConfigOptions (@PublicEvolving)
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? (docs/content/docs/sql/reference/dml/insert.md, docs/content.zh/docs/sql/reference/dml/insert.md, generated config option docs)

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

2.1.235 (Claude Code) with Sonnet 5

@flinkbot

flinkbot commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@raminqaf raminqaf left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a comment

Comment on lines +305 to 324
/** Whether every table scanned by {@code rel} declares a watermark. */
private static boolean allSourcesHaveWatermarks(RelNode rel) {
if (rel instanceof TableScan) {
final TableSourceTable table =
((TableScan) rel).getTable().unwrap(TableSourceTable.class);
// An unresolvable scan can't be proven to lack a watermark, so it isn't treated as a
// reason to fall back - matching the same convention used for the physical-tree check.
return table == null
|| !table.contextResolvedTable()
.getResolvedSchema()
.getWatermarkSpecs()
.isEmpty();
}
for (RelNode input : rel.getInputs()) {
if (!allSourcesHaveWatermarks(input)) {
return false;
}
}
return true;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can simplify this

private static boolean allSourcesHaveWatermarks(RelNode rel) {
    if (rel instanceof TableScan) {
        final TableSourceTable table =
                ((TableScan) rel).getTable().unwrap(TableSourceTable.class);
        return table == null
                || !table.contextResolvedTable().getResolvedSchema().getWatermarkSpecs().isEmpty();
    }
    return rel.getInputs().stream().allMatch(DynamicSinkUtils::allSourcesHaveWatermarks);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also noticed that collectSourcesWithoutWatermarks does the same check

@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants