Support config yaml

[ctrlaltdilj]

Support Flink 1.19 config.yaml configuration

Flink 1.19 introduced new config.yaml format. There are some dependencies on this new format in 1.19 that prevent old flink-conf.yaml from being able to configure properties.

This PR, adds support to be able to set a config.yaml file which is able to be consumed by the operator as well as deployments.

Brief change log

• added support to be able to set config.yaml by replacing flink-conf.yaml in the defaultConfiguration

Verifying this change

This change added tests and can be verified as follows:

• Build image
• create a values.yaml similar to
• Extended integration test for recovery after master (JobManager) failure
• Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes
• The public API, i.e., is any changes to the CustomResourceDescriptors: no
• Core observer or reconciler logic that is regularly executed: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? not documented, will add

[ctrlaltdilj]

I do think this would be better supported via a flag on configuration on FlinkDeployment, as it enables operator update independently from FlinkDeployments

I am happy to make that change and would prefer that. Any thoughts?

[mateczagany]

Hi, first of all, thank you for the contribution, but for better observability please try to always create a JIRA before opening a PR.
Unless trying to push a fix for a release candidate, please use the main branch for your PRs. Flink version is already 1.19.1 on the main branch.

[ctrlaltdilj]

Thanks @mateczagany for all the info! I have requested a JIRA account, and will file a ticket

[ctrlaltdilj]

@mateczagany
Created following ticket: https://issues.apache.org/jira/browse/FLINK-35744

[gyfora]

Would be great if this would be "automatic" based on whether the user specified flink-conf.yaml vs conf.yaml. Do you think that's possible?

[ljankows]

Can we also change the version to 1.19.1 as it was already released?