[FLINK-37088][k8s]support flink applicaton taskmanager log mounting

[baimafeima-yf]

What is the purpose of the change

flink application support host mount log for the JobManager/TaskManager Container

Brief change log

• add FlinkLogDecorator to the decorator chains in KubernetesJobManagerFactory and KubernetesTaskManagerFactory.

Verifying this change

This change is already covered by existing tests..

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive):no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented?no--will add an intro doc in later PR.

[flinkbot]

CI report:

• 1d09ce3 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[davidradl]

I am not a k8s expert. I have some basic questions:

• we should document these new config options.
• are there not issues in supplying default locations to write logs in a PR, as suddenly there will be a lot of data written to locations that was previously not written. I would assume that /apps/log/flink and /opt/flink/log are folders and could get full - so size would need to be managed; maybe having criteria to delete / archive older logs. If there is new management required of these logs, then the considerations should be documented.
• It would be helpful to describe in the Jira in more detail what the problem is and how this fixes it.
  • what is the life cycle of HostPathVolumeSource - it it associated with the pod, so are we trying to keep
    logs that would have disappeared with the container, in storage associated with the pod and we would
    lose the logs if the pods die.
• I wonder what the impact is of enabling this for all containers after this PR.

[baimafeima-yf]

Thank you very much for your review. Yes, we can configure a container's PersistentVolume (PV) and PersistentVolumeClaim (PVC) to achieve log mounting.
However, when starting a Flink application via code invocation, we found that the log mounting was not effective. Therefore, we modified this section of the code to implement log mounting and added a toggle kubernetes.decorator.log-mount.enabled. If users encounter the same situation as us, they only need to enable this toggle to achieve log mounting; it is disabled by default.

[davidradl]

Reviewed by Chi on 16/01/2025 Go back to the submitter with review comments.