[Core feature] Pod Template support for init containers (ie. copilot)

[Sovietaced]

Motivation: Why do you think this is important?

The copilot init container downloads task metadata and if this task metadata is hosted in an on premises data store requests may fail to the data store due to TLS verification issues. The only way to resolve this right now is with a custom copilot image but this could also be handled with a pod template that specifies root certificate volume mounts (as we already do for normal flyte task containers). I propose that we allow users to specify pod template configurations for init containers in the same way that we do for normal containers.

ie.

            volumeMounts:
              - name: my-cert-bundle
                readOnly: true
                mountPath: /etc/ssl/certs

Goal: What should the final outcome look like, ideally?

In our fork we modified pod_helper to look for an init container with the name default-init.

Describe alternatives you've considered

Forcing users to build custom images, however this may have limitations if those custom images also need arguments from environment variables.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[kumare3]

@Sovietaced how about always make the user provided init go first? But i like the idea of default-init. would you mind upstreaming this?

[Sovietaced]

@Sovietaced how about always make the user provided init go first? But i like the idea of default-init. would you mind upstreaming this?

I can try and make the behavior similar to the regular containers, looking for user provided container and fall back to default.