Introduce RetryOnFailure lifecycle management strategy

[matheuscscp]

Closes: #1278

TODO:

• Automatic tests.

This PR implements the API enhancements described in #1278.

I tested this thoroughly, including in the scenario of pods in ImagePullBackOff due to unexisting image tags. When the tags are pushed, the HelmRelease object eventually converges without manual intervention.

The behavior of the CLI commands flux reconcile hr and flux reconcile hr --force is the same for a HelmRelease in the failed state: a release action is attempted. If the HelmRelease is Ready, i.e. in the deployed state, the behaviors of these two commands are not the same, but they match the behaviors for a HelmRelease object that is not using this feature. In this case, the behavior is no-op for flux reconcile hr, and perform a release action for flux reconcile hr --force.

Because the retry strategies cause a time-based requeue (and do not end in a terminal state), running the flux reconcile hr command when the release is failed, with or without --force, results in an immediate release action, and does not remove any in-flight RequeueAfter operations. In particular, if an in-flight RequeueAfter completes during a reconciliation, another reconciliation will run immediately after. I observed this many times due to using a very short .retryInterval for speeding up my tests.

Another common behavior that I noticed while testing the install strategy is that, when the source object (e.g. OCIRepository) is created together with the HelmRelease, two reconciliations are processed: the one produced by "requeue dependency" and the watch event from the source object becoming Ready. This results in the first retry happening immediately after the first attempt, i.e. in two consecutive failed installs.

It's also important to note that using the install strategy causes the transition of .status.lastAttemptedReleaseAction from install to upgrade. So if this upgrade fails, the upgrade strategy configuration will now be the one driving the behavior, the install strategy configuration will no longer be at play. Therefore, to successfully achieve a HelmRelease that is automatically retried without any remediations, the spec must look like this:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
spec:
  install:
    strategy: # the install strategy will cause the first retry, which will be an upgrade (the first one)
      name: RetryOnFailure
  upgrade:
    strategy: # the upgrade strategy will cause the subsequent retries, which will all be upgrades
      name: RetryOnFailure

[stefanprodan]

I suggest we apply jitter for the retries to spread out the reconciliations

[matheuscscp]

After deeply understanding the state machine, it turns out that it's simpler to introduce both action strategies in the same PR.

[matheuscscp]

This check is a tautology, it's basically the negation of the check right above. I removed it in this separate commit: 079ae1b

[matheuscscp]

Here, this error was already recorded in a condition and was sent in Kubernetes and notification-controller events. But no logs show up because we are handling it here and returning a different error, so I'm logging it here

[matheuscscp]

I tested this PR in the following scenarios:

1. Chart build fails due to templating error. In this case, the HelmReleaseReconciler was returning the error to controller-runtime, resulting in retries with exponential backoff. To fix this I applied this diff.
2. Build works, but YAML apply fails. I tested this by making the chart apply a resource in a namespace it didn't have RBAC for.
3. Apply succeeds but Deployment fails due to image pull error.
4. Apply succeeds and Deployment becomes Ready, but Helm tests fail.
5. Install succeeds, then I delete the Helm storage secret. This works as expected, a Helm install is performed in the next reconciliation and succeeds.

In all the cases above, after fixing the issue in scenario 1, the controller behavior matches the expectations:

• Except for scenario 5, retries with upgrade after the respective .retryInterval.
• A status condition is updated with the error.
• A log is emitted with the error (after applying this diff).
• Kubernetes and notification-controller events are sent with the error.