[WIP] [Feat-v2] Enable memory increase on OOM failure

[Mecoli1219]

Tracking issue

#2234

Why are the changes needed?

This is not completed yet.

What changes were proposed in this pull request?

• FlyteIDL
  • Follow @kumare3 suggestion, update the RetryStrategy IDL
• FlyteAdmin
  • Update the flyteadmin to handle launching execution with adjusted resource
  • Handle both node's Retry (store with_overrides value) and task's Retry (task specified RetryStrategy)
  • Add new unit tests
• FlytePropeller
  • Store the historical OOM failures number
  • Create OnOOMConf for taskExecutionMetadata so that FlytePlugins can use it.
  • Add new unit tests
• FlytePlugins
  • Update the logistic to compute the updated resource when OOM
  • Implement the multiplication of Resource.Quantity (@b05702117 is working on it)
  • Add new unit tests

How was this patch tested?

I have manually tested it with the debugger to ensure that it is executed as expected, but we will need to create more unit tests later.

Labels

• added: For new features.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Previous version PR: #6263

Docs link

[flyte-bot]

Code Review Agent Run Status

• Limitations and other issues: ❌ Failure - The AI Code Review Agent skipped reviewing this change because it is configured to exclude certain pull requests based on the source/target branch or the pull request status. You can change the settings here, or contact the agent instance creator at [email protected].

[codecov]

Codecov Report

Attention: Patch coverage is 16.82464% with 351 lines in your changes missing coverage. Please review.

Project coverage is 33.76%. Comparing base (c0eb6de) to head (885cc11).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
flyteidl/gen/pb-go/flyteidl/core/literals.pb.go	1.04%	95 Missing ⚠️
.../tasks/pluginmachinery/core/mocks/on_oom_config.go	0.00%	83 Missing ⚠️
...yteworkflow/v1alpha1/mocks/ExecutableNodeStatus.go	0.00%	50 Missing ⚠️
.../flyteworkflow/v1alpha1/mocks/MutableNodeStatus.go	0.00%	25 Missing ⚠️
...tepropeller/pkg/compiler/transformers/k8s/utils.go	35.71%	18 Missing ⚠️
...ller/pkg/controller/nodes/task/taskexec_context.go	34.61%	17 Missing ⚠️
...ginmachinery/core/mocks/task_execution_metadata.go	38.46%	12 Missing and 4 partials ⚠️
...r/nodes/interfaces/mocks/node_execution_context.go	48.00%	10 Missing and 3 partials ⚠️
...plugins/go/tasks/pluginmachinery/flytek8s/utils.go	28.57%	9 Missing and 1 partial ⚠️
...ler/pkg/apis/flyteworkflow/v1alpha1/node_status.go	0.00%	6 Missing ⚠️
... and 6 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6293      +/-   ##
==========================================
- Coverage   33.82%   33.76%   -0.06%     
==========================================
  Files        1329     1330       +1     
  Lines      147808   148213     +405     
==========================================
+ Hits        49993    50051      +58     
- Misses      92973    93311     +338     
- Partials     4842     4851       +9     

Flag	Coverage Δ
unittests-datacatalog	48.01% <ø> (ø)
unittests-flyteadmin	50.11% <73.07%> (+0.02%)	⬆️
unittests-flytecopilot	30.99% <ø> (ø)
unittests-flytectl	58.09% <ø> (ø)
unittests-flyteidl	6.77% <1.04%> (-0.02%)	⬇️
unittests-flyteplugins	48.77% <14.72%> (-0.24%)	⬇️
unittests-flytepropeller	36.44% <18.71%> (-0.09%)	⬇️
unittests-flytestdlib	50.38% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Mecoli1219]

@b05702117 @paullongtan feel free to push your commits to this PR

[flyte-bot]

Code Review Agent Run Status

• Limitations and other issues: ❌ Failure - The AI Code Review Agent skipped reviewing this change because it is configured to exclude certain pull requests based on the source/target branch or the pull request status. You can change the settings here, or contact the agent instance creator at [email protected].