Argo CD not detecting or applying changes in structured spec.config.processors field for OpenTelemetry Collector CR

[Iamkishore486]

I’m observing that Argo CD is not detecting or applying changes made to nested fields inside the spec.config section of an OpenTelemetry Collector custom resource, even though the configuration is defined as structured YAML (not a multiline string).

• Updated a nested parameter inside the OpenTelemetry Collector CR:

spec:
  config:
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 77

• Argo CD behavior:

argocd app manifests <app-name> → shows updated value

argocd app diff <app-name> → no output

Application remains in Synced state

No sync is triggered

• However, in the cluster below command still shows the old value

kubectl get opentelemetrycollector servicemonitor -o yaml

• What I expected
  Argo CD should detect a difference in the nested field (spec.config.processors.memory_limiter.limit_percentage) and mark the application as OutOfSync, triggering a sync to apply the updated configuration.

• What actually happens

Argo CD does not detect any difference and does not attempt to apply the updated configuration, even though the rendered manifests clearly contain the new value.

• Confirmation

The spec.config field is defined as structured YAML, not a multiline string (|)

No ignoreDifferences or resource.customizations are configured

No errors are observed in Argo CD logs

• Environment

Argo CD version: v3.3.2

Kubernetes version: v1.34.3

Application type: Helm 

Resource: OpenTelemetry Collector CRD

• Additional details
  No ignoreDifferences or resource.customizations are configured.

• Workarounds observed

Forcing a sync (argocd app sync --force) applied changes 

Adding/changing annotations forces Argo CD to detect a diff

• Questions

Is Argo CD expected to detect changes in nested structured fields under spec.config for CRDs like OpenTelemetry Collector?

Could this behavior be related to how Argo CD performs diffing on CRDs with loosely defined schemas?

Are there any known limitations or recommended configurations for handling such cases?

[abhay1999]

Great question — this is a known pain point with CRDs that have loosely-typed or deeply nested fields like spec.config in the OpenTelemetry Collector.

Root Cause

The most likely cause is that the OpenTelemetry Operator’s admission webhook is normalizing spec.config before it is stored in etcd. When Argo CD runs its diff, it compares the desired manifest (from Git/Helm) against the live resource (from the cluster). If the webhook rewrites or reorders the nested fields during admission, the live state in etcd ends up structurally different from what Argo CD rendered — even though the semantics are the same.

The clue is your observation:

• argocd app manifests → shows the new value (desired state from Git/Helm)
• argocd app diff → no output (Argo CD thinks desired == live)
• kubectl get ... -o yaml → shows the old value

This means Argo CD’s diff engine is comparing the rendered manifest against a normalized/mutated live state and finding them equal, even though the actual cluster object is stale.

How to Confirm

Run:

argocd app diff <app-name> --local <path-to-manifests>

Also check if the OTel Operator has a mutating webhook:

kubectl get mutatingwebhookconfigurations | grep opentelemetry

Recommended Fix

Use resource.customizations in your Argo CD ConfigMap to tell the diff engine how to handle this CRD:

# argocd-cm ConfigMap
resource.customizations: |
  opentelemetry.io/OpenTelemetryCollector:
    ignoreDifferences: |
      jsonPointers:
      - /spec/config/processors

Or, if you want Argo CD to correctly detect real changes, configure managedFieldsManagers to exclude the operator’s manager from the diff:

resource.compareoptions: |
  ignoreManagedFieldsManagers:
  - opentelemetry-operator

This tells Argo CD to ignore fields last-written by the operator’s webhook, so only your Git-sourced changes are considered.

Answers to Your Questions

1. Yes, Argo CD should detect changes in nested structured fields — but only when the live resource in etcd actually reflects those changes. If a webhook is normalizing the field before storage, the diff will be empty.
2. This is likely CRD schema + webhook normalization, not a fundamental Argo CD limitation.
3. Configuring ignoreManagedFieldsManagers or resource.customizations is the recommended path. You may also want to file an issue with the OpenTelemetry Operator if the webhook is silently discarding valid config changes.

Hope this helps unblock you!

[Iamkishore486]

Thank you @abhay1999 for your suggestions and I appreciate your time and efforts in responding to the discussion. I tested both of your suggestions individually and combiningly but unfortunately both didn't work. I still see the same behaviour in argocd, Last sync still not picking latest commits.

[cp319391]

Check kubectl get opentelemetrycollector <name> -o json | jq '.metadata.managedFields' — if the OTel Operator owns spec.config.processors,
server-side apply is silently dropping your changes (which is why --force works).
Add this to your Application's syncPolicy:

syncOptions:
  - ServerSideApply=true
  - ServerSideApplyForceConflicts=true.       ##### not recommended 

It is important to see which fields are owned by Operator and which one by argocd, in case where server-side apply is enabled.

check that by below commands

kubectl get opentelemetrycollector demo -o json --show-managed-fields | \
  jq '.metadata.managedFields[] | select(.manager == "manager") | .fieldsV1'

or

kubectl get opentelemetrycollector demo -o json --show-managed-fields | \
  jq '.metadata.managedFields[] | select(.manager == "argocd-controller") | .fieldsV1'

[Iamkishore486]

Thank you @cp319391 for the responding to the discussion, I have verified if managedFields is owned by operator but we got null as output and we see no managedFields under metadata. Also, verified manager manged field and got below output and similar with argocd-controller

kubectl get opentelemetrycollector <otelcollector> -o json --show-managed-fields | jq '.metadata.managedFields[] | select(.manager == "manager") | .fieldsV1'
{
  "f:status": {
    ".": {},
    "f:image": {},
    "f:scale": {
      ".": {},
      "f:replicas": {},
      "f:selector": {},
      "f:statusReplicas": {}
    },
    "f:version": {}
  }
}

[cp319391]

could you check output for

kubectl get opentelemetrycollector <name> -o json --show-managed-fields | \
  jq '.metadata.managedFields[] | {manager, operation, fieldsV1}'

for me the output is something like below, which says spec.config. changes are controlled by argocd-controller. who is the controller/manager in your case.

{
  "manager": "argocd-controller",
  "operation": "Apply",
  "fieldsV1": {
    "f:metadata": {
      "f:annotations": {
        "f:argocd.argoproj.io/tracking-id": {}
      }
    },
    "f:spec": {
      "f:config": {
        "f:exporters": {
          "f:debug": {}
        },
        "f:processors": {
          "f:memory_limiter": {
            ".": {},
            "f:check_interval": {},
            "f:limit_percentage": {}
          }
        },

[Iamkishore486]

Thank you @cp319391 for the new command, even I see argocd-controller controls the changes as below

{
  "manager": "argocd-controller",
  "operation": "Apply",
  "fieldsV1": {
    "f:metadata": {
      "f:annotations": {
        "f:argocd.argoproj.io/tracking-id": {}
      },
      "f:labels": {
        "f:opentelemetry.io/target-allocator": {}
      }
    },
    "f:spec": {
      "f:config": {
        "f:exporters": {
          "f:debug": {
            ".": {},
            "f:verbosity": {}
          },
          "f:signalfx/splunk": {
            ".": {},
            "f:access_token": {},
            "f:drop_histogram_buckets": {},
            "f:realm": {}
          }
        },
        "f:extensions": {
          "f:health_check": {
            ".": {},
            "f:endpoint": {}
          }
        },
        "f:processors": {
          "f:batch": {
            ".": {},
            "f:send_batch_size": {},
            "f:timeout": {}
          },
          "f:filter/drop_metrics": {
            ".": {},
            "f:error_mode": {},
            "f:metrics": {
              ".": {},
              "f:metric": {}
            }
          },
          "f:k8sattributes": {
            ".": {},
            "f:extract": {
              ".": {},
              "f:labels": {}
            }
          },
          "f:memory_limiter": {
            ".": {},
            "f:check_interval": {},
            "f:limit_percentage": {},
            "f:spike_limit_percentage": {}
          },
          "f:resource": {
            ".": {},
            "f:attributes": {}
          },
          "f:resource/remove-attributes": {
            ".": {},
            "f:attributes": {}
          }
        },
        "f:receivers": {
          "f:prometheus": {
            ".": {},
            "f:config": {
              ".": {},
              "f:scrape_configs": {}
            }
          }
        },
        "f:service": {
          "f:extensions": {},
          "f:pipelines": {
            "f:metrics": {
              ".": {},
              "f:exporters": {},
              "f:processors": {},
              "f:receivers": {}
            }
          }
        }
      },
      "f:env": {},
      "f:mode": {},
      "f:podDnsConfig": {
        "f:options": {}
      },
      "f:ports": {},
      "f:replicas": {},
      "f:serviceAccount": {}
    }
  }
}
{
  "manager": "manager",
  "operation": "Update",
  "fieldsV1": {
    "f:status": {
      ".": {},
      "f:image": {},
      "f:scale": {
        ".": {},
        "f:replicas": {},
        "f:selector": {},
        "f:statusReplicas": {}
      },
      "f:version": {}
    }
  }
}

[llavaud]

Hi,

I took a quick look at the CRD in the chart, and I believe you're running into this issue.

CRDs sometimes come in two flavors: a lightweight version that accepts anything but doesn't play well with Argo CD's server-side diff feature, and a full version where all fields are properly defined.

In the OpenTelemetry case, it looks like they're using a mix of both. As you can see, the spec.config field has no schema defined and accepts arbitrary content via x-kubernetes-preserve-unknown-fields: true — that's likely the root cause of your problem.

Unless you can get hold of a full CRD definition, I'm afraid the only workaround is to disable the server-side diff feature.

[Iamkishore486]

@llavaud : Thank you for the update. Based on the details shared in your issue, it does make sense that fields configured with x-kubernetes-preserve-unknown-fields: true are being ignored by Argo CD when nested changes are introduced. I was able to confirm the same behavior through a discussion I created in the OpenTelemetry community as well: - open-telemetry/opentelemetry-collector#15230, but one observation I made when dealing with opentelemetry CRD is that -

When changes are made at the spec.config level, Argo CD is able to detect the differences and successfully sync the resource.
However, when changes are made deeper within the configuration, such as under spec.config.processors, spec.config.exporters, or similar nested fields, those changes appear to be ignored by Argo CD.

While reviewing the CRD, I also observed that the spec.config field is configured with x-kubernetes-preserve-unknown-fields: true, it is conflicting with the statement that fields having x-kubernetes-preserve-unknown-fields: true will be ignored.