Skip to content

Fixing Argo CD Sync Conflicts with Gateway API and Envoy Gateway 1.8+

If you recently bumped your infrastructure to Envoy Gateway 1.8+, you might see warnings similar to the ones I had on my setup, about the new Gateway API admission assets:

ValidatingAdmissionPolicy/safe-upgrades.gateway.networking.k8s.io is part of applications argocd/gateway-api-crds and envoy-gateway-chart

ValidatingAdmissionPolicyBinding/safe-upgrades.gateway.networking.k8s.io is part of applications argocd/gateway-api-crds and envoy-gateway-chart

Even if you set skipCrds: true on your main Envoy Gateway installation, the synchronization failure persists across both the policy and its binding. Here is a look at what changed in recent versions and how to decouple them permanently.

Why now

This issue seems entirely new since I upgraded to Envoy Gateway 1.8.1.

  1. The Sub-Chart Shift: To prevent Helm secret release payloads from exceeding the strict 1MB size limit due to expanding API schemas, Envoy Gateway v1.8.0 moved its schema engine into a standalone sub-chart (gateway-crds-helm).
  2. Upstream Gateway API Changes: Around the same time, Gateway API (v1.5+) introduced native declarative validation policies and bindings (safe-upgrades.gateway.networking.k8s.io) to catch dangerous CRD modifications at the control-plane level.

The conflict arises because the new Envoy Gateway charts bake both the ValidatingAdmissionPolicy and its companion ValidatingAdmissionPolicyBinding into the CRD installation tree and the core controller runtime templates by default.

If you manage your upstream Gateway API structures via an independent Argo CD app (argocd/gateway-api-crds), you instantly get a three-way tug-of-war over who owns these two cluster-scoped resources. Because validation configurations are treated as standard runtime objects rather than raw CRDs, traditional Helm flags like skipCrds: true do not intercept either of them.

A solution

1. Update Envoy Gateway CRD App Manifest

In my standalone envoy-gateway-crds application, I added crds.gatewayAPI.enabled=false. This restricts the sub-chart to deploying only Envoy-specific schemas (ClientTrafficPolicy, BackendTrafficPolicy, etc.):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: envoy-gateway-crds
spec:
  source:
    chart: gateway-crds-helm
    helm:
      parameters:
        - name: crds.envoyGateway.enabled
          value: "true"
        - name: crds.gatewayAPI.enabled # <-- Exclude upstream Gateway API CRDs & Validation Engine
          value: "false"

2. Update Primary Envoy Gateway Runtime Manifest

In my primary controller deployment chart, I passed crds.gatewayAPI.safeUpgradePolicy.enabled=false alongside the standard skipCrds toggle. This stops the runtime templates from generating the duplicate policy and binding:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: envoy-gateway
spec:
  source:
    chart: envoy-gateway
    helm:
      parameters:
        - name: skipCrds
          value: "true"
        - name: crds.gatewayAPI.safeUpgradePolicy.enabled # <-- Exclude runtime admission policy & binding templates
          value: "false"

3. Clear Out the Conflicted State

Finally, I cleared both mismatched resources manually so Argo can start fresh:

# Delete the disputed cluster-scoped assets
kubectl delete validatingadmissionpolicy safe-upgrades.gateway.networking.k8s.io
kubectl delete validatingadmissionpolicybinding safe-upgrades.gateway.networking.k8s.io

And in Argo CD UI, after running a standard Sync, there is now a clean, single ownership over both objects.

Side-note: Trade Replace=true for ServerSideApply=true

If you are dealing with large schemas like Gateway API, you might have legacy sync notes advising you to use Replace=true to skip the 256 KB client-side annotation size limit (kubectl.kubernetes.io/last-applied-configuration).

While Replace=true clears the size ceiling, it triggers aggressive replacement routines that don't play well with active validation webhooks. The modern GitOps pattern is to swap it for ServerSideApply:

syncPolicy:
  syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true # Handled directly on the API server, eliminating size caps safely

Published: 2026-06-13