Published: 2026-07-23

Talos Upgrades, Kyverno, and the Node That Wouldn't Drain¶

A routine Kubernetes upgrade on my single-node Talos cluster turned into a good reminder about admission webhooks, Helm value merging, and the order of operations during a node drain.

The symptom: every pod is Pending¶

After the upgrade, the node looked like this:

NAME     STATUS                     ROLES           VERSION   OS-IMAGE
 talos0   Ready,SchedulingDisabled   control-plane   v1.36.2   Talos (v1.13.7)

Nearly every pod was Pending, and events repeated:

Internal error occurred: failed calling webhook "validate.kyverno.svc-fail":
Post "https://kyverno-svc.kyverno.svc:443/validate/fail?timeout=10s":
dial tcp 10.109.140.191:443: connect: connection refused

Why it happened¶

talosctl upgrade-k8s drains the node before upgrading the control plane. On a single-node cluster that means evicting every workload, including Kyverno. Kyverno's validating webhook uses failurePolicy: Fail, so as soon as Kyverno became unreachable the API server refused every eviction.

The drain never finished, but the node was already cordoned. After rebooting, Kubernetes kept Unschedulable: true, and Kyverno couldn't reschedule because there was nowhere to put it.

The immediate fix was manual:

kubectl uncordon talos0

Once Kyverno started, its webhook came back and the cluster recovered.

Preventing it during the next upgrade¶

I now disable Kyverno automatically before any upgrade that drains nodes:

1. Save current replica counts for all deployments in the kyverno namespace.
2. Scale them to 0.
3. Run the Talos / Kubernetes upgrade.
4. Restore the saved replica counts.

The save is skipped if the temp file already exists, so retries don't overwrite the original replicas.

This can also be done by hand:

kubectl scale deployment -n kyverno --replicas=0 --all
# run the upgrade
kubectl scale deployment -n kyverno kyverno-admission-controller --replicas=1
kubectl scale deployment -n kyverno kyverno-background-controller --replicas=1
kubectl scale deployment -n kyverno kyverno-cleanup-controller --replicas=1
kubectl scale deployment -n kyverno kyverno-reports-controller --replicas=1

For multi-node clusters, running Kyverno with multiple replicas across nodes is a better long-term fix, since it survives individual node drains without any manual steps.

The Helm surprise¶

After the node recovered, the Argo CD chart upgrade (from 9.5.22 to 10.1.4) failed with:

Error: UPGRADE FAILED: argo-cd/templates/dex/deployment.yaml:155:22
  executing ... at <.Values.dex.startupProbe.enabled>:
    nil pointer evaluating interface {}.enabled

The new chart expects .Values.dex.startupProbe.enabled, but the existing release's values didn't have a startupProbe key. Using --reuse-values told Helm to use the old values wholesale and skip the new chart defaults.

The fix was a single flag change:

- helm upgrade argo-cd argo/argo-cd ... --reuse-values ...
+ helm upgrade argo-cd argo/argo-cd ... --reset-then-reuse-values ...

--reset-then-reuse-values loads the new chart's defaults first, then overlays your custom values on top. The missing startupProbe field gets its default, templates render cleanly, and any values you actually changed are preserved.

I verified with a dry run first:

helm upgrade argo-cd argo/argo-cd --namespace argocd \
  --version 10.1.4 --skip-crds --reset-then-reuse-values --dry-run

Takeaways¶

• Validating webhooks with failurePolicy: Fail are a drain-time single point of failure on single-node clusters. Scale them down or make them HA.
• --reuse-values is risky across major chart versions. Use --reset-then-reuse-values to pick up new defaults.
• kubectl get nodes and kubectl get events are enough to diagnose most of this. The cordoned node and webhook errors told the whole story.