Troubleshooting
- Nodes are stale but nothing is marked
- Nodes are marked but never replaced
- The cluster is rotating nodes constantly
- Nodes were expired instead of rotated
- A node I reverted went away anyway
- Everything stopped after a Karpenter upgrade
Nodes are stale but nothing is marked
STALE is climbing and DRIFTED stays at zero.
Check the policy is healthy and the selector matches what you think:
kubectl get bestbefore default-pool -o yaml | yq '.status'
kubectl get nodeclaims -l karpenter.sh/nodepool=default
If Ready is False with InvalidSelector, the selector parses on the API server but not in
k-belt, usually an operator such as In with no values. Fix the selector.
If the counts look right, look at the NodeClaims themselves. k-belt skips any NodeClaim without both
karpenter.sh/nodepool-hash and karpenter.sh/nodepool-hash-version, because Karpenter’s drift
check ignores those too:
kubectl get nodeclaim <name> -o jsonpath='{.metadata.annotations}'
NodeClaims carrying bestbefore.k-belt.io/paused are skipped, which is the point of that
annotation.
If the policy sets maxConcurrent, STALE will exceed DRIFTED until the rotation completes.
k-belt re-checks the remaining NodeClaims every 30 seconds, and each time a slot frees up it marks
the oldest of them.
Nodes are marked but never replaced
DRIFTED sits at the same number for hours. k-belt has done its part, so the answer is on
Karpenter’s side:
kubectl describe node <node> | grep -A5 Events
DisruptionBlocked gives the reason. The common ones are a PodDisruptionBudget that allows no
evictions, a pod with karpenter.sh/do-not-disrupt, pods that cannot be scheduled anywhere else, and
a Drifted budget of zero. Disruption lists the full set.
Worth checking your own budget first, since it is the easiest to get wrong:
kubectl get nodepool default -o jsonpath='{.spec.disruption.budgets}'
The cluster is rotating nodes constantly
Every replacement node is marked again shortly after it arrives. That means maxAge is shorter than
the time a full rotation takes, so the pool never gets ahead of itself. Raise maxAge or raise the
budget; Sizing maxAge has the arithmetic.
Nodes were expired instead of rotated
Karpenter deleted nodes in a burst even though BestBefore was running. Compare expireAfter with
maxAge:
kubectl get nodepool default -o jsonpath='{.spec.template.spec.expireAfter}'
If the gap between them is smaller than a full rollout, nodes reach expireAfter while they are
still queued behind the budget, and expiration deletes them regardless. Widen the gap or speed the
rollout up.
A node I reverted went away anyway
Reverting only helps before Karpenter commits. If the NodeClaim already had the DisruptionReason
condition, or the Node already had the karpenter.sh/disrupted taint, the replacement was already
under way.
Everything stopped after a Karpenter upgrade
Karpenter occasionally changes how it hashes NodePools, and bumps karpenter.sh/nodepool-hash-version
with it. k-belt handles the migration in both directions, but only while it is running. If k-belt
was down during the upgrade, it catches up on start, so check the controller is healthy:
kubectl -n k-belt-system logs deploy/k-belt --tail=50
If rotations are still not happening after that, the contract k-belt relies on may genuinely have changed. Please open an issue with your Karpenter version.