Monitoring
Status
kubectl get bestbefores
NAME MAX AGE MATCHED STALE DRIFTED AGE
default-pool 504h 42 6 6 2m11s
spot-pool 168h 18 0 0 2m11s
| Column | Meaning |
|---|---|
MATCHED |
NodeClaims the selector matches |
STALE |
Matched NodeClaims older than maxAge |
DRIFTED |
NodeClaims this policy has marked that Karpenter hasn’t replaced yet |
In a healthy rotation DRIFTED tracks STALE and both fall to zero. If the policy sets
maxConcurrent, DRIFTED stays at that limit and STALE stays above it until the rotation
completes, which is expected.
Without a limit, a STALE count that stays above DRIFTED means k-belt could not mark those
NodeClaims: they are paused, another policy owns them, or they are missing Karpenter’s hash
annotations.
The Ready condition carries configuration problems, such as a selector the API server accepted but
k-belt cannot parse:
kubectl get bestbefore default-pool -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'
Events
k-belt writes events on the NodeClaim, not on the policy:
kubectl get events --field-selector involvedObject.kind=NodeClaim | grep BestBefore
4m Normal BestBeforeExceeded nodeclaim/default-8vbh7 NodeClaim is older than BestBefore default-pool maxAge 504h; marked drifted for graceful replacement
1m Normal BestBeforeRestored nodeclaim/default-gjpbg No BestBefore considers the NodeClaim stale any more (previously drifted by default-pool); restored its nodepool hash
Karpenter’s own events on the Node explain why a marked node has not been replaced yet. Look for
DisruptionBlocked.
Metrics
The controller serves these on its metrics endpoint, labelled by policy:
| Metric | Meaning |
|---|---|
kbelt_bestbefore_matched_nodeclaims |
NodeClaims the selector matches |
kbelt_bestbefore_stale_nodeclaims |
Matched NodeClaims older than maxAge |
kbelt_bestbefore_drifted_nodeclaims |
NodeClaims marked and not yet replaced |
kbelt_bestbefore_oldest_drift_timestamp_seconds |
When the longest-standing drift started, 0 if none |
The alert worth having
A node that stays marked for a long time is a rotation that has stalled, and it will eventually be force-expired instead. Alert on the age of the oldest outstanding drift:
- alert: BestBeforeRotationStalled
expr: |
kbelt_bestbefore_drifted_nodeclaims > 0
and time() - kbelt_bestbefore_oldest_drift_timestamp_seconds > 86400
for: 15m
annotations:
summary: "BestBefore has had a node waiting to rotate for over a day"
Pick a threshold from your own numbers: comfortably longer than a normal rollout, comfortably shorter
than expireAfter - maxAge.
The chart protects the metrics endpoint with authn/authz by default. To scrape it, bind your
scraper’s ServiceAccount to the k-belt-metrics-reader ClusterRole, or set
metrics.serviceMonitor.enabled if you run the Prometheus Operator.