Scaling down the replicas of a Vault with Raft looks like a matter of changing a number in a YAML. That's how I did it in a GKE test environment, it seemed to go fine, and one step later I was left without quorum. I'll tell you what happened and the procedure I should have followed. It applies to any reduction, not just 5 to 3: I use that case because it's easy to reason about.
How it all started
The idea was to have Vault in high availability spread across three GKE zones. With a 3-node cluster we had already done it and it had gone well. But the test environment had 5 replicas, all in the same zone, so we had to go down to 3 before spreading them.
To spread them we used required anti-affinity and topologySpreadConstraints. And there was one more detail: volumes are tied to the zone where they were created, and the nodes in the new zones didn't exist yet, because we were the first to use them. For the autoscaler to create them, we had to delete the volume of the pod we wanted to move.
So I did it in two steps:
- I went from 5 to 3 by deleting
vault-4andvault-3, with their PVCs and PVs. - I deleted the
vault-2pod and its PV to force it to be born in another zone.
The second step left the cluster without quorum.
What I thought and what it was
I took for granted that Vault managed itself: when two pods disappeared, the cluster would readjust to 3 and have the tolerance to lose one more. That's not how it went.
Raft doesn't know you've shut down a pod. To it, those two nodes are still voters that don't respond. With 5 voters the quorum is 3, and with 3 live pods there was none to spare: failure tolerance was 0. When I deleted another pod, 2 of 5 were left, and without a majority there's no leader.
| What I thought | What it was | |
|---|---|---|
| Raft voters | 3 | 5 |
| Quorum | 2 | 3 |
| Live pods | 3 | 3 |
| Failure tolerance | 1 | 0 |
Notice that going from 5 to 3 in one go doesn't break quorum: 3 live pods still meet the 3 of 5. What breaks it is the margin. While the pods are terminating there's no tolerance left, and if something goes wrong at that moment there's no way back. Something similar happens if you skip steps, like going straight from 5 to 2: then you do lose the whole quorum.
I checked it afterwards on a local test cluster (kind, Vault 1.20.4) so I wouldn't rely on my memory alone. With 5 voters and 4 pods, autopilot state said Failure Tolerance: 1; with 5 voters and 3 pods, Failure Tolerance: 0 and Healthy: false. In another test, with 4 voters and 3 pods, deleting a third pod (2 of 4) left the cluster without a leader, and neither list-peers nor autopilot state responded: error 500.
What about automatic cleanup? The documentation explains that Autopilot comes enabled, but dead server cleanup (cleanup-dead-servers) has to be enabled by hand and requires configuring min-quorum. Also, the default threshold for considering a server dead is 24 hours. In my case it wasn't going to act in time, even if it had been on.
With the 3-node cluster from before it didn't happen to us because its Raft configuration said 3. When a pod was deleted, 2 of 3 were left, which is quorum.
The right procedure
This is the procedure we arrived at, which matches the one in the official Integrated Storage tutorial. Before the commands, two rules that would have saved me the headache:
Remove the peer from Raft before shutting down the pod. If you shut it down first, Raft keeps counting a voter that doesn't respond and you're left without tolerance until you remove it. I measured exactly that: with 4 voters and 3 pods, Failure Tolerance: 0. Doing it the other way around (remove-peer and then scale down) the tolerance never dropped below 1 at any step, and I ended up with 3 voters and tolerance 1.
You might think that with the pod still running retry_join will put it back into the cluster and undo what you just did. In my test it didn't happen: the peer stayed out, and when the pod restarted retry_join failed with node has been removed from the HA cluster. All vault data for this node must be cleaned up before it can be added back. An expelled node doesn't come back on its own.
Scale down one at a time and check between steps. Going replica by replica doesn't give you more tolerance in itself; what it gives you is the chance to look at autopilot state before touching the next pod and to stop if something doesn't add up.
The pattern of each step looks like this:
step-down if leader → remove-peer → check peers → scale down → check the pod has terminated → check peers
To avoid mistakes when substituting names, I set a variable first (I write the namespace as <namespace> in every command):
vault_name="vault" # name of the StatefulSet and prefix of the pods
I use ${vault_name}-0 as the pod to run the commands from. In a StatefulSet the pods are removed in reverse order, so 0 is always still there.
One thing before starting: list-peers, remove-peer, autopilot state and step-down need a token. Without it, the API answers 403 permission denied. Also, step-down requires sudo on sys/step-down in its policy: with a policy that only had update, it gave me a 403 too. An export VAULT_TOKEN in your terminal doesn't reach the container, so I pass it in the command itself with env VAULT_TOKEN=<your-token>.
Before you start
With the cluster healthy. If any of this fails, don't continue:
# All pods Running and Ready
kubectl get pods -n <namespace>
# Unsealed, HA active and with a leader
kubectl exec ${vault_name}-0 -n <namespace> -- vault status
# 5 peers, 1 leader and 4 followers, all voters
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft list-peers
# Healthy: true and Failure Tolerance: 2
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft autopilot state
Phase 1: the change in Git and stopping ArgoCD
The change itself is one line in the manifest, which goes up with its merge request:
replicas: 3
If ArgoCD has auto-sync enabled, as soon as you merge it will start scaling down the pods on its own, without you controlling the order. So disable it before merging:
ArgoCD UI → Application vault → App Details → Disable Auto-Sync
Phase 2: removing the peers and scaling down the pods
Since Vault runs in a StatefulSet, Kubernetes removes the pods in reverse order: first vault-4, then vault-3. The process is identical for each one and you have to finish it completely before moving on to the next.
First look at how the peers are doing:
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft list-peers
You should see something like this:
Node Address State Voter
---- ------- ----- -----
5bdda2aa-8079-7887-230b-f974c2cd3487 vault-0.vault-internal:8201 leader true
3f1e9c12-4521-6643-ab1d-e829c1ab4512 vault-1.vault-internal:8201 follower true
7ca2d341-9901-2234-cc3e-f112b3cd7821 vault-2.vault-internal:8201 follower true
a91c04be-33d7-4f0a-8c52-0b6e7d19f6a3 vault-3.vault-internal:8201 follower true
e2b7d5f0-6a18-4c93-9d41-5c8a02e7b1d4 vault-4.vault-internal:8201 follower true
Watch the Node column: those aren't the pod names, but UUIDs that Vault generates the first time each node starts. remove-peer takes that UUID, not the pod name. To find out which one is vault-4's, look at the Address column, which does carry the pod name.
If vault-4 is the leader, do a step-down on it first and confirm with list-peers that another pod has taken over. Leadership could come back to the same node, so if it still is, repeat it:
kubectl exec ${vault_name}-4 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator step-down
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft list-peers
Now, with the pod still running, you remove it from Raft with the UUID you saw in list-peers for vault-4:
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> \
vault operator raft remove-peer <uuid-of-vault-4>
In my test the expelled pod restarted on its own and stayed sealed. Don't be alarmed if you see it: it's the node you just kicked out, and you're about to shut it down. Scale down to 4 replicas and wait for it to disappear completely:
kubectl scale sts ${vault_name} --replicas=4 -n <namespace>
kubectl wait --for=delete pod/${vault_name}-4 -n <namespace>
And the check. list-peers was what we used to see how the cluster was doing, but it doesn't give the information that matters most in this operation: how much failure tolerance is left. For that, also look at Autopilot's state:
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft list-peers
# Expected: 4 peers, 1 leader, 3 followers
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft autopilot state
# Look for: Healthy: true and Failure Tolerance: 1
With 4 voters the quorum is still 3, so the tolerance is 1. Autopilot takes a few seconds to reflect the changes: right after shutting down a pod, mine still showed the previous tolerance. Wait a bit and look again before continuing, and don't continue until both commands say what's expected.
Repeat the same with vault-3: step-down if it's the leader, remove-peer <uuid-of-vault-3>, scale --replicas=3, kubectl wait --for=delete pod/${vault_name}-3 and the same double check. With 3 peers you should see 1 leader, 2 followers and Failure Tolerance: 1. The quorum has gone like this:
5 peers → quorum 3 (removing vault-4)
4 peers → quorum 3 (removing vault-3)
3 peers → quorum 2 ✅ final state
Until you run remove-peer on a pod, its PVC keeps its Raft data and it's your way out if something goes wrong, which is why the PVCs aren't deleted until phase 4. After remove-peer it no longer works: I'll tell you what happens further down.
Phase 3: giving control back to ArgoCD
Since the merge request already left replicas: 3, the real state and the desired one match. Turn auto-sync back on and ArgoCD will have nothing to correct:
ArgoCD UI → Application vault → App Details → Enable Auto-Sync
Phase 4: deleting the orphaned PVCs
When everything is done and the cluster is healthy, the PVCs of the deleted pods remain. They aren't deleted on their own: by default a StatefulSet keeps them when scaling down (persistentVolumeClaimRetentionPolicy with Retain). First check that everything is still fine:
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft autopilot state
# Expected: Healthy: true, 3 peers
And then you delete them. The names depend on your chart; here I use data-vault-N as an example, so look at them first with the get:
kubectl get pvc -n <namespace>
kubectl delete pvc -n <namespace> data-${vault_name}-3 data-${vault_name}-4
kubectl get pvc -n <namespace>
# Expected: only data-vault-0, data-vault-1 and data-vault-2
Finish with a vault status to confirm that HA is still active and there's no stuck standby.
The volumes and the safety net
Here's what I'd do differently: don't delete the PVCs before phase 4. I deleted those of vault-3 and vault-4 as soon as I scaled them down, and with that I was left with no way out.
The PVCs are your recovery mechanism, but only as long as you haven't run remove-peer on that pod. There are two situations:
Before remove-peer. The pod is still a valid Raft peer and its PVC keeps the data. Scaling back up is enough and it rejoins on its own. This was my case: I hadn't run remove-peer at any point, so with the volumes intact it would have been enough to go back up to 5 and recover the cluster, and then investigate calmly before continuing with the three zones. I tested it on the local cluster: after losing quorum, the pod came back with its PVC and joined as a follower without touching anything else (without auto-unseal I had to unseal it by hand).
kubectl scale sts ${vault_name} --replicas=5 -n <namespace>
kubectl wait --for=condition=Ready pod/${vault_name}-3 pod/${vault_name}-4 -n <namespace>
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft list-peers
# Expected: 5 peers, 1 leader, 4 followers
After remove-peer. The node has already been expelled from the cluster. If you bring it back up with its PVC, it will start with Raft data that the cluster no longer recognizes and it won't join: it'll tell you so in the log with the same node has been removed from the HA cluster. All vault data for this node must be cleaned up before it can be added back. You have to delete the PVC first so the pod starts clean and joins as a new peer through retry_join:
# Delete the obsolete PVC of each expelled peer
kubectl delete pvc -n <namespace> data-${vault_name}-4
kubectl delete pvc -n <namespace> data-${vault_name}-3
# Scale back up: they'll start without data and join on their own
kubectl scale sts ${vault_name} --replicas=5 -n <namespace>
kubectl wait --for=condition=Ready pod/${vault_name}-3 pod/${vault_name}-4 -n <namespace>
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft list-peers
On the local cluster the nodes joined with a new UUID, as voting followers.
All of this assumes you have auto-unseal configured (in my local test I didn't and had to unseal the pods by hand after every restart). And if the quorum doesn't recover by itself, it's time for the emergency procedure with peers.json, which I cover in Vault - Recovering the Raft quorum.
To wrap up
What I take away is that "it works" and "it has margin" aren't the same thing. With 3 live pods everything responded, the API answered and there wasn't a single error, but the cluster was one step from falling over and I didn't know it. list-peers showed me 5 voters without telling me how many were responding, and no alarm signal. It was autopilot state that gave me the key information (Healthy: false and Failure Tolerance: 0), and since then it's my way of checking the cluster's state in any operation of this kind. By the time I looked, I had already deleted the safety net.
The rule I keep is simple: don't destroy what's left over until you've checked that the change that justified removing it works.
If you want to go deeper, this is the documentation I used:
- Reference for
vault operator raft(remove-peer,list-peers,autopilot) - Reference for
vault operator step-down - Integrated Storage concepts
- Integrated Storage internals (quorum, fault tolerance, scaling)
- Official Integrated Storage tutorial
- Lost quorum recovery
