VAULT - RECOVERING THE RAFT QUORUM

Some mistakes teach you more than any tutorial. While scaling a Vault down from 5 to 3 replicas in a GKE test environment, I broke the Raft quorum. I'll tell you how I recovered it with a peers.json, and when you shouldn't touch it.

Afterwards I repeated the incident and the procedure on a local cluster (kind, Vault 1.20.4) to check every step, and a couple of traps came up that the official procedure doesn't tell you about. You'll see them marked as "watch out" throughout the post.

What had happened

I was spreading Vault across several GKE zones, with required anti-affinity and topologySpreadConstraints. Volumes are tied to the zone and the nodes in the new zones didn't exist yet, so to force the autoscaler to create them I had to delete the volume of the pod I wanted to move.

Before that I had gone from 5 to 3 replicas in one go, deleting vault-4 and vault-3 along with their PVCs and PVs, and without removing them from Raft with remove-peer. I thought Vault would readjust by itself and that I'd be left with the tolerance to lose a node. In reality Raft still counted 5 voters, with a quorum of 3 and 3 live pods: tolerance 0. When I deleted the volume of vault-2 to move it to another zone, vault-0 and vault-1 were left, 2 of 5, and the cluster was left without a leader.

The first thing I tried was scaling down, but I quickly started reading the documentation instead of carrying on trying things blindly. In the end I fixed it with the official peers.json procedure that I describe here. The right procedure for scaling down is in Vault - Reducing from 5 to 3 replicas.

What quorum is and when you lose it

Vault replicates data with Raft (integrated storage) and needs a simple majority of nodes to elect a leader. The official formula is ceil((N + 1) / 2):

In my case the Raft configuration said 5 voters and there were only 2 live pods. With fewer than 3 there's no majority, no leader, and API calls return an HTTP 500, including vault operator raft list-peers. Watch out: a 500 can also come from a TLS or storage problem, so always confirm with vault status before touching anything.

When to use this procedure (and when not to)

It's a last resort. Use it only if all of this holds:

• A leader can't be elected: vault status shows HA Mode: standby and there's no active node. Watch out: Active Node Address may show as <none>, but in my local test it showed the IP of the leader I had just lost. Don't rely on that field alone: what counts is that list-peers fails with a 500.
• list-peers fails or returns no configuration.
• The nodes still alive don't add up to a quorum and the missing ones aren't going to recover. In my case 2 pods were left with the PVC in Bound, out of 5 voters.

Don't use it if the failure is a network partition, if the pods are restarting but not lost, or if the leader is still reachable. If there are two or more live but isolated nodes, applying this can cause a split-brain. First confirm that the missing nodes are really lost.

Step 1: find the survivor and make a backup

In the commands I use a variable with the name of the StatefulSet (and <namespace> for the namespace, which you substitute). Set it in your terminal before starting; you'll need it in steps 3 and 4:

vault_name="vault"   # name of the StatefulSet and prefix of the pods

We're going to recover from a single node. Pick as the survivor a pod that has its PVC in Bound and returns Initialized: true, Sealed: false and HA Mode: standby. vault status needs no token:

kubectl get pvc -n <namespace>
kubectl exec ${vault_name}-0 -n <namespace> -- vault status

Before going on, a copy of the Raft data. It's a copy that's better than nothing, not a guarantee. I tested it by restoring it into a separate pod, with the same image: after applying a peers.json to it, it started, unsealed with the original key and kept my data (a policy created earlier). But that was with the cluster receiving no writes; a tar with Vault running can leave the Raft files inconsistent, and a snapshot (vault operator raft snapshot save) doesn't work without a leader. It also needs tar in the image (the Vault one has it through busybox):

kubectl exec ${vault_name}-0 -n <namespace> -- tar czf /tmp/vault-raft-backup.tar.gz /vault/data
kubectl cp <namespace>/${vault_name}-0:/tmp/vault-raft-backup.tar.gz ./vault-raft-backup.tar.gz

Step 2: stop the rest of the live pods

The documentation says to stop the servers that are still running but not healthy, and, in a cluster of 5 nodes or with non-voting nodes, to stop the other nodes as well before recovering. In my case 2 live pods with a valid PVC were left (vault-0 and vault-1) and, if you leave the second one running, it carries on with the 5-peer configuration. For the single-survivor procedure it has to be stopped.

If you've deleted a pod's volume, the StatefulSet recreates it with a new, empty volume (in my local test, Initialized: false, not joining anything). It's no use as a survivor either, and it also gets stopped by the scale-down.

Since the StatefulSet removes the pods in reverse order, scaling to 1 leaves only ${vault_name}-0:

kubectl scale statefulset ${vault_name} --replicas=1 -n <namespace>
kubectl wait --for=delete pod/${vault_name}-1 -n <namespace>

This assumes your survivor is -0, as in my case. If it isn't, scaling down won't work for you, because it always takes the highest ordinals, and you'd have to stop the other pods in some other way that I don't cover here.

The pods you stop keep Raft data with the old configuration. Further down you'll see what to do if they don't join afterwards.

Step 3: the node-id and the address

Read the node's identifier, a UUID that Vault generates the first time it starts, and put in local variables the two things the file in the next step needs:

node_id=$(kubectl exec ${vault_name}-0 -n <namespace> -- cat /vault/data/node-id)
echo "${node_id}"
# Example: 5bdda2aa-8079-7887-230b-f974c2cd3487

The address is the headless service's, on port 8201, which is the cluster port (not the API's 8200):

address="${vault_name}-0.${vault_name}-internal.<namespace>.svc.cluster.local:8201"

Step 4: the peers.json

Create in Raft's data directory a peers.json that lists only the surviving node. Watch out how you create it: I do it from your terminal, passing the content through standard input. The first time I tried it locally I wrote it with a cat > ... << EOF inside the pod, and there ${vault_name} doesn't exist, so the address ended up as -0.-internal.<namespace>.svc.cluster.local:8201:

kubectl exec -i ${vault_name}-0 -n <namespace> -- sh -c 'cat > /vault/data/raft/peers.json' << EOF
[
  {
    "id": "${node_id}",
    "address": "${address}",
    "non_voter": false
  }
]
EOF

# Check that what was written is what you expect
kubectl exec ${vault_name}-0 -n <namespace> -- cat /vault/data/raft/peers.json

The id is the UUID from step 3, the address is the headless service's and non_voter must be false so the node can be leader. Review the cat before going on, because Vault doesn't complain about a bad address: in my test with the broken address it accepted the file, elected a leader and everything looked fine. The problem showed up later, when the leader changed: the other nodes tried to resolve that absurd address, the node was left without contact and failure tolerance dropped to 0. And the file gets deleted anyway, so if you get it wrong you'll have to create it again (and repeat step 5). If you want a copy, save it before restarting.

Step 5: restart and check

Delete the vault-0 pod. Kubernetes recreates it and Vault reads the file when the node is unsealed: with auto-unseal that should be at startup (I haven't tested it, my local cluster didn't have it), but if you don't have it, the file stays there while the pod is sealed and you'll have to unseal by hand after the restart:

kubectl delete pod ${vault_name}-0 -n <namespace>
kubectl get pods -n <namespace> -w

When it's Running, check in the logs that it processed it:

kubectl logs ${vault_name}-0 -n <namespace> | grep -i 'raft recovery'

You should see it start the recovery, find the new configuration and delete the peers.json. The file disappearing is the confirmation that it was applied. It's a single-use emergency mechanism.

Step 6: is there a leader?

vault status needs no token, but list-peers does. Watch out: an export VAULT_TOKEN in your terminal doesn't reach the container, so pass it in the command itself (or get into the pod and run vault login):

kubectl exec ${vault_name}-0 -n <namespace> -- vault status
kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft list-peers

At first it may show as standby while auto-unseal finishes. Wait a few seconds and repeat until you see HA Mode: active. In list-peers a single node should appear as leader.

Giving it high availability back

This is the most dangerous trap of the procedure. It looks like scaling back up and letting retry_join do the rest would be enough, but the pods you stopped in step 2 keep the old Raft configuration in their PVC, and that can turn against you.

In my local test, after recovering a 3-voter cluster with a single survivor, I scaled up to 3 without touching anything. The two pods that come back, with their old configuration, make a majority between them (2 of 3), so they formed another cluster, with its own leader. Result: two pods with HA Mode: active, each with a different view in list-peers, and both with the vault-active=true label by which the vault-active service chooses where to send traffic, so requests could land on different clusters. It's a split-brain, the same risk I warned you about at the beginning, but caused by the procedure itself. In my local test with 5 voters and one old pod it didn't happen: that pod alone, with its 5-voter configuration, doesn't make a majority. But it depends on how many old nodes come back, and there's no point in taking the chance.

What I'd do by default is leave the other pods clean before scaling them up. They're still stopped from step 2, so the PVC is deleted right away (names depend on your chart, look at them first with kubectl get pvc). This is never done on the survivor's pod, or on the leader:

kubectl get pvc -n <namespace>
kubectl delete pvc data-${vault_name}-1 data-${vault_name}-2 -n <namespace>
kubectl scale statefulset ${vault_name} --replicas=3 -n <namespace>

They start empty, join the leader through retry_join as new peers and replicate everything from it (without auto-unseal you'll have to unseal them by hand; in my test I had to do it twice: the first response still said Sealed true). In my test they ended up with 3 voters, Healthy: true and Failure Tolerance: 1.

If you had already scaled up without deleting anything, check as soon as possible that there aren't two active nodes (I cover it in the next section). If there are, scale back down to 1 replica, delete the PVCs of the others and scale up again. And if a node with old data has nobody to form a majority with, it doesn't join but doesn't warn you either: in my test vault status showed it as standby and unsealed, and only list-peers revealed it wasn't in the cluster (in its logs, repeated elections against the old nodes).

Before calling it done

A plain vault status doesn't tell you whether the cluster is really healthy. The command that does is:

kubectl exec ${vault_name}-0 -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft autopilot state

Notice that Healthy must be true, Failure Tolerance must be 1 or more (if it's 0, it works but can't survive even one node going down) and every node must show as alive and as Voter.

But this only looks at the cluster from the pod where you run it. In my test's split-brain, autopilot state from the survivor said Healthy: true (with 1 voter and tolerance 0) and knew nothing about the other cluster. That's why it's worth doing two more checks:

# Run list-peers from each pod: they must give the same list
for i in 0 1 2; do
  kubectl exec ${vault_name}-$i -n <namespace> -- env VAULT_TOKEN=<your-token> vault operator raft list-peers
done

# Only one pod with vault-active=true
kubectl get pods -n <namespace> -L vault-active

Look at the Address column too: all the addresses must have valid pod names. One like -0.-internal... is the sign that the peers.json was written wrong.

You should also know what the risk is. The documentation warns that peers.json can cause entries in the Raft log that were never committed to be treated as committed. And since you've lost several servers, the committed state you're left with may be incomplete. That is, a recent write may be missing, or one that was never committed may appear. That's why it's the last resort and why it isn't automated. Secrets stored a long time ago are still there as long as the survivor's PVC is intact (in my local test, a policy created before the recovery was still there after two recoveries in a row); what can fail are recent writes.

To wrap up

What I take away from this is that I could have spared myself the whole thing, for two reasons. The first, that I should have scaled down one at a time, removing each peer from Raft with remove-peer and checking with autopilot state that failure tolerance was still 1. The second, that if I hadn't deleted vault-4 and vault-3 along with their volumes, it would have been enough to scale back up to 5 replicas and recover the cluster without touching peers.json. That only works if the PVCs still exist and you haven't run remove-peer on those pods: afterwards, their volumes have obsolete data and you have to delete them before scaling back up. With the cluster recovered I could have calmly figured out the problem before continuing with the 3-zone distribution. Deleting what's left over too early took that way out from me.

If you have to use it, take the copy before touching anything.

Official documentation:

• Recovering from lost quorum
• Integrated Storage and recovery with peers.json
• Integrated Storage internals
• Reference for vault operator raft
• Vault quorum recovery on Kubernetes (HashiCorp Support)
• Recovering a permanently lost quorum with Raft (HashiCorp Support)