VAULT - AUTO-UNSEAL WITH GCP KMS

If you've run Vault on Kubernetes, sooner or later you hit the awkward moment: a pod restarts, comes up sealed, and somebody has to go unseal it by hand with the Shamir keys. It's only a few minutes, sure, but while Vault is sealed it serves nothing, and everything that depends on it sits and waits. And if the restart comes from a GKE node upgrade, the problem repeats pod after pod.

Before taking it to a real environment, I ran a spike in a separate GCP project to get it clear in my head: Vault in HA with Raft on GKE, auto-unsealing with Cloud KMS. I'll tell you how it went and, above all, where the part that really gives you trouble is: it isn't the seal "gcpckms" block, which is four fields, it's getting the pod to authenticate against Google without carrying any credentials.

What auto-unseal does

Vault encrypts its data with a key (the root key), and that key is itself stored protected. By default it's protected by the Shamir keys that you hand out to people. With auto-unseal, a KMS key protects it instead: when a pod starts, it asks KMS to decrypt the root key and unseals itself. Vault never sees the KMS key, it just asks KMS to encrypt and decrypt.

One detail worth having clear from the start: with auto-unseal, initializing no longer gives you unseal keys but recovery keys. They're for sensitive operations (regenerating the root token, for example), but they can't decrypt the root key, so if KMS becomes unavailable they won't get you out of trouble. Keep them just as safe anyway.

The chicken-and-egg problem

There's a conceptual trap here: Vault is what stores your platform's secrets, but to start it needs Google credentials to talk to KMS. And those credentials can't be... in Vault.

The clean way out is to have no credentials at all. No service account JSON mounted as a secret, which would also be a secret to store and rotate outside Vault. With Workload Identity, the Vault pod identifies itself to Google with its own Kubernetes identity and there's no key to look after.

Step 1: the cluster

The only things that change compared to a normal cluster are the workload pool and the APIs you have to enable. These are the commands, trimmed down to what matters:

PROJECT_ID="<your-project>"
WORKLOAD_POOL="${PROJECT_ID}.svc.id.goog"

gcloud services enable \
  container.googleapis.com \
  iam.googleapis.com \
  cloudkms.googleapis.com

gcloud container clusters create vault-dev-cluster \
    --zone=us-central1-a \
    --num-nodes=3 \
    --machine-type=e2-small \
    --disk-size=50 \
    --workload-pool=${WORKLOAD_POOL} \
    --enable-ip-alias

--workload-pool is what turns on Workload Identity in the cluster. Without it, nothing else works even if the Terraform is perfect. Three e2-small nodes for a spike. Watch your quota: with the default boot disk (100 GB per node), creation failed on me because of the region's SSD_TOTAL_GB quota (250 GB in us-central1, in my project). Adding --disk-size=50 to the command fixed it.

Step 2: the key and the identity, with Terraform

Three pieces: the KMS key, a Google service account (GSA) with permissions only on that key, and the link between the GSA and the Kubernetes ServiceAccount.

resource "google_service_account" "vault_sa" {
  account_id   = var.vault_sa          # "vault-sa"
  display_name = "Vault Service Account"
}

resource "google_kms_key_ring" "vault" {
  name     = "vault-keyring"
  location = "global"
}

resource "google_kms_crypto_key" "vault" {
  name            = "vault-key"
  key_ring        = google_kms_key_ring.vault.id
  rotation_period = "86400s"           # 1 day, just for the spike
  purpose         = "ENCRYPT_DECRYPT"
}

# Encrypt and decrypt with the key
resource "google_kms_crypto_key_iam_binding" "vault_encrypter_decrypter" {
  crypto_key_id = google_kms_crypto_key.vault.id
  role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
  members       = [google_service_account.vault_sa.member]
}

# Vault also reads the key's metadata at startup
resource "google_kms_crypto_key_iam_binding" "vault_viewer" {
  crypto_key_id = google_kms_crypto_key.vault.id
  role          = "roles/cloudkms.viewer"
  members       = [google_service_account.vault_sa.member]
}

# Workload Identity: "this Kubernetes ServiceAccount can act as this GSA"
resource "google_service_account_iam_binding" "vault_ksa_binding" {
  service_account_id = google_service_account.vault_sa.name
  role               = "roles/iam.workloadIdentityUser"
  members = [
    "serviceAccount:${var.project_id}.svc.id.goog[${var.k8s_namespace}/${var.vault_sa}]"
  ]
}

A few things will save you time:

Three permissions, and two roles in my case. Vault's documentation asks for useToEncrypt, useToDecrypt and cryptoKeys.get on the key. The first two come with cryptoKeyEncrypterDecrypter; the third has to be added separately, and it's an easy mistake because that role's name sounds like it covers everything. I used cloudkms.viewer, which includes it but is broader than necessary. I checked it: without that permission the pod doesn't start (Permission 'cloudkms.cryptoKeys.get' denied), and a custom role with only cryptoKeys.get on the key was enough for it to start.

The Workload Identity member has a very specific format: PROJECT.svc.id.goog[NAMESPACE/SA_NAME]. The namespace and the name of the Kubernetes ServiceAccount have to match the pod's exactly. Here they're vault/vault-sa, and if the chart created the SA with a different name, it simply wouldn't work and Google wouldn't tell you anything.

iam_binding is authoritative. It replaces all the members that had that role on that key, it doesn't add one. For a spike I don't care, but in a shared project I'd use iam_member, which only adds.

Rotation. I set the key to rotate daily just for testing. A realistic value is 30 days, because every active version of the key costs money until it's destroyed. And rotating has a consequence for Vault that isn't obvious, which I cover below.

Something my spike doesn't have but I'd add: a lifecycle { prevent_destroy = true } on the key. If someone deletes it, Vault's data becomes undecryptable and no Raft backup saves you, because what it contains is encrypted with a key that no longer exists. It's the only place in the whole setup where a careless terraform destroy has the kind of consequences that really hurt.

I applied this Terraform as is (with the google provider v8.7.0) and it created the six resources. One thing to keep in mind if you repeat it in the same project: Cloud KMS doesn't let you delete a key ring, so you'll have to rename it.

If applying the Terraform gives you an application default credentials error, it's your machine and not the project: the message itself brings the fix, gcloud auth application-default login.

Step 3: Vault with the official Helm chart

With the identity ready, the Vault part takes a few lines of values. These are the values that matter, simplified (I removed probes, resources and logging, which aren't relevant here):

server:
  image:
    repository: "hashicorp/vault"
    tag: "2.0.3"
  # Only for the spike; the chart's default is OnDelete
  updateStrategyType: "RollingUpdate"
  ha:
    enabled: true
    replicas: 2
    raft:
      enabled: true
      config: |
        ui = true

        listener "tcp" {
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_disable     = 1
        }

        storage "raft" {
          path = "/vault/data"

          retry_join {
            leader_api_addr = "http://vault-0.vault-internal.vault.svc.cluster.local:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-1.vault-internal.vault.svc.cluster.local:8200"
          }
        }

        seal "gcpckms" {
          project    = "<your-project>"
          region     = "global"
          key_ring   = "vault-keyring"
          crypto_key = "vault-key"
        }

  serviceAccount:
    create: true
    name: vault-sa
    annotations:
      iam.gke.io/gcp-service-account: "vault-sa@<your-project>.iam.gserviceaccount.com"

Notice two things. First, there are no credentials in the seal "gcpckms" block: if Workload Identity is set up right, Vault gets them from the metadata server transparently. Second, the iam.gke.io/gcp-service-account annotation is the Kubernetes half of the link we declared earlier in Terraform. The link is declared in two places, and if you only do one of them, it fails just the same.

One more detail: if you create the Workload Identity binding right before installing, the pod's first start can fail with a 403 on getAccessToken. That's IAM propagation: for me it recovered on its own in about four minutes, without touching anything.

A note on the two replicas: it's the minimum to see a node join the cluster and unseal itself, but with two servers the quorum is two and the failure tolerance is zero, according to the table in the Vault Raft documentation. That means every pod restart is an outage. For anything real, that same documentation recommends five. And tls_disable = 1 is spike stuff: don't copy it into an environment with real traffic.

Step 4: initialize and check that it really works

With both pods up, initialize Vault on the first one. The command works without any extra flags and, with auto-unseal, gives you five recovery keys and the root token:

kubectl -n vault exec vault-0 -- vault operator init

What to do with those keys in a real Vault (not leaving them on anybody's screen) deserves another post.

Then, inside the pod (kubectl -n vault exec -it vault-0 -- sh) and after a vault login with the root token that init gave you, vault status and the lines I look at are these:

vault status
# Seal Type            gcpckms
# Sealed               false
# Storage Type         raft
# HA Enabled           true

And wait for both pods to show up as voters in the Raft cluster:

until [ "$(vault operator raft list-peers | awk '{print $4}' | grep -c true)" -eq 2 ]; do
  echo "Esperando a que los pods se unan al cluster de Raft..."
  sleep 5
done
vault operator raft list-peers
vault operator raft autopilot state

But what really convinces me is the test of restarting everything and seeing whether it comes back on its own, with nobody touching anything. Mind what it measures: it checks the auto-unseal, not that the service survives a restart (I get to that below). Vault's Helm chart uses OnDelete by default, which is the right thing in a real environment. For quick tests I change it to RollingUpdate in the values (that's why it appears in the step 3 snippet), so I can restart all the pods with a single command. Without that change, rollout restart restarts nothing with the default value. But notice why the default is what it is: every restart or upgrade causes a short outage because of the leader election, and with RollingUpdate you don't control the order. Kubernetes knows nothing about Vault's roles and goes from the highest ordinal to the lowest. Picture a five-node cluster where the leader is vault-4: leadership can move to vault-3, which Kubernetes restarts next, then to vault-2, and so on, with several elections and several short outages. With OnDelete you control the flow (you delete the pods one by one, standbys first and the leader last), at the cost of needing delete permissions in Vault's namespace. For this check, with no traffic, RollingUpdate is fine:

kubectl -n vault rollout restart statefulset vault
kubectl -n vault rollout status statefulset vault --timeout=300s

rollout status waits for each pod to be Ready, and with auto-unseal that means it has unsealed itself.

Then run the same checks again. If both pods come back as Sealed false and as voters without anybody giving them a key, auto-unseal works. It's cheaper to find out a link is broken in the spike than on a Saturday at 3 a.m.

What auto-unseal doesn't fix

Auto-unseal takes the person out of the loop, but it doesn't take away the restart. A pod that starts stays sealed until KMS decrypts the root key for it, and in the meantime it serves no requests. What happens around it depends on which one you restart, and this is what Vault's Raft documentation says:

• A standby in a cluster of three or more: the leader keeps serving and there's no outage.
• The active node: the cluster has to elect a new leader. With the default values (performance_multiplier = 5), the heartbeat and election timeouts are on the order of five seconds. That's seconds, not minutes, but it's not zero. In my spike I measured it once, with Vault 2.0.3: I deleted the active pod of two voters and queried /v1/sys/leader every half second from another pod. About 16 seconds went by between the first failed request and the last moment without a leader. It's a single run, take it as an order of magnitude.
• All of them at once: without a majority of nodes started and unsealed there's no leader, and the cluster is unavailable. With auto-unseal that wait no longer depends on anybody, but it's there.
• Two nodes, like in my spike: any restart is an outage, because of the quorum.

And one more detail about how you restart. The official upgrade guide for Kubernetes insists on using OnDelete and not RollingUpdate, because you have to upgrade the standbys before the active node: delete the pods one by one, first taking each standby out of Raft with vault operator raft remove-peer, and the active one last. For the spike, with no real traffic, RollingUpdate is convenient for me and, for the same reason, I skip taking the node out of Raft. With production traffic the efficient thing is to keep leader elections to a minimum, like the guide does: standbys first and the leader last. One practice of mine that the guide doesn't mention: before deleting the leader's pod I run vault operator step-down. That way I hand over leadership on purpose, instead of letting Vault sort it out when it notices the leader has disappeared. It's a single election, and a controlled one.

When it doesn't start

The symptom is always the same (Vault can't talk to KMS and stays sealed) even though the cause can be in three different places. The order I'd look in, from cheapest to most expensive:

# 1. Is the ServiceAccount annotation what you think it is?
kubectl -n vault get sa vault-sa -o yaml | grep iam.gke.io

# 2. Does the pod say who it is? (indicative, proves nothing on its own)
kubectl -n vault exec vault-0 -- \
  wget -qO- --header "Metadata-Flavor: Google" \
  http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email

Careful: that call does not prove the identity. With the ServiceAccount annotated but without the binding, /email still returns the GSA's email. And with an unannotated ServiceAccount it returns PROJECT.svc.id.goog. The real test is to ask for a token:

kubectl -n vault exec vault-0 -- \
  wget -qO- --header "Metadata-Flavor: Google" \
  http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token

If it returns a token and the email is your GSA's, the identity works and the problem is in the permissions on the key (is the viewer missing?) or in the key ring name or the region in the seal block. If it returns 403 Forbidden, the annotation is there but the binding is missing or its member doesn't match. And if the email is PROJECT.svc.id.goog, the ServiceAccount is missing the annotation. That discriminator saves you a lot of going around in circles.

Rotating the KMS key: which version Vault really uses

If you keep only one thing from this section, make it this: don't destroy the old versions of the KMS key, not even after rotating it and restarting the pods. Vault may still need them to unseal.

Imagine Vault was initialized when the primary was version 1 and you then rotate the key up to version 4. The root key was encrypted with version 1 and stays that way: according to the Cloud KMS documentation, rotating re-encrypts nothing and doesn't disable the old versions, and the GCP seal documentation says they shouldn't be disabled or deleted because they're used to decrypt old data.

So far, logical. What I expected is that, when a pod starts after a rotation, Vault would re-encrypt the root key with the primary version, because there's a function in the code (UpgradeKeys, in seal_autoseal.go) that does exactly that when the key identifier doesn't match the current one. I tested it on Vault 2.0.3 and it's not that simple.

To know when a disabled version has really stopped working, I encrypted a file with that version and decrypted it with gcloud kms decrypt every 30 seconds. After disabling it, the first failure came a little over 60 seconds later and three in a row at 125: a decrypt at 30 seconds still worked, and anyone who trusts it will think the version is no longer needed. Each test went like this: a clean Vault (chart 0.34.0, two voters), rotate the key once, do what the table says, disable the version Vault was initialized with, wait for the decrypt to fail three times in a row, delete both pods and see whether they started.

When they didn't start, the error was cryptoKeyVersions/1 is not enabled, current state is: DISABLED, and enabling the version again recovered them without touching Vault. In the log, every start says value has no key-id in common with currently healthy seals: Vault knows the identifier doesn't match.

What matters are the last two rows: the same scenario gave different results. Neither a restart, nor a failover, nor a step-down guarantees that the root key gets re-encrypted, and the log doesn't show when it happens. I don't know what triggers it, and from the outside you can't tell. I also didn't test whether waiting for hours does it on its own. These are few runs on a single two-voter cluster, but for the rule it's enough that sometimes it doesn't happen.

What the documentation lists as explicit ways to re-encrypt the root key is a root key rekey (with auto-unseal, by presenting recovery keys) or a change in the seal block, which triggers a migration. Don't mix it up with vault operator rekey -target=recovery, which changes the recovery keys, or with vault operator rotate, which rotates Vault's internal keyring: neither touches the root key.

If Vault still has the root key encrypted with version 1 and you destroy it, it won't be able to unseal, and the recovery keys won't save you, because they can't decrypt the root key.

There's a reversible way to check it in a spike: rotate the key, encrypt a file with the old version, disable it (don't destroy it) and wait for the decrypt of that file to fail three times in a row. Then delete the pods. If Vault starts, it no longer depended on it; if it doesn't, enable it again and everything is as it was. Repeat it, because a single result proves nothing, and do it in a test environment, never in one with traffic.

To wrap up

The funny thing is that the "Vault" part of all this takes about a dozen lines and the rest of the post is about identity. Auto-unseal is like automatic certificates: once you have it set up right, you forget it exists, which is exactly what it's for. The difference between remembering to unseal Vault and Vault unsealing itself comes down to a chain of four links (key, GSA, ServiceAccount, annotation) where any one that fails gives the same symptom.

One last thing: this spike ran on a cluster without a mesh. Afterwards I took it to an environment with Istio in ambient mode, with ztunnel, and it has its own issues, which I'll leave for another post.

My advice: set up the pod restart test before you actually need it, and repeat it every time you touch the identity configuration.