Series: Vault as Simple SSO for Multiple Kubernetes Clusters (Part 4 of 5)
In Part 3 we enabled the OIDC auth method and the two Kubernetes secrets engines, but they were not yet connected to anything real. In this post we wire everything together using only the EC2 public DNS name for external references. We configure Vault OIDC to trust Keycloak, we tell the k8s engines how to reach the kind clusters, we create the policies that allow an OIDC identity to request tokens, and then we walk through the full CLI flow from your laptop:vault login -method=oidc role=human, generate short-lived Kubernetes tokens withvault write k8s-*/creds/..., and use them directly withkubectl.
Prerequisites
The full lab must be running on the EC2:
- Keycloak on port 8080, reachable at the public DNS
- Vault dev server listening on
0.0.0.0:8200 - Two kind clusters (
cluster-aon 6443,cluster-bon 6444) vault-demo-saServiceAccount + admin binding in both clusters- The
vaultOIDC client already exists in thesrerealm (from Part 3)
On the EC2 host:
source ./scripts/vault-k8s-lab/env.shPUBLIC_DNS=lab.onlysre.dev VAULT_ADDR=http://lab.onlysre.dev:8200 KEYCLOAK_URL=http://lab.onlysre.dev:8080
Verify reachability:
curl -sI ${KEYCLOAK_URL} | head -1
vault status
kind get clusters
kubectl config get-contexts
HTTP/1.1 302 Found
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 2.0.3
Build Date 2026-06-17T12:39:45Z
Storage Type inmem
Cluster Name vault-cluster-e525ecfb
Cluster ID c0b25d9a-8ec9-b535-dfb8-81c098c4e4c0
HA Enabled false
cluster-a
cluster-b
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* kind-cluster-a kind-cluster-a kind-cluster-a
kind-cluster-b kind-cluster-b kind-cluster-b
From your laptop (replace the hostname with the current instance):
export PUBLIC_DNS=lab.onlysre.dev
curl -k https://${PUBLIC_DNS}:6443/version
curl -k https://${PUBLIC_DNS}:6444/version
curl http://${PUBLIC_DNS}:8200/v1/sys/health
curl -I http://${PUBLIC_DNS}:8080
Keycloak Realm and Client (Public DNS Only)
We use the public DNS for everything except the single localhost:8250 callback used by the future kubectl plugin.
Log into the admin CLI using the public address:
docker exec keycloak /opt/keycloak/bin/kcadm.sh config credentials \
--server http://${PUBLIC_DNS}:8080 \
--realm master \
--user admin \
--password admin
Logging into http://lab.onlysre.dev:8080 as user admin of realm master
Ensure the sre realm exists and has SSL disabled (demo only):
docker exec keycloak /opt/keycloak/bin/kcadm.sh get realms/sre --fields realm,enabled,sslRequired 2>/dev/null || \
docker exec keycloak /opt/keycloak/bin/kcadm.sh create realms -s realm=sre -s enabled=true
docker exec keycloak /opt/keycloak/bin/kcadm.sh update realms/sre -s sslRequired=NONE
docker exec keycloak /opt/keycloak/bin/kcadm.sh update realms/master -s sslRequired=NONE
docker exec keycloak /opt/keycloak/bin/kcadm.sh get realms --fields realm,enabled,sslRequired
[
{
"realm" : "sre",
"enabled" : true,
"sslRequired" : "none"
},
{
"realm" : "master",
"enabled" : true,
"sslRequired" : "none"
}
]
Recreate the vault confidential client with the correct redirect URIs (public DNS + localhost for the plugin):
docker exec keycloak /opt/keycloak/bin/kcadm.sh delete clients/$(docker exec keycloak /opt/keycloak/bin/kcadm.sh get clients -r sre -q "clientId=vault" --fields id --format csv | tail -1 | tr -d \") -r sre 2>/dev/null || true
docker exec keycloak /opt/keycloak/bin/kcadm.sh create clients -r sre \
-s clientId=vault \
-s enabled=true \
-s clientAuthenticatorType=client-secret \
-s standardFlowEnabled=true \
-s "redirectUris=[\"http://localhost:8250/oidc/callback\",\"http://${PUBLIC_DNS}:8200/ui/vault/auth/oidc/oidc/callback\",\"http://${PUBLIC_DNS}:8200/oidc/callback\"]" \
-s "webOrigins=[\"*\"]"
Created new client with id 'f431fad9-f7d2-4d72-bfcc-9f6fee817f59'
Retrieve the client secret (this is the value you will use in the next step):
CLIENT_ID=$(docker exec keycloak /opt/keycloak/bin/kcadm.sh get clients -r sre -q "clientId=vault" --fields id --format csv | tail -1 | tr -d \")
docker exec keycloak /opt/keycloak/bin/kcadm.sh get clients/$CLIENT_ID/client-secret -r sre
{
"type" : "secret",
"value" : "TzmZBdO2yryQAqc1jIYtJMXBmzB8LQKk"
}
Create (or ensure) a non-admin demo user:
docker exec keycloak /opt/keycloak/bin/kcadm.sh create users -r sre -s username=demo -s enabled=true 2>/dev/null || true
docker exec keycloak /opt/keycloak/bin/kcadm.sh set-password -r sre --username demo --new-password demo123 --temporary=false
docker exec keycloak /opt/keycloak/bin/kcadm.sh get users -r sre --fields username,enabled
[
{
"username" : "demo",
"enabled" : true
}
]
Configure Vault OIDC Auth Method (Public DNS Only)
Start a fresh Vault dev server bound to all interfaces (if it is not already running):
pkill vault 2>/dev/null || true
nohup vault server -dev -dev-listen-address="0.0.0.0:8200" > /tmp/vault.log 2>&1 &
sleep 5
vault status
Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 2.0.3 Build Date 2026-06-17T12:39:45Z Storage Type inmem Cluster Name vault-cluster-e525ecfb Cluster ID c0b25d9a-8ec9-b535-dfb8-81c098c4e4c0 HA Enabled false
Enable the OIDC auth method:
vault auth enable oidc
vault auth list
Success! Enabled oidc auth method at: oidc/ Path Type Accessor Description Version ---- ---- -------- ----------- ------- oidc/ oidc auth_oidc_598738b5 n/a n/a token/ token auth_token_1fda26c6 token based credentials n/a
Configure the OIDC method using the public DNS discovery URL (no localhost here):
vault write auth/oidc/config \
oidc_discovery_url="http://${PUBLIC_DNS}:8080/realms/sre" \
oidc_client_id="vault" \
oidc_client_secret="TzmZBdO2yryQAqc1jIYtJMXBmzB8LQKk" \
default_role="human"
vault read auth/oidc/config
Key Value --- ----- bound_issuer n/a default_role human jwks_ca_pem n/a jwks_pairs [] jwks_url n/a jwt_supported_algs [] jwt_validation_pubkeys [] namespace_in_state true oidc_client_id vault oidc_discovery_ca_pem n/a oidc_discovery_url http://lab.onlysre.dev:8080/realms/sre oidc_response_mode n/a oidc_response_types [] provider_config map[] unsupported_critical_cert_extensions []
Create the human role. It allows the three redirect URIs we registered in Keycloak and will later carry the k8s-creds policy:
vault write auth/oidc/role/human \
user_claim="sub" \
allowed_redirect_uris="http://localhost:8250/oidc/callback,http://${PUBLIC_DNS}:8200/ui/vault/auth/oidc/oidc/callback,http://${PUBLIC_DNS}:8200/oidc/callback" \
policies="default"
Success! Data written to: auth/oidc/role/human
Dedicated Policy for Kubernetes Token Generation
Following the same pattern we used for KV in Part 1, we create a small policy that only allows update on the creds paths:
vault policy write k8s-creds - <<'EOF'
path "k8s-a/creds/*" {
capabilities = ["update"]
}
path "k8s-b/creds/*" {
capabilities = ["update"]
}
EOF
vault policy read k8s-creds
path "k8s-a/creds/*" {
capabilities = ["update"]
}
path "k8s-b/creds/*" {
capabilities = ["update"]
}
Attach both default and k8s-creds to the OIDC role:
vault write auth/oidc/role/human \
user_claim="sub" \
allowed_redirect_uris="http://localhost:8250/oidc/callback,http://${PUBLIC_DNS}:8200/ui/vault/auth/oidc/oidc/callback,http://${PUBLIC_DNS}:8200/oidc/callback" \
policies="default,k8s-creds"
vault read auth/oidc/role/human
Key Value --- ----- ... policies [default k8s-creds] ... user_claim sub
Configure the Kubernetes Secrets Engines
We use 127.0.0.1 (plus the mapped host ports) for kubernetes_host because that is what the kind CA certificates are valid for. The Vault process runs on the EC2 host and can reach the kind API servers on those addresses.
First extract the CA certificates from the kind kubeconfigs:
kubectl config view --raw --minify --context kind-cluster-a \
-o jsonpath="{.clusters[0].cluster.certificate-authority-data}" | base64 -d \
> ~/certs/cluster-a-ca.crt
kubectl config view --raw --minify --context kind-cluster-b \
-o jsonpath="{.clusters[0].cluster.certificate-authority-data}" | base64 -d \
> ~/certs/cluster-b-ca.crt
ls -l ~/certs/
-rw-rw-r-- 1 ubuntu ubuntu 1107 ... cluster-a-ca.crt -rw-rw-r-- 1 ubuntu ubuntu 1107 ... cluster-b-ca.crt
Configure k8s-a:
vault write k8s-a/config \
kubernetes_host="https://127.0.0.1:6443" \
kubernetes_ca_cert=@/home/ubuntu/certs/cluster-a-ca.crt
vault read k8s-a/config
Key Value --- ----- disable_local_ca_jwt false kubernetes_ca_cert -----BEGIN CERTIFICATE----- ... kubernetes_host https://127.0.0.1:6443
Create the role that maps to the existing ServiceAccount:
vault write k8s-a/roles/cluster-a-admin \
allowed_kubernetes_namespaces="*" \
service_account_name="vault-demo-sa" \
token_default_ttl="15m"
vault read k8s-a/roles/cluster-a-admin
Key Value --- ----- allowed_kubernetes_namespaces [*] name cluster-a-admin service_account_name vault-demo-sa token_default_ttl 15m
Repeat for k8s-b:
vault write k8s-b/config \
kubernetes_host="https://127.0.0.1:6444" \
kubernetes_ca_cert=@/home/ubuntu/certs/cluster-b-ca.crt
vault write k8s-b/roles/cluster-b-admin \
allowed_kubernetes_namespaces="*" \
service_account_name="vault-demo-sa" \
token_default_ttl="15m"
Test Token Generation From the CLI (on EC2)
Before moving to your laptop, prove that token generation works on the EC2 host itself:
vault write k8s-a/creds/cluster-a-admin kubernetes_namespace=default
Key Value --- ----- lease_id k8s-a/creds/cluster-a-admin/... lease_duration 15m lease_renewable false service_account_name vault-demo-sa service_account_namespace default service_account_token eyJhbGciOiJSUzI1NiIs...
vault write k8s-b/creds/cluster-b-admin kubernetes_namespace=default
Key Value --- ----- lease_id k8s-b/creds/cluster-b-admin/... lease_duration 15m lease_renewable false service_account_name vault-demo-sa service_account_namespace default service_account_token eyJhbGciOiJSUzI1NiIs...
At this point the EC2 side is fully wired.
End-to-End Flow From Your Laptop (Pure CLI)
From this point on, you do not need the Vault UI. Everything is done with the vault CLI on your laptop.
All steps below run on your laptop. The CLI flow is the recommended way once you have vault installed locally.
Set the address (use the public DNS):
export VAULT_ADDR="http://lab.onlysre.dev:8200"Log in with OIDC (this will open your browser for the Keycloak login, then return a Vault token):
vault login -method=oidc role=humanYou will be redirected to the Keycloak login page for the sre realm. Log in as demo / demo123.
After a successful login, your terminal will show a Vault token. This token carries the policies default and k8s-creds.
Generate a short-lived Kubernetes token for cluster-a:
vault write k8s-a/creds/cluster-a-admin kubernetes_namespace=defaultKey Value --- ----- lease_id k8s-a/creds/cluster-a-admin/... lease_duration 15m lease_renewable false service_account_name vault-demo-sa service_account_namespace default service_account_token eyJhbGciOiJSUzI1NiIs...
Copy the service_account_token value.
Test it against cluster-a (public DNS, skip TLS verify because kind uses self-signed certs that do not contain the public name):
kubectl --server https://lab.onlysre.dev:6443 \
--token "<paste-the-service_account_token>" \
--insecure-skip-tls-verify \
get nodesNAME STATUS ROLES AGE VERSION cluster-a-control-plane Ready control-plane ... v1.31.0
Also test namespace-scoped access:
kubectl --server https://lab.onlysre.dev:6443 \
--token "<paste-the-service_account_token>" \
--insecure-skip-tls-verify \
get podsRepeat for cluster-b:
vault write k8s-b/creds/cluster-b-admin kubernetes_namespace=defaultkubectl --server https://lab.onlysre.dev:6444 \
--token "<paste-the-service_account_token>" \
--insecure-skip-tls-verify \
get nodesYou can run the two vault write commands one after another to have fresh tokens for both clusters. The tokens are real Kubernetes JWTs bound to vault-demo-sa.
What Just Happened (End-to-End CLI Flow)
- You ran
vault login -method=oidc role=humanon your laptop (withVAULT_ADDRpointing at the public EC2 DNS). - Vault redirected your browser to Keycloak (
srerealm) using one of the allowed redirect URIs registered on thehumanrole. - You authenticated as the
demouser. Keycloak issued a signed ID token. - Vault validated the ID token against the discovery document, looked up the
humanrole, and issued a Vault token carrying the policiesdefaultandk8s-creds. - When you ran
vault write k8s-a/creds/cluster-a-admin ..., Vault used the configuration atk8s-a/config(CA +kubernetes_host=127.0.0.1:6443+ its service account JWT) to call the Kubernetes TokenRequest API for the ServiceAccountvault-demo-sa. - Kubernetes returned a short-lived real ServiceAccount JWT (15 min TTL).
- You pasted that token into
kubectl --server https://...6443 --token ... --insecure-skip-tls-verify.
The exact same sequence works for cluster-b using the k8s-b mount and port 6444.
Summary of the Important Values (for this run)
- Public DNS:
lab.onlysre.dev - Vault (laptop):
export VAULT_ADDR=http://lab.onlysre.dev:8200 - Keycloak:
http://lab.onlysre.dev:8080 - OIDC login (CLI):
vault login -method=oidc role=human - OIDC discovery (used by Vault):
http://lab.onlysre.dev:8080/realms/sre - OIDC client secret:
TzmZBdO2yryQAqc1jIYtJMXBmzB8LQKk - Generate k8s token (cluster-a):
vault write k8s-a/creds/cluster-a-admin kubernetes_namespace=default - Generate k8s token (cluster-b):
vault write k8s-b/creds/cluster-b-admin kubernetes_namespace=default - Kubernetes secrets engine hosts (internal):
https://127.0.0.1:6443and:6444 - Demo user (Keycloak):
demo/demo123
What's Next
In Part 5 we eliminate the manual token copy step completely. We will build a small kubectl credential plugin (bash) that:
- Checks whether you already have a valid Vault token (
vault token lookup) - If missing or expired, runs
vault login -method=oidc role=human(opens browser automatically) - Calls
vault write -format=json k8s-a/creds/cluster-a-admin kubernetes_namespace=default - Outputs a proper
ExecCredentialobject that kubectl understands - Caches the result so the next
kubectlcommand is instant (no browser)
After that you will be able to do:
kubectl get nodes --context cluster-a
# (browser opens on first use, then cached)
kubectl get pods --context cluster-b
Next you will have true browser-based SSO with zero manual token handling.