Part 3: Keycloak Realm + OIDC Auth + Kubernetes Secrets Engines

Series · Vault as Simple SSO for Multiple Kubernetes Clusters · Part 3 of 5

Series: Vault as Simple SSO for Multiple Kubernetes Clusters (Part 3 of 5)

In Part 2 you provisioned the lab on EC2: Keycloak on 8080, Vault listening on all interfaces at 8200, and two kind clusters. Now you will configure identity in Keycloak and enable the two building blocks in Vault (OIDC auth and the Kubernetes secrets engines) that will later let a human obtain short-lived tokens for either cluster.

Prerequisites

The lab from Part 2 must be running on the EC2 instance. Confirm the environment variables are set in your current shell:

source ./scripts/vault-k8s-lab/env.sh

PUBLIC_DNS=lab.onlysre.dev
VAULT_ADDR=http://lab.onlysre.dev:8200
KEYCLOAK_URL=http://lab.onlysre.dev:8080

Verify Keycloak and Vault are reachable on the EC2 host:

curl -sI ${KEYCLOAK_URL} | head -1
vault status

HTTP/1.1 302 Found
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         2.0.3
Build Date      2026-06-17T12:39:45Z
Storage Type    inmem
Cluster Name    vault-cluster-2cad9942
Cluster ID      5c80e347-367a-cb56-c9be-b2233af0f54f
HA Enabled      false

You will use the Keycloak admin CLI (kcadm.sh) by running it inside the running container. No separate install is required.

Logging Into Keycloak as Admin

Keycloak is running with the bootstrap admin user admin / admin. Use kcadm.sh inside the container to configure credentials against the master realm:

docker exec keycloak /opt/keycloak/bin/kcadm.sh config credentials \
  --server http://localhost:8080 \
  --realm master \
  --user admin \
  --password admin

Logging into http://localhost:8080 as user admin of realm master

This writes a temporary session file inside the container. All subsequent kcadm.sh commands in this post run through docker exec keycloak.

Briefly inspect the master realm (the default realm that ships with Keycloak):

docker exec keycloak /opt/keycloak/bin/kcadm.sh get realms/master --fields realm,enabled

{
  "realm" : "master",
  "enabled" : true
}

You will not use the master realm for the demo. You will create a dedicated realm called sre.

Creating the 'sre' Realm

Create a new realm named sre and enable it:

docker exec keycloak /opt/keycloak/bin/kcadm.sh create realms \
  -s realm=sre \
  -s enabled=true

Created new realm with id 'sre'

List realms to confirm:

docker exec keycloak /opt/keycloak/bin/kcadm.sh get realms --fields realm,enabled

[
  {
    "realm" : "master",
    "enabled" : true
  },
  {
    "realm" : "sre",
    "enabled" : true
  }
]

The sre realm is now ready to hold the OIDC client that Vault will use.

Creating the Confidential OIDC Client for Vault

Vault will act as an OIDC client. Create a confidential client named vault inside the sre realm.

The client must:

Be confidential (has a client secret)
Have Standard Flow enabled (for browser-based login)
Allow the redirect URIs that Vault and the future kubectl plugin will use

Set the redirect URIs to cover both laptop testing and the Vault UI on the EC2 public DNS:

docker exec keycloak /opt/keycloak/bin/kcadm.sh create clients -r sre \
  -s clientId=vault \
  -s enabled=true \
  -s clientAuthenticatorType=client-secret \
  -s standardFlowEnabled=true \
  -s 'redirectUris=["http://localhost:8250/oidc/callback","http://lab.onlysre.dev:8200/ui/vault/auth/oidc/oidc/callback","http://lab.onlysre.dev:8200/oidc/callback"]' \
  -s 'webOrigins=["*"]'

Created new client with id '4204d17f-851d-4bc9-96ee-2e94c8dd5633'

Retrieve the generated client secret. You will need this value when configuring Vault later:

CLIENT_ID=$(docker exec keycloak /opt/keycloak/bin/kcadm.sh get clients -r sre -q "clientId=vault" --fields id --format csv | tail -1 | tr -d '"')
docker exec keycloak /opt/keycloak/bin/kcadm.sh get clients/$CLIENT_ID/client-secret -r sre

{
  "type" : "secret",
  "value" : "FhT8zbYof6TIhXJK2cVXF8vXGCdVaGJS"
}

Save the secret (FhT8zbYof6TIhXJK2cVXF8vXGCdVaGJS in this run) for the next post. Treat it like any other credential.

Verify the client settings:

docker exec keycloak /opt/keycloak/bin/kcadm.sh get clients -r sre -q "clientId=vault" --fields clientId,enabled,standardFlowEnabled,clientAuthenticatorType

[
  {
    "clientId" : "vault",
    "enabled" : true,
    "clientAuthenticatorType" : "client-secret",
    "standardFlowEnabled" : true
  }
]

Keycloak is now ready to issue tokens for the vault client to users who log into the sre realm.

Enabling OIDC Authentication in Vault

With the realm and client in place, enable the OIDC auth method in Vault:

vault auth enable oidc

Success! Enabled oidc auth method at: oidc/

List auth methods to see it alongside the token method:

vault auth list

Path      Type     Accessor               Description                Version
----      ----     --------               -----------                -------
oidc/     oidc     auth_oidc_10b620fc     n/a                        n/a
token/    token    auth_token_9f8ea698    token based credentials    n/a

Understanding the OIDC Auth Method in Vault

OpenID Connect (OIDC) is an identity layer built on top of OAuth 2.0. It lets an application (called the Relying Party) verify a user's identity by delegating authentication to an external OpenID Provider (OP). The provider issues a signed ID token (a JWT) that the application can validate without ever seeing the user's password.

In this series:

The end user is you (a human).
The OpenID Provider is Keycloak running in the sre realm.
The Relying Party is Vault's OIDC auth method.

Vault never stores or sees your Keycloak password. It only receives and validates a signed assertion from Keycloak.

The Complete OIDC Login Flow with Vault

Here is exactly what happens when you authenticate:

Initiation
You open the Vault UI and select the OIDC method, or you run:
```
vault login -method=oidc role=human
```
Vault generates a cryptographically random state value and nonce, then constructs an authorization URL.
Browser Redirect to Keycloak
Your browser is redirected to Keycloak's authorization endpoint inside the sre realm. The URL contains:
- client_id=vault
- redirect_uri (one of the three you registered earlier, including localhost:8250/oidc/callback and the PUBLIC_DNS Vault UI callback)
- response_type=code
- scope=openid profile email
- state and nonce for CSRF and replay protection
Authentication at Keycloak
You see the Keycloak login page for the sre realm. You enter your credentials. Keycloak validates them against its own user database (or whatever identity store you connected).
Authorization Code Returned
On success, Keycloak redirects your browser back to the redirect_uri with an authorization code and the same state.
Back-Channel Code Exchange (Vault → Keycloak)
Vault receives the code at its callback endpoint. Vault then makes a direct HTTPS POST to Keycloak's token endpoint:
- It authenticates using the client_id + client_secret you will configure.
- It sends the authorization code, redirect_uri, and the original state.
- Keycloak returns an ID token (JWT) and optionally an access token.
ID Token Validation
Vault fetches Keycloak's public signing keys from the JWKS URI listed in the discovery document. It validates the ID token:
- Signature is valid and was signed by Keycloak
- iss (issuer) matches the expected issuer from discovery
- aud (audience) contains the client_id
- exp has not passed
- nonce matches the value Vault generated in step 1
- sub claim is present
Role Mapping and Vault Token Issuance
Vault looks up the role you specified (human). Using the user_claim (commonly sub), it identifies the authenticated identity. It then attaches the policies listed on that role and issues a Vault token. That Vault token is what you use for all further Vault operations.

This entire dance happens in a few seconds. The only secret that ever leaves Keycloak is the short-lived authorization code and the resulting ID token — never your password.

Why Redirect URIs Must Be Pre-Registered

Keycloak only allows redirects to URIs you explicitly listed when creating the client. This prevents an attacker from tricking Vault (or any OIDC client) into sending the authorization code to a site the attacker controls.

That is why you registered three URIs for the vault client:

http://localhost:8250/oidc/callback (used by the kubectl plugin and manual laptop testing)
http://${PUBLIC_DNS}:8200/ui/vault/auth/oidc/oidc/callback (Vault UI running on EC2)
http://${PUBLIC_DNS}:8200/oidc/callback (direct callback path)

The Two Configuration Objects You Will Create

After enabling the method, two pieces of configuration control behavior:

auth/oidc/config
Contains the connection to Keycloak:
- oidc_discovery_url → http://${PUBLIC_DNS}:8080/realms/sre/.well-known/openid-configuration
- oidc_client_id
- oidc_client_secret
- default_role
auth/oidc/role/human
Controls what happens after successful login:
- user_claim="sub"
- allowed_redirect_uris (must match the ones registered in Keycloak)
- policies (the Vault policies this role receives)
- ttl and max_ttl

You will write these exact objects in Part 4.

OIDC vs userpass — Deeper Comparison

Aspect	userpass (Part 1)	OIDC (Part 3+)
Where identity lives	Inside Vault (local username + password hash)	External IdP (Keycloak `sre` realm)
Credential transmission	Password sent directly to Vault	Never sent to Vault; only signed ID token
Login experience	CLI prompt or API call	Browser redirect to Keycloak
Password storage	Vault must store hashes	Keycloak stores passwords; Vault stores nothing
SSO across systems	Not possible	Natural — same Keycloak login works for Vault + other apps
Revocation	Delete user in Vault	Disable user or client in Keycloak
Multi-cluster story	Each Vault needs its own users	One Keycloak realm can feed many Vaults

OIDC turns Vault into a consumer of identity rather than the source of truth. This is the foundation for the "single sign-on" experience the rest of the series demonstrates.

Enabling the Kubernetes Secrets Engines

Vault can generate short-lived Kubernetes ServiceAccount tokens on demand using the Kubernetes secrets engine. You will mount two separate engines so each cluster has its own path:

vault secrets enable -path=k8s-a kubernetes
vault secrets enable -path=k8s-b kubernetes

Success! Enabled the kubernetes secrets engine at: k8s-a/
Success! Enabled the kubernetes secrets engine at: k8s-b/

Confirm:

vault secrets list

Path               Type              Accessor                   Description
----               ----              --------                   -----------
agent-registry/    agent_registry    agent-registry_df7537c4    agent registry
cubbyhole/         cubbyhole         cubbyhole_4e3ffed5         per-token private secret storage
identity/          identity          identity_b02496e4          identity store
k8s-a/             kubernetes        kubernetes_5db4f109        n/a
k8s-b/             kubernetes        kubernetes_17c3096b        n/a
secret/            kv                kv_0b3756a9                key/value secret storage
sys/               system            system_ab89613e            system endpoints used for control, policy and debugging

(The kv/ engine may be present from earlier experiments; it is not used in this series.)

Understanding the Kubernetes Secrets Engine in Depth

The Kubernetes secrets engine is fundamentally different from every other secrets engine you have seen so far. It does not store data. It does not generate passwords. It acts as a trusted broker that asks a real Kubernetes cluster to mint a fresh ServiceAccount token on demand.

The Problem It Solves

Traditional Kubernetes access for humans usually means one of these patterns:

Long-lived static tokens embedded in kubeconfig files.
Certificates that are hard to rotate and revoke.
Shared service accounts whose tokens never expire.

These approaches violate least privilege. A human who only needs to look at pods in one namespace ends up holding cluster-admin forever.

The Kubernetes secrets engine replaces that with:

On-demand token generation.
Short TTLs (you will use 15 minutes).
Explicit mapping from a Vault identity to a specific Kubernetes ServiceAccount.

How Kubernetes ServiceAccount Tokens Actually Work

Since Kubernetes 1.21, the recommended way to obtain a token is the TokenRequest API (not the legacy secrets of type kubernetes.io/service-account-token).

When you call:

kubectl create token vault-demo-sa --namespace default

the kube-apiserver performs these steps:

Validates that the calling identity may create tokens for that ServiceAccount.
Creates a JWT signed by the cluster's private key.
Embeds claims: sub (the ServiceAccount), aud, exp, iss, and optionally bound pod metadata.
Returns the token. It is valid for the requested TTL (default 1 hour, max is controlled by the cluster).

The token is a real, first-class JWT. Any kubectl or client that presents it is authenticated exactly as if it were the ServiceAccount.

Vault's Kubernetes secrets engine simply automates this call on your behalf.

Internal Architecture of the Vault Engine

When you enable vault secrets enable -path=k8s-a kubernetes, Vault creates an isolated mount. Inside that mount live two important concepts:

Config (cluster connection)
Stored at k8s-a/config. Contains:
- kubernetes_host — the URL Vault will use to reach the Kubernetes API.
- kubernetes_ca_cert — the CA bundle used to verify the API server certificate.
- Optional service_account_jwt or client certificate if Vault itself authenticates to Kubernetes using a dedicated ServiceAccount.
Roles
Stored at k8s-a/roles/<name>. Each role is a policy document that says:
- Which ServiceAccount(s) this role may request tokens for (service_account_name or service_account_names).
- Which namespaces are allowed (allowed_kubernetes_namespaces).
- Token TTLs (token_default_ttl, token_max_ttl).
- Optional audience, extra audiences, and token type.

When a caller with a valid Vault token writes to k8s-a/creds/cluster-a-admin, Vault:

Checks the caller's policies permit the path.
Reads the role definition.
Builds a TokenRequest object using the ServiceAccount and namespace from the role.
Posts it to the Kubernetes API using the configured kubernetes_host and CA.
Receives the signed token.
Returns it (plus metadata) to the caller, wrapped in a Vault lease.

The lease lets Vault track and revoke the credential if needed, even though Kubernetes tokens themselves are not stored anywhere inside Vault.

The Two-Cluster Pattern You Are Building

You deliberately created two separate mounts:

k8s-a/  → cluster-a (port 6443)
k8s-b/  → cluster-b (port 6444)

This gives you:

Separate configuration (different CAs, different hosts).
Separate roles and policies.
Clear audit trails ("who asked for a token for cluster-b at 14:32").
The ability to grant different humans different levels of access per cluster without cross-contamination.

Comparison to the KV v2 Engine from Part 1

Dimension	KV v2 (Part 1)	Kubernetes (Part 3+)
What it stores	Static key/value pairs you write ahead of time	Nothing. It generates tokens on demand.
Source of truth	Vault's storage	The real Kubernetes cluster's TokenRequest API
Data returned	Whatever you previously wrote	A fresh, short-lived JWT signed by the cluster
TTL behavior	Controlled by KV engine or lease	Controlled by the role's `token_default_ttl` and cluster
Revocation	Delete the KV version or revoke lease	Token naturally expires; Vault can also revoke the lease
Typical use	Application config, feature flags	Human access to kubectl, CI jobs, temporary debugging

The policy model is identical. A policy that allows read on demo-kv/data/demo-secret is conceptually the same as a policy that allows update on k8s-a/creds/cluster-a-admin. The path and capability determine access.

The Role + Creds Lifecycle (Step by Step)

Configure the cluster connection (Part 4):

vault write k8s-a/config \
  kubernetes_host="https://10.0.1.23:6443" \
  [email protected]

Create a role (Part 4):

vault write k8s-a/roles/cluster-a-admin \
  allowed_kubernetes_namespaces="*" \
  service_account_name="vault-demo-sa" \
  token_default_ttl="15m"

Human authenticates via OIDC (Part 4 UI flow):
Vault issues a token with policy k8s-a-creds.

Human requests a token:

vault write k8s-a/creds/cluster-a-admin kubernetes_namespace=default

Vault calls the Kubernetes TokenRequest API using the CA and host from step 1.
Kubernetes returns a signed token valid for 15 minutes.
The human pastes that token into a kubectl command or the plugin stores it temporarily.
After 15 minutes (or when the lease is revoked), the token stops working. No cleanup required on the cluster.

Security Properties You Get for Free

The generated token is bound to a specific ServiceAccount. It cannot be used to impersonate another account.
The TTL is short by design. Even if the token is exfiltrated, the blast radius is limited.
Vault's audit log records exactly which Vault identity requested the token and when.
You can rotate the Kubernetes CA or change the ServiceAccount without touching any human credentials.
You can disable the entire k8s-a mount and instantly cut off all access to that cluster from Vault.

What You Will Still Do Manually in Part 4

At the end of this post the engines are enabled but not configured. In Part 4 you will:

Extract the CA certificate from each kind cluster's kubeconfig.
Discover the reachable kubernetes_host from Vault's perspective (the EC2 private IP plus the mapped host port).
Write the real k8s-a/config and k8s-b/config objects.
Create the two roles with the exact ServiceAccount name that already exists in each cluster (vault-demo-sa).
Create Vault policies that allow the OIDC role to call the creds endpoints.
Test generation from the CLI before moving to the browser UI flow.

Once those pieces are in place, the "generate credentials" button in the Vault UI will return a working token you can feed directly to kubectl.

Skeleton Role and Credential Commands

The actual wiring (CA certificates, kubernetes_host, attaching policies) happens in Part 4. For now, look at the shape of the commands you will run.

Create a role for cluster-a (the command below is illustrative; it will be completed with real values later):

vault write k8s-a/roles/cluster-a-admin \
  allowed_kubernetes_namespaces="*" \
  service_account_name="vault-demo-sa" \
  token_default_ttl="15m"

Create the matching role for cluster-b:

vault write k8s-b/roles/cluster-b-admin \
  allowed_kubernetes_namespaces="*" \
  service_account_name="vault-demo-sa" \
  token_default_ttl="15m"

Once configured, a logged-in user can request a token:

vault write k8s-a/creds/cluster-a-admin kubernetes_namespace=default

A successful response (shown here for illustration) looks like:

Key                     Value
---                     -----
lease_id                k8s-a/creds/cluster-a-admin/...
lease_duration          15m
lease_renewable         true
service_account_token   eyJhbGciOiJSUzI1NiIs...
service_account_name    vault-demo-sa
service_account_namespace default

At this stage the commands are not yet functional because the engines have no cluster configuration. You will finish the configuration in Part 4.

How the Pieces Fit Together

You now have three configured components:

Keycloak realm sre + client vault — the source of human identity.
OIDC auth method in Vault — the mechanism that turns a Keycloak login into a Vault token.
Kubernetes secrets engines at k8s-a and k8s-b — the mechanism that turns a Vault token (with the right policy) into a short-lived token for a specific cluster.

In Part 1 the mapping was:

Auth method (userpass) → policy → secrets engine (kv)

Here the mapping becomes:

Auth method (oidc) → policy → secrets engine (kubernetes at k8s-a or k8s-b)

The next post will connect the OIDC client secret and discovery URL, configure the engines with real cluster details, create the policies, and walk through the end-to-end flow from the Vault UI.

What's Next?

In Part 4 you will:

Configure the OIDC auth method with the real discovery URL and redirect URIs
Configure each Kubernetes secrets engine with its cluster's CA and reachable API address
Create Vault policies that allow the OIDC role to generate credentials
Perform the manual browser flow from your laptop: log into Vault via Keycloak, navigate to a secrets engine, generate a token, and use it with kubectl

Next you will wire the pieces and obtain a real short-lived Kubernetes token through the browser.