Series · Vault as Simple SSO for Multiple Kubernetes Clusters · Part 1 of 5
Series: Vault as Simple SSO for Multiple Kubernetes Clusters (Part 1 of 5)

Let's begin by launching a Vault development server and exploring the core building blocks you'll use throughout this series: authentication methods, secrets engines, paths, and policies. Everything is kept deliberately simple so you can clearly see how these pieces fit together — before you swap in OIDC from Keycloak and the Kubernetes secrets engine in later parts.

Versions

Let's start by checking the version of the Vault CLI on the lab machine:

vault version
Vault v1.17.3 (c3c8e0e5c3c8e0e5c3c8e0e5c3c8e0e5c3c8e0e5), built 2025-04-02T15:43:36Z

The Vault Mental Model (Everything Else Builds on This)

Vault deliberately separates three concerns:

  1. Identity — “Who or what are you?” (handled by auth methods)
  2. Authorization — “Given that identity, what are you allowed to do?” (handled by policies attached to tokens)
  3. Secret storage / generation — “Where do the actual secrets live or how are they created?” (handled by secrets engines mounted at paths)

These three are connected by one primitive: the token.

  • You authenticate once (userpass, OIDC, Kubernetes SA, etc.).
  • Vault verifies you and issues a token.
  • The token carries a list of policy names.
  • Every subsequent request presents only the token.
  • Vault builds an ACL from those policies and decides allow/deny for the requested path + capability.

This is why swapping the auth method from userpass (this post) to oidc (later posts) or swapping the secrets engine from kv to kubernetes changes almost nothing about the policy rules or the request flow.

Vault is default-deny. Nothing is allowed unless a policy explicitly grants it.

Starting Vault in Development Mode

Why development mode is perfect for learning

Running vault server -dev does many things automatically that a real Vault would require separate steps for:

  • Auto-initializes (creates the master key material).
  • Auto-unseals (no Shamir unseal or auto-unseal with KMS needed).
  • Uses in-memory storage only (inmem) — data vanishes on restart.
  • Prints a root token immediately.
  • Listens on localhost only by default.
  • Skips TLS, audit devices, and most production hardening.

This is terrible for production but perfect for learning the exact CLI commands, path model, and policy evaluation without any operational noise.

Later in this series you will start Vault listening on all network interfaces so your laptop can reach it, but the mental model stays exactly the same.

Let's start the server:

vault server -dev

You will see output similar to this:

==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
                 Storage: inmem
                 Version: Vault v1.17.3, built 2025-04-02T15:43:36Z

==> Vault server started! Log data will stream in below:

...

You may need to set the following environment variable:

    $ export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: 7v2vP8i9vN0kL3mX5qR7tY9uI2oP4aS6dF8gH0jK2lM4n
Root Token: hvs.CAESIJx8vK3pQ9rT2yU7iO0pL5mN8bV2xW4eR6tY8uI0oP

Development mode should NOT be used in production installations!

In a new terminal (or new SSH session), export the two values you need for the rest of this post:

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='hvs.CAESIJx8vK3pQ9rT2yU7iO0pL5mN8bV2xW4eR6tY8uI0oP'

Inspecting a Fresh Vault

Let's inspect what a brand new Vault looks like right after it starts. Run these three commands:

vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.17.3
Build Date      2025-04-02T15:43:36Z
Storage Type    inmem
Cluster Name    vault-cluster-1e76691a
Cluster ID      c4ca5956-383b-20cc-7a7d-494e0b01f045
HA Enabled      false

vault status answers: “Is it up? Is it sealed? What storage is it using?”

vault auth list
Path      Type     Accessor               Description                Version
----      ----     --------               -----------                -------
token/    token    auth_token_xxxx        token based credentials    n/a

Only the built-in token auth method exists. Every other auth method must be explicitly enabled.

vault secrets list
Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_xxxx        per-token private storage
identity/     identity     identity_xxxx         identity store
secret/       kv           kv_xxxx               key/value secret storage
sys/          system       system_xxxx           system endpoints

secret/ is the default KV mount (v1). sys/ is where mounts, policies, and configuration live. You'll ignore both and create your own mounts.

Understanding Auth Methods

An auth method is a plugin that turns external credentials into a Vault token plus a set of policy names.

You enable it at a path:

vault auth enable userpass

After this command, auth/userpass/ becomes a valid authentication endpoint.

Common auth methods and when they are used:

MethodTypical identity sourceUsed in this series?Notes
userpassusername + passwordOnly in this post (demo)Simple for learning
oidc / jwtKeycloak, Okta, GitHub, etc.Yes (Parts 3–5)The real SSO method
kubernetesPod ServiceAccount JWTIndirectly (via secrets engine config)How Vault talks to k8s clusters
approleRoleID + SecretIDYes (this post)Machine / service auth
aws / gcpCloud instance identityNot in this seriesCloud-native auth

The auth method only decides who you are and which policies get attached to your token. It never decides “can you read this secret?” — that is purely the job of policies.

To make this concrete, this post shows two different auth methods side by side:

  • userpass: you log in with a username and password (great for humans during demos).
  • approle: an application or script logs in with a RoleID + SecretID pair (the normal way for machines and CI).

Both will produce tokens that carry the exact same policy. This is the heart of how Vault works: the login method is just the front door. After that, only the token and its policies matter.

Enabling Auth Methods

Vault starts with almost nothing enabled. You explicitly turn on the methods you want.

Let's enable both userpass (for humans) and AppRole (for applications) right now:

vault auth enable userpass
vault auth enable approle
Success! Enabled userpass auth method at: userpass/
Success! Enabled approle auth method at: approle/

You now have two new authentication paths:

  • auth/userpass/ — for username + password logins
  • auth/approle/ — for machine / application logins using RoleID + SecretID

The rest of this post will use both so you can see how different auth methods produce tokens that carry the same policies.

How Login Produces a Token

When you authenticate, Vault:

  1. Calls the auth method to verify the provided credentials.
  2. Looks up the policies declared for that identity.
  3. Creates a new token record in its internal storage.
  4. Returns the token (and usually writes it to ~/.vault-token).

The token is now the only thing that matters for future requests. The original username or OIDC sub claim is usually kept only as metadata.

Important fields you will see in every login and token-create output:

  • token — the actual secret you send with every request (via X-Vault-Token header or VAULT_TOKEN env).
  • token_accessor — a non-sensitive identifier for the token. Useful for auditing and revocation without knowing the token value.
  • token_policies — policies that came from the auth method / identity.
  • identity_policies — policies that came from entity/group membership (more advanced identity features).
  • policies — the final merged list Vault will use for ACL evaluation.
  • token_duration / token_renewable — how long the token lives and whether it can be renewed.

Understanding Token Duration (TTL)

token_duration (also called TTL — Time To Live) tells you how long the token will be valid.

  • After this time passes, the token stops working and you must log in again.
  • token_renewable: true means Vault will let you (or your application) extend the token's life before it expires.
  • Most tokens also have a maximum TTL. Even if you keep renewing, you cannot go past the maximum. After that, you must re-authenticate.

Different auth methods and roles set different defaults:

  • In the userpass login above, you got 768h0m0s (32 days). This is the long default that userpass uses in dev mode.
  • In the AppRole example (coming up), you set token_ttl=1h on the role, so every token created from that role only lasts 1 hour.
  • Later in this series, when you use the Kubernetes secrets engine, the tokens it generates for Kubernetes will often be even shorter (15 minutes or less).

This is a deliberate security design. Short-lived tokens mean that even if a token leaks, it quickly becomes useless.

You can control duration in two main ways:

  1. When creating a token manually: 
    vault token create -ttl=30m -policy=demo-policy
  2. At the role / auth method level (recommended for apps):
    • userpass: set on the user
    • approle: set on the role (token_ttl, token_max_ttl)
    • oidc / kubernetes: set on the role

The shorter the duration, the smaller the window of risk. This is why you'll see very short TTLs when you start generating real Kubernetes service account tokens later in the series.

Creating a Demo User

Let's create a user and say: anyone who logs in successfully as this user must receive the policy named demo-policy.

vault write auth/userpass/users/demo \
  password="demo123" \
  policies="demo-policy"
Success! Data written to: auth/userpass/users/demo

Note that you are only attaching a policy name. The actual policy document does not need to exist yet.

Logging In as the Demo User

Run the login command:

vault login -method=userpass username=demo

(Enter demo123 when prompted.)

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.CAESIJ0demoTokenForUserpassDemo1234567890abc
token_accessor       8f2c3a1b4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a
token_duration       768h0m0s
token_renewable      true
token_policies       ["default" "demo-policy"]
identity_policies    []
policies             ["default" "demo-policy"]

Notice the very long token_duration of 768h0m0s (32 days). This is the default that the userpass auth method uses in development mode. In real usage you would usually configure much shorter lifetimes.

The token you received now carries demo-policy. Every request made with this token will be evaluated against the rules in that policy (once you create it).

Authenticating Applications with AppRole

Userpass is great for humans. For applications, CI/CD jobs, and automated services, Vault offers AppRole.

The idea is simple:

  • You create a role and attach policies to it.
  • The application gets two pieces:
    • role_id: a non-secret, stable identifier.
    • secret_id: a secret value (can be rotated, delivered securely, or generated per instance).
  • The app logs in with both and receives a token carrying the role's policies.

This keeps identity proof separate from what the token is allowed to do.

Let's create an AppRole and log in with it. (AppRole was already enabled earlier when you ran vault auth enable approle.)

Create the role and attach the same policy:

vault write auth/approle/role/demo-role \
  token_policies="demo-policy" \
  token_ttl=1h
Success! Data written to: auth/approle/role/demo-role

Fetch the role ID (this is not secret):

vault read auth/approle/role/demo-role/role-id
Key      Value
---      -----
role_id  01234567-89ab-cdef-0123-456789abcdef

Generate a one-time secret ID:

vault write -f auth/approle/role/demo-role/secret-id
Key                Value
---                -----
secret_id          89abcdef-0123-4567-89ab-cdef01234567
secret_id_accessor 1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d

Now log in using the role:

vault write auth/approle/login \
  role_id="01234567-89ab-cdef-0123-456789abcdef" \
  secret_id="89abcdef-0123-4567-89ab-cdef01234567"
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.CAESIJ2approleTokenForDemoRoleWithPolicy1234
token_accessor       9f8e7d6c5b4a39281706f5e4d3c2b1a0f9e8d7c6b5a4
token_duration       1h0m0s
token_renewable      true
token_policies       ["default" "demo-policy"]
policies             ["default" "demo-policy"]

Notice the much shorter token_duration of 1h0m0s. This came from the token_ttl=1h you set when creating the AppRole role. Every token issued by this role will only be valid for one hour.

You now have a valid token that carries exactly the same demo-policy — but it was obtained through AppRole instead of a username and password.

This is the core idea behind auth methods:

  • Different methods prove identity in different ways (password, OIDC, Kubernetes JWT, RoleID+SecretID, cloud instance identity, etc.).
  • They all result in the same thing: a token with policies attached.
  • Policies (not the login method) decide what the token can actually do.

Understanding Secrets Engines

A secrets engine is a mounted plugin responsible for storing or generating secrets at a path prefix.

  • kv (key/value) stores data you explicitly write.
  • database, aws, kubernetes, pki, etc. generate short-lived credentials on demand when you ask for them.

You choose both the engine type and the mount path. Using custom mount paths (demo-kv, k8s-a, k8s-b) instead of the default secret makes it obvious which cluster or purpose each engine serves.

KV Version 2 vs Version 1

KV v2 is versioned:

  • Every write creates a new version.
  • You can see history, roll back, and soft-delete specific versions.
  • Actual secret values live under /data/.
  • Metadata (versions, deletion times, custom metadata) lives under /metadata/.

This is why policies and CLI output always show paths like demo-kv/data/demo-secret.

You'll use a custom path called demo-kv so the path structure matches the pattern you'll use later for the Kubernetes engines.

Enabling a KV v2 Secrets Engine at a Custom Path

vault secrets enable -path=demo-kv kv-v2
Success! Enabled the kv-v2 secrets engine at: demo-kv/

Vault Paths — The Only Language Vault Speaks

Everything you do in Vault is a path. Examples you have already touched or will touch in this series:

  • auth/userpass/users/demo — configure a userpass identity
  • auth/oidc/config and auth/oidc/role/human — OIDC configuration (later)
  • demo-kv/data/demo-secret — the secret data itself
  • demo-kv/metadata/demo-secret — version history
  • sys/policy/demo-policy — the policy document
  • k8s-a/config — tell the Kubernetes engine how to reach a cluster
  • k8s-a/roles/cluster-a-admin — define a role that can generate tokens
  • k8s-a/creds/cluster-a-admin — the path that actually returns a short-lived service account token

Policies are written against these exact paths. The path you see in CLI output is the same string you put in a policy rule.

Writing and Reading a Secret

Let's store a secret and read it back. Notice the /data/ path that KV v2 always uses.

vault kv put -mount=demo-kv demo-secret foo=bar
== Secret Path ==
demo-kv/data/demo-secret

======= Metadata =======
Key                Value
---                -----
created_time       2026-06-21T19:12:34.123456789Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

vault kv get -mount=demo-kv demo-secret
== Secret Path ==
demo-kv/data/demo-secret

======= Metadata =======
Key                Value
---                -----
created_time       2026-06-21T19:12:34.123456789Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

====== Data ======
Key         Value
---         -----
foo         bar

The -mount flag is the clean modern way to separate the mount point from the secret name. You will see the full mount/data/name path in every policy you write.

Policies — Path + Capabilities + Default Deny

A policy is a small HCL document that says:

At this path, these actions (capabilities) are allowed.

Vault evaluates policies on every single request using a strict default-deny model:

  • No matching rule → denied (403).
  • Most specific rule wins (an exact path beats a /* glob).
  • When a token has multiple policies, Vault takes the union (most permissive wins).
  • An explicit deny always wins.

Capabilities

CapabilityWhat it permitsTypical CLI/API examples
readRead data that already existsvault kv get, reading a secret
listList keys or list mountsvault kv list, vault secrets list
createCreate new data at a path that has no data yetFirst write to a path
updateModify data or create new versionsvault kv put, vault write k8s-a/creds/...
deleteDelete data or specific versionsvault kv delete
sudoPerform privileged operations (rare in app policies)Certain sys/ endpoints
denyExplicitly forbid even if another policy allowsBreak-glass overrides

Creating a Policy That Only Allows Reading the Secret

Use a heredoc so the rule is easy to read and version-control:

vault policy write demo-policy - <<EOF
path "demo-kv/data/demo-secret" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: demo-policy

This policy grants exactly one capability on exactly one path. Nothing else is allowed.

How Policies Connect Identity to Secrets (The Model You'll Reuse)

This is the single most important diagram in the entire series.

Today (Part 1):

You can prove your identity with different auth methods. Both produce a token with the same policy attached.

Example with userpass:

userpass login (username "demo" + password)
    ↓
Vault issues token carrying "demo-policy"
    ↓
Request to demo-kv/data/demo-secret
    ↓
Policy allows read → success

Example with AppRole:

approle login (role_id + secret_id)
    ↓
Vault issues token carrying "demo-policy"
    ↓
Request to demo-kv/data/demo-secret
    ↓
Policy allows read → success

In both cases the auth method only proves identity. The policy decides what the token can do.

Later (Parts 3–5):

OIDC login via Keycloak (role "human")
    ↓
Vault issues token carrying policies you attach to the OIDC role
    ↓
Request to generate a Kubernetes token
    ↓
Vault builds ACL
    ↓
path "k8s-a/creds/cluster-a-admin" { capabilities = ["update"] }
    ↓
Vault (using its own cluster config) calls the real Kubernetes API
    ↓
Returns a short-lived service account token

The auth method changed. The secrets engine changed. The policy evaluation rules and path model stayed exactly the same.

This is why learning userpass + KV first is so valuable.

Creating a Limited Token and Proving the Policy Works

Create a fresh token that only carries demo-policy:

vault token create -policy=demo-policy -ttl=1h
Key                  Value
---                  -----
token                hvs.CAESIJ1limitedTokenWithOnlyDemoPolicy123456
token_accessor       1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c
token_duration       1h0m0s
token_renewable      true
token_policies       ["default" "demo-policy"]
identity_policies    []
policies             ["default" "demo-policy"]

This token was created with an explicit short lifetime (-ttl=1h). This is the pattern you'll use more and more as you move toward real workloads and Kubernetes tokens.

Use the limited token to successfully read the secret:

VAULT_TOKEN=hvs.CAESIJ1limitedTokenWithOnlyDemoPolicy123456 \
  vault kv get -mount=demo-kv demo-secret
== Secret Path ==
demo-kv/data/demo-secret

...
====== Data ======
Key         Value
---         -----
foo         bar

Now attempt to write a new version (the policy only granted read):

VAULT_TOKEN=hvs.CAESIJ1limitedTokenWithOnlyDemoPolicy123456 \
  vault kv put -mount=demo-kv demo-secret foo=newvalue
Error making API request.

URL: PUT http://127.0.0.1:8200/v1/demo-kv/data/demo-secret
Code: 403. Errors:

* permission denied

The permission denied error shows that the policy is being enforced correctly.

What's Next?

In Part 2 you will set up the full lab environment with Keycloak and two Kubernetes clusters. Then you will start connecting the concepts you learned here to real OIDC authentication and the Kubernetes secrets engine.

Next you will build the actual lab and replace userpass with OIDC while replacing KV with the Kubernetes secrets engine.