Part 5: Custom kubectl Credential Plugin – Full Browser SSO

Series: Vault as Simple SSO for Multiple Kubernetes Clusters (Part 5 of 5)

In Part 4 we used vault login -method=oidc role=human from the laptop, then manually copied the service_account_token from vault write k8s-*/creds/... into kubectl --token .... In this final post we remove the copy step entirely with a tiny kubectl credential plugin. The plugin derives the target cluster from your current context, checks for a valid Vault token, runs the OIDC login (opening your browser to Keycloak) only when needed, fetches a fresh short-lived token from the right k8s-* mount, and returns it to kubectl in the proper ExecCredential format.

---

Prerequisites

Everything from Part 4 must still be running on the EC2:

• Keycloak on port 8080
• Vault on port 8200
• Two kind clusters (cluster-a on 6443, cluster-b on 6444)
• vault-demo-sa ServiceAccount + cluster-admin binding in both clusters
• OIDC auth method + human role configured with the k8s-creds policy
• k8s-a and k8s-b Kubernetes secrets engines configured with the kind CAs

On your laptop you need:

• vault CLI
• kubectl
• jq
• The public DNS of the EC2 (used for everything):

export PUBLIC_DNS=lab.onlysre.dev
export VAULT_ADDR="http://${PUBLIC_DNS}:8200"

Quick reachability check from the laptop:

curl -k https://${PUBLIC_DNS}:6443/version
curl -k https://${PUBLIC_DNS}:6444/version
curl -I ${VAULT_ADDR}/v1/sys/health
vault status

---

The Credential Plugin

We will use a single small bash script called vault-k8s-creds.sh.

It does exactly four things:

1. Looks at the current kubectl context (or the server address passed by kubectl) to decide whether we are talking to cluster-a or cluster-b.
2. Runs vault token lookup. If there is no token or it is expired, it runs vault login -method=oidc role=human (this opens your browser to Keycloak).
3. Calls vault write -format=json k8s-*/creds/... for the correct mount/role and extracts the service_account_token.
4. Prints a client.authentication.k8s.io/v1 ExecCredential object containing the token and an expirationTimestamp (15 minutes minus a 2-minute safety buffer).

Place the script somewhere on your PATH and make it executable.

Example:

mkdir -p ~/.local/bin
cp scripts/vault-k8s-lab/vault-k8s-creds.sh ~/.local/bin/vault-k8s-creds.sh
chmod +x ~/.local/bin/vault-k8s-creds.sh

# Make sure it's on PATH
export PATH="$HOME/.local/bin:$PATH"
which vault-k8s-creds.sh

/home/you/.local/bin/vault-k8s-creds.sh

---

Kubeconfig with exec

We tell kubectl to use the plugin for our two contexts instead of embedding tokens.

Here is a minimal ~/.kube/config (or a separate file you can point at with KUBECONFIG):

apiVersion: v1
kind: Config
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: "https://lab.onlysre.dev:6443"
  name: cluster-a
- cluster:
    insecure-skip-tls-verify: true
    server: "https://lab.onlysre.dev:6444"
  name: cluster-b

users:
- name: vault-cluster-a
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: vault-k8s-creds.sh
      env:
      - name: VAULT_ADDR
        value: "http://lab.onlysre.dev:8200"
      provideClusterInfo: true
      interactiveMode: IfAvailable
- name: vault-cluster-b
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1
      command: vault-k8s-creds.sh
      env:
      - name: VAULT_ADDR
        value: "http://lab.onlysre.dev:8200"
      provideClusterInfo: true
      interactiveMode: IfAvailable

contexts:
- context:
    cluster: cluster-a
    user: vault-cluster-a
  name: cluster-a
- context:
    cluster: cluster-b
    user: vault-cluster-b
  name: cluster-b

current-context: cluster-a

Save this as ~/.kube/config (back up your existing one first) or use a dedicated file:

export KUBECONFIG=$HOME/.kube/vault-lab.yaml

Verify the contexts exist:

kubectl config get-contexts

CURRENT   NAME        CLUSTER     AUTHINFO          NAMESPACE
*         cluster-a   cluster-a   vault-cluster-a
          cluster-b   cluster-b   vault-cluster-b

---

Full End-to-End Flow (First Login)

Start with a completely clean state (no cached Vault token):

rm -f ~/.vault-token
vault token lookup 2>&1 || true

Error looking up token: Error making API request.

URL: GET http://lab.onlysre.dev:8200/v1/auth/token/lookup-self
Code: 403. Errors:

* permission denied

Now run a command against cluster-a for the first time. The plugin will notice there is no valid Vault token and start the OIDC login:

kubectl get nodes --context cluster-a

You will see output like this (the long URL is printed to stderr by the vault CLI):

vault-k8s-creds: no valid Vault token, starting OIDC login...
Complete the login via your OIDC provider. Launching browser to:

    http://lab.onlysre.dev:8080/realms/sre/protocol/openid-connect/auth?client_id=vault&code_challenge=9NvOPV9aeim4BuY10K9Qv4HTtZMFLmAXbMUtASM2Ds4&code_challenge_method=S256&nonce=n_9laxhxsv7aK1h00qsDdQ&redirect_uri=http%3A%2F%2Flocalhost%3A8250%2Foidc%2Fcallback&response_type=code&scope=openid&state=st_EYPt2dy4ogN6hchYcSxG


Waiting for OIDC authentication to complete...
Unable to connect to the server: getting credentials: decoding stdout: couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }

At this point the process is paused waiting for the browser redirect. Open the printed URL in your browser, log in as demo / demo123 in the sre realm, and complete the consent if prompted. Once the login finishes (the browser will try to redirect to http://localhost:8250/oidc/callback), the vault CLI receives the token, writes ~/.vault-token, and the plugin continues.

After a successful login, the same kubectl command (or a new one) succeeds:

NAME                      STATUS   ROLES           AGE   VERSION
cluster-a-control-plane   Ready    control-plane   66m   v1.31.0

Run the command again right away (cached path):

kubectl get nodes --context cluster-a

This time you get an immediate successful response with no browser, no "starting OIDC login" message, and no extra output from the plugin — the Vault token is still valid and kubectl uses the previously returned expirationTimestamp.

---

Switching Clusters

Switch to the other context:

kubectl config use-context cluster-b
kubectl get nodes --context cluster-b

NAME                      STATUS   ROLES           AGE   VERSION
cluster-b-control-plane   Ready    control-plane   66m   v1.31.0

The plugin detected "cluster-b" (from the context name or the server in KUBERNETES_EXEC_INFO), selected the k8s-b mount and cluster-b-admin role, obtained a fresh short-lived token from Vault, and returned it to kubectl.

You can now switch freely with no manual token handling:

kubectl get pods --context cluster-a
kubectl get pods --context cluster-b

No more copying tokens.

---

The Plugin Source

Here is the complete script we use:

#!/usr/bin/env bash
#
# vault-k8s-creds.sh
#
# kubectl exec credential plugin.
# Derives the target cluster (a or b) from the current kubectl context.
#
# Requirements:
#   - vault CLI configured for OIDC against the lab
#   - jq
#
# Behavior:
#   - Reuses existing Vault token (from \`vault token lookup\`)
#   - If no/expired Vault token → runs \`vault login -method=oidc role=human\`
#     (opens browser for Keycloak login)
#   - Fetches short-lived SA token from the correct k8s-* mount
#   - Returns ExecCredential v1 with token + expirationTimestamp (13 min)

set -euo pipefail

# --- Determine cluster from current context ---

get_context() {
  # Best source when provideClusterInfo: true
  if [[ -n "\${KUBERNETES_EXEC_INFO:-}" ]]; then
    # Try to extract server and guess
    local server
    server=$(echo "\$KUBERNETES_EXEC_INFO" | jq -r '.spec.cluster.server // ""' 2>/dev/null || true)
    if [[ "\$server" == *":6443"* ]]; then
      echo "cluster-a"
      return 0
    fi
    if [[ "\$server" == *":6444"* ]]; then
      echo "cluster-b"
      return 0
    fi
  fi

  # Fallback to current-context name
  if command -v kubectl >/dev/null 2>&1; then
    kubectl config current-context 2>/dev/null || true
  fi
}

CONTEXT_NAME=$(get_context)

case "\$CONTEXT_NAME" in
  *cluster-a*|cluster-a)
    MOUNT="k8s-a"
    ROLE="cluster-a-admin"
    ;;
  *cluster-b*|cluster-b)
    MOUNT="k8s-b"
    ROLE="cluster-b-admin"
    ;;
  *)
    echo "vault-k8s-creds: cannot determine cluster from context '\$CONTEXT_NAME'" >&2
    echo "Expected context names containing 'cluster-a' or 'cluster-b'." >&2
    exit 1
    ;;
esac

# --- Vault token handling (reuse or re-login) ---

if ! vault token lookup >/dev/null 2>&1; then
  echo "vault-k8s-creds: no valid Vault token, starting OIDC login..." >&2
  vault login -method=oidc role=human
fi

# --- Get short-lived Kubernetes token ---

OUT=$(vault write -format=json "\${MOUNT}/creds/\${ROLE}" kubernetes_namespace=default)

TOKEN=$(echo "\$OUT" | jq -r '.data.service_account_token')

if [[ -z "\$TOKEN" || "\$TOKEN" == "null" ]]; then
  echo "vault-k8s-creds: failed to get token from Vault" >&2
  exit 1
fi

# --- Expiration: 15m - 2min safety buffer ---

if EXP=$(date -u -d '+13 minutes' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null); then
  :
elif EXP=$(date -u -v+13M +%Y-%m-%dT%H:%M:%SZ 2>/dev/null); then
  :
else
  EXP=$(date -u +%Y-%m-%dT%H:%M:%SZ)
fi

# --- Emit ExecCredential ---

cat <<EOF
{
  "apiVersion": "client.authentication.k8s.io/v1",
  "kind": "ExecCredential",
  "status": {
    "token": "\${TOKEN}",
    "expirationTimestamp": "\${EXP}"
  }
}
EOF

Save it, make it executable, and put it on your PATH as shown earlier.

---

Why expirationTimestamp Matters

By returning expirationTimestamp, we tell kubectl: "this token is good until this time". kubectl will cache the credential and avoid calling the plugin again until the timestamp is close or passed. This gives us fast repeated commands without hammering Vault or re-triggering logins.

---

Reset / Logout

To force a fresh OIDC login on the next command:

rm -f ~/.vault-token
vault token revoke -self 2>/dev/null || true

To completely start over (including any cached Kubernetes credentials kubectl may hold):

rm -f ~/.vault-token
kubectl config delete-context cluster-a 2>/dev/null || true
kubectl config delete-context cluster-b 2>/dev/null || true
# re-apply the kubeconfig snippet or restart your shell

---

Troubleshooting

• "vault-k8s-creds: command not found" — the script is not on PATH when kubectl invokes it. Use an absolute path in the command: field of the exec stanza or ensure ~/.local/bin (or wherever you put it) is in PATH for non-interactive shells.
• "jq: command not found" — install jq on the laptop.
• Browser does not open — vault login -method=oidc should open it automatically. If it doesn't, check $BROWSER or run the login manually once.
• Wrong cluster — make sure your context name contains cluster-a or cluster-b, or that the server URL ends in :6443 / :6444.
• Token expired too soon — our roles use 15-minute TTL. We return it 2 minutes early as a safety buffer.

---

Summary of Values Used in This Post

• Public DNS: lab.onlysre.dev
• VAULT_ADDR: http://lab.onlysre.dev:8200
• OIDC login: vault login -method=oidc role=human
• Plugin script: vault-k8s-creds.sh (derives mount from context)
• Contexts: cluster-a, cluster-b
• Kubernetes tokens: 15 min TTL, generated on demand via the k8s-a / k8s-b secrets engines

---

What We Deliberately Simplified

• No TLS or production-grade certificate management (we use --insecure-skip-tls-verify and kind self-signed certs).
• Everything runs with wide-open security groups (0.0.0.0/0).
• A single demo user (demo) with cluster-admin in both clusters.
• In-memory Vault storage (data disappears on restart).
• No token caching beyond what the Vault CLI and kubectl's expirationTimestamp already provide.
• No namespace isolation or fine-grained RBAC beyond the demo.

---

Cleanup

When you are done with the entire lab:

# On the EC2
kind delete cluster --name cluster-a
kind delete cluster --name cluster-b
docker stop keycloak || true
# If you ran Vault in a container:
docker stop vault || true

# On your laptop
rm -f ~/.vault-token

---

That's the Series

You now have a complete, minimal, end-to-end story:

1. Two kind clusters.
2. Keycloak as the identity provider.
3. Vault OIDC + Kubernetes secrets engines as the glue.
4. A one-command browser login from the laptop that gives you short-lived tokens for any cluster you have access to.
5. A tiny custom kubectl plugin that makes the whole thing feel like magic.

All without ever manually pasting a token again.

Thanks for following along.