Series · Vault as Simple SSO for Multiple Kubernetes Clusters · Part 2 of 5
Series: Vault as Simple SSO for Multiple Kubernetes Clusters (Part 2 of 5)

In Part 1 you explored auth methods, secrets engines, and policies with a local dev server. Now you will build the actual lab environment on an AWS EC2 instance: Keycloak for identity, a Vault server reachable from your laptop, and two separate kind clusters. Everything you do here sets the stage for replacing the demo auth and KV engine with real OIDC and the Kubernetes secrets engine.

Install the CLI Tools

All command execution in this series happens on an Ubuntu 26.04 EC2 instance. Before you bring up Keycloak, Vault, or the clusters, install the required CLI tools.

The setup-ec2.sh script performs these steps automatically. Here are the commands it runs so you can see exactly what gets installed:

sudo apt-get update -y
sudo apt-get install -y curl wget jq unzip apt-transport-https ca-certificates gnupg lsb-release
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker "$USER" || true
sudo systemctl enable --now docker
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.24.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
curl -Lo kubectl https://dl.k8s.io/release/v1.31.1/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/kubectl
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update -y
sudo apt-get install -y vault

Once the tools are installed, make the helper scripts executable and source the environment file:

chmod +x scripts/vault-k8s-lab/*.sh
source ./scripts/vault-k8s-lab/env.sh

You can now run the convenience script that starts the entire lab in one go:

./scripts/vault-k8s-lab/start-lab.sh

Or follow the manual steps in the rest of this post (recommended for learning).

Versions

Before you begin, verify the tools on the EC2 host. The commands below show the versions used when this lab was built.

docker --version
kind version
kubectl version --client
vault version
Docker version 29.5.3, build d1c06ef
kind v0.24.0 go1.22.6 linux/amd64
Client Version: v1.31.1
Kustomize Version: v5.4.2
Vault v1.17.3 (c3c8e0e5c3c8e0e5c3c8e0e5c3c8e0e5c3c8e0e5), built 2025-04-02T15:43:36Z

Keycloak runs as a container. You will use quay.io/keycloak/keycloak:26.1.0.

What You Are Building

You will end up with this layout on a single EC2 instance:

  • Keycloak listening on port 8080 (exposed to the internet for OIDC)
  • Vault listening on port 8200 on all interfaces (so your laptop browser and CLI can reach it)
  • Two kind clusters:
    • cluster-a with its API reachable on host port 6443
    • cluster-b with its API reachable on host port 6444
  • A demo ServiceAccount in each cluster that Vault will later use to mint short-lived tokens
  • All external references will use the EC2 public DNS hostname (for example lab.onlysre.dev)

Once this is running, Part 3 will introduce OIDC and the Kubernetes secrets engines on top of this foundation.

One-Time EC2 Setup

Provision an Ubuntu 26.04 EC2 instance with a public IP. In the security group, temporarily allow inbound traffic from 0.0.0.0/0 on these ports:

  • 8200 (Vault)
  • 8080 (Keycloak)
  • 6443 (cluster-a)
  • 6444 (cluster-b)

SSH into the instance and run the setup script (this installs Docker, kind, kubectl, Vault, jq, and detects the public hostname):

# copy the script to the instance if needed, then:
chmod +x scripts/vault-k8s-lab/setup-ec2.sh
./scripts/vault-k8s-lab/setup-ec2.sh

The script prints something like:

PUBLIC_DNS=lab.onlysre.dev
VAULT_ADDR=http://lab.onlysre.dev:8200
KEYCLOAK_URL=http://lab.onlysre.dev:8080

Copy the PUBLIC_DNS value. You will use it for almost every URL and server address from now on.

Source the small helper that exports the variables for the rest of your session:

source ./scripts/vault-k8s-lab/env.sh
PUBLIC_DNS=lab.onlysre.dev
VAULT_ADDR=http://lab.onlysre.dev:8200
KEYCLOAK_URL=http://lab.onlysre.dev:8080

Starting Keycloak

Run Keycloak in development mode with proper host access so it can be reached from both the EC2 host and your laptop:

docker run -d --name keycloak \
  -p 8080:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e KC_BOOTSTRAP_ADMIN_USERNAME=admin \
  -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:26.1.0 start-dev

Check that the container is running:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | grep keycloak
keycloak   Up 42 seconds   0.0.0.0:8080->8080/tcp, 8443/tcp, 9000/tcp

Verify it responds on the public address:

curl -I http://${PUBLIC_DNS}:8080
HTTP/1.1 200 OK
...

You should now be able to open http://${PUBLIC_DNS}:8080 in your browser and see the Keycloak welcome page.

Starting Vault on All Interfaces

In Part 1 you ran vault server -dev which only listened on localhost. For the lab you need your laptop to reach the Vault UI and API, so you bind to all interfaces:

export VAULT_ADDR="http://${PUBLIC_DNS}:8200"
vault server -dev -dev-listen-address="0.0.0.0:8200"

You will see the familiar development banner, but the listener address will show 0.0.0.0:8200. Copy the Root Token value — you will use it for initial setup.

Example output (truncated):

==> Vault server configuration:

             Api Address: http://0.0.0.0:8200
                     Cgo: disabled
         Cluster Address: https://0.0.0.0:8201
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
            Recovery Mode: false
                  Storage: inmem
                  Version: Vault v1.17.3
              Version Sha: c3c8e0e5c3c8e0e5c3c8e0e5c3c8e0e5c3c8e0e5

==> Vault server started! Log data will stream in below:

WARNING! dev mode is enabled! ...

Root Token: hvs.CAESIJ1rootTokenForLabSetupOnlyChangeMe123456

Development mode should NOT be used in production installations!

...

In another terminal on the same EC2 host, export the address and verify:

export VAULT_ADDR="http://${PUBLIC_DNS}:8200"
vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             false
Total Shares       1
Threshold          1
Version            1.17.3
Build Date         2025-04-02T15:43:36Z
Storage Type       inmem
Cluster Name       vault-cluster-1a2b3c
Cluster ID         4d5e6f7a-8b9c-0d1e-2f3a-4b5c6d7e8f9a
HA Enabled         false

Open the Vault UI from your laptop at http://${PUBLIC_DNS}:8200 and log in with the root token. This confirms the network path works.

Creating the Two kind Clusters

You will create two separate clusters so you can later demonstrate multi-cluster access.

cluster-a (API on host port 6443)

First, create the kind configuration file (or copy it from scripts/vault-k8s-lab/kind-configs/cluster-a.yaml):

cat > cluster-a.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 6443
    hostPort: 6443
    listenAddress: "0.0.0.0"
    protocol: TCP
EOF

Create the cluster:

kind create cluster --name cluster-a --config cluster-a.yaml

You will see kind pull the node image (first time only) and bring up the control plane. When it finishes, the Kubernetes API for this cluster is available on the EC2 host at https://127.0.0.1:6443.

cluster-b (API on host port 6444)

Create the second config:

cat > cluster-b.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 6443
    hostPort: 6444
    listenAddress: "0.0.0.0"
    protocol: TCP
EOF
kind create cluster --name cluster-b --config cluster-b.yaml

Both clusters now run on the same EC2 host but listen on different host ports.

Setting Up kubectl Contexts

Export the kubeconfig for each cluster:

kind export kubeconfig --name cluster-a
kind export kubeconfig --name cluster-b --kubeconfig ~/.kube/config-b

Merge them so you can easily switch:

KUBECONFIG=~/.kube/config:~/.kube/config-b kubectl config view --flatten > ~/.kube/config
chmod 600 ~/.kube/config

List the contexts and clusters:

kind get clusters
kubectl config get-contexts
cluster-a
cluster-b
CURRENT   NAME              CLUSTER           AUTHINFO          NAMESPACE
*         kind-cluster-a    kind-cluster-a    kind-cluster-a
          kind-cluster-b    kind-cluster-b    kind-cluster-b

Verify each cluster is reachable from the EC2 host:

kubectl cluster-info --context kind-cluster-a
kubectl cluster-info --context kind-cluster-b

You should see the control plane addresses for each.

Creating the Demo ServiceAccount in Both Clusters

Vault will later generate tokens for a ServiceAccount that already exists in each cluster. Create a simple admin ServiceAccount (this is intentionally permissive for the demo):

cat > sa-and-binding.yaml <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-demo-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-demo-sa-admin
subjects:
- kind: ServiceAccount
  name: vault-demo-sa
  namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
EOF

Apply to both clusters:

kubectl apply -f sa-and-binding.yaml --context kind-cluster-a
kubectl apply -f sa-and-binding.yaml --context kind-cluster-b
serviceaccount/vault-demo-sa created
clusterrolebinding.rbac.authorization.k8s.io/vault-demo-sa-admin created

Confirm the account exists:

kubectl get sa vault-demo-sa --context kind-cluster-a
kubectl get sa vault-demo-sa --context kind-cluster-b

Verifying Reachability from Your Laptop

This is the critical step that proves the lab will be usable from outside the EC2 instance.

On your laptop (not the EC2), set the hostname:

export PUBLIC_DNS=lab.onlysre.dev

Test the Kubernetes APIs (the -k flag skips TLS verification because kind uses self-signed certs):

curl -k https://${PUBLIC_DNS}:6443/version
curl -k https://${PUBLIC_DNS}:6444/version

You should receive JSON containing the Kubernetes version for each cluster.

Test Vault:

curl http://${PUBLIC_DNS}:8200/v1/sys/health
{"initialized":true,"sealed":false,"standby":false,"performance_standby":false,"replication_performance_mode":"disabled","replication_dr_mode":"disabled","server_time_utc":1750000000,"version":"1.17.3","cluster_name":"vault-cluster-1a2b3c","cluster_id":"4d5e6f7a-8b9c-0d1e-2f3a-4b5c6d7e8f9a"}

Test Keycloak:

curl -I http://${PUBLIC_DNS}:8080

If all four commands succeed, your laptop can reach every component of the lab.

Record the Public DNS for the Rest of the Series

Set these once per terminal session (or put them in your shell profile):

export PUBLIC_DNS=lab.onlysre.dev
export VAULT_ADDR="http://${PUBLIC_DNS}:8200"

Use $PUBLIC_DNS (or the literal hostname) for:

  • Browser URLs (Vault UI, Keycloak)
  • OIDC discovery URLs and redirect URIs
  • kubectl --server flags
  • Every example in the remaining posts

Artifacts Created

The following files are now in your workspace (and committed in the repo under scripts/vault-k8s-lab/):

  • kind-configs/cluster-a.yaml
  • kind-configs/cluster-b.yaml
  • env.sh
  • setup-ec2.sh
  • start-lab.sh (optional convenience script that brings everything up)

You can re-run setup-ec2.sh on a fresh EC2 and then use start-lab.sh (or follow the manual steps above) to recreate the entire environment.

What's Next?

You now have a running lab with two Kubernetes clusters, Keycloak, and a Vault server that your laptop can reach.

In Part 3 you will:

  • Create a realm and OIDC client in Keycloak
  • Enable the OIDC auth method in Vault
  • Enable the Kubernetes secrets engine at k8s-a and k8s-b
  • Prepare the configuration pieces that will let a logged-in human obtain short-lived tokens for either cluster

Next you will introduce Keycloak for identity and the Kubernetes secrets engine for generating real service account tokens.