Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

OpenBao for Bouncer

OpenBao stores the secret values. Bouncer's database holds metadata, but the running server handles plaintext when a device views or saves a secret, and when someone creates a personal credential. Its role can read viewing paths and personal credential namespaces, write configured save paths and namespaces, and issue short-lived tokens for requesters. OpenBao enforces that scope; Bouncer enforces the approval policy (How Bouncer protects access).

Set OpenBao up before you install the server. You can use an OpenBao you already run, or set up a dedicated one with deploy/openbao as this page describes. Either way, Bouncer reaches it through the narrow role that the role kit (deploy/openbao-role-kit) creates in Install the server, step 2.

Terms used on this page

  • Sealed and unsealed. OpenBao encrypts everything it stores. When it starts, it is sealed: it can't read its own data until it is given the key to unseal it. While sealed, it answers nothing useful, so Bouncer can't issue any tokens.
  • Unseal shares (Shamir). By default, the unseal key is split into several shares when OpenBao is first set up, and a minimum number of them, the threshold, is needed to unseal. This page uses three shares with a threshold of two.
  • Root token. A token with every permission, which OpenBao prints once, when it is initialised. Bouncer never uses it.
  • Admin token. Any token allowed to manage OpenBao's policies, token roles and login methods. The root token is one. The role kit needs one every time it runs.
  • KV version 2. The OpenBao secrets engine Bouncer reads from. Each secret is a set of named fields at a path, for example the field password at apps/billing/db-password.
  • Raft snapshot. A backup of OpenBao's own storage.

A dedicated OpenBao with deploy/openbao

deploy/openbao runs OpenBao 2.7.2 as a single Raft node in Docker. It is not highly available: after a restart it stays sealed until someone unseals it. Run it on a dedicated Linux VM with rootful Docker. Keep swap off or encrypted.

You need two machines for the steps below:

  • the OpenBao VM, with a copy of the Bouncer kit at ~/bouncer: a release download's compose kit, unpacked (Get Bouncer), or a clone of the repository (ask whoever gave you access to Bouncer for the repository link);

  • your recovery workstation, a machine you trust. It needs:

    • ssh access to the VM, and a network route to OpenBao's address (for example on the same Tailscale network), because bao and curl talk to OpenBao directly;
    • its own copy of the Bouncer kit at ~/bouncer, the same way, for the role kit;
    • bash 4.4 or later (the role kit's scripts stop on anything older; macOS ships bash 3.2), the bao command, jq, curl, python3 and age;
    • an age key pair that you already use for recovery material.

    The unseal shares and the root token only ever exist on this machine, in memory or encrypted.

The commands use <identity> for your age private key file, <recipient> for its public key, and <vault folder> for a folder off the VM where the encrypted recovery file lives.

1. Start it

On the OpenBao VM:

sudo install -d -m 0755 /etc/bouncer/openbao
sudo install -m 0644 ~/bouncer/deploy/openbao/openbao.hcl /etc/bouncer/openbao/openbao.hcl
cd ~/bouncer/deploy/openbao
cp .env.example .env          # set OPENBAO_API_ADDR to the address clients will use
docker compose up -d
sudo tailscale serve --bg --https=8443 http://127.0.0.1:8200
docker compose exec -T openbao bao status

OpenBao listens on 127.0.0.1:8200 on the VM, and tailscale serve puts HTTPS on port 8443 in front of it (TLS and networking). Never bind the plain listener to your LAN, and never publish OpenBao to the internet.

Expected result: bao status exits 2 and shows a new, empty vault:

Seal Type          shamir
Initialized        false
Sealed             true
Storage Type       raft

2. Initialise it once

Initialising creates the unseal shares and the root token. OpenBao shows them only once, so send them straight into encryption, never to a terminal or a file on the VM.

First check that age works with your key, and that the destination is writable. A failure after initialising can lose the only copy of the shares:

echo dummy | age -r <recipient> | age -d -i <identity>       # prints "dummy"
touch <vault folder>/write-test && rm <vault folder>/write-test

Then, on the recovery workstation:

set -o pipefail
umask 077
ssh admin@openbao.example.com 'cd ~/bouncer/deploy/openbao && docker compose exec -T openbao bao operator init -format=json -key-shares=3 -key-threshold=2' |
  age -r <recipient> -o <vault folder>/openbao-init.json.age

Check that the file decrypts and holds three shares and a root token, without printing any of them:

age -d -i <identity> <vault folder>/openbao-init.json.age |
  jq -e '(.unseal_keys_b64 | length) == 3 and (.root_token | length) > 0' > /dev/null &&
  echo "The file decrypts and holds 3 shares and a root token."

Expected result: that sentence, and nothing else. Keep a second encrypted copy somewhere else, and test that you can reach it.

Three shares encrypted to one key give you a backup, not three separate keyholders. If you want different people to hold the shares, encrypt each one to its holder instead.

3. Unseal it

After every restart, give OpenBao two of its shares. Each share goes as a small JSON object ({"key":"<share>"}) on standard input, so it never appears on screen or as a command-line argument. From the recovery workstation:

for n in 0 1; do
  age -d -i <identity> <vault folder>/openbao-init.json.age |
    jq -c --argjson n "$n" '{key: .unseal_keys_b64[$n]}' |
    ssh admin@openbao.example.com 'cd ~/bouncer/deploy/openbao && docker compose exec -T openbao bao write sys/unseal -'
done

bao operator unseal doesn't work for this, because it reads the share from a terminal. Each bao write prints the progress; after the second share, sealed is false. Then check on the VM:

cd ~/bouncer/deploy/openbao
docker compose exec -T openbao bao status

Expected result: it exits 0, and shows:

Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            3
Threshold               2
Version                 2.7.2
Storage Type            raft
HA Enabled              true
HA Mode                 active

Restart the container once (docker compose restart openbao), check that bao status says Sealed true again, and unseal it again. Then you know the procedure works before you need it.

4. Make a KV mount and a test secret

Bouncer reads secrets from a KV version 2 mount. On a new vault there isn't one yet. Use the root token for this first setup, read straight from the encrypted file into your shell, never printed. On the recovery workstation:

export BAO_ADDR=https://openbao.example.com:8443
read -r BAO_TOKEN < <(age -d -i <identity> <vault folder>/openbao-init.json.age | jq -r .root_token)
export BAO_TOKEN
bao secrets enable -path=kv -version=2 kv
printf '%s' 'not-a-real-secret' | bao kv put -mount=kv bouncer-test/hello value=-
bao kv get -mount=kv -field=value bouncer-test/hello

Expected result: Success! Enabled the kv secrets engine at: kv/, then the new secret's version 1, then not-a-real-secret.

Use this test secret as your first resource while you check everything works. In the role kit's resources file it is the line:

bouncer-test   kv   bouncer-test/hello

and in Bouncer's config a resource bouncer-test with kv: {mount: kv, path: bouncer-test/hello}. Add your real secrets once the role kit's check and a first request work.

5. Run the role kit, then revoke the root token

Keep the same shell open, with the root token in BAO_TOKEN, and follow Install the server, step 2. If you'd rather run the kit with less than root, its README shows how to make a scoped bouncer-kit-admin token from the root token (./setup.sh --print-admin-policy). Such a token is a child of the root token, so it stops working when you revoke root below. It can't be your way back in afterwards.

Before you revoke the root token, decide how you will get an admin token again. The role kit needs one every time it runs, and you may need it in a hurry: the emergency stop runs on OpenBao with an admin token. There are two ways:

  • Your own admin login on OpenBao, such as a user and password or your organisation's single sign-on. Give it at least the kit's admin policy (./setup.sh --print-admin-policy), plus the SSH kit's (./ssh/setup-ssh.sh --print-admin-policy) if you use SSH hosts. Before revoking root, log in with it and run ./setup.sh --verify and ./emergency-stop.sh --dry-run to check that it can read everything the stop needs. The SSH kit's admin policy must match your kit.env and ssh/ssh-resources.
  • Make a new root token from the unseal shares when you need one. OpenBao 2.5.3 and later turn this off by default, and the kit's openbao.hcl doesn't turn it on, so it needs a config change and two restarts, each followed by an unseal. See Make a new root token. This takes several minutes, which matters in an emergency. The tokenless endpoints it uses (sys/generate-root) have also been deprecated since OpenBao 2.6.0, in favour of the authenticated sys/generate-root-token, so a later version may remove them. An admin login of your own is the approach that will last.

Then revoke the root token:

bao token revoke -self
unset BAO_TOKEN

A token that was created from the root token, such as a bouncer-kit-admin token, is revoked with it.

Make a new root token

This is tested only in our sandbox, against OpenBao 2.7.2. Use it when you have no other admin token.

  1. On the VM, allow root generation from the shares. Add this line inside the listener "tcp" block of /etc/bouncer/openbao/openbao.hcl:

      disable_unauthed_generate_root_endpoints = false
    

    Then docker compose restart openbao, and unseal it.

  2. On the recovery workstation, start the attempt with a one-time password. It is sent to OpenBao to start the attempt, but it stays off the screen and out of files:

    export BAO_ADDR=https://openbao.example.com:8443
    OTP_LENGTH=$(curl -s "$BAO_ADDR/v1/sys/generate-root/attempt" | jq .otp_length)
    OTP=$(head -c 64 /dev/urandom | base64 -w0 | tr -dc 'A-Za-z0-9' | head -c "$OTP_LENGTH")
    NONCE=$(jq -n --arg otp "$OTP" '{otp: $otp}' |
      curl -s -X PUT --data @- "$BAO_ADDR/v1/sys/generate-root/attempt" | jq -r .nonce)
    
  3. Give it two shares, the same way as for unsealing. The second answer holds the new token, encrypted with your one-time password:

    for n in 0 1; do
      age -d -i <identity> <vault folder>/openbao-init.json.age |
        jq -c --arg nonce "$NONCE" --argjson n "$n" '{key: .unseal_keys_b64[$n], nonce: $nonce}' |
        ssh admin@openbao.example.com 'cd ~/bouncer/deploy/openbao && docker compose exec -T openbao bao write -format=json sys/generate-root/update -'
    done | jq -r 'select(.data.complete) | .data.encoded_token' > encoded-token
    
  4. Decode it into your shell, without printing it:

    read -r BAO_TOKEN < <(OTP="$OTP" python3 -I -c '
    import base64, os, sys
    enc = base64.b64decode(sys.stdin.read().strip() + "==")
    print(bytes(a ^ b for a, b in zip(enc, os.environ["OTP"].encode())).decode())' < encoded-token)
    export BAO_TOKEN
    rm encoded-token; unset OTP NONCE OTP_LENGTH
    bao token lookup -format=json | jq -c '.data.policies'     # ["root"]
    

    (bao operator generate-root -decode doesn't work here: it needs a token before it decodes one.)

  5. Do what you needed the token for, then bao token revoke -self and unset BAO_TOKEN.

  6. Take the line out of openbao.hcl again, docker compose restart openbao, and unseal it.

Auto-unseal with the VM's TPM

The repository also has an alternative to typing in shares after every restart: deploy/openbao/auto-unseal/. It uses OpenBao's static seal, whose key is held in a credential that only the VM's TPM 2.0 (Trusted Platform Module, a security chip, here a virtual one) can decrypt at boot. OpenBao then unseals itself after a reboot or a container restart.

Know what it does and doesn't protect before you choose it:

  • It protects against someone who copies the VM's disk without its TPM state.
  • It doesn't protect against root on the running VM, or anyone who controls the hypervisor. A virtual TPM is software, and a VM backup that includes the TPM state is a copy of the vault that unseals itself.
  • If you lose both the TPM state and the escrow copy of the seal key, the vault can't be recovered, even from snapshots.

Not verified for this page. The project tested this only in a virtual machine with a software TPM. We didn't run it for this guide. A complete procedure for configuring and recovering auto-unseal on your own server is not published in this manual. Do not choose it without an independently tested recovery plan.

Bouncer and a sealed OpenBao

While OpenBao is sealed or unreachable:

  • the Bouncer server keeps running (from 0.5.6; older servers exited and restarted in a loop), and answers almost every call, /v1/health included, with HTTP 503 and backend_sealed or backend_unreachable;
  • nothing is issued, and clients say it's an outage, not a refusal: the CLI exits 12 or 13, and the apps say the server's vault is sealed or out of reach;
  • revokes are still accepted through bouncer-admin and the public API. Bouncer stops new issuance at once and revokes existing tokens at OpenBao once it answers. SSH certificates depend on each host fetching the updated revocation list. In 0.5.6 and earlier, hosts can't fetch it while OpenBao is unavailable, so a revoked certificate can keep starting new logins until OpenBao recovers and each host fetches the new list, or until it expires. From 0.5.7, Bouncer serves the list from its database during the outage, once it has read the CA public key (SSH revocation);
  • once you unseal OpenBao, the server notices by itself within a minute, sooner if something calls it.

From 0.5.6, BOUNCER_WAIT_FOR_OPENBAO isn't needed and is ignored. When Bouncer is unavailable explains every message. Monitor OpenBao's /v1/sys/health as well (Health checks).

Audit logging in OpenBao

OpenBao's audit log records every request made to it, so everything Bouncer does in OpenBao is recorded there too. See Audit for how this fits with Bouncer's own approval log.

With deploy/openbao, it's already on. The kit's openbao.hcl declares a file audit device that writes to /openbao/logs/audit.log, on the container's openbao-audit volume. Because it's declared in the config file, it can't be turned off through the API. To check it, on the recovery workstation, with an admin token:

bao audit list -detailed

and on the VM:

cd ~/bouncer/deploy/openbao
docker compose exec -T openbao ls -l /openbao/logs/

Expected result: a file/ device with file_path=/openbao/logs/audit.log, and that file growing as you use OpenBao.

The container can only write to /openbao/file (its data) and /openbao/logs, so keep audit files in /openbao/logs. To rotate the file, rename it inside that folder and send OpenBao a HUP signal, so it opens a new one:

docker compose exec -T openbao /bin/sh -c 'mv /openbao/logs/audit.log /openbao/logs/audit.log.1 && kill -HUP 1'

Check that a new request adds a line to the new audit.log before you compress or copy away the old file. Don't use copy-and-truncate, which can lose records, and don't use docker compose kill -s HUP, which can stop the container coming back after a reboot. The repository's deploy/openbao/README.md explains both.

With an OpenBao you already run, turn on at least one audit device yourself, in its server configuration. OpenBao refuses requests it can't write to at least one audit device, so consider two.

Backups of OpenBao

deploy/openbao/snapshot.sh and restore.sh take and restore Raft snapshots. A snapshot doesn't include the audit log, the host configuration or Bouncer's PostgreSQL database. See Backup and restore.

How this was checked

We ran steps 1 to 5, the audit check and Make a new root token in a sandbox on one machine: the kit's own compose.yaml and openbao.hcl with only the ports and project name changed, OpenBao 2.7.2, a throwaway age key, and docker compose exec in place of ssh. Every command above is the one we ran, with addresses and folders changed. The outputs are from that run. The role kit's setup, verify and check passed against the new KV mount; the issued secret_id lasted 32 days until we raised the AppRole mount's maximum lease time (Install, step 2d); after bao token revoke -self, the old root token was refused. tailscale serve, a second machine and a real recovery workstation weren't part of the run, and nor were TPM auto-unseal or an admin login of your own.

Upgrade the role kit (from 0.5.7)

This section is for an existing installation moving to 0.5.7. A new installation can skip it.

From that release, upgrade the role kit before the server, using that release's kit or a later compatible kit. The newer kit lets the server check each resource's OpenBao setup, and you need it before you turn on credential lifecycle.

  1. Keep the pre-upgrade backup and the old release's files.
  2. In the new release's deploy/openbao-role-kit, copy resources and kit.env from the old kit, and ssh/ssh-resources if you use SSH hosts. Log in to OpenBao with an authorised admin token. Run ./setup.sh --dry-run and read every change it proposes.
  3. Apply it with ./setup.sh. A kit policy that nobody edited by hand needs no extra flag. If it asks for --accept-unrevocable-tokens, stop and find out what differs before you continue.
  4. With the single-use check login from Install, step 2c, run ./check.sh to check the role's permissions. Do not give it the admin login.
  5. Run ./setup.sh --verify. It must exit 0.
  6. Run the new server's configuration check as its service user. It must exit 0 before you switch to the new server or add a resource to the live config.

If an earlier upgrade already used the 0.5.6 kit, do these steps again with the newer kit. On an installation that is otherwise up to date, only the bouncer-server policy changes: it can now read the token-role and resource-policy settings the check needs. An installation whose resource policies an older kit wrote also gets policies that refuse reads of older secret versions. No token role should change.

If you upgrade the server before the kit, it keeps issuing tokens as before. It logs a warning once for each resource and reports upgrade_needed, and the configuration check exits 4. That's expected, but it isn't a pass: without those settings the server can't confirm a role or policy is right, and older policies can still allow reads of earlier secret versions. Until you upgrade the kit, OpenBao's audit log can also show a permission-denied entry about once a minute for each resource. Failed creates and config reloads can cause extra checks.

Keep using the newer kit's scripts afterwards, including its emergency stop, even if you roll the server back. The 0.5.6 server works with the newer policy. An older kit reads the added lines as a hand edit: its --verify reports drift, its emergency stop cannot confirm a stop, and its setup.sh asks for --accept-unrevocable-tokens.

This procedure was checked against the source for 0.5.7. The project's automated tests, with fake data and OpenBao 2.7.2, ran the kit setup and the configuration check for a ready kit, an older kit, a missing role and a sealed OpenBao. The full procedure hasn't been run end to end.

This manual covers Bouncer 0.5.7. Built from source commit cdd3b5958d5be0b33e865b1b56b770f6bccffaf7 (workspace version 0.5.7+34).