Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Backup and restore

Everything Bouncer knows is in its PostgreSQL database: devices, requesters (with the hashes of their keys), requests, grants and the approval log. No secret value is ever in it. Still treat backups as sensitive: they show what was approved, for whom and when.

The secrets themselves are in OpenBao, which has its own backups (OpenBao snapshots).

What to back up

WhatHowNotes
Bouncer's databasebackup.sh, daily and before every upgradeThis page
Your config file and the role kit's resources and kit.envKeep them in your own Git repositoryWithout them you can't rebuild the server
OpenBao's datadeploy/openbao/snapshot.shPlus the unseal material, kept separately
The secret files in /etc/bouncer/secretsDon't back them up. Make new ones insteadSee Install, steps 2d and 3

Take a backup

The server can keep running.

cd ~/bouncer/deploy/compose
BOUNCER_BACKUP_DIR=~/bouncer-backups ./backup.sh manual

Expected result: it prints the new file's path on its last line, for example /home/admin/bouncer-backups/bouncer-20261115T062835Z-manual.dump. In our test run the dump was 52 KB.

backup.sh runs pg_dump inside the PostgreSQL container, checks that the result really is a PostgreSQL dump, and keeps the newest 30. The folder is created with mode 0700 and each file 0600. Its settings:

VariableDefaultMeaning
BOUNCER_PROJECTbouncerThe Compose project name
BOUNCER_BACKUP_DIR/var/backups/bouncerWhere dumps go
BOUNCER_BACKUP_KEEP30How many dumps to keep

backup.sh uses the docker command. With Podman, take the dump directly instead:

mkdir -p -m 0700 ~/bouncer-backups
podman-compose exec -T -u postgres postgres pg_dump --format=custom --dbname=bouncer \
  > ~/bouncer-backups/bouncer-$(date -u +%Y%m%dT%H%M%SZ).dump

Copy backups off the server. A backup that only exists on the machine it protects doesn't help when that machine is lost.

Schedule backups

The kit has a systemd service and timer that run backup.sh daily into /var/backups/bouncer. On a server you installed by hand (Install the server), install them like this:

cd ~/bouncer/deploy/compose
sudo install -D -m 0755 backup.sh /usr/local/libexec/bouncer-backup
sudo install -m 0644 systemd/bouncer-logical-backup.service systemd/bouncer-logical-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now bouncer-logical-backup.timer
systemctl list-timers bouncer-logical-backup.timer

Expected result: list-timers shows the timer with its next run, some time in the next day. To try it at once: sudo systemctl start bouncer-logical-backup.service, then look in /var/backups/bouncer.

The service uses the docker command and the Compose project name bouncer, which is the name compose.yaml gives. With Podman, schedule the podman-compose command above with your own timer instead.

upgrade.sh and ship.sh can install the same timer with --install-backup-timer, but only as part of an upgrade with upgrade.sh, which also needs --release, --image, --version and --revision (Upgrade and roll back). Don't run them just to install the timer.

How often to back up, how long to keep backups and where to copy them are your decisions. Daily, kept for 30 days and copied to another machine, is a reasonable start.

Restore

Read this section to the end before you start. A restore replaces the database with the backup, and brings back everything as it was then, including things you have since revoked. The steps below put those revocations back before anyone can reach the server again.

The commands use docker compose. With Podman, use podman-compose in their place; it takes the same exec -T -u options. For the start step, use podman-compose up -d bouncer, then the curl --retry health check in Install, step 5, instead of Docker Compose's wait flags. We ran the restore with Docker only.

If you are rolling back an upgrade, follow the order in Roll back instead: it loads or rebuilds the old image in the middle of these steps, and leaves out the up and logs at the end of step 3.

What a restore brings back

These are all as they were when the backup was taken:

  • Devices revoked since the backup work again.
  • Requesters disabled since the backup work again.
  • A requester key replaced since the backup: the old key works again, and the new one is refused. Our test run showed this: after the restore, the replaced key was accepted and the new key was refused until we replaced it again.
  • Grants revoked since the backup are active again, until their end time.
  • Personal credentials: their sharing, their mode, and whether they were frozen, disabled or deleted.
  • Credential lifecycle expiries. A version expired after the backup is current and usable again because the record of its expiry is gone.
  • The SSH revocation list's version number, if you use SSH host resources. Hosts refuse a list older than the one they have, so revocations stop reaching them until the number catches up.

And two things a restore can't reach:

  • Tokens issued after the backup aren't in the restored database, so the kill switch can't see them. They end on their own when their lifetime runs out (5 minutes by default). The role kit's emergency stop can revoke them at once if needed.
  • Grants approved after the backup are gone. Their requesters ask again.

1. Record how things stand now

If the server still runs:

docker compose exec -T bouncer bouncer-admin device list > ~/before-restore-devices.txt
docker compose exec -T bouncer bouncer-admin requester list > ~/before-restore-requesters.txt

If it doesn't, write down every device you revoked, every requester you disabled and every key you replaced since the backup. Memory isn't a record. If you also have a recent approval log export (Audit), use it. Without either, plan to check every device and requester with the person it belongs to before step 6, and to replace every requester's key.

2. Cut off access to the server

Stop your HTTPS front end or block client access to it, so only this machine can reach Bouncer. Use the stop procedure for your proxy; the following command applies only to tailscale serve:

sudo tailscale serve --https=443 off

If this machine fetches the SSH revocation list itself, stop that too: sudo systemctl stop bouncer-krl.timer.

3. Replace the database

Run these commands from the Compose folder for the version being restored, with its .env selecting the matching image and the same Compose project name, database volume and config paths.

docker compose stop bouncer
docker compose exec -T -u postgres postgres dropdb --force bouncer
docker compose exec -T -u postgres postgres createdb --owner=bouncer bouncer
docker compose exec -T -u postgres postgres psql -c 'REVOKE ALL ON DATABASE bouncer FROM PUBLIC'
docker compose exec -T -u postgres postgres pg_restore --dbname=bouncer --no-owner --role=bouncer \
  --exit-on-error < ~/bouncer-backups/<file>.dump
docker compose up -d --wait --wait-timeout 120 bouncer
docker compose logs bouncer        # look for "bouncer-server is up"

up -d --wait returns once the server is healthy. If it fails or times out, keep client access closed and check the container logs and health checks.

If you are rolling back an upgrade, leave out the last two lines (up and logs): wait until Roll back has selected the old image and its matching database before starting Bouncer.

The server now runs on 127.0.0.1 only, because the front end is still off.

4. Put back what was revoked after the backup

Compare the lists from step 1 with the restored ones:

diff <(docker compose exec -T bouncer bouncer-admin device list) ~/before-restore-devices.txt
diff <(docker compose exec -T bouncer bouncer-admin requester list) ~/before-restore-requesters.txt

This is what the device diff looked like in our test, for a device revoked after the backup:

< dev_01jbx2a7c9e1g3h5j7k9m1n3p5  Alice's phone  alice  Android  active   yes  ...
---
> dev_01jbx2a7c9e1g3h5j7k9m1n3p5  Alice's phone  alice  Android  revoked  yes  ...

Then:

  • revoke each device that is active now but was revoked before: docker compose exec bouncer bouncer-admin device revoke <device-id>;

  • disable each requester that is active now but was disabled before: docker compose exec bouncer bouncer-admin requester disable <name>;

  • replace requester keys that changed after the backup. From 0.5.7, the requester list has a KEY ID column: a short label for each requester's current key, 12 lower-case hexadecimal characters, that isn't secret. For each requester, compare its KEY ID in the list you saved in step 1 with the one in the restored list. If they differ, the key was replaced after the backup, even when the KEY ISSUED times match. Ignore LAST USED here.

    If either list has no KEY ID column, or you couldn't save a list in step 1, you can't compare keys this way. If you have an approval log export, kept off this server, that covers every requester_key_issued entry since the backup, replace the keys it shows were issued after the backup. Otherwise replace every active requester key. Matching KEY ISSUED times don't prove a key is unchanged.

    Replace each of those keys, write the new key to a private file, and give it to whoever runs that requester:

    (umask 077; docker compose exec -T bouncer bouncer-admin requester rotate-key <name> > <name>.key)
    
  • if you revoked any grant or pulled the kill switch after the backup, or you aren't sure, pull it now:

    docker compose exec bouncer bouncer-admin revoke --all
    

If any resource or personal storage area has lifecycle on, keep outside access closed until you have dealt with expiries. Use the approval log export kept off this server to find every credential_expired entry since the backup. For each affected credential, save a replacement value in OpenBao (for example with bao kv put -mount=<mount> <path> <field>=-, as an OpenBao administrator), rotating it at its issuer first if needed. The - reads the value from standard input. Include every field the secret needs, because kv put replaces its contents. OpenBao's newest version becomes current. If the export does not cover the whole period, replace the current value of every lifecycle credential. Do this before reopening access; expiring a value again through the app needs access open. See Restore or roll back.

If you use personal credentials, freeze every storage area until the owners have checked their credentials, because sharing, modes and deletions are also back as they were:

docker compose exec bouncer bouncer-admin personal freeze --namespace <namespace-id>

The freeze prints the IDs it froze. Then, with each owner:

  • Credentials they deleted after the backup: leave them frozen. Their values are still in OpenBao, because deleting a personal credential only marks its record deleted.
  • Credentials that were disabled at the backup come back disabled, and personal freeze leaves them alone. Nobody can use a disabled credential, but its owner can turn it on again, so if they had deleted it since, ask them to delete it again.
  • For each frozen credential they still want, turn it back on with bouncer-admin personal reassign <credential-id> --owner <person>, one at a time. It comes back with the sharing it had at the backup, so the owner then removes any sharing they had withdrawn since.

There's no command that lists personal credentials. To see whose each one is, read the restored database:

docker compose exec -T -u postgres postgres psql --dbname=bouncer \
  -c 'SELECT id, namespace, owner, name, status FROM personal_credentials ORDER BY namespace, owner, name'

The repository's deploy/compose/README.md ("Backups") has the full procedure.

5. If you use SSH host resources

Wait until the longest credential_ttl of your SSH host resources, plus five minutes, has passed since you stopped the server in step 3. Then, on each host that fetches the revocation list, do steps 1 and 2 of Reset host version state: stop its timer and any fetch in progress, and remove only its version file. Leave the timer stopped. Steps 3 to 5 come in step 6, once access to the server is open again.

The repository's deploy/compose/README.md explains why the order matters.

6. Open access again

Restore the same HTTPS front end or access rule you stopped in step 2. For tailscale serve:

sudo tailscale serve --bg --https=443 http://127.0.0.1:8080

With SSH host resources, each host's timer is still stopped from step 5. On each host, start a fetch yourself (step 3 of Reset host version state), then check that the host took the restored revocation list and that sshd uses it through RevokedKeys. If a fetch fails, follow step 4 of that section.

sudo systemctl start bouncer-krl.service
journalctl -u bouncer-krl.service -g 'bouncer-krl:' -n 1

Expected result: a line saying installed version N (written by the server at ...), or unchanged version N if the host already had the restored list. Check that every host shows the same version. The server can't confirm delivery itself: it reports SSH revocation as unknown with pending: krl_delivery. An open SSH session is unaffected.

Once every host has confirmed the restored list, resume its timer: sudo systemctl start bouncer-krl.timer.

OpenBao snapshots

deploy/openbao/snapshot.sh DIRECTORY saves a Raft snapshot of OpenBao, checks it can be read, and records a checksum. Run it with a dedicated snapshot identity (with read on sys/storage/raft/snapshot), never Bouncer's AppRole or the root token:

BAO_ADDR=https://openbao.example.com:8443 BAO_TOKEN_FILE=~/.openbao-snapshot-token \
  deploy/openbao/snapshot.sh ~/openbao-snapshots

A snapshot needs the matching unseal material to be useful, so keep both, separately and encrypted. To restore, use deploy/openbao/restore.sh on an isolated recovery instance, after you have stopped Bouncer and cut off client access. Restoring OpenBao can bring back a secret value you have since rotated, so reconcile revocations afterwards, as for Bouncer's database. The repository's deploy/openbao/README.md has the commands.

Not verified for this page. OpenBao snapshot and restore weren't part of our test run. The project's deploy/openbao/test-in-docker.sh tests them against throwaway servers.

How this was checked

In our test run we took a backup with backup.sh, then replaced a requester's key and revoked a device, then followed the versions of steps 1, 3 and 4 that were written at the time, with Docker. The restored server accepted the old key and showed the device as active; the diff showed both changes; after step 4 the device was revoked again and only the newest key worked.

That run took place before KEY ID existed, so it did not use the KEY ID comparison in step 4. That comparison is checked against the source and the project's automated tests, which replace a key twice in the same second and confirm each new key gets a different KEY ID. It has not been tried in a restore run.

The kit's own test (41 checks, all passed) also covers backup and restore. Steps 2, 5 and 6 (Tailscale and SSH hosts), the personal-credential steps, the backup timer and Podman were not part of that run; those parts come from the repository's deploy/compose/README.md and the scripts.

This manual covers Bouncer 0.5.7. Built from source commit cdd3b5958d5be0b33e865b1b56b770f6bccffaf7 (workspace version 0.5.7+34).