Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Manual for Bouncer 0.5.7. Chapters state version and distribution limits. Source cdd3b59, site build 20 November 2026. Your server or app may run an earlier version.

On this page

Upgrade and roll back

The server updates its database itself: each version carries its database migrations (numbered SQL changes), and runs the new ones when it starts. Version 0.5.6 has migrations through 0015. Version 0.5.7 adds more, including 0017, which stores the token records described in Audit. Numbers can be skipped, so check the list in the release you install rather than expecting every number in turn. Back up before starting a new server version. An older image may need the matching database backup to run again.

Check the server's own version. Updating an app or the CLI doesn't upgrade the server, so don't assume a migration has run because a client is newer. The Bouncer project's own production server runs build 0.3.8+25. Version 0.5.7 is published as a private GitLab release (Platforms and status).

For 0.5.7, upgrade the role kit first. Follow Upgrade the role kit, even if an earlier upgrade already used the 0.5.6 kit. If you upgrade the server first, it keeps issuing tokens as before but warns that the kit needs upgrading, and you can't turn on credential lifecycle until you upgrade it. Before you switch to the new server image or add a resource, the new server's configuration check (--check-config) must exit 0.

Read the release notes first. Also compare the new compose.yaml, .env.example and bouncer.yaml.example with the ones you have, for settings you need to add. Some config settings are refused by older servers, so a release may ask you to upgrade the server before you use a new setting.

From 0.5.7, upgrade any stand-alone bouncer-admin with the server. The copy inside the server container is upgraded with it. A bouncer-admin from 0.5.6 or earlier, such as one a scheduled export job runs on the host, parses each page but writes entries one by one. It stops at the first token_issued, backend_revoke, resource_provisioning_changed or ssh_ca_changed entry with Can't write an entry as JSON. Earlier entries, including those from the same page, have already been written. With --cursor-file, it does not save the cursor, so a retry repeats that output. See Audit.

0.5.7 also adds migration 0018 for provisioning events and 0019 for retained SSH CA public keys, after 0017 for token records. This release also adds ssh_ca_changed approval-log entries; upgrade stand-alone admin clients as described above. Let Bouncer read its public CA successfully after upgrading and before rotating the CA. Returning to 0.5.6 requires the pre-upgrade database backup and, on SSH destinations, the deliberate host version-state reset after reconciling revokes made since the backup. Do not delete migration records or edit their checksums to force the older binary to start: it cannot show the newer log and provisioning-event kinds for affected approvers. See Roll back. This was checked against the source. The earlier release-download test didn't run these later migrations.

Upgrade from a release download

If you installed from a release download, follow the new release's INSTALL-SERVER.md, "Upgrading to a later release". In short: download the new release's files and check them (sha256sum -c bouncer-server-<new version>-SHA256SUMS), load its image and check its labels, and unpack its kit next to the one you run, without the .env lines (upgrade.sh copies your .env across). For 0.5.7, first apply the new kit and run the configuration check as described above. Then run the new kit's upgrade.sh (below) with the new release's values:

cd ~/bouncer-releases/bouncer-server-<new version>/deploy/compose
./upgrade.sh --release ~/bouncer-releases/bouncer-server-<new version> \
  --image localhost/bouncer-server:<new short commit> --version <new version> --revision <new commit>

It takes a backup first, and puts the old release back if the new one doesn't come up healthy. The old release's folder and image stay, for a rollback.

Upgrade by hand

This is for a server built from the source:

cd ~/bouncer
git describe --tags --always > ~/bouncer-previous-version   # needed to roll back
git fetch && git checkout <new tag or commit>
cd deploy/compose
BOUNCER_BACKUP_DIR=~/bouncer-backups ./backup.sh pre-upgrade   # see Backup and restore
docker compose build --build-arg BOUNCER_VERSION="$(git describe --tags --always)" \
  --build-arg BOUNCER_REVISION="$(git rev-parse HEAD)"
docker compose up -d --wait --wait-timeout 120
docker compose logs bouncer                     # look for "bouncer-server is up"
curl -s http://127.0.0.1:8080/v1/health         # check the version in the answer

up -d replaces only the containers whose image or settings changed. The database volume stays. If a migration fails, the server stops at start-up instead of serving.

--wait waits for the containers' health checks before the next command. If it fails or times out, stop here and inspect the logs and health checks. Do not treat a container that is still starting as a failed migration. This wait does not verify every resource or replace the post-upgrade request test.

Afterwards, docker image prune removes old images.

Upgrade with upgrade.sh

deploy/compose/upgrade.sh does the same on the server and puts the old release back if the new one doesn't come up healthy:

deploy/compose/upgrade.sh --release <new release folder> --image <image> \
  --version <version text> --revision <commit> [--install-backup-timer] [--dry-run]

The release folder holds the new release's deploy/ folder (for example from git archive <commit> deploy), and the image must already be loaded. In order, the script:

  1. finds the running release from Docker's labels, and copies its .env into the new release with the new image, version and revision;
  2. checks the image exists, and that OpenBao answers and is unsealed (otherwise, from 0.5.6, the new server would answer 503 and the upgrade would roll back);
  3. stops the server, so nothing changes after the backup, and takes a backup with backup.sh;
  4. starts the new release and waits for /v1/health (240 seconds by default, BOUNCER_WAIT);
  5. if that fails, restores the backup (only when the new server ran migrations) and starts the old release again, then exits 1;
  6. on success, records the new release in CURRENT and the old one in PREVIOUS in the releases folder.

--dry-run prints the new .env without changing anything. --install-backup-timer also installs the daily backup timer (Backup and restore).

Build elsewhere with ship.sh

For a server too small to compile Rust, deploy/compose/ship.sh builds the image on the machine you run it on, from exactly one commit, copies the image and that commit's deploy/ folder to the server over SSH, and runs upgrade.sh there:

deploy/compose/ship.sh admin@bouncer.example.com <commit> [--version <text>] [--dry-run] [--install-backup-timer]

Run it from a clean checkout that has the commit. On the server it needs Docker, and sudo -n for mkdir, tee and chmod under /opt/bouncer. Each release lands in /opt/bouncer/releases/<full commit>. Old releases and images aren't removed, so a rollback stays possible.

ship.sh also uses sudo -n chown on the destination, so its account needs that permission.

Roll back

Going back to an older version after a newer one has migrated the database doesn't work by itself. The older server refuses a database that has migrations it doesn't know. So a rollback is a restore of the backup you took before the upgrade. Everything that happened since that backup is lost, and the restore caveats in Backup and restore apply. Read that section first.

Migration 0017, from 0.5.7, is one of these. To go back to 0.5.6, restore the pre-upgrade backup. Do not delete migration records to make the older server start: it cannot show the new token records, so the app's log fails for approvers once a token has been issued, and so does the export.

A source build reuses the image name localhost/bouncer-server:local, so the upgrade replaced the old image and you rebuild it before you start anything. A release download keeps each image under its commit name, so you load the old one. Do it in this order:

  1. Do restore steps 1, record how things stand, and 2, cut off access, while the new version still runs.

    Before stopping Bouncer, save the newer database and export the full approval log from the current release's Compose folder. The export needs the running server's admin socket. Keep both files off this server before continuing; the log enables the exact requester-key reconciliation in restore step 4 if it covers every entry since the pre-upgrade backup. Use your own backup host and a private destination folder:

    ROLLBACK_DUMP=$(BOUNCER_BACKUP_DIR=~/bouncer-backups ./backup.sh before-rollback)
    (umask 077; docker compose exec -T bouncer bouncer-admin log export > ~/before-rollback-log.jsonl)
    scp "$ROLLBACK_DUMP" ~/before-rollback-log.jsonl backup-admin@backup.example.com:/srv/backups/bouncer/
    

    Check that the dump, log export and off-server copy all succeeded before continuing. If the log is incomplete, rotate every active requester key during reconciliation.

  2. Stop the current server from the folder it runs from: the new release's deploy/compose for a release download, or ~/bouncer/deploy/compose for a source build. Run docker compose stop bouncer there. Then use the route that matches your installation:

    From a release download: use the old release's image and compose kit, with the .env you kept before the upgrade. No source checkout or image build is needed. In the folder holding the old download, verify its sums as in Install, step 1a. For 0.5.5, check, load and inspect it:

    sha256sum -c SHA256SUMS --ignore-missing
    gunzip -c 'bouncer-server-<old version>-image.tar.gz' | docker load
    docker image inspect 'localhost/bouncer-server:<old short commit>' \
      --format '{{index .Config.Labels "org.opencontainers.image.revision"}} {{index .Config.Labels "org.opencontainers.image.version"}}'
    

    The labels must match the old release's INSTALL-SERVER.md. If you still have the old release folder and its .env, use them. If not, unpack the old compose archive and restore its .env from your pre-upgrade copy:

    tar -xzf 'bouncer-server-<old version>-compose.tar.gz' -C ~/bouncer-releases
    

    Do not unpack over an existing release folder. In its deploy/compose, confirm BOUNCER_IMAGE, BOUNCER_VERSION and BOUNCER_REVISION name the old release, with the same Compose project name, database volume and config paths as before. Stay in this old kit folder for steps 3 and 4:

    cd "$HOME/bouncer-releases/bouncer-server-<old version>/deploy/compose"
    

    From a source checkout: check out the version you recorded before the upgrade and build its image, as in Install, step 4:

    cd ~/bouncer/deploy/compose
    git checkout "$(cat ~/bouncer-previous-version)"
    docker compose build --build-arg BOUNCER_VERSION="$(git describe --tags --always)" \
      --build-arg BOUNCER_REVISION="$(git rev-parse HEAD)"
    
  3. For both routes: if the new version added settings the old one refuses (for example require_hardware_key before 0.5.7, credential_type before 0.5.0, or push.fcm_also before build 0.3.8+25), take them out of /etc/bouncer/config/bouncer.yaml now. The old server will not start with them. Removing a security setting such as require_hardware_key lets weaker keys approve on the older server. Tell the affected approvers, or keep the resource out of use until you upgrade again.

    Do restore step 3, replace the database, without its last two lines (up and logs). Use the pre-upgrade dump that matches the old server. This discards database changes made since that backup. Use the lists and off-server log you saved above for reconciliation in restore steps 4 and 5. Do not edit _sqlx_migrations or reverse SQL changes by hand.

  4. Start the old version from its compose folder. up -d recreates the container using the old image selected in that folder's .env:

    docker compose up -d --wait --wait-timeout 120
    docker compose logs bouncer                     # look for "bouncer-server is up"
    curl -s http://127.0.0.1:8080/v1/health         # the old version in the answer
    
  5. Carry on with restore steps 4, 5 and 6. For SSH destinations, reconcile revokes made since the backup before resetting their version state. Restoring the database, including a rollback to 0.5.6 after migration 0019, can put the server's KRL version below what hosts already hold.

Release notes that say a version adds no migrations don't mean an older server can safely use the newer database. Roll back with the matching backup as above, or follow the release's own rollback instructions if it has them.

upgrade.sh does a rollback for you when the new release fails its health check during the upgrade: it starts the old release folder with the image it ran before, so nothing needs rebuilding as long as you give each release its own image name (ship.sh names each one after its commit). A rollback later, after people have used the new version, is a restore as above.

Reset host version state after a restore or rollback

A database restore also puts back the SSH revocation list's version number. Hosts may already hold a higher version, and they refuse lists with a lower version, so later revokes may not reach them. Going back to 0.5.6 with the pre-upgrade backup has the same risk. Keeping the version number across a restore is not implemented and is tracked in issue #283.

Resetting a host's version state removes that protection, so do it only when all of these are true:

  • you have confirmed that the restored server is the one you meant to restore;
  • you have put back the revokes made since the backup, as in the restore procedure;
  • the longest credential_ttl of your SSH host resources, plus five minutes, has passed since you stopped the old server. The restored database doesn't know about grants made after the backup, so it can't revoke their certificates. Waiting lets them expire, so a restored list can't let one of them log in again.

If you can't reconstruct a revoke, wait until every certificate it affected has expired, or keep that account's Bouncer principals file removed until then. A successful fetch doesn't make up for missing revoke history. Don't reset because of an older-version error you can't explain.

Before you start, open a separate SSH session to each host without using a Bouncer certificate, and keep it open, so you can still get in if something goes wrong.

On each host:

  1. Stop scheduled fetches and any fetch in progress:

    sudo systemctl stop bouncer-krl.timer
    sudo systemctl stop bouncer-krl.service
    
  2. Remove only the version file:

    sudo rm -f /var/lib/bouncer-krl/version
    

    That's the kit's default path. If /etc/bouncer/krl.env sets BOUNCER_KRL_STATE_DIR, remove the version file in that folder instead, and use the same path in step 4. Don't remove the installed list (/etc/ssh/bouncer/revoked.krl), the host's trusted SSH CA public-key file (/etc/ssh/bouncer/trusted-user-ca.pub) or last_ok.

  3. Fetch the list from the restored server. Use the service, which reads /etc/bouncer/krl.env; running the pull script directly doesn't read it:

    sudo systemctl start bouncer-krl.service
    
  4. Check that the fetch succeeded and installed a list from the restored server:

    journalctl -u bouncer-krl.service -n 20 --no-pager
    sudo cat /var/lib/bouncer-krl/version
    

    If the fetch failed, stop here. The host keeps its installed list and its record of the last successful fetch. While the timer is stopped, no scheduled fetch runs to report staleness, so keep checking it yourself with sudo /usr/local/sbin/bouncer-krl-pull --check. It reports STALE once the freshness limit passes. If /etc/bouncer/krl.env sets BOUNCER_KRL_STATE_DIR or BOUNCER_KRL_MAX_AGE, give --check the same settings. This only checks freshness; fix the cause, then repeat step 3.

  5. Once the fetch has succeeded, resume scheduled fetches:

    sudo systemctl start bouncer-krl.timer
    systemctl list-timers bouncer-krl.timer
    

From then on, the host again refuses any list with a lower version than the one it just installed.

The version reset, and what happens when a fetch fails and recovers, were checked with real KRL files and the unmodified pull script against a local fake server. This wasn't a production restore or a test on a physical host.

The OpenBao role kit

Run the role kit from the same release as the server (deploy/openbao-role-kit), with your admin token. For 0.5.7, do this before switching the server, as described above; for earlier releases, do it after each upgrade.

Copy resources and kit.env from the old release's deploy/openbao-role-kit into the new one first, and ssh/ssh-resources if you use SSH hosts. Run the kit and emergency stop from that folder afterwards. Without kit.env, defaults apply: using --accept-unrevocable-tokens could replace settings such as address binding with those defaults.

./setup.sh --verify   # what differs from what this version of the kit writes
./setup.sh            # apply it
./setup.sh --verify   # must now exit 0

From 0.5.3, the kit's resource policies also stop issued tokens reading older versions of a secret (?version=N). On an install set up by a kit up to 0.5.2, the first --verify lists each resource policy as OLDER, not DRIFT, and setup.sh upgrades them without --accept-unrevocable-tokens. On an install set up by the 0.5.3 or 0.5.4 kit, nothing should be OLDER. When moving from the 0.5.6 kit to the next release's kit, the first --verify also lists policy bouncer-server as DRIFT: the earlier policy lacks the read-only provisioning lines, and setup.sh restores it without a flag. That one is expected. Stop and investigate any other DRIFT (deploy/openbao-role-kit/README.md, "Upgrading the kit"). The emergency stop works on an install that isn't upgraded yet.

After setup.sh, check that bouncer-server (or <prefix>-server) has the line # kit-policy-format: 2: bao policy read bouncer-server | grep -x '# kit-policy-format: 2'. Kits after 0.5.5 also show it in ./setup.sh --verify, as ok policy bouncer-server has the format line "# kit-policy-format: 2". The 0.5.3 and 0.5.4 kits didn't write it, and until it's there, the kit relies on weaker checks to tell its own older policies from a hand edit (see "Known limits" under "Upgrading the kit" in deploy/openbao-role-kit/README.md).

Phone and PC apps

Upgrade the server before the apps when a release says so. A newer app checks what the server supports (for example credential_types in the health answer) and doesn't send what an older server would refuse.

How this was checked

The original kit test restarted the stack with the database kept and checked that existing records survived. It did not run the whole rollback procedure.

A later test with fake data followed this page's release-download route from 0.5.5 to the unpublished 0.5.6 build and back, using the matching database backup and the health waits shown above. It checked that the requester, grant and paired device survived. Both versions had the same migrations, so it didn't test going back across a database change. A local copy stood in for the copy to the example backup host, so copying to another machine wasn't tested.

That run used a software stand-in for an approving device and the new kit's upgrade.sh for the upgrade. It did not run a full source-build rollback, ship.sh, or a server without lifecycle support against reconciled data. The source-build and ship.sh descriptions are checked against source. For a source build, rebuilding the old image before starting anything follows from the image name both versions share.

This manual covers Bouncer 0.5.7. Built from source commit cdd3b5958d5be0b33e865b1b56b770f6bccffaf7 (workspace version 0.5.7+34).