Upgrades
How Grounded upgrades without downtime, how to pin digests and verify images, and what each release needs.
Grounded upgrades without downtime. New pods migrate the database while the previous version keeps serving, and for the length of a rollout old and new code share one database. That works because every migration is expand/contract: each release's schema works with the previous release's code, and destructive steps ship at least one release later. CI checks it on every release with an upgrade test.
How to upgrade
Read the release notes for anything to do before or after. Don't skip a release that its notes mark as required.
Take a backup, or check that last night's ran.
Bump the base and the image together in your overlay: change every ?ref= to the new tag, and pin the new digest of ghcr.io/ncecere/grounded (and ghcr.io/ncecere/grounded-ocr if you run OCR). Verify the digest first (below).
Let it roll. Each new pod's migrate init container migrates under an advisory lock (concurrent runs wait; only the first does any work), then new pods start while old ones keep serving (maxUnavailable: 0, maxSurge: 1). Workers roll at the same time and share the job queue.
Check /readyz, the admin Overview, and grounded doctor.
Stopping pods drain: they fail readiness, wait SHUTDOWN_DELAY for load balancers, then give in-flight requests (including streamed chats) and running jobs SHUTDOWN_TIMEOUT to finish. Jobs still running are cancelled and retried elsewhere.
Rolling back means pinning the previous digest: because of expand/contract, the previous release runs on the newer schema. Migrations are never rolled back. Downgrading across a release isn't supported; to go back further, restore a backup taken before the upgrade.
Pinning digests
Images are published for linux/amd64 and linux/arm64. Release tags are vX.Y.Z, vX.Y and latest-release; release candidates get only their own tag, such as v0.2.1-rc.1. There's no latest tag. Tags can move, so deploy by digest:
docker buildx imagetools inspect ghcr.io/ncecere/grounded:v0.2.1 # prints the index digestimages:
- name: ghcr.io/ncecere/grounded
newTag: v0.2.1 # informational
digest: sha256:<digest> # what is deployedVerifying images
Each image is built only in CI, scanned with Trivy, and signed with cosign using keyless signing tied to the repository's GitHub Actions workflows. Verify before you deploy:
cosign verify ghcr.io/ncecere/grounded@sha256:<digest> \
--certificate-identity-regexp '^https://github.com/ncecere/grounded/\.github/workflows/' \
--certificate-oidc-issuer https://token.actions.githubusercontent.comFor a stricter check, use --certificate-identity https://github.com/ncecere/grounded/.github/workflows/image.yml@refs/tags/v0.2.1. The same works for grounded-ocr. Then:
docker run --rm ghcr.io/ncecere/grounded@sha256:<digest> version
# grounded v0.2.1 (<commit>)Each image carries an SPDX SBOM and SLSA provenance as attestations, covered by the signature:
IMAGE=ghcr.io/ncecere/grounded@sha256:<digest>
docker buildx imagetools inspect "$IMAGE" --format '{{ json (index .SBOM "linux/amd64").SPDX }}' > sbom.spdx.json
docker buildx imagetools inspect "$IMAGE" --format '{{ json (index .Provenance "linux/amd64").SLSA }}' > provenance.jsonSince v0.2.0, each GitHub release also attaches the SBOMs per platform, the digest files and a checksums.txt (sha256sum -c checksums.txt).
Release by release
| From → to | Migrations | What to do |
|---|---|---|
| v0.2.0 → v0.2.1 | None (schema stays at 34) | Bump ?ref= and the digests. Some pages moved; old links redirect. |
| v0.1.0 → v0.2.0 | 00030 to 00034, additive only | Bump ?ref= and the digest. If you vendored the base, re-vendor or switch to the remote base. The image is public, so the private-registry component can go. New optional settings for OCR, SSO groups and evaluations. |
| First install: v0.1.0 | — | Install on Kubernetes. |
Security fixes are released as a patch of the latest minor release; older minor releases don't get fixes, so stay on the latest release. Before 1.0, minor releases may include breaking changes; the release notes say how to adapt.
Moved pages in v0.2.1
Admin → Legal holds and Admin → Profile migrations became tabs of Retention and Embedding profiles; a team's Crawl domains became a tab of Data sources; an agent's Versions tab moved to the version badge in the editor's header. The evaluations switch moved from Admin → Limits to Admin → Overview → Features. Old addresses redirect, so bookmarks keep working.