Upgrading RunR

A RunR installation is upgraded in place with helm upgrade of its release. The database, the API volume, the generated secrets and every region keep running. Never uninstall and install again: helm uninstall deletes the chart's PersistentVolumeClaims (database, API volume) and the generated runr-secret, whose APP_KEY encrypts the secrets in the database.

Procedure#

1. Back up the database. With the chart's PostgreSQL:

   kubectl -n runr exec deploy/runr-postgres -- pg_dump -U runr -Fc runr > runr-$(date +%F).dump

2. Keep APP_KEY. The chart reads it from the existing runr-secret on every upgrade (lookup), so nothing has to be done for an install the chart created. Do not set secret.appKey for the first time in the same upgrade (see Key rotation for changing it). Tools that render the chart without cluster access (helm template, some GitOps controllers) cannot read the Secret: give them secret.appKey with the current key, or manage the Secret yourself (secret.create=false).

3. Upgrade the release with the new chart and its images. The chart comes from BRF's public Helm repository, added as brf-tech at install (Kubernetes (Helm)); --version and the two image tags pin the release:

   helm repo update brf-tech
   helm upgrade runr brf-tech/runr --version <version> -n runr --reuse-values \
     --set api.image.tag=<version> --set ui.image.tag=<version>

   --reuse-values keeps the values of the running release; keys the new chart added take their defaults. --reset-then-reuse-values does the same but starts from the new chart's values.yaml. Values files work as before (-f values.yaml); new settings such as branding.* can be added in the same upgrade.

4. Check GET /api/version and kubectl -n runr logs job/runr-bootstrap.

What runs during the upgrade, in order:

Functions keep serving throughout: their traffic goes through each region's ingress, not through the console. Existing Knative Services, CronJobs and sandboxes are not redeployed by the upgrade; they pick up the new version's manifests on their next deploy. region:reconcile changes only what a release lists below (e.g. the image namespace of 0.5.1).

Between the migration job and the end of the rollout the previous version still serves the console — 27 seconds in the e2e run below (migration job 9 s, then the new pods ready). Do not edit projects, deploy or change regions in that time: the previous version cannot read values the migration has just encrypted, shows env vars as empty and would save that. Its region status checks fail in that window and the new version's next check reports the region again.

0.5.1 → 0.5.2#

Tracing backend#

region:reconcile (step tracing) brings the tracing backend of every region to this release where tracing is enabled, and installs nothing anywhere else:

It installs a missing backend and updates one whose chart version or image differs from this release's: chart open-telemetry/opentelemetry-collector 0.176.0 with ghcr.io/brf-tech/reqs/opentelemetry-collector-contrib:0.162.0, chart grafana-community/tempo 3.2.0 with ghcr.io/brf-tech/reqs/tempo:3.1.0 — RunR's mirrors of the Docker Hub images 0.5.0 and 0.5.1 pulled directly (otel/opentelemetry-collector-contrib, grafana/tempo), so regions no longer depend on Docker Hub's anonymous pull limit for them. It runs what php artisan cluster:install-tracing runs: a Helm upgrade (or install) of the otel-collector and tempo releases with their values rendered anew, the span-metrics PodMonitor and Knative's tracing configuration. The collector and Tempo pods roll once — spans sent while the collector restarts are dropped, Tempo keeps its traces on its volume — and the region's tracing status is installing until they run. Regions on one cluster share the backend: the first region of the cluster with tracing enabled decides it.

Tempo chart 3.2.0 applies the retention (observability.tracing.tempoRetentionHours, default 72 hours, or the region's own) where Tempo deletes old traces; with chart 3.1.0 the setting did not reach that part of Tempo, which kept traces for its default of 14 days (the chart's upgrade notes). After the update traces older than the retention are removed: raise it before the upgrade to keep them longer.

Before the upgrade:

• php artisan region:reconcile --dry-run lists what the step would install or update.
• To keep tracing off on regions without their own setting, set observability.tracing.installWithServing=false (or defaultMode=off) in the upgrade; for one region, set its tracing mode to off in the console.
• Regions that pull from a registry of your own: mirror the two images there and set observability.tracing.collectorImage / tempoImage (the same versions).

observability.tracing.collectorChartVersion and tempoChartVersion are empty by default (the versions above). The values the charts of 0.5.0 and 0.5.1 set, 0.175.1 and 3.1.0, which --reuse-values keeps, count as empty; set another version to pin one.

Image namespace ghcr.io/brf-tech/runr#

RunR's images are published under ghcr.io/brf-tech/runr, so the organization's other packages never share a name with RunR's. The tags and digests are the ones 0.5.1 published directly under ghcr.io/brf-tech (and releases up to 0.5.0 under ghcr.io/runrsh); the runr- prefix of the component images is gone:

The upgrade moves every reference an existing installation holds, from 0.5.1 and from earlier releases alike, the way the 0.5.1 move did: Helm values (template helper runr.image), the database (migration move_images_to_runr_namespace and tenant migration 0055, then php artisan images:move-registry in the bootstrap job), the region workloads, the region Traefik and the WebAssembly installer (region:reconcile: image-registry, traefik-image, wasm-sandbox). Only RunR's own packages move: another package directly under ghcr.io/brf-tech, a mirror of your own and the region registry's <registry>/ghcr.io/… path stay as they are. The system images policy passes ghcr.io/brf-tech/runr/** (region:reconcile, admission-policy, before the workloads move) and keeps passing ghcr.io/brf-tech/** and ghcr.io/runrsh/** for the pods that keep running with an image of the earlier names.

Two kinds of region workloads move their pod template but keep their running pods, because a restart costs more than the move; a pod created later — on a new node, or after the pod was deleted or evicted — starts from ghcr.io/brf-tech/runr:

region:reconcile reports them as moved, with what keeps running; such a pod runs its image from the earlier name until it is replaced. A Redis Deployment paused by someone else, or in the middle of a rollout, is listed as kept: and moves at the first run after it is resumed or the rollout is over.

php artisan images:move-registry --dry-run and php artisan region:reconcile --dry-run show what moves before the upgrade's bootstrap job does it.

A registry of your own that holds copies of RunR's images under their paths (an air-gapped registry, the CLI's RUNR_RUNTIME_REGISTRY) needs them under the new paths: copy them from ghcr.io/brf-tech/runr. Image settings that point into such a registry stay as they are.

0.5.0 → 0.5.1#

Image namespace ghcr.io/brf-tech#

0.5.2 moves these references on to ghcr.io/brf-tech/runr (Image namespace ghcr.io/brf-tech/runr); an upgrade from 0.5.0 straight to 0.5.2 moves them there in one step.

RunR's images are published under ghcr.io/brf-tech — the same image names, tags and digests that releases up to 0.5.0 published under ghcr.io/runrsh. The upgrade moves every reference an existing installation holds (also one that comes from 0.4.x); only the exact prefix ghcr.io/runrsh/ moves, references into other registries — a mirror of your own, the region registry's <registry>/ghcr.io/… path — stay as they are:

The node installers of Kata (with its devmapper) and gVisor — a new pod re-runs the install on its node, and kata-deploy's cleanup restarts the container runtime — and the region's queue Redis (runr-redis), whose streams live in memory, move their pod template but keep their running pods (region:reconcile does not count the image namespace alone as a Kata change); how is described under Image namespace ghcr.io/brf-tech/runr. The system images policy keeps passing ghcr.io/runrsh for those pods.

In the console's namespace the chart's PostgreSQL, Redis, gateway and Mailpit roll once with their moved images, together with the new version's API and UI: PostgreSQL is unavailable for a few seconds (an API pod that starts meanwhile restarts until it is back) and Redis starts empty — the usual rule of the rollout window applies, do not deploy or edit projects until it is over.

php artisan images:move-registry --dry-run and php artisan region:reconcile --dry-run show what moves before the upgrade's bootstrap job does it.

Outside the installation:

• CLI: runr downloads and update checks use the brf-tech/runr releases; with Go: go install github.com/brf-tech/runr/cli@latest.
• GitHub Action: uses: brf-tech/runr/actions/deploy@<tag>.
• Terraform: the provider's address is brf-tech/runr. Change source in required_providers and move the state once: terraform state replace-provider registry.terraform.io/runrsh/runr registry.terraform.io/brf-tech/runr.

0.4.x → 0.5.0#

Roles and API tokens#

0.4.x had no permission checks for single-tenant users. The upgrade gives every existing user a role (migration assign_roles_to_existing_users):

Two things 0.4.x stored but did not enforce are enforced now:

• Tenant viewers could change and deploy projects in 0.4.x although the role was named Viewer. 0.5.0 keeps them read-only. Give a viewer who needs to work on projects the Member or Developer role.
• API token scopes. Tokens with * keep full access (within their user's role). Tokens with 0.4.x scopes get those scopes as permissions (projects:read → projects.view, logs.view, metrics.view, git.pull, …; the full list is in permissions.md), so a token created with projects:read reads but no longer deploys. A token created without any scope has no permissions; create a new one with the scopes it needs.

Users the previous version created while the upgrade rolled out get the default role from the bootstrap job (php artisan rbac:assign-missing, safe to run again).

Secrets at rest#

The migration encrypt_secret_columns encrypts env vars, version snapshots, region kubeconfigs, registry and Redis passwords, webhook secrets and the other columns listed in security.md with APP_KEY. Values still in plaintext stay readable; kubectl -n runr exec deploy/runr-api -- php artisan secrets:encrypt encrypts anything written in plaintext later (it prints Encrypted 0 plaintext value(s) when there is nothing left). On large installs the migration job's 300-second deadline applies to the whole migration run.

Copies outside the database are not rewritten: the git history of each project on the API volume (commits of 0.4.x contain .config.json with the env values), the code ConfigMaps of deployed versions and the env of running Knative Services keep their plain values until those versions are deployed again.

Secret env vars#

0.4.x had plain env vars only; they stay plain after the upgrade and keep their values. To turn one into a secret, move it in one request and deploy:

curl -X PUT https://runr.example.com/api/projects/<id> -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"config":{"env_vars":{"PLAIN_VAR":"…"},"secret_env_vars":{"API_TOKEN":"…"}}}'

The deploy writes the Kubernetes Secret <project-slug>-secret-env in the region and the Knative Service reads the variable from it (secretKeyRef); the value no longer appears in the Service, the version or the API's project resource (security.md).

White-label#

branding.* values apply at the upgrade that sets them (no rebuild): product and company name, logos (URLs, data: URIs or ConfigMap / Secret references), colours (one base colour or the full 300–900 scale) and the login texts — white-label.md. GET /api/branding shows the result.

Billing (multi-tenant)#

0.5.0 calculates what each tenant owes and leaves invoicing to your billing system, which reads the amounts through the billing API and receives signed billing events. The upgrade migration turn_payment_records_into_charges turns the 0.4.x payment records into charges:

The payment fields (Pay link, due date, paid and refunded amounts) are not carried over — your billing system holds them. The 0.4.x billing service integration is replaced as follows:

Set billing.webhookUrl (and keep or set billing.webhookSecret) to receive events; without a URL the events are kept for GET /api/billing/events.

Region Traefik image#

The region Traefik runs ghcr.io/brf-tech/reqs/traefik:v3.7.4, RunR's mirror of the docker.io/traefik image its pinned chart runs, so its replicas — and a node that schedules one later — no longer pull from Docker Hub and its anonymous pull limit. region:reconcile moves the Traefik of existing regions that runs the same Traefik version from Docker Hub (a Helm upgrade of the traefik release with its values reused and only the image set; the Traefik pods roll once, one at a time) and leaves a Traefik of another version alone. To pull it from a registry of your own, set knative.traefikImage (the same Traefik version) before the upgrade.

Kata Containers#

Of RunR's Kata drop-ins only Dragonball's changed: on regions whose Kata install includes Dragonball, region:reconcile writes the new Dragonball drop-in to the nodes (no kata-deploy run, no restart); regions with the other hypervisors only keep theirs (from 0.4.1, whose guests still boot systemd, every drop-in is written once). Dragonball guests get 1 vCPU + the pod's CPU limits from 0.5.0 on (kata-hypervisors.md); a Dragonball function gets them at its next deploy.

Testing the upgrade path#

runtime/ci.sh test-helm-upgrade repeats the upgrade on a Kind cluster (runtime/ci.sh test-upgrade does the same for a Docker Compose install): it installs the chart and images of main (--from=<ref>; the API and UI images of that ref's chart values at its appVersion, built from that ref when they are not public) with the chart defaults, creates data through the old API — the administrator and a second user, the same cluster as a region (registry, Knative Serving, Traefik, the region agent), a Python function with two env vars, a full-access and a projects:read token — and runs helm upgrade --reuse-values (--values-mode=reset: --reset-then-reuse-values) to the checkout's chart and images with a test brand (Acme Cloud, logos as data: URIs, a primary 300–900 scale) while a prober calls the function every second. It then checks:

• the pre-upgrade migration succeeded, no migration is pending, /api/version answers (it reports the chart's appVersion) and every deployment runs the new images;
• both users sign in; the administrator is Owner, the other user Admin and still lists projects, regions and users;
• env vars are encrypted in the database (eyJpdiI6…), the plaintext value is in no row, the region kubeconfig is encrypted, the API returns the values, secrets:encrypt finds nothing;
• the * token works; the projects:read token reads projects, gets env values hidden and a 403 on deploy; its stored scopes are translated;
• the function answered every probe during the upgrade and answers afterwards with its env;
• a variable moved to secret plus a new secret land in <slug>-secret-env, the Knative Service references them and contains no value, the function reads them after the redeploy;
• the region reports Knative Serving ready and its agent reconnected to the new Reverb;
• /api/branding returns the brand (names, palette, logos) and serves the logo from its data: URI;
• nothing of ghcr.io/runrsh is left (the image namespace up to 0.5.0): no workload of the cluster — control plane and region, Deployments, StatefulSets, DaemonSets, CronJobs, Knative Services — references it, no running pod apart from the ones region:reconcile keeps running on purpose (the sandbox node installers and the queue Redis), and no database row of the central and the tenant schemas (a CaaS project and, multi-tenant, a tenant-schema row of a ghcr.io/runrsh image are created before the upgrade); the region agent runs ghcr.io/brf-tech/runr/agent.

--edition=multi-tenant installs edition: multi-tenant (UI images with the -mt suffix) and, instead of the region and the function, creates a tenant that the platform administrator owns with a tenant admin, member and viewer. After the upgrade every one of them signs in to the tenant, GET /api/members shows owner / admin / member / viewer, the tenant has the custom role Member and the member's permissions include settings.manage, sso.manage and projects.deploy, the viewer lists projects but gets a 403 on POST /api/projects, and secrets:encrypt finds no plaintext in the central or the tenant schema.

Measured on a 20-core host with Kind 1.32 (main = 0.4.4 → this release, --reuse-values): the upgrade's migration job took 9 s, the new pods were ready 27 s after it started, the bootstrap job (agent check, region:reconcile with every step up to date, rbac:assign-missing) 6 s; the prober's 38 requests during the upgrade all answered 200.