Regions

A region is one Kubernetes cluster that runs functions. The control plane reaches the cluster's API with the region's kubeconfig, installs what projects need there — Knative Serving with Traefik as its ingress, an image registry, Prometheus and Loki, the tracing backend and the runr-agent — and deploys every project of the region into it. The runr-agent runs in the region and connects back to the control plane over WebSocket: it reports metrics, pod sessions and events (monitoring.md).

This page describes what a cluster needs before it becomes a region, how to connect one, every setting of a region, region status, moving a region to another cluster, regions in multi-tenant installs and removing a region. The components RunR installs and how they interact are in architecture.md; the runr regions commands are in cli.md.

• How projects use regions
• Before you connect a cluster
• Connecting a region
• Region settings reference
• Region status
• Capacity, overcommit and dedicated workers
• Moving a region to another cluster
• Changing the base domain
• Regions in multi-tenant installs
• Removing a region
• API reference
• Artisan commands

How projects use regions#

Every project runs in one region, picked when the project is created (knative_region_id); a project can be moved to another region later. The region decides:

• where the project's workloads run: in the region's project namespace, which is named like the region (a region eu-2 runs its projects in the namespace eu-2);
• its address: an HTTP project answers at https://<project slug>.<region>.<base domain>. The base domain is the platform's (Administration → Platform, Base Domain; PATCH /api/settings with base_domain); <region>.<base domain> is the region's function domain;
• what the project can use: queue triggers need the region's Redis, builds (runtime plugins with a build step, WebAssembly functions) its registry, sandboxed runtimes its gVisor or Kata Containers, Wasm functions its WebAssembly support, GPU projects its GPU support, bindings its data services;
• whether the project is admitted: the region's status and capacity.

In the console, regions are under Administration → Platform → Knative Regions; the pencil of a region opens its tabs (General, Workers, Registry, Operator, Serving, Redis, Data services, Isolation, GPU, Migration, Supply chain). Changing a region needs regions.manage or the permission of the area (API reference).

Before you connect a cluster#

Kubernetes version#

RunR reads the cluster's version when it checks the kubeconfig and refuses a cluster older than Kubernetes 1.30. The minimum is the Helm value knative.minK8sMinor (environment variable KNATIVE_MIN_K8S_MINOR, default 30).

Knative and Kubernetes versions#

RunR installs Knative Serving through the Knative Operator. The Knative versions it offers for a cluster come from this table (App\Services\Helpers\KnativeVersionHelper::KNATIVE_K8S_COMPAT):

The Serving step offers the up to five newest Knative versions that the cluster supports and the Knative Operator chart repository has (GET /api/knative-regions/{id}/compatible-versions, runr regions knative versions), the newest first and preselected: a Kubernetes 1.32 cluster, for example, is offered Knative 1.20, 1.19, 1.18, 1.17 and 1.16. The operator's chart version is picked the same way.

Nodes#

• Container runtime. The nodes reach RunR's registry through a node mirror (DaemonSet runr-registry-mirror) that writes containerd's hosts.toml for it. On k3s and rke2 containerd reads it as it is. On plain containerd the mirror also sets config_path in config.toml, and containerd has to be restarted once: the mirror's log says so, and php artisan registry:rolling-restart <region-id> restarts it node by node (Artisan commands).
• Privileged pods. RunR runs privileged DaemonSets on the nodes — the registry's node mirror and a short-lived image pre-pull in the registry namespace, and for sandbox runtimes and WebAssembly their installers in runr-sandbox — and cAdvisor in observability with the host's PID namespace and host paths. A cluster that enforces Pod Security baseline or restricted on every namespace has to exempt these namespaces.
• Sandbox runtimes. Kata Containers needs containerd (plain, k3s, rke2) or CRI-O nodes and /dev/kvm (kata-hypervisors.md). WebAssembly support: wasm.md; GPU nodes: ai.md.
• Dedicated workers are optional: without them projects run on every schedulable node (Dedicated workers).

Storage#

RunR's own components ask for these volumes. Where no StorageClass is named the cluster's default class is used, so the cluster needs a default StorageClass unless every class is named.

Before it installs the registry on a PVC, RunR creates a test volume of the chosen class and waits 60 seconds for it to bind. A registry on S3 needs the bucket instead (endpoint, bucket, region, access key, secret key).

Ingress, external address and DNS#

The Serving step installs Traefik as Knative's ingress (chart traefik/traefik 40.3.0, image ghcr.io/brf-tech/runr/reqs/traefik:v3.7.4 or the Helm value knative.traefikImage) in the Knative Serving namespace. Its Service is of type LoadBalancer and listens on ports 80 and 443: the cluster needs a load-balancer implementation that gives the Service an external IP address or hostname. RunR reads that address as the region's external address (external_ip), at the end of the Serving install and with every status check.

Then create one DNS record for the function domain:

The region's General tab shows the record under DNS Configuration, the API as dns_instruction of GET /api/knative-regions/{id}. Every minute the status check resolves dns-check.<region>.<base domain> from the control plane and shows whether it reaches the external address (dns_resolved): for an IP address, the address is among the answers (A or AAAA, through any CNAME); for a load balancer's host name, the record is a CNAME to it or resolves to one of its addresses. The Serving install derives the function domain from the base domain; a base domain set later reaches the regions without a function domain when it is saved, or with the next status check. A changed base domain moves the regions with a migration (Changing the base domain).

HTTPS on the function domain is answered with the region Traefik's default certificate until the region has a certificate of its own (Function TLS). A load balancer in front of Traefik that sends PROXY protocol or X-Forwarded-For has to be named in ingress_trusted_ips (General).

Images and Helm charts#

RunR runs Helm in the control plane: for every install the control plane fetches the chart from its repository and applies it through the cluster's API.

The control plane also fetches the Knative Serving CRDs from the Knative release on github.com. The nodes pull the images: RunR's from ghcr.io/brf-tech/runr (the agent, the mirrors of Traefik, Redis, the Collector and Tempo, the function wrapper and the runtimes), the Kaniko build image, cAdvisor from gcr.io, and the images of the charts above from their own registries. The internal registry is a pull-through cache of ghcr.io and Docker Hub, and its node mirror sends the nodes' Docker Hub pulls through it.

Images of your own reach a region without outside access by upload through RunR (images.md), model weights the same way (huggingface.md).

Network reachability#

The agent's <base> is, in this order: the region setting agent_ws_url; RunR's in-cluster gateway when the region is the cluster RunR runs in; the environment variable RUNR_AGENT_WS_URL (Helm extraEnv); the console URL APP_URL (https:// becomes wss://). A cluster that cannot reach the console's public address needs agent_ws_url (General). The agent authenticates with a token of its region (monitoring.md). Request replay captures go to Helm observability.replay.ingestUrl instead when it is set (observability.md).

The API answers only for the console's host, except under /api/internal/: these callbacks from the regions are also answered for the hosts of the addresses RunR hands out — the in-cluster gateway, every region's agent_ws_url and app_url, RUNR_AGENT_WS_URL, observability.replay.ingestUrl and WORKFLOWS_PUBLIC_API_URL (a changed region setting within a minute). Any other host gets 400.

The kubeconfig#

RunR uses the kubeconfig's current context and reads from it the cluster's server and certificate-authority-data (or insecure-skip-tls-verify) and the user's token, or client-certificate-data and client-key-data: the kubeconfig carries its credentials in these fields. RunR stores it encrypted, never returns it in API responses and hands it out only through the kubeconfig download (security.md).

The kubeconfig is RunR's only access to the cluster, also for a region on the cluster RunR runs in: the control plane's own service account only reads the kube-system namespace to recognise that cluster. Give the kubeconfig cluster-admin rights. An install creates namespaces, CRDs (Knative's, Traefik's, kube-prometheus-stack's), ClusterRoles and ClusterRoleBindings (the runr-agent's, cAdvisor's, Traefik's Knative networking role and the charts' own), a PriorityClass (observability-critical), an IngressClass, RuntimeClasses, node labels and taints, privileged DaemonSets and Helm releases in several namespaces, and RunR uses the service proxy and pods/exec (deployment shells, sandboxes).

k3s, rke2 and local clusters#

• k3s's own Traefik. When an IngressClass traefik belongs to another controller, the region's Traefik serves the class traefik-knative and leaves the existing one alone.
• Containerd paths. The registry's node mirror finds the containerd configuration of k3s and rke2 (/var/lib/rancher/{k3s,rke2}/agent/etc/containerd) and needs no containerd restart there.
• A cluster on the same machine. A kubeconfig whose server is 127.0.0.1 or localhost and that RunR cannot reach that way is rewritten to host.docker.internal, without the CA check, for a control plane that runs in containers next to a local cluster.
• RunR's own cluster. RunR recognises the cluster it runs in by its kube-system namespace: the agent then connects to the in-cluster gateway, and the console's certificate can serve the function domain (Function TLS).

Connecting a region#

In the console#

Administration → Platform → Knative Regions → + Add Region opens a wizard of seven steps. It needs regions.manage and, for the registry step, registry.manage. Once a step is done, the steps before it are read-only.

1. Connection. The region name — lowercase letters, digits and hyphens, starting with a letter and ending with a letter or digit, at most 63 characters, unique — and the kubeconfig (upload or paste), with the Tenancy module the region mode (Platform users), and with the Billing module the region's billing type — pay as you go or fixed resources (Billing). RunR connects, checks the Kubernetes version and that the cluster has no namespace of that name yet. Nothing is created in this step.

2. Workers & Registry. The worker nodes to dedicate (optional), the registry and, under Existing monitoring (optional), a Prometheus and a Loki already running in the cluster.

   • Internal Registry (Install Zot) on a PVC (StorageClass, size) or on S3 (endpoint, bucket, region, access key, secret key), or External Registry (address, optional username and password).
   • Existing monitoring takes in-cluster Service URLs (http(s)://<service>.<namespace>.svc[.cluster.local]:<port>[/path]) and a Loki tenant ID; RunR then installs no Prometheus or Loki of its own for that component. With the Tenancy module only Platform users, and tenant owners for their own region, may set them.

   Save & Next creates the region: RunR tests the storage (or the external registry, from a pod), installs Zot (release runr-zot in runr-registry, chart zot/zot 0.1.125) with its node mirror, pre-pulls common images on the nodes, creates the region and its project namespace, applies the worker selection and queues RunR's managed Redis.

3. Operator. RunR first adopts a Knative Operator and Serving already running in knative-operator and knative-serving. Install Operator installs the Knative Operator (chart knative-operator/knative-operator in the chosen namespace, default knative-operator) and the Knative Serving CRDs.

4. Serving. The namespace (default knative-serving) and the Knative version. The install applies, in this order: the project namespace's ResourceQuota and LimitRange when an organization ceiling is set (architecture.md); Traefik; the KnativeServing resource with Traefik as ingress; the function domain <region>.<base domain>; the external address; Prometheus (release knative) and Loki (loki-stack) unless existing ones were given, with a PriorityClass and PodDisruptionBudgets; the runr-agent; and the tracing backend (observability.md).

5. Redis. The Redis of queue-triggered projects: RunR's managed Redis queued in step 2 (Deployment runr-redis in the project namespace), or an external one (Redis).

6. Sandbox. The Isolation tab: Kata Containers, gVisor, WebAssembly support and the RuntimeClass of each workload kind (Sandbox runtimes and WebAssembly).

7. Migration. The region's move to another cluster (Moving a region to another cluster). Finish closes the wizard.

The Operator and Serving steps can be skipped; a region without Serving runs no HTTP projects. Then create the DNS record that the General tab shows (Ingress, external address and DNS).

With the CLI#

runr regions check eu-2 --kubeconfig eu-2.yaml          # version, nodes, StorageClasses, the namespace
runr regions create eu-2 --kubeconfig eu-2.yaml --registry internal --storage-class fast-ssd --wait
runr regions knative install eu-2 operator --wait
runr regions knative install eu-2 serving --wait

Every flag is in cli.md.

With the API#

POST /api/registry-setup/start creates a region together with its registry, as the console does. It answers 202 with a setup token and runs in the background:

jq -n --rawfile kc eu-2.yaml '{name: "eu-2", kubeconfig: $kc, nodes: [],
    registry_mode: "internal", storage_type: "pvc",
    pvc_config: {storageClass: "fast-ssd", size: "10Gi"}}' |
  curl -sS -X POST "$API/registry-setup/start" -H "Authorization: Bearer $TOKEN" \
    -H 'Content-Type: application/json' -d @-
# {"setup_token":"…","setup_id":"…"}

curl -sS "$API/registry-setup/$SETUP_TOKEN/status" -H "Authorization: Bearer $TOKEN"
# {"status":"ready","phase":"done","region_id":"01J…",…}     status: pending, installing, ready or error

curl -sS -X POST "$API/knative-regions/$REGION/install-operator" -H "Authorization: Bearer $TOKEN"
curl -sS -X POST "$API/knative-regions/$REGION/install-serving" -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{"version": "1.20"}'

The status of a setup is kept for 30 minutes; a setup in error starts again with POST /api/registry-setup/{token}/retry and the same body. POST /api/knative-regions takes name, kubeconfig, nodes, is_active (false: the region is created inactive) and the Redis and monitoring fields and creates the region at once; given registry_mode and the registry fields (registry_storage_type, registry_storage_config, or registry_address, registry_username, registry_password) it then sets the registry up in the background like POST /api/knative-regions/{id}/registry (Registry). Both refuse (422) a cluster below the minimum Kubernetes version and a name whose namespace exists on the cluster.

Region settings reference#

One table per area. "Console" names the region's tab (Administration → Platform → Knative Regions → the region's pencil); the API paths are under /api.

General#

Set them in the console (General → Settings), with runr regions set REGION --set KEY=VALUE (cli.md) or with the API, which changes only the keys it is sent:

curl -sS -X PUT "$API/knative-regions/$REGION/settings" -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"request_timeout": 600, "agent_ws_url": "wss://runr.internal.example.com",
       "ingress_trusted_ips": ["10.0.0.0/24"]}'

Request timeout#

A project's timeout is the timeoutSeconds of its Knative revision: Knative ends a request that runs longer with 504. For a request to run that long, what is in front of it waits as long:

• Traefik. The region Traefik's entry points web and websecure close a request after their read and write timeouts. Both are the longest request_timeout of the regions on the cluster plus 10 seconds (at least 120 seconds), so Knative's own answer arrives first.
• Knative. Knative refuses a revision whose timeoutSeconds exceeds its max-revision-timeout-seconds (600 seconds by default). With a request_timeout above 600 seconds RunR raises it to that value (spec.config.defaults of the KnativeServing resource); it is never lowered.

A changed request_timeout reaches both right away (a Helm upgrade of the region's Traefik in the background — its pods roll once — and a merge patch of the KnativeServing resource); region:reconcile brings existing regions there with every upgrade.

Workers and capacity#

Registry#

The registry is chosen when the region is created (wizard step 2, runr regions create --registry, POST /registry-setup/start, or POST /knative-regions with the registry fields) or set up later in the region's Registry tab, with runr regions registry set REGION --registry internal|external (cli.md) or with POST /knative-regions/{id}/registry (registry.manage), which takes the registry fields of POST /registry-setup/start (registry_mode, storage_type, pvc_config, s3_config, external_address, external_username, external_password). The setup tests the storage or the external registry from the cluster and installs Zot in the background (registry_status installing, then ready, or error with the reason in registry_status_message). An external registry is replaced this way; an internal one is removed first, because its volume holds the region's images.

runr regions registry rm REGION or DELETE /knative-regions/{id}/registry (registry.manage) removes the registry: for internal the node mirror, the Helm release and the namespace (with its volume) when RunR created it; for external RunR forgets it.

Knative Operator and Serving#

Installing adopts what already runs on the cluster — another region's Knative on the same cluster, or one installed before — with every install path, the full install (POST /knative-regions/{id}/install-knative, runr regions knative install REGION) included: an operator whose Deployments are available (and the KnativeServing CRD registered) is recorded as the region's and not installed again, a Ready KnativeServing is adopted and not applied again. An operator install that does run (a cluster without a working operator) first removes the previous operator release; it keeps the Knative Serving CRDs while Knative Services exist on the cluster.

Upgrade moves Knative Serving to the next minor version the cluster supports — Knative upgrades one minor version at a time, so other versions are refused (422). RunR moves the Knative Operator's Helm release to that minor's chart first when it is older, applies the Serving CRDs of the version and sets the KnativeServing resource's spec.version; the operator then rolls Knative Serving's components while the region's Knative Services keep running. The Serving status is installing until the operator reports the new version Ready. Regions on one cluster share Knative Serving, so it is upgraded for all of them.

Uninstall Serving (POST /knative-regions/{id}/uninstall-serving) deletes the KnativeServing resource, Traefik — which stays while it serves other Ingresses, such as the console on the same cluster, unless remove_traefik is set — and, once Traefik is gone, the Serving namespace when RunR created it. Prometheus, Loki, the runr-agent and the tracing backend stay. Uninstall Operator (POST /knative-regions/{id}/uninstall-operator, once Serving is gone) removes the operator and the Knative Serving CRDs, and with them every Knative Service left on the cluster. POST /knative-regions/{id}/uninstall-knative (runr regions knative uninstall REGION) removes Serving and the operator release in one go and sets both components to not_installed.

Redis#

The Redis of queue triggers. A new region gets RunR's managed Redis unless it is created with another mode (redis_mode, --redis).

Change it in the console's Redis tab (Auto Installed or External Redis, Test Connection, Remove Redis), with runr regions redis set|test|rm (cli.md) or with POST /knative-regions/{id}/redis (mode, address, password), POST /knative-regions/{id}/test-redis (a redis-cli PING from a pod in the cluster) and DELETE /knative-regions/{id}/redis. Removing deletes RunR's managed Redis and the region's Redis settings.

Observability#

PUT /knative-regions/{id}/monitoring takes prometheus_url, loki_url and loki_org_id (the keys it is sent; who may set them is as at creation): a URL points the region at that in-cluster Prometheus or Loki, null back at RunR's own, which is installed right away when the region's Serving is ready (else with the Serving install). A component switched to an existing instance keeps RunR's own release running — regions on the same cluster may still use it; remove it with helm uninstall when none does. GET /knative-regions/{id} shows the region's choice under monitoring.

Removing the tracing backend (DELETE /observability/regions/{id}/tracing, the Tracing tab's Remove) sets the region's tracing mode to off, so region:reconcile does not install it again; choosing a mode brings it back.

php artisan cluster:patch-observability [REGION-ID] applies RunR's Prometheus and Loki values again; observability.md describes the install.

Sandbox runtimes and WebAssembly#

The console's Isolation tab (wizard step 6); sandbox.manage.

POST /knative-regions/{id}/uninstall-sandbox with runtime and DELETE /knative-regions/{id}/wasm remove them; both are refused while pods still use the runtime, and the mappings that used a removed Kata Containers or gVisor go back to runc.

The Sandbox API has its own settings per region (PUT /sandbox-regions/{id}, sandboxes.manage, or runr sandbox regions set): whether the region offers sandboxes (enabled, default on), allowed_classes (default: every installed class), default_class (default: the strongest allowed), allow_runc (default off), default_ttl_seconds (default 900), max_ttl_seconds, max_cpu_m (50–4000), max_memory_mb (64–8192) and max_disk_mb (64–10240) (sandbox-api.md, cli.md).

GPU and model storage#

The console's GPU tab; regions.gpu.manage. Every setting is described in ai.md and huggingface.md.

Function TLS#

The certificate the region's Traefik serves the function domain with. Console: General → Function-domain TLS (once Serving is ready); runr regions function-tls set; PUT /knative-regions/{id}/function-tls (regions.manage).

RunR writes the Secret runr-function-tls and Traefik's default TLSStore in the Knative Serving namespace and records the expiry (function_tls_expires_at). A default TLSStore that RunR did not create is replaced only with replace_existing: true.

Remove certificate (runr regions function-tls rm, DELETE /knative-regions/{id}/function-tls) switches function TLS off: RunR deletes its Secret and its default TLSStore (a TLSStore it did not create stays), and the region's Traefik answers HTTPS with its own default certificate again.

Data services#

The console's Data services tab; regions.data_services.manage; POST/DELETE /knative-regions/{id}/data-services/{kv|object|postgres}. Everything runs in runr-data (bindings.md).

Removing a service is refused while bindings use it, and deletes its data.

Supply chain and data location#

The console's Supply chain tab; regions.supply_chain.manage; PUT /knative-regions/{id}/supply-chain. SBOMs, scanning, the policy gate, signing and the admission policy of the region are described in supply-chain.md. The same call sets the region's data location (data_location, up to 64 characters: letters, digits, spaces, _, . and -, starting with a letter or digit; the tab's Data residency section): the place data residency policies match (supply-chain.md), and the place a migration of the region has to stay in.

Billing#

With the Billing module on, the region's Billing tab (billing.view; changes billing.manage) and Settings → Billing → Regions & prices show and set:

A region shared by the platform with a tenant keeps the platform's type and prices; the tenant adds its markup when it accepts the offer (Billing with tenancy).

Skupper and demo projects#

• Skupper. Private database connections of bindings run over Skupper v2, which the cluster's operators install. GET /knative-regions/{id}/skupper shows whether the cluster runs it, POST /knative-regions/{id}/skupper/site creates the site runr in the project namespace (cli.md).
• Demo projects. General → Demo Projects (POST/DELETE /knative-regions/{id}/demo-projects, runr regions demo install|rm) deploys four demo projects — a frontend, a backend, a queue-triggered e-mail sender and a scheduled statistics job — once Serving, Redis and the registry are ready.

Region status#

Every region has a status (region_status): active, inactive or maintenance.

The status changes in the console (General → Region Status), with runr regions set REGION --set status=maintenance or with PUT /api/knative-regions/{id} and {"region_status": "maintenance"}.

A region that is not active runs a small status responder in the Knative Serving namespace (Deployment and Service runr-status-responder, 10m CPU / 16 MiB requested) and an IngressRoute runr-region-block-<region> for the region's Traefik — one per region, as the regions of a cluster share its Knative Serving namespace. The IngressRoute matches, on HTTP and HTTPS and with the highest priority, every host the region answers on:

• every host under its function domain (*.<region>.<base domain>) and, during a base-domain migration, under the domain it moves to;
• the project custom domains routed on the region;
• the domains of the region's Knative Services, such as a tenant's own base domain, and the domain a tenant's base domain moves to (Organization domain).

The responder answers 404 or 503 as above. The project pods keep running, so the region answers again as soon as it is active: that change removes the IngressRoute and its Middleware.

The block is applied when the status changes, when Knative Serving of a region that is not active becomes ready (installed or adopted later) and when the hosts it covers change (a base-domain migration); region:reconcile applies it to every region with Knative Serving ready that is not active — platform regions and, with the Tenancy module, every tenant's own regions, each in its tenant's context — and removes a block left on an active one. A region created inactive (runr regions create --inactive, is_active: false) has the status inactive; the upgrade to 0.5.8 gives rows written before with is_active: false the status inactive, in the central schema and in every tenant schema.

A region that is not active is also left out of the per-minute metric snapshots, the daily log retention and the region maintenance commands that act on every active region (Artisan commands).

Capacity, overcommit and dedicated workers#

The region capacity is what the region's nodes leave for RunR workloads; admission splits it into an always-on budget and a FaaS overcommit budget with one overcommit factor per resource (faas_cpu_overcommit, default 3, and faas_memory_overcommit, default 1.5, each 1–10). How the capacity is measured, what counts against each budget and how the defaults were chosen: Region capacity and admission.

Dedicated workers reserve nodes for project workloads with the taint runr.dev/dedicated=projects:NoSchedule and a matching label; with workers selected, only those nodes count towards the capacity. What moves onto the workers and what stays on the other nodes: Dedicated workers.

Moving a region to another cluster#

A region migration keeps the region — its id, name, settings, function domain and projects — and replaces its cluster. Start it in the console (Migration → Upload New Kubeconfig, Start Migration), with runr regions migration start REGION --kubeconfig FILE (cli.md) or with the API:

jq -n --rawfile kc new-cluster.yaml '{kubeconfig: $kc}' |
  curl -sS -X POST "$API/knative-regions/$REGION/migrate" -H "Authorization: Bearer $TOKEN" \
    -H 'Content-Type: application/json' -d @-
curl -sS "$API/knative-regions/$REGION/migration-status" -H "Authorization: Bearer $TOKEN"

A region that is already moving answers 422. A region with a data location (data residency) moves only within it: the request names the new cluster's location (data_location in the API, --data-location in the CLI), and it has to be the region's.

The move runs as a queued job (MigrateRegionJob, at most 30 minutes) and sets migration_status as it goes:

1. pending: RunR saves the current kubeconfig and external address, checks that the new cluster answers and switches the region to the new kubeconfig (in_progress).
2. The dedicated workers: the selected nodes that exist on the new cluster get the label and the taint; when none of them exists there the selection is cleared (projects run on every schedulable node) and is made again in the Workers tab.
3. It installs on the new cluster the Knative Operator and Knative Serving with everything a Serving install brings (Traefik, Prometheus and Loki, the runr-agent, the tracing backend); an operator or a Ready KnativeServing already running there is kept. Then the managed Redis (an in_cluster or external Redis keeps its address), the internal registry (a Zot already serving in the registry namespace is kept; the region's registry address follows the new cluster) and Kata Containers and gVisor where the region had them ready.
4. What else the region had: WebAssembly support (on the selected nodes that exist there, else on every eligible node), GPU support with its settings, the data services RunR runs (managed, and KV on the region's Redis; an external object store needs nothing on the cluster), the admission policy, function TLS (the console's certificate, or the uploaded one copied from the old cluster's Secret), the Skupper site of the project namespace when projects use private database connections, and the status block of a region that is not active. A step that fails is logged and the move goes on.
5. It adds the region's function domain to Knative's domain configuration on the new cluster.
6. It deploys again what ran on the old cluster: for every project with a live deployment, the version that deployment runs, with its deploy overrides. The deploy checks of every deploy apply (a stopped project is refused, and so is one in a fixed region whose allocation it would exceed; the reason is logged, and the refused project's deployments are shut down with the reason as their message — nothing of it runs on the new cluster, so nothing is metered for it until it is deployed again). A stopped project moves with the region and stays stopped — it is deployed on the new cluster when it is resumed — and a project that runs nothing is not deployed.
7. dns_check: point the wildcard DNS record of the function domain at the new cluster's external address (external_ip and dns_instruction of the region, GET /api/knative-regions/{id}).

The data of the data services stays on the old cluster: the new cluster's are empty, and their content is moved with the services' own tools (a dump of the Postgres databases, a copy of the buckets). A private database connection's Skupper link is issued by the database side (an access token) and is created again for the new cluster (bindings.md).

The status check resolves dns-check.<function domain> every minute (Ingress, external address and DNS); once it reaches the new address the move is completed. It also completes 48 hours after it reached dns_check, or when you complete it (Migration → Complete Migration, runr regions migration complete, POST /api/knative-regions/{id}/complete-migration; only in dns_check). Completing forgets the old kubeconfig and address; RunR removes nothing from the old cluster. While a move runs, a banner in the console counts down the 48 hours.

A region whose cluster no longer answers opens in the console on its Migration tab, so it can be moved to a new cluster from there. The tab asks for the new cluster's data location when the region has one.

A move in dns_check or failed can be rolled back (Migration → Rollback Migration, runr regions migration rollback, POST /api/knative-regions/{id}/rollback-migration): the region gets its old kubeconfig and external address back and every project that runs is deployed there again, the same way as in step 6. What the move installed on the new cluster stays there.

Changing the base domain#

The platform's base domain (Administration → Platform, Base Domain; PATCH /api/settings with base_domain) gives every region that follows it its function domain <region>.<base domain>. A region follows the base domain while its function domain (knative_domain of its settings) is <region>.<base domain> or not set yet; a region with a function domain of its own keeps it whatever the base domain is.

• The first base domain goes to every region with Knative Serving that has no function domain: Knative's domain configuration and the region's knative_domain.
• A changed base domain starts a migration on every region that follows the old one (domain_migration_status dns_check). The region keeps its function domain — its projects' addresses do not change yet — and also answers on <region>.<new base domain> (domain_migration_target of GET /api/knative-regions/{id}): a small proxy in the project namespace (Deployment, Service and IngressRoutes runr-domain-alias, 10m CPU / 32 MiB requested per pod, two pods) sends every <project>.<region>.<new base domain> to the project's Knative Service. Create the wildcard DNS record of the new domain, pointing at the same external address.
• Completion. The status check (every minute) and the scheduler (hourly) check dns-check.<region>.<new base domain>; once it reaches the region's external address the new domain becomes the function domain: Knative's domain configuration and the KnativeServing resource name it, the region's knative_domain is it — project URLs, function TLS, the status block and request replay follow — and the old domain is no longer served. Without that DNS answer both stay served. Migration → Complete domain migration, runr regions migration complete REGION --domain or POST /api/knative-regions/{id}/complete-domain-migration completes it at once (for DNS the control plane resolves differently from the clients).
• Rollback. Setting the base domain back to the old one during the migration ends it: the region keeps its function domain and the new one is no longer served. A third base domain moves the second address to <region>.<third base domain>; the function domain stays.
• A cleared base domain removes the function domain from the regions that follow it.

During the migration HTTPS on the new domain is answered with the certificate the region Traefik serves by default — the function certificate when one is set. An uploaded certificate whose names cover both domains (*.<region>.<old base domain> and *.<region>.<new base domain>) serves both; one of the old domain only is replaced after the completion. Custom domains with a CNAME to a project's default host (<project>.<region>.<old base domain>) point it at the new default host before the completion. With the Tenancy module a tenant's own base domain is handled by the tenant's settings and moves the same way — on the platform regions through an alias of the tenant's own, on its own regions through this migration (Organization domain).

Regions in multi-tenant installs#

With the Tenancy module (tenancy.md) a region is either a platform region or a tenant-owned region.

Platform regions are connected by Platform users outside any tenant and live in the central schema. A Platform user assigns a platform region to a tenant with a CPU and memory quota; the assignment copies the region into the tenant's schema without its kubeconfig, and RunR reads the kubeconfig from the platform region whenever it acts on the cluster for the tenant. Tenant users see only the regions assigned to their tenant, as a summary (name, mode, their own project count, the GPU summary) without cluster, component or capacity details; the kubeconfig download and the region's capacity budgets answer 403. Platform regions are read-only for Tenant users. With the Billing module on, an assignment is an offer: the region is copied into the tenant only after a tenant administrator accepted it with a markup (Shared regions with billing).

A quota is the tenant's share of the region capacity: on a managed region the quotas of all tenants add up to at most the region capacity (and the organization ceiling, Helm quota.cpu / quota.memoryGiB, when one is set, architecture.md); an assignment or change above that answers 422. A quota of 0 sets no limit for that resource and does not count towards the sum. When the request leaves a quota out and an organization ceiling is set, the quota is the ceiling or what is still free on the region, whichever is smaller. How admission splits a quota into the always-on and the FaaS overcommit budget: Region capacity and admission.

Region mode (region_mode) decides whether quotas apply on a platform region:

The mode is set with region_mode on POST /api/knative-regions, POST /api/registry-setup/start (the console wizard's first step) or PUT /api/knative-regions/{id}; the console shows it on the region's General tab. Only platform administrators set it: from anyone else the field is ignored (the region keeps its mode, a new one is managed). Platform users are not held to tenant quotas on any region.

Tenant-owned regions are connected inside a tenant by its users with regions.manage (owners and admins by default, permissions.md); they live in the tenant's own schema and are never assigned to other tenants. Every route that changes a region or hands out its kubeconfig checks who may do it (region.mutable):

region:reconcile brings the tenant-owned regions up to date too: the cluster steps for the active ones, the status block for every one with Knative Serving ready.

Removing a region#

Removing a region — the trash button in the console's region list, offered once the region has no projects, runr regions rm REGION or DELETE /api/knative-regions/{id} — deletes RunR's record of the region. Nothing is removed from the cluster. Everything RunR installed keeps running there, the projects' Knative Services included, and they keep answering on the function domain while DNS points at the cluster. The region's runr-agent is turned away from then on: its token belongs to a region RunR no longer knows.

In RunR's database the region's projects stay, without a region; the region's metric snapshots, deployment sessions, gateways and log retention setting are deleted with it (with the Tenancy module also its tenant assignments), and its bindings stay with the region cleared. Closed bills keep the region's name in their line items.

To take a cluster out of service completely:

1. Delete the region's projects, or move them to another region.

2. Remove what RunR installed, from the region's pages in the console or with the API, while the region still exists:

   What	Console	API
   Demo projects	General → Demo Projects	DELETE /knative-regions/{id}/demo-projects
   Data services (after their bindings)	Data services	DELETE /knative-regions/{id}/data-services/{service}
   GPU support	GPU → Remove	POST /knative-regions/{id}/gpu/uninstall
   WebAssembly, Kata Containers, gVisor	Isolation	DELETE /knative-regions/{id}/wasm, POST /knative-regions/{id}/uninstall-sandbox
   Admission policy	Supply chain	DELETE /knative-regions/{id}/supply-chain/admission
   Function TLS	General → Remove certificate	DELETE /knative-regions/{id}/function-tls
   Tracing backend	Tracing tab	DELETE /observability/regions/{id}/tracing
   Redis	Redis → Remove Redis	DELETE /knative-regions/{id}/redis
   Knative Serving and Traefik, then the operator	Serving → Uninstall Serving, Operator → Uninstall Operator	POST /knative-regions/{id}/uninstall-serving (remove_traefik), then …/uninstall-operator
   Registry	Registry → Remove registry	DELETE /knative-regions/{id}/registry (runr regions registry rm)

3. Remove the region.

4. Remove what is left with kubectl and helm: the observability namespace (Helm releases knative and loki-stack, cAdvisor, the runr-agent and its Secret runr-agent-token), the project namespace (the region name), whichever of runr-registry, runr-sandbox, runr-sbx-*, runr-data, cnpg-system, runr-supply-chain, cosign-system and runr-gpu remain, the ClusterRoles and ClusterRoleBindings runr-agent and cadvisor-runr, the PriorityClass observability-critical, and on the nodes the labels runr.dev/dedicated, node-role.kubernetes.io/knative-worker, runr.dev/kvm and runr.dev/wasm*, the taint runr.dev/dedicated and the hosts.toml files the registry's node mirror wrote into containerd's configuration.

API reference#

Every route is under /api and needs a bearer token or a console session. The permissions are those of permissions.md (api/config/route_permissions.php); a|b means either. Every route that changes a region — every POST, PUT and DELETE below except the four kubeconfig previews and the registry tests — and the kubeconfig download also pass the region mutability check (Regions in multi-tenant installs).

Region

Knative

Redis, registry, Skupper

Sandbox runtimes, WebAssembly, GPU

Data services, observability, supply chain

With the Tenancy module, the tenant assignment routes are listed in Regions in multi-tenant installs.

Artisan commands#

Run them in the API container (php artisan …; with Compose docker compose exec api php artisan …).