runr_model_service (Resource)

A model service: a Hugging Face model (or a model of the organization's library) served by an engine (vLLM, TGI, TEI, RunR's transformers / diffusers server or a custom image) on CPU or GPU in a region. online services download the model from the Hub into the region's model volume; offline services serve the region's offline copy and have no egress. Settings left out are RunR's pick (engine, device, size, scaling); a change of a running service redeploys it. Bind it to projects with runr models bind (RUNR_MODEL_<ALIAS>_* variables).

Example Usage#

data "runr_region" "eu" {
  name = "eu-1"
}

# Embeddings on CPU: RunR picks the engine (TEI), size and scaling.
resource "runr_model_service" "embed" {
  hub_id    = "sentence-transformers/all-MiniLM-L6-v2"
  region_id = data.runr_region.eu.id
  min_scale = 1 # keep one instance warm
}

# Chat on a GPU, served from the region's offline copy (no egress).
resource "runr_model_service" "chat" {
  name      = "chat"
  hub_id    = "Qwen/Qwen2.5-7B-Instruct"
  revision  = "main"
  region_id = data.runr_region.eu.id
  mode      = "offline"
  device    = "gpu"
  engine    = "vllm"
  gpu = {
    count      = 1
    memory_min = 24000
  }
  args    = ["--max-model-len", "8192"]
  timeout = "45m"
}

output "chat_openai_base" {
  value = runr_model_service.chat.openai_base
}

Schema#

Required#

• region_id (String) Region id (data.runr_region). Changing it creates a new service.

Optional#

• args (List of String) Arguments appended to the engine's (the only arguments of a custom image).
• cpu (String) Kubernetes CPU quantity (default: from the model size).
• device (String) cpu or gpu (default: RunR's pick for the model).
• engine (String) vllm, tgi, tei, transformers, diffusers or custom (default: RunR's pick for the task and device).
• env_vars (Map of String, Sensitive) Extra environment variables (stored encrypted); they win over RunR's.
• gpu (Attributes) GPUs of a gpu service (default {count = 1}). (see below for nested schema)
• health_path (String) Readiness path of a custom image (default: TCP).
• hub_id (String) Hugging Face Hub id (organization/name); the model is added to the library on first use. Exclusive with model_id.
• image (String) Image instead of the engine's (a mirror, a newer release; the image of a custom engine).
• max_scale (Number) Maximum instances (default 1).
• memory_mb (Number) Memory in MiB (default: from the model size).
• min_scale (Number) Minimum instances (default 0; 1 keeps one warm).
• mode (String) online (default; from the Hub, cached in the region) or offline (the region's copy, no egress).
• model_id (String) Id of a model of the library (uploaded and imported models). Computed for hub_id.
• name (String) Service name (default: from the model's name). Changing it creates a new service.
• port (Number) Port a custom image listens on (default 8080).
• revision (String) Hub branch, tag or commit of hub_id (default main); the model is pinned to the commit it points to.
• running (Boolean) Deploy the service (true, default) or keep it stopped. A failed service is deployed again on the next apply.
• scale_to_zero (Boolean) Scale to zero instances when idle (default true).
• served_name (String) Model name of the OpenAI API (default: the model's name).
• timeout (String) How long wait waits (Go duration; large models take a while the first time).
• timeout_seconds (Number) Request timeout in seconds (default 300).
• wait (Boolean) Wait until a deployed service is ready.

Read-Only#

• api (String) API of the engine: openai, tei, runr or custom.
• id (String) Model service id.
• openai_base (String) <url>/v1 for engines with OpenAI-compatible routes.
• service_name (String) Knative Service name.
• status (String) stopped, waiting_snapshot, deploying, ready or failed.
• status_message (String) Reason of the status, if any.
• url (String) Cluster-local URL (projects of the region reach it; bound projects get it as RUNR_MODEL_<ALIAS>_URL).

Nested Schema for gpu#

Required:

• count (Number) Number of GPUs.

Optional:

• memory_min (Number) Least GPU memory per GPU in MiB.
• type (String) GPU product (node label), e.g. NVIDIA A100-SXM4-80GB.

Import#

Import is supported using the following syntax:

The terraform import command can be used, for example:

# <model service id or name>
terraform import runr_model_service.chat chat