Model gateway (LiteLLM)

The model gateway is optional, and most installs start without it. Without the gateway, users bring their own agent subscriptions: they sign in to Claude or Codex directly from the desktop app, exactly as they would on their own machine. Nothing about the base install requires the gateway.

Add the gateway when you want, as an admin:

• Central provider keys. Your Anthropic, OpenAI, or xAI keys live in one place (the LiteLLM proxy container), not on laptops and never inside sandboxes.
• Per-user short-lived keys. The Proliferate server mints a short-lived virtual key for each user; agents and sandboxes only ever see the public gateway URL and that virtual key. Your real provider credentials never reach them.
• Budgets. Spending limits per user and per organization, enforced at the gateway.

How it works

The gateway is LiteLLM, and the standard Docker Compose stack can run it for you: litellm and its own Postgres database (litellm-db) ship as an optional service group in docker-compose.production.yml, off by default and turned on with a Compose profile. The image is Proliferate's own build (ghcr.io/proliferate-ai/proliferate-litellm), preloaded with the model list Proliferate's harnesses expect (current Claude, GPT, and Grok models, each mapped to your own provider key). You supply provider keys and a master key; the Proliferate server uses the master key to mint and revoke per-user virtual keys, and sandboxes call the gateway's public URL with those virtual keys. The master key itself stays between the Proliferate server and LiteLLM, and is never sent to sandboxes.

Prefer to run LiteLLM yourself (a different host, a managed instance, or a custom model list) instead of the bundled service? See external LiteLLM instead below. The Proliferate server only cares about the base URL, public URL, and master key; it does not care who runs the process behind them.

Set up

AGENT_GATEWAY_ENABLED=true
 
# Provider keys the bundled LiteLLM container calls out with. Set the ones
# your team uses; an unset key just means that provider's models 400.
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
XAI_API_KEY=xai-...
 
# LiteLLM's own master key, and Postgres password for its key/spend store.
# Generate both with `openssl rand -hex 32`.
LITELLM_MASTER_KEY=sk-<random-value>
LITELLM_POSTGRES_PASSWORD=<random-value>
 
# The Proliferate server authenticates to LiteLLM's admin API with this; it
# must be the exact same value as LITELLM_MASTER_KEY above.
AGENT_GATEWAY_LITELLM_MASTER_KEY=sk-<same-random-value>
 
# Internal Docker network address of the bundled service (the Compose
# service name is `litellm`, listening on port 4000).
AGENT_GATEWAY_LITELLM_BASE_URL=http://litellm:4000

{$SITE_ADDRESS} {
  encode zstd gzip
  reverse_proxy api:8000
}

llm.company.com {
  encode zstd gzip
  reverse_proxy litellm:4000
}

AGENT_GATEWAY_LITELLM_PUBLIC_BASE_URL=https://llm.company.com

docker compose --env-file .env.runtime -f docker-compose.production.yml \
  --profile agent-gateway pull
docker compose --env-file .env.runtime -f docker-compose.production.yml \
  --profile agent-gateway up -d

Environment variable reference

Bundled-service-only (skip these in external mode):

Budgets

AGENT_GATEWAY_DEFAULT_USER_BUDGET_USD (default $5) caps what a single user can spend through their personal virtual key. Once a shared sandbox or automation run authenticates as the organization, that traffic is capped separately by AGENT_GATEWAY_DEFAULT_ORG_BUDGET_USD (default 0, meaning uncapped). These are lifetime caps with no automatic reset: when a key hits its budget, the gateway rejects further model calls until an admin raises that key's budget in LiteLLM directly. Provider spend itself is whatever your provider accounts bill; LiteLLM's own logs are the place to watch usage per key.

The bundled model list

The bundled image ships a fixed, checked-in model list mapping model names (current Claude, GPT, and Grok releases) to your provider keys; it is not a general-purpose "add any model" proxy. That covers every model Proliferate's supported harnesses ask for. If you need a different provider (Azure OpenAI, Bedrock, a self-hosted model server) or a custom model list, run LiteLLM yourself instead.

External LiteLLM instead

If you'd rather run LiteLLM outside the bundled Compose services, skip the agent-gateway profile entirely and point the same three AGENT_GATEWAY_LITELLM_* variables at your own instance: its admin URL, its public URL, and its master key. The Proliferate server does not distinguish between the bundled container and an externally run one; it just calls whatever URLs you configure. This is the path for a custom model list, a managed LiteLLM service, or a LiteLLM instance you already run for other purposes.

Removing the gateway

Set AGENT_GATEWAY_ENABLED=false, run ./update.sh, and if you were running the bundled service, stop it too:

docker compose --env-file .env.runtime -f docker-compose.production.yml \
  --profile agent-gateway down

Users fall back to signing in to their agents directly with their own subscriptions, the same as an install that never enabled the gateway.