Agent runtime

Role and HTTP contract#

The backend calls the runtime over HTTP with a task, the agent profile and the context it assembled. The runtime returns validated structured output and, if the agent wants to act, a list of tool_requests. It never executes a tool and never calls an external system: the model may be misled by untrusted text, but it can only propose.

When RUNTIME_TOKEN is set, the runtime requires Authorization: Bearer <token> on every route except GET /healthz, compared in constant time; without it, it answers 401. The endpoints are:

curl http://localhost:8000/healthz
# {"status":"ok","mode":"live","provider":"anthropic",
#  "configured_providers":["anthropic","custom"],"default_provider":"anthropic"}

Simulation and live mode#

The HTTP contract is the same in both modes; only the engine changes.

• Simulation: on with SIMULATION=true or when no provider is configured. It returns plausible plans, results, tokens and costs with simulated latency (SIM_LATENCY_SCALE=0 removes the delay), with no network and no cost, and it ignores the provider policy because no data leaves the process.
• Live: the CrewAI engine calls the provider and validates the output against the StructuredOutput schema. LLM_DIRECT_CALL=true replaces the CrewAI loop with a single chat call.
• Degraded: if a provider is configured but the live engine cannot start, the runtime answers in simulation and GET /healthz returns status: degraded with a warning.

Model providers#

There are five providers. One counts as configured when its key is in the environment (for custom, the base URL is enough) or when the request carries the organization's own key. A custom model is passed as openai/<name>, and a free local model can be priced at 0 with CUSTOM_PRICE_IN_PER_M and CUSTOM_PRICE_OUT_PER_M.

Organizations can bring their own key. It is sealed with AES-256-GCM under a key wrapped by CONNECTIONS_KEK, no API response returns it beyond its last four characters (key_last4), and it travels in clear only on the internal hop from the backend to the runtime.

Provider order and fallback#

For each call the runtime builds an attempt list. Each source adds only the providers not listed yet, so every configured and permitted provider ends up as a fallback:

• MODEL_PROVIDER_ORDER_<ROLE>: the agent role in capitals (for example MODEL_PROVIDER_ORDER_LEGAL) or the call kind: PLAN, REVIEW, CONSULT, SYNTHESIZE, SUMMARIZE, ROUTE, COUNTERARGUMENT.
• role_providers from the organization's model policy.
• The request's preferred_providers.
• The global MODEL_PROVIDER_ORDER (default deepseek,anthropic,openai,kimi,custom).

# Runtime (agent-runtime)
SIMULATION=false                 # true: scripted engine, no network, no cost
DEEPSEEK_API_KEY=...             # one configured provider is enough
ANTHROPIC_API_KEY=...
CUSTOM_LLM_BASE_URL=https://llm.internal.example/v1   # OpenAI-compatible
MODEL_PROVIDER_ORDER=anthropic,deepseek,custom
MODEL_PROVIDER_ORDER_LEGAL=anthropic                  # per role / call kind
ALLOWED_PROVIDERS=anthropic,custom                    # operator ceiling
RUNTIME_TOKEN=<shared secret>    # same value in the backend

# Backend
RUNTIME_URL=http://agent-runtime:8000
RUNTIME_TOKEN=<shared secret>
BUDGET_USD=50                    # organization budget (default 50)
REQUEST_BUDGET_CAP_USD=5         # default cap per request
AGENT_BUDGET_USD=20              # default monthly cap per agent

Data policy: ALLOWED_PROVIDERS#

The data policy is applied before anything is sent. The effective list is the operator's ALLOWED_PROVIDERS intersected with the request's allowed_providers: a provider outside the list is neither called nor used as a fallback, and a request can narrow the operator ceiling but never widen it.

The backend fills allowed_providers from each organization's model policy (GET/PUT /api/v1/settings/model-policy, owner and admin only, every change audited). When no provider is left it sends the sentinel value none, so the runtime refuses the call instead of widening the set. With byo_only, only the organization's own keys may answer. Failures are explicit:

{
  "detail": {
    "code": "no_allowed_provider",
    "message": "No allowed model provider is configured for this request; nothing was sent.",
    "providers_tried": [],
    "allowed_providers": ["anthropic"],
    "configured_providers": ["deepseek"]
  }
}

What leaves the process and redaction#

Only the prompts of each call reach a provider. The contract has no fields for credentials, connection_id, API URLs or tokens, so they cannot be sent. A prompt may contain:

• The user's request text and the agent's persona, responsibilities and tools.
• Outputs of dependency tasks, agent memory and external content.
• Tool results that the backend already filtered and redacted.
• Consult or synthesis questions.

Tools: the model asks, the backend executes#

A tool request is a proposal. The backend checks the agent's tool list, the arguments, the policy engine and the autonomy level, and may create an approval for a person. Only then does it run the tool, with credentials from the encrypted vault that never leave the backend. The result goes back to the model redacted and delimited as untrusted data.

Capabilities that the provider manifest marks as irreversible, such as Gmail's email.send, always require a human decision, whatever the agent's autonomy. An approved action is claimed at most once, so a restart cannot run it twice. Tools without a real connection use in-memory fake adapters, which is how simulation works end to end.

{
  "tool_requests": [
    { "tool": "email", "action": "send", "risk": "high",
      "args": { "to": "customer@example.com", "subject": "Proposal", "body": "..." } }
  ]
}

Code sandbox#

The code.run action lets an agent ask to execute Python. It is available only when the backend has SANDBOX_URL and SANDBOX_TOKEN; without them it is denied with sandbox_unavailable. By default every run is an approval that shows the code, the input files and the purpose, and it is bound to the code hash and the SHA-256 of every input: changing either requires a new approval.

After approval the backend sends the run to a separate runner that starts a disposable container:

• No network (--network none), a read-only root, a non-root user and all capabilities dropped.
• Time, memory, process and output-size limits (SANDBOX_MAX_RUNTIME_SECONDS, SANDBOX_MEMORY_MB, SANDBOX_PIDS, SANDBOX_MAX_OUTPUT_BYTES) and an allow-list of output file types.
• The runner's environment is not forwarded, so no secrets reach the code.
• The kill switch, an agent pause or read-only mode abort a running job.

Cost accounting and budgets#

Prices live in one table, model_prices.json, in USD per million tokens, byte-identical in the backend and the runtime (a test in each service enforces it). A model missing from the table is billed at the table's default rate and logged once. PRICE_IN_PER_M and PRICE_OUT_PER_M force a rate for every model.

Before each call the backend reserves the maximum estimate against the request, the agent and the organization. Afterwards it records the greater of the runtime's reported cost and its own recalculation, so it does not depend on the runtime's accuracy. Spending is read from a ledger in Postgres and survives restarts.