Models & providers

Providers, model roles, named models, and where each inference call goes

Every inference call in graph resolves through one system: a provider (who serves the model), a role (which job the call is doing), and optionally a named model (a specific entry selected at the point of use). Cost tuning is pure config — strong model where judgment lives, fast model where volume lives — and this page is the canonical map of where each call goes. The TOML syntax lives in the configuration reference.

Providers

[providers.anthropic]
type = "anthropic"
api_key = "${ANTHROPIC_API_KEY}"

[providers.local]
type = "openai_compat"
base_url = "http://localhost:11434/v1"    # Ollama, vLLM, LM Studio…

type	Serves
anthropic	the Anthropic API
openai	the OpenAI API
openai_compat	any OpenAI-compatible endpoint — Ollama, vLLM, LM Studio, hosted gateways
bedrock	AWS Bedrock — roadmap; the config key is accepted but the provider is a stub

A provider whose config can’t be honored — most commonly an unset ${VAR} behind its api_key, or the bedrock stub — is configured but not usable: the config still loads and every command that never resolves a model to it keeps working, while the first call that does errors naming the reason (for a missing variable, the variable and the config path that references it).

Requests retry transient failures (429, 5xx, connection errors, timeouts) up to 3 attempts with backoff. Structured output uses each provider’s native mechanism — a forced tool on Anthropic, json_schema with a json_object fallback on OpenAI-compatible servers that reject strict schema mode — and a result that fails to parse or validate gets one repair-role fix-up pass before erroring. Anthropic requests retry once without temperature when a model rejects the parameter.

Roles

Each pipeline role resolves to a model, falling back to default:

[models]
default = { provider = "anthropic", model = "claude-sonnet-5" }
solver  = { provider = "anthropic", model = "claude-haiku-4-5", temperature = 0.4 }

Role	Fires when
chat	every agent-loop turn in ask/chat and the workbench’s chat pane
planner	plan_and_execute authors or revises a plan, and draft-only planning in the workbench
solver	a solver-mode plan synthesizes its report
repair	a structured output fails to parse or validate — one fix-up pass
judge	an infer gate on an exit or decide step needs a yes/no verdict

The common cost setup: a strong model for chat/planner (they do the judgment), a fast model for solver/repair/judge (they do the volume). graph plan run on an authored plan touches only solver (or nothing) — the cost table maps invocations to calls.

Named models

Beyond the fixed roles, [models.named.<name>] entries are referenceable wherever a model name is accepted. That’s exactly three places:

1. a prompt tool’s model field
2. builtin__infer’s model input
3. the model: override on an infer gate (exit/decide)

Role names resolve in those same places, with their usual fallback to default; names may not shadow role names (config load fails); an unknown name fails the call listing what is configured — never a silent fallback.

[models.named.nano]
provider = "anthropic"
model = "claude-haiku-4-5"
description = "fast and cheap; small self-contained tasks like per-item map bodies"

The description is a planner-facing routing signal: when named models exist, builtin__infer’s catalog schema advertises them with guidance to prefer the smallest adequate model, so planner-authored plans route small chunks of work — per-item map bodies especially — to cheap models on their own. Write descriptions for that audience.

Provider failover

Any model entry — a role or a named model — can carry ordered fallbacks for outages:

[models.chat]
provider = "anthropic"
model = "claude-sonnet-5"
fallbacks = [
    { provider = "openai", model = "gpt-5" },
    { provider = "local", model = "llama3", temperature = 0.3 },
]

Each candidate names its own provider and model (model names rarely carry across providers); temperature optionally overrides — otherwise the primary’s effective temperature carries over. Every referenced provider must exist under [providers], checked at startup so a typo’d fallback surfaces immediately rather than mid-outage.

Semantics:

• A call moves to the next candidate only on outage-shaped errors — the transient class the retry layer recognizes — and only after the failing provider’s own retries are exhausted. Permanent errors (4xx, parse/schema failures) propagate immediately: a bad request would fail everywhere, and a fallback would only mask it.
• Streaming fails over only while the stream is being established; once tokens flow, a mid-stream error surfaces as-is.
• Fallbacks apply wherever the entry resolves — the agent loop, every pipeline role, structured output, and named-model calls. Each failover is logged as a warning on stderr.