Configuration

Semantic Router uses one canonical YAML document across the CLI, Dashboard, Helm, and Operator. The top-level structure is:

version:
listeners:
providers:
routing:
entrypoints:
recipes:
global:

Most deployments begin with version, listeners, providers, and one top-level routing profile. Add entrypoints and recipes when one deployment needs several isolated policies. Add global settings only for shared services or runtime behavior that differs from the built-in defaults.

What belongs where

Section	Owns
version	Canonical schema version. Use v0.3.
listeners	Public Router listeners and timeouts.
providers	Logical provider models, physical backend endpoints, pricing, capabilities, and defaults.
routing	The default recipe: model cards, signals, projections, decisions, strategy, algorithms, and route plugins.
entrypoints	Public virtual model aliases mapped to named recipes.
recipes	Additional isolated routing profiles that share providers and global infrastructure.
global	Router services, stores, integrations, observability, learning, and router-owned model assets.

Keep these boundaries clear:

• signals detect facts;
• projections combine evidence;
• decisions define eligibility and route policy;
• algorithms choose or coordinate candidate models;
• plugins add behavior at route-specific hook points; and
• providers bind logical model names to inference endpoints.

Provider pricing belongs beside each concrete model under providers.models[].pricing. It accepts an optional uppercase three-letter currency plus non-negative prompt_per_1m, completion_per_1m, cached_input_per_1m, and cache_write_per_1m rates. Routing model cards do not repeat deployment prices or credentials.

Use Protocol Compatibility to choose the model's backend api_format. Then see Backend Target Compatibility before moving its bindings between Docker, Helm, the Operator, and Dashboard workflows. The target matrix distinguishes canonical pass-through from Kubernetes discovery and records which URL, path, weight, and provider fields each surface preserves.

Router-wide debugging surfaces stay closed by default. global.services.observability.profiling serves Go pprof endpoints, and only when it is explicitly enabled; it then binds 127.0.0.1:6060 so profiles never reach a routable interface without an explicit bind change. The switch is read once at startup, so changing it requires a Router restart. See API and Observability.

Built-in category/domain classification uses the local variant selector when no remote backend is configured. To call a named external classifier, attach a backend under global.model_catalog.modules.classifier.domain and resolve its model from global.model_catalog.external[] with model_role: classification. The shared backend fields are protocol, contract, model, and optional deadline_ms; category currently supports http_classify with the full label_distribution.v1 response contract. Omit backend to retain local behavior. The deprecated use_modernbert and use_mmbert_32k keys remain readable, while generated canonical configuration uses variant: candle, variant: modernbert, or variant: mmbert32k.

The Routing Pipeline explains the design. Capability pages under Capabilities document each signal, projection, decision, algorithm, plugin, and global block.

Capability catalog

Use this catalog to choose a reusable building block, then open its guide for configuration details. The inventory comes from config/fragments/; each one-line goal comes from the matching guide's Overview. The documentation build regenerates this block and fails if the checked-in catalog has drifted.

Signals

Family and type	Use it to	Reusable fragment	Guide
authz — heuristic signal	authz turns identity and policy bindings into reusable routing inputs under routing.signals.role_bindings.	config/fragments/signal/authz/	Guide
classifier — learned signal	classifier exposes reusable label scores from a local native sequence classifier, a remote sequence classifier, or a configured external LLM.	config/fragments/signal/classifier/	Guide
complexity — learned signal	complexity estimates whether a request is easy, medium, or hard by comparing it with configured example sets.	config/fragments/signal/complexity/	Guide
context — heuristic signal	context detects requests that need a larger effective context window.	config/fragments/signal/context/	Guide
conversation — heuristic signal	conversation routes on chat structure and protocol facts, such as message count, developer instructions, available tools, explicit tool-use constraints, or an active tool loop.	config/fragments/signal/conversation/	Guide
domain — learned signal	domain classifies the request topic family.	config/fragments/signal/domain/	Guide
embedding — learned signal	embedding matches requests by semantic similarity to representative examples.	config/fragments/signal/embedding/	Guide
event — heuristic signal	event routes structured event-like requests by event type, severity, urgency, or domain-specific action code.	config/fragments/signal/event/	Guide
fact-check — learned signal	fact-check decides whether a prompt should be treated as evidence-sensitive traffic.	config/fragments/signal/fact-check/	Guide
hallucination — learned signal	hallucination checks the model's answer against the grounding context the request carried, such as tool results or retrieved documents, and reports the claims that context does not support.	config/fragments/signal/hallucination/	Guide
input-modality — heuristic signal	input_modality deterministically matches which kinds of input — text, image, audio, or video — are present in the parsed request.	config/fragments/signal/input-modality/	Guide
jailbreak — learned signal	jailbreak detects prompt-injection and jailbreak attempts before the Router commits to a route.	config/fragments/signal/jailbreak/	Guide
kb — learned signal	kb binds routing signals to the output of a named knowledge base instance.	config/fragments/signal/kb/	Guide
keyword — heuristic signal	keyword matches explicit words and phrases in the request.	config/fragments/signal/keyword/	Guide
language — heuristic signal	language detects the request language and exposes it as a routing signal.	config/fragments/signal/language/	Guide
metadata — heuristic signal	metadata matches bounded string values supplied by the caller in request metadata.	config/fragments/signal/metadata/	Guide
modality — learned signal	modality detects whether a request should stay in text generation, switch into image generation, or support both.	config/fragments/signal/modality/	Guide
pii — learned signal	pii detects sensitive personal data in requests.	config/fragments/signal/pii/	Guide
preference — learned signal	preference infers response-style preferences from examples and classifier settings.	config/fragments/signal/preference/	Guide
reask — learned signal	reask detects when the current user turn semantically repeats recent user turns in the same conversation.	config/fragments/signal/reask/	Guide
structure — heuristic signal	structure detects request-shape facts such as many explicit questions, ordered workflow markers, or dense constraint phrasing.	config/fragments/signal/structure/	Guide
user-feedback — learned signal	user-feedback detects correction, dissatisfaction, or escalation feedback from the conversation.	config/fragments/signal/user-feedback/	Guide

Selection algorithms

Family and type	Use it to	Reusable fragment	Guide
automix — selection algorithm	automix is an experimental selector that ranks candidate models by configured quality and cost plus internal verification and escalation estimates.	config/fragments/algorithm/selection/automix.yaml	Guide
hybrid — selection algorithm	hybrid combines Elo ratings, Router-DC description similarity, AutoMix's one-model value estimate, and cost into one weighted candidate score.	config/fragments/algorithm/selection/hybrid.yaml	Guide
kmeans — selection algorithm	kmeans sends a request to the model assigned to its nearest learned cluster.	config/fragments/algorithm/selection/kmeans.yaml	Guide
knn — selection algorithm	knn chooses a candidate from the models that performed well on the most similar recorded requests.	config/fragments/algorithm/selection/knn.yaml	Guide
latency-aware — selection algorithm	latency_aware ranks eligible candidates using observed TTFT and TPOT percentiles and selects the lowest relative-latency score.	config/fragments/algorithm/selection/latency-aware.yaml	Guide
mlp — selection algorithm	mlp runs a trained neural classifier on CPU to map a request to a candidate model.	config/fragments/algorithm/selection/mlp.yaml	Guide
multi-factor — selection algorithm	multi_factor ranks candidates by a configurable combination of quality, latency, cost, and load, then rejects any candidate that violates a hard limit.	config/fragments/algorithm/selection/multi-factor.yaml	Guide
prompt — selection algorithm	prompt uses a concrete helper model to select exactly one model from the matched decision's modelRefs.	config/fragments/algorithm/selection/prompt.yaml	Guide
router-dc — selection algorithm	router_dc embeds the request and each model description, then selects the candidate with the strongest semantic similarity.	config/fragments/algorithm/selection/router-dc.yaml	Guide
static — selection algorithm	static provides deterministic model choice without metrics or learned state.	config/fragments/algorithm/selection/static.yaml	Guide
svm — selection algorithm	svm uses a trained linear or RBF support-vector classifier to map request features to a candidate model.	config/fragments/algorithm/selection/svm.yaml	Guide

Looper algorithms

Family and type	Use it to	Reusable fragment	Guide
confidence — looper algorithm	confidence tries candidate models in order and stops when response confidence reaches a configured threshold.	config/fragments/algorithm/looper/confidence.yaml	Guide
fusion — looper algorithm	fusion asks several models to analyze a request and a judge model to synthesize one final answer.	config/fragments/algorithm/looper/fusion.yaml	Guide
ratings — looper algorithm	ratings calls every candidate model and returns one OpenAI-compatible choice per successful model. max_concurrent limits parallel work; it does not limit the total number of candidates executed.	config/fragments/algorithm/looper/ratings.yaml	Guide
remom — looper algorithm	remom runs several candidate models across bounded rounds and synthesizes their responses into one answer.	config/fragments/algorithm/looper/remom.yaml	Guide
workflows — looper algorithm	workflows runs a bounded, multi-step Router Flow behind one OpenAI-compatible model name.	config/fragments/algorithm/looper/workflows.yaml	Guide

Plugins and bundles

Family and type	Use it to	Reusable fragment	Guide
content-safety — plugin bundle	Content Safety combines supported route-local safety plugins into one reusable policy.	config/fragments/plugin/content-safety/	Guide
context-compression — route plugin	context_compression is a route-local request plugin that reduces large tool/function outputs before the selected provider receives the request.	config/fragments/plugin/context-compression/	Guide
fast-response — route plugin	fast_response is a route-local plugin that returns a deterministic fallback message immediately.	config/fragments/plugin/fast-response/	Guide
hallucination — route plugin	hallucination is a route-local plugin for fact-checking and response-quality screening after the decision already matched.	config/fragments/plugin/hallucination/	Guide
header-mutation — route plugin	header_mutation is a route-local plugin for adding, updating, or deleting downstream headers.	config/fragments/plugin/header-mutation/	Guide
memory — route plugin	memory is a route-local plugin for retrieving and storing conversation memory.	config/fragments/plugin/memory/	Guide
rag — route plugin	rag retrieves external context for a matched route before generation.	config/fragments/plugin/rag/	Guide
request-params — route plugin	request_params is a route-local plugin that validates and trims OpenAI Chat Completions request bodies before they are forwarded to backends.	config/fragments/plugin/request-params/	Guide
response-cache — route plugin	response_cache is the route-local plugin for reusing exact or semantically compatible prior responses.	config/fragments/plugin/response-cache/	Guide
response-jailbreak — route plugin	response_jailbreak is a route-local plugin for screening the model response before it is returned.	config/fragments/plugin/response-jailbreak/	Guide
router-replay — route plugin	router_replay is a route-local plugin for overriding replay/debug capture on one route.	config/fragments/plugin/router-replay/	Guide
system-prompt — route plugin	system_prompt is a route-local plugin for inserting or modifying the system prompt on matched traffic.	config/fragments/plugin/system-prompt/	Guide
tool-selection — route plugin	tool_selection is a decision plugin that controls how tools are chosen for a matched route.	config/fragments/plugin/tool-selection/	Guide
tools — route plugin	tools is a route-local plugin for tool filtering and semantic tool selection.	config/fragments/plugin/tools/	Guide

Minimal example

version: v0.3

listeners:
  - name: http-8899
    address: 0.0.0.0
    port: 8899
    timeout: 300s

providers:
  defaults:
    model: local/general
  models:
    - name: local/general
      provider_model_id: my-served-model
      backend_refs:
        - name: primary
          endpoint: host.docker.internal:8000
          protocol: http
          provider: vllm

routing:
  strategy: priority
  modelCards:
    - name: local/general
      modality: text
      capabilities: [chat]
  signals:
    keywords:
      - name: needs_explanation
        operator: OR
        keywords: ["explain", "walk me through"]
  decisions:
    - name: explanatory_answer
      description: Prefer an explanatory answer when the request asks for one.
      priority: 100
      rules:
        operator: AND
        conditions:
          - type: keyword
            name: needs_explanation
      modelRefs:
        - model: local/general

global:
  services:
    observability:
      metrics:
        enabled: true

Catalog-backed models

Built-in support is additive to the same version: v0.3 hierarchy. Set the optional canonical catalog identity and use a stable Provider ID on the backend; the Router and CLI then materialize the Model Card, reasoning family, native model mapping, protocol, request path, and provider defaults:

providers:
  defaults:
    model: production
    reasoning_effort: medium
  models:
    - name: production
      catalog: openai/gpt-5.6-sol
      backend_refs:
        - provider: openai
          api_key_env: OPENAI_API_KEY

The name remains the request-facing alias. A handwritten override targets the canonical card with routing.modelCards[].name: openai/gpt-5.6-sol. Private or newly released vLLM/SGLang models simply omit catalog and may keep using a handwritten card under their alias. api_format: openai|responses|anthropic is unchanged and remains an explicit protocol override; it never chooses a Provider. If this config declares a listener, every physical model must define backend_refs with an explicit Provider ID. A metadata-only external-gateway config with listeners: [], and a built-in virtual model whose recipe resolves its pool, may remain backendless. The local vllm-sr serve workflow owns Envoy transport and therefore rejects a backendless physical model even when it supplies its legacy default listener for an empty listener list. Use the external-gateway deployment profile for state-only protocol metadata. If providers.defaults.reasoning_effort is omitted, each model uses its reasoning-family default; saved canonical YAML does not add an unconfigured global effort.

Multiple backend_refs on one alias are homogeneous replicas. HTTP replicas may use different hosts, ports, and weights. HTTPS replicas may vary by port and weight but must keep one DNS hostname. Provider, wire protocol, native model ID, credential, headers, effective request path, and TLS semantics must also match. Use separate aliases for heterogeneous providers so request metadata always follows the upstream Envoy selects. See the Model and provider Day-0 guide for the complete contribution and evaluation workflow.

Classifier backend failures remain Unknown while the complete boolean tree is evaluated. Set rules.on_unknown to no_match, match, or fail_request to resolve an undetermined terminal result. Omitting it preserves the existing classifier-family error behavior.

Requests using an automatic model alias enter the default routing profile. A concrete provider model name is a direct pass-through request and bypasses recipe signals, decisions, route plugins, cache, learning, and session routing.

Validate and serve

vllm-sr validate --config config.yaml
vllm-sr serve --config config.yaml

Validation catches schema errors, unresolved references, incompatible recipe boundaries, invalid provider bindings, and unsupported plugin or algorithm settings before the Router starts.

For portable model-free Recipes, set routing.decisions[].algorithm.minimum_candidates to the smallest pool that preserves the decision's intended behavior. Empty built-in assets remain valid, while a published Entrypoint is rejected if its concrete assignments do not meet the declared cardinality.

Environment references and secrets

Keep credentials outside the YAML file:

api_key: ${MODEL_API_KEY}

Supported string substitutions are:

• ${VAR} and $VAR;
• ${VAR:-default} when VAR is unset or empty;
• ${VAR-default} when VAR is unset; and
• $$ for a literal $.

For a custom Recipe, authorize required host variables explicitly with --recipe-env NAME. Kubernetes deployments place sensitive environment values in Secrets rather than ConfigMaps or Helm values. See Security Hardening.

Entrypoints and recipes

An entrypoint maps one or more public model aliases to a recipe. A recipe owns its signal, projection, decision, algorithm, plugin, cache, replay, learning, and routing state. Providers, stores, and router-owned classifier assets may be shared without allowing policy state to cross recipe boundaries.

Set max_response_bytes on external LLM classifier entries and the MCP classifier module to cap one upstream classifier response.

In the schema, entrypoints[].model_names lists the public aliases, entrypoints[].recipe selects a named recipe, and recipes[].routing contains that recipe's policy.

If no decision matches, the recipe uses providers.defaults.model. The virtual entrypoint name never reaches a backend.

See Models, Entrypoints, and Serving for built-in virtual models, CLI serving, backend binding, forking, packaging, and migration. See Virtual Models for the complete schema.

Configuration workflows

The canonical document can be authored or applied through several interfaces:

• local CLI and YAML;
• Dashboard setup and visual routing tools;
• Helm or vllm-sr serve --target k8s;
• the Kubernetes Operator; and
• the routing DSL.

Configuration Workflows explains which interface owns which part of the document and how to avoid competing sources of truth.

Reference sources

• config/config.yaml is the exhaustive canonical example.
• config/fragments/ contains reusable signal, decision, algorithm, and plugin fragments.
• Providers and routing tutorials describe shared runtime configuration.
• Unified Config Contract v0.3 records the design behind the current contract.

Avoid copying the exhaustive example as an application config. Start with the smallest document that describes the deployment, then add only the capabilities and services it uses.