llm router api

An LLM router API built for the request path

Sending every user turn or agent step to a frontier model burns latency and margin on routine lookups. Score destination, complexity and approval gates in one fast pass with Clef-Flash before waking a heavy model.

Agent & LLM router cost-savings simulator
Est. monthly LLM savings
$2,048/mo (-73%)
Clef-Flash router pass cost
$8.55/mo

100% frontier LLM ($2.50/1M in, $10/1M out): $2,805/mo → Routed stack: $757/mo (~20,000 calls/day · fits Scale).

curl --fail-with-body https://clef-flash.com/api/v1/decisions \
  -H "authorization: Bearer $CLEF_API_KEY" \
  -H "content-type: application/json" \
  --data-binary @- <<'CLEF_DECISION'
{
  "model": "cloudflare/clef-flash",
  "threshold": 0.85,
  "state": "Customer asks: Where do I download last month PDF receipt and update our billing email?",
  "questions": {
    "destination": {
      "type": "choice",
      "instructions": "Which execution path should handle this request?",
      "criteria": {
        "docs_lookup": "standard product how-to, receipt download or settings lookup",
        "billing_workflow": "deterministic account/invoice state read via internal API",
        "frontier_reasoning": "multi-system debugging, ambiguous contract dispute or custom code"
      }
    },
    "complexity": {
      "type": "score",
      "instructions": "How much reasoning depth does this state require?",
      "criteria": [
        "trivial_lookup",
        "single_step",
        "multi_step",
        "frontier_escalation"
      ]
    },
    "require_human_gate": {
      "type": "noul",
      "instructions": "Does this action mutate billing, credentials or production data?",
      "criteria": {
        "true": "writes, deletes, charges, revokes keys or changes customer permissions",
        "false": "read-only query, search, draft or reversible local check"
      }
    }
  }
}
CLEF_DECISION

Cloudflare reported 38.8 ms median (122.4 ms p95) model latency for Clef-Flash on 1 Oct 2026; network round-trip depends on caller region.

Pro API key · $19 / 30 days · no auto-renewal

Ship this in production with a $19 Pro key

2,000 decisions/day, 8 concurrent requests in flight, text + JSON + image states, and an API key emailed in about a minute.

Why routing with a chat model defeats the point

Teams often build their first agent router by asking a general chat model to emit a JSON tool name. That adds 400 to 900 milliseconds of time-to-first-token, pays for output generation, and still requires a fallback parser when the model wraps its JSON in prose — eating the exact latency budget the router was supposed to save.

Clef-Flash is a 9B decision model post-trained specifically to score closed options in a single forward pass. Cloudflare’s published benchmark measured 38.8 ms median model latency (122.4 ms at p95), and upstream input costs $0.019 per million tokens with zero output-token billing.

Three routing decisions in a single API call

A production agent gate rarely asks just one question. When a user prompt or tool trace arrives, you want to know which worker or tool owns it, how hard the task is, and whether executing it touches billing or destructive state.

POST /api/v1/decisions answers all three over the same state in one round trip: a choice question for the destination, an ordered score question for reasoning depth, and a noul question for human-in-the-loop approval. If the winning route’s probability drops below your threshold, needsReview flips to true so your orchestrator escalates safely.

  • Destination (choice): SQL warehouse, vector KB, deterministic API or frontier LLM.
  • Reasoning tier (score): trivial lookup through multi-step synthesis.
  • Safety gate (noul): probability that the step mutates money, keys or production records.

Router architectures compared

Embedding routers are fast until two intents share vocabulary ("cancel my webhook retry" vs "cancel my annual plan"). Small chat LLMs understand nuance at the cost of chat latency and schema drift. A decision model combines full cross-attention over up to 24,576 served tokens (64K on the model card) with a closed probability distribution.

Router typeZero-shot new routesNegation & nuanceCalibrated confidenceOutput parsing risk
Embedding cosine similarityNeeds anchor tuningWeak on negationUnbounded cosine scoreNone
Small chat LLM + JSON modeYesGoodNo native probability vectorMalformed / extra keys
Clef-Flash Decision APIYes, edit criteria liveFull 9B attentionProbabilities sum to 1.0Zero — closed schema

Sizing concurrency for bursty agent workflows

Agent loops fan out in bursts rather than polite steady averages, which is why Clef-Flash plans cap concurrent requests in flight instead of imposing an artificial requests-per-second throttle: 8 in flight on Pro ($19 / 30 days, 2,000 decisions/day), 32 on Scale (20,000/day), 64 on Studio (40,000/day), and 128 on Agency (100,000/day).

When an agent transcript grows beyond Clef-Flash’s 120,000-character ceiling, route that call to cloudflare/clef-omni on the same API key (up to 200,000 characters / 65,529 served tokens) without changing your client code.

Frequently asked questions

How much latency does Clef-Flash add in front of an LLM?

In Cloudflare’s published benchmark on 1 October 2026, Clef-Flash scored 38.8 ms median and 122.4 ms p95 model latency. End-to-end HTTP time from your server includes network transit plus Worker validation; test from your own region with the free endpoint or a $19 Pro key.

What should my router do when confidence is low?

Pass a threshold (for example 0.85) in your POST /api/v1/decisions body. Whenever the top choice probability is below 0.85, the response marks needsReview: true — use that signal to fall back to your frontier model or ask the user a clarifying question.

Can I change route definitions per tenant without deploying weights?

Yes. Route labels and their criteria descriptions are sent in each request and have no cross-request state, so multi-tenant SaaS platforms can pass each customer’s enabled tools dynamically (2 to 12 choices per question).

How does concurrency enforcement work on paid keys?

Each API key tracks active in-flight requests via a Durable Object (8 on Pro, 32 on Scale, 64 on Studio, 128 on Agency). If you burst past your plan’s in-flight ceiling, excess calls receive 429 concurrency_exceeded immediately without spending credits or daily quota.

Keep reading

Model notes

One short email when a new decision model lands or a benchmark changes. No spam, unsubscribe in one click.