An LLM router API built for the request path
Sending every user turn or agent step to a frontier model burns latency and margin on routine lookups. Score destination, complexity and approval gates in one fast pass with Clef-Flash before waking a heavy model.
100% frontier LLM ($2.50/1M in, $10/1M out): $2,805/mo → Routed stack: $757/mo (~20,000 calls/day · fits Scale).
curl --fail-with-body https://clef-flash.com/api/v1/decisions \
-H "authorization: Bearer $CLEF_API_KEY" \
-H "content-type: application/json" \
--data-binary @- <<'CLEF_DECISION'
{
"model": "cloudflare/clef-flash",
"threshold": 0.85,
"state": "Customer asks: Where do I download last month PDF receipt and update our billing email?",
"questions": {
"destination": {
"type": "choice",
"instructions": "Which execution path should handle this request?",
"criteria": {
"docs_lookup": "standard product how-to, receipt download or settings lookup",
"billing_workflow": "deterministic account/invoice state read via internal API",
"frontier_reasoning": "multi-system debugging, ambiguous contract dispute or custom code"
}
},
"complexity": {
"type": "score",
"instructions": "How much reasoning depth does this state require?",
"criteria": [
"trivial_lookup",
"single_step",
"multi_step",
"frontier_escalation"
]
},
"require_human_gate": {
"type": "noul",
"instructions": "Does this action mutate billing, credentials or production data?",
"criteria": {
"true": "writes, deletes, charges, revokes keys or changes customer permissions",
"false": "read-only query, search, draft or reversible local check"
}
}
}
}
CLEF_DECISION
Cloudflare reported 38.8 ms median (122.4 ms p95) model latency for Clef-Flash on 1 Oct 2026; network round-trip depends on caller region.
Ship this in production with a $19 Pro key
2,000 decisions/day, 8 concurrent requests in flight, text + JSON + image states, and an API key emailed in about a minute.
Why routing with a chat model defeats the point
Teams often build their first agent router by asking a general chat model to emit a JSON tool name. That adds 400 to 900 milliseconds of time-to-first-token, pays for output generation, and still requires a fallback parser when the model wraps its JSON in prose — eating the exact latency budget the router was supposed to save.
Clef-Flash is a 9B decision model post-trained specifically to score closed options in a single forward pass. Cloudflare’s published benchmark measured 38.8 ms median model latency (122.4 ms at p95), and upstream input costs $0.019 per million tokens with zero output-token billing.
Three routing decisions in a single API call
A production agent gate rarely asks just one question. When a user prompt or tool trace arrives, you want to know which worker or tool owns it, how hard the task is, and whether executing it touches billing or destructive state.
POST /api/v1/decisions answers all three over the same state in one round trip: a choice question for the destination, an ordered score question for reasoning depth, and a noul question for human-in-the-loop approval. If the winning route’s probability drops below your threshold, needsReview flips to true so your orchestrator escalates safely.
- Destination (choice): SQL warehouse, vector KB, deterministic API or frontier LLM.
- Reasoning tier (score): trivial lookup through multi-step synthesis.
- Safety gate (noul): probability that the step mutates money, keys or production records.
Router architectures compared
Embedding routers are fast until two intents share vocabulary ("cancel my webhook retry" vs "cancel my annual plan"). Small chat LLMs understand nuance at the cost of chat latency and schema drift. A decision model combines full cross-attention over up to 24,576 served tokens (64K on the model card) with a closed probability distribution.
| Router type | Zero-shot new routes | Negation & nuance | Calibrated confidence | Output parsing risk |
|---|---|---|---|---|
| Embedding cosine similarity | Needs anchor tuning | Weak on negation | Unbounded cosine score | None |
| Small chat LLM + JSON mode | Yes | Good | No native probability vector | Malformed / extra keys |
| Clef-Flash Decision API | Yes, edit criteria live | Full 9B attention | Probabilities sum to 1.0 | Zero — closed schema |
Sizing concurrency for bursty agent workflows
Agent loops fan out in bursts rather than polite steady averages, which is why Clef-Flash plans cap concurrent requests in flight instead of imposing an artificial requests-per-second throttle: 8 in flight on Pro ($19 / 30 days, 2,000 decisions/day), 32 on Scale (20,000/day), 64 on Studio (40,000/day), and 128 on Agency (100,000/day).
When an agent transcript grows beyond Clef-Flash’s 120,000-character ceiling, route that call to cloudflare/clef-omni on the same API key (up to 200,000 characters / 65,529 served tokens) without changing your client code.
Frequently asked questions
How much latency does Clef-Flash add in front of an LLM?
In Cloudflare’s published benchmark on 1 October 2026, Clef-Flash scored 38.8 ms median and 122.4 ms p95 model latency. End-to-end HTTP time from your server includes network transit plus Worker validation; test from your own region with the free endpoint or a $19 Pro key.
What should my router do when confidence is low?
Pass a threshold (for example 0.85) in your POST /api/v1/decisions body. Whenever the top choice probability is below 0.85, the response marks needsReview: true — use that signal to fall back to your frontier model or ask the user a clarifying question.
Can I change route definitions per tenant without deploying weights?
Yes. Route labels and their criteria descriptions are sent in each request and have no cross-request state, so multi-tenant SaaS platforms can pass each customer’s enabled tools dynamically (2 to 12 choices per question).
How does concurrency enforcement work on paid keys?
Each API key tracks active in-flight requests via a Durable Object (8 on Pro, 32 on Scale, 64 on Studio, 128 on Agency). If you burst past your plan’s in-flight ceiling, excess calls receive 429 concurrency_exceeded immediately without spending credits or daily quota.
Keep reading
- Intent classification API
- Jev alternatives
- AI content moderation API
- Decision model vs LLM
- Pricing & API keys
Model notes
One short email when a new decision model lands or a benchmark changes. No spam, unsubscribe in one click.