clef flash benchmark

Clef Flash Benchmarks: Measure the Complete Decision

Cloudflare's published Clef Flash results are vendor-reported measurements, not an independent benchmark performed by this site. To evaluate your application, measure the complete path to a valid, accepted decision, including failures and review work.

Summarize your own observations

One attempt per line: elapsed milliseconds,outcome. Outcomes: accepted, incorrect, invalid, timeout, error. No header; maximum 500 rows. Include the time spent on failed attempts. Nothing is uploaded or saved; this tool does not run a model.

Median and nearest-rank p95 include all attempts. A small sample cannot establish tail latency. Cost per accepted decision is undefined if none were accepted. Repeated attempts remain separate rows.

Before you measure

A fast decision model can be useful in routing and classification, but a published inference number does not tell you how long a customer waits in your application. Input preparation, network transit, queueing, response validation and a retry can all sit between the button press and a usable result.

This guide explains how to read the published Clef Flash results and create a small evaluation you can reproduce. It contains no new model run and no invented performance score. The worksheet accepts your observations and calculates summaries; it does not turn sample numbers into a live model benchmark.

Attribute the published result to its actual source

Cloudflare's announcement and model card describe its own Clef evaluation. The often repeated median of 38.8 milliseconds and p95 of 122.4 milliseconds belong to that reported test. They must not be labeled as measurements made independently by this website or as a guaranteed response time for its playground.

Keep the source link beside any number you cite. Include what was measured and the stated setup when available. If a condition is missing, mark it as unknown. Do not fill the gap with a plausible machine, request length, region or concurrency level simply to make a comparison table look complete.

Define the user's completed task before the timer

Write one sentence describing the useful outcome. For ticket routing, that might be a valid queue decision based on the supplied message. For moderation, it might be a policy label with a visible review path. A request returning HTTP success is only a transport event; the output still needs to satisfy the task.

Choose a few acceptance checks you can apply consistently. Required fields should be present, values should be in the allowed set, and the selected action should agree with the labeled case. Separate format failures from incorrect judgments, because fixing one does not establish that the other has improved.

Record the exact service and model configuration

The open model, Workers AI endpoint and an independent web application are different execution environments. Record which one you test, the model identifier, version when exposed, input type and configurable options. Preserve that record with the sample set.

A model update or a changed wrapper can alter the result without any visible change to your application. A useful evaluation should let you answer which configuration produced each output. If the exact upstream model is not exposed, say that the test evaluates the product endpoint rather than claiming an isolated model comparison.

Make a small dataset that represents actual work

Use inputs you are allowed to evaluate and remove unnecessary personal information. Include ordinary requests, longer examples, ambiguous messages and cases where the safe outcome is review rather than an automatic action. Write expected labels before seeing the model's predictions.

Keep a held-out set for the final check after changing the question or schema. Otherwise, repeated editing against the same examples can make a configuration look unusually good. If human reviewers disagree on a case, record the disagreement and clarify the policy instead of pretending there is a single obvious ground truth.

Measure end-to-end latency separately from inference

Start the application timer before submitting the request and stop it when a usable decision has passed validation. Save any provider-reported timing in a separate field. Do not subtract network or parsing time from the user-facing measurement unless the chart clearly describes a different metric.

Use monotonic timing inside the application where possible. Record the same start and end boundaries for every candidate. A comparison becomes misleading when one timer includes retries and another ends as soon as the provider acknowledges a queued request.

Reproduce the check with one fixed request

This site's release checks include a small script, scripts/benchmark-cli.mjs, that sends one fixed request to the endpoint and times each attempt from the caller, beside the latencyMs the server reports for the same attempt. It prints one "milliseconds,outcome" row per attempt, the format the worksheet on this site accepts, so a run can be pasted into it without editing.

The request is deliberately fixed: a short duplicate-charge message and one three-label question (billing, technical, account). Fixing it is what makes two runs comparable, and changing it invalidates the comparison. The script carries no labelled answers, so it never reports accuracy and never emits "incorrect". A decision counts as accepted only when the response is a 200 whose distribution covers exactly the declared labels and sums to one.

  • Keyed endpoint: CLEF_FLASH_KEY=… node scripts/benchmark-cli.mjs --url https://clef-flash.com --runs 30 --json run.json
  • Anonymous endpoint: node scripts/benchmark-cli.mjs --url https://clef-flash.com --free --runs 20 (the free allowance is 25 per session per day; a longer run stops at the allowance instead of counting refusals as latency)
  • Read the summary, not only the median: attempts, accepted, failure categories, the caller's median and p95, and the server's median and p95 taken from its own latencyMs.

Include unsuccessful attempts in the observation log

For every attempt, capture an outcome category such as accepted, incorrect, invalid response, timeout or service error. Store the elapsed time even when the request fails. If the application retries, link the attempts to the same task so you can also measure the total wait.

A successful-request-only chart can hide a system that makes users start over frequently. Report the failure count next to timing summaries and state whether any attempts were excluded. Excluding a known setup mistake can be reasonable, but keep the reason visible and do not quietly remove slow results.

Explain median and p95 without hiding the sample size

The median describes the middle of the observed durations; it is not the worst wait. A p95 summarizes the slower part of a sample, but its usefulness depends on how many representative observations you collected. A handful of requests cannot establish a dependable tail-latency promise.

If the worksheet uses the nearest-rank method, say so and display the sample count. Keep cold starts and already-running tests in separate groups. Avoid combining different input lengths, hardware setups and queue conditions into a single number unless that mixture reflects the traffic you actually intend to serve.

Compare errors at the action boundary

Review what would happen after a wrong decision. Routing a ticket to a neighboring internal queue may be recoverable. Rejecting an important customer request or blocking a legitimate action can have a different cost. A high overall accuracy can still conceal a failure pattern that matters to your product.

Break down results by action type and important input groups. Include the review rate so a model that sends many cases to a person is not presented as fully automated. Keep acceptance rules stable while comparing configurations; otherwise the comparison may reflect a changed policy rather than a better model.

Evaluate probability behavior on held-out examples

A sharply peaked output distribution is not proof that the decision is correct. Inspect whether higher-scoring predictions in your labeled set actually have fewer errors. Check ambiguous cases and confident mistakes directly rather than relying only on a single summary.

Choose any automatic-action threshold according to the observed task behavior and the consequences of mistakes. Record the model and question configuration used when selecting it. If either changes, rerun the relevant examples before assuming that an earlier review policy remains suitable.

Compute cost per accepted decision

Start with the charges observed for the tested service and add the work required to obtain acceptable results. Include retries where billed, manual correction, and local infrastructure if you operate the model yourself. Use the provider's current terms rather than copying a price from a different product.

Divide the relevant cost by the number of accepted decisions, while keeping the raw request count visible. A cheap request that regularly needs another request or a person may not be a cheap completed task. If no result was accepted, show that the ratio is undefined instead of displaying zero.

Change one bottleneck and repeat the same check

If input preparation dominates the wait, investigate that path before switching models. If many outputs fail schema checks, inspect the question contract. If long requests are the main problem, compare a task-preserving reduction of input with the original version. Let the observed failure choose the next experiment.

Keep each change small enough to reverse and rerun the same acceptance cases. Separate correctness, latency and cost in the scorecard so an improvement in one cannot conceal a regression in another. Do not declare success because a single example suddenly looks faster.

Turn the benchmark into a release decision

A useful result is a clear statement that a named configuration meets the requirements for a named task under recorded conditions. Keep the worksheet, accepted cases and failure examples with the release record. That makes the evidence useful when the provider or application changes.

The existing Clef Flash playground can help explore the workflow, while this article explains what you need to measure. Do not treat the presence of an embedded demo as proof of the official speed figures. Ship a decision path only when you can show its useful result and its behavior when a request fails.

Try it here

Run this through the model

Paste your own state below and define a typed question. The frame is the live Clef-Flash demo; the panel on the playground page calls the API directly.

Clef-Flash community-hosted

Community-hosted demo of the 9B model. Runs live in your browser session.

Open the playground

Try the decision workspace

See pricing

Prefer code? The API takes the same state and schema.

Frequently asked questions

Did this site independently measure 38.8 ms?

No. That is a Cloudflare-reported result. This article does not claim a new independent model benchmark.

Why does my application take longer than the reported model timing?

Your measured path may include network transit, queueing, preparation, validation and retries. Record the timer boundaries and compare like with like.

Should failed requests be removed from the report?

Keep failure counts and durations visible. You can separately summarize successful requests, but that must not hide the complete task's failure rate or waiting time.

Does the worksheet run Clef Flash?

No. It summarizes observations you enter. Model execution belongs to the actual playground or configured API endpoint.

Can I rerun the endpoint check myself?

Yes. The method and the fixed request are in the release check described above; run it against an endpoint you control. This article still publishes no independent result of its own.