Apache-2.0 · OSS alpha · v0.1.1
recall sits in front of any OpenAI- or Anthropic-compatible proxy and answers one question on the hot path: have we already semantically answered a prompt close enough to this one? On a hit it replays a stored completion without ever calling the model. One static binary, no external vector database, no Python.
Hit — served from memory
113 µs p50
Full get(): exact-hash shortcut → embed → ANN search → threshold decide. One process, zero network hops, no GC pause. p99 ~258 µs.
Miss — goes upstream
the model’s latency
recall hands the caller back the query vector so the fresh completion is stored for next time. You pay for the call, once.
What sits between them
one threshold
A similarity cutoff. Set it too loose and the cache serves a wrong answer; too tight and it never hits. §02 is about not picking that number by hand.
Measured on a dev laptop with the production static embedder
(model2vec/potion-base-8M), 2000 iterations, in-process with no
network and no external services. Your absolute numbers will differ —
the shape is the point, and recall bench reproduces it.
§01 — the hot path
recall bench splits the lookup into its two terms so you can
see which one you are actually paying. It is not close.
full hit, 113.1 µs p50 — model2vec/potion-base-8M
111.5 µs query → vector (embedder-bound) ~1.6 µs ANN search + threshold decide 0 network hops
That ratio is the whole architectural bet. A typical semantic cache pays a round trip to a model server or an embedding API, and a round trip to a vector database. recall removes both by running a static embedder and the index in the same process — so the only thing left to optimise is the embedder, and the only thing left to choose is the threshold.
| embedder | hit-rate | embed p50 | lookup p50 | lookup p99 |
|---|---|---|---|---|
| hash-v1 (default stub) | 50.0% | 10.6 µs | 22.8 µs | 38.1 µs |
| model2vec/potion-base-8M | 100.0% | 111.5 µs | 113.1 µs | 257.5 µs |
hash-v1 embedder is a deterministic blake3 stub. It
is fast because it does not capture semantics — on the same workload it
reaches 50% hit-rate against model2vec's 100%, and that 50% is
just the exact-match half. Latency and quality are set by the same
choice. The number worth quoting is the slower one: 113 µs with a real
embedder.
§02 — the decision boundary
A single static cosine cutoff — the commonly documented
0.8 — is wrong across a real embedding space. Different
namespaces have different densities, and the cost of being wrong is not
symmetric: a missed hit costs one model call, a false hit serves
somebody the wrong answer.
So the adaptive policy targets an operator-chosen false-hit rate
rather than a similarity number, and learns a separate cutoff per
namespace. It is off by default — --policy adaptive
— and recall-eval runs static@0.8,
static@best and adaptive over a controllable-density
workload so you can see the hit-rate each reaches at a fixed false-hit
budget rather than taking the claim on trust.
The counter-metric
A wrong hit is the failure mode that grows with context size, because a single pooled embedding of a megaprompt is lossy. --verify-sample 0.1 re-calls the upstream for a fraction of hits and compares the served answer to a fresh one.
Only at temperature 0
That check is only meaningful when a sampled model would agree with itself. A non-zero mismatch rate means the threshold is too loose for that traffic — tighten --tau, or turn on verify-on-hit.
§03 — before you believe any of it
How fast a hit is has nothing to do with what you save. Savings are
hit_rate × avg_tokens × price — so on unique,
large-context traffic they are near zero no matter how quick the cache is.
Measure that first, on your own log, before anything else here matters.
Input and output are counted separately on purpose. Output costs roughly three to five times input, so a single lumped number mis-prices a skewed mix — and a million-token-context workload is almost all input. Per hit the saving scales with prompt size: one hit on a 1M-token context credits about 1M input tokens. The tilt is big per hit, but only on prompts that actually recur.
§04 — a result we would rather not print
Brute force is exact but linear. The optional pure-Rust HNSW index is
sublinear and 14–25× faster at 50,000 entries. It is also
the case that its recall@1 ≥ 0.98 gate is validated at
dims=32 — and collapses at the width a real deploy
actually runs.
| dims | corpus | brute p50 | hnsw p50 | speedup | recall@1 |
|---|---|---|---|---|---|
| 32 | 10,000 | 566 µs | 85 µs | 6.66× | 0.9960 |
| 32 | 50,000 | 4,470 µs | 181 µs | 24.64× | 0.9860 |
| 256 | 10,000 | 1,563 µs | 400 µs | 3.91× | 0.5900 |
| 256 | 50,000 | 10,607 µs | 733 µs | 14.48× | 0.2560 |
256 is exactly the width of potion-base-8M, the bundled
static embedder — so that bottom row is the regime a real deploy
lands in. Raising --top-k does not recover it: the true
neighbour is never visited, so this is search breadth and graph degree,
not ranking.
The honest read is not “HNSW is broken”. This is measured on
random unit vectors, which are the worst possible input for any ANN
index — no cluster structure, every point near-equidistant. Real
embeddings cluster, so recall@1 on real data is materially higher. What is
true is that the current ef/M configuration was
tuned for the low-dimensional unit test and its gate has not been
validated at production width or scale.
--index hnsw.
Before you do that at scale, run recall ann-bench for your own
dimensions and corpus size and raise ef_search/M
until recall@1 holds. The measurement that produced this table ships in
the binary.
§05 — where it actually is
The MVP loop — embed, ANN search, threshold decision, hit or miss
— runs end to end, and the proxy caches both OpenAI
/v1/chat/completions and Anthropic /v1/messages,
each in its own namespace partition, streaming and non-streaming.
Landed in M2
redb store, with cross-restart lookup rehydration — reopening rebuilds the index and exact-map, so cached hits survive a restart, not just the blobsOpt-in, with a caveat
--index hnsw) — see §04 before enabling it at scale--policy adaptive)Getting it
Built from source. There is no published crate — the name recall on crates.io belongs to an unrelated flashcard tool — and no prebuilt binary or image yet.
The quality trade, stated
potion-base-8M scores 51.08 on MTEB against all-MiniLM-L6-v2's 55.93 — about 91% of the quality, at roughly 8 MB and microsecond encodes. That is the trade the whole latency story rests on, and it is a real trade.
Competitor figures quoted in the project's benchmark document are published numbers, not run on this hardware. They are there to frame the architecture — removing the model-server round trip and the vector database round trip — not to claim a head-to-head win on identical inputs.
Apache-2.0
Every number on this page comes from a command that ships in the binary. Run them on your own traffic; the answer may well be that a semantic cache is not what your workload needs.