Semantic caching is usually introduced as a performance move. Instead of keying a cache on an exact string, you key it on meaning: embed the request, find a prior request that is close enough, and return the stored answer. It cuts latency and cost, and for expensive inference paths the savings are real.
But reuse by meaning changes what the cache is. A traditional cache returns the same answer to the same question. A semantic cache returns a stored answer to a different question it judged similar enough. The moment one caller receives an answer that was produced for another caller, the cache has made a trust decision. That is the core claim: a semantic cache is an authorization boundary, and treating it as pure infrastructure is how systems leak.
A cache hit is an implicit grant
When request B gets a cached response that was generated for request A, B inherits A’s context. Whatever data, tenant scope, or permission set shaped A’s answer is now being served to B.
If A and B belong to the same user under the same policy, that is fine. If they do not, the cache just moved information across a boundary that no policy engine was asked to approve. Nothing in the embedding distance measured that. Cosine similarity has no opinion about identity, tenancy, or clearance.
The failure mode is precise: two requests that are semantically close but authorized differently. Similar text, different entitlement. The cache is optimized to find exactly those matches, so the risk is not incidental. It is structural to how the cache works.
Similarity and authorization are two different tests
The fix is not to abandon semantic caching. It is to stop conflating two questions that feel like one:
- Is this request close enough to reuse an answer? That is a similarity test, and embeddings answer it well.
- Is this caller entitled to the answer we would reuse? That is an authorization test, and embeddings say nothing about it.
A safe semantic cache runs both. Similarity gates whether an answer is a candidate for reuse. Policy gates whether this caller may receive it. A hit requires passing both, not just the first.
In practice that means the cache key is not the embedding alone. It is the embedding plus the authorization context that produced the original answer: identity or role, tenant or scope, data classification, and any policy inputs that changed what the model was allowed to see. Answers generated under different authorization contexts live in different partitions, even when their text is identical.
The cached answer carries provenance
An answer is not a neutral string. It carries the provenance of the context that made it: which sources were retrieved, which tools were called, which policy decisions applied. When a semantic cache serves that answer to a new caller, it is replaying that provenance without re-checking it.
So reuse has to be a provenance-aware decision. The question is not only “have we answered something like this before,” but “is this caller entitled to an answer built from that context.” If the provenance included a source the new caller cannot access, the hit is a violation even when the text looks harmless.
This is where semantic caching rejoins the rest of the platform. The same policy boundaries that govern the live inference path have to govern the cached path, or the cache becomes a way to route around them.
Where the check belongs
The authorization check belongs in the mediation layer, alongside the cache, not inside application code that calls it. If every caller has to remember to re-authorize a cache hit, some caller eventually will not. Putting the check next to the cache keeps the boundary in one place and makes reuse observable: every hit records which policy allowed it, which is exactly the evidence a trust posture depends on.
That also keeps the optimization honest. A semantic cache that logs authorized reuse is faster and auditable. A semantic cache that reuses by similarity alone is faster and quietly wrong.
Closing
Semantic caching is worth doing. It is one of the few inference optimizations that pays for itself immediately. But the thing that makes it powerful, reuse by meaning across near-matches, is the same thing that makes it a boundary. The cache decides who receives whose answer.
Treat it accordingly: similarity to find candidates, policy to authorize them, provenance to keep the check honest. A cache hit should be a decision the system can explain, not an accident of distance in embedding space.