Semantic Similarity Is a Product Decision, Not a Number

An AI feature can return an answer that sounds close to a reference answer and still fail the user.

Imagine a support assistant that responds to a refund question with a concise, friendly explanation of an older policy. Its wording may be a strong semantic match for a previous answer. It may even look more polished than the current policy text. But if the policy changed last week, the answer is wrong in the way that matters.

This is the trap in treating similarity as a generic quality score. Similarity is powerful because it lets a system compare things that do not share the same words: two support tickets, an image and a caption, a product question and a paragraph in an internal handbook. It is limited because the representation and the metric do not know the product decision by themselves.

The practical question is not, "Which score says these strings are alike?" It is, "What kind of likeness would make this workflow safer or more useful?"

That shift turns embeddings and evaluation metrics from mysterious machine learning (ML) utilities into design tools. It also makes their failure modes easier to see.

Similarity Starts With a Representation

Software can compare text literally. That is useful for identifiers, exact commands, hashes, and regulated wording. It fails quickly when the same intent is expressed in different language.

"Can I return this after opening it?" and "Is an opened item eligible for a refund?" have little word overlap, yet they may require the same policy. A semantic system tries to preserve that relationship by converting each item into an embedding: a fixed-length list of numbers that represents learned features of the item.

The individual values are not meaningful labels such as contains-refund-policy. Their value is relational. An embedding model maps items it considers related into nearby regions of a vector space. A search service can embed a query, retrieve nearby document chunks, and give those chunks to an application or language model.

That pattern appears in far more than chat retrieval:

• grouping duplicate bug reports that use different vocabulary,
• finding existing help articles before an agent writes another answer,
• matching product photos to descriptions,
• measuring whether a generated summary preserved the meaning of a source,
• detecting a question that is too far from a feature's supported scope.

Some models learn a shared space for more than one modality. A text prompt about a red running shoe and an image of one can be near each other even though one is language and the other is pixels. This makes text-to-image search and image-based catalog discovery possible. It does not mean the model has verified the color, product variant, rights, or availability. It means the model found a useful correspondence in its representation.

Cosine Similarity Answers a Narrow Question

Once a system has vectors, it needs a way to compare them. Cosine similarity is common because it compares the direction of two vectors while discounting their raw length. Vectors pointing in a similar direction receive a higher score than vectors pointing apart.

That is a sensible default for many embedding models. It is also easy to misread.

Here is an intentionally small TypeScript implementation:

function cosineSimilarity(left: number[], right: number[]) {
  if (left.length !== right.length) {
    throw new Error('Embedding dimensions must match')
  }

  let dotProduct = 0
  let leftMagnitude = 0
  let rightMagnitude = 0

  for (let index = 0; index < left.length; index += 1) {
    dotProduct += left[index] * right[index]
    leftMagnitude += left[index] ** 2
    rightMagnitude += right[index] ** 2
  }

  return dotProduct / Math.sqrt(leftMagnitude * rightMagnitude)
}

The function has no idea whether a support policy is current, whether the requester is allowed to see it, or whether a close-looking answer contains a dangerous negation. It only measures a geometric relationship established by the embedding model.

That boundary should affect every product decision built on the score.

A high score can mean the candidates discuss the same topic. It cannot, by itself, establish:

• factual correctness,
• legal or policy authority,
• time validity,
• permission to disclose the retrieved content,
• coverage of every condition in a user request,
• absence of a harmful omission.

The last two are especially relevant to generated answers. "Customers can cancel within 30 days" may be close to "Customers can cancel within 30 days unless the item was activated." For the user holding an activated device, the missing clause is the entire outcome.

The Metric Must Match the Job

Semantic retrieval and output evaluation sound similar, but they often need different measures.

For retrieval, the question is usually: did we put the right source material in the candidate set? Cosine similarity against document embeddings is a reasonable tool. The system can improve it with metadata filters, hybrid keyword search, reranking, and a quality check on the final set.

For generated text, a global embedding score may hide important local differences. Metrics such as BERTScore compare contextual token representations, which can make them more sensitive to meaningful word substitutions than a single document-level vector. MoverScore uses an optimal-transport-style comparison to account for how information is distributed across the two texts. These are useful additions when a reference answer exists and paraphrasing is expected.

Neither makes an open-ended task objectively solved. A metric rewards the properties it can see. The following product goals need separate signals:

Product goal	Similarity can help with	What needs another check
Documentation search	Retrieving material about the same task	Access control, revision status, source authority
Support-answer drafting	Detecting a drift from approved examples	Current policy, exceptions, tone, escalation rules
Code explanation	Matching the intended behavior of a snippet	Whether the code compiles, is secure, and fits the repository
Image catalog search	Linking a visual query to likely products	Exact SKU, stock, rights, and safety constraints
Meeting summary	Checking broad coverage of the discussion	Decisions, owners, dates, and sensitive content

This is why a team should write the evaluation contract before choosing a score. If a search result must be both relevant and authorized, relevance alone is not an acceptable definition of success. If a generated invoice summary must preserve amounts and dates, test those facts directly instead of assuming a strong semantic match implies they survived.

Build a Retrieval Evaluation Set Around Decisions

Embedding quality is often evaluated with a benchmark that contains many retrieval, classification, clustering, and similarity tasks. The Massive Text Embedding Benchmark, commonly called MTEB, is a useful way to understand how a general embedding model behaves across such tasks.

It is not a release gate for a product. A model can perform well on a broad public benchmark while losing the distinctions your users depend on: product names, internal abbreviations, local languages, archived versus current documents, or data that uses unusual formatting.

Build a smaller task-specific set alongside any public benchmark. For an internal engineering assistant, include examples such as:

• a question whose answer exists in a current document,
• a plausible question whose closest document is obsolete,
• a request with a tenant or role that must not reach a relevant document,
• two documents that use similar language but establish different procedures,
• an acronym that changes meaning between teams,
• a query in the languages the product actually supports,
• a question where the right outcome is "not enough evidence."

For each example, record more than one accepted result when appropriate. Real retrieval often has several useful documents. Then measure candidate recall: how often did the system include at least one acceptable source in its top results? Review the misses by type instead of accepting one average number.

The review can expose an engineering fix that a model swap would not. A poor result may come from a document chunk split across a table, an index that missed the document's revision identifier, or a client that stripped a product code before search. Similarity metrics reveal the symptom; the surrounding system still owns the cause.

A Threshold Is a Policy, Not a Constant

Applications frequently add a rule such as "only show a result if its score is above 0.82." It feels like a safety boundary. It is actually a policy choice that depends on the model, corpus, language, and cost of the two error types.

If the threshold is too high, users receive unnecessary "no answer" results. If it is too low, the system supplies a confident-looking but weakly related source. Neither error is universally worse.

For a low-stakes discovery feature, a broader candidate set with visible uncertainty may be appropriate. For a workflow that proposes a financial action, a weak match should trigger a request for clarification or a human review. The product should not silently convert a geometric score into authority.

Calibrate the threshold with examples from the intended workflow. Plot or inspect scores for accepted matches, clear misses, and deceptive near-misses. Then re-evaluate when any of these change:

• the embedding model or its version,
• document chunking or preprocessing,
• the corpus's language or domain mix,
• metadata filters and access-control logic,
• the user interface that makes a result look more or less final.

Even the order of operations matters. Filtering retrieved results after a broad semantic search can leak sensitive titles or snippets into logs, caches, or an intermediate response. Apply tenant, role, and retention constraints before a candidate can become context for a model or visible content for a user.

The Frontend Can Preserve or Destroy the Signal

A web interface is where an uncertain similarity match becomes a user decision. Hiding that uncertainty turns an assistive feature into an implied guarantee.

For a documentation assistant, a useful result can show the source title, revision date, and a short excerpt that explains why it was selected. It can offer adjacent matches without pretending they are citations. If the system lacks a strong source, it should say so plainly and provide a route to search or escalate.

This does not require exposing raw similarity scores. Those scores are rarely meaningful to users. It requires exposing the evidence and the product state:

type SearchResultState =
  | { status: 'loading' }
  | { status: 'results'; documents: DocumentMatch[] }
  | { status: 'insufficient-evidence'; query: string }
  | { status: 'restricted'; message: string }
  | { status: 'failed'; retryAfterMs?: number }

Those cases ask different things of a person. "No relevant authorized evidence" should not look the same as a network failure. A restricted document should never be teased as a likely answer. A streamed answer should not hide the fact that its sources are still being gathered or verified.

The interface also produces feedback. A user who opens a cited document, rewrites a query, rejects a recommendation, or marks a suggestion as outdated gives a product-specific signal. Collect it with consent, minimize retained personal data, and avoid training or ranking directly on behavior that reflects unequal access or coercion. A click can mean relevance, but it can also mean confusion.

Similarity Is One Instrument in a Better Evaluation System

Embeddings make a large class of AI features practical. They let a system compare meaning across wording and, in some cases, across media. They are especially valuable for retrieval because they can bring useful evidence into reach before a language model writes anything.

But semantic closeness is not truth, permission, freshness, completeness, or product value. Treat it as a signal with a clear job. Pair it with data-quality checks, authorization, task-specific examples, observability, and an interface that does not overstate what the system knows.

That is the difference between using an embedding score to make a system look intelligent and using it to make a real decision workflow more dependable.