An AI Feature Is Not Ready Until You Can Name Its Failure

An AI feature can look successful long before anyone knows whether it helps.

Picture a support assistant that gives polished answers, gets a few thumbs-up reactions, and is quickly added to a product. Six months later, the team can say how many conversations it had. It cannot say whether it resolved more cases, created more escalations, exposed the wrong information, or trained users to trust answers they should have checked.

That is not a missing dashboard. It is a missing product contract.

Evaluation-driven development applies a familiar engineering idea to AI systems: define how a behavior will earn its place before building the behavior. Test-driven development does not guarantee good software, but writing an assertion before an implementation forces a team to make a claim precise. An AI evaluation plan serves the same purpose. It makes the team state what the feature is meant to accomplish, what it must not do, and what evidence would change a release decision.

The model is then one component in a system that can be measured and improved. It is no longer the feature's entire explanation.

Start With a Job, Not a Capability

"Use AI to help with support" is not an evaluable requirement. It names a technology and a broad department. It does not identify a user, a decision, an authority boundary, or an observable result.

A narrower claim is much more useful:

For routine account-access cases, prepare a draft response from the customer's authorized account state and current policy so an agent can resolve cases with fewer edits without disclosing information from another account.

That statement gives engineering something concrete to evaluate. It has inputs, an expected output, a person responsible for the final action, and a dangerous failure. It also exposes where an ordinary program may be better than a language model. An account identifier should come from an authenticated database query, not from fluent text generation. The model may help explain the result; it should not invent the result.

This framing is especially useful for frontend teams. A chat-shaped interface can hide several different jobs: finding documentation, explaining an error, drafting a message, changing a record, or guiding a multi-step task. Each has a different completion signal. Treating them all as "good conversation" produces a metric that is easy to raise and hard to trust.

Before selecting a model, write down:

• the user decision or task the feature changes,
• the evidence it is allowed to use,
• the cases it should handle and the cases it should escalate,
• the consequences of a wrong, unsafe, late, or overconfident result,
• the product outcome that would make continued operation worthwhile.

Those choices do not constrain innovation. They prevent a demonstration from becoming a production dependency without an owner.

Turn Quality and Task Completion Are Different Signals

Generative systems invite turn-by-turn evaluation because each reply is visible. A reviewer can ask whether a response is clear, factual, friendly, or formatted correctly. Those checks matter, but they are not necessarily what the user came to accomplish.

Consider an assistant helping an engineer debug a failed deployment. It might ask a sensible question about the runtime version, summarize a log accurately, and give three plausible next steps. Every turn can look good while the engineer still cannot identify the root cause after twenty messages.

Turn-based evaluation asks whether an individual output meets a standard. Task-based evaluation asks whether the system helped complete the job, often including the number of turns, corrections, handoffs, and time required.

Neither replaces the other. A task can be completed through an unsafe intermediate response, so output-level checks still matter. A system can also generate attractive replies that never improve the work. The evaluation needs both perspectives.

Question	Useful signal	A misleading shortcut
Is this response usable?	Groundedness, format validity, safety, clarity	A single overall quality score
Did the user finish the job?	Completion, correction rate, escalations, time to resolution	Counting messages or clicks
Is the feature worth operating?	Measured benefit against cost, risk, and maintenance	Assuming usage proves value

For a coding assistant, a task outcome might include whether a proposed change builds, passes the relevant tests, and reduces the next debugging step. For a document assistant, it might include whether users locate the current approved policy without opening several stale pages. The precise metric changes with the product. The discipline is to choose it deliberately.

Translate Quality Into Separately Testable Claims

"Make the answer good" compresses too much. A language model can be fluent but wrong, helpful but unauthorized, accurate but unusably slow, or creative when a fixed policy statement was needed.

Break the contract into properties with distinct failure modes. A knowledge assistant might need:

• domain capability: it can interpret the terminology and request type the feature supports;
• instruction following: it produces the required JSON, headings, language, length, or tool arguments;
• factual consistency: material claims are supported by the supplied, current evidence;
• safety and privacy: it avoids restricted disclosure, harmful advice, and prohibited content;
• interaction quality: its response makes the next user action clear;
• cost and latency: it meets a budget that keeps the workflow usable.

Some requirements are deterministic. A JSON response can be validated against a schema. A tool call can be rejected when its arguments would touch another tenant. A code suggestion can be compiled or tested. Use those checks directly rather than asking a model whether its own JSON or code looks correct.

Other requirements need judgment. A draft may be technically accurate yet omit the one condition that changes a customer's eligibility. This is where a rubric, source evidence, sampled human review, or an evaluator model can help. The important distinction is that the review criterion stays visible. "Supported by the current policy and includes eligibility exceptions" can be debated and audited. "High quality" cannot.

The contract should include negative examples too. For every claim about what the feature does, name what it must refuse, avoid, or hand off. A useful evaluation set contains routine cases alongside ambiguous requests, stale documents, missing authorization, adversarial instructions embedded in retrieved text, and inputs where the right result is "I do not have enough evidence."

A Rubric Is Product Design Written Down

Teams often treat a rubric as an annotation detail to create after the product exists. It is more valuable earlier, because it exposes disagreements that code cannot settle.

Suppose reviewers are asked whether a support answer is helpful. One may reward a warm tone; another may reject any answer that does not provide a next step; a third may care most about policy accuracy. Their disagreement is not noise to average away. It is a prompt to decide what the product promises.

An effective rubric has a small number of observable dimensions, clear pass and fail examples, and an escalation state for genuine uncertainty. It should be tested with people before it is automated. If experienced reviewers cannot apply it consistently, an evaluator model will only make the ambiguity cheaper and harder to see.

That feedback changes implementation. A finding that answers omit policy exceptions might require better retrieval, a different source format, an interface that reveals citations, or a prompt that asks the model to distinguish a rule from an exception. A finding that users do not act on otherwise correct answers might require a different response structure rather than a stronger model.

Latency Is a Product Requirement, Not a Provider Statistic

A response that eventually arrives may still be unusable. Autoregressive models generate output token by token, so latency depends on more than a provider's average request time. Prompt size, generated length, tool calls, retrieval, queueing, and retries all affect what a person experiences.

Measure the timing that matches the interaction:

• time to first token for a streamed explanation,
• time between tokens when a pause makes the interface feel stalled,
• time to complete for a structured result or tool-backed action,
• P90 or P95 latency when the slow tail damages trust more than the median does.

The frontend makes this concrete. A streaming answer can show useful partial content quickly, but it still needs a cancellable request, a stable loading state, and an honest failure path. A form that cannot submit until a model has produced valid structured output may need a timeout and an alternate manual route rather than an endless spinner.

Latency also forces a tradeoff between model quality, response length, and cost. There is no universally correct point on that tradeoff. A background document classification job can wait longer than an agent assisting a customer on a live call. The product owner should decide which constraint is non-negotiable before a benchmark score makes that decision by accident.

Connect Model Quality to a Real Outcome, Carefully

Business metrics are necessary, but they are easy to misread. More daily activity can mean a feature is useful. It can also mean users are trying repeatedly to get a correct answer. Longer sessions can mean engagement or confusion. More drafted support replies can be a productivity improvement or a new queue of responses that need expensive review.

Map the product contract to an outcome with a plausible causal path. For example, a team might test whether a grounded support draft reduces agent edit time while holding policy-violation and escalation rates below a defined threshold. A staged experiment can compare the workflow with and without the feature, while accounting for user mix and changes in demand.

The same care applies to social consequences. Optimizing recommendation engagement can favor compulsive use or extreme material. Optimizing short-term resolution can encourage an assistant to end a conversation before the user understands a safety-critical next step. A metric is not neutral simply because it is easy to calculate.

The strongest evaluation plan makes those tradeoffs explicit. It does not wait for a dashboard anomaly to reveal what the feature was optimizing.

Build the Feature Around Evidence of Value

Evaluation-driven development is not paperwork placed in front of a model API. It is a way to make product decisions reversible and accountable.

Define the user job. Separate output quality from task completion. State the claims the system must satisfy and the failures it must prevent. Design the interface and latency budget around the real workflow. Then connect the result to a meaningful outcome without mistaking activity for value.

With that foundation, model changes, prompt revisions, and new retrieval strategies become experiments against a contract. Without it, each improvement is merely a more convincing guess.