A Foundation Model Choice Is an Architecture Decision

The question "Which model should we use?" often starts too late.

Before a team compares capability scores, it has usually made a more consequential decision: where will user data go, who operates the inference path, which controls can the product change, and what happens when the provider changes the behavior behind an API?

These are architecture decisions. A highly ranked model can still be impossible to use for a healthcare workflow with a strict data boundary, an offline application, a product that requires token-level confidence, or a service that cannot tolerate a provider outage. Conversely, a self-hosted model can be technically possible but economically reckless when a small team needs global reliability next month.

Model selection becomes clearer when it is treated as a sequence of constraints and tradeoffs rather than a search for a universal winner.

Filter on Constraints Before Comparing Quality

Not every model attribute is equally negotiable. A team can often improve a prompt, add retrieval, tune a smaller model, or change a response template. It usually cannot patch an unacceptable license, make an external API become on-device software, or reconstruct undisclosed training-data lineage after launch.

Treat those as hard attributes. Filter candidates on them before a benchmark, a model arena, or a polished demo gets a vote.

Typical hard constraints include:

• where prompts, attachments, outputs, and logs may be processed and retained;
• the model license and restrictions on commercial use, redistribution, fine-tuning, or output handling;
• regional availability, procurement requirements, and contractual data-processing terms;
• whether the product must run offline, on-device, or inside a private network;
• required input and output modalities, context limits, and supported languages;
• a maximum latency or per-request cost that the workflow cannot compromise on;
• the degree of safety behavior and content control the product is permitted to configure.

The terms "open source," "open weight," and "available to download" should not be treated as synonyms. Weights can be inspectable while the training data remains unknown. A permissive-looking license can still impose conditions that do not fit a product. A hosted service can offer strong contractual protections but remain unsuitable for a particular residency rule. Read the current model and provider terms with the people accountable for legal and security decisions; model branding is not a compliance review.

This ordering prevents a common failure mode: a team spends weeks proving that an attractive candidate performs best, then discovers it cannot cross the data or deployment boundary that the product requires.

A Model API Buys Leverage and Creates a Dependency

A hosted model API is often the fastest route from a narrow product idea to evidence. The provider owns model serving, capacity planning, hardware failures, many performance improvements, and some safety mechanisms. The application team can focus on the user workflow, prompts, retrieval, tools, and interface.

That leverage is real. It is particularly useful when the product needs a strong general-purpose model, elastic demand, multiple regions, or capabilities that would be expensive to operate independently. An API may also expose features such as structured output, tool calling, content moderation, batching, or cached prompts that simplify an application design.

It is still a dependency with a roadmap. A provider may retire a model, alter a safety behavior, adjust rate limits, change pricing, introduce a new API shape, or route requests differently. An upstream upgrade can shift the tone or tool-use behavior of a feature even when application code has not changed.

Design for that reality:

type ModelRequest = {
  systemInstructions: string
  messages: Array<{ role: 'user' | 'assistant'; content: string }>
  responseSchema?: object
  maxOutputTokens: number
}

type ModelResponse = {
  text: string
  finishReason: 'complete' | 'length' | 'safety' | 'tool-call' | 'failed'
  usage: { inputTokens: number; outputTokens: number }
  modelVersion: string
}

interface ModelGateway {
  generate(request: ModelRequest): Promise<ModelResponse>
}

This kind of boundary should not become an elaborate abstraction built for imaginary providers. It is valuable when it centralizes the product's actual invariants: authorized context construction, timeout policy, model-version logging, schema validation, rate-limit handling, and a tested fallback. The goal is to prevent one vendor-specific response shape from quietly spreading through the UI and business logic.

Self-Hosting Moves the Boundary, Not the Responsibility

Deploying open weights can keep inference within a controlled environment. That can be essential for sensitive data, disconnected operation, customized serving, or a product that needs low-level access unavailable from an API. It also allows a team to choose when and how safety policies are applied rather than accepting a provider's fixed behavior.

Control comes with an operating burden. Someone must own capacity, GPU scheduling, model loading, quantization, autoscaling, rollouts, incident response, observability, and abuse prevention. A model that runs well on a developer laptop can behave very differently under concurrent load, long prompts, or a region failure.

There is also a security distinction worth preserving. Keeping data in a private environment does not automatically make the system private. Logs, embeddings, traces, backups, support access, prompt-injection paths, and model outputs need the same retention and authorization controls as any other sensitive service. Self-hosting changes the locations where those controls are enforced; it does not eliminate the controls.

On-device deployment sharpens the tradeoff further. It can improve privacy, offline availability, and perceived latency by keeping data near the user. It also constrains model size, battery use, memory, bundle delivery, and update strategy. A frontend or mobile team must decide how a model is downloaded, versioned, verified, and rolled back without turning a product update into a multi-gigabyte surprise.

Data Lineage Is a Product Risk, Not a Model Trivia Question

Users and organizations increasingly ask what data informed a model and what happens to the information they submit. The answer may affect procurement, intellectual-property policy, user trust, and whether a feature can be used with customer code or internal documents.

No model choice makes lineage simple. A proprietary provider may disclose only broad information about its training sources. An open-weight release may enable deeper inspection of the artifact but still lack a complete dataset history. A team building or fine-tuning a model has more control over its training inputs, but then owns consent, provenance, retention, and removal obligations for that data.

Keep two questions separate:

1. What can enter a request today? This is an application data-flow and processor agreement question.
2. What knowledge or rights shaped the model? This is a provenance, licensing, and policy question.

Answering the first does not settle the second, and a claim of transparency about the second does not authorize arbitrary customer data in the first. Product copy, privacy notices, administrator controls, and engineering logs should describe the real boundary rather than the most reassuring version of it.

Safety Controls Are Part of the Interface Contract

Hosted providers may block or alter certain outputs. That can be a valuable baseline for harmful-content prevention, but it can also conflict with an intended use case. A game character system, for example, might need fictional dialogue that triggers an overly broad content filter. A compliance feature may need to quote a user's unsafe request in order to explain why it cannot be completed.

The wrong response is to assume either that a provider's guardrails are sufficient or that removing them is the path to product freedom. The product needs its own policy. Define what the user can request, what evidence the model may receive, what content requires transformation or refusal, and where a human must intervene. Then test how the selected model and deployment path behave against that policy.

This is visible in the frontend. A generic error such as "the model failed" is not enough when the request was blocked by a safety rule, timed out, exceeded a quota, or returned an invalid tool proposal. Those states imply different next actions and different privacy expectations.

type AssistantResult =
  | { status: 'ready'; draft: string; sources: Source[] }
  | { status: 'needs-review'; draft: string; reason: string }
  | { status: 'cannot-help'; message: string }
  | { status: 'unavailable'; retryAfterMs?: number }

An honest UI protects users from interpreting an implementation constraint as a factual answer or a product decision.

Cost Has a Shape

An API's price per input and output token is easy to see. The total cost of a workflow is not. It includes retries, abandoned streams, retrieval, tool calls, observability, evaluation, cache misses, support, and the engineering work needed to make failures recoverable.

Self-hosting shifts the shape again. The marginal cost of another request may be low when capacity is already paid for, but idle hardware, peak provisioning, model operations, and specialized staff are real costs. A model deployment that looks cheaper at a high projected volume may be more expensive while demand is uncertain or traffic is spiky.

Compare representative workflows, not only a single short prompt. Record input size, generated length, tool use, peak concurrency, and the latency that the user actually experiences. Then include a cost for the fallback and failure behavior. An assistant that must fall back to a premium model for hard cases is not priced by its cheapest successful response.

Choose a Model Strategy That Can Change

The first serious release should make future choices easier, not lock the product to a guess. Keep an evaluation set that represents the user jobs, source documents, risk cases, languages, and interaction paths that matter. Record the model version and configuration with important requests. Measure quality, cost, and latency separately. Test provider upgrades and fallback behavior before they become production incidents.

There is no permanent endpoint in the API-versus-open-weights decision. Capabilities change, policies change, costs change, and a product's evidence boundary can become more or less demanding over time.

The useful question is not which model is best in the abstract. It is which model strategy gives this product the required capability while preserving the privacy, control, resilience, and economics that its users and operators can actually support.