Dataflow Through Databases and Services: Why Compatibility Is a Running System

The most useful way to think about dataflow is not as a transport problem, but as a compatibility problem that keeps changing shape.

Sometimes data is written to a database and read later by the same application in a newer version. Sometimes it is copied into archives or warehouses. Sometimes it moves through REST APIs, RPC systems, queues, or service meshes. In each case, the writer and the reader are separated by time, deployment history, and often by language or team boundary.

That is why the same encoded record can need to survive many different readers.

This article looks at the major places where data flows in a real system and the constraints each one creates: databases, archival storage, services, and the networking infrastructure that sits between them.

A Database Record Is a Message to Your Future Self

When one process writes to a database and later reads the same record back, the cleanest mental model is that the write is a message to a future version of the same application.

That framing is useful because the future version may not be identical to the current one.

Rolling deploys mean newer and older code can overlap. Long-lived records mean old data can remain in its original encoding for years. Multiple instances of the same service may not upgrade at the same time.

So the database has to tolerate this reality:

• newer code reading older data,
• older code reading newer data,
• missing fields that need defaults,
• old formats that cannot be rewritten immediately.

If that sounds like compatibility again, that is because it is.

Data Outlives Code

One reason schema evolution becomes so important is that code gets replaced much faster than data does.

A server release may take minutes. A stored record may stick around for years.

That means a single database can contain values written at very different times, and those values may not all share the same schema version. The system cannot assume that every row is a fresh record in the newest format.

In practice, this leads to a few patterns:

• adding nullable columns or fields instead of rewriting everything at once,
• filling in missing values on read,
• performing migrations asynchronously,
• rewriting historical data only when it is worth the cost.

This is also why archival dumps and snapshots are often encoded using the latest schema available at export time. If you are copying the data anyway, you might as well make the copy consistent.

Archival Storage Wants a Different Kind of Encoding

Once data is written once and then treated as immutable, the storage goal changes.

At that point, the system is often better served by an encoding that is compact, schema-aware, and convenient to process in bulk. The same data that is awkward as a live row-by-row update may be ideal as an object container file or a warehouse-friendly columnar export.

That is why archival paths and analytics paths frequently reuse the same core encoding ideas but shape them differently.

The important point is that archival data is still a compatibility problem. It just has fewer writes and more readers.

Services Expose Application-Specific Contracts

When data flows through services, the network layer adds a second dimension of compatibility.

The web browser, a mobile app, a backend service, and a third-party consumer may all be clients of the same service. They may also be running different versions of their code.

That means service APIs need to think about compatibility in a direction-specific way:

• request formats need to remain understandable by the server,
• response formats need to remain understandable by the client,
• versioning must work even when the provider does not control every consumer.

This is why REST, OpenAPI, gRPC, and Avro RPC all matter. They are not just transport choices. They are ways of formalizing the contract between systems that evolve independently.

RPC Is Not a Local Function Call

Remote procedure calls are attractive because they promise to make distributed systems feel local.

That promise is misleading.

A local function call is predictable. A network call is not.

The network can lose packets, delay responses, time out, or split a request from its response. The remote service may be slow, overloaded, or unavailable. Retrying may repeat an operation that already succeeded but whose response was lost.

It also costs more to move data across a network than to pass a reference in memory. Arguments must be encoded. Responses must be decoded. Clients and servers may use different languages with different numeric and string semantics.

That is why making a remote service look exactly like a local object usually produces confusion instead of clarity.

REST is popular partly because it leans into the reality that HTTP requests are network interactions, not function calls.

Web Services Need Explicit Versioning

Web services are one of the most common places where dataflow and compatibility meet.

OpenAPI documents, RESTful endpoints, gRPC schemas, and other service definitions exist because clients need to know what to send, what to expect back, and how to evolve with the server over time.

The versioning problem is not simple:

• URL versioning is easy to understand but can fragment APIs,
• header-based versioning is flexible but less visible,
• per-client version configuration can help large platforms but requires operational tooling.

There is no perfect universal answer. There is only the requirement that the server and client agree on which contract they are speaking.

For frontend engineers, this often shows up in a very practical way. A browser app may keep a tab open for hours while the backend deploys several times. If the response shape changes in a breaking way, the user feels it immediately.

Load Balancing and Service Discovery Are Part of Dataflow

The network path is not just client to server. It is client to discovery to load balancer to service instance, and sometimes through a sidecar or mesh on both sides.

That extra infrastructure exists because services move, scale, fail, and roll out independently.

Service discovery answers a simple question: where is the service right now?

Load balancing answers a different question: which healthy instance should handle the next request?

Service meshes add policy, routing, observability, and sometimes encryption across that traffic.

These layers are useful because the topology is dynamic. A server can be replaced, a region can shift load, or a deployment can bring up a new version alongside the old one. DNS can help, but it often changes more slowly than service instances do. Discovery systems and meshes exist to handle a more active environment.

The Real Compatibility Problem Is Operational

It is easy to describe compatibility as a data-format problem. In production, it becomes an operational problem.

If a service provider does not control all of its clients, it may need to support old API versions for a long time. If a database schema is changed under a live application, the system may need to handle mixed records during the transition. If a deployment uses rolling upgrades, new and old code paths must coexist safely.

That means the architecture needs to support:

• partial rollouts,
• mixed-version traffic,
• replayable or idempotent operations when retries happen,
• safe defaults for missing or extra data,
• observability into which version is doing what.

These are not edge-case concerns. They are the normal shape of a changing system.

What Frontend Engineers Should Carry Forward

Frontend engineers feel dataflow problems as UX problems.

If a save request succeeds but a read comes from a stale replica, the UI looks wrong. If an API response shape changes between deploys, the app can break on refresh. If a retry repeats a non-idempotent action, users may see duplicate work or duplicate side effects.

The right response is not to pretend the network is local. It is to design the client around eventual reality:

• optimistic updates where appropriate,
• clear loading and syncing states,
• retries only when the server can safely deduplicate them,
• explicit freshness indicators when data may lag,
• tolerant decoders that can handle older and newer payloads.

The more the system evolves, the more the UI becomes a compatibility surface.

Conclusion

Dataflow is where evolution becomes visible.

Databases, archival copies, APIs, RPC frameworks, load balancers, and service discovery layers all create slightly different expectations about who writes the data, who reads it, and how long the contract must survive.

Once you see those expectations clearly, the system becomes easier to reason about. Compatibility is not a one-time migration task. It is a running property of the architecture.