Encoding and Evolution: Why Data Formats Outlive the Code That Wrote Them

Data Systems

Most teams do not think about data encoding until a deploy exposes the fact that their old code and their new code are both reading the same data at the same time.

That is when encoding stops being a low-level implementation detail and becomes an architecture concern.

The reason is simple: applications do not live in one version for very long. Rolling upgrades, cached data, archived snapshots, mobile clients that update slowly, and background jobs that keep running older code all create a world where old and new representations coexist. In that world, a data format is not just a way to turn objects into bytes. It is a contract about what future readers are allowed to understand.

This article focuses on the parts of that contract that matter most in real systems: why built-in serialization usually ages badly, why textual formats are useful but limited, and why backward and forward compatibility should be treated as first-class design constraints rather than afterthoughts.

Encoding Is a Translation Problem, Not a Storage Problem

In memory, programs usually work with objects, arrays, maps, and references. On disk or over the network, they need a flat sequence of bytes.

Those two representations are not interchangeable. A pointer that makes perfect sense inside one process means nothing to another process. An object graph that is easy for the CPU to traverse in memory may need to be flattened into a self-contained message before it can be stored or sent.

That translation step is encoding on the way out and decoding on the way back in.

This matters because the shape of the encoded bytes determines who can read them later. If the format is tied too tightly to one language runtime, one object model, or one release of the codebase, the data becomes brittle fast.

Built-In Serialization Usually Optimizes the Wrong Thing

Languages often provide built-in encoding helpers such as Java serialization, Python pickle, or Ruby Marshal. They are attractive because they make saving and restoring objects feel effortless.

The problem is that convenience is not the same as longevity.

Built-in serialization often has three major weaknesses:

  • it is tied to one programming language,
  • it may require decoding arbitrary classes, which creates security risk,
  • it usually treats versioning as an afterthought.

That last point is the one that hurts most in production. If the encoding cannot tolerate a field being added, removed, renamed, or interpreted differently later, then the format becomes a migration trap.

For transient use inside one process or one short-lived cache, that may be acceptable. For durable storage or public APIs, it usually is not.

JSON, XML, and CSV Are Portable, but They Still Make Tradeoffs

Textual formats are popular for a reason. They are easy to inspect, widely supported, and good at crossing language boundaries.

JSON in particular fits the mental model of frontend and JavaScript engineers well because it resembles the data structures we already use in application code:

const profile = {
  userName: 'Martin',
  favoriteNumber: 1337,
  interests: ['daydreaming', 'hacking'],
}

That familiarity is useful, but it should not be confused with completeness.

Text formats bring several real limitations:

  • XML is verbose and easy to overcomplicate.
  • CSV has no nested structure and no built-in schema.
  • JSON does not distinguish integers from floating-point numbers, which matters once values grow beyond JavaScript's safe integer range.
  • JSON and XML do not naturally represent raw binary data, so teams fall back to Base64 and pay a size penalty.

That is why a format can feel simple on the surface and still create awkward edge cases in the system behind it.

If your app uses JavaScript end to end, large numeric identifiers are a good example. A number may be perfectly valid in a database or backend language and still become lossy once it passes through a JSON parser in the browser.

The format is portable. The semantics are not always portable unless you make them explicit.

Schema Flexibility Is Useful Only if You Handle Compatibility Explicitly

One of the biggest reasons teams choose schema-on-read or schemaless storage is flexibility.

That flexibility is real. It lets new fields appear without forcing every old record to be rewritten immediately. It lets application code evolve while historical data still remains in the system.

But flexible schemas do not remove the need for compatibility rules. They move that responsibility into the application.

Two directions matter:

  • Backward compatibility means newer code can read data written by older code.
  • Forward compatibility means older code can read data written by newer code.

Those are not abstract definitions. They describe whether a rolling deploy works without breaking the live system.

If your backend deploys new writers before all readers have been upgraded, older readers will encounter newer data. If your clients update slowly, your server may have to serve both old and new shapes for a long time.

That is why compatibility is not a nice-to-have. It is the thing that keeps mixed-version systems running.

Versioned Data Is the Normal Case, Not the Edge Case

It is tempting to think of data as if it were rewritten all at once into a single latest format. Real systems rarely behave that way.

Data outlives code.

Some rows were written five minutes ago. Others were written five years ago. Some records live in a live database. Others live in backups, exports, data lakes, event logs, or cache layers that were never fully rewritten.

That means one database or one API endpoint can easily contain multiple historical versions of the same shape at once.

For that reason, schema changes must usually be designed so that:

  • new code can tolerate old data,
  • old code can ignore new data,
  • missing fields can fall back to sensible defaults,
  • obsolete fields can be retired without breaking current readers.

That is the actual work of evolvability.

Frontend Engineers Feel This First in the UI

Frontend code tends to expose compatibility mistakes quickly because the browser often lags behind the server.

A mobile app may not update for weeks. A browser tab may stay open across several deploys. A cached response may outlive the code that created it. A user may refresh a page while one service instance is still on the old version and another is already on the new one.

Those are not rare incidents. They are ordinary production conditions.

If your UI crashes when a field is missing, or silently corrupts a numeric value, or cannot tolerate an extra property in the payload, the issue is not just client-side robustness. It is a broken compatibility contract.

The useful habit is to treat API payloads and stored records as future-facing data. Ask not only, "Can I read this now?" but also, "Will the next version of the app still be able to read it?"

The Right Question Is Not Schema or No Schema

The right question is whether the system can evolve safely.

Sometimes a loose format is fine, especially for transient data or for inputs whose shape is inherently messy. Sometimes a stricter contract is better because it gives better validation, better tooling, and fewer surprises.

The important thing is not to confuse flexibility with safety.

If you need a format that survives rolling upgrades, mixed client versions, long-lived storage, and changing product requirements, then compatibility becomes part of the architecture, not an implementation detail hidden inside serialization code.

That is the real lesson of encoding and evolution: the bytes do not just represent data. They represent your willingness to change the system without breaking the past.