API Gateways, Service Discovery, and Service Mesh: The Traffic Control Layer of Modern Systems

Architecture Patterns

As soon as you have multiple services, one uncomfortable fact shows up: requests need choreography.

How does the browser know where to send traffic? How do services find one another when instances appear and disappear? Where do auth, routing, retries, and observability live? These are not one problem. They are three adjacent problems solved by three different layers: API gateway, service discovery, and service mesh.

API Gateway: One Front Door, Many Backends

An API gateway is the entry point that sits between clients and internal services. It is often the layer your frontend actually talks to.

Responsibilities commonly placed at the gateway:

  • routing requests to the correct service,
  • authentication and authorization,
  • request aggregation,
  • rate limiting,
  • response shaping,
  • and sometimes caching.

For frontend teams, the gateway is often a blessing because it reduces client complexity.

// Frontend talks to one boundary
const dashboard = await fetch('/api/dashboard').then((res) => res.json())

// Gateway fans out internally
// - profile service
// - notifications service
// - feature flags service
// - billing summary service

That is basically a backend-for-frontend pattern. The browser sees one contract. The gateway absorbs service sprawl.

The Gateway Is Not a Magic Dumpster

Teams often overuse the gateway by stuffing business logic into it. That turns the gateway into a second monolith sitting in front of microservices.

Good gateway logic:

  • auth checks,
  • routing,
  • protocol translation,
  • lightweight aggregation.

Dangerous gateway logic:

  • domain rules,
  • long-lived orchestration,
  • deep data transformation,
  • hidden coupling to internal schemas.

If every product change requires gateway surgery, the gateway stopped being infrastructure and became an architectural bottleneck.

Service Discovery: How Services Find Live Instances

In dynamic environments, services cannot hardcode addresses of other services because instances come and go during deployments, autoscaling, and failures.

Service discovery solves that.

Instead of calling a fixed IP, a service asks a registry for the current healthy instances of another service.

checkout-service wants inventory-service
  -> asks registry for healthy inventory instances
  -> receives list of endpoints
  -> picks one and sends request

Common models:

  • Client-side discovery: the caller asks the registry and chooses an instance.
  • Server-side discovery: a load balancer or proxy asks the registry and forwards traffic.

Frontend apps rarely talk to registries directly, but they depend on this working correctly because unstable discovery becomes intermittent latency and random failures at the edge.

Service Mesh: Networking Concerns Moved Out of Application Code

A service mesh is an infrastructure layer, usually implemented with sidecar proxies or ambient networking, that handles cross-cutting service-to-service concerns:

  • mTLS
  • retries
  • timeouts
  • traffic policies
  • observability
  • canary routing

The reason meshes exist is simple: if every service team implements retries, TLS, tracing, and routing differently, the system becomes impossible to reason about.

With a mesh, the application code can stay focused on business behavior while networking policy lives in the platform layer.

What Frontend Engineers Need to Care About

Even if you never configure a service mesh yourself, these layers shape your API experience:

  • Gateway aggregation determines how many round-trips your page needs.
  • Discovery and load balancing determine latency variance and failure frequency.
  • Mesh retry behavior determines whether a request fails fast or stalls awkwardly.

That last point matters. Automatic retries can improve resilience, but they can also amplify latency.

Frontend timeout: 3s
Gateway timeout: 5s
Mesh retries: 2 attempts at 2s each

Result: the client gives up while the backend is still retrying.

If those layers are not coordinated, the user experiences confusing failures: duplicate spinners, late responses, and error states that appear after the action already looked complete.

Choosing the Right Tool for the Problem

Use an API gateway when you need a clean client-facing boundary.

Use service discovery whenever service instances are dynamic and callers need a reliable way to find healthy endpoints.

Use a service mesh when service-to-service networking policy is becoming inconsistent, duplicated, or operationally risky across teams.

Do not use a mesh to avoid thinking about service boundaries. Do not use a gateway as a place to hide all product complexity. And do not talk about service discovery as if it is interchangeable with either of the other two.

Conclusion

API gateways, service discovery, and service meshes all deal with traffic, but they solve different coordination problems at different layers. The gateway provides a stable front door. Discovery answers "where is the service right now?" The mesh standardizes how service-to-service traffic behaves.

For frontend engineers, understanding these layers explains why some APIs feel cohesive and fast while others feel fragmented, jittery, and impossible to reason about.