WAN limitations: cost, backhaul and circuit convergence

The load changed, the design did not

Most traditional WANs were designed when branch users mostly reached servers in a company data centre. That made a hub-and-spoke shape sensible: every branch had a line to the centre, and the centre held the servers, the firewalls and the internet connection. Then the applications moved. Email, file sharing, customer relationship systems and video meetings became cloud services, and the same branch lines now carry traffic that has no business in the data centre.

The weaknesses below are not faults of any one product. They follow from how the design was built.

Limitation 1: cost of private capacity

A private WAN service gives assurance, and the provider charges for it. When a branch needs more bandwidth, the choices are to buy more of the expensive service or to leave the application squeezed. Internet access to the same site is normally cheaper per unit of capacity, but a traditional router has no simple way to use it for site-to-site traffic with the same confidence.

• Capacity on the private circuit is bought for the busiest hour, then sits partly idle.
• The cheaper internet line often sits idle too, because it is held as a backup only.
• Prices differ between providers, regions and contract lengths, so treat any number you hear as local to that deal.

Limitation 2: backhaul through the data centre

Branch internet traffic is often sent over the WAN to the data centre, inspected there, and then sent out to the internet. The reason is usually security: one set of firewalls and one policy are easier to run than one per branch. The price is paid in delay and in load.

1. A user at the branch opens a SaaS page. The request goes from the branch to the data centre across the WAN.
2. The central firewall inspects it and sends it out to the internet.
3. The reply comes back the same way. The data centre sends it down the WAN again to the branch.

In practice, complaints about slow video meetings or slow web applications at branches often turn out to be a backhaul path, not a failing line. Tracing the route from a branch client shows whether the traffic passes through the data centre.

Limitation 3: slow, manual provisioning

Adding a site means ordering a circuit, waiting for the provider, configuring a router by hand or from a template, and building the routing and tunnels for that one site. Each step is separate, and the circuit lead time is outside your control. Changing a policy across a hundred sites means editing a hundred configurations, with the usual risk of one being different from the rest.

• Configuration drift. Over the years, sites that were meant to match end up with small differences that surprise whoever troubleshoots later.
• Per-device policy. QoS (quality of service) and routing choices live in each router, so a policy change is many changes.
• Little visibility. Per-application performance across the WAN is hard to see when the data sits in separate tools on separate boxes.

Limitation 4: circuit convergence

Convergence is the time the network takes to settle on a new working path after a change. On a WAN with two circuits, the question is how fast a branch moves traffic to the second circuit when the first one fails or gets worse.

A router normally learns that a path is gone by one of three means: the interface goes down, a routing protocol neighbour times out, or a tracking feature such as IP SLA (a Cisco feature that sends probes and reports whether a target answers) marks a route as unreachable. Each takes time, and the timers are set per device.

The third row is the important one. Routing decides by reachability and a metric, not by how the path is performing. A voice call can suffer for a long time on a line that every routing table still calls healthy.

Complexity at the edges

Mixing MPLS and internet circuits means mixing routing styles: BGP or static routes to the provider on one side, tunnels and NAT on the other. Each addition brings its own failure modes, and the design that grows out of many such additions is hard for a new engineer to read. Security adds more: a central firewall is easy to reason about, but moving the policy to branches multiplies the places it must be kept right.

Mistakes in diagnosing these problems

• Blaming the cloud provider for a path that is longer than it should be. Check whether the traffic is being backhauled before opening a ticket.
• Reading a line that is up as a line that is good. Interface state says nothing about loss, delay or jitter (variation in delay).
• Buying capacity to fix a design problem. More bandwidth on a backhauled path leaves the extra distance in place.
• Assuming two circuits mean two usable paths. Without a way to measure and steer, the second often sits unused.

Test yourself on WAN limitations