Hierarchical SD-WAN and Multi-Region Fabric

When one flat fabric stops being practical

In a full mesh every site can build a tunnel to every other site. The number of site pairs is n × (n − 1) ÷ 2, so it grows with the square of the number of sites. A network of 100 sites has 4,950 pairs. A network of 1,000 sites has 499,500. Each pair can need a tunnel for every colour combination, and each tunnel runs BFD. The numbers show the shape of the growth, not any limit of a platform, which depends on the model and the release.

Tunnel count is only one of three pressures.

• Control-plane load. Every site's routes and TLOCs reach every other site through the SD-WAN Controllers (formerly vSmart), and every policy is evaluated against all of them.
• Edge load. A small branch router holds state for tunnels and routes to sites it may not need to talk to.
• Operational blast radius. A policy mistake or a routing event in one part of the world is felt everywhere, and one team owns everything.

The hierarchical idea

The answer is the one used elsewhere in networking: add a hierarchy. Branches connect to a regional hub or gateway, and the gateways connect the regions to each other. A branch in Europe no longer builds tunnels to a branch in Asia. Its traffic goes to a regional gateway, across to the other region's gateway and down to the far branch. The cost is a longer path and a gateway that carries transit traffic. The gain is that each branch holds only the state for its own region and the gateways.

This can be built by hand in a flat fabric, using hub-and-spoke topology and control policy. Multi-Region Fabric builds the same idea into the fabric as a feature.

How Multi-Region Fabric is organised

The exact names, limits and behaviours vary by release, so read this as the model and check the details in your release's guide.

• Core region (region 0). The region that joins the others. It carries traffic between regions.
• Access regions. Numbered regions, each holding a group of sites, for example one per country or continent. The numbers are an example of a plan, not a rule.
• WAN Edge routers in an access region build tunnels with the other edges in the same region, for traffic that stays inside the region.
• Border routers belong to an access region and to the core region at the same time. They carry traffic between the two, and so between regions.

A branch in one region reaches a branch in another by a hierarchical path: its own region to a border router, across the core region to a border router of the destination region, then down to the far branch. Border routers also re-originate routes. A border router learns routes in one region and advertises them into the other with itself as the way to reach them, so a branch learns that a prefix in another region is reachable through a border router instead of seeing every remote site as a direct peer.

Flat and hierarchical designs compared

Design points that decide whether it works

• Border router redundancy and capacity. Border routers become transit for everything between regions. Plan at least two per region, in different failure domains (see Controller HA and failure domains), and size them for the inter-region traffic, not only their own site.
• The extra hop. Inter-region traffic crosses two border routers. That adds latency and may take a longer geographic path. Voice and other delay-sensitive traffic between regions needs a check against the budget.
• Region size. Too large and the original problem returns inside the region. Too small and traffic hairpins through border routers. Group sites by where most of their traffic goes.
• Transport colours per region. The access region and the core can use different transports. Decide which colours exist in which region, and check that border routers have a usable colour in both.
• Policy per region. Control and data policy can be applied for a region, which is a benefit (smaller blast radius, local ownership) and a risk (two regions with different intent for the same traffic).

Troubleshooting an inter-region path

Start by writing down the region of the source site, the region of the destination, and the border routers that serve each. Then check the path in legs.

1. Source site to its border router. Are the tunnels up inside the source region, and does the site learn the destination prefix with a border router as the next hop?
2. Across the core. Are the border routers connected to each other in region 0, and are those tunnels up on a colour both can use?
3. Destination border router to the destination site. Does the border router have tunnels into the destination region, and a route to the prefix there?
4. The return path. Repeat in the other direction. Asymmetric paths through two different border router pairs are easy to create and hard to see.

At each leg the questions are the ones from earlier lessons: is the route present and resolved, are the tunnels up, does a policy change the choice. What is new is the middle: a branch will not show a direct tunnel to the far site, because there is not one by design, and that absence is not a fault.

Mistakes when moving to regions

• Expecting a direct tunnel between branches in different regions and treating its absence as a bug.
• One border router per region, which makes a single device the way out for every site in the region.
• Border routers sized like branch routers while carrying the traffic of a whole region.
• Regions drawn by organisation chart, not by where the traffic goes, so most flows cross the core.
• Assuming that a feature or number from one release applies to another.

Test yourself on regions