Course contents

Unit 8: High availabilityLesson 8.3 (2 of 4 in this unit)19 of 25 in the Advanced Cisco SD-WAN course

Dual-hub primary/secondary data centers

Two hubs in two data centres: how branches choose between them, what BFD does when the primary fails, why return traffic is the real design problem, and how to test it.

Advanced · 15 min read

You'll learn to design two hubs as primary and DR, choose how branches prefer one, predict how long failover takes, and test that return traffic follows the same path.

Technical definition

Dual-hub design in Cisco SD-WAN is a layout in which every branch builds tunnels to two hub sites, each in its own data centre, and both hubs offer routes to the same applications. One hub is usually preferred and the other takes over when the preferred one, or its data centre, fails.

Data centre A needs maintenance on Saturday

Forty branches reach their applications through HUB-01 in data centre A. HUB-02 in data centre B is the disaster-recovery site. Someone wants to take HUB-01's data centre down for maintenance, and asks whether the branches will simply move. The honest answer is that they will if the design and the data centre side were built for it and tested, and this lesson is about the parts that decide that.

The design

HUB-01 (site 10) and HUB-02 (site 20) are WAN Edge routers with their own system IPs and site IDs. Each has its own tunnels to every branch, and both learn the application prefix, 10.20.1.0/24, from their data centre and advertise it into the overlay. A branch therefore has two routes to the same prefix, one via each hub, and chooses between them using OMP.

The design
The designYou want; Make the two routes; Result. Active/active: Equal Branches share traffic across both hubs, within the equal-cost limit Primary and DR: Unequal: a higher OMP route preference on HUB-01's routes Branches use HUB-01 and move to HUB-02 only when HUB-01's routes are gone or unresolved Per-application preference: Different by application, with a policy per traffic class Voice via one hub, bulk transfers via the otherYou wantMake the two routesResultActive/activeEqualBranches share traffic across bothhubs, within the equal-cost limitPrimary and DRUnequal: a higher OMP routepreference on HUB-01's routesBranches use HUB-01 and move toHUB-02 only when HUB-01's routes aregone or unresolvedPer-applicationpreferenceDifferent by application, with apolicy per traffic classVoice via one hub, bulk transfers viathe other

On small screens, swipe sideways or focus the diagram and use the arrow keys.

Read each row across: the left box names the item, the others describe it.

Among valid routes a higher OMP route preference wins, and the TLOC preference is considered after it; the full order is in the best-path lesson. The common mistake is leaving both at the default and finding that a "primary and DR" design is carrying production traffic through the DR site.

policy lists site-list HUB-01-SITE site-id 10 ! site-list BRANCHES site-id 101-103 ! ! control-policy PREFER-HUB-01 sequence 10 match route site-id 10 ! action accept set preference 200 ! ! ! default-action accept ! ! apply-policy site-list BRANCHES control-policy PREFER-HUB-01 out ! !

Structure of a Controller control policy, shortened; it is normally built in SD-WAN Manager. Routes originated by site 10 get preference 200 (higher is better) when sent out to the branch sites. The direction (out, towards the branches) is the part that is easy to get backwards, so verify the result in the branches' OMP tables. The preference value is an example.

Dual-hub design: BR-01 reaches the data centre through either hub
Dual-hub design: BR-01 reaches the data centre through either hubBR-01 builds tunnels to both HUB-01 in data centre A and HUB-02 in data centre B. Normally the preferred path is through HUB-01. When HUB-01 is taken out for maintenance, BR-01 uses HUB-02. Both hubs can reach the data centre network, and the return traffic must be steered back to the same hub that the branch used.BR-0110.255.2.1HUB-01DC A, 10.255.1.1HUB-02DC B, 10.255.1.2Data centre 10.100.0.0/16preferredbackupWhen HUB-01 is down, BR-01 uses HUB-02; the data centre must send replies back through the same hub.

On small screens, swipe sideways or focus the diagram and use the arrow keys.

The forward path is the easy half. Check the return path: if the data centre sends replies through the other hub, the flow breaks or becomes asymmetric.

What happens when HUB-01 fails

What happens when HUB-01 fails
What happens when HUB-01 failsCondition; What happens; Where BR-01's traffic goes. Normal: BR-01 has two routes to 10.20.1.0/24: via HUB-01 with preference 200, via HUB-02 with the default HUB-01. HUB-02 carries nothing HUB-01 loses its WAN: BFD to HUB-01 stops; after about seven seconds with default timers the tunnels are down, its TLOCs are unusable and its route is unresolved HUB-02, for new and existing flows. Applications that kept state on HUB-01's side reconnect HUB-01's data centre link fails: The prefix leaves HUB-01's service-side table, so it stops advertising it; an OMP route exists only while the hub has the prefix HUB-02. This works only if the hub learns the prefix dynamically: a static route or default that stays in the table would keep attracting traffic into a black holeConditionWhat happensWhere BR-01's traffic goesNormalBR-01 has two routes to 10.20.1.0/24:via HUB-01 with preference 200, viaHUB-02 with the defaultHUB-01. HUB-02 carries nothingHUB-01 loses its WANBFD to HUB-01 stops; after aboutseven seconds with default timers thetunnels are down, its TLOCs areunusable and its route is unresolvedHUB-02, for new and existing flows.Applications that kept state onHUB-01's side reconnectHUB-01's data centrelink failsThe prefix leaves HUB-01'sservice-side table, so it stopsadvertising it; an OMP route existsonly while the hub has the prefixHUB-02. This works only if the hublearns the prefix dynamically: astatic route or default that stays inthe table would keep attractingtraffic into a black hole

On small screens, swipe sideways or focus the diagram and use the arrow keys.

Read each row across: the left box names the item, the others describe it.

Two different failures, two different mechanisms: BFD notices a dead hub; a withdrawn route removes a hub that can no longer reach the application.

The return path is the hard part

Everything above steers traffic from the branch towards the data centre. Replies take whatever route the data centre has back to the branch prefix. If HUB-01 and HUB-02 both advertise the branch prefixes into the data centre network (by OSPF, BGP or similar) with equal cost, the core may send replies through HUB-02 while the branch sent requests through HUB-01. That is asymmetric routing. It works for simple traffic and breaks a stateful firewall or load balancer that sees only half the connection.

  • Make the data centre agree. Give the DR hub's advertisements into the data centre a worse metric (a higher cost in OSPF, a longer AS path or lower local preference in BGP) so the primary is preferred in both directions.
  • Make failover work in both directions. When HUB-01 can no longer reach the overlay, it must stop advertising the branch prefixes to the data centre, or replies keep going to a hub that cannot deliver them.
  • Check it, do not assume it. Trace a flow in both directions before and after a failover.

If the data centre side is BGP, the BGP course gives the tools for steering the return path.

Checking the branch's view

Example output written for this lesson, simplified; real output has more fields and a different layout
BR-01#show sdwan omp routes 10.20.1.0/24
VPN  PREFIX         FROM PEER    PATH ID  LABEL  STATUS  TLOC IP      COLOR         PREF
10   10.20.1.0/24   10.255.0.11  101      1002   C,I,R   10.255.1.1   mpls          200
10   10.20.1.0/24   10.255.0.11  102      1002   C,I,R   10.255.1.1   biz-internet  200
10   10.20.1.0/24   10.255.0.11  201      1002   R       10.255.1.2   mpls          -
10   10.20.1.0/24   10.255.0.11  202      1002   R       10.255.1.2   biz-internet  -
BR-01 has four paths: two via HUB-01 (system IP 10.255.1.1) with preference 200, chosen and installed, and two via HUB-02 (10.255.1.2), resolved (R) but not chosen. Each hub is reached over both of BR-01's transports. If the HUB-01 lines vanish or lose their R, the HUB-02 lines are what the branch falls back to.
Example output written for this lesson, simplified; real output has more fields and a different layout
BR-01#show sdwan bfd sessions | include 10.255.1
10.255.1.1   10   up   mpls          mpls          172.16.1.2    172.16.1.1     12346  ipsec  7   1000   3:04:10:02
10.255.1.1   10   up   biz-internet  biz-internet  203.0.113.10  198.51.100.20  12346  ipsec  7   1000   3:04:09:58
10.255.1.2   20   up   mpls          mpls          172.16.1.2    172.16.2.1     12346  ipsec  7   1000   3:04:10:01
10.255.1.2   20   up   biz-internet  biz-internet  203.0.113.10  198.51.100.40  12346  ipsec  7   1000   3:04:09:57
The standby hub needs live tunnels as well. If the DR hub's tunnels are down, the failover has nothing to fail over to, which only shows up during a failure. Checking that the standby tunnels are up is part of every health check.

A failover test plan

Run every test in a maintenance window or a lab, with a rollback ready and one person watching a continuous ping from a branch to an application address in both directions. Record the evidence for each: the time, the output of the commands above, and how long traffic stopped.

A failover test plan
A failover test planTest; Expected; Evidence to keep. Normal state: Branches use HUB-01; HUB-02 tunnels up and idle; return path via HUB-01 OMP routes and BFD sessions at a branch; a trace each way HUB-01 loses its transports: Branches move to HUB-02 in about seven seconds; replies follow Ping loss duration; OMP routes after failover; return trace HUB-01 loses its data centre link: The prefix is withdrawn and branches use HUB-02 HUB-01's service-side table; branch OMP routes Restore HUB-01: Traffic returns to HUB-01 only as intended; no flapping Time to return; whether open sessions broke Controller unreachable: Existing routes and tunnels keep working; no new policy until it returns Forwarding unchanged; OMP peer stateTestExpectedEvidence to keepNormal stateBranches use HUB-01; HUB-02 tunnelsup and idle; return path via HUB-01OMP routes and BFD sessions at abranch; a trace each wayHUB-01 loses itstransportsBranches move to HUB-02 in aboutseven seconds; replies followPing loss duration; OMP routes afterfailover; return traceHUB-01 loses its datacentre linkThe prefix is withdrawn and branchesuse HUB-02HUB-01's service-side table; branchOMP routesRestore HUB-01Traffic returns to HUB-01 only asintended; no flappingTime to return; whether open sessionsbrokeController unreachableExisting routes and tunnels keepworking; no new policy until itreturnsForwarding unchanged; OMP peer state

On small screens, swipe sideways or focus the diagram and use the arrow keys.

Read each row across: the left box names the item, the others describe it.
Why test with the Controller unreachable?

Because the two planes fail separately. Losing the Controller does not stop forwarding that already works, but it does stop everything that depends on a route or policy update, including a failover that needs a new route. Knowing which of your failure cases still work without the Controller is part of knowing how long an outage you can survive.

Where dual-hub designs go wrong

  • Equal preference left in place, so the DR hub carries production traffic and nobody notices until it is overloaded.
  • Return path never configured, giving asymmetric traffic that a firewall drops during or after failover.
  • A static route or default route advertised into the overlay, so a hub that has lost the data centre keeps attracting traffic.
  • DR hub capacity sized for a quiet standby, not for the whole network arriving at once.
  • Failover tested once at installation and never again, while policies, addresses and software versions changed.

Test yourself on dual-hub design

Predict · scenario 1

Both hubs advertise 10.20.1.0/24 with the same preference. How does BR-01 forward traffic to it?

Predict · scenario 2

HUB-01 loses all its WAN links. About how long until BR-01 stops sending traffic to it, with default BFD timers?

Predict · scenario 3

HUB-01 is up and has tunnels, but its link to the data centre core has failed. Its route to 10.20.1.0/24 was learned from the core by OSPF. What happens?

Predict · scenario 4

After failing over to HUB-02, one application stops working although pings succeed. The data centre has a stateful firewall. What do you check first?

FAQ

If both hubs advertise the same prefix, do branches use both?
By default, yes if the routes are equally good: the branch can share traffic across them, up to the router's equal-cost limit. That is active/active. For primary and DR you give one hub's routes a higher preference so branches use it first.
How long does failover take?
Detecting a dead hub takes about seven seconds with the default BFD timers (hello 1 second, multiplier 7). After that the branch picks the other hub's route. Sessions that were open usually have to reconnect, and stateful devices in the data centre may add their own delay.
Why not just run two hubs and leave routing in the data centre alone?
Because the data centre decides the return path. If its routers send replies via the hub the branch is not using, traffic arrives one way and returns another, and a firewall that tracks connections drops it.