Two data centres, each the home of different applications
The earlier dual-hub lesson treated one data centre as the primary and the other as the disaster-recovery site. This scenario is less tidy. HUB-01 (site 10) serves DC-A and HUB-02 (site 20) serves DC-B, and both data centres host applications:
| Application | Prefix (example) | Primary | Replica |
|---|---|---|---|
| ERP | 10.20.1.0/25 | DC-A | DC-B |
| File and collaboration servers | 10.20.1.128/25 | DC-B | DC-A |
| Voice gateways | 10.20.20.0/24 in VPN 20 | DC-A | None |
The branches (BR-01, BR-02, BR-03A and BR-03B, sites 101 to 103) should use the primary data centre of each application, in both directions, and move to the other one if the primary fails. Two constraints matter. Traffic must not be asymmetric, because the firewalls in the data centres track connections. And the voice gateways have no replica, so for them there is nothing to fail over to.
On small screens, swipe sideways or focus the diagram and use the arrow keys.
Layer 1: advertise prefixes that can be told apart
Both hubs advertise both server prefixes into VPN 10 (corporate). They differ only in preference, which is set in the next layer. One point here decides whether the rest works: the two prefixes must be advertised as the two /25 routes, not as a covering 10.20.1.0/24. A router chooses the most specific matching prefix first, and OMP preference is only compared between routes for the same prefix. A /24 summary would hide the difference between the halves, and a leftover /24 from an older design could quietly override the /25 policy for traffic it still matches.
The voice gateways are advertised only by HUB-01, and only in VPN 20. VPN 20 is the voice service VPN in the lab, and the prefix 10.20.20.0/24 is an example.
Layer 2: one preference per prefix and per hub
OMP route preference is a value carried with an OMP route. Among otherwise valid routes for the same prefix, the higher value wins, and it is compared before the TLOC preference. The Controller can change it with a control policy that matches routes by prefix and by originating site. The values below are examples. Only the order matters.
| Prefix | Route from HUB-01 (site 10) | Route from HUB-02 (site 20) | Branches use first |
|---|---|---|---|
| 10.20.1.0/25 (ERP) | 200 | 100 | DC-A |
| 10.20.1.128/25 (files) | 100 | 200 | DC-B |
| 10.20.20.0/24 (voice, VPN 20) | 200 | Not advertised | DC-A |
Every cell is set explicitly, including the lower values. Relying on a default for the second hub works until a release or a policy changes the default, and then the table in the design document and the table in the network disagree. The structure of a policy that does this follows.
policy
lists
prefix-list ERP
ip-prefix 10.20.1.0/25
!
prefix-list FILES
ip-prefix 10.20.1.128/25
!
site-list BRANCHES
site-id 101-103
!
!
control-policy PREFERRED-DC
sequence 10
match route
prefix-list ERP
site-id 10
!
action accept
set
preference 200
!
!
!
sequence 20
match route
prefix-list ERP
site-id 20
!
action accept
set
preference 100
!
!
!
sequence 30
match route
prefix-list FILES
site-id 10
!
action accept
set
preference 100
!
!
!
sequence 40
match route
prefix-list FILES
site-id 20
!
action accept
set
preference 200
!
!
!
default-action accept
!
!
apply-policy
site-list BRANCHES
control-policy PREFERRED-DC out
!
!Outline of the policy structure only, shortened, and not tested on a device. Verify the keywords, the way a route's originating site is matched, and the effect of the direction in your release and in the policy builder of SD-WAN Manager. The preference values are examples. The direction out means towards the branch sites: the Controller edits the routes it sends to them.
The default action is accept, so every other route passes through unchanged. If it were reject, the first sequence you forgot would remove a whole application from the branches.
Why not TLOC preference?
TLOC preference looks like the obvious tool and does not fit. It is a property of a transport locator (system IP, colour and encapsulation, see TLOC components), so it applies to everything reached through that locator. Raise it on HUB-01 and the branches prefer HUB-01 for ERP, for files and for voice. The file servers then travel to DC-A and cross the data-centre interconnect to the primary copy in DC-B, which is the detour the design was meant to avoid. Use TLOC preference to rank the transports of one hub, and OMP route preference to rank the data centres per prefix.
Layer 3: the data-centre side must agree
The policy so far steers traffic from the branch to the data centre. The reply is routed by the data centre network, using the branch prefix (for example 10.10.1.0/24 for BR-01) as its destination. This is where the design is most likely to fail, for a reason that the SD-WAN side cannot see.
A return route is chosen by destination only. The data centre network does not know whether a reply to 10.10.1.0/24 belongs to an ERP session or a file session. So the return path cannot be preferred per application with a plain metric on the branch prefix. What can be controlled is the choice per data centre.
- Separate routing domains per data centre. The ERP servers in DC-A reply to a DC-A core, which prefers HUB-01 for every branch prefix. The file servers in DC-B reply to a DC-B core, which prefers HUB-02. Each side agrees with its own primary automatically. The metrics (OSPF cost, or BGP local preference or AS-path length) are set so that each hub is the best exit for its own data centre, and the other hub, reached across the interconnect, is only a backup.
- Failover has to be mirrored. When the branch moves ERP to DC-B, the request enters through HUB-02 and reaches the replica in DC-B. Its reply goes to the DC-B core, which prefers HUB-02. The path is symmetric again without any change, as long as the replica sits behind the DC-B core and not behind DC-A's.
- One shared routing domain across both data centres. This is the hard case. A shared core has one best exit per branch prefix, so one of the two applications returns through the hub its branch did not use. Options are source-based routing in the core, a firewall or load balancer per data centre that forces replies back, or source NAT on the hub so the reply returns to the hub that sent the request. Each has a cost in complexity and in what the servers see as the client address. Treat this as a design review item before any SD-WAN policy is written.
The earlier lesson covers the same problem for a primary and DR pair, and the BGP course has the attributes used on the data centre side.
What the branch should see
BR-01#show sdwan omp routes vpn 10 VPN PREFIX FROM PEER PATH ID LABEL STATUS TLOC IP COLOR PREF 10 10.20.1.0/25 10.255.0.11 101 1002 C,I,R 10.255.1.1 mpls 200 10 10.20.1.0/25 10.255.0.11 102 1002 C,I,R 10.255.1.1 biz-internet 200 10 10.20.1.0/25 10.255.0.11 201 1002 R 10.255.1.2 mpls 100 10 10.20.1.0/25 10.255.0.11 202 1002 R 10.255.1.2 biz-internet 100 10 10.20.1.128/25 10.255.0.11 101 1002 R 10.255.1.1 mpls 100 10 10.20.1.128/25 10.255.0.11 102 1002 R 10.255.1.1 biz-internet 100 10 10.20.1.128/25 10.255.0.11 201 1002 C,I,R 10.255.1.2 mpls 200 10 10.20.1.128/25 10.255.0.11 202 1002 C,I,R 10.255.1.2 biz-internet 200
Pair this with show sdwan bfd sessions from the dual-hub lesson: the fallback hub needs live tunnels before the failure, not after it.
Failure cases and expected behaviour
On small screens, swipe sideways or focus the diagram and use the arrow keys.
The second row decides how good the design really is. A hub detects its own death through BFD, but it detects a dead application only through routing. A service-side check (tracking, a dynamic route that depends on the server) is what turns an application failure into a withdrawn prefix. Without it the design only protects against hub and transport failures.
Test plan
Run these in a lab or a maintenance window, with a continuous ping or a connection test from a branch to an address in each prefix, and a trace from the data centre back to the branch. Keep the evidence listed.
On small screens, swipe sideways or focus the diagram and use the arrow keys.
Why test the stateful device in both directions?
A ping can pass a firewall that still drops a TCP connection, because the firewall may treat each echo as a simple flow and policy can differ by protocol. A connection with a handshake that leaves by one hub and returns by another is normally dropped. Test with a real TCP connection to the application port, in both directions, after every failover step.
When the replica is not identical
The network moves packets to the data centre that the policy picks. It cannot know whether that copy is current, writable or even running the same version. A failover can succeed in the network sense and still fail for users: the file replica may be read-only, the ERP replica may be an hour behind, or a licence server may exist only in DC-A. Agree with the application owners which replicas are meant to serve users, in which mode, and what the planned switchover is. Write it in the same document as the preference table, so a change in either one prompts a review of the other.
Test yourself on per-application data centre choice
You raise the TLOC preference on HUB-01 so that ERP prefers DC-A. After the change, the file servers (primary in DC-B) are reached through HUB-01 as well. Why?
ERP is advertised as 10.20.1.0/25 by both hubs. An old design still advertises 10.20.1.0/24 from HUB-02 with a higher preference. What does a branch use for a server at 10.20.1.20?
The ERP application in DC-A stops, but HUB-01 and its tunnels are healthy. Branches still send ERP traffic to DC-A. The prefix is a static route on HUB-01. What is the likely fix?
Both data centres share one routing domain. Files and ERP now use different hubs, but a firewall in DC-A drops file replies. What is the most likely cause?