Controller HA and failure domains

One question per failure

Every redundancy decision in Catalyst SD-WAN answers the same question: if this fails, what stops, and what carries on? The fabric is built from separate planes, so the answers differ a great deal. The management plane (the SD-WAN Manager, formerly vManage) can be down while every packet is still forwarded. The control plane (the SD-WAN Controllers, formerly vSmart) can be down for hours with only a slow loss of function. The data plane (the WAN Edge routers, formerly cEdge or vEdge, and their tunnels) is where a failure is felt by users at once.

The components, what each failure removes

The rest of the lesson takes the rows in order of how often they matter in practice.

SD-WAN Manager: a cluster, and a second site

A Manager can run as a single node or as a cluster, and a cluster is typically three nodes. A cluster shares the work and survives the loss of a node. It does not survive the loss of the site that holds all of its nodes. For that, a standby Manager or cluster is built at another site, with its data replicated from the active one. If the active site is lost, an administrator starts the switchover. It is not automatic and it takes time, and the devices have to start talking to the standby afterwards. Support, replication behaviour and the exact procedure differ by release, so check your release's disaster recovery guide before writing any promise into a design.

While the Manager is down the network keeps forwarding. What is lost is the ability to see and change it: a failed link may go unreported, and a configuration change cannot be pushed. In practice the Manager is the component people notice first, because it is the one they look at, and the one that matters least for traffic.

SD-WAN Controllers: two, in different places

Each WAN Edge keeps control connections to a limited number of Controllers. On many releases that number is two, so with two or more Controllers deployed, losing one leaves the other. Spread the Controllers across data centres or cloud availability zones, so one power or network failure does not take all of them. A Controller group (affinity) setting lets you decide which Controllers a given set of edges prefers, which also spreads the load across them. Check the figures for your release.

If every Controller is unreachable, the edges do not stop. They keep the routes, TLOCs, keys and policy they hold, and they keep forwarding. They hold them for a graceful-restart period after the control connection is lost. On current releases this is long, commonly given as 12 hours, but treat that as a figure to check on your release. A short Controller outage is therefore survivable. What it cannot do is react to change: a new site cannot learn routes, a withdrawn prefix is not withdrawn, and a failover that depends on a route update does not happen. Tunnel failure detection still works, because BFD runs between the edges themselves.

SD-WAN Validators: reachable by name

A Validator is the first contact for every device. It checks identity and tells the device which Manager and Controllers to use. Deploy more than one, and configure the edges with a DNS name that resolves to several Validator addresses, so that a device can try the next one. With one Validator down, nothing already connected breaks. A new device cannot join until a Validator answers, and a device that loses its control connections and needs to reconnect may not be able to find a Controller. That is why a Validator outage can look harmless for a day and then matter after a site reboots.

Edge and transport redundancy

At the edge, the failure domain is the router. A branch with one WAN Edge is down when that router is down, however many transports it has. A branch that must survive this has two edges and a shared gateway for the LAN, usually with VRRP. The lab on VRRP and EIGRP on the service side builds this at site 103, where BR-03A and BR-03B serve one site.

Transport redundancy is a separate question. BR-01 has MPLS and internet, so it survives losing either. BR-02 has internet only, so losing it removes the site's WAN even though the router is healthy. Two colours on one router, two routers on one site and two hubs in two data centres each protect against a different failure. A design should say which of the three it is buying.

Placing components by failure domain

• Controllers: at least two, in separate data centres, cloud zones or power and network domains. Two Controllers on the same hypervisor are one failure domain.
• Validators: more than one, also separated, with a DNS name covering them.
• Manager: cluster nodes in one site, standby at a second site. A cluster adds capacity and node resilience, not site resilience.
• Hubs: one per data centre, as in the dual-hub design.
• Shared dependencies: DNS, NTP and the certificate authority are failure domains too. A Validator that cannot be resolved is as good as down.

A health check you can run

Two commands on a WAN Edge give a quick view of the control plane. Run them in normal operation and keep the output as a baseline, because a baseline makes a later failure obvious.

BR-01#show sdwan control connections
PEER    PEER  PEER         SITE  DOMAIN  PEER           PRIVATE        STATE  UPTIME
TYPE    PROT  SYSTEM IP    ID    ID      PRIVATE IP     PORT
vsmart  dtls  10.255.0.11  1     1       192.0.2.11     12346  up     12:05:20:11
vbond   dtls  -            0     0       192.0.2.12     12346  up     12:05:20:30
vmanage dtls  10.255.0.10  1     0       192.0.2.10     12346  up     12:05:19:58

BR-01#show sdwan omp summary
oper-state               UP
admin-state              UP
personality              vedge
omp-uptime               12:05:19:50
routes-received          9
routes-installed         9
routes-sent              2
tlocs-received           7
tlocs-installed          7
tlocs-sent               2

Outage versus forwarding

When something fails, ask which plane it belongs to. A Manager outage, a Validator outage and a Controller outage leave forwarding alone for a time. A tunnel or edge failure does not. The dual-hub design lesson shows a failover that depends on BFD and so works without any Controller, and one that depends on a route being withdrawn and so does not. Knowing which of your failure cases need the Controller is the same as knowing how long a Controller outage you can survive.

Mistakes that remove the redundancy quietly

• Two Controllers on the same host or in the same rack, which is one failure domain with two names.
• A Manager cluster described as disaster recovery, when all three nodes are in one room.
• A standby Manager that has not been switched to in a test, so its procedure and its effect on the devices are unknown.
• One Validator address configured on the edges instead of a name, so a single outage blocks reconnecting devices.
• Redundant components that were healthy at installation and were not checked again.

Test yourself on what each failure removes