One hub, two hundred spokes, one voice requirement
HUB-01 (site 10) is the head office. About 200 branch sites are spokes, and most have two transports: an MPLS circuit and an internet line. BR-01 (site 101) is one of them. Voice runs between the spokes and gateways behind HUB-01, and the requirement is plain: voice prefers MPLS, and moves to the internet path when the MPLS quality drops. BR-02 (site 102) has only an internet line, so for it the policy has nothing to choose between. Site IDs and colours are examples.
The mechanics of measurement and SLA classes are in the application-aware routing lesson. This one covers what changes when there are many tunnels converging on one place.
Each direction is decided by the sender
AAR is applied by the router that sends the traffic. A spoke chooses a path for traffic going to the hub, using its own measurements of its own tunnels and the policy applied to its site. The hub chooses a path for traffic going to a spoke, using the hub's measurements of its tunnels to that spoke and the policy applied to the hub's site. The two decisions are separate.
On small screens, swipe sideways or focus the diagram and use the arrow keys.
For voice, the third row is the one that works. The paths may end up different in the two directions, for example spoke to hub over the internet and hub to spoke over MPLS. That is acceptable for voice only if both directions meet the SLA, because a call is bad if either side is bad. Stateful devices on the hub side, such as a firewall in front of the gateways, also need to cope with asymmetric paths, which is the same issue as in the dual-hub lesson.
Site lists and where to apply the policy
A site list is a named group of site IDs. Two are enough here: one for the hub and one for the spokes. The same policy can be applied to both lists, or two policies with different SLA values and preferences can be used if the directions need different treatment.
policy
lists
site-list HUB
site-id 10
!
site-list SPOKES
site-id 101-300
!
vpn-list VOICE-VPN
vpn 20
!
!
sla-class VOICE
loss 2
latency 150
jitter 30
!
app-route-policy VOICE-AAR
vpn-list VOICE-VPN
sequence 10
match
dscp 46
!
action
sla-class VOICE preferred-color mpls
!
!
!
!
!
apply-policy
site-list SPOKES
app-route-policy VOICE-AAR
!
site-list HUB
app-route-policy VOICE-AAR
!
!Outline of structure only, shortened and not tested on a device. It shows one policy applied to two site lists, so that both directions are steered. Verify the keywords in your release and how SD-WAN Manager builds the same policy. The thresholds, the site range and the colour are examples, not defaults. Loss is in per cent, latency and jitter in milliseconds.
The site range 101-300 stands for the spokes in this example. In a real network the list is whatever your site numbering plan makes of it. A spoke that is missing from the list gets no steering in its outbound direction, and nothing in the network reports that as an error.
On small screens, swipe sideways or focus the diagram and use the arrow keys.
What the hub measures
A spoke has few tunnels: with MPLS and internet to one hub and the same colours at both ends, that is two per hub. The hub has all of them. Each tunnel has a BFD session, and each session is measured and recorded. The arithmetic for this network:
| Design choice | Tunnels per spoke to HUB-01 | Sessions on HUB-01 (200 spokes) |
|---|---|---|
| Two colours, restrict set (same colour pairs only) | 2 | 400 |
| Two colours, no restrict (every colour pair) | 4 | 800 |
| Add a second hub with the same layout | 4 (two per hub) | 400 on each hub |
| Internet-only spoke such as BR-02 | 1 | 1 per such spoke |
With the default one-second BFD hello, 400 sessions mean about 400 probes per second sent by the hub, plus the same number received. That is small for the bandwidth of a hub, but the sessions also use memory, and the per-interval statistics for every tunnel use more still. Platform limits for the number of tunnels and BFD sessions differ by model and release: look them up for the hub you plan to buy, with room to grow.
- Restrict colours so that MPLS only builds tunnels to MPLS. Doubling the sessions for combinations nobody uses is the most common waste. See TLOC components.
- Think before shortening the measurement settings. A shorter poll interval and a smaller multiplier make AAR react faster, and produce more statistics and more sensitivity. Multiplied by 200 spokes, the cost of a setting chosen for one site is paid at the hub.
- Do not measure what no policy uses. Colours that carry no AAR traffic still have BFD sessions. They are still needed for failure detection, but not for steering.
A fault at the hub moves every spoke at once
Suppose HUB-01's MPLS uplink becomes congested at 14:00. Every spoke's MPLS tunnel to the hub shows loss and delay, because they share that uplink. Spokes and hub all decide, more or less together, that MPLS no longer meets the VOICE class and move voice to the internet path. Two things follow:
- The hub's internet uplink now carries the voice of 200 sites at once. If it is smaller than the MPLS one, it can become the next bottleneck, the internet tunnels start missing the SLA, and no tunnel qualifies.
- When MPLS recovers, all sites may move back together, in a burst, and the cycle can repeat.
The options are modest, and they should be chosen on purpose. Size the internet uplink at the hub for the voice load that can move, not only for normal use. Use a measurement multiplier that smooths over short congestion, so a few bad intervals do not trigger a move. Check whether your release has variance settings that make the router move back only when the original path is clearly better. Decide what the policy does when no path meets the SLA (shared, best available, or dropped): with 200 sites, that choice is the most important one in the policy. A second hub with its own uplinks spreads the risk, at the price of more tunnels and a return-path design.
Checking the two ends
BR-01#show sdwan app-route stats REMOTE SYSTEM IP LOCAL COLOR REMOTE COLOR INDEX PACKETS LOSS LATENCY JITTER 10.255.1.1 mpls mpls 0 600 0 32 3 10.255.1.1 mpls mpls 1 600 0 33 2 10.255.1.1 biz-internet biz-internet 0 600 0 24 2 10.255.1.1 biz-internet biz-internet 1 600 0 25 4
HUB-01#show sdwan app-route stats | include 10.255.2 10.255.2.1 mpls mpls 0 600 0 31 3 10.255.2.1 biz-internet biz-internet 0 600 0 23 2 10.255.2.2 biz-internet biz-internet 0 600 0 41 6 10.255.2.3 mpls mpls 0 600 4 38 5 10.255.2.3 biz-internet biz-internet 0 600 0 26 3
show sdwan policy from-vsmartRun on the hub and on a spoke. Both should list the app-route policy. A hub that does not list it is not steering hub-to-spoke traffic, which is the quiet failure in the table above. The command name varies a little between releases.
Spoke-to-spoke traffic
If the topology is strictly hub-and-spoke, spokes have no tunnels to each other. A flow from BR-01 to BR-02 goes to HUB-01, is forwarded by the hub, and goes on to BR-02. There are two tunnels and two measurements:
- The latency of the call is about the sum of both legs plus the processing at the hub. Two legs of 40 ms each give an end-to-end figure near 80 ms, even though each tunnel passes a 150 ms class on its own.
- The SLA is measured per tunnel, not end to end. The policy cannot ask for "less than 150 ms between BR-01 and BR-02". If the end-to-end limit matters, set the per-leg thresholds lower than the end-to-end target.
- The first leg is chosen by the policy at the sending spoke. Whether the policy applies again at the hub to traffic that arrives from one tunnel and leaves by another depends on how it is applied and on the release. Check it. If it does not apply, the second leg follows routing.
The alternative is to allow direct spoke-to-spoke tunnels, built on demand in some designs. That removes the double leg and the hub load for those flows, and adds tunnels and policy cases between spokes. Whether to allow them is a topology decision made before the AAR policy, not by it.
Validation table
On small screens, swipe sideways or focus the diagram and use the arrow keys.
Why test on one spoke first, with the hub still running?
The scale effects only appear with many spokes, so a lab with three spokes cannot show them. What a small test can show is the logic: which router moves, in which direction, and what the policy does when no path qualifies. Do that first. Then estimate the scale effects from the table of sessions and the hub uplink sizes, and confirm them during a staged rollout, a few spokes at a time, watching the hub's resources.
Test yourself on AAR with many spokes
The VOICE app-route policy is applied to the spokes site list only. A spoke's MPLS path degrades. What happens to voice from the hub to that spoke?
HUB-01 serves 200 spokes with MPLS and internet. Colours are not restricted, so every colour pair builds a tunnel. Roughly how many BFD sessions does the hub hold, and what is a cheaper design?
At 14:00 the hub's MPLS uplink is saturated, and several hundred voice calls move to the internet path at the same time. Calls then get worse. What is the most likely cause and a sensible response?
BR-01 calls BR-02 through HUB-01. Each leg has 70 ms latency, and the VOICE class limit is 100 ms per tunnel. The users say the call is delayed. Is the SLA broken?