Course contents

Unit 5: Onboarding and trustLesson 5.5 (5 of 5 in this unit)25 of 27 in the Cisco SD-WAN course

Onboarding verification and rollback

The order of checks that prove a new router is working, from control connections to BFD and routes, and the rollback options from smallest to most disruptive.

Intermediate · 10 min read

You'll learn to check a newly onboarded router in the right order, read control connection and BFD output to tell what is and is not working, and choose a rollback that fits the failure.

Technical definition

Onboarding verification is the set of checks that prove a new WAN Edge is a working member of the fabric: it is seen by the Manager, its control connections are up, its tunnels to other sites are up, and it learns and uses routes. Rollback is the planned way back to a known state if those checks fail.

Up in the GUI, silent on the wire

BR-01 shows as reachable in the Manager, and the site engineer reports that nothing at the branch can reach the data centre. Both statements can be true. The Manager checks the control plane, and user traffic needs the data plane and routes as well. A fixed order of checks keeps this from becoming a guess. The commands below are for your own router or a lab; the sample output is written for the lesson.

The order of checks

The order of checks
The order of checksStep; Question; Where to look. 1: Is the device registered and seen? The Manager's device list 2: Are control connections up, on each transport? show sdwan control connections 3: Are tunnels to other sites up? show sdwan bfd sessions 4: Are routes learned and installed? show ip route, and the OMP views 5: Does user traffic follow them? A ping or traceroute between hosts in the same VPNStepQuestionWhere to look1Is the device registered and seen?The Manager's device list2Are control connections up, on eachtransport?show sdwan control connections3Are tunnels to other sites up?show sdwan bfd sessions4Are routes learned and installed?show ip route, and the OMP views5Does user traffic follow them?A ping or traceroute between hosts inthe same VPN

On small screens, swipe sideways or focus the diagram and use the arrow keys.

Read each row across: the left box names the item, the others describe it.

Each step depends on the one before. If step 2 fails, steps 3 to 5 will fail too, and looking at routes at that point finds nothing useful.

The order of checks
The order of checksFive stages from connection to traffic.1 Deviceis joined2 Controlconnections3 Tunnelsto sites4 Routes5 Traffic

On small screens, swipe sideways or focus the diagram and use the arrow keys.

Stop at the first failure.

Step 2: control connections

Control connection checks
Control connection checksCheck each connection.Validatorinitial contactControllerroutes and policyManagerconfiguration

On small screens, swipe sideways or focus the diagram and use the arrow keys.

All should be up.
show sdwan control connections

One line per connection. Look for a line for the Controller and one for the Manager, in the up state, on each transport that should carry them. See the control connections lesson for how to read it.

Example output written for this lesson, simplified; real output has more fields and a different layout
BR-01#show sdwan control connections
PEER TYPE  PROT  PEER SYSTEM IP  SITE ID  LOCAL COLOR   STATE
vsmart     dtls  10.255.0.11     1        biz-internet  up
vsmart     dtls  10.255.0.11     1        mpls          up
vmanage    dtls  10.255.0.10     1        biz-internet  up
The Controller is up on both transports and the Manager on one. That is a healthy result if the design allows management on one transport only. If the Manager is meant to be reachable on both, one line is missing.

Step 3: tunnels to other sites

Example output written for this lesson, simplified; real output has more fields and a different layout
BR-01#show sdwan bfd sessions
SYSTEM IP    SITE  STATE  SOURCE COLOR  REMOTE COLOR  DST PUBLIC IP
10.255.1.1   10    up     biz-internet  biz-internet  198.51.100.20
10.255.1.2   20    up     biz-internet  biz-internet  203.0.113.30
10.255.1.1   10    down   mpls          mpls          192.0.2.40
BR-01 has working tunnels to HUB-01 and HUB-02 over the internet transport, and the MPLS tunnel to HUB-01 is down. Control connections can be fine while this is true, because tunnels are built separately from them. The next question is why: the underlay on the MPLS circuit, restrict or colour settings, or a policy.

Learn more: Control Connections (DTLS and TLS)TLOCLogical Overlay and Encrypted Tunnel Construction

Steps 4 and 5: routes and traffic

With tunnels up, check that the service side has learned the routes it needs and that they are installed in the right VPN. A route from the overlay that is not installed is a routing or policy question, not an onboarding one, and later units cover it. For a first test, ping from a host behind BR-01 to a host in the same VPN behind HUB-01. A failed ping does not by itself prove a tunnel problem, since a host firewall can drop ICMP, so repeat with a different test if the routes and tunnels look right.

Reading the result

What you seeWhat it meansGo back to
No control connectionsReachability, trust or identity failedManual Bootstrap and PKI and Certificates
Control up, Manager sees device, no configurationNothing is attached yetTemplates and Configuration Groups
Control up, BFD sessions missingTunnel layer: underlay, NAT, colours, policyTLOC
BFD up, no routes in the VPNRouting or OMP policy; not an onboarding faultThe routing units later in the course

Rolling back

Plan the way back before the change, and pick the smallest one that works. In rising order of disruption:

Rolling back
Rolling backDetach the template and restore the earlier state.DetachtemplateRestorepreviousVerifycontrol connections

On small screens, swipe sideways or focus the diagram and use the arrow keys.

Keep a known good copy.
  1. Revert the last change. Detach the template or group, or restore the previous version of the template. For a push that failed, check whether your release already reverted it.
  2. Fix the variable or the shared setting. Most failures come from a wrong value, so correct it and push again.
  3. Redo the onboarding step. A fresh token for a virtual router if the first one expired or was used, a corrected root certificate, a corrected bootstrap line.
  4. Reset the SD-WAN configuration on the router. An IOS XE command exists for this. It removes the configuration the Manager pushed, so confirm in your release's guide what is kept.
  5. Leave controller mode. The router returns to autonomous mode and reloads. Use it to start a lab device from scratch.
request platform software sdwan config reset controller-mode disable

Both disrupt the router and can drop its connection. Run them on a lab or your own device, from the console, after reading your release's guide. Command forms and results are release dependent.

In practice, the rollback most often needed is the first or second, and the console matters more than any command: when a push removes the router from the Manager, the console is the way back in.

Mistakes in verification

  • Stopping at the Manager's status and calling the router done.
  • Checking only one transport. A router with two colours should show both working.
  • Testing with ping alone and reading a failure as a tunnel fault.
  • Resetting the whole router for a problem that was one wrong variable.
  • Starting a risky change on a remote site with no console or out-of-band access.

Test yourself on verification

Predict · scenario 1

Control connections on BR-01 are up on both transports, and show sdwan bfd sessions lists nothing. What does that suggest?

Predict · scenario 2

In the example output, the MPLS BFD session to HUB-01 is down while the internet sessions are up. What is a sensible first check?

Predict · scenario 3

A template push to a remote branch changes its only transport interface and the router drops off the Manager. Which preparation would have helped most?

Predict · scenario 4

Which is the least disruptive rollback for a template push that failed because one variable was wrong?

FAQ

Is a green status in the Manager enough proof of a good onboarding?
No. It shows the Manager can see the device. Tunnels to other sites and routes in the right VPNs are separate checks, and a device can be reachable by the Manager and still forward no user traffic.
Does a failed attempt harm the fabric?
A failed onboarding of one new router normally affects only that router. A pushed template or group that is shared with other devices is different: check how many devices it touches before pushing, and prefer a small first deployment.
How do I remove an SD-WAN configuration and start again?
There are commands to reset the SD-WAN configuration, and to leave controller mode, on IOS XE routers. They disrupt the router, and the exact forms are release dependent, so read your release's guide first and use them on a lab or your own device.