Why a second edge changes everything
At a single-edge site, redistribution is a one-way pipe each way and is hard to get wrong. At site 103 there are two WAN Edges, BR-03A (10.255.2.3) and BR-03B (10.255.2.4), both in OSPF area 0 with the LAN core in service VPN 10. Each edge can pass routes between OMP and OSPF. When both do, a route can leave the overlay through one edge, cross the LAN, and re-enter the overlay through the other. This lab builds that setup, shows the failure, and fixes it. Site IDs, system IPs and prefixes are examples.
The two directions
On small screens, swipe sideways or focus the diagram and use the arrow keys.
- For OSPF, the commonly described default is that internal routes (intra-area and inter-area) go into OMP and external routes do not, until you enable them. Verify this for your release.
- OMP routes redistributed into OSPF normally appear in the core as external routes (Type 5 LSAs, advertised by an ASBR, an edge acting as the boundary router). Verify the type in your release.
- OMP routes are described as having an administrative distance of 251 on IOS XE SD-WAN, which is high, so an IGP route to the same prefix wins. Check the value on your release with
show ip route vrf 10 <prefix>.
That last point is what turns a mistake into an outage. If an edge learns a remote prefix both from OMP and from OSPF, the OSPF copy wins. The edge then forwards towards the LAN instead of into the tunnel, and the packet goes to the other edge, which may do the same thing. See the administrative distance lesson if that term is new.
How the loop forms, step by step
- HUB-01 advertises 10.20.1.0/24 into OMP. Both BR-03A and BR-03B receive it.
- BR-03A redistributes OMP into OSPF. The core now has 10.20.1.0/24 as an external route from BR-03A.
- BR-03B hears the OSPF route from the core. If BR-03B also advertises OSPF external routes into OMP, it announces 10.20.1.0/24 to the overlay as a route it owns.
- BR-03B has two copies of the prefix: the OMP one (distance 251) and the OSPF one (a lower distance). The OSPF copy wins, so BR-03B sends traffic for 10.20.1.0/24 across the LAN towards BR-03A.
- BR-03A has the real OMP route and sends it into a tunnel. In the good case that works. If BR-03A also prefers an OSPF copy learned from BR-03B's advertisement, packets bounce between the two edges until the TTL (time to live) reaches zero.
Whether you get a bounce, a detour or a black hole depends on which combination of redistribution and route preference is configured. All of them are wrong, and all of them start with two redistribution points.
On small screens, swipe sideways or focus the diagram and use the arrow keys.
Recognising the symptoms
- A traceroute from a site 103 host shows the same two hops repeating (BR-03A, then BR-03B, then BR-03A) and ends with TTL exceeded messages.
- Latency to a remote site jumps, or only some flows fail.
- High CPU on an edge as it forwards the same packets repeatedly.
- A remote site sees site 103 advertising prefixes that belong to somewhere else.
Check first: which edge installs the prefix from which source. Then look at who originates the prefix in OMP.
BR-03B#show ip route vrf 10 10.20.1.0 Routing Table: 10 Routing entry for 10.20.1.0/24 Known via "ospf 10", distance 110, metric 20, type extern 2 Last update from 10.10.3.2 on GigabitEthernet5, 00:04:12 ago Routing Descriptor Blocks: * 10.10.3.2, from 10.255.2.3, 00:04:12 ago, via GigabitEthernet5
HUB-01#show sdwan omp routes 10.20.1.0/24 VPN PREFIX FROM PEER PATH ID LABEL STATUS TLOC IP COLOR PREF 10 10.20.1.0/24 10.255.0.11 301 1002 C,I,R 10.255.2.4 biz-internet -
The loop-prevention toolkit
On small screens, swipe sideways or focus the diagram and use the arrow keys.
route-map OMP-TO-OSPF permit 10
set tag 3000
!
route-map OSPF-TO-OMP deny 10
match tag 3000
!
route-map OSPF-TO-OMP permit 20
!
router ospf 10 vrf 10
redistribute omp subnets route-map OMP-TO-OSPFSketch only. On IOS XE SD-WAN the redistribution into OMP and the route policy that filters it are normally built in SD-WAN Manager templates, and the keywords here (redistribute omp, how the second route-map is attached) are not verified for your release. The tag value 3000 is an example. Verify the keywords in your release before use.
The idea is two plain-language rules: tag everything that comes out of the overlay, and refuse anything with that tag on its way back in. It works at a dual-edge site because the tag travels with the OSPF route to the other edge, where the deny rule catches it.
Tasks
Task 1: map the redistribution points
List every point at site 103 where a route moves between OMP and OSPF.
Show answer
Four, if both edges redistribute in both directions: BR-03A OMP into OSPF, BR-03A OSPF into OMP, BR-03B OMP into OSPF and BR-03B OSPF into OMP. Each pair can feed the other edge's pair. A good first step is to reduce this number.
Task 2: apply the tag rules
On both edges, tag routes from OMP into OSPF and deny that tag from OSPF into OMP. What should the OSPF routes look like in the core?
Show answer
10.20.1.0/24, 10.10.1.0/24 and 10.10.2.0/24 appear in the core with tag 3000. Check with show ip ospf database external and look for the tag value on those LSAs.
BR-03B#show ip ospf database external 10.20.1.0 OSPF Router with ID (10.255.2.4) (Process ID 10 VRF 10) Type-5 AS External Link States LS age: 212 Link State ID: 10.20.1.0 (External Network Number) Advertising Router: 10.255.2.3 Network Mask: /24 Metric Type: 2 (Larger than any link state path) Metric: 20 External Route Tag: 3000
Task 3: confirm the overlay is clean
Which source should the data centre prefix show in the branch's routing table now?
Show answer
OMP, not OSPF. The OSPF copy still exists in the OSPF database, but check that the prefix is installed from OMP. If an OSPF copy is still installed instead, look at administrative distance on the edge and at whether your release lets you prefer OMP for those prefixes.
BR-03B#show sdwan omp routes 10.20.1.0/24 VPN PREFIX FROM PEER PATH ID LABEL STATUS TLOC IP COLOR PREF 10 10.20.1.0/24 10.255.0.11 101 1002 C,I,R 10.255.1.1 biz-internet - 10 10.20.1.0/24 10.255.0.11 201 1002 R 10.255.1.2 biz-internet -
Optional part: eBGP to a legacy router
A legacy router behind site 103 may speak eBGP (external BGP, between different autonomous systems) with the edges. The same pattern applies with BGP tools. Mark routes from OMP with a community when sending them to the legacy router. Filter on that community when accepting routes from BGP into OMP at the other edge. Keep AS-path loop detection on its default behaviour unless you have a reason: two edges in one AS across a legacy router can reject each other's routes and break the path rather than cause a loop. See the BGP course for communities and AS paths.
Break and fix
Fault 1: tag set, but not filtered on the way back
Symptom: the core shows tag 3000 on remote prefixes, yet HUB-01 still sees BR-03B as an origin of 10.20.1.0/24.
Diagnosis and fix
The first rule works and the second does not. Check that the route policy that denies tag 3000 is attached to the OSPF-into-OMP direction on both edges, and that the OSPF routes it sees really carry the tag. Fix the attachment, then clear the route from OMP and re-check with show sdwan omp routes.
Fault 2: external OSPF routes not advertised into OMP
Symptom: a static default route redistributed into OSPF at the LAN core is missing at remote sites.
Diagnosis and fix
The edge advertises only internal OSPF routes into OMP unless external ones are enabled for that VPN. Confirm the route is in show ip route vrf 10 on the edge, then enable the external route type for advertisement. When doing so, keep the tag filter in place, since external routes are exactly what can loop.
Fault 3: one edge does both jobs and its failure takes the site
You reduced redistribution to BR-03A only. When BR-03A fails, the core loses all remote prefixes.
Diagnosis and fix
A single redistribution point is a single point of failure. Either accept it knowingly and test the failure, or use both edges with the tag rules from this lab, and different metrics so the core prefers one. Re-run the failure test and record how long the core takes to converge.
What to record
Keep the redistribution map (who redistributes what, in which direction), the tag value and where it is set and denied, the administrative distances you saw on your release, and a before and after of show sdwan omp routes for one remote prefix. Note the release and model, since keywords differ.
Test yourself on redistribution
BR-03B shows 10.20.1.0/24 installed from OSPF, next hop BR-03A, although HUB-01 is the origin. Why is the OSPF route winning over the OMP one?
You want to stop routes taken from OMP into OSPF at BR-03A from returning to OMP at BR-03B. What is the clean approach?
A traceroute from a site 103 host to 10.20.1.10 alternates between 10.10.3.2 and 10.10.3.3 and ends in TTL exceeded. What is the best first check?
Remote sites do not receive a default route that the LAN core injects into OSPF as an external route. The edge has it in its VPN 10 table. What is the likely reason?