A fibre cut that took the branch offline for three hours
Northwind Freight runs its headquarters behind FGT1 and a depot behind FGT2. Depot staff open delivery notes on FILE1 (10.0.1.5) through the tunnel to-BR1, which leaves FGT1 on port1 over the fibre ISP. Last month the fibre was cut by road works. ISP2, the broadband line on port4 added for SD-WAN, stayed up the whole time, but the tunnel couldn't use it: a phase 1 is tied to one interface. The depot was cut off until the fibre was repaired.
The fix is a second tunnel, to-BR1-B, from port4 to the same branch FortiGate, plus routes that prefer the fibre tunnel and fall back to the broadband one on their own.
- 1. Normally: Both tunnels are up; the routes prefer to-BR1 over ISP1.
- 2. ISP1 fails: DPD declares to-BR1 dead, its route is withdrawn, and traffic uses to-BR1-B.
Counting the paths: partial or full redundancy
Each tunnel joins one local interface to one remote address. The number of possible tunnels is the number of local Internet links multiplied by the number of remote ones.
| Design | HQ links | Branch links | Tunnels | Survives |
|---|---|---|---|---|
| Single tunnel | 1 | 1 | 1 | Nothing on the path |
| Partially redundant (this lab) | 2 (port1, port4) | 1 | 2 | Loss of either HQ ISP |
| Fully redundant | 2 | 2 | up to 4 | Loss of any one ISP at either site |
The NSE 4 task wording, "meshed or partially redundant", maps onto this table: partial redundancy is what most branches can afford, and the same building blocks scale up to the four-tunnel version when the branch gets a second line.
What moves the traffic from to-BR1 to to-BR1-B
Nothing in IPsec itself fails over. The FortiGate combines three ordinary mechanisms, and each has to be in place:
The weak link in that chain is step 1. Without DPD, a tunnel whose peer has vanished can stay "up" for the rest of its SA lifetime, its route stays in the table, and traffic disappears into it.
Step 1: make sure port4 can reach the branch
FortiOS sends a tunnel's IKE and ESP packets out of the interface its phase 1 is bound to, and it needs an active route to the remote gateway through that interface. In the routing lesson, the ISP2 default route had distance 20, which keeps it out of the routing table while ISP1 is up. That is fine for Internet traffic, but it leaves to-BR1-B with no usable route to 198.51.100.2, so it can't come up.
config router static
edit 2
set gateway 192.0.2.1
set device "port4"
set distance 10
set priority 10
next
endSame distance as the ISP1 default route (10), higher priority value (10 against 1). Both default routes are now in the routing table: new Internet sessions still prefer port1, and port4 has an active route for the backup tunnel. Route 2 is the ISP2 default route from the routing lesson; use your own entry number.
If port1 and port4 are SD-WAN members, the SD-WAN default route already covers both and this step is not needed. In that design the tunnels can become SD-WAN members too; the Dual VPN tunnel wizard under Network › SD-WAN builds that version.
Learn more: Routing on a FortiGateSD-WAN
Step 2: the backup phase 1 and phase 2 on FGT1
config vpn ipsec phase1-interface
edit "to-BR1"
set dpd on-idle
set dpd-retrycount 3
set dpd-retryinterval 5
next
edit "to-BR1-B"
set interface "port4"
set ike-version 2
set peertype any
set net-device disable
set proposal aes256-sha256 aes256gcm-prfsha384
set dhgrp 19 14
set remote-gw 198.51.100.2
set psksecret <another-long-random-key>
set dpd on-idle
set dpd-retrycount 3
set dpd-retryinterval 5
next
end
config vpn ipsec phase2-interface
edit "to-BR1"
set auto-negotiate enable
next
edit "to-BR1-B"
set phase1name "to-BR1-B"
set proposal aes256-sha256 aes256gcm
set pfs enable
set dhgrp 19 14
set src-subnet 10.0.1.0 255.255.255.0
set dst-subnet 10.0.3.0 255.255.255.0
set auto-negotiate enable
next
endto-BR1-B is the same tunnel as to-BR1 except for the interface (port4) and its own key. Shorter DPD timers on both tunnels mean a dead peer is noticed in roughly 15 seconds (3 retries, 5 seconds apart) instead of waiting on the default timers. auto-negotiate keeps the phase 2 SAs up without waiting for traffic, so the backup is ready before it is needed.
dpd has three settings. on-demand (the default) probes only when the FortiGate sends traffic into the tunnel and hears nothing back. on-idle probes when the tunnel is quiet. A backup tunnel is quiet most of the time, so on-idle is the setting that tells you it is healthy before you depend on it. dpd-retrycount defaults to 3.
FGT2 gets the mirror image. It has one Internet link, so both of its phase 1 entries use port1:
| FGT2 entry | interface | remote-gw | Selectors (src → dst) |
|---|---|---|---|
| to-HQ (existing) | port1 | 203.0.113.2 | 10.0.3.0/24 → 10.0.1.0/24 |
| to-HQ-B (new) | port1 | 192.0.2.2 | 10.0.3.0/24 → 10.0.1.0/24 |
FGT2 tells the two tunnels apart by the address the IKE packets come from: 203.0.113.2 matches to-HQ, 192.0.2.2 matches to-HQ-B. Use the same DPD timers and auto-negotiate on both sides.
Step 3: routes that prefer to-BR1
config router static
edit 22
set dst 10.0.3.0 255.255.255.0
set device "to-BR1-B"
set priority 10
next
endRoute 20 (via to-BR1) and the distance-254 blackhole (route 21) already exist from the site-to-site lesson. Route 22 has the same default distance (10) as route 20 but a higher priority value, so both are installed and to-BR1 (priority 1) carries new sessions.
Why priority and not a higher distance? A route with a higher distance waits outside the routing table. If FGT2 ever sends a packet through to-HQ-B while FGT1 still prefers to-BR1 (one side noticed a failure before the other, or FGT2's routes differ), FGT1's reverse path check finds no active route to 10.0.3.0/24 through to-BR1-B and drops it. With equal distance, both routes are active and either tunnel can receive traffic. On FGT2, add the matching routes: 10.0.1.0/24 via to-HQ (priority 1) and via to-HQ-B (priority 10).
Step 4: one set of policies for both tunnels
config firewall policy
edit 10
set dstintf "to-BR1" "to-BR1-B"
next
edit 11
set srcintf "to-BR1" "to-BR1-B"
next
endPolicies 10 (HQ-to-BR1) and 11 (BR1-to-HQ) now accept either tunnel, so the rules can't drift apart. The alternative is a second pair of policies for to-BR1-B, which is what the Fortinet example uses; either works, but a missing backup policy is easy to miss until the day it is needed.
On FGT2, do the same with to-HQ and to-HQ-B. Putting each pair of tunnels in a zone is a third option and keeps the policy list short when there are many branches.
A standby tunnel instead of two live ones
FortiOS can also keep the backup phase 1 down until the primary fails, with the monitor option (CLI only). The backup then negotiates only when DPD declares the primary dead, which saves a little IKE traffic but adds negotiation time to every failover.
config vpn ipsec phase1-interface
edit "to-BR1-B"
set monitor "to-BR1"
set monitor-hold-down-type delay
set monitor-hold-down-delay 300
next
endmonitor names the primary phase 1 this tunnel backs up. The hold-down settings control failback: immediate (the default), after a delay in seconds, or at a set time of day. Waiting 300 seconds keeps a flapping ISP1 from bouncing traffic back and forth.
Two live tunnels with route priority are easier to verify, because you can see the backup is up before you need it. The monitor approach suits metered links where even idle IKE traffic costs money.
What the VPN wizard builds for you
For a new site, the VPN wizard creates the whole first tunnel in one pass. Choose the Site to Site template, a FortiGate peer that is accessible with a static address, the outgoing interface and the local LAN:
VPN Setup: Site to Site
- Tunnel name
- to-BR2
- Template
- Site to Site
- Remote site device type
- FortiGate
- Remote site device
- Accessible and static
- IP address
- 198.51.100.10
- Remote site subnets
- 10.0.4.0/24Becomes an address object, a route and the phase 2 selector.
- Outgoing interface that binds to tunnel
- port1
- Local interface
- port2
- Local subnets
- 10.0.1.0/24
Remote site
Local site
The wizard then creates, all tagged with the comment "Created by VPN wizard":
- Address objects and groups for the local and remote subnets, with routing allowed on them.
- The phase 1 and phase 2, with the address groups as phase 2 selectors.
- A static route to the remote group through the tunnel, and a distance-254 blackhole route.
- Two firewall policies, LAN to tunnel and tunnel to LAN, without NAT.
For the second tunnel of a redundant pair, the Custom template is the better fit: it creates only the phase 1 and phase 2, and you add the route (with its priority) and the policy interfaces yourself, as in steps 3 and 4. Running the Site to Site template twice would add a second blackhole and a second pair of policies for the same subnets.
Checking both tunnels and the routes
FGT1 # get vpn ipsec tunnel summary 'to-BR1' 198.51.100.2:0 selectors(total,up): 1/1 rx(pkt,err): 8812/0 tx(pkt,err): 8790/0 'to-BR1-B' 198.51.100.2:0 selectors(total,up): 1/1 rx(pkt,err): 6/0 tx(pkt,err): 6/0
FGT1 # get router info routing-table all S 10.0.3.0/24 [10/0] via to-BR1 tunnel 198.51.100.2, [1/0] [10/0] via to-BR1-B tunnel 198.51.100.2, [10/0]
FGT1 # diagnose vpn tunnel list name to-BR1-B name=to-BR1-B ver=2 serial=7 192.0.2.2:0->198.51.100.2:0 ... ... dpd: mode=on-idle on=1 idle=5000ms retry=3 count=0 seqno=412 ...
idle=5000ms is the 5-second retry interval and count=0 is what a healthy tunnel shows; a rising count is the first sign of trouble.Test failover in a maintenance window
set status down on port1. That drops every session using port1, moves HQ Internet traffic to port4 as well as the branch traffic to to-BR1-B, and lasts until you set port1 back up. Warn the depot and HQ users first, and keep a console or port2 session open so you don't lose your own access.config system interface
edit "port1"
set status down
next
endUndo with set status up. A downed interface removes to-BR1 at once; it tests the routing, not DPD (the exercise below tests DPD).
With port1 down, the routing table should show only the route via to-BR1-B, a continuous ping from BR-PC to FILE1 should resume after a short gap, and get vpn ipsec tunnel summary should show the backup's counters climbing. Set port1 up again: once to-BR1 has renegotiated, new sessions return to it.
Reading the VPN event logs when a tunnel flaps
Every phase 1 and phase 2 change is logged as a VPN event: Log & Report › System Events, then the VPN Events subtype (shown when VPN is enabled in System › Feature Visibility). The fields that matter are the tunnel name, the action and the local and remote addresses:
logdesc="IPsec connection status changed" action="tunnel-down" remip=198.51.100.2 locip=203.0.113.2 outintf="port1" vpntunnel="to-BR1" duration=86211 ... logdesc="IPsec phase 1 SA deleted" action="delete_phase1_sa" remip=198.51.100.2 locip=203.0.113.2 outintf="port1" vpntunnel="to-BR1" ... logdesc="IPsec connection status changed" action="tunnel-up" remip=198.51.100.2 locip=203.0.113.2 outintf="port1" vpntunnel="to-BR1" ...
Three questions narrow down a flapping tunnel quickly:
- Does the interface flap too? System events for port1 going down and up at the same times point at the ISP or the cable, not IPsec.
- Only one tunnel, or both? If
to-BR1andto-BR1-Bdrop together, the problem is at the branch end (FGT2 or its single link). - Are the DPD timers too tight for the link? A busy broadband line that loses a few probes can trigger DPD although the peer is fine. Raising
dpd-retryintervalon that tunnel trades slower failover for fewer false alarms.
In practice, the ticket that matters is rarely the failover itself. It is the backup tunnel that has been down for months without anyone noticing, found on the day ISP1 fails. A weekly look at the IPsec widget, or an alert on tunnel-down events for the backup, prevents that.
Hub-and-spoke or full mesh when the third site arrives
With more branches, the question becomes which sites get tunnels to which. A full mesh of n sites needs n(n−1)/2 tunnels: 3 for three sites, 45 for ten, before doubling them for redundancy.
| Hub-and-spoke | Full mesh | |
|---|---|---|
| Tunnels (10 sites) | 9 | 45 |
| Branch to branch | Through the hub (extra hop, hub bandwidth) | Direct |
| Adding a site | Configure the new spoke and the hub | Touch every existing site |
| Weak point | The hub (so it is often an HA cluster or one of two hubs) | Configuration effort and mistakes |
In a hub-and-spoke design, spoke-to-spoke traffic enters the hub through one tunnel and leaves through another. The hub therefore needs a policy from the tunnel interface back to the tunnel interface (or between the spoke tunnels), and each spoke needs routes for the other spokes' subnets pointing at its hub tunnel. A partial mesh sits in between: hub-and-spoke for most sites, plus direct tunnels between the few branches that exchange a lot of traffic.
The wizard's Hub-and-Spoke template builds the hub side and produces configuration keys for the spokes. For many sites, ADVPN (Auto-Discovery VPN) keeps a hub-and-spoke configuration but lets spokes build short-lived direct tunnels on demand; it is beyond this lesson.
Where redundant tunnels go wrong
The backup tunnel stays down
get vpn ipsec tunnel summary shows to-BR1-B with selectors(total,up): 1/0 and zero counters, and diagnose vpn ike gateway list name to-BR1-B shows no established IKE SA. The primary works.
Show the cause and the fix
FGT1 has no active route to 198.51.100.2 through port4: the ISP2 default route still has a higher distance, so it is not in the routing table. get router info routing-table all shows only the port1 default route. Fix: give the ISP2 route the same distance and a higher priority (step 1), or add a more specific route to 198.51.100.2 that is active through port4. Check that ISP2 doesn't block UDP 500 and 4500.
Failover happens, but branch traffic still drops
During the test the routing table shows only the route via to-BR1-B, the tunnel counters on to-BR1-B rise for outgoing traffic, but nothing comes back from the branch.
Show the cause and the fix
The packets leave FGT1 through the backup tunnel, so FGT1's side is fine. On FGT2, policy 11's equivalent only lists to-HQ, so traffic arriving on to-HQ-B matches no policy and is denied (debug flow on FGT2 shows the implicit deny, policy 0). Fix: add to-HQ-B to FGT2's policies, then check FGT2 has its own routes via to-HQ-B. A redundant design is two configurations; test from both ends.
ISP1 is down, but traffic still goes into to-BR1 for minutes
The VPN event log shows no tunnel-down for to-BR1 until long after the outage began, and the routing table keeps the route via to-BR1.
Show the cause and the fix
DPD is disabled on to-BR1 (set dpd disable), or set very loosely, so FGT1 has no way to notice the peer has gone. The tunnel stays up until its SA lifetime ends, and packets vanish into it. If port1 itself had gone down, the interface state would have removed the route at once; an upstream failure needs DPD. Fix: enable DPD (on-idle) with sensible retry values on both tunnels and both FortiGates.
Replies from the branch are dropped after a one-sided failover
FGT1 # (debug flow on FGT1, trimmed) msg="vd-root:0 received a packet(proto=6, 10.0.3.20:51544->10.0.1.5:445) from to-BR1-B. flag [S], ..." msg="reverse path check fail, drop"
Show the cause and the fix
FGT1's route via to-BR1-B was configured with a higher distance (for example 15) instead of a higher priority. It sits in the routing database but not in the routing table, so the reverse path check finds no route back to 10.0.3.20 through to-BR1-B. Fix: use the same distance and a higher priority, so both routes are active. Then align FGT2's preferences with FGT1's so both sides normally use the same tunnel.
Try it: prove the backup carries the depot
- Build
to-BR1-Bandto-HQ-Bas in steps 1 to 4, on both FortiGates. - Confirm both tunnels show
1/1inget vpn ipsec tunnel summary, and that the routing table holds both branch routes. - Start a continuous ping from BR-PC to FILE1. In a maintenance window, set port1 down and count the lost replies.
- Find the
tunnel-downevent forto-BR1in the VPN event log and note its time. - Set port1 up again. Which tunnel does a new SMB session to FILE1 use, and how can you tell from the counters?
Extra: on a lab FortiGate only, block the tunnel upstream of port1 (for example, drop UDP 500/4500 and ESP on the lab ISP router) so port1 stays up. Compare failover time with DPD enabled and disabled on to-BR1.
- One phase 1 and phase 2 per path, each bound to its own WAN interface, on both FortiGates.
- An active route to the remote gateway through each WAN interface.
- DPD on every tunnel; on-idle so the quiet backup is checked too.
- Tunnel routes with equal distance and different priority, plus one blackhole.
- Policies that include every tunnel interface, at both ends.
Test yourself on redundant and meshed VPNs
FGT1 has two ISPs and FGT2 has one. How many tunnels give the most redundancy between them?
Routes to 10.0.3.0/24: via to-BR1 distance 10 priority 1, via to-BR1-B distance 10 priority 10. Both tunnels are up. Which tunnel do new sessions to the branch use?
ISP1's upstream network fails but port1 stays physically up. DPD is disabled on to-BR1. What happens to branch traffic?
The VPN event log shows tunnel-down then tunnel-up for to-BR1-B every 3 to 4 minutes, while to-BR1 is stable and port4 stays up. What is the most likely cause?
A company has 12 branches that mostly talk to HQ, plus two warehouses that exchange large files daily. Which design fits?