Course menu

Module 5: Users and VPNsLesson 5.2 (2 of 5 in this module)20 of 33 in the FortiGate Administrator course

IPsec VPN concepts

What IPsec protects, IKE phase 1 and phase 2, IKEv1 versus IKEv2, proposals, Diffie-Hellman and PFS, NAT traversal and route-based tunnels.

Intermediate · 12 min read

What you will learn

After this lesson, you can explain what phase 1 and phase 2 each agree, follow an IKEv2 exchange, and know which settings must match on both ends of a FortiGate tunnel.

  • IKE phases
  • Proposals and DH
  • NAT-T
  • Route-based VPN

IPsec is a suite of protocols that protects IP packets between two gateways. IKE (Internet Key Exchange) authenticates the peers and agrees keys in two phases: phase 1 builds a secure control channel (the IKE SA), and phase 2 builds the security associations that encrypt user traffic with ESP. On a FortiGate, a route-based tunnel appears as a virtual interface that routes and policies use like any other.

In simple terms: Two firewalls first prove who they are and agree a secret way to talk (phase 1). Then they agree how to lock the real traffic (phase 2) and send it through a tunnel.

A real-life situation

The company opens a branch office behind FGT2. Branch users need the file server at headquarters, and the traffic must cross the Internet without anyone being able to read or change it. Leasing a private line is expensive; an IPsec tunnel between FGT1 and FGT2 over the existing Internet links does the job.

What the tunnel does to a packet

1. The original packet (inside the private network)

IP headersrc 10.0.3.20dst 10.0.1.5
TCP headerdst port 445
Datafile request

2. The tunnelled packet (crossing the internet)

New IP headersrc 198.51.100.2dst 203.0.113.2
ESP headerSPI, sequence no.
🔒 Original IP header10.0.3.20 →10.0.1.5
🔒 TCP header
🔒 Data
🔒 ESP trailer
ESP authintegrity check

Encrypted: nobody on the path can read or change this part

FGT2 encrypts the branch PC's packet and wraps it in a new header between the two FortiGates' public addresses. On the Internet only 198.51.100.2 ↔ 203.0.113.2 with ESP is visible.

Building the tunnel: the IKEv2 exchange

FGT1HQ
FGT2branch
  1. Phase 1: IKE_SA_INITAgree encryption, integrity and DH group; exchange Diffie-Hellman values and nonces. From here on everything is encrypted.
  2. My proposals (AES-256, SHA-256, DH 14), my DH public value and a nonce.SA_INIT from FGT1 to FGT2.SA_INIT
  3. SA_INITI accept AES-256, SHA-256, DH 14; here is my DH value and nonce.SA_INIT from FGT2 to FGT1.
  4. Phase 1: IKE_AUTHEach side proves its identity (PSK or certificate). The first IPsec SA is created in the same exchange.
  5. I am FGT1, here is my proof. For user traffic: AES-256/SHA-256, selectors 10.0.1.0/24 ↔ 10.0.3.0/24.AUTH from FGT1 to FGT2.AUTH
  6. AUTHProof checks out. Here is mine; selectors accepted.AUTH from FGT2 to FGT1.
  7. Tunnel upTwo IPsec SAs exist, one per direction, each with its own SPI and keys. User traffic flows in ESP.
  8. ESPBranch PC's packet, encrypted with the outbound SA.ESP from FGT2 to FGT1.
IKEv2 sets up the IKE SA and the first IPsec SA in four messages. The pre-shared key itself is never sent; each side proves it knows the key.

IKEv1 reaches the same result with more messages: main mode (6 messages, identities protected) or aggressive mode (3 messages, identities sent in clear, used for some dial-up setups) for phase 1, then quick mode (3 messages) for each phase 2.

What each phase agrees

Phase 1 (IKE SA)Phase 2 (IPsec SA)
ProtectsIKE's own messagesUser traffic (ESP)
AuthenticationPre-shared key or certificatesInherited from phase 1
ProposalEncryption, integrity, DH groupEncryption, integrity, PFS (DH group) if enabled
Selectors–Local and remote subnets
FortiOS default lifetime86 400 s (24 h)43 200 s (12 h)

Diffie-Hellman lets the peers agree a shared secret over an open network. Larger groups are stronger: group 14 (2048-bit) is a reasonable minimum, and elliptic-curve groups 19–21 are stronger and faster. PFS repeats the DH exchange for each phase 2, so one stolen key doesn't unlock later traffic.

Keeping the tunnel healthy

  • Dead peer detection (DPD): if the peer stops answering, the FortiGate tears the SAs down so routing can fail over instead of sending into a dead tunnel.
  • NAT traversal: ESP has no ports, so it can't pass through most NAT devices. When IKE detects NAT on the path, both sides switch to UDP 4500 and wrap ESP in UDP.
  • Rekeying: before a lifetime runs out, new SAs are negotiated so traffic continues without interruption.

Route-based or policy-based?

A route-based (interface-mode) tunnel appears as a virtual interface, here to-BR1. Routes send traffic into it, and normal firewall policies allow it in each direction. A policy-based tunnel is selected by a firewall policy with action IPsec instead. Route-based tunnels are the FortiOS default and recommended choice: they work with dynamic routing, SD-WAN and redundant tunnels, and keep VPN policies looking like all the others.

Why it works this way

Two strangers on the Internet first need a private, authenticated channel before they can safely agree keys for bulk traffic. Splitting the work into two phases means the expensive authentication happens once, while user traffic keys can be changed often and cheaply.

Common mistakes

  • Different proposals or DH groups on each side: phase 1 or phase 2 fails with "no proposal chosen".
  • Selectors that aren't mirror images, or PFS on one side only.
  • Blocking UDP 500/4500 or ESP on an upstream firewall.
  • Weak choices kept for compatibility: DES, 3DES, MD5, SHA-1 and DH groups 1, 2 or 5.

Key takeaways

✅ Key takeaways
  • Phase 1 authenticates the peers and builds the IKE SA; phase 2 builds the IPsec SAs for user traffic.
  • IKEv2 needs four messages for the IKE SA and first IPsec SA; IKEv1 uses main or aggressive mode plus quick mode.
  • Proposals, DH groups, PFS and mirrored selectors must match on both ends.
  • NAT-T uses UDP 4500; DPD detects dead peers; route-based tunnels are the FortiOS default.

Check yourself

Predict · scenario 1

Which phase agrees the subnets whose traffic the tunnel will carry?

Predict · scenario 2

There is a NAT router between FGT2 and the Internet. Which port does the tunnel end up using?

Predict · scenario 3

What does PFS add?

FAQ

IKEv1 or IKEv2?
IKEv2 for new tunnels: fewer messages, built-in NAT traversal and dead peer detection, EAP for remote users, and clearer error messages. Use IKEv1 only when the other end doesn't support IKEv2.
What has to match on both FortiGates?
Phase 1: IKE version, mode (IKEv1), authentication (the same pre-shared key or trusted certificates), at least one common proposal and DH group, and compatible peer IDs. Phase 2: a common proposal, PFS and DH group if used, and mirror-image selectors (one side's local is the other side's remote).