A week in the life of the FGT1 cluster
Since the last lesson, Northwind Freight's headquarters firewall is a pair: FGT1-A (priority 200) and FGT1-B (priority 100), active-passive, heartbeats on port7 and port8, session pickup enabled, override disabled. Three requests are in this week's queue:
- The monitoring team wants to poll the CPU and memory of each unit separately, not just "the firewall".
- Video-call users complained that their calls dropped during last month's failover test, although web browsing carried on.
- A FortiOS 7.6 patch release has to go on during Saturday's maintenance window, without taking the office offline.
Each request touches a different part of cluster operations. First, what the cluster does on an ordinary day.
What the cluster keeps in step, and what it doesn't
In active-passive mode only the primary forwards traffic. It synchronises its configuration to the other units with a mix of incremental updates (each change as it is made) and periodic full checks, all over the heartbeat links using TCP and UDP port 703. A handful of settings stay on each unit:
| Synchronised from the primary | Kept per unit |
|---|---|
| Policies, objects, routes, VPNs, security profiles | Host name |
| Interfaces (except the reserved management one) | HA device priority and override |
| Most HA settings (mode, group, heartbeat, session pickup) | Reserved management interface settings and its gateway |
| FortiToken Mobile tokens | Licences and subscriptions; dashboard widgets |
Logs are per unit too. In the GUI, the event log viewer has an HA device selector for reading the other unit's messages. From the primary's CLI, execute ha manage opens a session on another unit:
execute ha manage 1 adminConnects to cluster member 1 (the secondary here) as the admin user; the prompt changes to that unit's host name. Type exit to return to the primary.
The health check that starts every cluster task:
FGT1-A # get system ha status HA Health Status: OK Model: FortiGate-VM64 Mode: HA A-P Group Name: FGT1-HA ... ses_pickup: enable override: disable Configuration Status: FGVM0000000A(updated 1 seconds ago): in-sync FGVM0000000B(updated 2 seconds ago): in-sync ... Primary : FGT1-A , FGVM0000000A, HA cluster index = 0 Secondary : FGT1-B , FGVM0000000B, HA cluster index = 1
out-of-sync, System › HA and the HA Status dashboard widget mark it in red, and hovering over it names the configuration tables whose checksums differ.Session pickup: which connections survive a failover
The previous lesson enabled session-pickup. It synchronises the primary's TCP session table to the other units, so most TCP connections passing through resume on the new primary. It doesn't cover everything, which is exactly what the video-call complaint is about:
| Traffic | After a failover |
|---|---|
| TCP through the cluster | Resumes, with session pickup enabled |
| UDP and ICMP (VoIP media, many video calls, QUIC/HTTP/3) | Resumes only with session-pickup-connectionless as well |
| Sessions with flow-based inspection | Resume, but are not inspected after the failover |
| Sessions with proxy-based inspection | Restart: the proxy's state isn't synchronised |
| IPsec tunnels ending on the cluster | Stay up: SAs are synchronised whether or not session pickup is on |
| Admin sessions to the cluster (GUI, SSH), FortiManager and FortiAnalyzer connections | Restart |
config system ha
set session-pickup enable
set session-pickup-connectionless enable
set session-pickup-delay enable
set session-sync-dev "port6"
endconnectionless adds UDP and ICMP sessions (it needs session-pickup enabled first). delay synchronises only sessions older than 30 seconds, which cuts sync traffic on a cluster full of short web sessions at the cost of those short sessions restarting. session-sync-dev moves session sync off the heartbeat links onto port6 (cabled directly between the units); if port6 fails, sync falls back to the heartbeat links.
For Northwind's video calls, session-pickup-connectionless is the fix. The other two are tuning for busy clusters: enable them when heartbeat traffic or CPU on the secondary shows the session sync load, not by default. The video team's calls are long-lived, so session-pickup-delay wouldn't hurt them, but short DNS lookups and API calls would no longer be carried across.
Giving each unit its own management address
The cluster's interfaces use virtual MAC addresses and move with the primary role, so 10.0.1.1 on port2 means "whichever unit is primary at the moment". A monitoring system polling 10.0.1.1 sees the primary only. A reserved management interface fixes that: each unit keeps one port (up to four) with its own address, its own hardware MAC address, and its own default route for management.
Northwind uses port5 on both units, connected to the management VLAN 10.0.99.0/24 behind the core switch at 10.0.99.1. On FGT1-A:
config system interface
edit "port5"
set ip 10.0.99.11 255.255.255.0
set allowaccess ping https ssh snmp
next
end
config system ha
set ha-mgmt-status enable
config ha-mgmt-interfaces
edit 1
set interface "port5"
set gateway 10.0.99.1
next
end
set ha-direct enable
endha-mgmt-interfaces gives port5 its own default route (dst defaults to 0.0.0.0/0) that only management traffic uses. ha-direct makes this unit send its own syslog, FortiAnalyzer and RADIUS/LDAP traffic out of port5, so each unit reports as itself (SNMP has its own ha-direct switch, below). Enabling ha-direct clears source-ip settings for logging, NetFlow and sFlow.
The interface settings and the management gateway are not synchronised, so repeat them on FGT1-B with its own address. From FGT1-A, execute ha manage 1 admin, then the same commands with set ip 10.0.99.12 255.255.255.0. Afterwards, https://10.0.99.11 and https://10.0.99.12 reach each unit directly, whichever is primary.
- SNMP: allow
snmpon port5, and enableha-directon the SNMP community host (and SNMP users) as well; the SNMP manager then polls each unit's own address for its CPU and memory. - FortiManager: manage the cluster through a cluster interface address, not a reserved management address.
- No spare port? An in-band management address (
set management-ipon an ordinary interface) gives each unit an extra address in that interface's subnet without reserving the port. It is not synchronised either, so set it on each unit.
In practice, the reserved management interface pays for itself on upgrade day: it is the only way to reach the secondary's GUI directly and see that it really runs the new firmware.
Changing HA settings without an unplanned failover
Make HA changes on the primary, in System › HA (edit the primary unit) or with config system ha. Most HA settings then synchronise. The ones to treat with care:
| Setting | What a change does |
|---|---|
| group-name, password, mode, group-id | Units only join if these match. A mismatch splits the cluster; a different group-id also changes the virtual MAC addresses. |
| hbdev, hb-interval, hb-lost-threshold | Heartbeats every 200 ms (hb-interval 2) and a failure after 6 lost (the default on many models). Raising them avoids false failovers on busy links but slows detection. |
| monitor | A monitored port that is down on the primary makes it lose the election. Check the cable on every unit before adding a port. |
| priority, override | Per unit, not synchronised. Set override the same way on every unit; mixed settings make elections unpredictable. |
| Anything that asks to reboot | Can reboot every unit at the same time. |
Settings that need a reboot reboot the whole cluster
y.Moving the primary role on purpose
To test failover, or to move traffic before maintenance on one unit, force it from the CLI of the primary:
Forced failover is for tests and maintenance windows
execute ha failover set 1 makes the current primary give up the role and keeps it secondary until you run execute ha failover unset 1; unsetting can trigger a second failover. Sessions that session pickup doesn't cover will restart. Fortinet's guidance is to use it only for testing, troubleshooting, maintenance and demonstrations.execute ha failover set 1
execute ha failover status
get system ha status1 is the cluster ID when virtual clustering is off. The status command shows whether the flag is set, and get system ha status explains the new choice of primary (the EXE_FAIL_OVER flag on the old one).
After unset, the normal election decides again, and the "Primary selected using" lines say which criterion won. Two other ways to move the role, both used in Fortinet's staged-upgrade procedure: with override disabled, diagnose sys ha reset-uptime on the unit that should give up the role resets its HA uptime; with override enabled, changing the priorities does it.
Learn more: High Availability (FGCP)
Saturday night: upgrading the cluster's firmware
A cluster is upgraded like a single FortiGate: the image is uploaded once, to the cluster, and the cluster handles the members. How it does that depends on upgrade-mode. The default, uninterruptible, upgrades without interrupting traffic:
Step 3 is a failover. Sessions follow the same rules as any failover: TCP sessions resume with session pickup, UDP needs the connectionless option, proxy-inspected sessions restart. Which unit is primary at the end depends on the election at step 5, so read the "Primary selected using" lines afterwards rather than assuming FGT1-A is back in charge.
Two details from the 7.6 documentation decide how the night goes. If every secondary crashes or stops responding during the upgrade, the primary carries on normally on the old firmware and isn't upgraded until at least one secondary rejoins. And the primary waits at most uninterruptible-primary-wait minutes (default 30, range 15 to 300) for a secondary to be considered upgraded and running before it starts its own upgrade.
Before uploading firmware to a production cluster
- Back up the configuration, and schedule a maintenance window even for an uninterruptible upgrade: step 3 is a real failover.
- Check the supported upgrade path with Fortinet's upgrade tool; the GUI wizard can follow the path for you.
- Confirm
get system ha statusshows both units in sync and healthy first, and thatupgrade-modeisuninterruptible. - Don't combine the upgrade with CLI changes that need a reboot: uninterruptible mode does not stagger those reboots.
In the GUI: System › Firmware & Registration, select the FortiGate, Upgrade, then choose the firmware (or upload a file), the schedule, and confirm on the review page, which backs up the configuration first. The CLI equivalent uploads the image from a TFTP server with execute restore image tftp <file> <server-ip>.
| upgrade-mode | What happens |
|---|---|
| uninterruptible (default) | Secondaries first, then the primary, as above |
| simultaneous | All units upgrade at once: quicker, but traffic stops while they reboot |
| local-only | Only the unit the image is uploaded to |
| secondary-only | Only the secondary units |
local-only and secondary-only are for a staged, multi-version upgrade: upgrade one unit, move traffic to it, test, and only then upgrade the rest. While the units run different builds, sessions and routes still synchronise but configuration changes do not, so freeze changes until every unit matches. Afterwards, set upgrade-mode back to uninterruptible.
set uninterruptible-upgrade enable (now upgrade-mode uninterruptible) or disable (now simultaneous). The upgrade-mode setting, with the local-only and secondary-only options, arrived in 7.4.1. This lesson is written for FortiOS 7.6; 7.4 has not yet been reviewed.After the upgrade, check each unit through its own management address:
get system status
get system ha statusget system status on each unit shows the running version and build; both must match. get system ha status should show both units in-sync and say why the current primary was selected.
Cluster problems an operator actually meets
The secondary's management address doesn't answer
https://10.0.99.11 opens FGT1-A, but 10.0.99.12 times out. The cluster is healthy and in sync.
Show the cause and the fix
The reserved management interface settings were configured on FGT1-A only. They are not synchronised, so FGT1-B's port5 has no address (or no management gateway). Fix: execute ha manage 1 admin from FGT1-A, then set port5's IP address and access, and the ha-mgmt-interfaces entry, on FGT1-B.
FortiAnalyzer only ever sees one serial number
Logs arrive, but all from the primary's serial number, even for events that happened on the secondary.
Show the cause and the fix
Without ha-direct, management services such as remote logging use a cluster interface, so the traffic leaves through the primary. Fix: enable ha-direct under config system ha so each unit logs out of its own reserved management interface, and make sure FortiAnalyzer can reach both management addresses.
The upgrade stopped halfway
FGT1-A # get system ha status ... Configuration Status: FGVM0000000A(updated 2 seconds ago): in-sync ... Primary : FGT1-A , FGVM0000000A, HA cluster index = 0 ...
get system status shows it still on the old build.Show the cause and the fix
FGT1-B didn't come back into the cluster after its upgrade (a failed boot, a heartbeat cable, or a problem with the new image). As designed, the primary keeps running the old firmware and waits rather than upgrading itself and leaving no working unit. Fix: traffic is unaffected, so there is time. Connect to FGT1-B's console, find out why it didn't rejoin, and fix or roll back that unit before letting the upgrade continue.
Everyone was logged out of the GUI at the failover test
Show why this is expected
Administrative sessions terminate on the cluster itself, and sessions terminated by the cluster are not failed over (IPsec SAs are the main exception). Users' traffic through the cluster continued; only the admin sessions had to log in again. Nothing needs fixing.
Try it: run Northwind's three tickets on a lab cluster
- Reserve port5 on both units, give them 10.0.99.11 and 10.0.99.12, and log in to each directly. Confirm each GUI shows its own serial number.
- Start a long download and a continuous ping through the cluster. Run
execute ha failover set 1on the primary. Which survives? Enablesession-pickup-connectionless, unset the flag, and repeat. - Check that
upgrade-modeisuninterruptible, upload a newer 7.6 patch to the primary, and record the time of each step from the HA event log on each unit. - Afterwards, note which unit is primary and explain why from the "Primary selected using" lines.
- Start every task with get system ha status: health, sync and the reason for the current primary.
- Remember what is per unit: host name, priority, override, reserved management interface, licences.
- Match the session pickup options to the traffic: connectionless for UDP, delay and sync ports only for load.
- Upgrade in a window even in uninterruptible mode, and verify both units afterwards.
Test yourself on cluster operations
Session pickup is enabled, connectionless is not. During a failover, which session drops?
FGT1-B's port5 is configured as a reserved management interface, but changes made to it on FGT1-A don't appear on FGT1-B. Why?
upgrade-mode is uninterruptible. The image is uploaded to the primary, FGT1-A. Which unit installs the new firmware first?
During an uninterruptible upgrade, the only secondary fails to boot the new image. What does the primary do?
An administrator changes hb-lost-threshold during business hours. The CLI asks for no reboot. What is the main risk?