Monday morning at Brightwater Analytics
At Brightwater Analytics, a fictional company, the help desk opens at 09:00 to a queue of tickets: some web sites won't open, while video calls that started earlier carry on. The FortiGate GUI shows a conserve mode banner. On Friday, policy 3 (LAN to Internet, port2 to port1) was changed to use a proxy-based antivirus profile and deep SSL inspection for all 400 users. Nobody has looked at FGT1's memory since. This lesson works the problem the way an administrator should: compare with normal, rule out the link layer, find the resource under pressure, then change the load, not the symptoms.
Knowing what normal looks like
"High" only means something compared with a normal day. Record these on a quiet day and a busy one, and keep the output with the configuration backup:
| Command | What it records |
|---|---|
get system performance status | CPU states, memory, network use, sessions, setup rate, uptime |
get system session status | Total sessions |
get system status | Firmware and FortiGuard engine versions |
execute tac report | A large snapshot of many diagnostics; run it again during a problem and compare |
Abnormal behaviour should reach you before users do. FortiOS reports CPU use above cpu-use-threshold (90% by default) and logs entering conserve mode as a critical system event, with an SNMP trap. An automation stitch can act on either, which is shown at the end of this lesson.
Learn more: Sizing a FortiGateLogging and Monitoring
Rule out the cable and the port first
Slow or failing traffic is not necessarily a resource problem. A damaged cable or a duplex mismatch produces the same complaints with a healthy CPU. Check the link status in Network › Interfaces (it follows the physical connection and can't be set in FortiOS), then the error counters:
FGT1 # diagnose netlink interface list port2 if=port2 family=00 type=1 index=4 mtu=1500 link=0 master=0 flags=up broadcast run multicast stat: rxp=48210933 txp=51022871 rxb=... txb=... rxe=18452 txe=0 rxd=0 txd=0 mc=2210 collision=0 @ time=... re: rxl=0 rxo=0 rxc=18452 rxf=0 rxfi=0 rxm=0 te: txa=0 txc=0 txfi=0 txh=0 txw=0
rxe (receive errors) and rxc (CRC errors) are rising on port2: frames arrive damaged, which points at the cable, the switch port or a speed and duplex mismatch. rxd counts frames dropped for lack of buffer space and collision only applies in half duplex. diagnose hardware deviceinfo nic port2 shows the same counters with longer names.Run it twice a minute apart: a counter that stays still since last week is history, one that climbs is a live problem. At Brightwater, the counters on port1 and port2 were flat, so the link layer was ruled out.
Learn more: Speed and Duplex
Connectivity problems: the sniffer and debug flow, used safely
When one flow fails and the FortiGate is otherwise healthy, follow the packet: did it arrive, which route and policy did it get, did the reply come back? The sniffer and debug flow answer those questions, and the packet flow lesson explains how to read them. What matters here is that both tools cost CPU, which is the resource you may be short of.
Diagnostics on a loaded FortiGate
- Filter every sniffer and debug flow to one host or port, and give it a packet count.
- Stop the sniffer with Ctrl+C, and end debugging with
diagnose debug disableanddiagnose debug reset. A forgotten trace keeps using CPU. - When the FortiGate logs to FortiAnalyzer or FortiCloud, debug flow output is also sent there as event log messages, so run it no longer than necessary.
- Sessions offloaded to NP6 or NP7 processors are not seen by debug flow and change what the sniffer shows; test with ICMP, which is not offloaded, or disable offload on a narrow test policy.
diagnose sniffer packet port2 'host 10.0.1.10 and port 443' 4 50
diagnose debug flow filter addr 10.0.1.10
diagnose debug flow trace start 20
diagnose debug enableA filtered sniffer (50 packets) and a filtered debug flow (20 packets) for one test PC. Run them one at a time on a busy unit.
Learn more: Understanding FortiGate Firewall Packet Flow
Reading the one-screen health check
FGT1 # get system performance status CPU states: 21% user 34% system 0% nice 41% idle 0% iowait 0% irq 4% softirq CPU0 states: 25% user 36% system 0% nice 35% idle 0% iowait 0% irq 4% softirq ... Memory: 8104668k total, 7212154k used (89%), 600402k free (7.4%), 292108k freeable (3.6%) Average network usage: 512330 / 498012 kbps in 1 minute, ... Average sessions: 61204 sessions in 1 minute, ... Average session setup rate: 22 sessions per second in last 1 minute, ... ... Uptime: 3 days, 1 hours, 14 minutes
Two other first lines are worth recognising. 0% user 0% system ... 100% idle on a FortiGate that passes no traffic means it isn't doing any work: look at links, interfaces and routing, not load. 1% user 98% system with slow traffic means the FortiGate is overloaded, often by scanning.
Finding the process behind the load
FGT1 # diagnose sys top 5 20 Run Time: 3 days, 1 hours and 14 minutes 22U, 0N, 35S, 41I, 0WA, 0HI, 4SI, 0ST; 7914T, 586F scanunitd 2610 S < 18.2 14.1 ipsengine 2745 S < 12.6 9.8 scanunitd 2611 S < 11.9 13.7 miglogd 2603 S 2.0 3.6 httpsd 2573 S 0.4 2.4 cmdbsvr 2528 S 0.0 5.0 ...
m to sort by memory, p by CPU, q to quit. Here the antivirus scanner, scanunitd, leads on both.| Process | Does |
|---|---|
ipsengine | IPS engine, also flow-based inspection |
scanunitd | Antivirus scanning |
iked | IKE negotiation for IPsec tunnels |
httpsd | The GUI over HTTPS |
newcli, sshd | CLI and SSH sessions |
cmdbsvr | The configuration database |
For memory specifically, diagnose sys top-mem lists the processes using the most memory, and diagnose sys top-all shows threads instead of processes. In the GUI, the Process Monitor (click the CPU or Memory widget on Dashboard › Status) shows the same list with a graph.
Killing a process
The Process Monitor can kill a process, the same as diagnose sys kill. Whatever that process was handling (scanning, VPN negotiation, logging) is interrupted, and the sessions it was serving may be dropped. Treat it as a last step, during a quiet period, ideally with Fortinet support involved; Kill & Trace leaves a crash log and backtrace for them.
Where CPU usually goes, and what to change
- Scanning everything. Antivirus, IPS and deep inspection on all traffic. Limit them to the policies and protocols that need them, and avoid scanning the same traffic twice.
- Logging every packet. Packet logging and logging to local memory or disk cost CPU; log to FortiAnalyzer or FortiCloud and drop events nobody reviews.
- Strong VPN encryption without hardware help. Use offload-capable settings where the model supports them.
- Forgotten diagnostics. Sniffers and debug traces left running.
- A nearly full disk. Move logs off the FortiGate.
In practice, when a FortiGate that was fine last month is struggling now, the cause is usually a change: a new profile, a wider policy, more users. The event log shows configuration changes; read it before reading process lists.
Conserve mode: three thresholds
The gap between red (88) and green (82) stops the FortiGate from flapping in and out of conserve mode when memory hovers around one value. What happens to traffic in conserve mode depends on two settings:
| Setting | Value | In conserve mode |
|---|---|---|
av-failopen (config system global), for proxy-based inspection | pass (default) | Traffic bypasses the antivirus proxy, with every profile that needs the proxy |
| off | New sessions that need the proxy are blocked; a policy with IPS only keeps working | |
| one-shot | Like pass, and stays bypassed after conserve mode ends, until the setting changes or the FortiGate restarts | |
fail-open (config ips global), for flow-based inspection | disable (default) | New sessions that need flow inspection are dropped |
| enable | New packets are allowed without scanning |
So with defaults, a FortiGate in conserve mode fails open for proxy inspection and closed for flow inspection. That explains Brightwater's mixed symptoms: traffic through the proxy profile passed uninspected, while new sessions through policies with flow-based IPS were dropped, and established sessions (the video calls) carried on. Decide both settings deliberately and write the choice down: availability or security, per organisation, not by accident.
FGT1 # diagnose hardware sysinfo conserve memory conserve mode: on total RAM: 7914 MB memory used: 7043 MB 89% of total RAM memory freeable: 285 MB 3% of total RAM memory used + freeable threshold extreme: 7518 MB 95% of total RAM memory used threshold red: 6964 MB 88% of total RAM memory used threshold green: 6489 MB 82% of total RAM
execute log filter category 1 and execute log display, and the kernel's entries with diagnose debug crashlog read.Getting FGT1 out of conserve mode
- Collect evidence first.
diagnose sys top 5 20 3,diagnose sys top-mem,diagnose sys session full-stat(session table statistics),get system performance statusanddiagnose hardware sysinfo memory. Save the output. - Find the change. The event log shows Friday's edit to policy 3.
- Reduce the load. At Brightwater, policy 3 went back to the previous flow-based profile, and proxy-based antivirus was applied only to a new policy for the finance subnet, which needed it. Memory can take some minutes to fall as the proxy's existing connections finish.
- Confirm the exit. Below 82% the FortiGate leaves conserve mode on its own;
diagnose hardware sysinfo conserveshowsoff. - Size the fix. If the load is legitimate, the answer is a larger model or less inspection, not a new threshold.
Clearing sessions or rebooting to free memory
diagnose sys session clear without a filter drops every session in the session table, so every user has to reconnect, and a reboot interrupts everything. Both free memory quickly and remove the evidence. Use them only after collecting output, in an agreed window.
Letting the FortiGate collect evidence for you
Conserve mode often starts at a busy hour when nobody is watching. An automation stitch with the Conserve Mode trigger can run the diagnostic commands the moment it happens and email the results (Security Fabric › Automation):
config system automation-trigger
edit "Conserve Mode"
set event-type low-memory
next
end
config system automation-action
edit "high_memory_debug"
set action-type cli-script
set script "diagnose sys top 5 20 3
diagnose sys session full-stat
get system performance status
diagnose hardware sysinfo memory"
set accprofile "super_admin"
next
edit "high_memory_email"
set action-type email
set email-to "noc@brightwater.example"
set email-subject "FGT1 conserve mode"
set message "%%results%%"
next
end
config system automation-stitch
edit "auto_high_memory"
set trigger "Conserve Mode"
config actions
edit 1
set action "high_memory_debug"
set required enable
next
edit 2
set action "high_memory_email"
set required enable
next
end
next
end%%results%% puts the script's output into the email. A High CPU trigger exists for the same purpose and uses cpu-use-threshold.
Learn more: Logging and Monitoring
Resource tickets to work through
"The Internet is slow" and CPU shows 98% system
diagnose sys top lists ipsengine and scanunitd at the top.
Show the next steps
The FortiGate is spending its CPU on inspection. Check which policies apply IPS and antivirus, and to what: a profile on a policy that carries backups or software updates between internal networks is a common culprit. Narrow the policies or the protocols scanned. If inspection is required at that volume, it is a sizing question.
CPU high at night, nobody in the office
Traffic is light, yet CPU stays high. An SSH session from yesterday afternoon is still logged in.
Show the next steps
Check whether a debug or sniffer started during the day is still running in that session. Close it with diagnose debug disable and diagnose debug reset (or end the session), and make stopping diagnostics part of every change checklist.
One site works, another fails, CPU and memory are normal
Only connections to one supplier fail. No conserve mode, no interface errors.
Show the next steps
It is a connectivity problem, not a resource problem. Use debug flow filtered on the supplier's address to see where the first packet stops (route, policy, NAT), as in the packet flow lesson.
Users on the LAN see slow transfers; port2 shows climbing rxc
Show the next steps
CRC errors mean damaged frames. Replace the cable, check the switch port, and make sure both ends agree on speed and duplex (preferably both on auto-negotiation). Then watch the counter again for a few minutes.
The thresholds, defaults and commands in this lesson are from the FortiOS 7.6 documentation. They have not yet been reviewed against 7.4 for this lesson. Output layouts change between builds and models, and new firmware can use more memory on the same hardware, so check memory headroom before an upgrade.
Try it: take a baseline and plan a conserve mode response
On a lab FortiGate, run get system performance status, diagnose sys top 5 20 1 and diagnose hardware sysinfo conserve, and write down CPU, memory, sessions and setup rate. Check av-failopen under config system global and fail-open under config ips global, and write one sentence on what your traffic would do in conserve mode. Then build the conserve mode stitch above with an email action to your own address, and note which commands you would add for your own network.
- Compare with a baseline; "high" is relative to a normal day.
- Rule out links with interface error counters before blaming load.
- Find the process with
diagnose sys top, then the change that caused it. - Conserve mode: red 88, extreme 95, green 82 by default; proxy fails open, IPS fails closed.
- Collect evidence before clearing sessions or rebooting, and stop every trace you start.
Test yourself on resource problems
FGT1 is in conserve mode with default settings. A new session matches a policy with a flow-based IPS profile only. What happens to it?
Memory is at 89% and the FortiGate entered conserve mode. Usage falls to 85% and stays there. Is it still in conserve mode?
get system performance status shows 0% user, 0% system, 100% idle, yet nothing passes through the FortiGate. Where should you look?
diagnose netlink interface list port2 shows rxc rising by hundreds per minute. CPU and memory are normal. Most likely cause?
During conserve mode an engineer wants to free memory now. Which step should come first?
These checks come together with policies, VPNs and HA in Troubleshooting tickets.