A real-life situation
Elmside Logistics, a fictional company, runs FGT1 from the course lab. Its LAN-to-Internet policy (policy 3) uses flow-based inspection with antivirus, web filtering and IPS. Two complaints arrive in the same week. The security team says that when a user downloads a known test virus, the browser shows a broken download instead of the block page they expected. The network team says that after someone switched policy 3 to proxy-based inspection to fix that, large software downloads now pause for several seconds at 99 % and CPU on FGT1 is higher.
Both teams are describing the same design choice from different sides. To answer them, you need to know where inspection sits in the packet path, and what the FortiGate actually does with the data in each mode.
- 1. Policy 3: PC1 (10.0.1.10) downloads a file from 198.51.100.80 over HTTPS. Profiles inspect the data on FGT1.
Where inspection sits in the packet path
The packet-flow lesson described eight stages for the first packet of a session. Inspection is stage 6, after the policy has accepted the session. Two points follow from that position:
- Profiles only see traffic that a policy has already allowed. A deny policy or the implicit deny ends the session before any inspection.
- The first packet (a TCP SYN) carries no data. Inspection is really about the data packets that follow, in both directions, for the whole life of the session.
Learn more: FortiGate Packet Flow
Flow-based inspection
In flow-based mode, the session stays one connection from client to server. The FortiGate copies the packets to the IPS engine, a separate process, which runs every flow-based profile in a single pass: IPS signatures, application control, and the flow versions of antivirus, web filter and file filter. The packets are forwarded while they are being inspected, so the user sees little extra delay.
- Low latency and memory. The engine keeps only the state it needs for each session, not whole files.
- Files are scanned on the fly. Flow-mode antivirus scans a file as it streams through. In recent releases the FortiGate typically holds back the last part of the file until the verdict, so a detected file arrives incomplete and is useless to the client.
- Blocking happens mid-stream. When a signature matches, the engine drops the packet and can reset the session. For plain HTTP it can often send a replacement page, but if the detection comes late in the transfer the user may only see a failed download.
- The connection is not changed. The client and server keep their own TCP session; the FortiGate does not rewrite the content.
Proxy-based inspection
In proxy-based mode, the FortiGate acts as a transparent proxy. The WAD process answers the client's connection itself and opens its own connection to the server. There are now two TCP sessions, joined by the proxy in the middle.
- Whole objects. The proxy reassembles the complete HTTP response, email or file (within the size limits in the protocol options profile), scans it, and only then sends it to the client.
- Content can be replaced. Because the client hasn't received anything yet when the verdict is made, the FortiGate can send a full replacement message (a block page or a notice in place of an attachment) instead of resetting the connection.
- More features. Some features only exist in proxy mode. In FortiOS 7.6 these include ICAP (sending content to an external scanning server), content disarm and reconstruction in antivirus, and web application firewall profiles. Individual cells of Fortinet's inspection mode feature comparison change between builds, so check the table for yours. Proxy-only profiles aren't supported on models with 2 GB of RAM or less.
- Higher cost. Buffering adds latency (the user waits while a large file is downloaded and scanned) and uses more memory per session. IPS and application control still run in the IPS engine, so two engines work on each proxied session.
Proxy mode applies to the protocols the proxy understands, such as HTTP, HTTPS, FTP and the mail protocols. Other traffic on a proxy-based policy is still inspected by the IPS engine.
Flow and proxy compared
| Flow-based | Proxy-based | |
|---|---|---|
| Engine | IPS engine (ipsengine) | WAD proxy, plus the IPS engine for IPS and app control |
| Connection | One end-to-end session | Two sessions, joined by the proxy |
| Data handling | Packets pass while scanned | Objects buffered, scanned whole, then sent |
| Latency | Low | Higher, grows with object size |
| Memory | Lower | Higher per session |
| When blocked | Drop or reset; replacement page where possible | Replacement message in place of the content |
| Extra features | Base set of all main profiles | Proxy-only features (release dependent) |
| Hardware offload | Partly possible on some models | Stays on the CPU |
| Typical use | Most traffic; the default | Where a proxy-only feature is needed |
How the mode is chosen
Per policy, in profile-based NGFW mode
In the default profile-based NGFW mode, each firewall policy has an inspection mode setting. Flow-based is the default for new policies.
Edit Policy: LAN-to-Internet
- Inspection Mode
- Flow-based Proxy-basedChanges which profiles can be selected below.
- AntiVirus
- OnElmside-AV-proxy
- Web Filter
- OnElmside-WF-proxy
- IPS
- Ondefault
- SSL Inspection
- deep-inspectionWithout decryption, neither engine sees HTTPS content.
- Protocol Options
- default
Security Profiles
config firewall policy
edit 3
set inspection-mode proxy
set ssl-ssh-profile "deep-inspection"
set av-profile "Elmside-AV-proxy"
set webfilter-profile "Elmside-WF-proxy"
set ips-sensor "default"
next
endinspection-mode is flow or proxy. The profiles referenced must have a feature set that fits the mode, or the change is rejected.
⚠ Changing the inspection mode affects live users
Switching a busy policy to proxy mode moves its web, mail and file traffic through the proxy, raising memory and CPU use and changing what users see when something is blocked. Existing sessions keep their old handling until they end. Make the change in a maintenance window, on one policy at a time, and watch get system performance status afterwards.
Profiles have a feature set
Some profile types, including antivirus and web filter, have their own feature set setting (flow-based or proxy-based). It hides options the chosen mode can't use and decides which policies can reference the profile. The GUI only offers profiles that fit the policy's inspection mode.
config antivirus profile
edit "Elmside-AV-proxy"
set feature-set proxy
next
endThe same setting exists in web filter profiles (config webfilter profile). Which other profile types carry it, and which combinations a release accepts, are in the administration guide for your build.
Per-policy inspection mode and profile feature sets work the same way in FortiOS 7.4 and 7.6. Which features are proxy-only, and which profile types have a feature set, have changed between releases (features have moved into flow mode over time). Check the release notes and the feature table for your exact build.
NGFW mode: profile-based or policy-based
The NGFW mode is set per VDOM and changes the whole policy model:
| Profile-based (default) | Policy-based | |
|---|---|---|
| Policies | Firewall policies with profiles attached | Security policies, plus a separate SSL inspection & authentication policy |
| Applications | Matched by an application control profile on the policy | Matched directly as a criterion in the security policy |
| Web categories | Matched by a web filter profile | Can be matched directly in the security policy |
| Inspection mode | Flow or proxy, per policy | Flow-based only |
In policy-based mode, the SSL inspection & authentication policy is checked first: it decides which traffic is decrypted and which users must authenticate. The security policies then allow or deny by source, user, destination, application and category.
config system settings
set ngfw-mode policy-based
endSet inside the VDOM you are changing. profile-based is the default value.
⚠ Switching NGFW mode is a redesign, not a toggle
Changing ngfw-mode changes how every policy in the VDOM is written and evaluated. Existing policies don't carry over as you might expect, and other settings (NAT handling among them) can change with the mode; check the administration guide for your release before switching. Plan it as a migration: back up the configuration, rebuild and test the policy set in a lab, and switch in a maintenance window with a rollback plan.
Where SSL inspection fits
Most traffic is HTTPS, and neither engine can inspect content it can't read. The SSL/SSH inspection profile on the policy decides what the engines get:
- Certificate inspection doesn't decrypt. Both modes see only the server name (SNI) and the certificate, so web filtering works by domain and application control by name, but antivirus can't see the files.
- Deep inspection decrypts the session first. Then the IPS engine or the proxy sees the clear-text data, inspects it, and the FortiGate re-encrypts it towards the client. Deep inspection works in both flow and proxy mode.
Decryption is CPU work (accelerated by content processors on many hardware models), and it is needed before any content profile can see the payload. On a busy policy, deep inspection plus proxy mode is the most expensive combination you can pick.
Learn more: SSL/TLS Inspection
Effects on hardware offload
The packet-flow lesson showed that established sessions can be handed to a network processor (NP). Inspection limits that:
- Proxy-based sessions stay on the CPU. The proxy has to terminate the connections and hold the data, which an NP can't do.
- Flow-based sessions can be partly accelerated on some models. Depending on the NP generation and the release, the NP can deliver packets to the IPS engine more efficiently (Fortinet calls this nTurbo), and the session can be offloaded once inspection no longer needs to see it, for example after application control has identified an allowed application. Content processors (CPs) can speed up parts of IPS pattern matching.
- The inspection itself runs on the CPU in the IPS engine or the proxy, so inspected traffic costs far more CPU than plain forwarded traffic.
Which inspected sessions can be offloaded, and how, depends on the NP and CP generation of your model and on the FortiOS release. Treat any rule of thumb as a starting point and check the hardware acceleration guide for your model.
Learn more: Security Processors and Hardware Acceleration
Memory, conserve mode and fail-open
Inspection needs memory: the IPS engine keeps state for every inspected session, and the proxy buffers whole objects. When memory use crosses a high threshold, the FortiGate enters conserve mode to protect itself. What happens to inspected traffic then is a choice between security and availability:
- Fail open: new sessions that need inspection pass uninspected until memory recovers. Users keep working, but traffic is unscanned for that time.
- Fail closed: sessions that need inspection are blocked. Nothing passes unscanned, but users see outages.
FortiOS has settings for this, such as av-failopen under config system global for the antivirus behaviour in conserve mode and fail-open under config ips global for the IPS engine. Defaults and options differ by release, so read the CLI reference for your build before changing them, and record the decision in your security policy rather than leaving it to the default.
diagnose hardware sysinfo conserveShows whether the FortiGate is in conserve mode and the memory thresholds it uses.
Verification
Four questions confirm how a session is being inspected:
| Question | Where to look |
|---|---|
| Which mode does the policy use? | show firewall policy 3 (look for inspection-mode) |
| Is this session inspected, and by what? | diagnose sys session list with filters |
| Which engine is using CPU? | diagnose sys top |
| Is the box under memory or CPU pressure? | get system performance status |
diagnose sys session filter clear
diagnose sys session filter src 10.0.1.10
diagnose sys session filter dport 443
diagnose sys session listFind PC1's HTTPS session and read its state flags and policy ID.
FGT1 # diagnose sys session list session info: proto=6 proto_state=11 duration=12 expire=3597 timeout=3600 ... ... state=log may_dirty ndr redir ... misc=0 policy_id=3 ... npu info: flag=0x00/0x00, offload=0/0, ips_offload=0/0, ... total session 1
state line marks a session sent to the IPS engine with ndr and a session redirected to the proxy with redir; a flow-based session shows the first but not the second. The offload fields are zero, as expected for a proxied session. Flag names vary between releases, so use them as hints alongside the policy's configured mode.FGT1 # diagnose sys top 5 10 Run Time: 12 days, 3 hours and 41 minutes ... ipsengine 411 S < 18.6 4.9 wad 392 S 11.2 3.8 wad 395 S 9.7 3.1 ...
ipsengine is the IPS engine (flow inspection, IPS, application control); wad is the proxy. Busy wad processes mean proxy-based policies are carrying real load. Press q to exit.FGT1 # get system performance status CPU states: 3% user 6% system 0% nice 76% idle 0% iowait 0% irq 15% softirq ... Memory: 8058540k total, 4212336k used (52.3%), ... ... Average sessions: 2140 sessions in 1 minute, ...
What can go wrong
Large downloads stall at the end in proxy mode
Users see a download reach 99 % and pause for several seconds, or a slow link time out, after the policy moved to proxy mode. The proxy is receiving the whole file from the server and scanning it before it releases the rest to the client.
Show the fix
This is how proxy mode works, so first ask whether the policy needs proxy mode at all. If it does, look at the protocol options profile: client comforting sends small parts of the file to the client while it is being buffered so the client doesn't give up, and the oversize limit decides what happens to files too large to buffer (pass unscanned or block). Both trade some protection for usability; check the exact options in your release and record the choice.
The block page only appears in proxy mode
In flow mode, a test virus over HTTPS produces a failed or truncated download; in proxy mode, the same test shows a clear block page. This is the Elmside security team's complaint.
Show the explanation
Nothing is broken. In flow mode most of the file has already reached the client when the verdict is made, so the FortiGate blocks by dropping the rest and resetting the session; a replacement page is only possible in some cases. The proxy holds the whole object, so it can replace it every time. Decide whether a clear message for users is worth the cost of proxy mode, and confirm in the antivirus log that the file was detected in both cases.
High CPU in ipsengine
diagnose sys top shows ipsengine near the top, and users report slow traffic through inspected policies.
Show the fix
- Check which policies have IPS and application control, and whether trusted high-volume traffic (backups, storage replication) is being inspected without need. Give it its own policy with fewer or no profiles.
- Use the most specific IPS sensor that fits, such as protect_http_server for WEB1, rather than every signature.
- Check that signatures and engines are current (
diagnose autoupdate versions); an engine update can change performance. - Check whether the model can offload or accelerate the inspected sessions, and compare the load with the model's data sheet figures for threat protection, not firewall throughput.
Policy-based mode: an application rule seems to let early packets through
In policy-based NGFW mode, a security policy that blocks an application still lets the TCP handshake and the first data packets through, and a packet capture shows them reaching the server.
Show the explanation
An application can only be identified after the IPS engine has seen some of its data, which takes a few packets. Until then, the session is matched on its other criteria (addresses, ports, users) and allowed to continue so identification can happen; once the application is known, the session is checked again and blocked or allowed. The details of this matching differ between releases. For a hard guarantee, also deny by address or port where possible, and don't treat an application rule as a way to stop the first packets.
Enterprise use and design guidance
- Start with flow mode everywhere. It covers IPS, application control, antivirus and web filtering at the lowest cost.
- Use proxy mode where a feature needs it. For example, a staff web policy that needs ICAP or content disarm and reconstruction. Keep guest, server-to-server and bulk traffic on flow mode.
- Size for inspection, not forwarding. Data sheets give separate figures for firewall, IPS, NGFW and threat protection throughput; the inspected figures are much lower. Proxy mode and deep inspection reduce real capacity further.
- Choose the NGFW mode once. Profile-based suits most organisations. Policy-based suits teams that want applications and users as first-class policy criteria and can rebuild their policy set for it.
Best practices
- Name profiles by feature set (for example
Elmside-AV-proxy) so the mode is obvious in every policy. - Decrypt only where you inspect content, and exempt sensitive categories such as banking and health.
- Change inspection mode on one policy at a time and compare
get system performance statusbefore and after. - Decide and document fail-open behaviour for conserve mode instead of relying on defaults.
- Test with a new connection after every change; existing sessions keep the handling they started with.
🛠 Practical exercise: see the two engines at work
On a FortiGate VM in a lab, give the LAN-to-Internet policy flow-based antivirus with deep inspection, and download the EICAR test file over HTTPS from PC1. Note what the browser shows and find the session in diagnose sys session list. Then create proxy-based antivirus and web filter profiles, switch the policy to proxy mode, and repeat. Compare the browser result, the session's state flags, and the ipsengine and wad lines in diagnose sys top during a large download. Write down two differences you can see, and which mode you would choose for this policy and why. Switch the policy back to flow mode when finished.
Key takeaways
- Inspection is stage 6 of the packet path: it only sees data in sessions a policy has accepted.
- Flow mode: the IPS engine scans packets as they pass, with low latency; blocks by drop or reset.
- Proxy mode: WAD terminates the connection and scans whole objects; replacement messages and extra features, more CPU and memory, no offload.
- Profile-based mode sets flow or proxy per policy; profile feature sets must match. Policy-based mode is flow only.
- Without deep inspection, neither engine sees HTTPS content.
Check yourself
Policy 3 on FGT1 uses flow-based antivirus with deep inspection. A user downloads a test virus over HTTPS. What do they most likely see?
You set inspection-mode proxy on a policy, but the CLI rejects the AV profile you reference. Most likely cause?
diagnose sys top shows several busy wad processes on FGT1. What does that tell you?
An engineer wants the staff web policy fully offloaded to the NP for speed, and also wants proxy-based antivirus on it. What is the problem?
A VDOM is switched to ngfw-mode policy-based. Which statement is true?
Next, see how each profile behaves in Antivirus and IPS, Web filtering and Application control.