Stateless, stateful and next-generation firewalls

A real-life situation

Northgate Print Works, a fictional company, protected its office for years with access lists on its Internet router. To let staff browse the web, the router had to permit inbound TCP from any source port 80 or 443 to any destination port above 1023, because that is where the replies to web requests arrive. An external scan found that this rule let anyone reach several internal services that happened to listen on high ports, simply by sending from port 443. When the company installed a FortiGate, the inbound rule disappeared: the FortiGate allows replies only for connections that staff actually opened. This lesson explains why, and what the "next-generation" features add on top.

The FortiGate examples use the course lab:

Why the type of firewall matters

Every firewall answers the same question for each packet: allow or deny? What changes between types is how much information it has when it answers. A packet filter knows one packet. A stateful firewall knows the connection. An NGFW knows the connection, the application, the user and, when it can decrypt, the content. More knowledge means tighter, simpler rules, but also more processing and more things that can go wrong.

Stateless packet filters

A stateless packet filter, such as an access control list (ACL) on a router, compares each packet with an ordered list of rules. A rule matches on header fields only: source and destination IP address, protocol (TCP, UDP, ICMP), source and destination port, and sometimes TCP flags. The first match decides. The filter keeps no memory: the hundredth packet of a download is judged exactly like the first.

The return-traffic problem

When PC1 opens a web page, it sends from an ephemeral port, a temporary source port chosen by its operating system, to port 443 on the server. The reply comes back from port 443 to that ephemeral port. A stateless filter on the inbound side must allow it with a rule like this:

The second rule is the weakness. The filter can't tell a real reply from a new packet an attacker crafts with source port 443. Ephemeral ranges also differ by system: the IANA range is 49152–65535, many Linux systems use 32768–60999, and older systems started at 1024. A filter has to allow the widest range its clients might use.

• The "established" trick: some ACLs can match TCP packets with the ACK or RST flag set, which a connection-opening SYN doesn't have. That blocks new inbound connections, but a crafted packet with the ACK flag still passes, so it is a filter on flags, not a record of connections.
• UDP and ICMP: there are no flags to check. Allowing DNS replies means allowing anything from source port 53.
• Multi-connection protocols: active FTP and SIP open extra connections on ports negotiated inside the data, which a header-only filter can't read.

Packet filters are fast and cheap, which is why they remain useful as a first coarse filter. They are not enough on their own at a network edge.

Stateful inspection and the session table

A stateful firewall evaluates its rules only for the first packet of a connection. If a rule allows it, the firewall records a session: protocol, source and destination addresses and ports, the interfaces, the rule that allowed it, any NAT, the connection state and an idle timer. Every later packet, in either direction, is checked against the session table. A reply is allowed because it matches an existing session, not because a rule permits inbound traffic. An unsolicited packet from port 443 matches no session and no inbound rule, so it is dropped.

TCP: tracking the connection state

Because it tracks the handshake, a stateful firewall drops a TCP packet that doesn't fit, such as an ACK for a connection whose SYN it didn't see. That defeats the crafted-packet trick that passes a stateless "established" rule.

UDP and ICMP: pseudo-sessions

UDP and ICMP have no connections, but a stateful firewall still creates a session for them. For UDP the session is keyed on addresses and ports; for ICMP echo it uses the addresses and the ICMP identifier. A DNS reply from the resolver is allowed because it matches the session created by the query. Since nothing marks the end of a UDP exchange, these sessions end when their idle timer expires, which is why UDP timers are usually much shorter than TCP ones.

Timeouts matter

A session lives as long as traffic refreshes its timer. Timers that are too short break quiet but legitimate connections, such as a database client that sends nothing for two hours and then finds its session gone. Timers that are too long fill the session table with dead entries. Session capacity is a real sizing limit for any stateful firewall.

Application-layer gateways and proxies

Some protocols negotiate extra connections inside their data. Active FTP tells the server which port to connect back to; SIP describes the media ports for a call in its message body. A firewall that reads only headers can't allow those connections without opening wide port ranges. Two answers exist:

• An application-layer gateway (ALG) reads the control connection, learns the negotiated ports, and opens a narrow, temporary "pinhole" for just that data connection, translating addresses inside the payload if NAT is in use. FortiOS calls these session helpers.
• A proxy goes further: it terminates the client's connection, opens its own connection to the server, and passes data between them, understanding the protocol fully. A web proxy can read every HTTP request, cache, and enforce rules on content. The cost is more processing and protocol support that must keep up with the applications.

Next-generation firewalls

By the late 2000s most applications travelled over HTTP and HTTPS, so "allow TCP 443" allowed almost everything. A next-generation firewall keeps stateful inspection and adds:

Application identification needs a few packets of the connection before it can recognise the application, so an NGFW often allows the start of a session and decides once the application is known.

Learn more: Application Control · Antivirus and IPS

What each type can and can't see

The limits of encrypted traffic

Most traffic is encrypted, which limits what any firewall can see without decrypting:

• Certificate inspection reads the TLS handshake: the server name (SNI) and, with TLS 1.2, the certificate. That is enough for web categories and many application signatures, but not for URLs, files or exploits inside the session.
• TLS 1.3 encrypts the server certificate, so the firewall relies mainly on the SNI unless it decrypts.
• Encrypted Client Hello (ECH), where browsers and servers support it, hides the real server name too. Support is still developing; check how your FortiOS release handles it.
• QUIC (HTTP/3 over UDP 443) is a different transport from TLS over TCP. Many administrators block QUIC with application control so browsers fall back to TCP, where inspection is mature.
• Deep inspection decrypts and re-encrypts with a CA certificate the clients must trust. It reveals content but costs processing, raises privacy questions (banking, health), and breaks applications that pin their certificates, which need exemptions.

Learn more: SSL/TLS Inspection

How a FortiGate implements each idea

Stateful sessions

Every allowed connection through a FortiGate creates a session, and replies are matched against it. You can read the table. Filter it first so you see only the connection you care about:

FGT1 # diagnose sys session list
session info: proto=6 proto_state=01 duration=73 expire=3586 timeout=3600 flags=00000000 ...
...
hook=post dir=org act=snat 10.0.1.10:51544->198.51.100.80:443(203.0.113.2:51544)
hook=pre dir=reply act=dnat 198.51.100.80:443->203.0.113.2:51544(10.0.1.10:51544)
...
misc=0 policy_id=3 ...
total session 1

For a count of sessions rather than the sessions themselves:

FGT1 # diagnose sys session stat
misc info:       session_count=812 setup_rate=37 exp_count=0 clash=0 ...
...

Session timers

The default idle timer for a session is set under config system session-ttl (3600 seconds by default), with optional entries there for specific protocols and ports. A firewall policy can override it with its own session-ttl, and so can a custom service. Handshake, closing and UDP idle timers are separate settings under config system global; their names and defaults are in the CLI reference for your release.

To keep a quiet application's sessions alive, raise the timer only on the policy that carries it, not globally:

Session helpers (ALGs)

FortiOS ships with session helpers for protocols such as FTP, TFTP and SIP, each bound to a protocol and port. show system session-helper lists them. When FTP control traffic on TCP 21 passes, the FTP helper reads the negotiated data port and allows that one data connection, with NAT applied inside the payload. SIP is often handled by a dedicated VoIP profile instead of the basic helper; which one applies depends on settings covered in VoIP material.

Next-generation features

On a FortiGate, NGFW features are security profiles attached to policies: application control, IPS, antivirus, web and DNS filtering, and an SSL/SSH inspection profile that decides how much of TLS they see. In the default profile-based mode, the policy matches on addresses, services and users, and the profiles inspect what the policy allows. In the policy-based NGFW mode, security policies can match applications directly.

Learn more: Understanding FortiGate Firewall Packet Flow

What can go wrong

An application disconnects after an hour of inactivity

Users of a reporting tool lose their database connection after lunch. The client sends nothing while idle, and the next query hangs or fails.

Active FTP or SIP calls partly work

The FTP login works but directory listings hang; or calls connect but there is no audio in one direction.

Application control doesn't identify an application

Logs show the traffic as generic HTTPS or as an unknown application, and the block rule doesn't match.

Replies are dropped as "no session matched"

Outbound traffic leaves through the FortiGate, but replies return through another path or interface.

Enterprise use and design guidance

• Layer the types. Stateless ACLs on edge routers can drop bogons and spoofed sources cheaply; the stateful NGFW behind them does the real policy work.
• Inspect according to risk. Full profiles with deep inspection on user Internet traffic; IPS on policies towards servers; plain stateful policies where inspection adds little, such as backup replication between trusted sites.
• Size for sessions, not only throughput. Concurrent sessions and new sessions per second limit a stateful firewall long before raw bandwidth in many networks.
• Keep traffic symmetric. Stateful inspection, NAT and HA designs all assume both directions pass through the same firewall.

Best practices

• Write policies for the initiating direction only; let sessions handle replies.
• Adjust session timers on narrow policies or services, not globally.
• Use application control with an appropriate SSL inspection profile, and know what certificate inspection can't see.
• Prefer protocols that don't need ALGs (SFTP, passive FTP) where you can.
• Watch session count and setup rate as health indicators.

Key takeaways

Check yourself

Next: where firewalls sit in a network, in Firewall placement, segmentation and zero trust. For the full path of a packet through a FortiGate, see Understanding FortiGate firewall packet flow.