Course contents

Unit 6: Advanced securityLesson 6.10 (6 of 11 in this unit)8 of 25 in the Advanced Cisco SD-WAN course

URL filtering

Category and reputation filtering, allow and block lists, what filtering can see in HTTPS with and without decryption, and block pages.

Advanced · 11 min read

You'll learn to choose categories, reputation and lists for a web policy, predict what the edge can see in HTTP and HTTPS, and work through a wrongly blocked or wrongly allowed site in order.

Technical definition

URL filtering on a WAN Edge is a security function that decides whether a web request is allowed, using the category of the site (for example news or gambling), its reputation score, and the organisation's own allow and block lists.

A policy for BR-01's corporate users

BR-01 (site 101) sends corporate users in VPN 10 straight to the internet over its internet circuit. The security team wants the WAN Edge (cEdge/vEdge) to stop access to sites that carry malware or break company rules, without a separate proxy appliance in every branch. URL filtering on the edge does that. It is one layer of several, and it depends on what the edge can see, which is where most surprises come from.

The three decisions in a web policy

ControlWhat it doesWatch for
Web categoriesMost known sites belong to one or more categories (for example search engines, adult content, file sharing). The category set is maintained by Cisco's threat intelligence group and is large. You choose which categories to block or alert onNew and rare sites may be uncategorised. Decide how to treat them
Web reputationA score for how risky a site appears to be. You set a threshold: sites scoring worse than it are blockedA strict threshold causes false positives on new but legitimate sites
Allow and block listsDomains or URLs you name yourself, overriding the categories for those entriesWhich list wins when a site is on both depends on the release: check, and test it

Each match has an action. The two basic ones are block and alert only. Alert only lets the request through and writes a log entry, which makes it a good first setting for a new category. Block stops the request, and the user normally sees a block page that says the access was denied. The page text can be adjusted, and it is worth including how to ask for an exception.

A web request meets the URL filtering policy on BR-01
A web request meets the URL filtering policy on BR-01A corporate user in VPN 10 requests a web page. BR-01 looks up the category and reputation of the URL and compares them with the policy. A permitted category goes on to the internet. A blocked category or low-reputation site is stopped and the user sees a block page. For HTTPS, the edge can see only the server name unless the traffic is decrypted by the TLS proxy.UserVPN 10BR-0110.255.2.1Category andreputation lookupAllowed siteBlock pagecategory blockedallowedblockedHTTP: full URL visibleHTTPS: server name only, unless decrypted

On small screens, swipe sideways or focus the diagram and use the arrow keys.

For encrypted pages the edge sees the requested name; to judge the full URL it needs the TLS proxy. Check the order: policy match, category, then the allow and block lists.

What the edge can see: HTTP, HTTPS and HTTPS with decryption

Filtering quality depends on the information in front of the edge, so it differs a lot by protocol.

What the edge can see: HTTP, HTTPS and HTTPS with decryption
What the edge can see: HTTP, HTTPS and HTTPS with decryptionTraffic; Visible to the edge; What filtering can do. HTTP (port 80, unencrypted): Full URL: host, path and query Filter whole sites and individual pages HTTPS without decryption: The requested server name (in the TLS handshake) and the server certificate Filter by domain only. Paths and content are invisible HTTPS with the TLS proxy: Everything, as clear text between the two TLS sessions Filter paths too, and let file and threat scanning workTrafficVisible to the edgeWhat filtering can doHTTP (port 80,unencrypted)Full URL: host, path and queryFilter whole sites and individualpagesHTTPS withoutdecryptionThe requested server name (in the TLShandshake) and the server certificateFilter by domain only. Paths andcontent are invisibleHTTPS with the TLSproxyEverything, as clear text between thetwo TLS sessionsFilter paths too, and let file andthreat scanning work

On small screens, swipe sideways or focus the diagram and use the arrow keys.

Read each row across: the left box names the item, the others describe it.

Two consequences follow. First, a block on example.com/games is not possible over undecrypted HTTPS, only a block on the whole domain. Second, content hidden in encrypted sessions is missed by anything else that relies on the clear text. The lesson on the HTTPS/TLS proxy, covers the cost and risks of decrypting, which is why many designs filter by domain on most traffic and decrypt only a selected part.

How does the edge learn the server name if the traffic is encrypted?

In the TLS handshake the client usually sends the name of the site it wants (the Server Name Indication field) before encryption starts. The server's certificate is also sent in a form the edge can read. Newer techniques that encrypt the name (Encrypted Client Hello) reduce this visibility, so the picture changes over time. Check what your release supports.

What the feature depends on

  • The inspection service on the router. URL filtering runs in a security function on the WAN Edge, which needs platform resources and may need a subscription. Whether your model supports it, and how much memory it needs, comes from the platform documentation. No figures are given here.
  • A cloud reputation service. The edge asks a cloud service for the category and score of a site, and keeps recent answers locally. That needs a working path from the router to the internet that the firewall allows.
  • DNS. The client resolves the name first. If DNS is broken, the page fails before filtering matters. DNS-level controls also act first: see DNS web layer security.

When the cloud service is unreachable, new sites cannot be classified. Failing open (allow) keeps users working and lets risky sites through. Failing closed (block) is safer and can stop all browsing during a service or path outage. Neither is correct for all organisations. Choose on purpose, check whether your release lets you set it, and test it.

Applying it by VPN

A URL filtering policy is part of the security policy in SD-WAN Manager and is applied to a service VPN or to a zone pair, depending on how the security policy is built in your release. With the lab's examples (VPN numbers are examples):

Applying it by VPN
Applying it by VPNVPN; Suggested stance. 10 corporate: Block high-risk categories, alert on borderline ones, reputation threshold set after a pilot 20 voice: No web filtering needed; keep it out of scope 30 guest: Stricter category set, since guests are unmanaged and the organisation carries the legal exposure 40 point-of-sale: Allow only the few sites the payment systems need; a small allow list is easier to defend than a large block listVPNSuggested stance10 corporateBlock high-risk categories, alert on borderline ones, reputation thresholdset after a pilot20 voiceNo web filtering needed; keep it out of scope30 guestStricter category set, since guests are unmanaged and the organisationcarries the legal exposure40 point-of-saleAllow only the few sites the payment systems need; a small allow list iseasier to defend than a large block list

On small screens, swipe sideways or focus the diagram and use the arrow keys.

Each item on the left, with what it means on the right.

Designing the policy without drowning in tickets

  • Start from categories that carry real risk or legal exposure, such as known malware hosts, phishing and adult content. Add others when there is a reason.
  • Run new categories in alert-only mode for a period and read the logs before switching to block.
  • Do not block by reputation without a review process. A score is a prediction, and it is wrong for some sites.
  • Write down an exception process: who asks, who approves, how long the exception lasts and how it is removed.
  • Keep the allow and block lists short, each entry with an owner and a reason.

A site is blocked, or is not: the order of checks

  1. Is the policy attached to the right VPN or zone on this router? Check the security policy in SD-WAN Manager, and that the device template was pushed.
  2. Is the inspection service running? If it is down, the feature may fail open or closed depending on the design.
  3. What did the edge see? If the site is HTTPS and not decrypted, only the domain counts, so a path-based rule cannot match.
  4. What category and reputation does the site have? Use the vendor's category lookup tool to check. Uncategorised sites follow the policy for that case.
  5. Is it on an allow or block list? Check both lists for the domain and for parent domains. Then check which list wins in your release.
  6. Can the router reach the reputation service? Test the path and any firewall rules.
  7. Is it really blocked by something else? DNS-layer blocking or a firewall rule can produce a similar symptom. The block page tells you which control acted, so ask the user what they saw.
Example output written for this lesson, simplified; real output has more fields and a different layout
BR-01#show utd engine standard statistics url-filtering
URL Filtering Statistics
  Requests inspected : (example)
  Allowed            : (example)
  Blocked            : (example)
  Alerted            : (example)
  Lookup failures    : (example)
Example only, with placeholders instead of numbers. The command name and the counter names depend on the release, so confirm them in the current documentation. The useful counters to look for are the ones that separate blocked from allowed, and any that count failed lookups: a rising lookup-failure count points at the cloud path rather than the policy.

Test yourself on URL filtering

Predict · scenario 1

A policy blocks example.com/games. Users reach example.com/games over HTTPS and the TLS proxy is not in use. Why is the page not blocked?

Predict · scenario 2

A new category is to be introduced to 300 users. What is the lowest-risk first step?

Predict · scenario 3

After a path outage to the cloud reputation service, browsing at BR-02 stops for all new sites. Which design setting explains this?

Predict · scenario 4

A user says a legitimate supplier site is blocked. The page shows the web filtering block page. What is the best sequence?

FAQ

Can URL filtering block a single page on an HTTPS site?
Only if the edge decrypts the traffic with the TLS proxy. Without decryption the edge sees the server name and certificate, not the path, so it can allow or block the whole site but not one page of it.
Which wins, the allow list or the block list?
It depends on the release and the policy options, so check the current documentation and test it in the lab. Do not assume. Keep the lists small and documented so that a conflict is easy to spot.
What happens if the reputation service cannot be reached?
The edge cannot get a category or score for sites it has not cached. Whether it then allows or blocks is a design decision that may be configurable in your release. Decide it on purpose and test it by cutting the path to the service in the lab.