Web Crawler System Design: URL Frontier, Politeness and Deduplication

A crawler looks simple: download a page, extract its links, download those, repeat. Written as a loop, it works for a thousand pages. At a billion pages it falls over in instructive ways: it hammers small sites with hundreds of parallel requests, downloads the same page under fifty URLs, gets lost in an infinite calendar, and spends most of its time waiting on DNS.

This post designs a crawler that scales and stays polite: capacity numbers, the URL frontier with priority and per-host queues, robots.txt as standardized in RFC 9309, URL and content deduplication, crawler traps, and the components around them, with a diagram of the whole loop.

Requirements and numbers

The goal: crawl about a billion pages a month for a search or analytics index, refresh important pages more often, and never overload a website.

The last row matters most. Discovery outruns fetching by a wide margin, so the crawler is always choosing what not to fetch. Prioritization is a core feature, not an optimization.

The crawl loop

1. Take the next URL from the frontier.
2. Check robots.txt for its host, from cache.
3. Resolve the host name, from a local DNS cache.
4. Fetch the page, respecting the host's politeness delay.
5. Store the raw response.
6. Parse it, extract links and normalize them.
7. Drop links already seen; add the rest to the frontier with a priority.
8. Fingerprint the content and skip near-duplicates in later processing.

Every step after the first is a place where a naive crawler wastes effort or causes harm.

The URL frontier

The frontier is the heart of a crawler: a queue that has to answer two different questions at once. Which URL matters most? And which host may we contact right now?

The classic answer is the Mercator design, described in the URL frontier chapter of Introduction to Information Retrieval:

• Front queues handle priority. A prioritizer assigns each new URL to one of F queues based on signals such as how often the page changes, its link popularity or the importance of its site.
• Back queues handle politeness. Each back queue holds URLs of exactly one host. A heap records, per host, the earliest time it may be contacted again.
• A fetcher thread takes the host at the top of the heap, fetches one URL from that host's back queue, and reinserts the host with a new earliest time. When a back queue empties, it is refilled from the front queues, favoring higher priorities.

Mercator's designers recommended roughly three times as many back queues as fetcher threads, so threads rarely wait for a polite host.

At scale, partition the frontier by host: hash the host name to choose a frontier shard, so all URLs of one host live in one place and politeness can be enforced locally, without coordination.

Politeness and robots.txt

A polite crawler sends one request at a time per host and waits between requests. A common heuristic from the same chapter: wait about ten times as long as the last fetch from that host took, so slow servers automatically get more breathing room. Back off further on 429 and 503 responses, and honor any Retry-After header.

robots.txt became a standard in 2022 with RFC 9309. The rules a crawler has to get right:

The asymmetry between 4xx and 5xx surprises people: a missing file means "go ahead", but a server error means "stay out", because the crawler cannot know what the rules would have said. Crawl-delay is not part of RFC 9309, and crawlers differ in whether they honor it. Identify your crawler with a descriptive user agent that includes a URL with contact details.

Deduplicating URLs

The same page hides behind many URLs: HTTP://Example.com:80/a/./b?utm_source=x#top and http://example.com/a/b are one page. Normalize before checking whether a URL was seen:

• Lowercase the scheme and host, drop default ports, resolve . and .. segments, and normalize percent-encoding, following RFC 3986 section 6.
• Drop the fragment, which never reaches the server.
• Remove known tracking parameters such as utm_*, carefully; other parameters can change the content.
• Respect rel="canonical" hints after fetching.

Then check a seen-URL set. With billions of URLs, a Bloom filter is the usual choice. Sized for one billion URLs at a 1% false-positive rate, it needs about 9.6 bits per URL, roughly 1.2 GB in total, with seven hash functions. A false positive means skipping a URL you have not actually seen, an acceptable cost at 1% for most crawls; keep an exact store for URLs you must not miss, such as sitemap entries.

Deduplicating content

Mirrors, printer-friendly versions and session parameters produce pages that are identical, or nearly so, under different URLs. Exact duplicates are easy: hash the normalized content and compare. Near-duplicates need a fingerprint that changes little when the page changes little. Simhash is the standard technique: a 64-bit fingerprint where similar documents differ in few bits. Google researchers reported in 2007 that, for a repository of eight billion pages, 64-bit simhash fingerprints with a threshold of three differing bits were a practical setting for detecting near-duplicates.

Crawler traps

Some sites generate infinite URL spaces, often by accident: calendars with a "next month" link forever, session IDs in every URL, faceted search with every filter combination, and relative links that keep growing a path. Defenses:

• Cap the URL length and the path depth.
• Cap pages per host per crawl cycle, scaled by the host's importance.
• Detect repeating path segments such as /a/b/a/b/a/b.
• Lower the priority of URLs whose content fingerprints keep matching pages you already have.

Sitemaps help in the other direction: they list the URLs a site wants crawled, up to 50,000 URLs or 50 MB uncompressed per sitemap file, with sitemap index files for more.

Around the loop: DNS, storage and recrawling

• DNS is a hidden bottleneck. At hundreds of fetches per second, a shared caching resolver close to the fetchers avoids sending every lookup upstream.
• Storage. Write raw responses to object storage in large batch files, and send parsed text and links onward to indexing through a stream.
• Recrawling. Pages change at different rates. Track how often each page changed on previous visits and schedule its next fetch accordingly; conditional requests with If-Modified-Since or ETag save bandwidth when nothing changed.
• JavaScript-heavy pages need a headless browser, which is far more expensive per page. Render only the hosts that need it.

The web crawler diagram

How to read it:

• The frame is the URL frontier: priority queues feed per-host queues, which enforce politeness.
• Fetchers take one URL per host at a time, consult the robots.txt cache and the DNS cache, and download from the internet.
• Raw pages go to the object store. The parser extracts links and text.
• New links pass the seen-URL filter before returning to the frontier, closing the loop. Content fingerprints flag near-duplicates.
• Parsed pages flow on to indexing through the stream on the right.
• The Markdown block lists the capacity numbers and politeness rules.

Drawing pipelines with loops

• Make the loop visible. A crawler is a cycle; drawing it as a straight pipeline hides the feedback that causes trap and duplicate problems.
• Show where politeness is enforced. Put per-host queues in the picture, not in a footnote.
• Mark caches as caches. robots.txt and DNS caches change the load on the outside world; label their expiry.
• Put numbers on the edges: pages per second in, links per page out.

For the general method, see the system design diagram example. Rate control toward each host uses the same ideas as the rate limiter system design, and the processing after the crawl looks like the data pipeline architecture diagram.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add a Core Frame named URL frontier and put two Queue nodes from the Data pack inside: priority queues and per-host queues.
3. Add a Compute node from the Cloud pack for the fetchers, a DNS node and an Internet node from the Network pack, and a Redis node from the Data pack for the robots.txt cache.
4. Add an Object Store from the Data pack for raw pages and another Compute node for the parser.
5. Add a second Redis node for the seen-URL filter, a Managed DB from the Cloud pack for content fingerprints, and a Kafka node from the Data pack for pages going to indexing.
6. Draw the loop with labeled edges and add a Markdown block from the Core pack with the numbers.

The example uses 13 diagram nodes, within the Free plan.

FAQ

What is a URL frontier? The data structure that decides what a crawler fetches next. It combines priority queues, which rank URLs by importance, with per-host queues and a schedule that keep the crawler from contacting any host too often.

How do crawlers stay polite? One connection per host at a time, a delay between requests that grows with the server's response time, backing off on 429 and 503 responses, and following robots.txt.

What happens if robots.txt returns an error? Under RFC 9309, a 4xx status means the crawler may crawl the site, while a 5xx status or a network error means it must assume everything is disallowed until it can read the file.

How do crawlers avoid downloading the same page twice? They normalize URLs and check them against a seen-URL set, often a Bloom filter, and they fingerprint content with exact hashes and simhash to catch the same page under different URLs.

How big does a Bloom filter for a billion URLs need to be? About 1.2 GB for a 1% false-positive rate, using roughly 9.6 bits per URL and seven hash functions. Lower error rates cost more bits per entry.

Start free — one seat, no credit card.