Design a Web Crawler

1Clarifying the requirements#

Pin down what "crawler" means before drawing a single box. The word covers everything from a 200-line script to the thing that feeds a search engine, and those are wildly different builds. Here is how I would open:

Candidate: What is the crawler feeding? A search index, a price tracker, an AI training corpus? Each one changes what I optimize for.

Interviewer: Assume it feeds a general search index. We want broad coverage of HTML pages.

Candidate: Do we need to render JavaScript, or is raw HTML enough for v1?

Interviewer: Raw HTML is fine. Flag JS rendering as a later add-on.

Candidate: What scale and freshness? One pass over the whole web, or constant re-crawling?

Interviewer: Target around 1 billion pages a month, and popular pages should get re-crawled more often than the long tail.

Candidate: Last one: do we respect robots.txt and avoid hammering any single site?

Interviewer: Yes. Politeness is a hard requirement. Getting us banned is a failure. That is where I want you to spend time.

What those answers just told me:

• The interview is about the frontier and politeness, not about parsing HTML.
• "Popular pages re-crawled more often" means freshness is adaptive, not a fixed cadence.
• "Getting banned is a failure" is the interviewer handing me the thesis: design the frontier around politeness, not as an afterthought.

Functional requirements

• Start from seed URLs and fetch pages over HTTP and HTTPS.
• Parse each page, extract outbound links, and feed new URLs back in.
• Store the raw page content for a downstream indexer.
• Respect robots.txt and per-host rate limits.
• Avoid fetching the same URL twice, and avoid storing duplicate content.
• Prioritize so important and fresh pages get crawled sooner.

Non-functional requirements

• Scale to about 1 billion pages per month, plus re-crawls.
• Politeness. Never more than a small, configurable request rate per host.
• Fault tolerance. A crashed worker must not lose queued work.
• Extensibility. Easy to add new content types (images, PDFs) or a JS renderer later.
• Freshness. Re-crawl high-value pages more often than static ones.

2Back-of-the-envelope estimation#

Turn "1 billion pages a month" into numbers you can design against. One number drives everything, and I will flag it when I reach it.

Fetch rate. A month is roughly 2.6 million seconds (30 days times 86,400).

1,000,000,000 pages / 2,600,000 sec ≈ 385 pages/sec average

• Average is a lie. Traffic is never flat, so plan for a peak of about 3x.
• That is roughly 1,200 pages/sec at the busy hours.
• This is the real write rate into the pipeline. Every component below has to survive 1,200 fetches/sec, not 385.

Storage.

• Average HTML page is about 100 KB raw. HTML gzips well, so call it 25 KB compressed.
• 1B pages/month x 25 KB = 25 TB/month of compressed content.
• Over a year that is 300 TB, before re-crawls.
• Keep two versions of popular pages and budget closer to 400 to 500 TB/year.
• This goes to object storage (S3-style), never a relational store. Blobs this size in a DB are a mistake.

Metadata (the number that bites).

• Each URL is a small row: url, hash, last-crawled, status, priority. Call it 500 bytes.
• 1B crawled URLs x 500 bytes = 500 GB.
• But the crawl graph is far bigger than pages crawled: you discover many more links than you fetch.
• At ~10 unique URLs seen per page, the known-URL set balloons to about 10 billion entries.
• Budget ~5 TB for URL metadata and dedup structures.

Bandwidth.

• 1,200 pages/sec x 100 KB raw (you download uncompressed from the origin; compression happens after).
• 1,200 x 100 KB = 120 MB/sec, about 1 Gbps inbound at peak.
• That is a meaningful pipe. It says: many workers across many machines, not one fat box.
• A single NIC and a single process will not carry 1 Gbps of fetches plus parsing.

The shape of the problem. Write-heavy, bandwidth-bound, with a huge dedup set and a hard politeness constraint. Keep those four in mind; everything downstream serves them.

3Core APIs#

A crawler is mostly internal, but it still needs a control surface. An operator seeds it, pauses it, and checks status. There is no public "crawl this" endpoint for end users; this surface is for operators only.

POST /v1/seeds
Content-Type: application/json

{
  "urls": ["https://example.com", "https://news.example.org"],
  "priority": 8,
  "max_depth": 5
}

GET /v1/status

{
  "pages_fetched_total": 48213900112,
  "fetch_rate_per_sec": 1174,
  "frontier_size": 940000000,
  "hosts_in_backoff": 312,
  "errors_last_hour": 20418
}

POST /v1/hosts/example.com/throttle
Content-Type: application/json

{ "max_rps": 0.5 }

Two choices worth defending:

• The throttle endpoint matters more than it looks. When a site owner emails that we are too aggressive, an operator dials one host down in seconds, no deploy. That is an incident tool, not a nice-to-have.
• Components talk over queues, not REST. The REST layer is just for humans and for kicking off work. The real traffic is frontier-to-worker.

4High-level design#

The spine is standard, so draw it fast. Workers pull URLs from a central URL frontier, download the page, parse it, extract links, and push new links back after a dedup check. Raw content lands in object storage for the indexer.

Why each box earns its place.

• URL Frontier. The heart of the thing, and not one plain queue. It decides what to crawl next and enforces politeness so we never flood a host. Interviewers push hardest here.
• Fetcher Workers. Stateless. Pop a URL, resolve DNS, check robots, download, hand off the bytes. Stateless means you run hundreds and scale by adding more, and a crash loses nothing because state lives in the frontier.
• DNS Cache. DNS lookups cost tens of milliseconds and you hit the same hosts repeatedly. A local cache turns that into a memory read. Without it, DNS alone becomes the bottleneck at 1,200 fetches/sec.
• robots Cache. Fetch and cache each host's robots.txt so you check rules without re-downloading it every request.
• Parser and Link Extractor. Pulls anchors, normalizes them (resolve relative paths, strip fragments, lowercase the host), emits clean absolute URLs.
• URL Dedup Filter. Answers "have we seen this URL?" fast. A Bloom filter in front of a persistent set.
• Content Dedup. Catches different URLs that return identical bodies (mirrors, session IDs in the URL). Hash the body, skip if seen.
• Content Store. Object storage holding compressed HTML, keyed by a content hash.

5Data model#

Use a few stores, each picked for its access pattern. The URL metadata (small rows, huge count, fast lookup) and the content (big blobs) are different beasts, so I do not force them into one place. I also break out hot politeness state, because it is read and written on every single fetch.

url_metadata (key-value or wide-column store like Cassandra, keyed by url_hash):

• Keying by url_hash gives a flat, uniform key for a store that holds 10B rows, and lets the frontier shard cleanly by host.

content_blob (object storage, not a DB table; shown as a schema for clarity):

host_politeness (small, hot, lives in Redis):

The one key choice I want to defend: content is keyed by hash, not by URL.

• Ten URLs that return the same bytes all point at one blob.
• Mirror sites and session-ID spam cost one write instead of ten.
• Content dedup falls out of the schema for free.

6Deep dives#

6.1 The URL frontier: priority and politeness together#

This is where candidates shine or fall apart, so I spend real time here. The frontier has to do two jobs that actively fight each other.

• Priority. Crawl important and fresh pages first.
• Politeness. Never send more than N requests per second to any single host.

Why the obvious designs both fail.

• One giant priority queue: a big news site has thousands of high-priority URLs, so workers slam it all at once and get us banned.
• Shard purely by host for politeness: you lose global priority ordering and waste budget on junk.

The trick is two layers of queues.

Front queues are prioritized.

• A URL's priority (PageRank-ish signals, freshness, depth) picks which front queue it enters.
• Higher-priority queues get drained more often, so a hot news article jumps ahead of a dead forum thread.

Back queues are keyed by host.

• Each back queue holds URLs for exactly one host.
• A worker is assigned a back queue, and a timer gates it: after a fetch, the host's next_slot_at is pushed out by 1 / max_rps.
• The worker will not touch that host again until the timer clears.
• At the default 0.5 rps, that is one fetch every 2 seconds for that host, no matter how many juicy URLs it has waiting.

So priority decides what gets pulled forward, and per-host back queues enforce how fast we hit each site. This is the single most important idea in the whole design. I say it out loud so the interviewer knows I know it.

6.2 Dedup at two levels#

We dedup twice, for two different reasons, and conflating them is a common mistake interviewers watch for.

URL dedup stops us queueing the same link a hundred times.

• Every page links to the homepage, the privacy policy, the login page.
• Before a URL enters the frontier, check a Bloom filter sized for ~10B URLs.
• A Bloom filter gives a probabilistic "definitely new" or "maybe seen" in a few bits per item.
• 10B URLs fit in roughly 20 GB of RAM instead of terabytes. That is the payoff for the 10B number from the estimation.

The catch: false positives.

• The filter can say "seen" for a genuinely new URL, and we skip it.
• At a ~1% false-positive rate, we lose about 1% of new URLs.
• For a general crawler chasing coverage, that is an acceptable trade.
• If it were not, back the filter with a disk-based set and consult disk only on a "maybe seen" hit, paying latency to buy correctness.

Content dedup catches the harder case: two different URLs returning the same bytes.

• Think ?sessionid=abc versus ?sessionid=xyz, or a site mirrored on three domains.
• Hash the fetched body (a plain content hash for exact dupes, or a 64-bit SimHash) and check before writing.
• If the hash exists, skip the store write and just record that this URL maps to the existing blob.

Near-duplicate detection is a v2 refinement.

• For pages that are 98% the same (a product page with a different sidebar ad), SimHash compares by Hamming distance and collapses near-dupes.
• Mention it as a refinement, not v1, because exact-hash dedup already kills the worst offenders.

6.3 Trap avoidance and freshness#

The web is hostile. Two problems to call out, because interviewers love them and they are easy to forget under pressure.

Crawler traps. Some sites generate infinite URLs.

• A calendar with a "next month" link forever, or faceted search with every filter combo.
• Follow links naively and one trap eats the whole crawl budget.
• Defenses: cap crawl depth per host, cap total URLs per host, and watch for URL patterns that explode (long query strings, repeating path segments).
• A per-host budget means no single domain monopolizes the frontier. A calendar that would spawn a million URLs gets cut off at its host cap.

Freshness and re-crawl. Different pages change at wildly different rates.

• A news homepage changes every few minutes; a 2015 blog post never changes.
• Crawling both at the same cadence wastes bandwidth.
• Set next_crawl_at adaptively: start every page at, say, a 7-day re-crawl.
• Shorten it for pages whose content hash keeps changing; lengthen it for pages that stay identical.
• Pages unchanged across the last five visits drift out to a 30-day cadence.
• A simple exponential adjustment, and it directly protects the 1 Gbps pipe and the 400 TB/year budget.

6.4 Crash safety and the fetch lifecycle#

One thing I will not skip: what happens when a worker dies mid-fetch. A URL marked fetching by a now-dead worker must not be lost, and it must not be stuck forever.

• Lease, do not just dequeue. A worker leases a URL from the back queue with a visibility timeout, say 60 seconds.
• Finish cleanly: the worker acks and the URL flips to done.
• Crash: the lease expires, the URL becomes eligible again, and fetch_attempts is incremented.
• Dead URL: after a few failed attempts, mark it error and stop retrying so a dead host does not churn forever.
• This is the same at-least-once pattern as SQS or a Kafka consumer group.
• It is exactly why fetchers stay fully stateless: the durable state is the lease and the metadata row, never anything in worker memory.

7Bottlenecks and trade-offs#

Here is where I think it breaks under load, and how I defend the calls I made.

The frontier is the single point of contention.

• Every worker reads and writes it. One Redis instance tips over near the 1,200 fetches/sec peak.
• Fix: shard the frontier by host hash across many instances, so host A always lands on shard 3.
• Politeness is naturally per-host, so sharding by host keeps each host's politeness state on one shard with no cross-talk.
• That is a clean shard key, and I lead with it if pushed on scaling.

DNS is a sneaky bottleneck.

• At 1,200 fetches/sec with cold DNS, resolution alone can dominate latency.
• The DNS cache handles the common case, but misses on a long tail of hosts still hurt.
• Run a local caching resolver on each worker box and pre-warm it for known hosts.

Bloom filter false positives cost coverage.

• I accepted a 1% miss rate for the memory savings.
• Coverage at web scale is already approximate, and 1% of 10B is noise against the pages that matter (they have many inbound links and get rediscovered through another path).
• If the interviewer wants zero misses, trade RAM for a disk-backed exact set and eat the latency on "maybe" hits.

Hot hosts versus politeness.

• A domain with millions of valuable pages is capped at, say, 2 requests/sec. Fully crawling it takes forever.
• No free lunch: negotiate a higher rate with the site owner (big sites often have crawler agreements), or accept slower coverage.
• I would not break politeness to fix it. Getting banned loses the whole site, not just the slow part.

Object storage write amplification.

• Re-crawling popular pages means writing many versions.
• Cap retained versions (say, the latest 2) and let the rest expire, unless the downstream needs history.
• This keeps the 400 TB/year number from drifting to a petabyte.

The trade-off I flag explicitly: raw HTML over JS rendering.

• Raw-HTML fetching keeps each fetch cheap: one HTTP GET, no headless browser.
• The cost: we miss content that only appears after JavaScript runs, a growing share of the web.
• v2 adds a separate, smaller pool of rendering workers for URLs flagged JS-heavy.
• A headless browser costs maybe 20x the CPU and memory of a plain fetch, so I would not make every worker a browser.

Let politeness and the read:discover ratio pick the architecture, not the urge to crawl fast.

8Summary#

The decisions that matter:

• Two-layer frontier. Front queues for priority, per-host back queues with timers for politeness. The single most important call, driven by the fact that the two goals fight each other.
• Stateless fetcher workers. Scale horizontally, lose nothing on crash, because state lives in the frontier and stores, protected by lease timeouts.
• Dedup twice. Bloom filter on URLs (cheap, 1% miss accepted), content hash on bodies (free near-dup and mirror handling).
• Content keyed by hash in object storage, metadata in a wide-column store, hot politeness state in Redis.
• Shard the frontier by host hash so politeness state and work both partition cleanly.
• Adaptive re-crawl via a next_crawl_at that stretches for static pages and tightens for changing ones.
• Politeness and robots as hard constraints, with trap avoidance via per-host depth and URL budgets.
• Out of scope by choice: JS rendering, images and PDFs, the indexer itself, near-dup SimHash. Naming them keeps me honest about what 45 minutes covered.

For how specific teams frame this at the whiteboard, see our company-specific system design guides.