Practice
Learn
← Quit
0 / 0 correct
System Design
Hard
1 of 40
Web crawler: how do you avoid re-crawling URLs you have already seen across billions of URLs?
A
Keep all URLs in one process's hash set
B
Compare full page content before fetching
C
Normalize URLs and check a distributed seen-set, often a Bloom filter backed by a KV store
D
Re-crawl everything; duplicates do not matter