Practice
Learn
← Quit
0 / 0 correct
System Design
Hard
1 of 10
Web crawler: how do you avoid re-crawling URLs you have already seen across billions of URLs?
A
Compare full page content before fetching
B
Normalize URLs and check a distributed seen-set, often a Bloom filter backed by a KV store
C
Keep all URLs in one process's hash set
D
Re-crawl everything; duplicates do not matter