Practice
Learn
← Quit
0 / 0 correct
System Design
Hard
1 of 566
Web crawler: what is a 'spider trap' and a defence?
A
Infinite generated URLs (calendars, session IDs); limit depth per host, cap URL length and pages per domain
B
A robots.txt file; ignore it
C
A slow DNS server; add caching
D
A firewall blocking crawlers; use more IPs