Web Crawler

What is a Web Crawler?
A web crawler (also called a spider or bot) is a program that systematically browses websites by following links from page to page to discover content. Search engines like Google run the best-known crawlers, but crawlers also power e-commerce data collection, site auditing, and archiving.

Crawling and scraping are related but distinct steps: the crawler discovers which URLs exist (for example, every product page in a category tree), while the scraper extracts structured data from those pages. In an e-commerce pipeline, a crawler might map a retailer's full catalog nightly, and the scraper then collects price and stock for each discovered product.

Well-behaved crawlers identify themselves with a user-agent string, respect robots.txt directives, and throttle their request rate so they never degrade the target site's performance.