Web Crawler
What is a Web Crawler?
A web crawler (also called a spider or bot) is a program that systematically browses websites by following links from page to page to discover content. Search engines like Google run the best-known crawlers, but crawlers also power e-commerce data collection, site auditing, and archiving.
Crawling and scraping are related but distinct steps: the crawler discovers which URLs exist (for example, every product page in a category tree), while the scraper extracts structured data from those pages. In an e-commerce pipeline, a crawler might map a retailer's full catalog nightly, and the scraper then collects price and stock for each discovered product.
Well-behaved crawlers identify themselves with a user-agent string, respect robots.txt directives, and throttle their request rate so they never degrade the target site's performance.
Related Terms
- Data CrawlingThe automated discovery of URLs across a site or the wider web, ahead of extracting data from them.
- Web ScrapingThe automated collection of publicly available data from websites at scale.
- Data ParsingThe process of extracting and transforming raw HTML or JSON data into a structured format.
- Rate LimitingRestricting how many requests a client may make to a server in a given time window.