Data Crawling

What is Data Crawling?
Data crawling is the automated discovery of pages across a website or the wider web: a crawler starts from a set of seed URLs, follows the links it finds, and builds a map of what exists. In e-commerce that map is usually a retailer's full catalogue — every category, every paginated listing, every product URL — refreshed on a schedule so newly added and delisted items are caught.

Data crawling vs data scraping. The two are often used interchangeably, but they name different halves of the same pipeline. Crawling answers which pages exist; scraping answers what is on them. A price monitoring run crawls a competitor's category tree to discover 40,000 product URLs, then scrapes price, stock and promotion fields from each one. Skip the crawl and you only ever monitor the products you already knew about. The distinction is covered at length in data crawling vs data scraping.

At scale, data crawling is mostly an exercise in restraint and bookkeeping: obeying robots.txt, throttling request rates so the target site is never degraded, deduplicating URLs that differ only by tracking parameter, handling pagination and infinite scroll, and recording which pages changed since the last pass so the next run can prioritise them.