Building a Scalable Web Scraping Architecture in Python & Go

Learn how to build a scalable, hexagonal architecture for web scraping using Python and Go. Master techniques to bypass PerimeterX & HUMAN Security.

What is a scalable web scraping architecture?

A scalable web scraping architecture is a system design that separates *what* you want to collect from *how* the collection happens, so that adding sources, rotating network strategies or swapping runtimes does not require rewriting business logic. In practice that means a ports-and-adapters (hexagonal) layout: a stable core surrounded by replaceable adapters for network, parsing, storage and scheduling.

The alternative — a script per site — works until roughly the tenth source, then becomes a maintenance surface nobody wants to own.

Why hexagonal architecture for scraping?

Scraping systems fail in a specific way: the outside world changes constantly while your business rules stay stable. A site adds Cloudflare, a marketplace changes its DOM, a proxy vendor is swapped, a category needs a headless browser instead of raw HTTP.

Hexagonal architecture puts each of those volatile concerns behind a port:

• Network adapter — how a request is made and fingerprinted
• Parser adapter — how a response becomes structured records
• Storage adapter — where records land
• Scheduler adapter — when and how often a job runs

The core — product models, matching rules, validation — never imports any of them. When a site starts blocking your TLS signature you replace one adapter, not a pipeline.

How do sites detect scrapers at the network layer?

Sites like Amazon and Walmart catch standard HTTP requests using TLS fingerprinting (JA3/JA4). Every HTTP client produces a distinctive handshake — cipher suite ordering, extensions, elliptic curves — and a Python client's fingerprint looks nothing like Chrome's, regardless of what the User-Agent header claims.

In our Network Adapter layer, instead of a standard HTTP client we use adapters that mimic browser behaviour at the TLS level. For Go-based systems we integrate the curl_cffi library with a CGO wrapper to reproduce Chrome's TLS handshake signature. On the Python side we use httpx with custom TLS adapters, or curl_cffi directly, to create connections that do not trip PerimeterX and HUMAN Security sensors on the first request.

Beyond TLS, three signals matter:

• HTTP/2 frame ordering and header casing, which differ between real browsers and most libraries
• Request timing, where perfectly regular intervals are a stronger tell than volume
• IP reputation, where datacentre ranges are scored differently from residential

Python or Go — which layer gets which?

This is the question teams argue about longest and it has a boring answer: use both, split by workload shape.

• Go for the fetch layer. Goroutines make tens of thousands of concurrent connections cheap, memory stays flat under load, and a single binary deploys anywhere. This is where throughput is won.
• Python for parsing, normalisation and matching. The data ecosystem is unmatched, and parsing is IO-light and logic-heavy — exactly where developer velocity beats raw speed.

Connect them with a queue rather than a function call. Go workers publish raw payloads; Python consumers transform them. Each side scales independently, and a parsing bug never takes down collection.

How should the queue and scheduler be designed?

Two rules prevent most production incidents:

• Schedule per source, not globally. Collection frequency should follow how fast each source actually changes. Running everything hourly wastes budget on stable categories and still lags fast-moving ones.
• Make retries idempotent and bounded. A failed fetch should be safe to repeat and must eventually stop. Unbounded retries against a blocking site turn a small failure into an IP ban.

Persist raw responses before parsing. Storage is cheap, re-collection is not — and when a parser bug is found three weeks later, replaying stored payloads beats re-crawling a marketplace that has since changed its layout.

What about observability?

Scraping systems fail partially and quietly. A selector breaks on one category and the pipeline keeps reporting success while that category silently returns nulls.

Track at minimum:

• Fetch success rate per source, not in aggregate
• Parse yield — records extracted versus pages fetched
• Field-level null rates, which catch broken selectors before anyone downstream notices
• Freshness distribution, so you know the age of your worst record, not your average one

Senkrondata runs this architecture in production behind a 97.2% data accuracy SLA across 500M+ rows and 23 countries. The validation layer described above is what makes that number defensible rather than aspirational.

Build or buy?

Building this stack is entirely feasible — the components above are the honest blueprint. What teams underestimate is that the work never finishes: anti-bot vendors ship changes continuously, and a scraper is only correct until the next deployment on the other side.

If your differentiator is what you do with the data rather than how you collect it, our E-commerce Scraper API and Scraper APIs expose this infrastructure directly. If you are deciding where crawling ends and scraping begins, our guide on data crawling vs data scraping covers the distinction.