Inside the Product Matching Engine: The Architecture That Matches Millions of SKUs Without Barcodes

From blocking and embedding-based similarity to LLM-assisted judgment and human review — inside the architecture of Senkrondata’s matching engine.

The hardest problem in competitor price tracking is invisible on screen: which SKU in your own catalog does that listing on the competitor's site actually correspond to? Barcodes aren't always there, names are written differently on every site, the same product sells in three different pack sizes. This is a retail-specific instance of a classic, hard problem known in the literature as entity resolution.

This piece is the technical follow-up to our introduction to why product matching matters so much. Here we look at the "how" instead of the "what" — the architecture underneath the engine.

Why the scale of the problem kills naive solutions

Hundreds of retail channels, millions of products. A "compare every product to every product" approach across two catalogs is O(n×m) — one million on each side means 10¹² comparisons. Repeating that every day is physically impossible.

That's why the matching engine isn't a single algorithm but a funnel: each stage starts cheap and tolerant, aggressively shrinking the candidate pool; the expensive, precise decision is saved for the very end, the smallest pool. The economics of the funnel are simple: the earlier you do cheap elimination, the less work reaches the expensive layer.

Stage 1 — Candidate generation (blocking)

The way to avoid the O(n×m) explosion is to only ever evaluate "plausible candidates" for each product. This technique is called blocking: each product is assigned one or more blocking keys that roughly describe it — normalized brand + category + initials, say, or characteristic tokens extracted from the name. Only products landing in the same block get compared.

A good blocking key balances two things:

• Recall: true matches must land in the same block (or they never meet).
• Reduction ratio: the block must be small enough (or elimination is pointless).

In practice a single key isn't enough; multiple blocking schemes run in parallel (one on brand, one on a barcode prefix, one on name tokens) and the results are unioned. This provides robustness against noise in any single key.

Stage 2 — Exact identity (deterministic matching)

Within a block, the cheapest and safest signal is checked first: the unique identifier. If a barcode/GTIN is present and consistent on both sides, matching reduces to a lookup — ambiguity is near zero.

But in production this ideal case breaks often: barcodes go missing, are mistyped at the source, or are never standardized in variant-heavy categories like apparel and fashion. A barcode alone can also mislead — the same GTIN is sometimes reused for a different pack or batch. So exact identity is the "great when available" layer, but with a verification reflex on top: even if barcodes match, if name/brand completely contradict each other, the match is flagged suspicious.

Stage 3 — Probabilistic similarity (lexical + semantic)

When exact identity is absent, the work falls to similarity scoring. Two kinds of signal are combined here:

• Lexical similarity: token-based measures (Jaccard, TF-IDF-weighted cosine, Levenshtein/edit distance). These are good at catching surface variation like "Men's Sport Shoe White 42" vs "Sport Shoe - White/42" but miss synonyms ("sneaker" vs "sport shoe").
• Semantic similarity: turning the product name/description into an embedding vector and measuring proximity in vector space (cosine similarity). This captures the same meaning written with different words. At scale, this is accelerated with an approximate nearest-neighbor (ANN) index.

The two signals combine into a weighted score. The critical detail: this stage is deliberately tolerant. Its job isn't to make the final call, it's to narrow the candidate pool to a handful of "probably the same" products. The threshold is kept loose here; the fine-grained decision is left to the next, more expensive layer.

Stage 4 — Contextual decision (LLM-as-a-judge, with hard constraints)

The narrowed pool — now maybe 1-5 candidates per product — is handed to the layer that reasons closest to a human. Here a language model decides whether two listings are actually the same product by evaluating context. The key is not to let the model run free; the decision is framed by a few hard constraints:

• Variant equality is mandatory: pack count (3-pack ≠ single), volume/weight, size, color, gender — these aren't "close," they're different products. If any contradicts, the match is rejected, whatever its score.
• Evidence required: the model must ground its decision in the attributes it compared; "the name just looks similar" isn't enough.
• Reject under uncertainty: if the model isn't sure, "no match" is the safe default (see the precision-recall trade-off below).

The funnel economics pay off exactly here: the expensive LLM decision runs only a few times per candidate — a tiny slice of total workload — because the earlier stages have already eliminated millions of pointless comparisons.

Stage 5 — Human review (human-in-the-loop)

Low-confidence matches automation can't resolve, and especially critical products, drop into an expert queue. This isn't a "fallback" — it's the system's calibration mechanism: expert decisions double as feedback data for improving thresholds and rules over time. Every manual decision makes future automatic ones a little more accurate.

Design principles: the choices that make the engine trustworthy

• Precision > Recall. A false positive (wrong match) silently turns into a wrong pricing decision and is hard to notice; a false negative (missed match) is a visible gap that can be closed later. So when in doubt, don't match. We calibrate the engine toward precision.
• Every match has provenance. Which layer produced a match (deterministic / probabilistic / LLM / manual) and at what confidence score is recorded. "What was this match based on?" always has an answer — essential for both auditability and debugging.
• A match isn't static. Catalogs change, competitor sites restructure their listings, new products appear. Matching is a continuously re-running process; a match can "decay" and need re-verification.

Scale and separating the read layer

Producing daily price-comparison reports over millions of matches isn't feasible by querying the operational database directly (where crawlers write constantly) — reporting load would choke write load. So match and price data are mirrored into a separate analytical store (a column-oriented, dedup-friendly engine) for reporting. Heavy analytical queries then stay fast and fresh without slowing the live data-collection pipeline. (We cover this split in more depth in a separate post.)

The bottom line

Product matching isn't "an algorithm" — it's a funnel architecture: generate candidates with blocking, resolve the easy cases with deterministic identity, narrow with probabilistic similarity, evaluate context with an LLM, close the uncertain cases with a human — and at every stage prefer precision over recall and keep every decision traceable. The trustworthiness of your competitor price data depends on the quality of these layers you never see on screen.

If you want your matching engine built with this kind of rigor, talk to the Senkrondata team.