Operational or Analytical? Why Price Data Lives in Two Databases

How are operational and analytical data separated? Why Senkrondata keeps its OLTP and OLAP layers apart across millions of price records.

On one side, 200+ crawlers write millions of price rows every day. On the other, a customer asks for a report showing "the price trend for these 50,000 products over the last 6 months." These two jobs sound like "the same data," but the load they put on a database is completely different — and mixing them in one system slows both down.

This is a classic distinction in data engineering: operational (OLTP) and analytical (OLAP) workloads want different optimizations.

Two different loads, two different natures

Operational writes look like this: a crawler updates a product's price — a single row, frequent, low-latency. The guarantee needed is that data stays consistent at the moment of the write, and two concurrent updates don't corrupt each other. This is the natural territory of a relational database optimized for row-level updates.

Analytical reads look like this: "which 1,000 products changed price the most last month, broken down by category?" That query scans, groups, and aggregates millions of rows. On a row-oriented database, a query like that can lock up the system for minutes — right on top of the tables crawlers are trying to write to at that same moment.

Putting both loads on the same database creates contention between them: a heavy report query slows down daily data collection, or the other way around.

The fix: write in one place, read in another

The standard way to resolve this contention is to physically separate the two layers:

1. Operational layer: the row-oriented relational database that crawlers and integrations write to, holding "the current correct state."
2. Analytical layer: a separate, column-oriented store optimized for large scans and aggregations, mirrored (copied) from the operational layer at regular intervals.

The mirror is one-directional: data flows from operational to analytical, never the other way. The reporting engine reads almost entirely from the analytical layer — without touching, or burdening, the operational database.

What this buys in practice

• Reporting speed: a monthly report running over millions of rows returns in seconds because the column store is built exactly for this kind of query.
• Uninterrupted data collection: crawlers keep writing even while the heaviest report is running; neither blocks the other.
• Independent scaling: as write load grows (more crawlers added) or read load grows (more reports/analysis), the two layers can be scaled independently.

The principles that keep it trustworthy

• The operational layer is the single source of truth. The analytical layer is a copy; in a conflict, the operational layer wins.
• The mirror has its own freshness guarantee. The analytical layer is correct "as of the last sync," not "just now" — knowing that gap matters for interpreting how current a report is.
• Writes and reads never mix. A report query never touches the operational layer in a way that would slow down data collection.

The bottom line

Storing "the same data" twice, for two different purposes, might look wasteful — it's the opposite. This separation is the design that makes uninterrupted data collection and fast, heavy reporting possible at the same time. The bigger the scale, the more this separation pays off.

If you want both fast collection and fast reporting over millions of records, talk to the Senkrondata team.