How QuestDB Handles Out-of-Order Data, Deduplication, and Durability

Updated

QuestDB treats all three as built-in ingestion guarantees rather than application-level workarounds. According to QuestDB's own product description, "durability, deduplication, and out-of-order correction are built in," and the stated goal is to "capture every event once" for high-frequency data such as ticks, orders, executions, and transactions. In practice this means you send events to a table, and QuestDB reconciles late arrivals, repeated sends, and crash recovery inside the database — your downstream SQL sees one consistent record.

The three guarantees, and what each one solves

Concern What QuestDB does What it means for your pipeline
Out-of-order data Out-of-order correction is built in A tick or trade that arrives after newer rows doesn't corrupt the timeline; you don't need to buffer and re-sort upstream
Deduplication Deduplication is built in Re-sent events (retries, at-least-once producers) don't inflate counts or double-count in risk, P&L, or surveillance queries
Durability Durability is built in Events survive process or node failure, so the "complete record" stays complete

The unifying idea is stated directly on the site: QuestDB keeps "the live stream and the complete record, unified," and "keeps years of history queryable" while computing on live data in milliseconds. Out-of-order handling, dedup, and durability are what make that single record trustworthy enough to serve both real-time computation and historical research.

Why this matters for the workloads QuestDB targets

QuestDB positions itself for capital markets, aerospace and robotics, and energy and infrastructure. In those settings the same event stream feeds several consumers at once:

  • Real-time paths — intraday risk and P&L, surveillance conditions, pre-trade signals — must act on the latest state without waiting for stragglers.
  • Historical paths — TCA, markouts, post-trade analysis, compliance, and research — must query the same events later and get the same answer.

If late or duplicate events were handled differently in each path, the two would disagree. Building correction into ingestion is what lets one SQL engine span "messaging, stream processing, and tiered storage" over a single record.

How to reason about it in your own design

  1. Assume producers can be late and can repeat. Design your ingestion so retries are safe to send again — deduplication is expected to absorb them rather than your application tracking sequence numbers.
  2. Don't pre-sort upstream. Out-of-order correction belongs in the database, so you can forward events as they arrive instead of holding them back to guarantee ordering.
  3. Query one table for both live and historical views. Because the live stream and the full record are unified, real-time dashboards and post-trade analysis can read the same data through standard SQL and open formats (Apache Parquet, Iceberg, Arrow).

What the published numbers tell you (and don't)

QuestDB publishes benchmark figures on v10.0: 19M rows/sec/server ingestion, 220M rows/sec streamed to Arrow, 2.3s to stream 500M rows, and 32ms to first Arrow batch. These describe throughput and streaming latency, not the specific cost of out-of-order correction or deduplication — so treat them as capacity signals for high-volume ingestion, and validate correction behavior against your own late/duplicate event patterns.

Practical caveats

  • The page evidence confirms these features exist and are built in, but does not document configuration knobs, dedup key semantics, or durability modes. Check QuestDB's documentation for the exact table-level settings before relying on defaults.
  • Pricing details aren't specified in the available material beyond an "upgrade" signal and a free enterprise trial; don't assume a particular tier includes specific durability or dedup options without confirming.
  • Deduplication and out-of-order correction have a cost. At millions of events per second, confirm that your chosen keys and table design keep ingestion within your latency budget.