What the Feature Store for ML Benchmarks Measure

Updated

Featurestore.org publishes three benchmark tracks for feature stores: an Offline API benchmark, an Online API benchmark, and a Feature Freshness benchmark. Each measures a different part of the feature-serving path — throughput and latency for batch/historical retrieval, latency and throughput for real-time lookups, and the delay between an event occurring and its feature becoming available for serving. The benchmarks are useful for comparing vendors on the dimensions that matter most in production, but they are not a single "score" — you have to read them against your own workload.

Offline API Benchmark

The offline API is what training pipelines and batch scoring jobs call to pull large volumes of historical feature values, usually joined across entities and time ranges.

What this benchmark is measuring:

  • Throughput — how many rows or how much data the store can return per unit of time when a training job requests a large slice.
  • Latency under load — how response time behaves as request size and concurrency grow, which matters when many training jobs share the store.
  • Point-in-time correctness overhead — the cost of retrieving features as they existed at a given timestamp, the operation that prevents label leakage in training sets.

This is the benchmark to read if your main use case is building training datasets or running nightly batch inference. A store that looks fast on the online benchmark can still be slow here, because offline retrieval stresses different infrastructure (typically a warehouse or data lake rather than a low-latency key-value store).

Online API Benchmark

The online API serves feature vectors to a model at inference time, usually by entity key (user ID, item ID, session ID).

What this benchmark is measuring:

  • Read latency — the per-request time to fetch a feature vector, which directly bounds how much of your serving latency budget the feature store consumes.
  • Throughput — requests per second the online store sustains, relevant when inference traffic spikes.
  • Behavior at scale — how latency holds up as the number of entities and features grows, not just at small scale.

Read this benchmark if you serve real-time predictions and the feature lookup sits on the critical path. The relevant question is not the single best latency number but the latency at the request rate and feature-vector width you actually expect.

Feature Freshness Benchmark

Freshness measures the gap between when source data is produced and when the corresponding feature is queryable in the online store.

What this benchmark is measuring:

  • End-to-end ingestion delay — time from event or batch arrival to availability for serving.
  • Update cadence — how frequently features are refreshed, and whether the store supports streaming writes versus only scheduled batch loads.
  • Consistency between offline and online — whether the value served online matches what the offline path would return for the same point in time.

This is the benchmark that separates stores built for streaming ingestion from those that only support batch. If your model depends on recent behavior — fraud signals, session context, recency-weighted counts — freshness is often the deciding factor, and a store with excellent online latency but hour-level freshness may still be unsuitable.

How to Use the Benchmarks to Assess a Store

The three benchmarks answer different questions, so compare vendors on the one that matches your workload rather than averaging them.

Your primary need Benchmark to weight most What to check
Training dataset generation, batch scoring Offline API Throughput on large slices; point-in-time retrieval cost
Real-time inference Online API Latency at your expected request rate and vector width
Recency-sensitive models Feature Freshness Ingestion delay; streaming vs. batch support
Mixed workloads All three Whether one path's strength comes at another's expense

Practical guidance:

  1. Match the benchmark to your access pattern. A recommendation model doing real-time lookups should be judged on the online benchmark; a churn model retrained nightly should be judged on the offline one.
  2. Look at the shape of the curve, not the peak. Latency and throughput at small scale say little; the behavior as load and feature count grow is what you will live with.
  3. Treat freshness as a design constraint. If the store cannot ingest streaming data, no amount of online latency tuning will make recency-dependent features viable.
  4. Confirm the benchmark conditions apply to you. Benchmark results depend on data volume, feature-vector width, and infrastructure; a vendor's published numbers are a starting point for your own load test, not a substitute for it.

The site also maintains a vendor comparison in its Feature Store Advanced Guide, covering options such as Hopsworks, Vertex AI Feature Store, and SageMaker Feature Store, which is the natural companion to the benchmarks when shortlisting candidates.