What Is Production ML and What Does It Take to Run Models in Production?

Updated

Production ML is the full lifecycle of training, deploying, and monitoring models on live data — not just building a model that scores well offline. A model is "in production" when it serves real users or systems, receives real inputs, and is maintained as data and behavior drift. Getting there requires infrastructure for feature management, deployment, and monitoring, plus attention to latency, cost, and governance as the number of models grows.

Why a Good Model Breaks After Launch

The gap between a notebook and a running system is where most ML projects fail. Two failure modes dominate:

  • Training-serving skew — the features computed during training differ from those computed at inference time. Different code paths, different data sources, or different timing all produce skew, and the model silently degrades.
  • Feature freshness — a model that needs a user's latest activity to make a real-time decision cannot rely on a batch job that ran last night. Stale features mean wrong predictions.

Both problems are structural, not modeling problems. They are solved by making the same feature definitions serve both training and inference, and by retrieving those features fast enough for the use case.

The Core Infrastructure Pieces

A production ML stack typically includes:

Component Role
Feature store Central repository for feature data, with fast retrieval and reuse across models
Model registry Versioned record of trained models and their metadata
Deployment pipelines Move a model from experiment to a serving endpoint
Monitoring Track prediction quality, drift, and system health after launch

Hopsworks frames the feature store as "the backbone of production ML," offering a central repository for feature data with sub-millisecond retrieval powered by RonDB. The point of centralizing features is reuse: Hopsworks cites Meta's observation that their top 100 features are used in over 100 models. Reuse is also a cost lever — Hopsworks claims up to 80% cost reduction by reusing features and streamlining development.

Latency and Scale: Real-Time vs. Batch

Real-time inference changes the requirements. If a decision must happen inside a user request, feature retrieval has to be fast. Hopsworks advertises sub-millisecond latency for end-to-end data retrieval and cites SIGMOD 2024 benchmarks showing 10x lower latency than SageMaker and Vertex.

Scale introduces a second set of constraints:

  • Compute management — GPUs for LLMs and other models need scheduling and quota management to keep utilization high. Hopsworks supports training at scale with Ray and serving with KServe/vLLM.
  • Data formats — working directly with Delta, Iceberg, and Hudi tables avoids migrations and conversions. Hopsworks claims 9–45x faster reads via a Python-native query engine.
  • Framework flexibility — Spark, Flink, Pandas, and DuckDB can coexist, so a team can start small and scale to petabytes.

Operational Concerns as Models Multiply

Once several models are live, the questions shift from "does it work?" to "can we run this sustainably?"

  • Data reuse — shared features across models reduce duplicated pipelines and inconsistent definitions.
  • Cost — reused features and managed compute directly affect the bill.
  • Governance and control — some organizations need air-gapped, on-premises, or hybrid deployment to keep full control over data and AI operations. Hopsworks lists sovereign AI as a supported mode.

Customer examples illustrate the range. Zalando, described as Europe's largest online fashion platform, built mission-critical ML infrastructure on Hopsworks for real-time personalization across 25 countries and 50 million customers. Clicklease moved from a messy microservice architecture with training-production skew to a streamlined platform for real-time fraud detection and credit decisioning.

What to Check Before Committing

If you are evaluating whether a platform supports production ML, verify these against your own workload:

  1. Does the feature store serve both training and online inference from one definition?
  2. What is the measured retrieval latency, and does it fit your real-time path?
  3. Can you deploy and monitor models without leaving the platform?
  4. Does it read your existing table formats without migration?
  5. Does it support your deployment model — cloud, on-premises, air-gapped, or hybrid?

Hopsworks publishes pricing at hopsworks.ai/pricing and offers a free start and a demo booking, so cost and onboarding terms can be checked directly rather than assumed.