How to Make Your First Inference Call or Deploy Your First Model on Baseten

Updated

Baseten supports two distinct first steps: calling a hosted model through Model APIs with an API key and the OpenAI SDK, or deploying your own model on dedicated GPUs using a config.yaml and the Baseten CLI. Choose the first if a supported model already fits your application and you want to start with an API call. Choose the second if you need a specific open-source model, a private checkpoint, or custom serving logic.

Path 1: Make your first inference call with Model APIs

Model APIs expose hosted endpoints for a curated set of models. Baseten manages the model, engine, hardware, and scaling, so you only supply an API key and a request.

What you need

  • A Baseten account with an API key
  • The OpenAI SDK installed in your environment
  • A model from the supported Model APIs list

Steps

  1. Get your API key from your Baseten workspace.
  2. Install the OpenAI SDK if you have not already.
  3. Send a request using the OpenAI client, pointing the base URL at Baseten's Model APIs endpoint and passing your API key.
  4. Read the response the same way you would from any OpenAI-compatible Chat Completions call.

The expected result is a normal chat completion response. Because Model APIs support the OpenAI Chat Completions API (and the Anthropic Messages API in beta), you can reuse existing client code and swap the endpoint and key.

When this path fits

  • A supported model meets your quality and latency needs.
  • You do not want to manage containers, GPUs, or scaling.
  • You want the fastest route from zero to a working call.

Path 2: Deploy your first model on dedicated infrastructure

Dedicated inference lets you deploy an open-source model, a private checkpoint, or a custom model on GPUs you control. Baseten still handles containers, GPU capacity across clouds and regions, scaling, and observability, but you define the model and its serving behavior.

Three ways to define a model

Approach Use it when What you write
config.yaml A supported architecture fits and you want no custom serving code A config file defining model, hardware, and inference engine
Python model class You need custom preprocessing, postprocessing, or dependencies A Python class plus supporting files
Custom Docker server You need full control over server behavior Your own Docker-based server

For a first deployment, the documented starting point is a Hugging Face model defined with config.yaml and deployed with the Baseten CLI. This avoids writing serving code while still giving you a dedicated deployment.

Steps for the config.yaml route

  1. Pick a supported model architecture from Hugging Face.
  2. Write a config.yaml that names the model, the hardware, and the inference engine.
  3. Deploy with the Baseten CLI. The CLI builds the container, provisions GPU capacity, and routes traffic to your deployment.
  4. Verify by sending a request to the deployment endpoint and checking the response.

The expected result is a running dedicated deployment you can call, with Baseten handling autoscaling, cold starts, and release promotion.

When to move to a Python class or Docker server

Switch from config.yaml when you need any of the following:

  • Custom preprocessing or postprocessing of inputs and outputs
  • Extra dependencies not covered by the default engine
  • Custom server behavior that the config file cannot express

How to choose between the two paths

Question Model APIs Dedicated inference
Do you need a specific or private model? No Yes
Do you need custom serving code? No Yes, via Python class or Docker server
Who manages hardware and scaling? Baseten Baseten, but you set autoscaling and hardware
Do you need dev/staging/production environments? Not the focus Yes, stable environments and release lifecycle
Fastest first call? Yes No

Start with Model APIs if a supported model works. Move to dedicated inference when model choice, customization, or environment control becomes a requirement.

Common sticking points

  • Skipping the "Choose how to start" decision. The docs split into "Call a hosted model" and "Deploy your model" for a reason. Picking the wrong entry point wastes time.
  • Reaching for a Docker server too early. If a supported architecture fits, config.yaml gets you deployed without serving code.
  • Assuming Model APIs cover every model. They cover a curated set. Anything outside that set belongs on dedicated inference.
  • Forgetting that pricing and limits are separate pages. Check the Pricing and limits and Billing and usage docs before committing to a path, since cost and quota behavior differ between hosted calls and dedicated deployments.