How does Deepgram's unified Voice Agent API work?

Updated

Deepgram's Voice Agent API combines speech-to-text, text-to-speech, and LLM orchestration in a single API call instead of requiring you to stitch together separate components. The flow runs from the user's audio through STT, LLM orchestration, and TTS, with your business logic and external systems able to feed context back into the conversation. This design is intended to reduce complexity, latency, and cost for teams building real-time voice agents.

The end-to-end flow

The platform describes the pipeline as a sequence of connected stages:

Stage Role
User Your customer or end user speaking naturally
Audio input A partner-provided transport layer that bridges streamed audio into Deepgram's APIs and plays back TTS audio to the end user, including telephony partners via PSTN/SIP
Speech-to-text (STT) Conversational speech recognition that detects end-of-turn and interruptions and streams transcripts in real time
LLM orchestration Coordinates context, memory, and AI reasoning
Text-to-speech (TTS) Converts AI responses into natural-sounding speech
Business logic Your custom application logic that executes domain-specific actions, integrates external APIs, or adds validation before results flow back into the agent
External systems Databases, APIs, and back-office systems the agent interacts with; results are fed back into the conversation context for grounding

What LLM orchestration handles

The orchestration layer is where conversation state and control live. According to the platform description, it manages:

  • Context: maintains conversation history and system settings.
  • System updates: adjusts role or behavior.
  • Prompt updates: refines instructions in real time.
  • Response injections: adds guidance or context.
  • Function calling: executes actions or fetches data.
  • LLM connection: connects to language models via API for response generation.

Why the unified approach matters

Stitching separate STT, LLM, and TTS services together means managing multiple integrations, passing state between them, and absorbing the latency of each handoff. Deepgram's stated benefit of the single API is reduced complexity, latency, and cost. The STT layer is built for conversation specifically — it detects end-of-turn and interruptions — which is the behavior a voice agent needs to respond naturally rather than waiting for a fixed silence threshold.

Where Flux STT and Flux TTS fit

The platform's current models are Flux STT and Flux TTS, described as the first conversation-aware speech platform built to natively understand turn-taking, handle interruptions, and carry conversational context. They are available in real time or batch, in the cloud or self-hosted.

  • Flux STT listens; Flux TTS speaks.
  • Flux TTS is described as holding tone, expression, and continuity across an entire conversation, with latency as low as 80ms for text-to-speech on the platform.
  • A promotional offer states that spending $1 on Flux TTS returns $1 back, up to $500, and that Flux TTS can be built with free through September 12th, with up to 45 concurrent streaming connections in the US and 5 in the EU and AU.

Choosing your path

The platform frames adoption as choosing the Voice AI journey that fits your business. In practice, the decision comes down to scope:

  • Use the unified Voice Agent API if you want one integration covering STT, orchestration, and TTS, and you want the platform to manage turn-taking and context.
  • Use individual APIs (speech-to-text, text-to-speech, audio intelligence) if you already have orchestration in place or only need one capability.

For teams evaluating the stack, the practical next step is to test the conversational behavior directly — particularly interruption handling and end-of-turn detection — since those determine whether a voice agent feels natural in production.