What Is Voice AI and How Do You Build a Voice Agent That Handles Real Calls?

Updated

Voice AI is software that holds a spoken conversation over a phone line: it listens to the caller, decides what to do, acts on that decision (looking up a record, booking a slot, sending a text), and replies in speech. You can build one two ways on a platform like Plivo — a no-code prompt-based agent builder for teams, or code using SIP trunking, audio streaming, and frameworks such as LiveKit or Pipecat. The choice depends on how much control and integration work you need. This guide covers the building blocks, both build paths, a real call flow, and the production checks that matter.

How Voice AI differs from a chatbot or basic IVR

A text chatbot reads and writes text. A basic IVR plays fixed menus and waits for keypad presses. Voice AI combines four things in a live loop:

  • Speech recognition turns the caller's audio into text.
  • An LLM interprets intent and decides the next step.
  • Tool calls let the agent act on real systems — check a calendar, look up a booking, send an SMS.
  • Text-to-speech returns a natural spoken reply.

The difference that matters in practice is the tool-call layer. A menu system can route a call; a voice agent can complete the task the caller asked for.

The core building blocks of a working voice agent

Every production voice agent needs the same four capabilities, plus a telephony layer to connect to the phone network:

Layer Job What to verify
Speech recognition Convert caller audio to text Handles accents, background noise, interruptions
LLM reasoning Understand intent, choose next action Reliable tool selection, not just chat
Tool calls Read/write to your systems Auth, error handling, confirmation steps
Text-to-speech Speak the reply naturally Natural pacing, barge-in support
Telephony Real phone numbers, in and out Coverage, latency, self-serve provisioning

Plivo's page describes this stack as carrier-grade telephony underneath, with the agent layer on top. Its stated figures are latency under 500 ms, 99.99% uptime, and coverage in 150+ countries — useful benchmarks to compare against when you evaluate any provider.

Two ways to build: prompt-based vs. code

No-code, prompt-based builder

Best when your team wants to launch an agent without engineering time. You describe the agent's role, the tasks it should handle, and the tools it can call, then deploy. Plivo positions this path for teams that want to move fast; the trade-off is less control over low-level call behavior.

Code and API path

Best when you need custom logic, your own orchestration, or integration with existing systems. Plivo lists several routes:

  • SIP trunking — give a LiveKit agent a real phone number over carrier-grade inbound and outbound trunks, self-serve.
  • Audio streaming — stream live call audio into a Pipecat pipeline, with bidirectional audio, background voice cancellation, and noise cancellation.
  • Programmable AI Agents — bring LiveKit or Pipecat, or build directly on the voice APIs, with SDKs for Python, Node, and Go, plus API and CLI access.

The page's framing is "full control, zero glue code" — meaning the telephony transport is handled for you so you don't build number provisioning and call routing yourself.

What a real call looks like end to end

Plivo publishes a recorded call from a dental clinic agent named Aria. The transcript shows the full loop in action:

  1. Answer and greet — "Thanks for calling Bright Smile Dental Clinic. This is Aria."
  2. Capture the request — the caller asks to reschedule a Friday cleaning.
  3. Look up the record — the agent asks for a booking ID or name, then confirms the existing appointment (cleaning with Dr. Patel, Friday at 11).
  4. Check availability and book — the agent offers Tuesday at 3 PM and books it.
  5. Confirm — it sends a confirmation text to the caller's phone.

The page notes the agent called four tools during the exchange. That is the pattern to design for: each step is a discrete tool call with a clear input and a verifiable result, not one monolithic prompt.

Production checks before you deploy

  • Latency — sub-500 ms round-trip is the target Plivo advertises; test with your own LLM and tools, since those add time.
  • Uptime — 99.99% is the stated figure; confirm what your contract actually guarantees.
  • Coverage — 150+ countries per the page; check the specific countries you need for number availability and regulations.
  • Pricing model — Plivo lists AI Agents at $0.03/min excluding telephony, and audio streaming + SIP from $0.0028/min. Telephony is billed separately, so model total cost per call, not just the agent rate.
  • Getting started — the page offers $10 in free credits to start; check the pricing page for current terms rather than assuming a free tier.

Choosing between the two paths

Pick the prompt-based builder if your agent's job is a well-defined conversation (appointments, order status, FAQs) and you want it live quickly. Pick the code path if you need custom tool orchestration, your own model, or integration with systems your team already runs. Both sit on the same telephony layer, so the decision is about control versus speed — not about call quality.