What Is an Agent Framework and What Should You Look For?

Updated

An agent framework is the layer between a raw LLM and a working agent: it handles tool and function calling, memory, multi-turn orchestration, and execution control so you don't rebuild that plumbing for every project. You need one when your agent has to do more than answer a single prompt — call tools, remember context across turns, and run safely for many users. If you're only prototyping a one-shot prompt, a direct LLM API call is enough; a framework starts paying off once you add tools, state, or multiple end users.

Framework vs. raw LLM API vs. single-purpose agent app

These three get conflated, but they solve different problems:

Layer What it gives you What you still build
Raw LLM API Text generation, function-calling primitives Tool routing, memory, retries, access control, logging
Agent framework Orchestration, tool calling, memory, execution control Your domain logic, data connections, UI
Single-purpose agent app A finished workflow (e.g., chat-with-your-database) Little, but limited to that app's scope

A framework is the middle option: more structure than an API, more flexibility than a packaged app. The trade-off is that you own integration and deployment, which is exactly where provider flexibility matters.

Capabilities worth comparing

Provider flexibility (LLM and tools)

The most common lock-in risk is a framework that only works with one model vendor or one data source. Check whether you can swap the LLM provider and the tool/data backend independently. Vanna, for example, states it works with any LLM provider and database, and lists Anthropic, OpenAI, Gemini, and Ollama alongside SQLite, PostgreSQL, MySQL, Snowflake, and BigQuery, with other databases covered in its docs. If a framework hard-codes one vendor, your migration cost later is the whole integration.

Access control and user-scoped execution

When an agent serves many end users, "who can see what" becomes a correctness and security requirement, not a feature. User-scoped execution means each request runs with the permissions of the requesting user, so one tenant can't read another's data through the agent. This is the core of multi-tenant agents: the framework must enforce scope at execution time, not rely on the prompt to behave.

Observability and memory

Observability (metrics, tracing) tells you what the agent actually did when something breaks. Memory lets the agent learn from prior interactions instead of starting cold each session. Both are easy to skip in a prototype and painful to retrofit, so treat them as evaluation criteria, not add-ons.

Judging production readiness

Production readiness is less about model quality and more about the operational surface around it. Concrete signals to look for:

  • Audit logs — a record of agent activity for compliance and debugging.
  • Quota management — limits so one user or tenant can't exhaust shared resources.
  • Data retention controls — defined storage and lifecycle for conversations and logs.
  • Access control — enforced per user, not per deployment.
  • No vendor lock-in — swappable LLM and database providers.

Vanna describes these as optional hosted admin features (access control, observability, agent memory via a hosted vector DB, secure file storage, audit logs, data retention), which is a useful pattern to expect: the open-source framework handles orchestration, and the operational layer is where hosted services enter. Note that "optional" and "hosted" mean these are separate from the OSS core — check the pricing page for what's included versus paid, since the source doesn't state costs.

A concrete example: the SQL agent

A SQL agent framework lets non-SQL users query company databases in natural language. The framework's job is the unglamorous part: translate the question into a query, run it under the requesting user's access scope, keep multi-turn context so follow-ups work, and log what happened. Vanna's SQL Agent is described as open source, multi-database, multi-turn, and access-controlled, and it offers Cloud, Self Host, and a Colab dummy database path (about 1 minute, no sign-in) versus a local install (pip install vanna, about 30 minutes). That split — fast sandbox to evaluate, self-host to deploy — is a reasonable thing to demand from any framework before you commit.

How to choose

  • Prototyping only? A raw LLM API or a sandbox path (like the Colab dummy database) is enough.
  • Serving multiple users or tenants? Require user-scoped execution and enforced access control.
  • Shipping to production? Require audit logs, quotas, retention controls, and observability.
  • Planning long-term? Require swappable LLM and database providers to avoid lock-in.

Match the framework to the stage you're actually at, and verify the operational features exist before you build on top of them.