vanna.ai

A production-ready agent is one that can safely serve many real users, not just answer a demo prompt. The difference is rarely the model. It is the operational layer around it: identity, access control, observability, quota, memory, and deployment choice. Vanna 2.0 frames this as "user-scoped execution, rich UI components, and quota management built-in," and its SQL Agent page lists the admin features that separate a prototype from something you can ship. If your agent only needs to answer questions for you on your laptop, you do not need most of this. If other people will use it against real systems of record, you do.

Demo agent vs. production-ready agent

A demo agent usually runs as a single trusted user with broad credentials, no per-user limits, and no record of what happened. That works until a second person logs in.

The pattern is consistent: production readiness is about constraining and recording what the agent does, not about making it smarter.

Why user-scoped execution is the gating requirement

The single most important property is that every request runs as a specific user, with that user's permissions. This is what "user-scoped execution" and "multi-tenant agents" mean in practice.

Consider a SQL agent that lets employees query company systems of record. If the agent connects with one admin database account, then a sales rep can read finance tables simply by asking nicely. The agent has no way to know the request should have been denied. User-scoped execution fixes this by binding the query to the requesting user's identity, so the database enforces the same rules it would for that person typing SQL directly.

This is why identity comes before capability. An agent that can do more than its user is allowed to do is a security incident waiting to happen, and no amount of prompt engineering substitutes for it.

The capabilities you must add before launch

Vanna's page lists these as optional hosted admin features. Treat them as the checklist for going live:

• Access control — control who sees what. Without it, multi-tenant isolation is impossible.
• Observability — real-time metrics and distributed tracing. You need to answer "why did this answer come out wrong?" after the fact.
• Agent memory — a hosted vector database for learning. Persistent memory is what lets the agent improve across sessions instead of starting cold every time.
• File system — secure file storage and retrieval. Agents that read or produce files need somewhere safe to do it.
• Audit logs — comprehensive activity tracking and compliance. Required when someone asks who ran what and when.
• Data retention — long-term storage for conversations and logs. Needed for debugging, review, and compliance windows.

Two of these deserve emphasis. Observability is not optional at scale: without tracing, a wrong answer is unexplainable, and an unexplainable agent cannot be trusted with real decisions. Agent memory and file storage affect reliability and user experience directly — a user who has to re-explain context every session will stop using the agent.

Choosing a stack without locking yourself in

Vanna's "Choose Your Stack" section is built around avoiding vendor lock-in: the framework works with any LLM provider and any database, and you select your options to generate configuration code.

• LLM providers shown: Anthropic, OpenAI, Gemini, Ollama
• Databases shown: SQLite, PostgreSQL, MySQL, Snowflake, BigQuery, and other databases via docs

The practical test for lock-in is whether you can swap one component without rewriting the agent. If your prompts, memory, and access rules are tied to a single vendor's proprietary format, you are locked in regardless of what the marketing says.

Open source, cloud, or self-host

The page presents three deployment paths:

1. Open source — pip install vanna, multi-database, multi-turn, access controlled. The page notes a 30-minute path and a 1-minute "Dummy Database in Colab" with no sign-in.
2. Cloud — the hosted option.
3. Self-host — run it yourself.

The admin features above (access control, observability, memory, file system, audit logs, retention) are described as optional hosted features, with a link to pricing. That distinction matters: if you go open source, you are responsible for providing equivalents of those capabilities yourself. If you want them managed, the hosted path is the one to evaluate — check the pricing page for current terms, since cost and plan details are not specified here.

A reasonable decision rule: start open source to validate that the agent answers useful questions against your data, then decide whether you want to operate the identity, observability, and retention layer yourself or pay for it hosted.

A concrete readiness test

Before calling an agent production-ready, walk through this with a second user account:

1. Log in as a user with limited permissions and ask for data they should not see. Expected result: denied, by the same rules the database enforces.
2. Ask a question that requires multi-turn context. Expected result: the agent remembers the prior turn.
3. Restart the service and repeat. Expected result: memory and files persist.
4. Trigger a wrong answer and try to trace it. Expected result: you can see the steps, not just the output.
5. Check the audit log for the above activity. Expected result: each action is recorded with the responsible user.

If any step fails, that is the gap between your prototype and production.

Multi-tenant agents are AI agents that serve many users or organizations from one deployment while keeping each tenant's data, memory, and permissions isolated. You need this design as soon as more than one person uses the same agent and those people shouldn't see each other's queries, files, or conversation history. The core requirement is user-scoped execution: every query, memory lookup, and file access runs as the requesting user, not as a shared service account. Vanna 2.0 is one framework that builds this in — its SQL Agent is described as "Open Source • Multi-database • Multi-turn • Access controlled," with optional hosted admin features for access control, observability, agent memory, file system, audit logs, and data retention.

Single-user vs. multi-tenant agents

A single-user agent can hold one set of credentials, one memory store, and one permission level. A multi-tenant agent must resolve all three per request.

The failure mode to avoid is "confused deputy" behavior: the agent runs with broad service credentials and a user tricks it into reading another tenant's data. Isolation has to be enforced at execution time, not just in the UI.

Scope execution per user

The central design rule: the agent should never hold credentials broader than the current user's. Each request carries an identity, and that identity determines what the agent can query, remember, and read.

Concretely, for a SQL agent this means:

1. Input — the user's natural-language question plus their authenticated identity (from your app's session or token).
2. Action — the agent generates SQL and executes it through a connection scoped to that user's permissions, not a shared admin connection.
3. Expected result — the user sees only rows their identity is allowed to see; the same question from a different tenant returns that tenant's data or nothing.

If you can't scope the database connection per user, enforce row-level security in the database itself and pass the tenant ID into every query. Either way, the isolation lives below the LLM, so a prompt injection can't cross tenants.

The same applies to memory and files. Agent memory (a vector store of past interactions) and any file storage must be partitioned by tenant. Vanna lists "Agent Memory: Hosted vector DB" and "File System: Secure file storage" as admin features, which is the layer where this partitioning normally happens.

Access control, audit logs, and retention

Isolation is necessary but not sufficient for production. You also need to answer "who did what, and can we prove it later?"

• Access control — decide who sees what. Map each user to a tenant and a role, and check that mapping before any data access. Vanna lists "Access Control: Control who sees what" as a hosted admin feature.
• Audit logs — record every query, its author, and its result scope. Vanna lists "Audit Logs: Comprehensive activity tracking & compliance."
• Observability — per-tenant metrics and distributed tracing so you can see which tenant is slow, erroring, or hitting limits. Vanna lists "Observability: Real-time metrics & distributed tracing."
• Data retention — conversations and logs accumulate fast across many tenants. Vanna lists "Data Retention: Long-term storage for conversations & logs," which is where you set per-tenant retention windows.

These are the features that turn a working demo into something you can operate for many customers. Note that Vanna presents them as optional hosted admin features, so the open-source framework alone doesn't give you the managed versions — plan for that distinction when choosing self-host vs. cloud.

Quotas and cost control across tenants

With one tenant, a runaway query is an annoyance. With many, one tenant can consume the shared LLM and database budget. Vanna's site lists quota management among its production features, alongside user-scoped execution and rich UI components. Design for:

• Per-tenant request or token limits.
• Per-tenant storage and retention caps.
• Alerts when a tenant approaches its limit.

Without per-tenant quotas, noisy-neighbor problems are inevitable.

Choosing a stack

Vanna works with any LLM provider and database, which matters for multi-tenant deployments because different tenants may need different backends. The site lists LLM providers (Anthropic, OpenAI, Gemini, Ollama) and databases (SQLite, PostgreSQL, MySQL, Snowflake, BigQuery, and others via docs).

Two practical decisions:

• Self-host vs. cloud. The framework is open source; the admin features (access control, observability, memory, file system, audit logs, retention) are optional hosted features. If you need those managed, check pricing; if you need full control, self-host and build the equivalent.
• Vendor lock-in. Vanna advertises "no vendor lock-in," and the provider/database matrix supports that — you can swap LLMs or databases per deployment. Verify this against your own requirements rather than assuming it.

For a first build, the fastest path is the SQL Agent over a single database with per-user connection scoping, then add memory partitioning, audit logs, and quotas as you onboard more tenants. The Colab dummy-database option is listed as a 1-minute, no-sign-in start; the pip install path is listed at about 30 minutes.

Common failure points

• Shared service credentials. The agent runs as an admin account and relies on the LLM to "not" leak data. Fix: scope credentials per user or enforce row-level security.
• Unpartitioned memory. All tenants share one vector store, so one tenant's history surfaces in another's answers. Fix: partition memory by tenant ID.
• No audit trail. You can't reconstruct who saw what. Fix: log every query with author and scope.
• Missing quotas. One tenant exhausts shared budget. Fix: per-tenant limits and alerts.
• Assuming OSS includes admin features. The hosted admin features are optional and separate. Fix: confirm what's in the open-source framework vs. the hosted offering before committing.

An agent framework is the layer between a raw LLM and a working agent: it handles tool and function calling, memory, multi-turn orchestration, and execution control so you don't rebuild that plumbing for every project. You need one when your agent has to do more than answer a single prompt — call tools, remember context across turns, and run safely for many users. If you're only prototyping a one-shot prompt, a direct LLM API call is enough; a framework starts paying off once you add tools, state, or multiple end users.

Framework vs. raw LLM API vs. single-purpose agent app

These three get conflated, but they solve different problems:

A framework is the middle option: more structure than an API, more flexibility than a packaged app. The trade-off is that you own integration and deployment, which is exactly where provider flexibility matters.

Capabilities worth comparing

Provider flexibility (LLM and tools)

The most common lock-in risk is a framework that only works with one model vendor or one data source. Check whether you can swap the LLM provider and the tool/data backend independently. Vanna, for example, states it works with any LLM provider and database, and lists Anthropic, OpenAI, Gemini, and Ollama alongside SQLite, PostgreSQL, MySQL, Snowflake, and BigQuery, with other databases covered in its docs. If a framework hard-codes one vendor, your migration cost later is the whole integration.

Access control and user-scoped execution

When an agent serves many end users, "who can see what" becomes a correctness and security requirement, not a feature. User-scoped execution means each request runs with the permissions of the requesting user, so one tenant can't read another's data through the agent. This is the core of multi-tenant agents: the framework must enforce scope at execution time, not rely on the prompt to behave.

Observability and memory

Observability (metrics, tracing) tells you what the agent actually did when something breaks. Memory lets the agent learn from prior interactions instead of starting cold each session. Both are easy to skip in a prototype and painful to retrofit, so treat them as evaluation criteria, not add-ons.

Judging production readiness

Production readiness is less about model quality and more about the operational surface around it. Concrete signals to look for:

• Audit logs — a record of agent activity for compliance and debugging.
• Quota management — limits so one user or tenant can't exhaust shared resources.
• Data retention controls — defined storage and lifecycle for conversations and logs.
• Access control — enforced per user, not per deployment.
• No vendor lock-in — swappable LLM and database providers.

Vanna describes these as optional hosted admin features (access control, observability, agent memory via a hosted vector DB, secure file storage, audit logs, data retention), which is a useful pattern to expect: the open-source framework handles orchestration, and the operational layer is where hosted services enter. Note that "optional" and "hosted" mean these are separate from the OSS core — check the pricing page for what's included versus paid, since the source doesn't state costs.

A concrete example: the SQL agent

A SQL agent framework lets non-SQL users query company databases in natural language. The framework's job is the unglamorous part: translate the question into a query, run it under the requesting user's access scope, keep multi-turn context so follow-ups work, and log what happened. Vanna's SQL Agent is described as open source, multi-database, multi-turn, and access-controlled, and it offers Cloud, Self Host, and a Colab dummy database path (about 1 minute, no sign-in) versus a local install (pip install vanna, about 30 minutes). That split — fast sandbox to evaluate, self-host to deploy — is a reasonable thing to demand from any framework before you commit.

How to choose

• Prototyping only? A raw LLM API or a sandbox path (like the Colab dummy database) is enough.
• Serving multiple users or tenants? Require user-scoped execution and enforced access control.
• Shipping to production? Require audit logs, quotas, retention controls, and observability.
• Planning long-term? Require swappable LLM and database providers to avoid lock-in.

Match the framework to the stage you're actually at, and verify the operational features exist before you build on top of them.

An AI agent is a system that uses a language model to decide what to do next — calling tools, fetching data, and chaining steps — rather than just answering a single prompt. To act on the real world, an agent needs external tools, because its training data is frozen and it can't browse, scrape, or write to your apps on its own. The practical way to give it those capabilities is to connect it to ready-to-run tools through APIs or marketplace integrations. Apify, for example, describes itself as "a marketplace of ready-to-run tools for AI" with "73,229 tools for your AI," which is the kind of catalog you'd plug an agent into.

Agent vs. chatbot vs. single prompt

The distinguishing feature is autonomy over steps. A chatbot waits for you to drive; an agent plans and executes, then reports back.

Why agents need external tools

A model's knowledge stops at its training cutoff and contains no live data about your niche, your competitors, or your own systems. Tools close that gap:

• Fresh data — current prices, posts, reviews, listings
• Actions — writing to a database, sending a message, triggering a workflow
• Structure — turning messy web pages into clean fields an agent can reason over

Without tools, an agent can only talk. With them, it can do.

How agents connect to tools

Three common patterns, from simplest to most integrated:

1. Direct API calls — the agent (or your code around it) hits an endpoint and gets JSON back. You handle auth and parsing.
2. Marketplace integrations — you pick a ready-made tool from a catalog and connect it to your agent. Apify's page lists this as "Easily connect with your AI agents," alongside "Ready-to-run or build your own."
3. MCP / framework adapters — the tool exposes itself in a format your agent framework understands. Apify's Website Content Crawler, for instance, "integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem."

The right choice depends on how much glue code you want to own. Marketplaces and adapters trade flexibility for speed.

Concrete example: crawling a site to feed an agent or RAG pipeline

Say you want an agent that answers questions about a documentation site.

1. Input: the site's URL(s).
2. Action: run a crawler. Apify's Website Content Crawler will "crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines." It "supports rich formatting using Markdown, cleans the HTML, downloads files."
3. Expected result: clean Markdown chunks you embed into a vector store.
4. Then: your agent retrieves relevant chunks at query time and answers with citations.

The crawler does the messy part (HTML cleanup, formatting); the agent does the reasoning. This split is the whole point of connecting tools.

Criteria for choosing agent tools

Judge each candidate on the same dimensions:

• Data source — does it cover the site/platform you actually need? (TikTok, Google Maps, Instagram, e-commerce, Facebook are all separate tools in Apify's catalog.)
• Output format — JSON for structured logic, Markdown for LLM/RAG input.
• Scheduling & monitoring — can it run on a schedule, or only on demand?
• Integration — native support for your framework (LangChain, LlamaIndex) vs. raw API.
• Cost — check the provider's pricing page; don't assume free.
• Reliability signals — usage counts and ratings. Apify shows these per tool (e.g., Google Maps Scraper: 616K runs, 4.7 from 1,817 reviews; TikTok Scraper: 291K runs, 4.8 from 371).

Common failure points

• Auth — API keys and tokens expire or lack scope; the agent fails silently.
• Rate limits — high-volume agent loops hit caps fast; add backoff.
• Stale data — a cached result looks valid but isn't; timestamp everything.
• Unstructured output — raw HTML breaks parsing; prefer tools that clean and format.
• Silent errors — an agent may treat a failed call as an empty result. Validate responses explicitly.

Bottom line

An AI agent is a goal-driven system that plans and calls tools; a chatbot just responds. To make an agent useful, connect it to tools that supply live data and actions — via direct APIs, a marketplace like Apify, or framework adapters. Pick tools by data source, output format, scheduling, integration, and cost, and guard against auth, rate-limit, and staleness failures before you ship.

A user-scoped agent runs every request under the identity, permissions, and data boundaries of the person who made it, instead of a shared service account. This matters whenever more than one user touches the same agent: if the agent authenticates as itself rather than as the caller, it can read and act on data the caller was never allowed to see. Vanna 2.0 frames this as "user-scoped execution" and pairs it with access control, agent memory, and audit logs, which are the three places user scoping actually gets enforced.

The core problem: shared-credential agents

Most early agent demos connect to a database or API with one set of credentials baked into the environment. That works for a single-user prototype and breaks the moment you ship to a team or a customer base.

When the agent runs as a shared service account:

• Data leaks across users. A query the agent generates for User A can return rows that belong to User B, because the database only knows the service account, not the caller.
• Access is over-broad. The service account usually has more permissions than any individual user, so the agent can read tables or run writes no single user should trigger.
• Actions are unauditable. Logs show the agent's identity, not the human behind the request. You cannot answer "who asked for this?" after the fact.

User scoping fixes all three by making the caller's identity the thing the agent executes as.

How user scoping connects to access control, memory, and audit logs

User scoping is not a single feature; it is a property that has to hold in three layers.

Vanna's hosted admin features list these directly: Access Control ("control who sees what"), Agent Memory (a hosted vector database for learning), and Audit Logs (activity tracking and compliance). The point of listing them together is that scoping only works if all three respect the same user boundary. An agent with row-level access control but shared memory will still leak information through remembered context.

Practical steps to scope an agent to a user

The exact API depends on your framework, but the sequence is consistent.

1. Pass identity into the agent at request time. The agent should receive the caller's identity (a user ID, a token, or a session) as part of the request, not read it from a global config. In Vanna's SQL Agent, access control is a first-class property of the agent, so the identity you pass determines what the generated SQL is allowed to reach.
2. Enforce permissions at the data layer, not just the prompt. Row-level and table-level filters belong in the query path or the database itself. A prompt instruction like "only show this user's rows" is not enforcement.
3. Isolate memory per user. If the agent stores conversation history or learned context in a vector database, partition it by user ID. Vanna's hosted Agent Memory is described as a hosted vector DB for learning, which is exactly the component that needs per-user isolation in a multi-tenant deployment.
4. Log the acting user on every action. Audit logs should record the human identity alongside the agent action, so you can reconstruct who triggered what.
5. Verify with a second account. The cheapest test: run the same request as two different users and confirm the results differ in the way permissions predict. If they match, scoping is not working.

Vanna works with any LLM provider and database (Anthropic, OpenAI, Gemini, Ollama; SQLite, PostgreSQL, MySQL, Snowflake, BigQuery, and others), so the scoping logic lives in your agent configuration and data layer rather than being tied to one vendor.

Common failure modes and how to detect them

• Credential reuse. The agent connects with one shared key. Detect it by checking whether your connection string or client is created once at startup versus per request.
• Missing tenant filters. The agent generates correct-looking SQL with no WHERE user_id = ... clause. Detect it by logging generated queries and grepping for the filter.
• Leaked conversation history. Memory is stored in one namespace. Detect it by asking the same follow-up question as two users and seeing whether the second user's answer references the first user's context.
• Audit gaps. Logs show the agent but not the user. Detect it by trying to trace a single action back to a person; if you cannot, the log is incomplete.

Each of these is a scoping failure at a different layer, which is why the fix is architectural rather than a single setting.

When user scoping is worth the work

If your agent serves exactly one user or runs entirely on data everyone is allowed to see, shared credentials are simpler and fine. The moment you have multiple users with different permissions, or you are shipping an agent to customers, user scoping stops being optional. Vanna's positioning—"build agents your users can actually use," with user-scoped execution and quota management built in—reflects that shift from demo to production, where identity is the first thing you have to get right.