kiln.tech

What is Kiln AI?

Kiln AI is a desktop workbench for building, evaluating, and improving AI systems, paired with an open-source Python library for deploying those systems in production. It targets teams rather than solo tinkerers: engineers, data scientists, and non-coding contributors such as PMs, QA staff, and subject-matter experts all work inside the same tool. The app runs on macOS, Windows, and Linux, and the underlying Python library is MIT-licensed.

What you actually do in it

The product groups its work into four stages:

• Build — RAG pipelines (index, chunk, retrieve), reusable "skills," tools and MCP integrations, sub-agents, plus structured JSON output and reasoning.
• Evaluate — evals scored by an LLM-as-judge, an AI eval builder that turns a plain-language intent into eval criteria, golden datasets, and human ratings.
• Improve — automatic prompt optimization against your evals, fine-tuning to distill into smaller models, and synthetic data generation with filtering and labeling.
• Collaborate — datasets versioned in Git alongside your code, plus review and feedback flows so the whole team can weigh in.

An AI assistant sits across all of it, letting you run experiments and optimize configurations through conversation rather than clicking through every screen.

Who it fits, and the trade-off

The strongest fit is a team that already ships an AI feature and keeps re-breaking it — the pitch is "solve a problem once, it stays solved," meaning eval results are tracked so a prompt or model change can't silently regress quality. If you're prototyping a single chatbot for yourself, the eval and collaboration machinery is overhead you won't use. Kiln also claims broad model coverage across cloud and local providers, which matters if you need to compare options or run offline.

Next step

If you want to judge it quickly, the practical test is to take one prompt or agent you already have, import it, and write a single eval that captures what "good" means to you. If that eval catches a regression you'd otherwise have missed on your next change, the tool has earned its place; if writing the eval feels like more work than the change itself, you're probably not at the scale where it pays off.

How does Kiln AI's evals platform help teams measure and maintain AI quality?

Kiln AI's evals platform is designed to turn AI quality into something a team can measure and track over time, rather than something judged by gut feel. According to the site, it "tracks quality across every axis," while optimizers search for the best-performing AI configuration. The stated goal is continuity: "Solve a Problem Once. It Stays Solved."

How measurement works in practice

The workbench couples evals with LLM-as-Judge scoring, so every change to a prompt, model, or retrieval setup can be scored against the same standard. Supporting pieces include golden datasets and human ratings, plus an AI Eval Builder that translates plain intent into eval definitions. That combination matters because it lets non-engineers contribute too — the site names PMs, QA staff, and subject-matter experts as people who can add feedback, ratings, and generated data without writing code.

From measurement to maintenance

Measurement only maintains quality if it feeds back into the build. Kiln closes that loop with auto-optimization (tuning prompts against your evals), fine-tuning to distill behavior into smaller models, and synthetic data generation with filtering and labeling. Datasets are versioned in Git, so eval results and the data behind them travel with the codebase instead of living in someone's notebook.

A concrete scenario

Imagine a support agent that starts answering well, then degrades after a model swap. With a golden dataset and judge-based evals in place, the regression shows up as a score drop on the next run. The team can then compare configurations, re-optimize the prompt, or fine-tune a smaller model — and keep the eval as the regression test going forward. The trade-off is upfront work: someone has to build meaningful datasets and eval criteria. Weak or narrow evals will happily certify a system that still fails in production.

Who benefits most

Engineers get an MIT-licensed Python library (3.10+) for deploying tasks in production, while the desktop app serves the wider team. Kiln states it supports cloud and local models and has tested model capabilities across a large library, which helps when choosing a cheaper or local option without losing measured quality.

A useful next step: before adopting any tool, pick one recurring failure in your current AI feature and write three to five eval cases for it. If a platform can run those consistently on every change and let a non-coder add feedback, it will maintain quality in practice. Kiln's own documentation at Kiln AI is the place to check how its eval and optimizer workflow maps to that test.

Can non-engineers like PMs and QA contribute to AI development in Kiln AI without coding?

Yes. Kiln AI is explicitly positioned so that PMs, QA staff, and subject-matter experts can contribute to AI quality without writing code. The page names these roles directly and describes their contribution path as feedback, ratings, evals and data generation rather than programming.

H3 What non-engineers actually do

• Give feedback and reviews. The collaboration section mentions "Feedback — Reviews from the whole team," so reviewers can respond to a task's output as users rather than as developers.
• Rate outputs. Human ratings sit alongside golden datasets and LLM-as-judge scoring in the evals area, which means a person can mark whether an answer was good or bad.
• Contribute to evals. The page describes replacing "vague specs with evals and golden data," which is exactly the kind of work a PM or QA lead can do: define what a good answer looks like, then check outputs against it.
• Generate and label data. Synthetic data generation, filtering and labelling are listed as improvement activities, and labelling in particular is judgement work, not coding work.
• Work through the AI assistant. Kiln's assistant can run experiments and optimize systems through conversation, which lowers the technical barrier for someone who understands the problem but not the pipeline.

H3 Where the coding line still falls

Engineers deploy through the open-source Python library and handle production integration. The app is the shared surface; the library is the developer surface. So a PM can define quality and judge results, but they are not expected to wire up retrieval, tools or deployment.

H3 A practical scenario

A support PM notices the assistant keeps giving overly long answers. Instead of filing a ticket with a vague complaint, they open the task, mark a batch of responses as too verbose, rate the good ones, and add a few examples of the tone they want. The engineer then runs an optimizer against that collected evidence. The PM's contribution is the standard, not the code.

H3 How to decide if this fits your team

Choose this style of workflow if your bottleneck is unclear quality criteria rather than missing engineering capacity. It suits teams where domain experts already know what "good" looks like but have no way to record it. It is less useful if your experts cannot spare time to rate outputs, since the evals are only as good as the human judgement behind them.

A reasonable next step: pick one task, have a PM and a QA reviewer rate twenty existing outputs, and see whether the disagreements reveal a specification gap worth turning into an eval.

How does Kiln AI's open-source Python library support deploying AI tasks to production?

Kiln AI splits its product into two surfaces: a desktop workbench for building and evaluating AI systems, and a separate open-source Python library (MIT-licensed, Python 3.10+) for running the tasks you built there. The production story is that split — you design, evaluate, and optimize in the app, then load and execute the same task definitions from code, rather than re-implementing prompts and pipelines for deployment.

What the library actually gives you

• Task loading from files: the documented pattern is Task.load_from_file(TASK_PATH), so a task defined in the workbench becomes a code artifact you can version and ship.
• A default run configuration per task via task.default_run_config(), giving you a starting point you can override per environment.
• Adapters as the execution layer — the example notes an adapter can run a task, which is what lets the same task target different model backends, including cloud or local models.
• Datasets versioned in your Git repo, which keeps evaluation data and task definitions alongside the code that deploys them.

Why this matters for a real deployment

Consider a team that has a RAG assistant with a retrieval step, a prompt, and a structured JSON output. In the workbench they build the retrieval chain, define evals with an LLM-as-judge, and auto-optimize the prompt against those evals. The production handoff is then a thin service: load the task file, pick a run config, run it inside their own application, and keep the eval suite in CI so a prompt or model change that regresses quality fails before release. Engineers keep deployment control; the people who wrote the evals and golden data do not need to touch the deployment code.

Trade-offs to weigh

The library is a runner, not a platform. It does not replace your API layer, queue, monitoring, or secrets management, and nothing here indicates it handles those for you. Treat it as the piece that guarantees the task you evaluated is the task you deploy.

Next step

Pick your single highest-value task, export it, and write the smallest possible wrapper around Task.load_from_file plus default_run_config() that runs in your staging environment. Then wire the same evals you used in the workbench into CI so the deployed task is checked on every change. For the app side and team workflows, see Kiln AI.

What does Kiln AI's Auto-Optimize and fine-tuning workflow do to improve model performance?

Kiln AI treats Auto-Optimize and fine-tuning as two different levers for improving a model or AI system, and the page groups them under an "Improve" step that sits after building and evaluating.

Auto-Optimize tunes prompts automatically against your evals. Instead of hand-editing a prompt and hoping it gets better, you define what "better" means through evals, and the optimizer searches for a prompt configuration that scores well on them. The page frames this as finding the "optimal AI configuration" and tracking quality across every axis. This is useful when your bottleneck is prompt quality rather than model capability — for example, an agent that keeps producing malformed structured output or misses edge cases in retrieval-augmented answers.

Fine-tuning is positioned as distilling behavior into smaller models. So the trade-off is different: Auto-Optimize keeps your current model and improves how you instruct it, while fine-tuning can move capability into a cheaper or faster model. The page also lists synthetic data generation (generate, filter, label) as part of this step, which is typically what you need to produce training examples when you don't have enough real labeled data.

A practical way to choose between them:

• Use Auto-Optimize when the underlying model is capable enough and the prompt, tools, or retrieval setup is the weak link.
• Use fine-tuning when you've validated a working configuration and want to reduce cost, latency, or dependence on a large model.
• Use synthetic data when fine-tuning is blocked by too few examples.

Both paths depend on evals and golden datasets being in place first, because otherwise you have no reliable signal that a change helped. The page also notes a Python library (MIT-licensed) for deploying tasks, so improvements validated in the app can be carried into production code.

Next step: before optimizing anything, write down the specific failure you want to fix and turn it into an eval. That single artifact determines whether Auto-Optimize or fine-tuning is the right move. If you want to see how the pieces fit together, start at Kiln AI.

How does Kiln AI handle running models locally versus in the cloud?

Kiln AI treats local and cloud models as interchangeable options inside one workbench, rather than as separate products or workflows. The page frames this as "Every model. Cloud or local." — the same task, evals and optimizers are meant to apply whether a model runs on your own hardware or through a hosted provider. A model library is included to compare capabilities, so you can pick a model based on what it can do rather than on where it happens to run.

What that means in practice

• You define a task once, then swap the model behind it. Evals and auto-optimization are described as tracking quality across changes, which is the point of keeping both options in one place: you can test a local model against a cloud one on the same golden dataset instead of rebuilding your test setup.
• The workbench is a desktop app (macOS, Windows, Linux), so local execution fits a laptop or workstation with enough compute. Cloud models are reached the same way — as another selectable model — which suits teams that don't want to provision GPUs.
• Deployment is decoupled from authoring. The open-source Python library (MIT-licensed, Python 3.10+) is how tasks move into production, so the local-versus-cloud choice can be revisited at deploy time rather than locked in when the task is built.

How to decide which to use

A concrete scenario: a team building a document-search assistant starts on a cloud model to get quality high, writes evals and a golden dataset, then tries a local model on the same evals. If scores hold, they switch the production run config; if not, they keep the cloud model for the hard queries and route easy ones locally.

As a next step, browse the model library and run one existing task against both a local and a cloud model using the same eval set — that single comparison tells you more than any general rule about which is better. For the wider ecosystem of local runtimes and hosted APIs, see Ollama and OpenRouter.