What Hamel Husain Means by "Evals" in AI Engineering

On hamel.dev, Hamel Husain uses "evals" to mean the practice of debugging, analyzing, and measuring AI systems—not just scoring model outputs, but applying data-science methods to understand why a system behaves as it does and whether changes actually improve it. He frames this as "bringing data science back to AI," and it is the stated focus of his teaching and writing. The idea applies to teams building LLM-powered products who need evidence—rather than intuition—about quality.

The core idea

Husain describes his current work as "helping teams debug, analyze, and measure their systems. I call this 'evals.'" Three verbs matter here:

• Debug — find out where and why an AI system fails on real inputs.
• Analyze — look at the data (inputs, outputs, traces) to locate patterns behind failures.
• Measure — track quality over time so you can tell whether a change helped.

This is broader than a benchmark score. It treats evaluation as an ongoing engineering loop tied to product improvement, which is why his writing connects evals to product decisions rather than to leaderboard rankings.

Why he argues it matters

Husain's background shapes the argument: he is a machine learning engineer with 20+ years of experience, has worked at Airbnb and GitHub (including early LLM research used by OpenAI for code understanding), and has led and contributed to popular open-source machine-learning tools. From that perspective, the gap he points to is teams shipping AI products without the measurement discipline that data science normally brings.

Two blog titles state the position directly: "Your AI Product Needs Evals" (3/29/24) and "'It's Hard to Eval' Is a Product Smell" (6/29/26). The second is the sharper claim—if a product is hard to evaluate, that difficulty is itself a signal about the product's design, not merely a testing inconvenience.

Where to start reading

The site lists a dated archive of long-form posts. For understanding evals specifically, these are the most direct entry points:

The site also points to a dedicated AI Evals Guide for answers to common questions, which is the natural next stop after the overview posts.

Learning with him

Husain co-teaches AI Evals for Engineers and PMs, described on the site as having over 5,000 students from 500+ companies, including OpenAI, Anthropic, and Google. The page directs readers to check for upcoming cohorts. This is a paid-style course offering in the sense that it is a structured class with enrollment—the site does not state pricing, so treat cost and access terms as things to confirm on the course page rather than assume.

A concrete example of the mindset

Suppose a team ships a support assistant and users report it "sometimes gives wrong answers." A score-only approach yields a number and little else. The evals approach Husain describes would instead: collect the failing conversations, inspect them for patterns (e.g., failures cluster on questions requiring account-specific data), form a hypothesis, change the system, and re-measure on the same collected cases to confirm the fix. The measurement is tied to a debugging question, which is the distinction he keeps drawing.

Practical takeaways

• Treat evals as debugging plus measurement, not a one-time benchmark.
• Build the evaluation set from real product inputs and failures.
• Use eval difficulty as a design signal—hard-to-eval features may need rethinking.
• Start with the overview and guide on the site, then move to the LLM-as-a-judge and tool-selection posts when you need methods and tooling.