简体中文
EnglishWebsite review
This review may include AI-generated content and is not individually verified. Domain, certificate and other facts come from the observed data on this page.
What is Hamel's Blog?
Hamel's Blog is the personal site of Hamel Husain, a machine-learning engineer with more than 20 years of experience across Airbnb, GitHub and open-source ML tooling. Its writing concentrates on applied AI engineering, with a strong emphasis on evals — the practices teams use to debug, analyze and measure AI systems — alongside broader notes on machine learning and data science.
What you'll find there
- An eval-focused blog. Long-form posts on topics such as LLM-as-a-judge, automated evals, decision models for evals, selecting eval tooling, and why "it's hard to eval" can signal a product problem.
- Practical AI product lessons. Articles on rapidly improving AI products, fine-tuning, RAG, prompt inspection, latency, tokenization and data curation — written from hands-on building experience rather than theory.
- A teaching track. Husain co-teaches an "AI Evals for Engineers and PMs" course and maintains an AI Evals Guide, with cohort dates posted on the page.
- A newsletter. Readers can subscribe to receive new posts.
Who it suits
Engineers, data scientists and product managers who are shipping LLM-based features and need concrete methods for measuring quality. If you are choosing between eval tools or setting up a first evaluation suite, the tool-selection and "field guide" posts are the natural entry points. Readers looking for introductory ML tutorials or general data-science career content will find less of that here.
A useful next step: start with "AI Evals: Everything You Need to Know," then read "Selecting The Right AI Evals Tool" once you know what you want to measure. For a broader practitioner community around LLM application building, Hamel's Blog links naturally to work discussed by others in the same space, such as the applied-AI writing at Simon Willison's Weblog.
How can I learn AI evals through Hamel Husain's course or guide?
Start with the free AI Evals Guide on Hamel's Blog to learn the core ideas, then join a cohort of the paid course, AI Evals for Engineers and PMs, which Husain co-teaches. The course page lists upcoming cohorts; the guide answers common questions about evals.
What each resource is for
| Resource | Best for | What you get |
|---|---|---|
| AI Evals Guide | Deciding whether evals matter for your team, or getting vocabulary fast | Answers to common eval questions, free to read |
| Course (co-taught) | Engineers and PMs who need to build and run an eval process on a real product | Structured teaching; the page reports 5,000+ students from 500+ companies, including OpenAI, Anthropic and Google |
| Blog | Self-directed depth on specific problems | Long-form posts such as "AI Evals: Everything You Need to Know", "Using LLM-as-a-Judge For Evaluation", "Selecting The Right AI Evals Tool" and "Your AI Product Needs Evals" |
A practical path
- Read the guide, then "Your AI Product Needs Evals" and "AI Evals: Everything You Need to Know" to see the full workflow.
- Pick one post that matches your blocker: "Using LLM-as-a-Judge For Evaluation" if you need scoring at scale, "Selecting The Right AI Evals Tool" if you are choosing tooling, "Do Automated Evals Work?" if you are skeptical of automation.
- If you want feedback and a schedule, check the course page for the next cohort. If you prefer to move at your own pace, follow the blog series and apply it to one feature in your product.
How to choose
Take the course if you are an engineer or PM who needs to ship an eval system and wants guided practice; the trade-off is cost and a fixed cohort schedule. Use the free guide and blog if you are exploring, on a tight budget, or only need one specific technique. Either way, plan to run evals on your own data within the first week, because the material is about debugging and measuring real systems, not abstract theory.
As a next step, write down one failure your AI product has today, then search the blog for a post that addresses it and try that method on a small sample before scaling up.
How should I choose the right AI evals tool for my team?
Start with the decision you are actually trying to make: are you choosing a tool to debug and improve an AI product, or a tool to gate releases with automated scores? Teams often conflate the two, and that mismatch is the most common reason an evals tool feels wrong after a few weeks.
Hamel Husain's blog is a strong starting point because it treats evals as a measurement and debugging discipline rather than a dashboard purchase. His writing on Hamel's Blog includes a dedicated post on selecting an AI evals tool, plus guides on LLM-as-a-judge, automated evals, and decision models for evals.
Match the tool to the job
| If your team needs to… | Prioritize | Accept as a trade-off |
|---|---|---|
| Understand why outputs fail | Fast iteration on test cases, easy inspection of traces and failures, support for human review | Less out-of-the-box automation; more manual curation early |
| Block regressions in CI | Deterministic checks, stable scoring, versioned datasets, clear pass/fail reporting | Rigid thresholds that may miss nuanced quality issues |
| Evaluate open-ended quality at scale | LLM-as-a-judge with calibration against human labels | Judge bias and drift; needs ongoing spot checks |
| Ship quickly with a small team | Low setup cost, integration with your existing stack, hosted option | Less control over data handling and custom metrics |
A practical sequence
- Write 20–50 real examples first. Pull from production logs or user complaints. If you cannot describe what "good" looks like for these, no tool will fix that — Husain frames this as a product problem, not a tooling one.
- Start with human review and simple code checks. Many teams only need a spreadsheet, a script, and a rubric before adopting a platform.
- Add LLM-as-a-judge only after you can validate it. Compare judge scores against your own labels on a sample; keep the judge prompt versioned.
- Then choose tooling based on whether you need hosted collaboration, CI integration, or open-source control.
Concrete scenario
A five-person team building a support assistant notices vague answers. Before buying anything, they label 40 failing conversations, find that most failures involve missing policy details, and add a retrieval check plus a judge rubric for "cites policy." Only then do they pick a tool — and the deciding factor becomes whether it can run their custom check in CI, not which vendor has the longest feature list.
Decision criteria to apply
- Data control: can you keep sensitive examples in your own environment?
- Custom metrics: can you plug in your own checks and judges, or are you limited to built-in scores?
- Workflow fit: does it meet your team where they work (notebooks, CI, dashboards)?
- Cost of maintenance: who recalibrates judges and updates datasets as the product changes?
For deeper grounding, Husain's AI Evals Guide and his long-form posts are the most directly relevant reading; Inspect AI is worth knowing if you want an open-source Python framework for writing evals as code.
How do I use LLM-as-a-judge to evaluate my AI product?
LLM-as-a-judge means using a language model to score or compare your product's outputs against defined criteria, so you can measure quality at a scale that human review can't cover. The practical way to start is to treat the judge as a measurement instrument: define what "good" means for your specific task, write a small set of scored examples yourself, then check whether the judge agrees with you before you trust its numbers on thousands of cases.
Hamel Husain's writing treats evals as the core discipline for building AI products, and his guide on using LLM-as-a-judge is the most directly relevant piece here. His broader argument, in posts like "Your AI Product Needs Evals" and "It's Hard to Eval Is a Product Smell," is that if you can't articulate what a good output looks like, that's a product-definition problem rather than a tooling problem. See Hamel's Blog for those essays and his AI Evals Guide.
A workable sequence
- Write the rubric in plain language. For a support-reply generator, criteria might be: answers the question asked, cites the correct policy, no invented refund amounts, appropriate tone. Vague criteria like "helpful" produce judges that agree with everything.
- Label a seed set by hand. Fifty to a few hundred real inputs with your own pass/fail or 1–5 scores. These are your ground truth.
- Build the judge prompt. Give the judge the input, the output, the rubric, and ask for a structured verdict with a short justification. Ask for reasoning before the score; it usually improves agreement.
- Measure agreement. Compare judge scores to your labels. If the judge disagrees on a chunk of cases, read those cases — often the rubric is ambiguous, not the judge.
- Iterate and re-check. Tighten criteria, add few-shot examples of tricky edge cases, and re-run agreement. Only then wire the judge into CI or a nightly batch.
Where judges work well and where they don't
Judges are strongest on rubric-checkable qualities: factual consistency against a provided source, instruction-following, format compliance, tone, and pairwise preference between two outputs. They are weaker on anything requiring deep domain verification, exact arithmetic, or knowledge the judge model may not have — there, deterministic checks or retrieval-grounded comparison do more work. Pairwise comparisons ("which of these two is better?") tend to be more reliable than absolute 1–10 scores, and they suit A/B testing of prompt changes.
Practical cautions
- Watch for position bias (judges favoring the first or longer answer) by swapping order and controlling for length.
- Use a strong model as judge, but validate against human labels rather than assuming capability equals calibration.
- Log judge justifications; they're how you debug disagreements and improve the rubric.
- Keep a human-reviewed sample in the loop permanently — judges drift as your product and inputs change.
For tooling, Husain has written about Hamel's Blog's coverage of Inspect AI, an open-source Python library for LLM evals, and about selecting the right evals tool. If you want a broader industry view of evaluation practice, Anthropic and OpenAI both publish evaluation guidance in their documentation.
A useful first step: pick your ten most common failure modes from real user logs, write a one-line pass/fail test for each, and hand-label 100 examples. That seed set will tell you quickly whether a judge is viable for your product or whether you need narrower, rule-based checks first.
How can I debug and improve an AI product that seems hard to evaluate?
Start by treating "hard to eval" as a product problem, not a measurement problem. If you cannot describe what a good output looks like for a specific user and task, no benchmark will rescue the product. Hamel Husain makes this argument directly in his writing on evals, including the post titled "'It's Hard to Eval' Is a Product Smell" on Hamel's Blog, and his broader AI Evals Guide is aimed at exactly this situation: teams that have shipped an LLM feature and need to debug, analyze and measure it rather than guess.
A practical debugging loop
- Collect real traces. Pull a sample of actual inputs and outputs from production or from realistic user sessions. Do not start from synthetic prompts.
- Read outputs by hand, in batches. Label each one pass/fail against a written definition of quality, and note why it failed. Husain's blog repeatedly returns to this: looking at data is the work, and tooling is secondary.
- Cluster the failures. Group them into a small number of recurring failure modes — wrong retrieval, ignored instruction, bad formatting, hallucinated fact. This tells you whether the problem is the prompt, the data, the retrieval step or the model.
- Write targeted checks per failure mode. Start with deterministic checks (schema validity, required fields, forbidden phrases) before reaching for model-graded judgments.
- Add LLM-as-a-judge only where rules fail. His guide on using LLM-as-a-judge covers calibrating a judge against your own human labels; an uncalibrated judge is just a second opinion you cannot trust.
- Re-run the same set after every change. A fixed, growing evaluation set turns "it feels better" into a before-and-after comparison.
Deciding what to measure
| Situation | Start with | Trade-off |
|---|---|---|
| Output has a fixed structure | Deterministic assertions | Cheap and reliable, but blind to tone and reasoning quality |
| Open-ended answers, clear criteria | Human labels on a sample, then a calibrated judge | Slower to set up, but catches the failures users actually hit |
| Multi-step pipeline (retrieval, tools) | Trace-level inspection of each step | More work per case, but localizes the fault instead of blaming the model |
| Fast-moving prototype | A small hand-checked set of 20–50 cases | Quick signal, but you must keep adding real cases or it goes stale |
For teams that want an off-the-shelf harness, Husain has written about Inspect AI, an open-source Python library for LLM evals, and about selecting the right evals tool — useful if you would rather adopt a framework than build scripts. If you want structured teaching rather than reading, his site lists a co-taught course, "AI Evals for Engineers and PMs," with details and upcoming cohorts on the page.
Next step: pick the single failure mode you see most often in your traces, write five concrete test cases for it, and make those the first entries in a versioned eval set you run on every change.
What are practical techniques for optimizing LLM latency and tokenization?
Hamel's Blog focuses on applied AI engineering, and two posts address your question directly: one on optimizing LLM latency and one on tokenization gotchas. Both are practitioner notes rather than tutorials, so treat them as starting points for your own profiling.
Latency optimization
The page lists "Optimizing LLM latency" as a blog entry, which signals the topic is covered as an engineering concern rather than a theory piece. Practical techniques generally fall into three layers:
- Request level: stream tokens so users see output sooner, cap
max_tokens, and trim prompts and context. Latency that matters to users is usually time-to-first-token plus generation time, and each has different levers. - Serving level: batching, KV-cache reuse, and faster inference servers. The blog also covers "vLLM & Large Models," which is the standard open-source route to higher throughput and lower per-request latency under load.
- Measurement: instrument each stage (queue, prefill, decode) before changing anything. Without a breakdown, you cannot tell whether the prompt, the model, or the server is the bottleneck.
A concrete scenario: a chat assistant feels slow. If time-to-first-token is high, shorten the system prompt and check queue depth; if generation is slow, consider a smaller model or streaming with a token budget.
Tokenization
"Tokenization Gotchas" is listed as a post, and the topic matters because token counts drive both cost and latency. Practical checks:
- Count tokens with the tokenizer matching your model, not a generic one; counts differ across model families.
- Watch for languages, code, and rare strings that tokenize inefficiently, inflating both cost and generation time.
- Validate truncation behavior: silently cutting context can remove instructions and degrade output.
- Set limits in tokens, not characters.
Where to go next
Start with the latency post for a concrete checklist, then read the tokenization post to align cost and speed estimates with your model's tokenizer. For broader evaluation work around these changes, Hamel's Blog also hosts an AI Evals guide and a course, which help you measure whether latency or tokenization changes actually improve the product.
Google Cloud DNS
Fastly
Fastmail
User reviews (0)