简体中文
EnglishWebsite review
This review may include AI-generated content and is not individually verified. Domain, certificate and other facts come from the observed data on this page.
What is Eugene Yan?
Eugene Yan is a personal website and professional blog for Eugene Yan, an AI engineer and writer. The site describes him as a Member of Technical Staff at Anthropic, working on recommendation systems and AI-powered products at scale, with prior ML/AI leadership at Amazon, Alibaba, Lazada, and a healthtech Series A. It is not a product or company page; it is where he publishes essays, talks, and prototypes.
What you'll find on the site
- Writing on practical AI engineering: recommendation systems, LLMs, evaluation (including LLM-as-judge), prompting, and building reliable AI products.
- Prototypes: small apps such as Obsidian Copilot, AlignEval, and AI Reading Club, shown as working examples rather than polished products.
- Talks and teaching: conference material plus pointers to ApplyingML.com and Applied-LLMs.org for applied ML/LLM guidance.
- A newsletter: the site says it goes to 11,800+ readers, covering RecSys, LLMs/AI, and lessons learned.
Who it's for
An ML engineer preparing an LLM feature for production might read the evaluation and system-pattern essays before choosing an approach. A technical lead hiring for an AI team could use the interview and leadership pieces as a discussion framework. A curious builder can browse the prototypes to see how ideas get tested quickly.
How it compares to other AI writing
| Site | Emphasis | Best for |
|---|---|---|
| Eugene Yan | Practitioner essays, evals, RecSys, prototypes | Engineers shipping AI systems |
| Lilian Weng | Deep technical surveys of ML/LLM topics | Readers wanting conceptual depth |
| Simon Willison | Frequent notes on tools and LLM practice | Tracking the fast-moving tool landscape |
A useful next step: open the writing index and start with the evaluation or LLM-system-patterns posts, since those map most directly to decisions you'll face when building with LLMs.
What does Eugene Yan write about on his newsletter?
Eugene Yan's newsletter covers recommendation systems, LLMs and AI, and his own learning notes. He describes it as a send-out on "RecSys, LLMs/AI, and what I'm learning," and it reaches more than 11,800 readers. The site's article list shows the range in practice: technical pieces on building and evaluating LLM systems, recommendation-system design, engineering and career advice, and yearly reviews.
What the topics look like in practice
- Building AI systems: patterns for LLM-based products, prompting fundamentals, and lessons from a year of building with LLMs.
- Evaluation: LLM-as-judge effectiveness, product evals in three steps, long-context Q&A evaluation, and cybersecurity evals.
- Recommendation systems: improving RecSys and search with LLM techniques, and hybrid LLM-RecSys training.
- Engineering and career: hiring ML/AI engineers, advice for new principal tech ICs, and how to work and compound with AI.
If you're deciding whether to subscribe, check the archive first: if the evaluation and RecSys posts match your work, the newsletter is likely a good fit; if you want broad AI news rather than hands-on system-building notes, it may be narrower than you need.
How can I learn to apply LLMs and ML effectively through his teaching resources?
Start with the two teaching sites Eugene Yan points to: Applying ML for machine learning in practice, and Applied LLMs for LLM application work. They are the structured learning paths; his main site, Eugene Yan, is where the broader essays, talk write-ups and prototypes sit. If you want a single place to see what he is currently working on and which resource fits, start there and follow the links to the teaching sites.
A practical way to work through the material
- Pick one concrete project first. A small internal Q&A tool over your own documents, or a search ranking improvement, gives every essay something to attach to.
- Read the applied LLM material in build order. Prompting fundamentals, then patterns for building LLM-based systems and products, then evaluation. Evaluation is the part most people skip and the part that decides whether a prototype survives contact with users.
- Use the ML material for the non-LLM half. Recommendation and search systems, feature work and model evaluation are covered there; many real products need both a classic ranking model and an LLM layer.
- Read the prototypes as worked examples, not products. Tools like AlignEval and Obsidian Copilot show how a narrow, well-scoped use case gets built; the value is in the scope decisions, not the feature list.
- Follow the talks and year-in-review posts for calibration. They summarise what changed in practice over a year, which is useful for deciding what to learn next.
Choosing between the resources
| If your goal is… | Start with | Why |
|---|---|---|
| Ship an LLM feature at work | Applied LLMs | Focused on building, prompting and evaluating LLM systems |
| Strengthen core ML and RecSys skills | Applying ML | Practical ML application and recommendation systems |
| See current work and examples | Eugene Yan | Essays, talks and prototypes in one place |
| Learn evaluation specifically | The eval essays on his main site | Covers LLM-as-judge and product evals in steps |
A useful decision rule: if you cannot yet describe how you would measure whether your LLM feature works, spend your first week on the evaluation material before anything else. For a reader scenario, a data scientist asked to add a document assistant to an internal wiki should read the prompting and system-pattern pieces, build a fifty-question eval set from real wiki queries, and only then choose a model or retrieval setup.
The site's newsletter is a low-effort way to keep up; treat it as a supplement to working through the guides, not a replacement for building something.
What are the key patterns for building LLM-based systems and products?
The key patterns are the recurring design choices that make LLM systems reliable enough to ship: retrieval, evaluation, guardrails, caching, and careful prompting, each chosen to fit the task rather than applied by default. Eugene Yan's writing on Eugene Yan treats these as a practical toolkit drawn from production experience, not a single architecture.
Patterns worth knowing
- Retrieval-augmented generation: fetch relevant documents, then let the model answer from them. Useful when answers must reflect private or current data; the trade-off is added latency and the need to keep the index fresh.
- Evaluation as a first-class step: build task-specific evals, including LLM-as-judge where human review does not scale. Yan's posts on product evals and LLM-evaluators stress checking judge reliability against human labels.
- Guardrails and defense in depth: input filtering, output checks, and constrained actions. Necessary for customer-facing or agentic systems; adds engineering overhead.
- Caching and cost control: reuse frequent responses and route simple queries to smaller models. Helps latency and spend, but risks stale answers.
- Prompting fundamentals: clear instructions, examples, and structured output. Cheap to try first, but fragile for complex reasoning without supporting patterns.
Choosing between them
| If your situation is… | Start with | Main trade-off |
|---|---|---|
| Answers need internal or fresh data | Retrieval | Index maintenance, latency |
| Quality is hard to measure | Evals before launch | Upfront effort, ongoing labeling |
| Users see output directly | Guardrails | Extra checks, some false positives |
| Volume and cost matter | Caching, model routing | Possible staleness |
A practical next step
Pick one user-facing task, write ten representative test cases with expected answers, and run them against a simple prompt-plus-retrieval version. If results vary, invest in evals before adding complexity; if they are stable but slow or costly, add caching. For deeper treatment, Yan also teaches applied material at Applying ML and Applied LLMs.
How do I evaluate LLM systems and LLM-as-judge approaches?
Start with the article that matches the question directly: Eugene Yan's "Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)" on Eugene Yan, which is the closest thing on the site to a practical guide to judge-based evaluation. Pair it with "Product Evals in Three Simple Steps" for a lighter, product-facing process, and "Evaluating Long-Context Question & Answer Systems" if your inputs are long documents.
What to take from the site
Eugene's framing, as reflected in his writing and talks, treats evaluation as an engineering discipline rather than a one-off benchmark: define what "good" means for the product, build a small but well-labeled set, and treat the judge as a component that itself needs validation. His "Patterns for Building LLM-based Systems & Products" and "What We've Learned From A Year of Building with LLMs" put evals in the same loop as prompting and retrieval, not after it.
A practical way to evaluate LLM-as-judge
- Build a human-labeled gold set first. 50–200 examples covering common cases and known failure modes. Without this, you cannot tell whether the judge is right.
- Measure the judge against humans. Report agreement (e.g., exact match on pass/fail, or correlation on scores) and inspect disagreements. A judge that agrees 70% of the time is not a reliable gate.
- Use pairwise or rubric-based prompts, not free-form scores. Ask for a verdict plus a short rationale, and require the judge to quote evidence from the output.
- Control for known biases. Position bias in pairwise comparisons, verbosity bias, and self-preference for outputs from the same model family. Swap order and test with a different judge model.
- Check stability. Re-run the same inputs; a judge that flips verdicts on identical inputs is too noisy to trust.
- Keep a human in the loop for high-stakes decisions. Use the judge for triage and regression detection, and route ambiguous cases to review.
Choosing between approaches
| Approach | Best for | Main trade-off |
|---|---|---|
| Deterministic checks (schema, exact match, regex) | Structured outputs, formatting, retrieval hit rate | Misses quality and nuance |
| Human review | Gold sets, high-stakes releases, calibrating judges | Slow and costly at scale |
| LLM-as-judge | Regression testing, large-scale triage, subjective quality | Needs its own validation; can be biased or unstable |
| Pairwise comparison | Choosing between two model or prompt versions | Position bias; harder to set an absolute bar |
A concrete scenario: a team shipping a support-answer assistant labels 150 real tickets, writes a rubric (correctness, groundedness in retrieved docs, tone), and runs an LLM judge nightly. They gate releases on judge–human agreement above a threshold and manually review every disagreement above a certain score gap. That turns the judge into a fast filter while humans keep authority over the decisions that matter.
For adjacent perspectives, Hamel Husain writes extensively on evaluation-driven development, and Anthropic publishes guidance on building evaluations for AI systems.
What advice does Eugene Yan offer for interviewing and hiring ML/AI engineers?
Eugene Yan's site points to a dedicated article, "How to Interview and Hire ML/AI Engineers" (7 Jul 2024), as the place where he sets out his hiring advice. The homepage itself only lists that piece among his most-read writing; it does not reproduce the recommendations. So the short answer: his interviewing and hiring guidance lives in that article at Eugene Yan, not in the site introduction.
What the site does establish
- He writes from direct experience leading ML/AI teams at Amazon, Alibaba, Lazada and a healthtech Series A, and currently works as a Member of Technical Staff at Anthropic. That background is the basis for his hiring perspective.
- His related writing on "Advice for New Principal Tech ICs" and on working with AI suggests he treats hiring as part of building reliable, scalable AI teams rather than a standalone checklist.
A practical next step
If you are designing an ML/AI interview loop, read the article first, then use it as a prompt for your own scorecard. A concrete reader scenario: a startup hiring its first ML engineer can map his categories onto a structured loop, separating coding ability, ML fundamentals, product sense and communication, and assigning each to a specific interview rather than testing everything at once.
For broader context on applying ML and LLMs in practice, he also points readers to Applying ML and Applied LLMs.
Cloudflare
Fastly
Google Workspace
User reviews (0)