What evidence does HotTea collect to measure AI fluency?

Updated

HotTea collects evidence from a real agentic coding session — not a proctored exam — and bundles it into a reviewer packet. Candidates work in an isolated developer VM with Claude, Codex, tests, git, and normal shell tooling, and the platform captures the transcript, terminal output, git snapshots, final diff, tests, hidden checks, and a reviewer note. A structured fluency analysis then quotes specific reasoning moments and adds probe responses and follow-up questions for the next interview.

The workspace the evidence comes from

Each applicant gets an isolated developer VM provisioned with Claude Code, Codex, tests, git, and a normal shell. There is no mock IDE and no locked browser, so the candidate works the way they actually work. The sample packet shows this in practice: a workspace VM is provisioned with Claude Code and a recorder, and the session runs for a bounded window (the sample shows 29 minutes and 3 AI turns).

Because the environment is a real repo rather than a simulated editor, the artifacts it produces are the same ones an engineer would leave behind on the job — which is what makes them usable as hiring evidence.

What lands in the reviewer packet

One packet per candidate, containing:

  • Transcript — the candidate's conversation with the agent, including what they asked for and how they responded to what came back.
  • Terminal output — the commands run and their results.
  • Git snapshots — intermediate states, not just the final answer.
  • Final diff — the actual change the candidate landed.
  • Tests — the test runs and their outcomes.
  • Hidden checks — checks the candidate did not see, used to verify the work independently.
  • Reviewer note — a human-written note attached to the packet.

The sample packet illustrates the shape of this: a Hatchways-style invite regression task, a candidate listed as a senior backend developer, 9/10 AI fluency, 4/4 hidden checks passed, and 3 agentic turns observed. Timestamped entries show the workspace being provisioned, the candidate asking Claude to map the invite flow and rejecting the first suggestion because of wrong branch order, then moving expiry validation ahead of the retryable membership branch to fix the root cause. A git snapshot, tests, and reviewer note are attached at the end.

How AI fluency is actually measured

The fluency analysis is not a single score pulled from nowhere. It is built from observable behaviors during the session:

  • How the candidate directs the agent — the framing and specificity of their prompts.
  • What they probe — which parts of the agent's output they interrogate rather than accept.
  • What they push back on — moments where they reject or correct a suggestion, as in the sample where the candidate rejected the first suggestion citing wrong branch order.
  • What they verify — how they confirm the change actually works, via tests and hidden checks.

The structured analysis quotes these reasoning moments directly, so a reviewer can read the candidate's own words rather than a summary. It also records probe responses and generates follow-up questions — what the candidate proved, what they did not prove, where reviewers disagreed, and what to ask next.

What the packet is designed to support

The stated purpose is a defensible technical hiring decision. GitHub-based assessments already show what a candidate can build; HotTea adds the AI-era audit trail for agentic engineering on top of that. The reviewer packet is meant to answer the questions a practical coding task alone cannot: how the candidate worked with an agent, where they exercised judgment, and what remains unverified.

Cost guardrails on the evidence collection

Sessions run under hard caps rather than soft warnings: per-session, per-message, and monthly token ceilings, plus limits on prompt length and candidate session count. Requests stop before overage. The founding access tier is listed at $199/mo, including 25 candidate sessions per month and an estimated 2M monthly token cap. Access activates when payment clears through Stripe Checkout, and a sample packet is available to open before committing.