How is HotTea different from proctored coding assessments?

Updated

HotTea is a paid-pilot assessment tool that replaces proctoring with an evidence trail. Instead of locking the browser and blocking AI, it gives each candidate a real developer VM with Claude, Codex, git, tests, and a normal shell, then records how they direct and verify the agent. The output is a reviewer packet — transcript, terminal output, git snapshots, final diff, tests, hidden checks, and a reviewer note — rather than a pass/fail proctoring verdict. It fits teams that already trust practical coding tasks but want a defensible record of AI-era engineering behavior.

The core difference: prevention vs. observation

Traditional proctored assessments assume AI use is cheating and try to prevent it. HotTea assumes candidates will use AI agents in the job, so it measures how well they do it.

Dimension Proctored coding assessment HotTea
Environment Locked browser, fake editor, webcam/mic monitoring Real isolated developer VM with Claude, Codex, tests, git, shell
AI use Blocked or flagged Expected — agentic turns are recorded and analyzed
What's captured Keystrokes, screen, violations Transcript, terminal output, git snapshots, final diff, tests, hidden checks, reviewer note
Output Pass/fail or score with integrity flags Hiring Signal Report: what the candidate proved, what they did not prove, where reviewers disagreed, what to ask next
Goal Detect rule-breaking Make the hiring decision defensible

What the reviewer packet actually contains

The sample packet on the site (a Hatchways-style invite regression task) shows the structure. A candidate named Alex Chen, tagged Senior Backend Developer, worked for 29 minutes across 3 AI turns and scored 9/10 on AI fluency with 4/4 hidden checks passed.

The timeline entries are the point:

  • 18:02 — Workspace VM provisioned with Claude Code and recorder.
  • 18:09 — Candidate asked Claude to map the invite flow, then rejected the first suggestion, citing wrong branch order.
  • 18:18 — Moved expiry validation ahead of the retryable membership branch — identified as the root cause.
  • 18:31 — Git snapshot, tests, and reviewer note attached to packet.

That is the AI-fluency evidence: how the candidate directs the agent, what they probe, what they push back on, and what they verify. A proctoring tool would have logged none of it — or flagged the AI use as a violation.

Why this matters for the hiring decision

GitHub-based assessments already show what a candidate can build. The stated gap HotTea targets is the AI-era audit trail: the record of agentic engineering behavior that a final repo alone doesn't reveal. Two candidates can submit identical diffs while one drove the agent deliberately and verified the result, and the other accepted the first suggestion without checking. The packet distinguishes them.

The design choice follows from that: no proctoring, no locked browsers, no fake editors. The candidate works the way they really work, and the reviewer reads the receipts.

Cost and guardrails

Founding-team access is listed at $199/mo, including 25 candidate sessions per month and an estimated 2M monthly token cap. The site describes hard caps on prompt length, message count, candidate session count, and monthly tokens — requests stop before overage rather than billing through. Access activates when payment clears via Stripe Checkout.

When HotTea is the better fit

Choose it if your team:

  • Already runs practical coding tasks and trusts them as signal
  • Wants evidence of how candidates work with AI agents, not proof they avoided AI
  • Needs a defensible record for hiring decisions — quoted reasoning moments, probe responses, and follow-up questions
  • Is willing to run a paid pilot and review packets manually

Stick with proctoring if your constraint is a compliance or exam-integrity requirement that mandates monitored, AI-free conditions. HotTea is not built for that; it is built for teams hiring agentic engineers who will use these tools on the job.