Evals Before Vibes: How to Know Your Agent Actually Works

The vibe trap

You built an agent. You chatted with it for 30 minutes. It seemed smart. You shipped it.

Three days later a user reports it confidently gave the wrong answer to a basic question. You test again — it works fine now. You shrug. The model must have been having a bad day.

This is the vibe trap: using informal impressions as your quality signal. It works at hobby scale. It fails the moment you have real users, real money, or real stakes.

Evals are the exit ramp.

What an eval actually is

An eval is a function that takes a prompt + expected behavior and returns a score. That's it. No fancy infrastructure required to start.

The simplest eval you can write:

interface EvalCase {
  input: string;
  expected: string;
  check: (output: string) => boolean;
}

const cases: EvalCase[] = [
  {
    input: "What is 2 + 2?",
    expected: "4",
    check: (out) => out.includes("4"),
  },
  {
    input: "Summarize: 'The quick brown fox jumps over the lazy dog'",
    expected: "mentions fox and dog",
    check: (out) => out.toLowerCase().includes("fox") && out.toLowerCase().includes("dog"),
  },
];

async function runEvals(model: (prompt: string) => Promise<string>) {
  let pass = 0;
  for (const c of cases) {
    const output = await model(c.input);
    if (c.check(output)) {
      pass++;
      console.log(`✓ ${c.input.slice(0, 40)}`);
    } else {
      console.log(`✗ ${c.input.slice(0, 40)}`);
      console.log(`  Got: ${output.slice(0, 100)}`);
    }
  }
  console.log(`\n${pass}/${cases.length} passed (${Math.round(pass / cases.length * 100)}%)`);
}

Run this on every prompt change. You now have a regression suite.

The three check types you need

1. Exact match — for structured outputs, codes, IDs, yes/no decisions. Binary. Fast.

2. Contains check — for prose outputs where exact wording varies but key facts must appear. "The output mentions 'Paris'" is more robust than "the output equals 'Paris is the capital of France'".

3. Model-graded — for complex tasks where you need semantic judgment. Ask a second (cheaper, faster) model: "Does this response correctly answer the question? Reply YES or NO with one reason." This scales to subjective quality, tone, and reasoning chains.

async function modelGrade(output: string, criterion: string): Promise<boolean> {
  const verdict = await callModel(
    `Criterion: ${criterion}\nOutput: ${output}\nDoes the output meet the criterion? Reply YES or NO only.`
  );
  return verdict.trim().startsWith("YES");
}

What to eval first

If you're starting from zero, eval in this order:

1. Happy path — the 5-10 inputs your agent was explicitly designed for. These should all pass. If they don't, you don't have a working agent, you have a demo.

2. Edge cases you've already seen fail — anything that broke in testing. Encode these immediately. Every bug is an eval case.

3. Adversarial inputs — empty inputs, malformed inputs, out-of-domain questions, prompt injection attempts. Your agent needs to degrade gracefully.

4. Regression set — a fixed set of 50-100 cases that you run on every change. Green means you didn't break anything. Red means you have a decision to make.

The eval workflow

Prompt change → Run evals → Check score delta → Ship if delta > threshold → Done

Your threshold depends on stakes. A toy project might accept -2%. A production agent handling customer data should require +0% or better. Set this explicitly before you start so you're not making the decision under pressure when your score drops by 4% after a "small" prompt tweak.

Tools worth knowing

• Braintrust — eval runs, logging, model comparison. Has a generous free tier.
• Promptfoo — open-source, runs locally, excellent for red-teaming.
• Langfuse — observability + eval, good if you want tracing alongside scores.
• DIY script — for simple agents, a 50-line TypeScript script is often enough. Don't over-engineer.

The uncomfortable truth

Most agentic failures are prompt failures, not model failures. You don't need a new model. You need to know what your current prompt breaks on and fix it.

Evals tell you that. Vibes don't.

Build the eval harness before you add the next feature. Future you will be grateful.