How we build Relevan with coding agents

More than half the commits on Relevan’s main branch carry a Claude co-author line. What makes that work is the repo around the agent: what it reads first, what stops it, and who checks its work.

Here’s what’s in ours, and the patterns you can copy.

One flow from idea to merged PR

We use Claude Code, with a copy of Matt Pocock’s engineering skills  in .claude/skills, pinned to one upstream commit. Our CLAUDE.md gives the agent one main flow:

It starts with /grill-with-docs. The agent interviews me about the idea, one question at a time, and writes each decision into the glossary and the plan. Our learning loop plan  quotes my answers word for word, like “staff queue is not realistic,” so a later session can’t reopen a settled call.

/to-spec and /to-tickets turn the plan into GitHub issues, each a thin vertical slice that names the tickets blocking it. /implement takes one ticket and works test-first at seams, the public boundaries we agree to test up front. It ends with /review-diff, where two fresh sub-agents check the change against our coding standards and against the spec.

The last step, /retro, is the one people skip. It fixes the environment the agent ran in. When /to-tickets made tickets with no parent issue from a plan file, one new section in our tracker doc fixed that for every session after.

Pattern 1: a hook that says no, and why

An instruction in CLAUDE.md is a suggestion. A hook is a rule. We create git worktrees only under .claude/worktrees/, and a PreToolUse hook makes sure of it:

When the path is outside the allowed folder, the script returns a deny:

jq -n --arg allowed "$allowed" '{
  hookSpecificOutput: {
    hookEventName: "PreToolUse",
    permissionDecision: "deny",
    permissionDecisionReason: ("Create git worktrees only under " + $allowed
      + "/<name>. Use EnterWorktree, or give git worktree add a path there.")
  }
}'

The reason is the important part. The agent reads it, so write it as an instruction that says what to do instead. Then the agent can fix its next call without stopping to ask you.

Two more hooks format and lint each file right after an edit, and lint every changed package before the agent stops.

Pattern 2: a skill for a judgment call

A hook can’t decide whether a database migration is safe. A skill can walk the agent through it. Our schema-namespace-check skill lists what counts as breaking, such as a dropped column or a new NOT NULL column with no default. A breaking change can go into its own schema namespace, so it can’t block other developers. A hook makes sure the agent loads the skill first:

if (/sql:generate/.test(command) && !/\bSCHEMA_CHECKED=1\b/.test(command)) {
  // deny: "Load the schema-namespace-check skill and classify the change
  // before you generate a migration. Then run the command again with
  // SCHEMA_CHECKED=1 at the start."
}

Pattern 3: a review loop with an exit

A merge to main needs one approving review and every review thread resolved. When a team member comments @claude review, a GitHub Action runs Claude Code as the reviewer. Our review-loop skill drives the cycle: it fixes the findings, resolves their threads, and asks for the next review until Claude approves. It stops after five rounds, or when the same finding comes back after a fix. Then a person decides. Without that exit, two agents can argue forever.

Many agents, one pipeline

We don’t run one agent at a time. We also use Capy , where a Captain is an agent that directs a crew of agent threads, each with its own machine and its own PR. This post is an example. A Captain and three crew threads wrote, rendered, and reviewed it and two sibling posts, and I edited them by hand.

Every PR that is ready for review then goes through the same gates:

1. Capy reviews each push automatically, and the agent that owns the PR triages each finding.
2. CI runs format, lint, and tests. When the PR changes more than docs, it deploys the changed packages to a dev stack for that PR and runs our Playwright end-to-end tests against it.
3. After the merge, main deploys to dev, runs the full end-to-end suite, and only then deploys to prod.

Our customers’ agents get skills too

The agents that use Relevan get the same treatment. Our API serves four skills at /skills/<name>/SKILL.md: setup, ranking signals, mapping, and feedback. Our llms.txt lists each one, and each index gets its own skill doc, generated from its live mapping. Every document links to the others, and a test fails if the installable bundle drifts from the source.

We test that surface the way a customer’s agent meets it. In October, we gave fresh agents only our public docs and API, and had them build and tune a product search index. They found six problems. The worst was a hidden monthly quota that stopped tuning halfway through. Now llms.txt documents the quota, and each search response on a capped plan carries X-Search-Quota-Remaining.

What’s next

These are plans. None of them run today.

• Evals as PR gates. Our open evals, decision-model-eval  and shots-to-success , inform our tuning decisions today. We want them to run on the PRs that change tuning or the agent-facing docs.
• The learning loop. We train per-index ranking models from feedback offline today. Plan 0043 adds scheduled runs, an offline gate for each tuning change, and a test on live traffic, or an expert review on a low-traffic index, before a change goes live.

What we’d tell another team

1. Write every hook denial as an instruction. The agent reads it.
2. Pair a hook with a skill when a rule needs judgment.
3. Give every agent loop a round cap and a way out to a person.
4. Run the retro, and fix the environment the agent works in.

If you’re building an agent that searches, point it at our llms.txt and let the relevan-setup skill build your first index. Try our free tier, or book a call to talk through your setup.