Nash AI

Challenge

Automated pull request review tends to fail in two ways. The comment lands on a line that isn’t part of the diff, so GitHub rejects it or it floats somewhere useless. Or the review arrives as a pile of low-confidence nitpicks that people learn to ignore.

Nash AI is a GitHub App built around those two failures: a review should be line-accurate and short enough to read, and the operator should be able to cap its cost and switch it off.

Approach

A pull request webhook is signature-verified and turned into a queued job. The API returns quickly, and an ARQ worker on Redis does the expensive work. The worker:

1. Fetches the PR, the diff, and commit context, then loads the repo’s .codereview.yml. That config is cached in Redis by (owner, repo, sha), so one run uses one snapshot of the rules.
2. Packs context in layers (project profile, then repo rules, then the review hunks and surrounding lines). Large PRs go through a two-stage chunked pass: bounded chunks first, then a synthesis pass for cross-chunk risks.
3. Runs a tool-using agent loop that can fetch files, search the codebase, and read file history, then finalizes into schema-validated findings.
4. Validates every finding against the fetched file contents and the set of lines GitHub will accept comments on. Findings that fail are dropped with a reason code such as line_not_in_diff or syntax_invalid_suggestion.
5. Filters by confidence, severity, category, and ignore paths, runs an editor pass to remove duplicates and weak claims, and posts one inline review.

Two decisions mattered most. Validation sits between the model and GitHub, so the model’s line numbers are never trusted on their own. And model choice is configuration: Anthropic, OpenAI, and Gemini are selectable per repo, with role-based routing for the primary reviewer, editor, challenger, and tie-break.

The same review engine also runs as an MCP server with eight tools, so another agent can drive a repository review and do its own reasoning. The dashboard is a Next.js app over the same API.

Role

Solo. I designed the pipeline, wrote the backend and the dashboard, set up the infrastructure, and wrote the evals and CI gates.

Tradeoffs

• Queue instead of inline review. The webhook returns fast and the worker can be sized separately, at the cost of Redis, a second deployed service, and job-state transitions to reason about.
• Precision over recall. The default confidence threshold is 0.85 and the editor pass removes weak claims, so Nash AI will stay quiet on some real issues. A review nobody reads catches nothing either.
• max_mode costs more. It adds a challenger model and a conditional tie-break, which means more model calls per review. It is opt-in per repository.
• Guardrails over autonomy. Per-installation hourly review limits, a daily token budget, and an ENABLE_REVIEWS kill switch exist so a bad prompt or a runaway repo is a config change, not an incident.
• A known gap. The LLM client tests are maintained separately and sit outside the CI coverage denominator until the baseline is recalibrated. The 60% floor covers the agent, telemetry, and API packages only.

Lessons

• Treat model output as untrusted input. Most of the quality work was validation, not prompting.
• Evals have to exist before the second prompt change. Without a golden set, “better” is a feeling.
• Observability earns its place early: trace metadata on every review (repo, PR, model, prompt version, run id) made false positives debuggable instead of mysterious.