The problem#
Coding agents are easy to start and hard to leave running. Left alone they can loop forever, spend more than the issue is worth, push to the wrong branch, or declare success on code that fails its own tests.
The approach#
Labelling an issue @agent fix this (or commenting @agent fix) starts one ephemeral run. The design rule is that the agent's own claim of success is never trusted: only a green full test suite inside the sandbox can open a PR.
- Hard caps: the agent loop stops at 30 turns, 30 minutes or $20 of provider-reported cost by default. If a provider does not report a cost, the cost stays unknown rather than being estimated.
- Verification gate: tests red means a comment on the issue and no PR. Tests green but coverage dropped by more than two points, or coverage could not be measured, opens the PR with a
needs-reviewlabel. - Write guard: the agent's file tools reject paths outside the repo and anything under
.github/workflows/, compared case-insensitively. - Provider-agnostic: the worker talks to an LLM interface with Anthropic, OpenRouter and OpenAI-compatible implementations.
Architecture#
The control plane is TypeScript on AWS: an API Gateway HTTP API in front of a webhook Lambda, an SQS queue, and a dispatcher Lambda that launches the worker. The worker is a Python container run as a one-off Fargate task, so nothing is running when there are no issues. State lives in three DynamoDB tables for runs, budgets and the invited installations, and everything is defined in AWS CDK.
The worker mints its own short-lived GitHub installation token on start and reads the GitHub App key from SSM Parameter Store, so no token is passed in the task environment. Traces go to Langfuse over its HTTP API. A dead-letter consumer comments on the issue when a message fails repeatedly.
Hover or tap any part to trace what it talks to. 14 parts, 17 connections. Swipe sideways to see it all.
- 1A labelled issue or an '@agent fix' comment triggers a webhook; the Lambda verifies the HMAC signature and enqueues a work item on SQS.
- 2The dispatcher claims the run with a conditional write on the delivery id, checks the invite allowlist and reserves a per-repo daily slot.
- 3It calls ECS RunTask with metadata-only overrides; the worker reads the GitHub App key from SSM and mints its own installation token.
- 4The worker clones the repo, creates a run-scoped branch and picks the Python or Node adapter.
- 5The agent loop searches, edits and runs tests through a path-guarded toolset until it calls finish or hits the turn, time or cost cap.
- 6The verification gate runs the full suite and coverage in the sandbox, then opens a PR, opens it with a needs-review label, or only comments.
- 7Secondary path: if the dispatcher keeps failing, SQS moves the message to the dead-letter queue, where a Lambda logs it and comments on the issue.
Running each webhook exactly once#
GitHub can deliver the same webhook more than once, and a duplicate here means paying for a second agent run. The dispatcher derives a run id from the delivery id and claims it with a conditional write before doing anything else. It then checks the allowlist and reserves a slot in a per-repo, per-day budget, which is decremented again if the Fargate launch fails.
UpdateExpression:
"SET prs_used = if_not_exists(prs_used, :zero) + :one, expiresAt = :expiresAt",
ConditionExpression: "attribute_not_exists(prs_used) OR prs_used < :ceiling",The slot is released again if the Fargate launch fails. The budget module also has a post-run dollar reconcile function, though I did not find the worker calling it yet.
What I learned#
Most of the design is about limits and failure modes rather than the agent itself: duplicate deliveries, runaway cost, and what the system does when the tests fail.