TL;DR: We wanted PR reviews like the big commercial bots, but with control over cost and where our code goes. We tried Cursor cloud agents, then per-repo GitHub Actions, compared open tools, tried Modal as a sandbox that wakes up on demand, and landed on Dobby: an org-wide GitHub App on Google Cloud Run, with a separate recipes repo for prompts.
This is the long version of that journey. Not a polished vendor pitch. More like the notes we wish we had when we started: what we tried, what we learned, and what we would do again.
What we were trying to solve
Commercial PR review tools are useful. They also raise three questions most teams ask eventually:
Where does our code go? Diffs (and sometimes full files) leave your company and go to someone else's stack.
Who decides how much we spend? Auto-review on every push can quietly burn a lot of model usage.
Who owns the behavior? Changing tone, focus areas, or “what is critical” often means waiting on a vendor.
We wanted something closer to a CodeRabbit-style experience, but with a lot more control on pricing and data.
For v1, the bar was intentionally small:
Review pull requests in a useful way.
Trigger from GitHub (when a PR opens, and when someone mentions the bot).
Post one clear comment people can skim.
Keep model choice, prompts, and secrets under our roof.
Later we could grow into deeper reviews, custom commands, Slack, and more. v1 did not need to be the full product. It needed to be a solid pipe we could extend.
Question you will probably ask: Do we need inline comments on day one?
We said no. Inline comments (notes on specific lines in the Files tab) are great UX. They are also harder. We started with one summary comment. That got us shipping.
Common pitfall: Copying the full commercial feature list into v1. You will spend months on polish and still not have org-wide triggers working. Ship the pipe first.
Step 1: What does “good enough” look like in our heads?
Before tools, we sketched the ideal flow:
Someone opens a PR (or mentions the bot)
|
v
A short-lived sandbox starts
|
v
An agent looks at the PR change
|
v
A review comment is posted
|
v
The sandbox stops (we stop paying for idle time)
In plain words: we did not want a big server sitting on all day. We wanted something that wakes up for the job, does the review, then goes away.
That mental model guided every later choice.
Question you will probably ask: Is this the same as “just run a script in CI?”
Almost. CI (continuous integration: automated checks on GitHub) can do this. The difference is whether you want that script in every repo, or one shared service for the whole organization.
Step 2: We already had Cursor. We used it as a lab.
We started with Cursor cloud agents: hosted AI agents that can work on a coding task for you.
What went well
Almost no infrastructure.
Fast way to test: “Does this review style even help our team?”
Easy to tweak prompts and see results.
What did not scale
Hard to run the same thing automatically for every PR across many repos.
You stay tied to that product’s runtime and billing.
It is a great lab. It is a weak factory for an org-wide bot.
Decision: Use Cursor to learn what a good review looks like. Do not make Cursor the production reviewer for the whole org.
Common pitfall: Falling in love with the demo quality of a hosted agent, then discovering you cannot wire it cleanly into GitHub App webhooks, cooldowns, and org policy.
Step 3: First real v1 on GitHub Actions (one repo at a time)
Next we built a workflow that lives inside a repository. When someone asks for a review (or a workflow runs), GitHub starts a temporary machine, our scripts run, and a comment is posted.
The brain of the review was Hermes: an open-source agent you run like a coding assistant in a terminal. It can look at files and run simple commands, then write a review. Models came through OpenRouter (one API that can talk to many model providers).
Repo with workflow installed
|
v
GitHub Actions runner starts
|
v
Collect PR title, body, diff
|
v
Hermes reviews (via OpenRouter)
|
v
Post comment on the PR
Why this was a good first production step
Familiar security model: secrets stay in GitHub for that repo.
You only pay (and only run) on repos that opt in.
No new cloud account required for compute.
What we learned the hard way
A run could take 4 to 5 minutes. Most of that was waiting on the model, not slow CPUs. Buying a “faster runner” barely helped.
The expensive line is usually tokens (model usage), not the machine minutes.
Installing the same workflow in many repos gets old fast.
Decision: Actions is fine for a few repos and for learning. It is painful as the only way to cover a whole organization.
Question you will probably ask: Should every push auto-review?
Be careful. Auto on every push is convenient and can become the main cost driver. Mentions, cooldowns, and “skip docs-only changes” matter more than people think.
Common pitfall: Optimizing CI minutes while the real bill is the model API.
Step 4: Looking around. Hawk, OpenRabbit, RabbitAI
Before inventing everything, we compared open tools in the same space. Short version:
| Tool |
What it optimizes for |
How “agent-like” it feels |
| Hawk |
Auto review, inline comments, product UI |
Batches files into prompts. Feels automatic. The model does not freely explore with tools. |
| OpenRabbit |
Run on your own Actions, keep it simple and cheap |
Straight pipeline: get files, ask model, post comments. |
| RabbitAI |
Graph of code impact + memory + retrieval |
Fixed multi-step graph (LangGraph-style). Strong structure, less free exploration. |
| Where we headed (Dobby) |
Control, consent, one clear comment, later extensibility |
Tool-using agent (Hermes) behind a strict outer gate (who can trigger, when, on which repos). |
What they do better than a from-scratch bot
What we did not want to copy blindly
Spend on every push with no consent.
Behavior locked inside a dashboard we do not own.
Hard-to-audit runs (“it commented and vanished”).
Decision: Steal good ideas (structure, severity, optional lenses). Keep ownership of triggers, prompts, and secrets.
Question you will probably ask: Should we just self-host one of these?
Maybe, if install speed matters more than control. We wanted a bot we could reshape for our org without fighting someone else’s product assumptions.
Step 5: Diff only, or clone the whole repo?
This question comes up in every design review.
| Approach |
Upside |
Downside |
| Mostly the PR diff (plus maybe the changed files) |
Faster, cheaper, less code sitting in the sandbox |
Can miss how the change fits the wider system |
| Full repo on disk |
Better for “does this break everything?” |
Slow on big repos, more data in the sandbox, longer billed time |
Decision for v1: Diff-first. Enough context to be useful. Full-repo understanding can be a deep mode later, not the default.
Question you will probably ask: Will diff-only reviews feel dumb?
Sometimes, on cross-cutting refactors. That is why we later added a deeper mention mode. Default stays concise so people actually read it.
Common pitfall: Cloning giant monorepos on every review “just in case.” You pay in time, disk, and tokens before you know if the extra context helps.
Step 6: Our own agent. Hermes, or a framework like LangChain?
Once we left “Cursor as the factory,” we needed an agent we could pin and extend.
Options we weighed:
Hermes: feels like a coding agent in a terminal. Good when the model should open files and poke around.
LangChain / LangGraph / similar: good when you want a fixed recipe of steps (fetch, analyze, summarize) with clear stages.
Decision: Hermes for the review worker. Keep the outer system boring: receive event, check policy, start job, post comment. That outer shell is what we extend for new triggers later.
Question you will probably ask: Is a free-roaming agent dangerous?
Yes, if you give it the whole company and no rules. We keep a small workspace, treat PR text as untrusted, and limit what the job is allowed to do. The outer policy layer matters as much as the model.
Common pitfall: Putting all intelligence in one giant prompt with no tools, then wondering why hard bugs are missed. Or the opposite: unbounded tools with no policy, and surprise spend.
Step 7: Looking for reusable sandboxes. Why Modal showed up.
Remember the ideal picture: wake up, review, stop.
Modal is a platform that runs your code in containers that start when called. That matched our mental model well. At the time we did not have a clear playbook for the same idea on GCP or AWS, so Modal was the natural experiment.
GitHub event
-> Modal webhook function (doorbell)
-> starts a review function (kitchen)
-> Hermes runs
-> comment posted
-> containers go away
What we liked
Fits “sandbox on demand.”
Easy to attach secrets to a function without putting them in git.
Fast path from idea to a public webhook URL.
Free plan reality check (high level)
You get a monthly pool of free compute credit, not infinite free forever.
Many CPU jobs can run at once.
There is no magic “N free reviews” number. You burn credit while the container is alive, including time waiting on the model.
Model API cost is separate. Free compute does not mean free tokens.
Decision: Modal was a strong v1 sandbox for a standalone bot. Worth learning even if you later move.
Question you will probably ask: Can the same sandbox run browser tests later?
Often yes. Platforms like Modal can run Playwright-style checks in a container. That is a later chapter, not a v1 requirement.
Common pitfall: Treating free compute credits as free product cost. The model bill still arrives.
Step 8: Secrets, simply
Wherever you run, you will ask: where do the keys live?
| Setup |
Where secrets usually live |
What to remember |
| GitHub Actions |
Repo or org secrets in GitHub |
Familiar. Per-repo install can drift. |
| Modal |
Secret bundles on Modal, injected when the function runs |
App secrets (GitHub App, model API) are not the same thing as “permission to deploy.” Deploy access is a separate login/token for the team. |
| GCP Cloud Run |
Google Secret Manager, mapped into the service |
The service account that runs the bot needs permission to read those secrets. Easy to forget. |
We kept real secrets out of git. Locally, a private .env file is fine for developers. Production reads from the platform’s secret store.
Common pitfall: One person creates production secrets from their laptop, nobody else can redeploy, and the team discovers this during an outage.
Step 9: Org-wide coverage. Stop installing workflows everywhere.
Once a few repos worked, the next pain was obvious: we did not want to configure every repository by hand.
What we needed:
GitHub App installed on the organization
|
v
One webhook URL for the whole org
|
v
Policy: auto review? mention? cooldown? skip bots?
|
v
Start a review job
|
v
Comment as the App
That is how Dobby took shape: a standalone org-wide bot. Same idea as the sandbox sketch, but one doorbell for many repos.
Decision: GitHub App + shared worker beats copying workflows into every repo.
Question you will probably ask: Should the webhook itself run the AI?
No. GitHub expects a quick response. If the model takes minutes, the webhook should only accept the event and queue work. Run Hermes in a separate worker.
Common pitfall: Doing the full review inside the webhook handler. It times out, retries, and you get duplicate reviews.
Step 10: Modal vs GCP Cloud Run (because our org already uses GCP)
Modal was easy. Our organization already ran workloads on Google Cloud Run (run containers on GCP, scale down when idle). So we compared for the long term.
|
Modal |
GCP Cloud Run |
| Speed to first demo |
Excellent |
Slower. More setup. |
| Day-to-day fit for us |
Great experiment platform |
Better long-term home, because the org already lives on GCP |
| Cost for a review-only bot |
Fine early on |
Usually cheaper and more predictable for us over time |
| Secrets |
Modal’s secret store |
GCP Secret Manager |
| Shape we ended on |
One app with doorbell + worker functions |
Two services: webhook and worker, with a task queue between them |
Cold start, in human terms
When nothing has run for a while, the first request can be slow while the platform starts a fresh container. That is a cold start. Keeping one instance warm avoids that, but you pay for idle time.
Our approach: Keep the webhook light so GitHub is happy. Let the worker take longer if it needs to. People care more that the bot eventually comments than that the model starts in 200ms.
Decision: Move Dobby to Cloud Run for production alignment and cost. Keep the same product shape: doorbell, then worker.
Question you will probably ask: Is Cloud Run always cheaper than Modal?
Not as a universal law. For our review-only workload inside an existing GCP org, Cloud Run won on long-term cost and ops familiarity. If your team has no GCP yet, Modal can still be the smarter first production home.
Common pitfall: Moving clouds for ideology before the product works. We moved after the flow was proven.
How the final architecture looks (Dobby on Cloud Run)
GitHub organization
|
GitHub App events
(PR open / mention)
|
v
+----------------------+
| Cloud Run webhook |
| verify + policy |
| (no Hermes here) |
+----------+-----------+
|
queue a task
|
v
+----------------------+
| Cloud Run worker |
| build context |
| run Hermes |
| post PR comment |
+----------------------+
|
v
PR summary comment
Recipes (prompts, command words, exclusions) load from a separate GitHub repo, so behavior can change without rebuilding the bot image every time.
Step 11: Moving from Modal to Cloud Run, without the jargon pile
What actually changed for us:
Split “receive GitHub events” and “run the review” into two services.
Put a queue between them so the webhook stays fast.
Point secrets at GCP’s secret manager instead of Modal’s store.
Point the GitHub App webhook at the new URL.
Redeploy when code changes, same as any container service.
Same story as Modal. Different house.
Common pitfall: Forgetting permissions. The webhook needs permission to create tasks. The worker needs permission to read secrets and talk to GitHub. Most “it deploys but nothing happens” bugs are permissions, not model quality.
Step 12: The Hermes image got heavy. We slimmed it down.
Early builds took a long time because the agent install pulled in a lot of optional pieces (things aimed at browser automation we did not need for a PR comment bot).
What it felt like: waiting forever for an image build, then waiting again on every small change.
What we did:
Pin a known Hermes version so builds are repeatable.
Skip parts we do not use for v1 reviews.
Reuse cached image layers so unchanged pieces are not rebuilt every time.
Decision: Treat the worker image like product infrastructure. A fat image slows every deploy and every cold start.
Question you will probably ask: Do we need a browser inside the review bot?
Not for v1 summary comments. Add it when you truly need UI checks as part of review.
Common pitfall: “Install the full agent with every optional tool” because it might be useful someday. Your builds will remind you every day.
Step 13: Once basic review worked, we shaped the product
With comments flowing, the next questions were product questions, not platform questions.
Review shape
We asked for:
Clear severity (critical / high / medium).
Critical and high first, so skimming works.
Honest “no major issues” when that is true.
A short wrap-up, not a wall of obvious restatement.
Concise vs deep
Default reviews were getting noisy: repeating things the author already knows.
So we split modes:
Default / @dobby review: concise.
@dobby review deep: slower, deeper pass.
Shared cooldown per PR so people do not burn two full runs by accident.
Small human feedback
When someone mentions Dobby, we add an eye reaction on that comment. It is a tiny signal: “we heard you.”
Step 14: Recipes. Change behavior without redeploying.
We did not want every prompt tweak to mean a full rebuild and redeploy.
So we added a recipes setup:
A separate GitHub repo holds a manifest (list of commands and prompts).
Words in a mention map to a recipe (review, review deep, help, later more).
Optional model override per recipe.
A list of repos to exclude from auto analysis.
Unknown commands get a help list instead of silence.
@dobby review -> standard recipe
@dobby review deep -> deep recipe
@dobby help -> list available recipes
excluded repo -> short "this repo is excluded" reply
Sketch of the idea:
recipes repo
manifest.yaml
prompts/
review.md
review-deep.md
Dobby worker
loads manifest
picks recipe from the mention
runs Hermes with that prompt
Decision: Prompts and exclusions live next to engineering PRs. The bot binary stays stable.
Question you will probably ask: Can the bot read the recipes repo if the App is installed on the org?
Usually yes, but only if the token you mint can see that repo. A token scoped too narrowly can look like a mysterious “not found” error even when the file is public to your org.
Common pitfall: Keeping the live recipes folder inside the bot repo forever. It drifts. Two sources of truth appear. Prefer one recipes repo as the source of truth, with a small builtin fallback inside the bot for emergencies.
What we would tell a team starting today
If you want something like Dobby, a practical order is:
Prove the review style in a lab (Cursor or a single repo).
Run one repo in Actions until the comment quality is acceptable.
Decide diff-first vs full clone on purpose, not by accident.
Pick an agent style (tool-using vs fixed pipeline) that matches your risk and cost comfort.
Put a doorbell in front (GitHub App) when you outgrow per-repo installs.
Choose a sandbox that fits your org (Modal for speed to learn, Cloud Run if you already live on GCP).
Never run the model inside the webhook.
Move prompts to recipes once people start asking for “just change the tone.”
Design consent early: mentions, cooldowns, exclusions, model tier.
Takeaways
Control over pricing and data is a product requirement, not a later hardening pass.
Cursor is a great lab. GitHub Actions is a great first factory. An org App is what makes it feel automatic.
Open clones teach UX lessons. They do not have to become your architecture.
Diff-first keeps v1 honest. Deep mode can spend more when someone asks.
Modal matches the “wake up and stop” dream. Cloud Run won for us because of org fit and long-term cost.
Most outages in this space are permissions, timeouts, and fat images, not “the model is dumb.”
Recipes turn a one-off bot into something the team can extend without waiting on a redeploy.
Where Dobby stands now
Dobby is our org-wide PR review bot:
GitHub App for the organization.
Webhook on Cloud Run for policy and queuing.
Worker on Cloud Run that runs Hermes and posts one structured comment.
Recipes in a separate repo for prompts, command words, optional models, and exclusions.
v1 is a simple PR reviewer. The same shape is ready for deeper modes and more advanced use cases when we need them.
What we are thinking of adding next
v1 proves the pipe. The next features are the ones people feel day to day when they open a PR or ask Dobby for help.
Inline comments: findings on the exact lines in the Files tab, not only one summary comment. This is what most people expect from a commercial review bot.
Apply-able suggestions: where a fix is clear, post a GitHub suggested change the author can accept in one click.
Slack: point Dobby at a PR from Slack and get the same review flow without living only in GitHub comments.
More recipes: commands beyond review and deep review (for example architecture notes, security focus, or a test plan), still edited in the recipes repo without a full redeploy.
Cost on the comment: a short note of rough model spend so the team can see what a review cost, not only that it ran.
Skip noisy PRs: docs-only or generated-file changes can skip the agent so people are not waiting on a review that adds little.
Optional merge gate: a status check teams can require, so a serious finding can block merge when the repo wants that.
Deeper context when asked: for hard PRs, optionally pull more of the repo than the diff, so Dobby can reason about callers and shared code, not only the patch.
Question you will probably ask: Will all of this land at once?
No. We will add them in the order that reduces friction first (inline comments and skip rules), then workflow surfaces (Slack, merge gate), then richer analysis.
Common pitfall: Shipping every “CodeRabbit feature” before the summary comment is trusted. Extend from a review people already read.
