A Clean Review Doesn't Mean It Works: CodeRabbit Now Runs Your App and Clicks Through It
TL;DR
On November 18, CodeRabbit — the company best known for AI code review — launched CodeRabbit Verify, a product that tests whether your application actually works after a change. It's in public beta and free during the beta.
The problem it targets fits in one sentence: code review reads code. It can't tell you that the checkout button is broken.
Verify boots your app inside an isolated sandbox, drives a real browser down a user path — open the page, fill the form, place the order, pay — and comes back with screenshots, session recordings, and reproduction steps. When it finds a bug, you can hand it to CodeRabbit's own agent for a fix, or export the whole finding as Markdown for whichever coding agent you already use.
The short version:
- This isn't a smarter comment bot. It moves AI from reading diffs to running applications.
- It only tests web apps, and only on cloud-hosted GitHub, GitLab.com, and Bitbucket Cloud. Native mobile apps, self-managed Git servers, and self-hosted CodeRabbit are all out.
- The beta is free, but there's no end date. CodeRabbit also hasn't said what it will cost afterward, or how fast the queue clears on busy repositories.
What it is, and how it differs from ordinary code review
Today's AI code review — including CodeRabbit's own core product — does essentially one thing: read the change, read the surrounding context, offer an opinion. It's good at "this is missing a null check" or "this loop can exit early." It has never started your application.
Which means an entire class of bug is invisible to it: the logic looks fine, the screen shows the right thing, and the behavior is still wrong.
The example in CodeRabbit's launch post is concrete. A $100 order with a SAVE20 coupon should come to $80. Verify ran the flow and found that the confirmation page did display $80 — while the application still recorded a $100 charge internally. The customer can apply the coupon, check out, and see the right number; the ledger is wrong. Any test that only asserts "the coupon was accepted" or "the confirmation page appeared" sails right past it.
That gap used to be covered by a human clicking through, or by a full end-to-end suite. Now agents write the code, the volume of changes has gone up, nobody can click through all of it, and nobody wants to maintain the E2E scripts. That's the slot Verify is aiming at.
Two ways to run it, deliberately kept apart
Verify has two fully independent modes. The documentation is explicit about this, and it's easy to conflate them:
1. Pull request runs. Verify reads the PR's commits, the relevant source, existing tests, and documentation, writes test scenarios for the behavior those changes affect, and runs them against the PR's code. Results come back as a single comment that stays updated.
Three ways to start one: post a new PR comment reading @coderabbitai verify; tick the checkbox in GitHub's walkthrough comment; or turn on automatic runs so every new PR gets tested.
2. User journey runs. Teams save critical flows — sign-up, login, checkout — as reusable journeys. Each one records who is acting, what they're trying to achieve, the steps, and the conditions that must hold. Then you run them against a branch, on demand or on a schedule.
The two modes deliberately don't cross: PR runs never use your saved journeys, and journey runs never post on pull requests. The first answers "did this change break something"; the second answers "are the core flows of the app still alive."
A repository's journey library caps at 50 journeys, each with up to 50 steps. An agent can scan the repository and propose journeys; the team edits and approves before anything enters the library.
How to use it: four steps
Step one, check the prerequisites. The repository needs to be on GitHub Cloud, GitLab.com, or Bitbucket Cloud; the org needs CodeRabbit Cloud; and Verify has to appear in the sidebar. If it isn't there, the org doesn't have access yet.
Step two, prepare an environment. Verify runs inside a CodeRabbit "coding environment" — the same one its Coding Agent uses. You write two scripts: a setup script (install dependencies, build; it runs once at preparation) and a startup script (run migrations, seed test data, bring up frontend and backend, ending with a readiness check). The environment must reach Ready status.
One thing that trips people up: setup runs once during preparation, and every subsequent run restores a snapshot and checks out the code under test. Don't put dependency installs in the startup script, or every run pays for it.
Step three, post the command. As a new comment on the pull request:
```
@coderabbitai verify
```
You can append an instruction — "check that the login form shows an error after a wrong password" — which shifts what Verify prioritizes. Three hard rules apply: the command must start a line and sit outside a fenced code block; each comment starts at most one run; and a reply inside a review thread is ignored outright — no run, no answer. You need write access to the repository.
Step four, read the results. A run produces screenshots, browser session recordings, text evidence, and reproduction steps. From a failure you can hit "Fix with AI" to hand it to CodeRabbit Agent, or package the finding, tested revision, environment, and evidence as Markdown and export it to Cursor, Claude Code, Codex, or whatever agent you're already running.
Who it fits, and who should skip it
Good fit:
- Teams building web apps or services that already have a working startup script and test accounts;
- Small and mid-size teams drowning in agent-generated PRs where nobody can click through each one;
- Anyone who wants coverage on sign-up, login, checkout, and payment without maintaining an E2E suite that goes red every other week.
Not a good fit:
- Native mobile apps — the docs say web apps and services only;
- Teams on GitHub Enterprise Server, self-managed GitLab, Bitbucket Data Center, Azure DevOps, or self-hosted CodeRabbit — none are supported;
- Repositories with no usable startup script and no test data — you'll get a wall of blockers and nothing else.
One scope note worth stating plainly: it tests application behavior, not code quality. Style, performance, and vulnerability scanning live elsewhere in CodeRabbit's lineup (Review, Secure, Deep Scan).
Pitfalls that will actually stop you
Not everyone has permission. Verify has no permissions of its own; it inherits Coding Agent permissions. Admin and Member roles have read and write by default — Billing Admins can't use it. A missing sidebar entry usually means the org isn't enabled.
Screenshots capture whatever is on screen. The docs are blunt about this: environment variables and secrets are redacted from text evidence, but screenshots and recordings show what they show. Keep production connection strings and real user data out of the test environment entirely.
Outcomes are granular — don't read Skipped as a pass. A run can end as Bugs (a defect was found), Blocked (an execution or infrastructure error), Blockers (missing credentials, permissions, integrations, or test data), Failed (scenarios failed but couldn't be classified), Passed (within the tested scope only), Inconclusive, Skipped (no scenarios selected — nothing ran), Superseded, or Generated — not run.
CodeRabbit's own gloss on Passed is that it "does not imply that every part of the application was tested." Skipped isn't a pass; it's an absence of testing.
Missing prerequisites surface as blockers, not bugs. No login credentials, no test account, no third-party integration — Verify won't force its way through and won't log it as a product defect. It reports the limitation separately with a suggested fix. Getting a real run requires closing those gaps first.
Automatic runs skip public repositories. Auto PR runs don't cover public GitHub repos; those need a manual @coderabbitai verify.
Cost and speed
- Free during the public beta: PR runs, full-repo journey runs, and fixes through CodeRabbit Agent. The end date is listed as undetermined.
- Post-beta pricing is unannounced. That matters here more than usual — a browser sandbox costs far more than reading text, and driving a real user path is a heavier operation than posting a comment.
- No performance numbers either. The docs describe a run moving through queuing, environment preparation, scenario generation, testing, and publishing, with new runs queuing behind unfinished ones on the same PR. Queue behavior on high-volume repositories isn't published.
- The real integration cost is the startup script. That's the one genuine barrier: your app has to boot reliably, in someone else's sandbox, on non-production data.
Why this matters
For the past year and a half, the AI coding race has been about who writes faster. That line is now moving downstream: verification after the code is written is becoming the bottleneck. An agent can open ten PRs a day; nobody wants to click through ten test passes a day.
The rest of the industry is moving the same direction. That same week, Google Labs shipped Stitch Loop (AI finds the problem after merge, writes the fix, opens the PR), Qodo talked up policy-driven automatic approvals, and GitHub made Copilot code review adaptive in effort. Everyone is crowding into the "after you commit" segment. CodeRabbit's bet is different in kind: don't compete on comment quality — go run the application.
Whether it lands comes down to two unglamorous things: how hard the startup script is to write, and whether the sandbox is fast and cheap enough. Neither has a public answer yet.
Next step
First, validate small. Pick the repository you're most confident can boot — ideally one with a one-command startup script and an existing test account. Don't start with your most complicated service.
Second, test one path with your first command. Login or checkout, one is enough. Judge whether the generated scenarios are sensible and the evidence is sufficient to act on before rolling it out.
Third, treat the journey library as an asset. The value isn't in a single PR run; it's in hardening sign-up, login, and payment into flows that get re-verified on every change. Once that's running, it's worth far more than one extra code comment.
Sources:
- CodeRabbit blog: https://www.coderabbit.ai/blog/introducing-coderabbit-verify
- CodeRabbit docs (Verify): https://docs.coderabbit.ai/verify
- Official press release (PR TIMES, 2026-11-19): https://prtimes.jp/main/html/rd/p/000000003.000194489.html
- IFAI report (2026-11-18): https://ifai.io/en/news/coderabbit-launches-verify-ai-testing-agent