The AI reviewer found a real bug. Its suggested fix would have broken my app.

TL;DR — I put an AI code reviewer on a pull request written by an AI coding agent. On the default setting it found nothing. On the strict setting it found a real vulnerability. And the patch it offered would have quietly broken every negative number in the exported file. I ship small browser tools written by Claude Code, and I am not a good enough reviewer to catch a security bug in code I did not write. That is the awkward kind of gap: the code looks fine, the page works, the tests pass. So I installed CodeRabbit on the repository and gave it something real to read: a CSV export for a pricing calculator. One row per material line, then other costs, total cost, selling price, profit, margin. About sixty lines of vanilla JS. My own checks passed first — a static site audit, plus a headless browser run of the tool, 14 of 14. Round 1: silence No actionable comments were generated in the recent review. That is the default. CodeRabbit ships a review profile called CHILL , tuned not to nag. For a team drowning in review comments that is probably right. For someone who cannot fully audit their own code, silence is the least useful answer available. So I committed a config file: # .coderabbit.yaml reviews : profile : assertive Same commit. Same diff. Same reviewer. Only the setting changed. Round 2: a real bug, checked the hard way The strict pass flagged CSV formula injection (CWE-1236) , and it was right. A spreadsheet treats a cell that begins with = , + , - or @ as a formula. Name a product =1+1 , export it, and the number two appears in the file the other person opens. Pick a nastier formula and it stops being a curiosity. My csvCell() escaped quotes and commas correctly and did nothing at all about this. What surprised me was how it checked. Folded into the comment was a shell command it had actually run against the repo — a ripgrep over every place a product name or unit flows into the exporter — to see whether something upstream already sanitised the value. It did n