Inside the FPT Cloud Evaluation: How CloudThinker Review Held the First Quality Gate
A merge request lands on Friday afternoon. It adds an endpoint that returns workspace data, the tests pass, and the senior reviewer who knows the tenant model is on leave. The change looks clean line by line. Whether it lets one tenant read another tenant's workspace depends on rules that aren't in the diff.
That's the tension in enterprise code review. Review quality depends on senior engineers, the scarcest people in the organization, while release pressure pushes review depth down exactly when risk is highest. For a provider running a multi-tenant platform, a cross-tenant data-access defect is a compliance incident, not a routine bug.
This is the engineering account of how FPT Cloud evaluated CloudThinker Review over five weeks on production merge requests: the challenge in each phase, the decision that addressed it, and the measured outcome. Roughly 97% precision, full coverage of the backend logic defects in the evaluated set including every security finding, and 68.6% of all verified defects caught at the merge request, before human review or QC testing.
The constraints
Three constraints shaped the evaluation:
- Security and tenant isolation are non-negotiable. The platform serves multiple enterprise tenants, so defects that cross tenant boundaries carry regulatory and contractual consequences.
- Senior review capacity is limited. Every merge request needs experienced review, but senior engineers can't scale linearly with delivery volume, and review depth drops under deadline pressure.
- Quality claims must be measurable. In a compliance-driven environment the review process itself has to be auditable. Adopting an AI reviewer needs quantitative evidence on real workloads, not anecdotes.
So the evaluation was a controlled measurement, not a demo. CloudThinker Review ran on production merge requests, FPT Cloud engineers verified every finding, and coverage was measured against a ground truth built from all defects found by any quality layer: AI review, human review or QC testing.
Why diff-only review breaks on a multi-tenant platform
Most AI reviewers, and most rushed human reviews, judge the change in front of them. That works for style and obvious bugs. It fails on the defects that matter most here, for structural reasons:
Intent isn't in the diff. Whether a query must filter by tenant, what range a quota field accepts, which config target a change should hit: all of that lives in specifications, tickets and people's heads. A reviewer that only sees the diff can't check a change against a rule it has never seen.
Standards live as institutional knowledge. Most engineering standards were never written down. They're enforced by whichever senior engineer happens to review, so coverage varies with the roster.
Review depth scales with people, not with code. More merge requests means thinner review per request. The risky ones get the same skim as the trivial ones.
Quality layers overlap without a map. Without measuring which layer catches what, teams can't tell whether review is catching logic defects or just conventions, and QC ends up absorbing the rest.
Five weeks, one variable: context
01Week 1
Baseline
Low accuracy. The reviewer saw the diff but not the intent.
02Week 2
Specifications
Requirement specs from the codebase and Jira loaded into every review.
03Week 3
Curated rules
Rules generated from code and review history, then curated by seniors.
04Weeks 4 to 5
Measurement
Production merge requests, every finding verified by engineers.
Week 1: baseline and the context gap
The first week set a baseline and surfaced the central challenge. Accuracy was initially low, and the limit was context, not model capability.
An AI reviewer working only on the diff can't see intended behavior: the tenant-isolation rules, the valid value ranges, or the domain conventions the team has built up over years. The baseline showed it. Findings were technically plausible but wrong for the business domain, and defects in logic that needs domain knowledge to judge were missed.
That diagnosis set the plan for the next two weeks. Instead of tuning the model, the team supplied the business context the reviewer lacked.
Week 2: requirement specifications from code and Jira
In the second week the reviewer was connected to the two systems of record where intended behavior is documented:
- Requirement specifications maintained in the codebase, so each review checks the change against documented behavior, not code hygiene alone.
- Jira issues, so each merge request brings its acceptance criteria, value bounds and business rules into the review.
01Context
Systems of record
- Merge requestThe diff and the files around it
- Requirement specsMaintained in the codebase
- Jira issuesAcceptance criteria, value bounds, rules
- Curated rule setEngineering standards, machine-readable
What the change is supposed to do
02Review
CloudThinker
- Requirement checkDoes the change match the spec
- Side effectsWhat else the change touches
- SecurityTenant isolation, secrets, access
- StandardsCurated rules applied every time
First quality gate
03Downstream
Humans and QC
- Human reviewJudgment and conventions
- QC testingRuntime behavior and UI rendering
Work only they can do
This changed the question the reviewer answers, from "is this code well formed?" to "does this change implement the documented requirement, without unintended side effects?" That reframing is what makes static business-logic defects detectable: a chart legend silently re-sorted, a column-width setting silently dropped, a configuration change applied to the wrong target.
Week 3: generated rules, curated by senior engineers
The third week took on FPT Cloud's internal engineering standards, most of which existed as institutional knowledge, not documentation. Instead of asking senior engineers to write a rulebook by hand, the team inverted the process:
- CloudThinker generated candidate rules automatically, from patterns in the codebase and recurring themes in historical review feedback.
- Senior engineers curated the generated rules, keeping the ones that reflected real intent, correcting inaccuracies, and adding the judgment-based standards only they could articulate.
The result is a machine-readable rule set the reviewer loads on every merge request. Senior expertise stops being a per-review bottleneck and becomes a durable asset applied the same way across the organization.
Weeks 4 and 5: methodology and results
With the context integrations in place, the last two weeks measured performance on a representative set of production merge requests. Engineers verified every AI-reported finding, and coverage was computed against the complete set of defects found by any layer.
| Metric | Result | Interpretation |
|---|---|---|
| Precision | ~97% | False positives were rare, so engineers acted on findings instead of triaging them |
| Backend logic coverage | Full | Every defect observable in the code itself was detected, security findings among them |
| Share of all defects | 68.6% | Caught at the merge request, before human review or QC testing |
Two notes on method:
- The figures are preliminary. Some of the evaluated merge requests hadn't completed QC testing at the time of measurement, so coverage on that subset is an upper bound.
- Coverage is reported in two scopes on purpose. The 68.6% figure charges code review with every defect found anywhere, including UI and runtime defects only QC can see. The backend logic figure describes the defect class code review is actually responsible for. You need both for an accurate picture.
How a context-aware review reads one merge request
The example below is illustrative: simplified code written for this post, not FPT Cloud's. It shows how the reviewer handles the defect class that mattered most in the evaluation, a cross-tenant IDOR.
The diff adds a handler that returns a workspace by ID:
@router.get("/workspaces/{workspace_id}")
def get_workspace(workspace_id: str, user: User = Depends(current_user)):
workspace = db.query(Workspace).filter(Workspace.id == workspace_id).first()
if workspace is None:
raise HTTPException(status_code=404)
return workspace
The readings. Line by line, nothing is wrong. Whether it's safe depends on where tenant isolation is enforced, and there are three possibilities:
- Middleware or a query scope enforces it globally. Then this handler inherits the filter and is fine.
- A shared helper enforces it, and handlers are expected to call it. Then this handler skips a required step.
- Nothing enforces it. Then any authenticated user can read any workspace by guessing or enumerating IDs.
The checks. The reviewer reads the files around the diff and the context it was given. The requirement spec says every workspace read is scoped to the caller's tenant. The curated rules say handlers must go through the scoped query helper, because there's no global filter. Sibling handlers in the same module all call it.
The finding. High severity, cross-tenant IDOR: the handler bypasses tenant scoping, so a user in tenant A can read workspace data from tenant B. The comment on the merge request names the spec line, the rule and a sibling handler that does it correctly, and proposes the fix:
workspace = scoped_query(Workspace, tenant_id=user.tenant_id).filter(
Workspace.id == workspace_id
).first()
A diff-only reviewer has no way to reach that finding. It needs the spec and the rule to know the filter was required.
Detection coverage across quality layers
01AI code review
68.6%
Technical, static business logic and every security finding, at the merge request.
02Human review
5.9%
Code-convention findings, already covered by rules generated in week 3.
03QC testing
25.5%
UI and runtime defects that only appear when the code executes.
The code-layer defects the AI reviewer didn't report were all code-convention violations. Each maps directly to rules already generated in week 3 that hadn't yet been loaded into the review context at measurement time. With the curated rule set applied, that gap closes.
AI review and QC cover different ground
The spread of defects across layers showed a clear division of responsibility, not overlapping effort.
Defects found by AI review (code and diff, in scope):
- Technical and code quality: a
NaNpx CSS value, duplicated logic, and a configuration change applied to the wrong target - Static business logic: a silently re-sorted chart legend and a silently dropped column-width setting
- Security, every finding in this category: cross-tenant IDOR vulnerabilities and credentials stored in plaintext
Defects found by QC (runtime execution, outside review scope):
- An application crash when expanding all rows over large datasets
- Layout regressions and distorted gauges after font and padding changes
- Drill-through, pagination and page-load performance issues
The pattern is precise: the AI reviewer's blind spot is runtime behavior and UI rendering, not business logic. Every defect observable in the code itself was detected. The QC-found defects only show up when the code executes and the interface renders. The two layers are complementary by scope, and neither replaces the other.
Severity of verified findings
| Severity | Share of findings |
|---|---|
| Critical | 0% |
| High | 37% |
| Medium | 37% |
| Low | 26% |
A single merge request accounted for nearly half of the High-severity findings, including:
- IDOR vulnerabilities allowing cross-tenant access to workspace data
- A partial failure incorrectly reported as
Succeeded - A silent data-copy failure that left the destination table empty
For a multi-tenant provider these are the most expensive defect classes that can reach production: a tenant-isolation breach, an integrity failure in status reporting, and silent data loss. All were found at the merge request, before human review, QC or deployment.
Why this beats the traditional review model
| Diff-only review | Context-engineered review (this evaluation) | |
|---|---|---|
| What it checks | Is the code well formed | Does the change implement the documented requirement |
| Business logic | Plausible findings, often wrong for the domain | Full backend logic coverage in the evaluated set |
| Security | Depends on who reviews | Every security finding in the evaluated set |
| Engineering standards | In senior engineers' heads | Generated, curated, loaded on every MR |
| Precision | Low at baseline | ~97% |
| Evidence for audit | Anecdotes | Verified findings, two-scope coverage |
The deeper difference is what the reviewer is allowed to know. The team didn't tune the model. Supplying intent (specs, tickets) and standards (curated rules) is what moved the result, because the defects that hurt a multi-tenant platform are defined by rules that never appear in the diff.
Path to production
The evaluation closed with three recommendations for rollout, in order of expected leverage:
- Load the curated rule set into the review context. This recovers the missed convention findings at negligible cost.
- Attach the feature specification to every merge request. This extends coverage from static logic into dynamic behavior and value-bound validation.
- Keep reporting coverage in two scopes. Separate in-scope and blended figures keep an accurate view of code review performance at scale.
Who decides what happens to a finding
Code review is advisory by nature, and it stays under your control. CloudThinker reads the merge request and its context freely and posts findings as review comments. Anything beyond a comment, such as blocking a merge or opening a follow-up issue, goes through Auto Mode, so you decide what runs on its own and what waits for approval.
Getting started
Connect your repository and Jira read-only, then ask from chat:
"Review this merge request against the linked Jira issue and our requirement specs. Flag anything that reads or writes tenant data without the tenant filter, and cite the spec line for each finding."
"Generate candidate review rules from our last six months of review comments. Group them by theme so our senior engineers can curate them."
"For last month's merge requests, show which defects review caught and which ones QC caught, in two scopes."
Conclusion
The deciding factor in enterprise-grade AI code review turned out to be context engineering, not model selection. The week 1 baseline, the specification integration in week 2 and the curated rules in week 3 map directly onto the move from low initial accuracy to roughly 97% precision with full backend logic coverage.
Three conclusions carry beyond this engagement:
- Low initial accuracy is usually a context problem. A reviewer that reads only diffs can't judge business intent. Specifications, issue context and codified standards close the gap.
- Institutional standards can be codified efficiently. Generated rules with senior curation capture engineering judgment without a manual documentation burden.
- Evaluation rigor is a requirement. Verified findings, ground-truth coverage, two-scope reporting and disclosed limitations are what make results usable in a compliance-driven adoption decision.
Where merge requests routinely touch tenant data and senior review capacity is tight, AI review works as the first quality gate. It catches what's observable in the code at the cheapest point in the pipeline, so human reviewers and QC can spend their time on the work that needs them.
Read the FPT Cloud case study for the business view of this engagement, or visit the Review product page and contact our team.