Manual Review
- Manual review = Human investigation for gray-zone cases where automation is uncertain
- Good candidates: high-value transactions, VIP customers, ML uncertain, customer appeals
- Bad candidates: clear fraud (auto-decline), clear legitimate (auto-approve), low-value
- Set decision-quality and response-time goals from the loss at stake, your staffing and fulfillment deadlines
- Feed confirmed outcomes back to ML models. Then test whether automation actually improves
Manual Review Without a Dedicated Analyst
No fraud analyst? Here's how to handle manual review without one:
The 2-minute rule: Start with two minutes per order. Check customer history, address and order pattern. Bigger or messier orders may need longer. Can't verify it? Hold the order and contact the customer: "We need to confirm a couple of details on order [ID] before we ship. Can you reply with [what you need]?" A timer doesn't prove fraud.
The time budget rule: Set the review hours you can spare each week. If case count x review time exceeds that budget, find what fills the queue. Test narrower review rules, improve the data or add help; tightening auto-declines can hide the workload by blocking good customers.
The value threshold rule: Start by comparing review time with the loss you could prevent. For a $100 physical order at 30% margin, product cost is $100 x 70% = $70 before shipping and fees. The $30 margin is what you lose if you block a good sale. Five minutes at $30/hour costs $30 x 5 / 60 = $2.50. Review pays only when its expected benefit beats that cost. Tune the cutoff to your own orders.
Manual review stops being practical when case volume x review time exceeds available hours or fulfillment deadlines. That's your upgrade signal. Compare a dedicated reviewer or automated fraud tool with the cost of missed fraud and blocked good orders.
When to Use Manual Review
Good Candidates for Review
| Scenario | Why Manual Review |
|---|---|
| Gray zone scores | ML uncertain, needs judgment |
| High-value transactions | Loss too big for automation error |
| VIP customers | False positive cost too high |
| Complex patterns | Multiple signals, needs synthesis |
| Appeals | Customer disputes automated decision |
Poor Candidates for Review
| Scenario | Better Alternative |
|---|---|
| Clear fraud signals | Auto-decline |
| Clear legitimate signals | Auto-approve |
| Low-value transactions | Risk-accept the loss |
| High volume attacks | Automated rules |
Review Queue Design
Prioritization
Priority Score =
(Transaction Value × Risk Score × Time Sensitivity)
÷ Analyst Capacity
Example queue settings, not service standards. Replace the amounts and deadlines with your order values, staffing and shipment cutoffs before using them.
| Priority | Criteria | SLA |
|---|---|---|
| Critical | >$5K, high risk, time-sensitive | 15 min |
| High | >$1K, high risk OR VIP | 1 hour |
| Medium | Medium risk, medium value | 4 hours |
| Low | Low value, marginal signals | 24 hours |
Queue Management
- Response-deadline tracking: Watch for overdue cases
- Automatic escalation: An overdue case goes to a named person
- Capacity planning: Staff to volume
- Skill-based routing: Complex cases to senior
The Review Process
Investigation Steps
1. Review automated decision reason
↓
2. Examine transaction/application details
↓
3. Check customer history
↓
4. Query external data (device, email, phone)
↓
5. Look for linked accounts
↓
6. Make decision
↓
7. Document rationale
Key Data Points
| Category | What to Check |
|---|---|
| Identity | Name, address, SSN verification |
| Device | Fingerprint, reputation, velocity |
| Behavior | Pattern vs. history |
| Network | Links to other accounts (see synthetic identity) |
| External | Email age, phone history, bureau |
Decision Framework
| Evidence | Decision |
|---|---|
| Clear fraud (Tier 1 indicators) | Decline, flag account |
| Supporting signals (multiple Tier 2) | Investigate or step up; a signal count alone doesn't prove fraud |
| Unclear but risky | Challenge (step-up verification) |
| Risky but VIP | Approve with monitoring |
| Clear legitimate | Approve, and add the customer to your allow list if your rules call for it |
Analyst Tools
Essential Features
- One screen: All the data in one view
- Decision shortcuts: One-click common actions
- Notes/comments: For handoffs and history
- Timer: Track review time
- Feedback loop: Outcome tracking
Nice-to-Have Features
- Similar case search: "Show me cases like this"
- Graph visualization: Network connections
- Communication tools: Contact customer if needed
- Quality scoring: Manager review integration
Quality Assurance
Review Sampling
Choose sampling coverage from reviewer experience, decision risk and observed errors. New analysts and high-impact decisions deserve more coverage; reduce it only when audited quality supports that choice.
| Reviewer or case | How to sample |
|---|---|
| New analyst | Start with a second review of each decision, until audits show it's no longer needed |
| Experienced analyst | Random coverage plus all high-impact or disputed decisions |
| Recurring errors | Expand coverage and coach before reducing it |
| Mature process | Reassess coverage when products, tools or attack patterns change |
Quality Metrics
| Metric | How to set the goal |
|---|---|
| Decision accuracy | Audited correct decisions / All audited decisions; choose a goal from the cost of errors and show unknowns |
| Documentation completeness | Record a reason and the evidence for every decision. Make that the rule |
| SLA adherence | Cases completed within your chosen deadline / Cases due; staff to fulfillment needs |
| False positive rate | Confirmed legitimate fraud blocks / All fraud blocks, using Fraud Metrics |
| Missed fraud | Confirmed fraud approved by review / All reviewed approvals; use mature outcomes |
Feedback Loop
- Track outcomes: Was decision correct?
- Feed to models: Confirmed outcomes, not just reviewers' calls, train the model
- Identify patterns: What do humans catch that ML misses?
- Update rules: Encode learnings
Scaling Manual Review
When Volume Exceeds Capacity
- Raise review threshold: Only highest risk. Decide first whether the orders that drop out get approved or declined, and test it
- Auto-decide more: Accept some error
- Reduce review scope: Focus on key signals
- Add staff: If sustainable
- Improve models: Long-term solution
Efficiency Improvements
| What to Try | How to Tell It Worked |
|---|---|
| Better data presentation | Measure time saved finding the evidence |
| Keyboard shortcuts | Measure repeat-task time and decision errors |
| Pre-computed insights | Compare investigation time and missed context |
| Decision templates | Compare documentation time and audited quality |
Next Steps
Setting up manual review?
- Define queue prioritization: Critical vs. low priority
- Design the review process: Step-by-step workflow
- Set quality targets: SLA and accuracy goals
Improving review efficiency?
- Check efficiency improvements: Quick wins
- Build better analyst tools: All the data in one view
- Implement feedback loop: Train the model on confirmed outcomes
Scaling beyond capacity?
- Raise review threshold: Only highest risk
- Improve models: Long-term solution
- Consider vendors: Outsource review
Related Topics
- Prevent Fraud: The fraud hub
- Evidence Framework: Tier 1/Tier 2 indicators
- Risk Scoring: Automated scoring
- Data Enrichment: Email age, phone history, IP signals for reviewers
- Device Fingerprinting: Device intelligence
- Behavioral Analytics: User behavior patterns
- Identity Verification: Document and biometric checks
- Fraud Metrics: Measuring performance
- Representment: fighting a chargeback with evidence
- Velocity Rules: Pattern detection
- Rules vs. ML: Detection approaches
- Friendly Fraud: First-party abuse cases