Manual Review
- Manual review = Human investigation for gray-zone cases where automation is uncertain
- Good candidates: high-value transactions, VIP customers, ML uncertain, customer appeals
- Bad candidates: clear fraud (auto-decline), clear legitimate (auto-approve), low-value
- Set decision-quality and response-time goals from the loss at stake, your staffing and fulfillment deadlines
- Feed decisions back to ML models to improve automation over time
Human investigation for complex fraud decisions.
Manual Review Without a Dedicated Analyst
Most SMBs don't have a fraud analyst. If that's you, here's how to handle manual review without one:
The 2-minute rule: Start with two minutes per order. Check customer history, address and order pattern. Bigger or messier orders may need longer. Can't verify it? Follow your documented escalation or step-up process. A timer doesn't prove fraud.
The time budget rule: Set the review hours you can spare each week. If case count x review time exceeds that budget, find what fills the queue. Test narrower review rules, improve the data or add help; tightening auto-declines can hide the workload by blocking good customers.
The value threshold rule: Start by comparing review time with the loss you could prevent. For a $100 physical order at 30% margin, product cost is $100 x 70% = $70 before shipping and fees. The $30 margin is what you lose if you block a good sale. Five minutes at $30/hour costs $30 x 5 / 60 = $2.50. Review pays only when its expected benefit beats that cost. Tune the cutoff to your own orders.
Manual review stops being practical when case volume x review time exceeds available hours or fulfillment deadlines. That's your upgrade signal. Compare a dedicated reviewer or automated fraud tool with the cost of missed fraud and blocked good orders.
When to Use Manual Review
Good Candidates for Review
| Scenario | Why Manual Review |
|---|---|
| Gray zone scores | ML uncertain, needs judgment |
| High-value transactions | Loss too big for automation error |
| VIP customers | False positive cost too high |
| Complex patterns | Multiple signals, needs synthesis |
| Appeals | Customer disputes automated decision |
Poor Candidates for Review
| Scenario | Better Alternative |
|---|---|
| Clear fraud signals | Auto-decline |
| Clear legitimate signals | Auto-approve |
| Low-value transactions | Risk-accept the loss |
| High volume attacks | Automated rules |
Review Queue Design
Prioritization
Priority Score =
(Transaction Value × Risk Score × Time Sensitivity)
÷ Analyst Capacity
Example queue settings, not service standards. Replace the amounts and deadlines with your order values, staffing and shipment cutoffs before using them.
| Priority | Criteria | SLA |
|---|---|---|
| Critical | >$5K, high risk, time-sensitive | 15 min |
| High | >$1K, high risk OR VIP | 1 hour |
| Medium | Medium risk, medium value | 4 hours |
| Low | Low value, marginal signals | 24 hours |
Queue Management
- Real-time SLA tracking - Monitor aging
- Automatic escalation - If SLA breached
- Capacity planning - Staff to volume
- Skill-based routing - Complex cases to senior
The Review Process
Investigation Steps
1. Review automated decision reason
↓
2. Examine transaction/application details
↓
3. Check customer history
↓
4. Query external data (device, email, phone)
↓
5. Look for linked accounts
↓
6. Make decision
↓
7. Document rationale
Key Data Points
| Category | What to Check |
|---|---|
| Identity | Name, address, SSN verification |
| Device | Fingerprint, reputation, velocity |
| Behavior | Pattern vs. history |
| Network | Links to other accounts (see synthetic identity) |
| External | Email age, phone history, bureau |
Decision Framework
| Evidence | Decision |
|---|---|
| Clear fraud (Tier 1 indicators) | Decline, flag account |
| Supporting signals (multiple Tier 2) | Investigate or step up; a signal count alone doesn't prove fraud |
| Unclear but risky | Challenge (step-up verification) |
| Risky but VIP | Approve with monitoring |
| Clear legitimate | Approve, whitelist signals |
Analyst Tools
Essential Features
- Single pane of glass - All data in one view
- Decision shortcuts - One-click common actions
- Notes/comments - For handoffs and history
- Timer - Track review time
- Feedback loop - Outcome tracking
Nice-to-Have Features
- Similar case search - "Show me cases like this"
- Graph visualization - Network connections
- Communication tools - Contact customer if needed
- Quality scoring - Manager review integration
Quality Assurance
Review Sampling
Choose sampling coverage from reviewer experience, decision risk and observed errors. New analysts and high-impact decisions deserve more coverage; reduce it only when audited quality supports that choice.
| Reviewer or case | How to sample |
|---|---|
| New analyst | Start with a second review of each decision until quality is demonstrated |
| Experienced analyst | Random coverage plus all high-impact or disputed decisions |
| Recurring errors | Expand coverage and coach before reducing it |
| Mature process | Reassess coverage when products, tools or attack patterns change |
Quality Metrics
| Metric | How to set the goal |
|---|---|
| Decision accuracy | Audited correct decisions / All audited decisions; choose a goal from the cost of errors and show unknowns |
| Documentation completeness | Record a reason and required evidence for every decision as your operating policy |
| SLA adherence | Cases completed within your chosen deadline / Cases due; staff to fulfillment needs |
| False positive rate | Confirmed legitimate fraud blocks / All fraud blocks, using Fraud Metrics |
| Missed fraud | Confirmed fraud approved by review / All reviewed approvals; use mature outcomes |
Feedback Loop
- Track outcomes - Was decision correct?
- Feed to models - Human decisions train ML
- Identify patterns - What do humans catch that ML misses?
- Update rules - Encode learnings
Scaling Manual Review
When Volume Exceeds Capacity
- Raise review threshold - Only highest risk
- Auto-decide more - Accept some error
- Reduce review scope - Focus on key signals
- Add staff - If sustainable
- Improve models - Long-term solution
Efficiency Improvements
| Initiative | Impact |
|---|---|
| Better data presentation | Measure time saved finding the evidence |
| Keyboard shortcuts | Measure repeat-task time and decision errors |
| Pre-computed insights | Compare investigation time and missed context |
| Decision templates | Compare documentation time and audited quality |
Next Steps
Setting up manual review?
- Define queue prioritization - Critical vs. low priority
- Design the review process - Step-by-step workflow
- Set quality targets - SLA and accuracy goals
Improving review efficiency?
- Check efficiency improvements - Quick wins
- Build better analyst tools - Single pane of glass
- Implement feedback loop - Train ML from decisions
Scaling beyond capacity?
- Raise review threshold - Only highest risk
- Improve models - Long-term solution
- Consider vendors - Outsource review
Related Topics
- Evidence Framework - Tier 1/Tier 2 indicators
- Risk Scoring - Automated scoring
- Data Enrichment - Email age, phone history, IP signals for reviewers
- Device Fingerprinting - Device intelligence
- Behavioral Analytics - User behavior patterns
- Identity Verification - Document and biometric checks
- Fraud Metrics - Measuring performance
- Chargeback Representment - Fighting disputes
- Velocity Rules - Pattern detection
- Rules vs. ML - Detection approaches
- Friendly Fraud - First-party abuse cases