Skip to main content

Manual Review

TL;DR
  • Manual review = Human investigation for gray-zone cases where automation is uncertain
  • Good candidates: high-value transactions, VIP customers, ML uncertain, customer appeals
  • Bad candidates: clear fraud (auto-decline), clear legitimate (auto-approve), low-value
  • Set decision-quality and response-time goals from the loss at stake, your staffing and fulfillment deadlines
  • Feed confirmed outcomes back to ML models. Then test whether automation actually improves

Manual Review Without a Dedicated Analyst​

No fraud analyst? Here's how to handle manual review without one:

The 2-minute rule: Start with two minutes per order. Check customer history, address and order pattern. Bigger or messier orders may need longer. Can't verify it? Hold the order and contact the customer: "We need to confirm a couple of details on order [ID] before we ship. Can you reply with [what you need]?" A timer doesn't prove fraud.

The time budget rule: Set the review hours you can spare each week. If case count x review time exceeds that budget, find what fills the queue. Test narrower review rules, improve the data or add help; tightening auto-declines can hide the workload by blocking good customers.

The value threshold rule: Start by comparing review time with the loss you could prevent. For a $100 physical order at 30% margin, product cost is $100 x 70% = $70 before shipping and fees. The $30 margin is what you lose if you block a good sale. Five minutes at $30/hour costs $30 x 5 / 60 = $2.50. Review pays only when its expected benefit beats that cost. Tune the cutoff to your own orders.

When to Upgrade

Manual review stops being practical when case volume x review time exceeds available hours or fulfillment deadlines. That's your upgrade signal. Compare a dedicated reviewer or automated fraud tool with the cost of missed fraud and blocked good orders.

When to Use Manual Review​

Good Candidates for Review​

ScenarioWhy Manual Review
Gray zone scoresML uncertain, needs judgment
High-value transactionsLoss too big for automation error
VIP customersFalse positive cost too high
Complex patternsMultiple signals, needs synthesis
AppealsCustomer disputes automated decision

Poor Candidates for Review​

ScenarioBetter Alternative
Clear fraud signalsAuto-decline
Clear legitimate signalsAuto-approve
Low-value transactionsRisk-accept the loss
High volume attacksAutomated rules

Review Queue Design​

Prioritization​

Priority Score =
(Transaction Value × Risk Score × Time Sensitivity)
÷ Analyst Capacity

Example queue settings, not service standards. Replace the amounts and deadlines with your order values, staffing and shipment cutoffs before using them.

PriorityCriteriaSLA
Critical>$5K, high risk, time-sensitive15 min
High>$1K, high risk OR VIP1 hour
MediumMedium risk, medium value4 hours
LowLow value, marginal signals24 hours

Queue Management​

  1. Response-deadline tracking: Watch for overdue cases
  2. Automatic escalation: An overdue case goes to a named person
  3. Capacity planning: Staff to volume
  4. Skill-based routing: Complex cases to senior

The Review Process​

Investigation Steps​

1. Review automated decision reason
↓
2. Examine transaction/application details
↓
3. Check customer history
↓
4. Query external data (device, email, phone)
↓
5. Look for linked accounts
↓
6. Make decision
↓
7. Document rationale

Key Data Points​

CategoryWhat to Check
IdentityName, address, SSN verification
DeviceFingerprint, reputation, velocity
BehaviorPattern vs. history
NetworkLinks to other accounts (see synthetic identity)
ExternalEmail age, phone history, bureau

Decision Framework​

EvidenceDecision
Clear fraud (Tier 1 indicators)Decline, flag account
Supporting signals (multiple Tier 2)Investigate or step up; a signal count alone doesn't prove fraud
Unclear but riskyChallenge (step-up verification)
Risky but VIPApprove with monitoring
Clear legitimateApprove, and add the customer to your allow list if your rules call for it

Analyst Tools​

Essential Features​

  • One screen: All the data in one view
  • Decision shortcuts: One-click common actions
  • Notes/comments: For handoffs and history
  • Timer: Track review time
  • Feedback loop: Outcome tracking

Nice-to-Have Features​

  • Similar case search: "Show me cases like this"
  • Graph visualization: Network connections
  • Communication tools: Contact customer if needed
  • Quality scoring: Manager review integration

Quality Assurance​

Review Sampling​

Choose sampling coverage from reviewer experience, decision risk and observed errors. New analysts and high-impact decisions deserve more coverage; reduce it only when audited quality supports that choice.

Reviewer or caseHow to sample
New analystStart with a second review of each decision, until audits show it's no longer needed
Experienced analystRandom coverage plus all high-impact or disputed decisions
Recurring errorsExpand coverage and coach before reducing it
Mature processReassess coverage when products, tools or attack patterns change

Quality Metrics​

MetricHow to set the goal
Decision accuracyAudited correct decisions / All audited decisions; choose a goal from the cost of errors and show unknowns
Documentation completenessRecord a reason and the evidence for every decision. Make that the rule
SLA adherenceCases completed within your chosen deadline / Cases due; staff to fulfillment needs
False positive rateConfirmed legitimate fraud blocks / All fraud blocks, using Fraud Metrics
Missed fraudConfirmed fraud approved by review / All reviewed approvals; use mature outcomes

Feedback Loop​

  1. Track outcomes: Was decision correct?
  2. Feed to models: Confirmed outcomes, not just reviewers' calls, train the model
  3. Identify patterns: What do humans catch that ML misses?
  4. Update rules: Encode learnings

Scaling Manual Review​

When Volume Exceeds Capacity​

  1. Raise review threshold: Only highest risk. Decide first whether the orders that drop out get approved or declined, and test it
  2. Auto-decide more: Accept some error
  3. Reduce review scope: Focus on key signals
  4. Add staff: If sustainable
  5. Improve models: Long-term solution

Efficiency Improvements​

What to TryHow to Tell It Worked
Better data presentationMeasure time saved finding the evidence
Keyboard shortcutsMeasure repeat-task time and decision errors
Pre-computed insightsCompare investigation time and missed context
Decision templatesCompare documentation time and audited quality

Next Steps​

Setting up manual review?

  1. Define queue prioritization: Critical vs. low priority
  2. Design the review process: Step-by-step workflow
  3. Set quality targets: SLA and accuracy goals

Improving review efficiency?

  1. Check efficiency improvements: Quick wins
  2. Build better analyst tools: All the data in one view
  3. Implement feedback loop: Train the model on confirmed outcomes

Scaling beyond capacity?

  1. Raise review threshold: Only highest risk
  2. Improve models: Long-term solution
  3. Consider vendors: Outsource review

Something wrong on this page? Email matt at paymentsandrisk.com and say what needs fixing.