Playbook: Running Fraud Operations
TL;DR
- Daily: Clear the review queue within your fulfillment deadlines, scan for spikes, process block list additions
- Weekly: Check confirmed legitimate orders among fraud blocks, review fraud losses and score distribution; report unknown outcomes
- Monthly: Full rule performance review, threshold check, retire stale rules, update velocity limits
- Quarterly: Re-run threshold sweep, review seasonal adjustments, audit allow/block lists
- You're operating five things: rules, model thresholds, the review queue, block/allow lists, and feedback. This playbook ties them together
This is the operational playbook for running fraud detection day-to-day. It ties together fraud rules, ML model feedback, review queues, and block/allow lists into a working system.
For crisis response (active fraud attack), use Survive a Fraud Attack instead. This playbook is for steady-state operations.
Workflow Overview
The Five Things You Operate
| Component | What It Is | Your Job |
|---|---|---|
| Rules | Your logic for known fraud patterns | Create, tune, retire. Building Fraud Rules |
| Model thresholds | Your sensitivity dial for ML scores | Set, monitor drift, re-calibrate. Risk Scoring |
| Review queue | Your gray zone for ambiguous transactions | Clear daily, maintain SLA. Manual Review |
| Block/allow lists | Your overrides for known good and bad actors | Add on confirmed fraud, prune expired entries. Building Fraud Rules |
| Feedback | Your improvement engine for ML | Verify chargebacks flow, send fraud reports fast. Fraud Model Feedback |
Everything else is a detail of one of these five.
Daily Operations
Starting time budget: Try 15-30 minutes, then adjust to case volume and fulfillment deadlines. The cadence and alert settings below are operating choices, not processor requirements.
□ Clear the review queue
- SLA: same business day for all items
- For each review item:
□ Check fraud signals (score, rules triggered, device, IP)
□ Decision: approve, decline, or escalate
□ If fraud confirmed: add email + device to block list
□ If legitimate: approve and consider adding to allow list
- If case count x review time exceeds available hours, investigate
which rules fill the queue. Test changes or add capacity before
deadlines are missed; don't auto-decline just to clear it
□ Scan for spikes
- Check: decline rate vs. comparable days; investigate outside your planned range
- Check: fraud alerts or flags vs. yesterday
- Check: review queue volume vs. normal
- If spike detected: follow Survive a Fraud Attack playbook
□ Process block list additions
- Add confirmed fraud identifiers (email, card hash, device)
- Set expiration dates (IP: 30-90 days, email: 12 months)
- Don't batch these. Block confirmed fraud immediately
Automating Daily Tasks
Most of the daily block list work can be automated. Configure your review tool to auto-add identifiers to the block list when a reviewer marks a transaction as fraud. Configure your processor to auto-block on chargeback. Manual daily work should be reviewing the queue and scanning for spikes.
Weekly Operations
Time budget: 30-60 minutes.
□ Review fraud losses vs. blocked transactions
- Pull: total fraud losses this week ($)
- Pull: total blocked transaction value this week ($)
- Compare to last week and 4-week average
- Rising fraud losses with stable blocks = new pattern getting through
- Rising blocks with stable fraud = rules may be too aggressive
□ Check false positive rate
- Start with a random sample you can investigate; expand if uncertain
- For each: confirmed fraud, confirmed legitimate, or unknown?
- Calculate: confirmed legitimate / all sampled fraud blocks
- Report sample count and unknown share, using Fraud Metrics definitions
- Compare lost good-order margin with prevented loss before changing rules
- Honor the loss cap and rollback chosen before any live test
□ Look for new patterns
- Review this week's confirmed fraud:
□ Same merchant category or product?
□ Same BIN range or card issuer?
□ Same geography or IP range?
□ Same time of day?
- If you see a cluster: write a rule in shadow mode
(see Building Fraud Rules)
□ Check score distribution for drift
- Pull fraud score histogram for the past week
- Compare to 4-week rolling average
- Investigate shifts outside your chosen range for this score scale and traffic mix
- See Fraud Model Feedback for drift response
Monthly Operations
Time budget: 1-2 hours.
□ Full rule performance review
- Pull trigger rate per rule (how often each rule fires)
- Pull precision per rule (fraud-in-hits rate)
- Fill in the rule performance table (template below)
- Action by result:
□ Prevented loss justifies customer and review cost: keep monitoring
□ Cost or capacity budget breached: pause or roll back as agreed
□ Uncertain outcomes: keep shadowing; don't infer a fraud percentage
□ Block rate outside its planned range: investigate traffic and rule changes
□ Threshold check
- Is your score distribution shifting? (Check weekly data)
- Has your block rate moved outside the range set from comparable traffic?
- Has your fraud-in-approved rate changed?
- If any shifted significantly: re-run threshold sweep
(see Risk Scoring)
□ Update velocity limits for seasonal changes
- Next month: peak season? Holiday? Back-to-school?
- If approaching peak: backtest legitimate purchase bursts and attack traffic
- Test seasonal adjustments with an owner, loss cap, end date and rollback
- See Velocity Rules for seasonal edge cases
□ Retire stale rules
- Identify rules that haven't triggered in 90 days
- Verify the fraud pattern they targeted is no longer active
- Retire (disable, don't delete. Keep a log in case the pattern returns)
□ Audit block/allow lists
- Remove expired block list entries (should be automatic if TTL is set)
- Review allow list entries: still justified?
- Check for block list entries without expiration dates. Add them
□ Send fraud reports
- Review any confirmed fraud not yet reported to your vendor
- Verify chargeback integration is still flowing
(see Fraud Model Feedback)
Monthly Rule Performance Template
Use the Fraud Metrics definitions. This example is a review rule, so it has flagged legitimate orders, not false blocks. All outcomes below are hypothetical and fully labelled.
| Rule Name | Triggers | Blocks | Reviews | Confirmed Fraud in Hits | Confirmed Legitimate in Hits | Legitimate Share of Hits | Action |
|---|---|---|---|---|---|---|---|
| Example: High value review | 500 | 0 | 500 | 150 | 350 | 70% | Test a tenure exception against costs and coverage |
Quarterly Operations
Time budget: 2-4 hours.
□ Re-run threshold sweep
- Follow the experiment in Risk Scoring
- Compare optimal thresholds to current settings
- Test a change when expected cost improves within your budgets, not at a fixed score-point shift
□ Review seasonal adjustments
- What's coming next quarter? Holidays, promotions, slow season?
- Pre-plan threshold adjustments for known peaks
- Set calendar reminders to loosen/tighten on specific dates
□ Full block/allow list audit
- Export block list. How many entries? How old is the oldest?
- Export allow list. Are any entries stale (customer inactive)?
- Prune lists. Smaller lists are faster and more accurate
□ Evaluate tool effectiveness
- Is your current fraud stack catching enough?
- Are you outgrowing your tools?
- Review: total fraud rate, false positive rate, review rate, net fraud cost
- Evaluate a tool for a measured gap; compare avoided loss with fees and customer impact
(see Fraud Vendors)
When Things Go Wrong
Fraud Spike
Trigger: Fraud losses breach the alert or dollar-loss limit you chose for comparable traffic. Check counts too; one case can move a small day's rate.
□ Immediate (first 30 minutes):
- Tighten rules: lower amount ceilings, reduce velocity limits
- Increase review queue priority
- Check: is this a known pattern or something new?
□ Within 2 hours:
- Investigate pattern: IPs, devices, BINs, email domains
- Deploy targeted block list entries
- Write a targeted rule in shadow mode (or live if high confidence)
□ Within 24 hours:
- Measure: is the spike contained?
- If not contained: follow Survive a Fraud Attack playbook
- If contained: keep new rules in place, monitor for 1 week
False Positive Spike
Trigger: Decline complaints rise or sampled legitimate fraud blocks breach your customer-impact budget. Investigate affected orders; no fixed legitimate share proves a rule is wrong.
□ Immediate:
- Identify which rule(s) are causing the most false positives
- Move the worst offender from "decline" to "review" (or shadow mode)
□ Within 24 hours:
- Analyze: what changed? New rule deployed? Score drift? Traffic mix change?
- Check: did someone tighten a threshold without testing?
□ Within 1 week:
- Re-test the rule in shadow mode
- Add exceptions for affected customer segments
- Monitor false positive rate daily until stable
New Attack Type
Trigger: You see fraud getting through that doesn't match any existing rules. Chargebacks from a pattern you haven't seen before.
□ Document the pattern:
- What do the fraudulent transactions have in common?
- IP range? Device type? Product category? Time pattern?
□ Write a rule in shadow mode:
- Target the identified pattern
- Set a sample and outcome window; a week alone doesn't establish fraud results
- Promote only within the pre-agreed loss, customer-impact and capacity limits
□ Report to your vendor:
- Send fraud reports for all confirmed fraud in this pattern
- This helps the ML model learn faster
□ Monitor for adaptation:
- Fraudsters adjust. Watch for the same pattern with slight variations
Vendor Outage
Trigger: Your fraud vendor or scoring service is down. Transactions are coming through unscored.
□ Immediate fallback:
- Enable basic processor rules (if not already active):
□ Amount cap: use the tested fallback amount in your incident plan
□ Velocity limit: use a tested count and window for your normal purchase bursts
□ Country mismatch: review (not decline) international orders
- Send all orders over your review threshold to manual review
□ During outage:
- Monitor block rate and fraud rate manually
- Be conservative: it's better to review more than to approve blind
□ After recovery:
- Re-score any transactions that were processed unscored (if possible)
- Review the manual decisions from the outage period
- Check for fraud that slipped through during the outage
Measuring Success
Track these four metrics to know if your fraud operations are working.
Use the Fraud Metrics definitions, with matching populations and mature outcome windows.
| Metric | Formula | How to set your target | When to investigate |
|---|---|---|---|
| Fraud rate | Confirmed fraud dollars / Transaction dollars | Loss budget and applicable written processor limits | Outside budget or a change in comparable traffic |
| Block rate | Fraud-control blocks / Total attempts | Your customer-impact budget | Outside the range you set from comparable traffic |
| False positive rate | Confirmed legitimate / All fraud blocks | Lost good-order margin vs. fraud prevented | Budget breached; show unknowns separately |
| Review rate | Transactions sent to review / Total attempts | Case count x review time vs. available hours | Backlog threatens fulfillment deadlines |
Net Fraud Cost Formula
Realized fraud-related cost = Confirmed unrecovered fraud losses
+ Estimated contribution lost on legitimate orders blocked
+ Review volume x Cost per review
+ Fraud-tool fees
Keep counts and dollars separate. Estimate lost contribution from legitimate orders you believe would've completed, not all declined revenue. Report estimated fraud prevented separately against a stated counterfactual; don't subtract it again from losses that already exclude blocked fraud.
For the false positive cost calculation, see Risk Scoring: Finding Your Thresholds. For fraud economics, see Fraud Economics.
Next Steps
Setting up fraud operations for the first time?
- Deploy the day-one rule set
- Start with just the daily checklist for the first month
- Add weekly and monthly checks as you build muscle
Already running operations but want to improve?
- Start the weekly false positive sampling if you're not doing it
- Build the monthly rule performance template
- Set up your first threshold sweep
In a crisis right now?
- Survive a Fraud Attack - Active attack response
- Reduce Chargebacks Fast - Chargeback ratio spiking
Related
- Building Fraud Rules - Starter rules, allow/block lists, shadow mode
- Fraud Model Feedback - Feedback loops and model monitoring
- Risk Scoring - Thresholds and cost calculations
- Data Enrichment - IP, email, phone intelligence vendors
- Velocity Rules - Rate-based detection and tuning
- Rules vs. ML - When to use each approach
- Manual Review - Review queue management
- Processor Rules Configuration - Platform-specific setup
- Survive a Fraud Attack - Crisis response playbook
- Fraud Metrics - Measuring detection performance
- Fraud Economics - Cost of fraud decisions
- Fraud Vendors - Third-party tools
- Experimentation - Testing changes safely