Playbook: Running Fraud Operations
On this page
TL;DR
- Daily: Clear the review queue (same-day SLA), scan for spikes, process block list additions
- Weekly: Check false positive rate (sample 10 blocked transactions), review fraud losses, check score distribution
- Monthly: Full rule performance review, threshold check, retire stale rules, update velocity limits
- Quarterly: Re-run threshold sweep, review seasonal adjustments, audit allow/block lists
- You're operating five things: rules, model thresholds, the review queue, block/allow lists, and feedback. This playbook ties them together
This is the operational playbook for running fraud detection day-to-day. It ties together fraud rules, ML model feedback, review queues, and block/allow lists into a working system.
For crisis response (active fraud attack), use Survive a Fraud Attack instead. This playbook is for steady-state operations.
Prerequisites
Before following this playbook, you should have:
- A fraud rule set deployed (see Building Fraud Rules)
- Access to your fraud dashboard (processor or vendor)
- A review queue process (see Manual Review)
- Understanding of your fraud metrics baseline
Workflow Overview
The Five Things You Operate
| Component | What It Is | Your Job |
|---|---|---|
| Rules | Your logic for known fraud patterns | Create, tune, retire. Building Fraud Rules |
| Model thresholds | Your sensitivity dial for ML scores | Set, monitor drift, re-calibrate. Risk Scoring |
| Review queue | Your gray zone for ambiguous transactions | Clear daily, maintain SLA. Manual Review |
| Block/allow lists | Your overrides for known good and bad actors | Add on confirmed fraud, prune expired entries. Building Fraud Rules |
| Feedback | Your improvement engine for ML | Verify chargebacks flow, send fraud reports fast. Fraud Model Feedback |
Everything else is a detail of one of these five.
Daily Operations
Time budget: 15-30 minutes for most SMBs.
□ Clear the review queue
- SLA: same business day for all items
- For each review item:
□ Check fraud signals (score, rules triggered, device, IP)
□ Decision: approve, decline, or escalate
□ If fraud confirmed: add email + device to block list
□ If legitimate: approve and consider adding to allow list
- If queue is consistently > 50 items/day, your thresholds
are sending too much to review. Tighten rules or adjust
score thresholds
□ Scan for spikes
- Check: decline rate vs. yesterday (> 2x = investigate)
- Check: fraud alerts or flags vs. yesterday
- Check: review queue volume vs. normal
- If spike detected: follow Survive a Fraud Attack playbook
□ Process block list additions
- Add confirmed fraud identifiers (email, card hash, device)
- Set expiration dates (IP: 30-90 days, email: 12 months)
- Don't batch these. Block confirmed fraud immediately
Automating Daily Tasks
Most of the daily block list work can be automated. Configure your review tool to auto-add identifiers to the block list when a reviewer marks a transaction as fraud. Configure your processor to auto-block on chargeback. Manual daily work should be reviewing the queue and scanning for spikes.
Weekly Operations
Time budget: 30-60 minutes.
□ Review fraud losses vs. blocked transactions
- Pull: total fraud losses this week ($)
- Pull: total blocked transaction value this week ($)
- Compare to last week and 4-week average
- Rising fraud losses with stable blocks = new pattern getting through
- Rising blocks with stable fraud = rules may be too aggressive
□ Check false positive rate
- Sample 10 blocked or declined transactions from the past week
- For each: was it actually fraud or a legitimate customer?
- Calculate: legitimate / total sampled = false positive rate
- Target: under 50% false positive rate
- Over 50%: identify the triggering rules and loosen or add exceptions
□ Look for new patterns
- Review this week's confirmed fraud:
□ Same merchant category or product?
□ Same BIN range or card issuer?
□ Same geography or IP range?
□ Same time of day?
- If you see a cluster: write a rule in shadow mode
(see Building Fraud Rules)
□ Check score distribution for drift
- Pull fraud score histogram for the past week
- Compare to 4-week rolling average
- If median shifted > 5 points: investigate
- See Fraud Model Feedback for drift response
Monthly Operations
Time budget: 1-2 hours.
□ Full rule performance review
- Pull trigger rate per rule (how often each rule fires)
- Pull precision per rule (fraud-in-hits rate)
- Fill in the rule performance table (template below)
- Action by result:
□ Fraud-in-hits > 30%, FP rate low → Keep, consider tightening
□ Fraud-in-hits 10-30% → Add conditions or tighten threshold
□ Fraud-in-hits < 10% → Loosen significantly or retire
□ Rule blocking > 1% of traffic → Probably too aggressive
□ Threshold check
- Is your score distribution shifting? (Check weekly data)
- Has your block rate changed > 20% from baseline?
- Has your fraud-in-approved rate changed?
- If any shifted significantly: re-run threshold sweep
(see Risk Scoring)
□ Update velocity limits for seasonal changes
- Next month: peak season? Holiday? Back-to-school?
- If approaching peak: loosen velocity limits 10-15%
- If leaving peak: tighten back to baseline
- See Velocity Rules for seasonal edge cases
□ Retire stale rules
- Identify rules that haven't triggered in 90 days
- Verify the fraud pattern they targeted is no longer active
- Retire (disable, don't delete. Keep a log in case the pattern returns)
□ Audit block/allow lists
- Remove expired block list entries (should be automatic if TTL is set)
- Review allow list entries: still justified?
- Check for block list entries without expiration dates. Add them
□ Send fraud reports
- Review any confirmed fraud not yet reported to your vendor
- Verify chargeback integration is still flowing
(see Fraud Model Feedback)
Monthly Rule Performance Template
| Rule Name | Triggers | Blocks | Reviews | Fraud Caught | False Positives | FP Rate | Action |
|---|---|---|---|---|---|---|---|
| Example: High value review | 500 | 0 | 500 | 45 | 350 | 70% | Add tenure exception |
Quarterly Operations
Time budget: 2-4 hours.
□ Re-run threshold sweep
- Follow the experiment in Risk Scoring
- Compare optimal thresholds to current settings
- If shifted > 10 points: update thresholds gradually (not all at once)
□ Review seasonal adjustments
- What's coming next quarter? Holidays, promotions, slow season?
- Pre-plan threshold adjustments for known peaks
- Set calendar reminders to loosen/tighten on specific dates
□ Full block/allow list audit
- Export block list. How many entries? How old is the oldest?
- Export allow list. Are any entries stale (customer inactive)?
- Prune lists. Smaller lists are faster and more accurate
□ Evaluate tool effectiveness
- Is your current fraud stack catching enough?
- Are you outgrowing your tools?
- Review: total fraud rate, false positive rate, review rate, net fraud cost
- If fraud rate > 0.1% consistently: consider upgrading tools
(see Fraud Vendors)
When Things Go Wrong
Fraud Spike
Trigger: Fraud losses jump 2x+ over baseline in a single day.
□ Immediate (first 30 minutes):
- Tighten rules: lower amount ceilings, reduce velocity limits
- Increase review queue priority
- Check: is this a known pattern or something new?
□ Within 2 hours:
- Investigate pattern: IPs, devices, BINs, email domains
- Deploy targeted block list entries
- Write a targeted rule in shadow mode (or live if high confidence)
□ Within 24 hours:
- Measure: is the spike contained?
- If not contained: follow Survive a Fraud Attack playbook
- If contained: keep new rules in place, monitor for 1 week
False Positive Spike
Trigger: Customer complaints about declined transactions jump, or your weekly FP sample shows > 60% legitimate.
□ Immediate:
- Identify which rule(s) are causing the most false positives
- Move the worst offender from "decline" to "review" (or shadow mode)
□ Within 24 hours:
- Analyze: what changed? New rule deployed? Score drift? Traffic mix change?
- Check: did someone tighten a threshold without testing?
□ Within 1 week:
- Re-test the rule in shadow mode
- Add exceptions for affected customer segments
- Monitor false positive rate daily until stable
New Attack Type
Trigger: You see fraud getting through that doesn't match any existing rules. Chargebacks from a pattern you haven't seen before.
□ Document the pattern:
- What do the fraudulent transactions have in common?
- IP range? Device type? Product category? Time pattern?
□ Write a rule in shadow mode:
- Target the identified pattern
- Run shadow for 1 week (accelerated from the normal 2 weeks)
- If fraud-in-hits > 30%: promote to live
□ Report to your vendor:
- Send fraud reports for all confirmed fraud in this pattern
- This helps the ML model learn faster
□ Monitor for adaptation:
- Fraudsters adjust. Watch for the same pattern with slight variations
Vendor Outage
Trigger: Your fraud vendor or scoring service is down. Transactions are coming through unscored.
□ Immediate fallback:
- Enable basic processor rules (if not already active):
□ Amount cap: decline orders over $X (5x your AOV)
□ Velocity limit: block > 3 charges per card per hour
□ Country mismatch: review (not decline) international orders
- Send all orders over your review threshold to manual review
□ During outage:
- Monitor block rate and fraud rate manually
- Be conservative: it's better to review more than to approve blind
□ After recovery:
- Re-score any transactions that were processed unscored (if possible)
- Review the manual decisions from the outage period
- Check for fraud that slipped through during the outage
Measuring Success
Track these four metrics to know if your fraud operations are working.
| Metric | Formula | Target | Red Flag |
|---|---|---|---|
| Fraud rate | Fraud losses / Total volume | Under 0.1% | Above 0.3% |
| Block rate | Declined transactions / Total transactions | Watch for drift | Sudden change > 20% |
| False positive rate | Legitimate blocked / Total blocked | Under 50% | Above 70% |
| Review rate | Manual reviews / Total transactions | Under 2% | Above 5% (queue bottleneck) |
Net Fraud Cost Formula
Net fraud cost = Fraud losses
+ (False positives x Average order value x Estimated LTV loss rate)
+ (Review volume x Cost per review)
- (Fraud prevented x Average fraud amount)
This is the number that matters. Optimizing fraud rate alone can drive false positives up. Optimizing false positives alone can let fraud through. Net fraud cost captures the full trade-off.
For the false positive cost calculation, see Risk Scoring: Finding Your Thresholds. For fraud economics, see Fraud Economics.
Next Steps
Setting up fraud operations for the first time?
- Deploy the day-one rule set
- Start with just the daily checklist for the first month
- Add weekly and monthly checks as you build muscle
Already running operations but want to improve?
- Start the weekly false positive sampling if you're not doing it
- Build the monthly rule performance template
- Set up your first threshold sweep
In a crisis right now?
- Survive a Fraud Attack - Active attack response
- Reduce Chargebacks Fast - Chargeback ratio spiking
Related
- Building Fraud Rules - Starter rules, allow/block lists, shadow mode
- Fraud Model Feedback - Feedback loops and model monitoring
- Risk Scoring - Thresholds and cost calculations
- Data Enrichment - IP, email, phone intelligence vendors
- Velocity Rules - Rate-based detection and tuning
- Rules vs. ML - When to use each approach
- Manual Review - Review queue management
- Processor Rules Configuration - Platform-specific setup
- Survive a Fraud Attack - Crisis response playbook
- Fraud Metrics - Measuring detection performance
- Fraud Economics - Cost of fraud decisions
- Fraud Vendors - Third-party tools
- Experimentation - Testing changes safely