Fraud Model Feedback
- Your vendor's ML model improves when you give it outcomes. Chargebacks, fraud reports, false positive flags
- Chargebacks arrive 30-120 days late. Send fraud reports the moment you confirm fraud, so the model gets a faster signal
- Score drift is real. The same score means different things over time. Check your distribution weekly
- Monitor four things: score distribution (weekly), block rate trend (weekly), fraud-in-approved rate (monthly), false positive sample (monthly)
- At minimum, check that chargebacks reach your fraud vendor. That's the baseline. Everything else is optimization
Your fraud vendor's ML model is only as good as the data it learns from. Every score comes from patterns it found in past transactions and their outcomes. Don't feed the outcomes back and it stagnates.
Here's how the loop works, what breaks it, and what to watch. For thresholds and cost math, see Risk Scoring. For rule-based detection, see Building Fraud Rules.
What Your Fraud Score Actually Means
Every fraud vendor gives you a number. It's the model's estimate that a transaction is fraudulent.
| Vendor | Score Range | Higher Means |
|---|---|---|
| Stripe Radar | 0-100 | Higher risk |
| Sift | 0-100 | Higher risk |
| Forter | Confidence % | Higher confidence in approve/decline decision |
| Signifyd | 0-1000 | Higher confidence the order is good |
| Adyen | 0-100+ | Higher risk |
The same score means different things at different businesses. A Radar 65 on a $20 digital download is a different bet than a 65 on a $2,000 electronics order. Your thresholds should track your margins, your fraud rate, and how much false positive pain you'll take.
For setting thresholds and calculating the cost trade-off, see Risk Scoring: Finding Your Thresholds.
How the Model Learns: The Feedback Loop
The model improves when you tell it what actually happened after it scored a transaction. That's the feedback loop.
The Three Types of Feedback
1. Chargebacks (automatic in most setups)
A customer disputes a charge with their bank. The chargeback flows from the card network through your processor. If your fraud vendor IS your processor (Stripe, Adyen), that's automatic. On a third-party vendor like Sift or Forter, check the integration is actually passing dispute data.
- Signal strength: High. A chargeback is a confirmed bad outcome.
- Delay: 30-120 days after the transaction. That's the core problem (see below).
2. Fraud reports (manual or semi-automated)
You flag a transaction as confirmed fraud before any chargeback lands. Maybe a customer called ("I didn't make this purchase"). Maybe your review team spotted a pattern.
- Signal strength: High, and much faster than waiting for a chargeback.
- Delay: As fast as you flag it. Days instead of months.
3. False positive flags (manual)
You tell the model a blocked transaction was actually legitimate. It's the signal most merchants forget to send.
- Signal strength: Medium. Helps the model learn what "good" looks like, not just what "bad" looks like.
- Delay: Whenever you catch it. Usually when a customer calls support about a block.
How Feedback Reaches Your Vendor
| Vendor | Chargebacks | Fraud Reports | False Positive Flags |
|---|---|---|---|
| Stripe | Automatic from disputes | Via Dashboard or API (Report as fraudulent) | Mark review items as legitimate |
| Sift | Requires $chargeback event via API | Requires $label event via API | Requires $label with is_fraud: false |
| Forter | Requires feedback API call | Requires feedback API call | Requires feedback API call |
| Signifyd | Automatic from guarantee claims | Via case management | Via case management |
| Adyen | Automatic from disputes | Via dispute management or API | Via risk rule feedback |
Make sure chargebacks reach your fraud vendor. That's automatic on most processor-integrated tools, like Stripe Radar and Adyen RevenueProtect. On a third-party tool like Sift or Forter, check the integration is sending chargeback events. Your vendor's docs will tell you.
That's the baseline. Everything else here is optimization on top.
The Feedback Delay Problem
Chargebacks arrive 30-120 days after the transaction. So the model is always learning from stale data.
Timeline:
Day 0: Transaction scored (model uses what it knows NOW)
Day 1-30: Fraud happens, goes undetected
Day 30: Customer notices unauthorized charge
Day 45: Customer files dispute with bank
Day 60: Chargeback reaches your processor
Day 75: Chargeback data feeds back to model
Day 90+: Model updates (next training cycle)
The model is learning from data that's 2-3 months old.
What This Means
The model is always one fraud cycle behind. If a new fraud pattern emerges in January, the model won't learn about it from chargebacks until March or April. Meanwhile, you're exposed.
What You Can Do
Send fraud reports immediately. Don't wait for the chargeback. When you confirm fraud through any channel, report it to your vendor right away.
| Signal | Speed | How to Send |
|---|---|---|
| Chargeback | 30-120 days | Automatic (verify integration) |
| Fraud report | 1-7 days | Manual via dashboard or API |
| Customer complaint ("I didn't buy this") | 1-3 days | Flag in review tool, report to vendor |
| Refund-before-chargeback | 1-14 days | Some vendors auto-learn from refund reason codes |
| Manual review decline | Same day | Review queue decision feeds back |
The faster you send outcomes, the faster the model adapts. Report fraud within 48 hours and you've handed your model a 2-3 month head start.
Some processors, Stripe and Adyen among them, learn from refund reason codes. Refund with reason "fraudulent" and the processor may read that as a fraud signal. Check your processor's docs.
When Models Go Wrong
ML models aren't set-and-forget. They degrade if nobody watches them.
Score Drift
The distribution of fraud scores shifts over time. A transaction that scored 60 last quarter might score 45 now. The underlying risk hasn't changed at all.
Why it happens: The model retrains on new data. New fraud patterns emerge. Seasonal traffic changes the mix. Or the vendor ships a model update.
How to spot it: Check your score distribution weekly. Median shifting, or tails changing shape? Your thresholds need a look.
Last quarter: Median score = 25 | 5% of transactions scored > 70
This quarter: Median score = 32 | 8% of transactions scored > 70
If your decline threshold is 70, your block rate just went from 5% to 8%.
Did fraud actually increase, or did the scores shift?
What to do: Re-test your thresholds quarterly. Run the threshold sweep experiment from Risk Scoring to recalibrate.
Fraud Pattern Changes
Fraudsters change tactics. A new attack hits you that the model has never seen.
Why it happens: Fraudsters adapt. If your model catches one pattern, they try another. New fraud techniques emerge industry-wide.
What to do: Rules cover the gap while the model catches up. When you spot a new pattern:
- Write a rule to catch it (see Building Fraud Rules)
- Report the fraud to your vendor (speeds model learning)
- The model learns the pattern from outcomes, eventually
That's why you need both. Rules are your fast response. ML is your long-term learner.
Cold Start
New business, new vertical, or new geography. The model has no history, so it's guessing.
Strategy:
- Lean heavily on rules early (they work without training data)
- Use conservative thresholds (more reviews, fewer auto-approvals)
- Weight the model higher as data accumulates (3-6 months)
- Send all outcomes (fraud and legitimate) to accelerate learning
For a detailed cold start strategy, see Risk Scoring: Cold Start Strategy.
Seasonal Shifts
Holiday shopping, back-to-school, end-of-year. That traffic looks nothing like your normal traffic. The model may flag real holiday behavior as anomalous.
What to do:
- November-December: Relax thresholds slightly (higher volume, more gift purchases, more new shipping addresses). Move borderline declines to review
- January: Tighten thresholds back (holiday fraud chargebacks start arriving, legitimate volume drops)
- Your seasonal peaks: If your business has specific peak periods (tax season, semester start, etc.), adjust thresholds around those dates
"Relax for the holidays" doesn't mean "turn off fraud detection." Fraudsters know you're relaxing. Adjust by 10-15%, not 50%. And monitor daily during peak periods.
What to Monitor
Weekly Checks
Score distribution: Is it shifting?
Pull a histogram of fraud scores for the past week. Compare to the previous 4-week average.
| What to Look For | What It Means | Action |
|---|---|---|
| Median score increasing | Model is scoring more transactions as risky | Check if fraud is actually rising, or if scores drifted |
| More transactions in the "review" band | Review queue will grow | Verify you have capacity, or adjust thresholds |
| Scores clustering at extremes | Model is more "decisive" (fewer gray areas) | Usually fine, but check false positive rate |
Block rate trend: Did something break?
| Signal | Possible Cause | Action |
|---|---|---|
| Sudden increase in blocks | New rule deployed too aggressively, model update, fraud spike | Investigate immediately |
| Gradual increase in blocks | Score drift, traffic mix change | Re-test thresholds |
| Sudden decrease in blocks | Rule disabled accidentally, model update | Check fraud-in-approved rate |
Monthly Checks
Fraud-in-approved rate: Are you missing fraud?
Fraud-in-approved = Chargebacks on approved transactions / Total approved transactions
If it's rising, your detection is getting worse. Either your thresholds are too loose, or something new is getting through.
False positive sample: Are you blocking good customers?
Pull a random sample of 10-20 declined or blocked transactions. Investigate each one:
- Was it actually fraud?
- Was it a legitimate customer?
- What rule or score triggered the block?
If more than 50% of your sample are false positives, your detection is too aggressive.
Quarterly Checks
Threshold re-calibration: Re-run the threshold sweep experiment. Scores drift. Last quarter's optimal threshold isn't this quarter's.
Monitoring Summary
| Cadence | What to Check | Red Flag |
|---|---|---|
| Weekly | Score distribution | Median shifted > 5 points |
| Weekly | Block rate | Changed > 20% from baseline |
| Monthly | Fraud-in-approved rate | Rising for 2+ consecutive months |
| Monthly | False positive sample (10-20 blocked txns) | Over 50% are legitimate |
| Quarterly | Threshold sweep | Optimal thresholds shifted > 10 points |
Next Steps
Just getting started with ML scoring?
- Verify chargebacks flow to your vendor (the baseline)
- Set initial thresholds using Risk Scoring
- Start weekly score distribution checks
Already have ML but want to improve it?
- Start sending fraud reports within 48 hours of confirmation
- Flag false positives in your review tool
- Run the threshold sweep experiment quarterly
Want the full operational picture?
- Running Fraud Operations - Daily/weekly/monthly cadence
- Building Fraud Rules - Rules as fast response to new patterns
- Processor Rules Configuration - Platform-specific setup
Related
- Risk Scoring - Thresholds, cost calculations, cold start
- Rules vs. ML - When to use each approach
- Building Fraud Rules - Starter rules, allow/block lists, shadow mode
- Velocity Rules - Rate-based detection
- Data Enrichment - IP, email, phone features for ML models
- Running Fraud Operations - Operational cadence playbook
- Processor Rules Configuration - Vendor-specific setup
- Manual Review - Review queue as feedback source
- Fraud Metrics - Measuring detection performance
- Fraud Vendors - Vendor ML capabilities
- Fraud Economics - Cost of fraud decisions
- Experimentation - Testing threshold changes