Risk Scoring

TL;DR

Risk score: a 0-100 number for how likely fraud is. Use it to split traffic into auto-approve, review and auto-decline
Rules: email domain, velocity, geo mismatch. Transparent, fast to deploy, catches known patterns
ML: complex patterns, novel fraud. Needs data science people and labeled training data
Best approach: combine both. Rules for known fraud, ML for the subtle stuff
Threshold tuning: run A/B tests. Price a false positive against a fraud loss. Adjust quarterly

A risk score is just a number. The question is whether it helps you decide better.

Your thresholds are bets. You're trading blocked good customers for blocked fraud. The "right" threshold depends on your margins, your fraud rate, and how many complaints you can stomach.

Experiment to Run: Score Threshold Sweep

Run 3 cutoffs in parallel on small slices of traffic:

Segment A: Score under 30 auto-approve, 30-60 review, over 60 auto-block
Segment B: Score under 40 auto-approve, 40-70 review, over 70 auto-block
Segment C: Score under 50 auto-approve, 50-80 review, over 80 auto-block

Metrics: Fraud loss + review cost + estimated false positive cost (use average order value × block rate × estimated good customer %)

Run length: 4 weeks. Chargebacks need time to show up

Decision: Pick the cutoff with lowest total cost. Probably not the tightest one.

What Is a Risk Score?

A risk score is a number on each transaction. It says how likely fraud is. Higher score, higher risk.

Common scales:

0-100 (higher = riskier)
0-1000 (more granular)
0-1 probability (true probability)

How it's used:

Score 0-30:   Auto-approve
Score 31-70:  Manual review
Score 71-100: Auto-decline

The thresholds depend on your risk tolerance, margins and review capacity. They aren't magic numbers. Every one is a trade-off between catching fraud and wrongly declining good customers.

Rules-Based Scoring

A rule is an explicit condition that moves a transaction's score up or down.

How Rules Work

Each rule checks a condition, then nudges the score:

IF email_domain = "tempmail.com" THEN +30
IF shipping_country != billing_country THEN +15
IF customer_has_previous_orders > 5 THEN -10
IF device_seen_on_fraud_before = true THEN +50
IF amount > $500 THEN +10

Final score = base score + sum of all triggered rules.

Types of Rules

Identity rules:

Email validity: deliverable, disposable domain, recently created.
Phone number validation against the carrier.
Name consistency across the data you've got.

Transaction rules:

Order amount, since high value means higher risk.
Product category, since some categories run hot.
Shipping method, since expedited is riskier.
Billing and shipping address mismatch.

Behavioral rules:

Time to checkout, since too fast means bot.
Session behavior: copy-paste against actual typing.
Failed attempts before one finally succeeds.

Velocity rules:

Orders per IP per hour.
Cards per email per day.
Shipping addresses per card per week.

Device/network rules:

Proxy or VPN on the session.
Device fingerprint seen on fraud before.
Geolocation against billing country, the thing AVS checks.

Rules: Pros and Cons

Pros:

Transparent, so you know why a transaction flagged.
Controllable: you can adjust for a new pattern today.
Explainable enough to satisfy customers, banks and auditors.
No training data required.

Cons:

Reactive: you write the rule after you've seen the fraud.
Brittle, because fraudsters learn your rules and adapt.
High maintenance, because rule sets don't stay small.
Blind to the subtle correlations a model would catch.

Machine Learning Scoring

ML models read your transaction history to find patterns that predict fraud. Including patterns nobody could write as a rule.

How ML Scoring Works

Training: Model is fed historical transactions labeled as fraud/legitimate
Learning: Model identifies features and patterns correlated with fraud
Scoring: For new transactions, model outputs fraud probability
Feedback loop: New fraud outcomes are fed back to improve the model

Types of ML Models

Supervised learning:

Learns from labeled examples: this was fraud, this wasn't.
The most common approach for fraud scoring.
It won't work without clean labeled history.

Unsupervised learning:

Finds anomalies without any labels at all.
Useful for catching brand new fraud types.
Higher false positive rate than supervised.

Neural networks:

Finds complex, non-linear patterns nothing else sees.
Black box, so explaining one score is harder.
It doesn't work without a lot of data.

ML: Pros and Cons

Pros:

Adaptive, because it learns new fraud patterns on its own.
Scalable to millions of transactions with no rule updates.
It catches correlations a person would never spot.

Cons:

Black box, so you can't explain one decision easily.
Data hungry, and it won't train without labeled history.
Cold start, so it's poor until the data arrives.
It'll learn the biases in your history too.

Combining Rules and ML

The systems that work best run both:

Transaction arrives
       ↓
Rules evaluate (known patterns)
       ↓
ML model evaluates (complex patterns)
       ↓
Scores combined
       ↓
Decision + explanation

Why both?

Rules catch known, obvious fraud instantly.
ML catches emerging patterns and subtle signals.
Rules explain what the model just flagged.
ML cuts down your rule maintenance load.

Cold Start Strategy

When you're launching, or short on data:

Lean on rules at launch. They work with no training data.
Give ML more weight as labeled outcomes pile up.
Don't switch rules off when ML arrives. Feed rule hits to the model as labels.
Push chargeback outcomes back in. That's how the model improves.

Example Combined System

Rule: Shipping to known fraud address     → +70 points
Rule: Email domain is disposable          → +20 points
Rule: Customer has 3+ successful orders   → -15 points
ML score: 0.35 (35% fraud probability)    → +35 points
                                          ___________
Final score:                              → 110 points → DECLINE

Setting Thresholds

Your threshold strategy depends on:

Factor	Lower Thresholds (stricter)	Higher Thresholds (looser)
Margin	Low margin (can't absorb fraud)	High margin (can absorb some fraud)
Product	Physical goods (lost forever)	Digital (can revoke access)
Chargeback ratio	Near network thresholds	Comfortable buffer
Customer experience	Less important	Critical to business
Review capacity	Large review team	Limited/no review team

The Trade-Off Curve

Your thresholds pick a point on the ROC curve. That's Receiver Operating Characteristic.

Lower threshold = catch more fraud, decline more good customers
Higher threshold = approve more good customers, let through more fraud

There's no "correct" threshold. It depends on what your business can tolerate. AUC, precision, recall and F1 help you find the balance. The call is still a business call.

Three-Tier Strategy

Tier 1: Auto-approve (low scores)

Fast for the customer, no manual work for you.
You accept some fraud slipping through.

Tier 2: Manual review (middle scores)

A human takes the ambiguous ones.
You can ask for extra verification.
It costs real money per order reviewed.

Tier 3: Auto-decline (high scores)

Blocks the obvious fraud outright.
It'll cost you some good customers too.
Offer an alternative payment method instead.

Finding Your Thresholds

Those "approve below 40, decline above 70" recommendations are someone else's guess. Here's how to find yours:

1. Calculate your cost of false positive:

Average order value × Gross margin × Probability customer never returns

If your AOV is $100, margin is 30%, and 50% of blocked customers never return: $100 × 0.3 × 0.5 = $15 per false positive

2. Calculate your cost of fraud:

Average fraud amount + Chargeback fee + Operational cost

If average fraud is $150, CB fee is $25, ops cost is $10: $185 per fraud

3. Find the break-even: At what threshold does the cost of false positives equal the cost of fraud prevented?

4. Test your hypothesis: Set thresholds based on your calculation. Run for 30 days. Measure actual costs. Adjust.

Where This Can Fool You

Score calibration: A score of 80 should mean 80% of those transactions are fraud. Check if yours does. Many vendor scores aren't well-calibrated.
Score drift: Model performance degrades over time. Re-test quarterly.
Feedback loops: never tell the model what was actually fraud and it goes stale. Make sure chargeback outcomes flow back.

Key Metrics

Fraud detection rate (True Positive Rate / Recall): What percentage of actual fraud did you catch?

Fraud detected / Total fraud × 100

False positive rate: What percentage of good transactions were wrongly declined?

Good transactions declined / Total good transactions × 100

Precision: Of transactions you flagged as fraud, how many actually were?

True fraud flagged / All transactions flagged × 100

Review rate: What percentage of transactions go to manual review?

Transactions in review / Total transactions × 100

Ideal: high detection, low false positives, a review rate you can staff.

Building vs. Buying

Build your own:

Full control of the rules and the models.
You can tune for your own fraud patterns.
You need data science people on staff.
You'll own the maintenance forever.

Buy a solution:

Faster to stand up.
The vendor sees fraud across many merchants.
Less control of the scoring logic.
A per-transaction cost that scales.

Hybrid:

Vendor for the ML and consortium data.
Your own rules layered on top.
The right answer for most merchants.

Vendor Landscape

Note: this space moves fast. Judge vendors on your own stack, geography and risk profile.

Category	Examples
Standalone fraud platforms	Forter, Riskified, Signifyd, SEON
Processor-integrated	Stripe Radar, Adyen Risk, Checkout.com FDP
Identity/device	Kount, ThreatMetrix, BioCatch
Rules engines	Splunk, Datadog (DIY)

Do You Need ML Scoring?

Most SMBs don't need a dedicated ML scoring engine. Here's a quick decision framework:

Your Situation	Do You Need ML?	Why
Under $1M/month	No.	If you're on Stripe, Radar IS your ML - it's included free. Adyen's RevenueProtect and Shopify's built-in tools also use ML under the hood. You already have it.
$1M-$5M/month, 10+ manual rules, 2+ hours/week maintaining them	Maybe.	When your rule set gets unwieldy and you're spending more time tuning rules than running your business, a dedicated scoring tool can automate what rules can't catch.
Over $5M/month or seeing fraud patterns your rules consistently miss	Yes.	At this volume, novel fraud patterns slip past static rules. ML adapts to patterns you haven't written rules for yet.

The Practical Answer

You don't have to choose between rules and ML. Start with rules. They're free in Stripe Radar, Adyen and most processor tools. When rules plateau, add ML. Plateau means you keep adding rules and fraud still gets through. That usually happens between $2M and $5M a month, not before.

Next Steps

Just getting started with scoring?

Use your processor's built-in scoring → Stripe Radar, Adyen Risk, etc.
Define three buckets → Auto-approve, review, auto-decline
Track your false positive rate → Customer complaints are the signal

Tuning your thresholds?

Run the threshold sweep experiment (see top of page) → Data beats intuition
Segment by transaction type → Different thresholds for different products
Track fraud rate AND false positive rate → Optimize the tradeoff, not just one metric

Building custom scoring?

Review rules vs. ML tradeoffs → Know when to use which
Start with rules on known patterns → ML for novel detection
Invest in feature engineering → Good features beat complex models

What Is a Risk Score?​

Rules-Based Scoring​

How Rules Work​

Types of Rules​

Rules: Pros and Cons​

Machine Learning Scoring​

How ML Scoring Works​

Types of ML Models​

ML: Pros and Cons​

Combining Rules and ML​

Cold Start Strategy​

Example Combined System​

Setting Thresholds​

The Trade-Off Curve​

Three-Tier Strategy​

Finding Your Thresholds​

Key Metrics​

Building vs. Buying​

Vendor Landscape​

Do You Need ML Scoring?​

Next Steps​

See Also​