Skip to main content

Risk Scoring

TL;DR
  • Risk score: a 0-100 number for how likely fraud is. Use it to split traffic into auto-approve, review and auto-decline
  • Rules: email domain, velocity, geo mismatch. Transparent, fast to deploy, catches known patterns
  • ML: complex patterns, novel fraud. Needs data science people and labeled training data
  • Best approach: combine both. Rules for known fraud, ML for the subtle stuff
  • Threshold tuning: run A/B tests. Price a false positive against a fraud loss. Adjust quarterly

A risk score is just a number. The question is whether it helps you decide better.

Your thresholds are bets. You're trading blocked good customers for blocked fraud. The "right" threshold depends on your margins, your fraud rate, and how many complaints you can stomach.

Experiment to Run: Score Threshold Sweep

Illustrative cutoff sweep: Start in shadow mode. These bands show the method, not vendor-approved settings. Replace them using your score scale and observed outcomes:

  • Segment A: Score under 30 would approve, 30-60 would review, over 60 would block
  • Segment B: Score under 40 would approve, 40-70 would review, over 70 would block
  • Segment C: Score under 50 would approve, 50-80 would review, over 80 would block

Metrics: Confirmed fraud loss, review cost and contribution lost on legitimate orders blocked. Estimate the last from affected orders x average value x margin x probability the sale is lost.

Run length: Set enough time for the outcomes you're measuring to mature. Four weeks can collect early signals without settling fraud. Any live cutoff test needs the owner, loss cap and rollback first.

Decision: Pick the cutoff with lowest total cost. Probably not the tightest one.

What Is a Risk Score?​

A risk score is a number on each transaction. It says how likely fraud is. Higher score, higher risk.

Common scales:

  • 0-100 (higher = riskier)
  • 0-1000 (more granular)
  • 0-1 probability (true probability)

How it's used:

Score 0-30: Auto-approve
Score 31-70: Manual review
Score 71-100: Auto-decline

The thresholds depend on your risk tolerance, margins and review capacity. They aren't magic numbers. Every one is a trade-off between catching fraud and wrongly declining good customers.

Rules-Based Scoring​

A rule is an explicit condition that moves a transaction's score up or down.

How Rules Work​

Each rule checks a condition, then nudges the score:

IF email_domain = "tempmail.com" THEN +30
IF shipping_country != billing_country THEN +15
IF customer_has_previous_orders > 5 THEN -10
IF device_seen_on_fraud_before = true THEN +50
IF amount > $500 THEN +10

Final score = base score + sum of all triggered rules.

Types of Rules​

Identity rules:

  • Email validity: deliverable, disposable domain, recently created.
  • Phone number validation against the carrier.
  • Name consistency across the data you've got.

Transaction rules:

  • Order amount, since high value means higher risk.
  • Product category, since some categories run hot.
  • Shipping method, since expedited is riskier.
  • Billing and shipping address mismatch.

Behavioral rules:

  • Time to checkout, since too fast means bot.
  • Session behavior: copy-paste against actual typing.
  • Failed attempts before one finally succeeds.

Velocity rules:

  • Orders per IP per hour.
  • Cards per email per day.
  • Shipping addresses per card per week.

Device/network rules:

  • Proxy or VPN on the session.
  • Device fingerprint seen on fraud before.
  • Geolocation against billing country, the thing AVS checks.

Rules: Pros and Cons​

Pros:

  • Transparent, so you know why a transaction flagged.
  • Controllable: you can adjust for a new pattern today.
  • Explainable enough to satisfy customers, banks and auditors.
  • No training data required.

Cons:

  • Reactive: you write the rule after you've seen the fraud.
  • Brittle, because fraudsters learn your rules and adapt.
  • High maintenance, because rule sets don't stay small.
  • Blind to the subtle correlations a model would catch.

Machine Learning Scoring​

ML models read your transaction history to find patterns that predict fraud. Including patterns nobody could write as a rule.

How ML Scoring Works​

  1. Training: Model is fed historical transactions labeled as fraud/legitimate
  2. Learning: Model identifies features and patterns correlated with fraud
  3. Scoring: For new transactions, model outputs fraud probability
  4. Feedback loop: New fraud outcomes are fed back to improve the model

Types of ML Models​

Supervised learning:

  • Learns from labeled examples: this was fraud, this wasn't.
  • The most common approach for fraud scoring.
  • It won't work without clean labeled history.

Unsupervised learning:

  • Finds anomalies without any labels at all.
  • Useful for catching brand new fraud types.
  • Higher false positive rate than supervised.

Neural networks:

  • Finds complex, non-linear patterns nothing else sees.
  • Black box, so explaining one score is harder.
  • It doesn't work without a lot of data.

ML: Pros and Cons​

Pros:

  • Adaptive, because it learns new fraud patterns on its own.
  • Scalable to millions of transactions with no rule updates.
  • It catches correlations a person would never spot.

Cons:

  • Black box, so you can't explain one decision easily.
  • Data hungry, and it won't train without labeled history.
  • Cold start, so it's poor until the data arrives.
  • It'll learn the biases in your history too.

Combining Rules and ML​

The systems that work best run both:

Transaction arrives
↓
Rules evaluate (known patterns)
↓
ML model evaluates (complex patterns)
↓
Scores combined
↓
Decision + explanation

Why both?

  • Rules catch known, obvious fraud instantly.
  • ML catches emerging patterns and subtle signals.
  • Rules explain what the model just flagged.
  • ML cuts down your rule maintenance load.

Cold Start Strategy​

When you're launching, or short on data:

  1. Lean on rules at launch. They work with no training data.
  2. Give ML more weight as labeled outcomes pile up.
  3. Don't switch rules off when ML arrives. Feed rule hits to the model as labels.
  4. Push chargeback outcomes back in. That's how the model improves.

Example Combined System​

Rule: Shipping to known fraud address → +70 points
Rule: Email domain is disposable → +20 points
Rule: Customer has 3+ successful orders → -15 points
ML score: 0.35 (35% fraud probability) → +35 points
___________
Final score: → 110 points → DECLINE

Setting Thresholds​

Your threshold strategy depends on:

FactorLower Thresholds (stricter)Higher Thresholds (looser)
MarginLow margin (can't absorb fraud)High margin (can absorb some fraud)
ProductPhysical goods (lost forever)Digital (can revoke access)
Chargeback ratioNear network thresholdsComfortable buffer
Customer experienceLess importantCritical to business
Review capacityLarge review teamLimited/no review team

The Trade-Off Curve​

Your thresholds pick a point on the ROC curve. That's Receiver Operating Characteristic.

  • Lower threshold = catch more fraud, decline more good customers
  • Higher threshold = approve more good customers, let through more fraud

There's no "correct" threshold. It depends on what your business can tolerate. AUC, precision, recall and F1 help you find the balance. The call is still a business call.

Three-Tier Strategy​

Tier 1: Auto-approve (low scores)

  • Fast for the customer, no manual work for you.
  • You accept some fraud slipping through.

Tier 2: Manual review (middle scores)

  • A human takes the ambiguous ones.
  • You can ask for extra verification.
  • It costs real money per order reviewed.

Tier 3: Auto-decline (high scores)

  • Blocks the obvious fraud outright.
  • It'll cost you some good customers too.
  • Offer an alternative payment method instead.

Finding Your Thresholds​

Those "approve below 40, decline above 70" recommendations are someone else's guess. Here's how to find yours:

1. Calculate your cost of false positive:

Average order value × Gross margin × Probability customer never returns

If your AOV is $100, margin is 30%, and 50% of blocked customers never return: $100 × 0.3 × 0.5 = $15 per false positive

2. Calculate your cost of fraud:

Average fraud amount + Chargeback fee + Operational cost

If average fraud is $150, CB fee is $25, ops cost is $10: $185 per fraud

3. Find the break-even: At what threshold does the cost of false positives equal the cost of fraud prevented?

4. Test your hypothesis: Set thresholds based on your calculation. Run for 30 days. Measure actual costs. Adjust. New thresholds change real decisions, so this is a live test: give it an owner, a loss cap and an outcome window, because 30 days won't show every dispute.

Where This Can Fool You
  • Score calibration: Check what the vendor's score means. An 80 on a risk scale isn't automatically an 80% fraud probability. For a score sold as a probability, compare its predictions with mature confirmed outcomes.
  • Score drift: Model performance degrades over time. Re-test quarterly.
  • Feedback loops: never tell the model what was actually fraud and it goes stale. Make sure chargeback outcomes flow back.

Key Metrics​

Fraud detection rate (True Positive Rate / Recall): What percentage of actual fraud did you catch?

Fraud detected / Total fraud × 100

False positive rate: For the site's operational measure, what share of fraud blocks were confirmed legitimate?

Confirmed legitimate transactions blocked / All fraud blocks x 100

Model evaluation often uses false positives / all legitimate transactions instead. Keep that measure separately labelled; see Fraud Metrics.

Precision: Of transactions you flagged as fraud, how many actually were?

True fraud flagged / All transactions flagged × 100

Review rate: What percentage of transactions go to manual review?

Transactions in review / Total transactions × 100

Ideal: high detection, low false positives, a review rate you can staff.

Building vs. Buying​

Build your own:

  • Full control of the rules and the models.
  • You can tune for your own fraud patterns.
  • You need data science people on staff.
  • You'll own the maintenance forever.

Buy a solution:

  • Faster to stand up.
  • The vendor sees fraud across many merchants.
  • Less control of the scoring logic.
  • A per-transaction cost that scales.

Hybrid:

  • Vendor for the ML and consortium data.
  • Your own rules layered on top.
  • The right answer for most merchants.

Vendor Landscape​

Note: this space moves fast. Judge vendors on your own stack, geography and risk profile.

CategoryExamples
Standalone fraud platformsForter, Riskified, Signifyd, SEON
Processor-integratedStripe Radar, Adyen Risk, Checkout.com FDP
Identity/deviceKount, ThreatMetrix, BioCatch
Rules enginesSplunk, Datadog (DIY)

Do You Need ML Scoring?​

Most SMBs don't need a dedicated ML scoring engine. Here's a quick decision framework:

Your SituationDo You Need ML?Why
Under $1M/monthStart with current tools.Processor screening may already cover the problem. Check the plan, missed fraud and review workload before adding a separate model.
$1M-$5M/month, 10+ manual rules, 2+ hours/week maintaining themMaybe.When your rule set gets unwieldy and you're spending more time tuning rules than running your business, a dedicated scoring tool can automate what rules can't catch.
Fraud patterns current tools consistently miss, at any volumeTest the gap.Evaluate added scoring against confirmed outcomes, full cost and blocked good orders. Sales volume alone doesn't establish a need.
The Practical Answer

You don't have to choose between rules and ML. Start with the tools your processor provides. Evaluate extra scoring when you can name a fraud pattern those tools miss and test whether the avoided loss beats tool fees and customer friction. Monthly sales alone don't answer that.

Next Steps​

Just getting started with scoring?

  1. Use your processor's built-in scoring → Stripe Radar, Adyen Risk, etc.
  2. Define three buckets → Auto-approve, review, auto-decline
  3. Track your false positive rate → Customer complaints are the signal

Tuning your thresholds?

  1. Run the threshold sweep experiment (see top of page) → Data beats intuition
  2. Segment by transaction type → Different thresholds for different products
  3. Track fraud rate AND false positive rate → Optimize the tradeoff, not just one metric

Building custom scoring?

  1. Review rules vs. ML tradeoffs → Know when to use which
  2. Start with rules on known patterns → ML for novel detection
  3. Invest in feature engineering → Good features beat complex models

See Also​

Something wrong on this page? Email matt at paymentsandrisk.com and say what needs fixing.