Payments Experimentation (Operator Field Manual)
- Shadow mode before enforcement: log decisions without blocking. Plan the sample and outcome window before you start; a week or two can reveal problems without proving fraud results
- Pick one primary decision metric per test, and name the guardrails upfront (fraud, customer impact, data quality). A test can track several metrics; it needs one declared rule for deciding
- Limit the eligible cohort to manage risk, then randomize comparable orders inside it between treatment and control. A test can cover all eligible traffic; the randomized control is what isolates the change
- Set stop rules before launch: max false positive rate, max revenue at risk per day. Then honor them, because you won't want to at the time
- Any test that changes real orders needs a risk owner, a loss cap and an outcome window first. Inconclusive means keep your current controls
Flipping fraud rules live without testing is how good orders start dying. Shadow first, then enforce. Treat payment rules like code: you wouldn't deploy straight to production, so don't do it here either.
Checked 3 November 2026. Adapt the steps to your stack.
What makes a test readable
- Shadow mode first. Log decisions, don't block. Measure false positives before going live.
- Pick a primary decision metric per test, with guardrails named upfront. Auth lift can be primary while fraud and customer impact guard the decision.
- Run by cohort, randomized inside it. Pick the eligible slice (method, BIN, country, device, card-present vs card-not-present), then assign treatment and control at random within it, and hold the assignment stable.
- Set stop rules before launch. Max false positive %, max revenue at risk. Honor them.
- Feedback loops lag. Use alerts and the banks' fraud reports (Mastercard's SAFE, Visa's TC40; ask your processor for them) to shorten the chargeback feedback delay.
Shadow Mode: The Foundation
Shadow mode runs your new rule without enforcing it. Log what happened and what the new rule would've done, then link both to confirmed outcomes. Logs alone don't reveal false positives: some fraud is delayed, and some blocked orders stay unknown.
How to Implement Shadow Mode
- Log both decisions - Actual outcome + shadow rule outcome
- Tag transactions - Mark "would-have-blocked" for tracking
- Don't affect the customer - Shadow decisions are invisible
- Track over time - Choose enough comparable traffic and follow-up for your decision; extend shadow mode when outcomes are unresolved
What to Measure in Shadow
| Metric | What It Tells You |
|---|---|
| Would-block rate | How aggressive is the new rule |
| False positive rate | Confirmed legitimate / All would-blocks; report unknowns separately |
| Detection rate | Confirmed fraud that would be flagged / All confirmed fraud in the measured population |
| Coverage | Which confirmed fraud patterns this rule would catch beyond current controls |
Shadow Decision Matrix
| Actual Outcome | Shadow Decision | Interpretation |
|---|---|---|
| Approved, confirmed legitimate | Would block | False positive |
| Approved, confirmed fraud | Would block | True positive |
| Approved, confirmed legitimate | Would allow | Correct allow |
| Approved, confirmed fraud | Would allow | Missed fraud |
| Outcome unresolved | Either | Unknown; a dispute alone doesn't establish fraud, and no dispute doesn't prove legitimacy |
Experiment Design
Pick the test type first, because the metric follows from it. For a first experiment, tighten one fraud rule. Everything else can wait.
Test Types
| Test Type | What You're Testing | Key Metric |
|---|---|---|
| Fraud rule tightening | New velocity limit | Block rate vs fraud rate |
| Fraud rule loosening | Relaxing a rule | Auth rate vs fraud increase |
| 3DS threshold | When to challenge | Conversion vs liability shift |
| Auth retry | Decline handling | Recovery rate vs cost |
| Checkout flow | Payment form changes | Conversion rate |
Cohort Selection
Limiting the eligible cohort is a risk decision, not what makes the test readable. A randomized treatment/control split can cover all eligible traffic. Within your cohort, randomize comparable orders and keep assignment stable. Pick cohorts that:
- Have enough observations for the effect you're trying to detect, given the baseline rate and uncertainty you can accept
- Represent meaningful segments
- Can be isolated
Good cohorts:
- Geographic (US vs EU vs APAC)
- Payment method (card vs wallet)
- Transaction type (CP vs CNP)
- BIN range (specific issuers)
- Device type (mobile vs desktop)
- Customer type (new vs returning)
Sample Size Guidelines
Traffic volume tells you how fast observations arrive, not how many prove a result. Plan the sample from your baseline rate, the smallest change worth acting on, and acceptable uncertainty.
| Decision Volume | Planning question | What still has to close |
|---|---|---|
| Under 100/day | Can a longer shadow run collect enough relevant outcomes? | Sample plan and fraud follow-up window |
| 100-500/day | How many days reach the planned sample in each cohort? | Sample plan and fraud follow-up window |
| 500-2000/day | Are segment and weekday patterns represented? | Sample plan and fraud follow-up window |
| Over 2000/day | Are repeated attempts or customers inflating the count? | Sample plan and fraud follow-up window |
If the planned sample or mature outcomes aren't available, leave shadow mode on and call it monitoring. No daily volume guarantees a conclusive test.
Before any live test
Shadow mode changes nothing a customer sees. A live test does: real orders get blocked, challenged or let through. So write these seven things down before one starts, and get a yes from whoever carries the losses:
- Who owns the risk. One role in your business approves the test and can stop it. In a one-person shop that's you. On a team, it's whoever answers for fraud losses. Not the vendor, and not the tool
- Which orders it touches. Name the eligible orders: the cohort, channel or amount band from cohort selection. Never let flagged orders through just to build a comparison group
- The most it can cost. A dollar cap on the fraud the test can let through and the good orders it can block. Reach it and the test stops
- How you undo it. A rollback you've tried, and who's allowed to pull it
- When it stops early. A stop rule set before launch
- When the results count. Fraud and disputes arrive weeks after the order. Set an outcome window long enough for the disputes you're measuring to land, and don't judge fraud before it closes. A change that only adds checks can expand sooner. 3DS on more of the eligible orders you named counts, if it doesn't replace any declines. Expand only if completed payments hold up, and stay inside your loss cap. Anything that loosens waits for the window. Time Frames has the network dispute windows
- What inconclusive means. Too few orders, or outcomes too young to read? Keep your current controls, plus any added check you've already expanded under point 6. Waiting is an answer
Run every live test under these seven, whether it's a vendor pilot, a 3DS rollout or a split rule test.
Test to Run
Week 1: Shadow Phase
- Choose one rule change - Example: tighten velocity on CNP high-risk BINs
- Implement shadow logging - Log would-block decisions
- Tag approved transactions - Mark those that would have been blocked
- Monitor daily - Check false positive rate
Week 2: Analysis
Calculate fraud results only from mature outcomes. Keep observing if disputes haven't had time to arrive.
- Calculate false positives - Good orders that would have blocked
- Calculate true positives - confirmed fraud, under your declared definition and outcome window, that the rule would have caught. Disputes and refunds are separate evidence; unresolved cases stay unknown
- Assess impact - Revenue at risk vs fraud prevented
- Decide: proceed, modify, or abandon
Week 3: Ramp (if proceeding)
- Enable on 10-25% of traffic - Real enforcement, limited scope, and only once the seven live-test points are written down
- Monitor hourly - Watch for unexpected blocks
- Check customer support - Any complaints about declines?
- Compare to control - Does reality match shadow?
Week 4: Full Rollout (if successful)
- Roll to 100% - Only if Week 3 metrics are stable and the outcome window you set has closed. A week of enforcement can't show the disputes that haven't been filed yet
- Document baseline - New normal for this rule
- Set ongoing alerts - Detect drift from baseline
- Plan next experiment - Continuous improvement
Metrics to Track
Primary Metrics (choose one per test)
| Metric | Definition | Target Direction |
|---|---|---|
| Auth rate | Approved / Attempted | Higher is better |
| Block rate | Fraud-control blocks / Attempted | Compare fraud prevented with customer impact |
| Fraud rate | Confirmed fraud / Transactions in the same measured population | Lower within your other budgets |
| Conversion | Completed / Started | Higher is better |
Secondary Metrics (monitor, don't optimize)
| Metric | Why Track It |
|---|---|
| False positive rate | Catch good-order blocking |
| Support tickets | Detect customer friction |
| Revenue per attempt | Net effect on business |
| Soft vs hard decline mix | Understand decline sources |
Analyst Calculations
Use the Fraud Metrics definitions; a dispute status alone isn't a fraud label.
Block rate = Fraud-control blocks / Total attempts
Shadow false positive rate = Confirmed legitimate would-blocks / All would-blocks
Detection rate = Confirmed fraud that would be flagged / All confirmed fraud
Relative auth change = (Treatment auth rate - Control auth rate) / Control auth rate, when the control rate isn't zero
Report the percentage-point difference too, beside unknown outcomes, sample counts and uncertainty. The formula alone isn't causal proof; the controlled design is.
Stop Rules
Define these before launch and honor them when they fire, because the moment a stop rule triggers is exactly when you won't want to obey it.
Example Stop Rules
Choose these limits before launch from your baseline, expected margin impact and loss budget. They are your settings, not industry cutoffs. For any rate difference, write whether you mean percentage points or relative percent.
| Condition | Action |
|---|---|
| Confirmed blocked-good margin exceeds your dollar cap | Pause experiment |
| Auth rate falls beyond the difference you approved vs. control | Pause and investigate |
| Decline complaints breach the limit set from comparable traffic | Pause and review |
| Fraud exposure or blocked-good margin reaches its agreed dollar cap | Roll back |
| Critical incident under your incident plan | Immediate rollback |
Rollback Requirements
One-click rollback or it isn't an experiment. Before launching any experiment:
- Confirm rollback is one-click (or automated)
- Test rollback in staging
- Document rollback procedure
- Assign rollback authority
Scale Callout
| Volume | Approach |
|---|---|
| Under $100k/mo | Start in shadow mode. Low case counts can leave tests inconclusive; sales volume alone doesn't decide significance. |
| $100k-$1M/mo | Choose a limited ramp from the loss cap and eligible cohort. Expand only within pre-agreed budgets once the outcome window closes. Document everything. |
| Over $1M/mo | Require rollback switch, alerting, daily review during ramp. Dedicated owner per experiment. |
Where This Breaks
- No labeled outcomes. If you can't tell good from bad orders, fix tagging first. Without confirmed labels you can't claim fraud detection or false positives. An auth or conversion test can still measure those outcomes while the fraud verdict stays inconclusive.
- Chargeback feedback lags. Dispute data arrives weeks late. Use alerts, SAFE, TC40 to shorten the loop.
- Testing during peak periods. Black Friday, promotions and holidays change who's buying. A simultaneous randomized control still reads for that period. Name the period, check assignment and mix for actual bias, and don't export the conclusion to normal weeks unqualified.
- Multiple simultaneous changes. A bundle measures the bundle. Splitting the credit between its parts needs a design built for it, such as a factorial test. For a first test, isolate one variable.
- No operator-dev handshake. Engineers deploy, operators don't know. Add "show me the shadow logs" checkpoint.
Common Experimentation Mistakes
| Mistake | Consequence | Prevention |
|---|---|---|
| No shadow period | Can block good orders from day one | Shadow decision rules before they enforce |
| Too short test | Inconclusive results | Minimum sample sizes |
| No stop rules | Runaway false positives | Define before launch |
| Multiple changes | Measures the bundle, not its parts | One variable at a time for a first test |
| No rollback plan | Stuck with bad rule | Test rollback first |
| Ignoring support signals | Customer friction unnoticed | Monitor tickets |
Experimentation Infrastructure
Minimum Requirements
- Shadow logging - Record shadow decisions separately
- Outcome tagging - Link transactions to disputes/refunds
- Cohort assignment - Random assignment, held stable: the same customer always lands in the same test group. Consistent assignment alone isn't randomization
- Metrics dashboard - Real-time visibility
- Alert system - Trigger on stop rule conditions
- Rollback mechanism - Quick revert capability
Nice to Have
- Statistical significance calculator - Built into dashboard
- Automatic ramping - Gradual traffic increase
- Experiment registry - Track all active/past tests
- Cross-experiment interference detection - Catch conflicts
Next Steps
Setting up your first experiment?
- Implement shadow mode - Log decisions without blocking
- Design the test - Test type, cohort, sample size
- Define stop rules - Before you launch
Running a test now?
- Follow the test steps - Shadow, analyze, ramp, roll out, once the outcome window has passed
- Track key metrics - Primary and secondary
- Know when to stop - Honor the rules
Building experimentation infrastructure?
- Meet minimum requirements - Shadow logging, tagging, cohorts
- Avoid common mistakes - No shadow, too short, no rollback
- Scale appropriately - By transaction volume
Related
- Reference - The whole reference section
- Processor Rules Configuration - Native fraud tools
- Velocity Rules - Rate-based detection
- Auth Optimization - Improving approval rates
- Processor Reporting Checklist - Data requirements
- Alerts Configuration - Monitoring setup
- Risk Scoring - Score thresholds
- 3D Secure - Authentication testing
- Rules vs. ML - Detection approaches
- Checkout Conversion - Friction impact
- Fraud Metrics - Measuring performance
- Chargeback Metrics - Dispute tracking
- Benchmarks - Performance targets