Skip to main content

Payments Experimentation (Operator Field Manual)

TL;DR
  • Shadow mode before enforcement: log decisions without blocking. Plan the sample and outcome window before you start; a week or two can reveal problems without proving fraud results
  • Pick one primary decision metric per test, and name the guardrails upfront (fraud, customer impact, data quality). A test can track several metrics; it needs one declared rule for deciding
  • Limit the eligible cohort to manage risk, then randomize comparable orders inside it between treatment and control. A test can cover all eligible traffic; the randomized control is what isolates the change
  • Set stop rules before launch: max false positive rate, max revenue at risk per day. Then honor them, because you won't want to at the time
  • Any test that changes real orders needs a risk owner, a loss cap and an outcome window first. Inconclusive means keep your current controls

Flipping fraud rules live without testing is how good orders start dying. Shadow first, then enforce. Treat payment rules like code: you wouldn't deploy straight to production, so don't do it here either.

Checked 3 November 2026. Adapt the steps to your stack.

What makes a test readable​

  • Shadow mode first. Log decisions, don't block. Measure false positives before going live.
  • Pick a primary decision metric per test, with guardrails named upfront. Auth lift can be primary while fraud and customer impact guard the decision.
  • Run by cohort, randomized inside it. Pick the eligible slice (method, BIN, country, device, card-present vs card-not-present), then assign treatment and control at random within it, and hold the assignment stable.
  • Set stop rules before launch. Max false positive %, max revenue at risk. Honor them.
  • Feedback loops lag. Use alerts and the banks' fraud reports (Mastercard's SAFE, Visa's TC40; ask your processor for them) to shorten the chargeback feedback delay.

Shadow Mode: The Foundation​

Shadow mode runs your new rule without enforcing it. Log what happened and what the new rule would've done, then link both to confirmed outcomes. Logs alone don't reveal false positives: some fraud is delayed, and some blocked orders stay unknown.

How to Implement Shadow Mode​

  1. Log both decisions - Actual outcome + shadow rule outcome
  2. Tag transactions - Mark "would-have-blocked" for tracking
  3. Don't affect the customer - Shadow decisions are invisible
  4. Track over time - Choose enough comparable traffic and follow-up for your decision; extend shadow mode when outcomes are unresolved

What to Measure in Shadow​

MetricWhat It Tells You
Would-block rateHow aggressive is the new rule
False positive rateConfirmed legitimate / All would-blocks; report unknowns separately
Detection rateConfirmed fraud that would be flagged / All confirmed fraud in the measured population
CoverageWhich confirmed fraud patterns this rule would catch beyond current controls

Shadow Decision Matrix​

Actual OutcomeShadow DecisionInterpretation
Approved, confirmed legitimateWould blockFalse positive
Approved, confirmed fraudWould blockTrue positive
Approved, confirmed legitimateWould allowCorrect allow
Approved, confirmed fraudWould allowMissed fraud
Outcome unresolvedEitherUnknown; a dispute alone doesn't establish fraud, and no dispute doesn't prove legitimacy

Experiment Design​

Pick the test type first, because the metric follows from it. For a first experiment, tighten one fraud rule. Everything else can wait.

Test Types​

Test TypeWhat You're TestingKey Metric
Fraud rule tighteningNew velocity limitBlock rate vs fraud rate
Fraud rule looseningRelaxing a ruleAuth rate vs fraud increase
3DS thresholdWhen to challengeConversion vs liability shift
Auth retryDecline handlingRecovery rate vs cost
Checkout flowPayment form changesConversion rate

Cohort Selection​

Limiting the eligible cohort is a risk decision, not what makes the test readable. A randomized treatment/control split can cover all eligible traffic. Within your cohort, randomize comparable orders and keep assignment stable. Pick cohorts that:

  • Have enough observations for the effect you're trying to detect, given the baseline rate and uncertainty you can accept
  • Represent meaningful segments
  • Can be isolated

Good cohorts:

  • Geographic (US vs EU vs APAC)
  • Payment method (card vs wallet)
  • Transaction type (CP vs CNP)
  • BIN range (specific issuers)
  • Device type (mobile vs desktop)
  • Customer type (new vs returning)

Sample Size Guidelines​

Traffic volume tells you how fast observations arrive, not how many prove a result. Plan the sample from your baseline rate, the smallest change worth acting on, and acceptable uncertainty.

Decision VolumePlanning questionWhat still has to close
Under 100/dayCan a longer shadow run collect enough relevant outcomes?Sample plan and fraud follow-up window
100-500/dayHow many days reach the planned sample in each cohort?Sample plan and fraud follow-up window
500-2000/dayAre segment and weekday patterns represented?Sample plan and fraud follow-up window
Over 2000/dayAre repeated attempts or customers inflating the count?Sample plan and fraud follow-up window

If the planned sample or mature outcomes aren't available, leave shadow mode on and call it monitoring. No daily volume guarantees a conclusive test.


Before any live test​

Shadow mode changes nothing a customer sees. A live test does: real orders get blocked, challenged or let through. So write these seven things down before one starts, and get a yes from whoever carries the losses:

  1. Who owns the risk. One role in your business approves the test and can stop it. In a one-person shop that's you. On a team, it's whoever answers for fraud losses. Not the vendor, and not the tool
  2. Which orders it touches. Name the eligible orders: the cohort, channel or amount band from cohort selection. Never let flagged orders through just to build a comparison group
  3. The most it can cost. A dollar cap on the fraud the test can let through and the good orders it can block. Reach it and the test stops
  4. How you undo it. A rollback you've tried, and who's allowed to pull it
  5. When it stops early. A stop rule set before launch
  6. When the results count. Fraud and disputes arrive weeks after the order. Set an outcome window long enough for the disputes you're measuring to land, and don't judge fraud before it closes. A change that only adds checks can expand sooner. 3DS on more of the eligible orders you named counts, if it doesn't replace any declines. Expand only if completed payments hold up, and stay inside your loss cap. Anything that loosens waits for the window. Time Frames has the network dispute windows
  7. What inconclusive means. Too few orders, or outcomes too young to read? Keep your current controls, plus any added check you've already expanded under point 6. Waiting is an answer

Run every live test under these seven, whether it's a vendor pilot, a 3DS rollout or a split rule test.


Test to Run​

Week 1: Shadow Phase​

  1. Choose one rule change - Example: tighten velocity on CNP high-risk BINs
  2. Implement shadow logging - Log would-block decisions
  3. Tag approved transactions - Mark those that would have been blocked
  4. Monitor daily - Check false positive rate

Week 2: Analysis​

Calculate fraud results only from mature outcomes. Keep observing if disputes haven't had time to arrive.

  1. Calculate false positives - Good orders that would have blocked
  2. Calculate true positives - confirmed fraud, under your declared definition and outcome window, that the rule would have caught. Disputes and refunds are separate evidence; unresolved cases stay unknown
  3. Assess impact - Revenue at risk vs fraud prevented
  4. Decide: proceed, modify, or abandon

Week 3: Ramp (if proceeding)​

  1. Enable on 10-25% of traffic - Real enforcement, limited scope, and only once the seven live-test points are written down
  2. Monitor hourly - Watch for unexpected blocks
  3. Check customer support - Any complaints about declines?
  4. Compare to control - Does reality match shadow?

Week 4: Full Rollout (if successful)​

  1. Roll to 100% - Only if Week 3 metrics are stable and the outcome window you set has closed. A week of enforcement can't show the disputes that haven't been filed yet
  2. Document baseline - New normal for this rule
  3. Set ongoing alerts - Detect drift from baseline
  4. Plan next experiment - Continuous improvement

Metrics to Track​

Primary Metrics (choose one per test)​

MetricDefinitionTarget Direction
Auth rateApproved / AttemptedHigher is better
Block rateFraud-control blocks / AttemptedCompare fraud prevented with customer impact
Fraud rateConfirmed fraud / Transactions in the same measured populationLower within your other budgets
ConversionCompleted / StartedHigher is better

Secondary Metrics (monitor, don't optimize)​

MetricWhy Track It
False positive rateCatch good-order blocking
Support ticketsDetect customer friction
Revenue per attemptNet effect on business
Soft vs hard decline mixUnderstand decline sources

Analyst Calculations​

Use the Fraud Metrics definitions; a dispute status alone isn't a fraud label.

Block rate = Fraud-control blocks / Total attempts
Shadow false positive rate = Confirmed legitimate would-blocks / All would-blocks
Detection rate = Confirmed fraud that would be flagged / All confirmed fraud
Relative auth change = (Treatment auth rate - Control auth rate) / Control auth rate, when the control rate isn't zero

Report the percentage-point difference too, beside unknown outcomes, sample counts and uncertainty. The formula alone isn't causal proof; the controlled design is.


Stop Rules​

Define these before launch and honor them when they fire, because the moment a stop rule triggers is exactly when you won't want to obey it.

Example Stop Rules​

Choose these limits before launch from your baseline, expected margin impact and loss budget. They are your settings, not industry cutoffs. For any rate difference, write whether you mean percentage points or relative percent.

ConditionAction
Confirmed blocked-good margin exceeds your dollar capPause experiment
Auth rate falls beyond the difference you approved vs. controlPause and investigate
Decline complaints breach the limit set from comparable trafficPause and review
Fraud exposure or blocked-good margin reaches its agreed dollar capRoll back
Critical incident under your incident planImmediate rollback

Rollback Requirements​

One-click rollback or it isn't an experiment. Before launching any experiment:

  • Confirm rollback is one-click (or automated)
  • Test rollback in staging
  • Document rollback procedure
  • Assign rollback authority

Scale Callout​

VolumeApproach
Under $100k/moStart in shadow mode. Low case counts can leave tests inconclusive; sales volume alone doesn't decide significance.
$100k-$1M/moChoose a limited ramp from the loss cap and eligible cohort. Expand only within pre-agreed budgets once the outcome window closes. Document everything.
Over $1M/moRequire rollback switch, alerting, daily review during ramp. Dedicated owner per experiment.

Where This Breaks​

  • No labeled outcomes. If you can't tell good from bad orders, fix tagging first. Without confirmed labels you can't claim fraud detection or false positives. An auth or conversion test can still measure those outcomes while the fraud verdict stays inconclusive.
  • Chargeback feedback lags. Dispute data arrives weeks late. Use alerts, SAFE, TC40 to shorten the loop.
  • Testing during peak periods. Black Friday, promotions and holidays change who's buying. A simultaneous randomized control still reads for that period. Name the period, check assignment and mix for actual bias, and don't export the conclusion to normal weeks unqualified.
  • Multiple simultaneous changes. A bundle measures the bundle. Splitting the credit between its parts needs a design built for it, such as a factorial test. For a first test, isolate one variable.
  • No operator-dev handshake. Engineers deploy, operators don't know. Add "show me the shadow logs" checkpoint.

Common Experimentation Mistakes​

MistakeConsequencePrevention
No shadow periodCan block good orders from day oneShadow decision rules before they enforce
Too short testInconclusive resultsMinimum sample sizes
No stop rulesRunaway false positivesDefine before launch
Multiple changesMeasures the bundle, not its partsOne variable at a time for a first test
No rollback planStuck with bad ruleTest rollback first
Ignoring support signalsCustomer friction unnoticedMonitor tickets

Experimentation Infrastructure​

Minimum Requirements​

  1. Shadow logging - Record shadow decisions separately
  2. Outcome tagging - Link transactions to disputes/refunds
  3. Cohort assignment - Random assignment, held stable: the same customer always lands in the same test group. Consistent assignment alone isn't randomization
  4. Metrics dashboard - Real-time visibility
  5. Alert system - Trigger on stop rule conditions
  6. Rollback mechanism - Quick revert capability

Nice to Have​

  1. Statistical significance calculator - Built into dashboard
  2. Automatic ramping - Gradual traffic increase
  3. Experiment registry - Track all active/past tests
  4. Cross-experiment interference detection - Catch conflicts

Next Steps​

Setting up your first experiment?

  1. Implement shadow mode - Log decisions without blocking
  2. Design the test - Test type, cohort, sample size
  3. Define stop rules - Before you launch

Running a test now?

  1. Follow the test steps - Shadow, analyze, ramp, roll out, once the outcome window has passed
  2. Track key metrics - Primary and secondary
  3. Know when to stop - Honor the rules

Building experimentation infrastructure?

  1. Meet minimum requirements - Shadow logging, tagging, cohorts
  2. Avoid common mistakes - No shadow, too short, no rollback
  3. Scale appropriately - By transaction volume

Something wrong on this page? Email matt at paymentsandrisk.com and say what needs fixing.