Skip to main content

Payments Experimentation (Operator Field Manual)

On this page

Most merchants flip fraud rules live without testing, then scramble when good orders die. Shadow first, then enforce. Treat payment rules like code deploys: stage, monitor, promote.

Last verified: Dec 2025. Experimentation frameworks evolve; adapt to your stack.

What Matters (5 bullets)​

  • Shadow mode first. Log decisions, do not block. Measure false positives before going live.
  • Pick a single success metric per test. Auth lift, fraud rate, CX impact. Not all three at once.
  • Run by cohort. Method, BIN, country, device, CP vs CNP. Never test on all traffic.
  • Set stop rules before launch. Max false positive %, max revenue at risk. Honor them.
  • Feedback loops lag. Use alerts/SAFE/TC40 to shorten the chargeback feedback delay.

Shadow Mode: The Foundation​

Shadow mode runs your new rule in parallel without enforcing it. Every transaction gets two decisions: actual (what happened) and shadow (what would have happened).

How to Implement Shadow Mode​

  1. Log both decisions - Actual outcome + shadow rule outcome
  2. Tag transactions - Mark "would-have-blocked" for tracking
  3. Don't affect the customer - Shadow decisions are invisible
  4. Track over time - 7-14 days minimum

What to Measure in Shadow​

MetricWhat It Tells You
Would-block rateHow aggressive is the new rule
False positive rateGood orders that would have been blocked
True positive rateBad orders correctly caught
CoverageWhat % of fraud would this catch

Shadow Decision Matrix​

Actual OutcomeShadow DecisionInterpretation
Approved, no disputeWould blockFalse positive (bad)
Approved, disputedWould blockTrue positive (good)
Approved, no disputeWould allowCorrect allow
Approved, disputedWould allowMissed fraud

Experiment Design​

Test Types​

Test TypeWhat You're TestingKey Metric
Fraud rule tighteningNew velocity limitBlock rate vs fraud rate
Fraud rule looseningRelaxing a ruleAuth rate vs fraud increase
3DS thresholdWhen to challengeConversion vs liability shift
Auth retryDecline handlingRecovery rate vs cost
Checkout flowPayment form changesConversion rate

Cohort Selection​

Never test on all traffic. Pick cohorts that:

  • Are large enough for statistical significance (500+ decisions)
  • Represent meaningful segments
  • Can be isolated

Good cohorts:

  • Geographic (US vs EU vs APAC)
  • Payment method (card vs wallet)
  • Transaction type (CP vs CNP)
  • BIN range (specific issuers)
  • Device type (mobile vs desktop)
  • Customer type (new vs returning)

Sample Size Guidelines​

Decision VolumeMinimum Test DurationNotes
Under 100/day2-4 weeksMay be inconclusive
100-500/day1-2 weeksStandard test period
500-2000/day3-7 daysFaster feedback
Over 2000/day1-3 daysCan iterate quickly

Test to Run (2-4 weeks)​

Week 1: Shadow Phase​

  1. Choose one rule change - Example: tighten velocity on CNP high-risk BINs
  2. Implement shadow logging - Log would-block decisions
  3. Tag approved transactions - Mark those that would have been blocked
  4. Monitor daily - Check false positive rate

Week 2: Analysis​

  1. Calculate false positives - Good orders that would have blocked
  2. Calculate true positives - Fraud/disputes that would have caught
  3. Assess impact - Revenue at risk vs fraud prevented
  4. Decide: proceed, modify, or abandon

Week 3: Ramp (if proceeding)​

  1. Enable on 10-25% of traffic - Real enforcement, limited scope
  2. Monitor hourly - Watch for unexpected blocks
  3. Check customer support - Any complaints about declines?
  4. Compare to control - Does reality match shadow?

Week 4: Full Rollout (if successful)​

  1. Roll to 100% - Only if Week 3 metrics are stable
  2. Document baseline - New normal for this rule
  3. Set ongoing alerts - Detect drift from baseline
  4. Plan next experiment - Continuous improvement

Metrics to Track​

Primary Metrics (choose one per test)​

MetricDefinitionTarget Direction
Auth rateApproved / AttemptedHigher is better
Block rateBlocked / AttemptedLower is usually better
Fraud rateDisputes / ApprovedLower is better
ConversionCompleted / StartedHigher is better

Secondary Metrics (monitor, don't optimize)​

MetricWhy Track It
False positive rateCatch good-order blocking
Support ticketsDetect customer friction
Revenue per attemptNet effect on business
Soft vs hard decline mixUnderstand decline sources

Analyst Calculations​

Block rate = Blocked transactions / Total attempts
False positive rate = (Would-block AND no dispute) / Would-block
True positive rate = (Would-block AND disputed) / Total disputed
Lift = (New auth rate - Baseline auth rate) / Baseline auth rate

Stop Rules​

Define before launch. Honor when triggered.

Example Stop Rules​

ConditionAction
False positive rate > 2%Pause experiment
Auth rate drops > 1% vs controlInvestigate
Support tickets spike 2xPause and review
Revenue at risk > $XRollback
Any P0 incidentImmediate rollback

Rollback Requirements​

Before launching any experiment:

  • Confirm rollback is one-click (or automated)
  • Test rollback in staging
  • Document rollback procedure
  • Assign rollback authority

Scale Callout​

VolumeApproach
Under $100k/moShadow only; avoid live blocks. Use alerts for spikes. No statistical significance for small tests.
$100k-$1M/moShadow → 25% ramp → full if false positives under 1%. Document everything.
Over $1M/moRequire rollback switch, alerting, daily review during ramp. Dedicated owner per experiment.

Where This Breaks​

  • No labeled outcomes. If you can't tell good from bad orders, fix tagging first. No experimentation without truth labels.
  • Chargeback feedback lags. 30-90 day delay on dispute data. Use alerts, SAFE, TC40 to shorten the loop.
  • Testing during peak periods. Black Friday, promotions, holidays skew results. Avoid or heavily caveat.
  • Multiple simultaneous changes. Can't attribute results. Isolate one variable per test.
  • No operator-dev handshake. Engineers deploy, operators don't know. Add "show me the shadow logs" checkpoint.

Common Experimentation Mistakes​

MistakeConsequencePrevention
No shadow periodBlocked good orders immediatelyAlways shadow first
Too short testInconclusive resultsMinimum sample sizes
No stop rulesRunaway false positivesDefine before launch
Multiple changesCan't attribute resultsOne variable at a time
No rollback planStuck with bad ruleTest rollback first
Ignoring support signalsCustomer friction unnoticedMonitor tickets

Experimentation Infrastructure​

Minimum Requirements​

  1. Shadow logging - Record shadow decisions separately
  2. Outcome tagging - Link transactions to disputes/refunds
  3. Cohort assignment - Deterministic customer/transaction bucketing
  4. Metrics dashboard - Real-time visibility
  5. Alert system - Trigger on stop rule conditions
  6. Rollback mechanism - Quick revert capability

Nice to Have​

  1. Statistical significance calculator - Built into dashboard
  2. Automatic ramping - Gradual traffic increase
  3. Experiment registry - Track all active/past tests
  4. Cross-experiment interference detection - Catch conflicts

Next Steps​

Setting up your first experiment?

  1. Implement shadow mode - Log decisions without blocking
  2. Design the test - Test type, cohort, sample size
  3. Define stop rules - Before you launch

Running a test now?

  1. Follow the 2-4 week timeline - Shadow, analyze, ramp, rollout
  2. Track key metrics - Primary and secondary
  3. Know when to stop - Honor the rules

Building experimentation infrastructure?

  1. Meet minimum requirements - Shadow logging, tagging, cohorts
  2. Avoid common mistakes - No shadow, too short, no rollback
  3. Scale appropriately - By transaction volume