Vendor Selection Guide
- The test to start with: Run the vendor on YOUR traffic in shadow mode, where it flags orders but doesn't block any. If it doesn't catch fraud you're actually seeing, it won't help you. It can't show what an order you blocked would have done.
- Before buying: Review a manageable set of disputes by cause and dollars. Match controls to those causes; billing confusion may need descriptors or refund policies, while confirmed fraud needs its own response
- SMBs: a 2-week shadow test. Turning on blocking is a live test, so it needs an owner, a loss cap and an outcome window first.
- Enterprise: Structured 4-round process with proof-of-value on live traffic.
- Your fraud tool's bill changed? Count the billed events first, before you pay for more or switch anything off.
Test the vendor on your own traffic before you sign anything, and confirm first that fraud is the real problem. Here's how to do both without spending three months on it.
Start Here: Do You Actually Need a Vendor?
| Your situation | What to do |
|---|---|
| Current controls meet your fraud-loss budget and written limits; no specific fraud pattern | You don't need a vendor yet. Configure your processor's built-in rules and revisit when something changes. |
| Specific fraud patterns persist after you've tuned your current tools | Run the 2-week shadow test below with one vendor. |
| Processor warning, limit breached, or active fraud attack | You need help now. Confirm the metric and deadline. Check chargeback alert coverage while evaluating a fraud tool. |
| Dedicated fraud team, $10M+ volume | Follow the enterprise evaluation process. |
Your processor's limit and the network program lines are separate checks. Visa enforces through VAMP and Mastercard through ECM. See Monitoring Thresholds for the definitions and conditions.
The bill changed
Your fraud tool, or your processor's screening product, now charges per screened transaction or check, or the charge went up. Put the notice beside your current plan and the first changed invoice. Write down:
- the included allowance, and what counts as one screened event
- the billed event count on the invoice
- any minimum, overage or monthly fee, and the date it starts
- what downgrading or canceling changes
Keep each cost on its own line. Authorization fees, the screening charge, any CAPTCHA or bot-protection fee, and payment or dispute fees are billed for different events. Adding them into one "fraud cost" hides which one moved.
On Stripe, check which Radar tier you're on and its allowance first: The Paid Radar Tiers has Stripe's published list. Your account's price and allowance come from your own plan, not a list.
Then compare the added fee with what you can measure over the same invoice period: confirmed fraud losses you avoided, review time, and good orders you declined. Don't count a vendor's predicted catch as a loss you avoided. If the numbers don't support a decision, call it inconclusive and keep your current controls. You don't need to buy or extend a trial to wait for better evidence.
Use this request:
Which events count as a paid screen on my plan? What's included, what's the overage, and when does this price start? Please show the total using my last complete billing period and explain what changes if I downgrade or cancel.
Scale Callout
A per-screen charge grows with the events your plan bills, not your order count. Take the count from the invoice, not an estimate.
Where This Breaks
A lower dispute count alone doesn't show a tool caused it. Order mix, refunds, declined good orders and dispute timing all move the number. If you can't separate them, the result's inconclusive.
Test to Run
Reconcile one complete invoice period against the plan and the notice: included and billed events, added fees, review hours, confirmed avoided losses and declined good orders. Keep current actions in place while you do. Pass when the event count and the fee reconcile and both sides of the comparison cover the same period.
The SMB Approach (Under $10M)
For a small team, start with three questions. Use the revenue bands as planning examples; procurement needs, fraud exposure and available staff decide how formal the evaluation should be:
-
What specific problem am I solving? (Not "fraud" in general. Be specific: chargebacks approaching your processor's limit? Card testing attacks? High false positive rate?)
-
What does "working" look like? (Chargeback ratio falls below the target you've agreed with your processor? Manual review queue shrinks by half? Specific fraud type stops?)
-
How will I know if it isn't working? (Set the outcome window before you start. Disputes can arrive weeks after the order, so a short test may not prove anything yet. Keep current controls while results mature, and if you can't read them, call it inconclusive.)
The 2-Week Test
For an SMB, this can be the whole evaluation. It doesn't change a single live decision:
Week 1:
- Sign up for trial or request demo access
- Connect the tool (confirm the integration time in the demo. Don't assume hours)
- Run in shadow mode (flag, don't block)
- Check what it flags. Do those transactions look risky to you?
Week 2:
- Keep it in shadow mode. Compare its would-block list with what your current controls did
- Count what it flagged that you'd have let through, and what it flagged that you know was a good customer
- Note the orders to check again once disputes have had time to arrive
Decision:
- If it flags fraud you'd have missed and its would-block list is mostly orders you'd have doubted: keep evaluating, and check those orders again when their disputes have had time to land
- If it flags good customers or misses obvious fraud: walk away
- If you can't tell either way: keep your current controls. That's inconclusive, not a verdict on your volume
Want it blocking live orders? That's a live test. Before you switch it on, write down the owner, the eligible orders, the loss cap, rollback, stop rule and outcome window.
Pull your last 50 chargebacks. Put them in front of the vendor during the demo and ask: "Which of these would you have caught?" If they can't answer specifically, the product is a black box. That's a useful diagnosis, not a performance estimate.
Every Vendor Claims They Stop Fraud
They all have impressive numbers. "95% detection rate." "Blocked $X million in fraud." "Trusted by [big logos]."
None of it tells you whether the tool will work on YOUR traffic.
What works for one merchant can wreck another. A model trained on a different population may over-decline your customers, and a shared signal can be wrong or stale. Treat that as a risk to test. Ask which global or custom model, and which inputs, the vendor uses for your account (Sift's services document, 2025, p. 6, is one example).
You can't trust any tool blindly. You have to test it on your own orders.
Test Without Changing Live Decisions
Don't pass flagged orders through just to build a comparison group, and don't loosen live thresholds to see what happens. Both put real money at risk to answer a question shadow mode answers without touching live orders.
Instead, let your current controls keep deciding while the tool scores the same orders in the background. Later, compare its scores with what actually happened on a defined set of orders you can observe, separating disputes from confirmed fraud. Orders your current controls blocked stay unknown, because you can't see what they'd have done. Judge the tool only after your sample and outcome window (test design, fraud metrics).
If a live split is still worth it, it's a pilot. It needs the live-test safeguards first: one owner, named orders, a loss cap, rollback, a stop rule and an outcome window.
Talk to Your Customers
This is the feedback loop that gets skipped. Don't skip it.
When a tool flags a transaction, you have options beyond "block" or "approve":
- Call the customer
- Email asking a clarifying question
- Request additional verification
What comes back:
"I ordered this for my daughter at college. The different shipping address is her dorm."
"Yes, I used my work VPN. That's why my IP looks weird."
"I disputed that charge at [other merchant] because they never shipped my order. I'm not a fraudster."
That last one is important. A shared risk signal can be wrong or stale. As hypotheticals, someone who filed a valid dispute, or whose stolen card a fraudster used, could end up looking risky. Ask what a signal means and look for a legitimate explanation (SEON on false positives is one provider's view).
Trust that data blindly and you can block good customers over something that happened somewhere else.
Build Feedback Loops
The tool doesn't know whether it was right. You have to tell it.
What to track:
- Every transaction the tool flagged as high risk
- What action you took (blocked, approved, reviewed)
- What actually happened (chargeback, no chargeback, customer complaint)
What to do with that data:
- Calculate its false positive rate on the orders whose outcomes you can see, keeping unknowns separate
- Identify patterns where the tool is wrong (certain products, customer types, geographies)
- Share outcomes with the vendor so they can tune the model
If you don't build the feedback loop, you're flying blind. You'll never know whether the tool is helping or hurting.
Different Business Models, Different Results
A "risky" customer for one merchant is a great customer for another.
- International shipping? Risky for some, normal for others.
- High-value first order? Suspicious for commodity goods, expected for luxury.
- Multiple failed payment attempts? Could be fraud, could be a card limit issue.
- New email address? Suspicious for some, but some customers create new emails for every merchant.
Ask how the vendor's model was trained. Test it against your own customers before you trust it.
When a vendor says "this customer is in our fraud consortium," ask: Why? What did they do? At what merchant?
Denylists are useful signals, not verdicts. Treat them as one input, not a decision.
What to Ask in a 30-Minute Demo
Skip the slides. Ask these questions:
-
"Show me your dashboard. Walk me through a flagged transaction." (Confusing after 5 minutes means confusing forever.)
-
"What data do you need from me to work?" (Some tools need deep integration. Others run off a processor connection alone. Know which one this is.)
-
"What's your pricing at my volume?" (Get a real number, not "it depends.") Expect resistance. The sales pages for Sift, Sardine, Forter, Riskified and Equifax's Kount line show no list price, so you'll need a quote. SEON publishes Starter at $699/month including 2,500 fraud checks. Signifyd describes a variable percentage of approved orders, with no charge on a fraud decline. MaxMind publishes per-query prices, such as $0.005 for minFraud Score. All read 3 November 2026. Compare billing units, features and contract terms, not just the numbers.
-
"Can I run in shadow mode first?" (If no, that's their term. Weigh it against buying blind.)
-
"What happens if I want to leave?" (Data portability, contract terms, exit process. Ask before you sign.)
Red Flags
Walk away if you see any of these:
- Won't let you test on your actual data
- No customers at your size or in your industry
- Won't put the price in writing before a pilot. A custom quote is normal in fraud tooling; refusing to write the quote down isn't
- High-pressure tactics ("this pricing expires Friday")
- Can't explain why a transaction was flagged
- Requires an annual contract with no exit clause
- Volume minimums you can't hit
Before you accept a quote on an older brand, confirm the current seller, support route and product. Kount's and Midigator's domains lead to Equifax, and Accertify's history describes its 2024 carve-out. Those changes alone aren't a reason to reject an accurately described product.
When to Skip Vendors Entirely
Start with your processor's built-in tools. A dedicated vendor earns its place by covering a measured fraud or workload gap, with benefits that beat its full cost. Annual revenue alone doesn't decide that.
Signs you might actually need a vendor:
- Chargeback ratio approaching your processor's limit
- A specific fraud pattern your processor keeps missing
- Manual review queue is drowning your team
- You have a fraud analyst who needs better tools
Signs you do NOT need a vendor yet:
- "Fraud feels like a problem" but you haven't quantified it
- You want to be "proactive" about fraud
- A vendor reached out with a scary pitch
- Your current controls meet your fraud-loss budget and limits
The Enterprise Approach (Over $10M)
If you have a dedicated fraud team, a procurement process and the volume to justify a formal evaluation, here is how larger organizations run vendor selection.
What is an RFP?
RFP stands for Request for Proposal. A formal document you send to several vendors, describing what you need and asking them to write back with how they would meet it and at what price.
Skip it unless one of these applies. The RFP is a procurement tool for organizations that need to:
- Compare 5+ vendors systematically
- Document the selection process for compliance or audit
- Negotiate enterprise contracts with legal review
- Justify the selection to a board or executive team
If none of those apply to you, skip the RFP and use the SMB approach above.
Structured Evaluation Process
For teams that need formal documentation:
Round 1: Requirements and Shortlist (1-2 weeks)
- Document your specific problem, baseline metrics, and success criteria
- Send requirements to 5-8 vendors
- Shortlist to 3-4 based on responses
Round 2: Demos and Technical Review (2-3 weeks)
- Structured demos against your use cases
- Technical architecture review
- Meet the team that will actually work your account
Round 3: Proof of Value (4-8 weeks)
- Run vendor in shadow mode on live traffic
- Compare vendor decisions against confirmed outcomes you can observe, keeping blocked and unresolved orders as unknown
- Measure detection rate and false positive rate once the outcome window closes
Round 4: Commercial Negotiation (2-4 weeks)
- Finalize pricing based on POV results
- Negotiate contract terms
- Reference checks
POC vs POV vs Pilot
These three get thrown around interchangeably. This guide uses them like this. Whatever the label, write down each test's data, whether anything changes live, who owns the decision and what counts as a result:
| Term | What It Is | When to Use |
|---|---|---|
| POC (Proof of Concept) | Vendor analyzes your historical data offline. Shows what they "would have" caught. | Quick filtering. |
| POV (Proof of Value) | Vendor runs on live traffic in shadow mode. You compare their decisions to your outcomes. | Checking performance before commitment, without changing live decisions. |
| Pilot | Vendor is live in production, making real decisions. | Optional final validation. Requires contract negotiation upfront, and the live-test safeguards before it starts. If the result is inconclusive, keep current controls. |
Recommendation: be careful with a POC, because cherry-picking results out of historical data is too easy. A POV gets you live traffic in shadow mode, though it still can't show what a blocked order would have done. Only approve a pilot after the risk owner accepts its orders, loss cap, rollback, stop rule and outcome window.
Evaluation Criteria
These weights are an illustrative starting scorecard and total 100%. Choose your own before comparing vendors. Use the Fraud Metrics definitions, with unknown outcomes separate.
| Criterion | Weight | What to Measure |
|---|---|---|
| Detection Accuracy | 25% | What percentage of known fraud did they catch? |
| False Positive Rate | 25% | Confirmed legitimate / All fraud blocks or would-blocks; report unknowns |
| Integration Effort | 20% | How long to implement? What resources required? |
| Total Cost | 15% | Per-transaction cost, implementation fees, ongoing support |
| Support Quality | 10% | Responsiveness, expertise, account management |
| Contract Terms | 5% | Exit provisions, data portability, price escalation |
Adjust the weights to your situation. If integration effort is your real constraint, weight it higher. If accuracy matters more than price, weight cost lower.
Contract Negotiation Points
Things to negotiate before signing:
- Performance guarantees: Can they commit to a detection rate? What happens if they miss?
- Pilot pricing: Lock in pricing from pilot through production
- Exit terms: What happens to your data? How long to transition out?
- Price escalation: What triggers increases? Cap annual increases.
- SLA credits: Real credits for downtime, not just apologies
Implementation Planning
For enterprise implementations:
- Assign an owner: One person accountable for success
- Phase any live change: Shadow mode, then a share of traffic, then more. A percentage ramp isn't a safeguard on its own: name the owner, eligible orders, loss cap, rollback, stop rule and outcome window first (the checklist). Don't expand before outcomes mature
- Run parallel: Keep old system running until new system is proven
- Plan for tuning: Launch is the beginning, not the end. Budget time for ongoing optimization.
- Set review cadence: Weekly during implementation, monthly after launch
Fraud Profile by Business Type
Two businesses the same size can face completely different fraud. Work out what you're actually fighting before you buy anything.
| Business Type | Primary Fraud | Secondary Fraud | Check First | What decides whether a tool pays |
|---|---|---|---|---|
| E-commerce (physical goods) | Third-party fraud (stolen cards) | Deliberately false nonreceipt or not-as-described claims. Genuine delivery and product complaints are service disputes (fraud classification) | Alert coverage, then processor rules for the cause | Avoided losses and review time vs. fees and blocked-good margin |
| SaaS / Subscriptions | Friendly fraud and billing confusion | Card testing on trial signups | Billing descriptors and refund policy | Billing-related disputes separately from fraud |
| Digital goods / Gaming | Third-party fraud + instant resale | Card testing, promo abuse | Current rules and missed patterns | Loss avoided on the product vs. friction and tool cost |
| Service businesses | Friendly fraud and billing disputes | Third-party fraud | Contracts, communication and documentation | Which losses a tool could actually prevent |
| Marketplaces | Seller fraud + buyer fraud | Account takeover | Controls on both sides and identity checks | Buyer and seller exposure, review hours and applicable requirements |
| B2B / Invoice | BEC / wire fraud | Fake invoices | Dual authorization and independent verification | Exposure from failed process controls, not a sales cutoff |
How to read this table:
- Primary fraud is the type to check first in your own losses.
- Check First names a control to evaluate against your own losses.
- What decides whether a tool pays lists the inputs that do: your loss rate, fees, margin and coverage. Revenue doesn't supply them.
Service businesses and B2B companies: start operational before software. Their fraud problems are operational: vague contracts, thin documentation, BEC aimed at accounts payable. The fixes are process. If customers dispute because they didn't understand the scope of work, a fraud scoring tool is an expensive way to not solve that.
If you're a marketplace: you're fighting fraud on both sides of the transaction. Buyer fraud looks like e-commerce fraud. Seller fraud looks like application fraud and synthetic identity. Price a dedicated platform against the measured losses on both sides.
The Question That Matters Most
Before you evaluate any vendor, answer this honestly:
Do you actually know why you're getting chargebacks?
Pull your last 50 disputes. Categorize them:
- True fraud (stolen cards)
- Friendly fraud (customer lying)
- Service issues (product problems, shipping delays)
- Billing confusion (didn't recognize the charge)
Work out the causes and the dollars. A fraud tool won't fix most billing or service disputes, but a small share of costly fraud can still justify one, and some products describe first-party abuse detection, as the provider's own claim. Check what the product covers. For the rest, you might need:
- Better billing descriptors (for recognition issues)
- Better customer service (for service issues)
- Better fulfillment (for shipping issues)
- Better product (for quality issues)
Fraud tools solve fraud problems. Make sure you actually have a fraud problem.
Next Steps
- Prevent Fraud: The fraud hub
Fraud tool's bill changed?
- Count the billed events: Allowance, overage, start date
- Check your Radar tier: If you're on Stripe
- Keep current controls while you decide: Inconclusive is allowed
SMB evaluating vendors?
- Answer the three questions: Problem, success, failure
- Run the 2-week shadow test: No live decisions change
- Know when to skip vendors: Check whether a measured fraud or workload gap justifies one
Enterprise running formal evaluation?
- Follow structured process: Rounds 1-4
- Understand POC vs POV vs Pilot: Which to use
- Negotiate contract terms: Performance guarantees, exit
Testing vendor claims?
- Test without changing live decisions: Shadow scores against real outcomes
- Build feedback loops: Track outcomes
- Ask the right demo questions: Force specifics
Related
- Fraud Vendors Overview for who to evaluate
- Data Enrichment for standalone IP, email, phone APIs
- Chargeback Vendors for dispute-specific tools
- Risk Scoring for understanding what vendors are doing under the hood