Vendor Selection Guide
- The only test that matters: Run the vendor on YOUR traffic in shadow mode. If it doesn't catch fraud you're actually seeing, it won't help you.
- Before you buy anything: Pull your last 50 chargebacks. If most aren't true fraud (stolen cards), a fraud tool won't fix your problem. You may need better billing descriptors or refund policies instead.
- SMBs: 2-week test with a kill-switch date. No multi-month evaluations needed.
- Enterprise: Structured 4-round process with proof-of-value on live traffic.
Test the vendor on your own traffic before you sign anything, and confirm first that fraud is the real problem. Most of what follows is how to do those two things without spending three months on it.
Start Here: Do You Actually Need a Vendor?
| Your situation | What to do |
|---|---|
| Chargeback ratio under 0.3%, no specific fraud pattern | You don't need a vendor yet. Configure your processor's built-in rules and revisit when something changes. |
| Chargeback ratio 0.3%-0.65%, seeing specific fraud patterns | Run the 2-week test below with one vendor. |
| Chargeback ratio above 0.65% or under active fraud attack | You need help now. Start with chargeback alerts (same-day impact) while evaluating a fraud tool. |
| Dedicated fraud team, $10M+ volume | Follow the enterprise evaluation process. |
The 0.65% line in that table is an internal warning marker, not a network one. Visa enforces through VAMP and Mastercard through ECM, both of which start higher. See dispute monitoring thresholds for the current numbers.
The SMB Approach (Under $10M)
Under $10M in annual volume, you don't need a formal evaluation process. You need to answer three questions:
-
What specific problem am I solving? (Not "fraud" in general. Be specific: chargebacks approaching 0.65%? Card testing attacks? High false positive rate?)
-
What does "working" look like? (Chargeback ratio drops below 0.5%? Manual review queue shrinks by half? Specific fraud type stops?)
-
How will I know if it isn't working? (Set a deadline. If the tool hasn't moved your key metric in 30 days, kill it.)
The 2-Week Test
For most SMBs, this is the entire evaluation process:
Week 1:
- Sign up for trial or request demo access
- Connect the tool (most modern tools take hours, not weeks)
- Run in shadow mode (flag, don't block)
- Check what it flags. Do those transactions look risky to you?
Week 2:
- Turn on blocking for one segment (high-risk orders, new customers, orders over $X)
- Watch your false positive rate (customer complaints, support tickets)
- Check: Are you blocking real fraud or annoying good customers?
Decision:
- If it caught fraud you'd have missed and the false positives are tolerable: keep it
- If it blocks good customers or misses obvious fraud: kill it
- If you can't tell either way: you lack the volume for this tool
Pull your last 50 chargebacks. Put them in front of the vendor during the demo and ask: "Which of these would you have caught?" If they can't answer specifically, the product is a black box.
Every Vendor Claims They Stop Fraud
They all have impressive numbers. "95% detection rate." "Blocked $X million in fraud." "Trusted by [big logos]."
None of it tells you whether the tool will work on YOUR traffic.
What works for one merchant wrecks another. A tool tuned for high-volume commodity e-commerce will over-decline a luxury brand. A tool trained on US fraud patterns will false-positive your international customers. A consortium that flagged someone for disputing a scammy merchant will block them from your legitimate store.
You can't trust any tool blindly. You have to test it on your own orders.
Test By NOT Taking Action
The best way to evaluate a fraud tool is to ignore some of what it tells you. Don't act on everything it flags.
Experiment 1: Split your flagged transactions
- 50% of flagged orders: follow the tool's recommendation (decline, void, or refund)
- 50% of flagged orders: let them through anyway
- Wait 60 days. Compare chargeback rates between the two groups.
If the tool is accurate, the "let through" group comes back with higher chargebacks. If the two rates look similar, the tool is selling you false positives.
Experiment 2: Vary your thresholds
- Score 80+: Block and measure what happens
- Score 60-79: Let through and track outcomes
- Score 40-59: Let through and track outcomes
You will find where the tool's accuracy actually lives. Usually a narrower band than the vendor claims.
Experiment 3: Do nothing on some cases
- Pick a random sample of "high risk" flagged orders
- No decline. No preemptive refund. No void.
- Wait and see whether the chargebacks show up
This tells you the real false positive rate, not the vendor's claimed rate.
Talk to Your Customers
This is the feedback loop most merchants skip.
When a tool flags a transaction, you have options beyond "block" or "approve":
- Call the customer
- Email asking a clarifying question
- Request additional verification
What comes back:
"I ordered this for my daughter at college. The different shipping address is her dorm."
"Yes, I used my work VPN. That's why my IP looks weird."
"I disputed that charge at [other merchant] because they never shipped my order. I'm not a fraudster."
That last one is important. Consortium data and denylists are full of legitimate customers. Someone who filed a valid dispute against a bad merchant is now flagged in shared databases. Someone whose card was stolen and used by a fraudster is now associated with fraud. Someone who had a billing dispute with their cable company is now "high risk."
Trust denylist data blindly and you're blocking good customers over something that happened at a different merchant.
Build Feedback Loops
The tool doesn't know whether it was right. You have to tell it.
What to track:
- Every transaction the tool flagged as high risk
- What action you took (blocked, approved, reviewed)
- What actually happened (chargeback, no chargeback, customer complaint)
What to do with that data:
- Calculate the tool's real false positive rate on YOUR traffic
- Identify patterns where the tool is wrong (certain products, customer types, geographies)
- Share outcomes with the vendor so they can tune the model
If you don't build the feedback loop, you're flying blind. You'll never know whether the tool is helping or hurting.
Different Business Models, Different Results
A "risky" customer for one merchant is a great customer for another.
- International shipping? Risky for some, normal for others.
- High-value first order? Suspicious for commodity goods, expected for luxury.
- Multiple failed payment attempts? Could be fraud, could be a card limit issue.
- New email address? Suspicious for some, but some customers create new emails for every merchant.
The vendor's model was trained on aggregate data. Your business isn't aggregate. Test everything against your own customers before you trust it.
When a vendor says "this customer is in our fraud consortium," ask: Why? What did they do? At what merchant?
A customer who disputed a legitimate fraud at a sketchy merchant is now "high risk" in shared databases. A customer whose card was compromised is now associated with fraud. A customer who had a billing dispute is now flagged.
Denylists are useful signals, not verdicts. Treat them as one input, not a decision.
What to Ask in a 30-Minute Demo
Skip the slides. Ask these questions:
-
"Show me your dashboard. Walk me through a flagged transaction." (Confusing after 5 minutes means confusing forever.)
-
"What data do you need from me to work?" (Some tools need deep integration. Others run off a processor connection alone. Know which one this is.)
-
"What's your pricing at my volume?" (Get a real number, not "it depends.") Expect resistance, because almost nobody in this market publishes rates. Sift, Sardine, Forter, Riskified and Kount were all checked on 2 August 2026 and none of them publishes a price. Signifyd publishes the model without the numbers: a percentage of the order total on approved orders, nothing charged when an order is declined for fraud. The one vendor with public pricing is SEON, at $699/month for 2,500 fraud checks, or roughly 28 cents a check (seon.io/pricing). Quote that number back at anyone who won't give you one, and ask what you get for the difference.
-
"Can I run in shadow mode first?" (If no, walk away. Any vendor worth buying lets you test without going live.)
-
"What happens if I want to leave?" (Data portability, contract terms, exit process. Ask before you sign.)
Red Flags
Walk away if you see any of these:
- Won't let you test on your actual data
- No customers at your size or in your industry
- Won't put the price in writing before a pilot. A custom quote is normal in fraud tooling; refusing to write the quote down isn't
- High-pressure tactics ("this pricing expires Friday")
- Can't explain why a transaction was flagged
- Requires an annual contract with no exit clause
- Volume minimums you can't hit
- Quotes you a brand that has been absorbed. Check the vendor's own domain first: as of 2 August 2026, midigator.com and kount.com both redirect into Equifax, and Accertify's own site describes its carve-out from American Express as completed
When to Skip Vendors Entirely
Under $1M in annual volume, you don't need a dedicated fraud vendor. Your processor's built-in tools - Stripe Radar, Adyen Risk - are almost always enough.
Signs you might actually need a vendor:
- Chargeback ratio approaching 0.65%
- A specific fraud pattern your processor keeps missing
- Manual review queue is drowning your team
- You have a fraud analyst who needs better tools
Signs you do NOT need a vendor yet:
- "Fraud feels like a problem" but you haven't quantified it
- You want to be "proactive" about fraud
- A vendor reached out with a scary pitch
- Your chargeback ratio is under 0.3%
The Enterprise Approach (Over $10M)
If you have a dedicated fraud team, a procurement process and the volume to justify a formal evaluation, here is how larger organizations run vendor selection.
What is an RFP?
RFP stands for Request for Proposal. A formal document you send to several vendors, describing what you need and asking them to write back with how they would meet it and at what price.
Most SMBs will never issue one. The RFP is a procurement tool for organizations that need to:
- Compare 5+ vendors systematically
- Document the selection process for compliance or audit
- Negotiate enterprise contracts with legal review
- Justify the selection to a board or executive team
If none of those apply to you, skip the RFP and use the SMB approach above.
Structured Evaluation Process
For teams that need formal documentation:
Round 1: Requirements and Shortlist (1-2 weeks)
- Document your specific problem, baseline metrics, and success criteria
- Send requirements to 5-8 vendors
- Shortlist to 3-4 based on responses
Round 2: Demos and Technical Review (2-3 weeks)
- Structured demos against your use cases
- Technical architecture review
- Meet the team that will actually work your account
Round 3: Proof of Value (4-8 weeks)
- Run vendor in shadow mode on live traffic
- Compare vendor decisions against your outcomes
- Measure detection rate and false positive rate
Round 4: Commercial Negotiation (2-4 weeks)
- Finalize pricing based on POV results
- Negotiate contract terms
- Reference checks
POC vs POV vs Pilot
These three get thrown around interchangeably. They aren't the same thing:
| Term | What It Is | When to Use |
|---|---|---|
| POC (Proof of Concept) | Vendor analyzes your historical data offline. Shows what they "would have" caught. | Quick filtering. Low effort but also low signal. |
| POV (Proof of Value) | Vendor runs on live traffic in shadow mode. You compare their decisions to your outcomes. | Validating performance before commitment. Best signal for most evaluations. |
| Pilot | Vendor is live in production, making real decisions. | Final validation. Requires contract negotiation upfront. |
Recommendation: skip the POC. Cherry-picking results out of historical data is too easy. POV on live traffic is the one that gives you real signal.
Evaluation Criteria
| Criterion | Weight | What to Measure |
|---|---|---|
| Detection Accuracy | 25% | What percentage of known fraud did they catch? |
| False Positive Rate | 25% | What percentage of good transactions did they block? |
| Integration Effort | 20% | How long to implement? What resources required? |
| Total Cost | 15% | Per-transaction cost, implementation fees, ongoing support |
| Support Quality | 10% | Responsiveness, expertise, account management |
| Contract Terms | 5% | Exit provisions, data portability, price escalation |
Adjust the weights to your situation. If integration effort is your real constraint, weight it higher. If accuracy matters more than price, weight cost lower.
Contract Negotiation Points
Things to negotiate before signing:
- Performance guarantees: Can they commit to a detection rate? What happens if they miss?
- Pilot pricing: Lock in pricing from pilot through production
- Exit terms: What happens to your data? How long to transition out?
- Price escalation: What triggers increases? Cap annual increases.
- SLA credits: Real credits for downtime, not just apologies
Implementation Planning
For enterprise implementations:
- Assign an owner: One person accountable for success
- Phase the rollout: Shadow mode → 10% of traffic → 50% → 100%
- Run parallel: Keep old system running until new system is proven
- Plan for tuning: Launch is the beginning, not the end. Budget time for ongoing optimization.
- Set review cadence: Weekly during implementation, monthly after launch
Fraud Profile by Business Type
Two businesses the same size can face completely different fraud. Work out what you're actually fighting before you buy anything.
| Business Type | Primary Fraud | Secondary Fraud | Buy First | ROI Breakeven |
|---|---|---|---|---|
| E-commerce (physical goods) | Third-party fraud (stolen cards) | Friendly fraud (INR, SNAD claims) | Chargeback alerts, then tune processor rules | ~$500K/yr revenue |
| SaaS / Subscriptions | Friendly fraud (60-80% of disputes) | Card testing on trial signups | Better billing descriptors + clear refund policy | ~$200K/yr revenue |
| Digital goods / Gaming | Third-party fraud + instant resale | Card testing, promo abuse | Fraud platform (Sardine, Kount, Sift) | ~$300K/yr revenue |
| Service businesses | Friendly fraud (90%+ of disputes) | Rare third-party fraud | Better contracts + communication + documentation habits | Rarely worth a tool |
| Marketplaces | Seller fraud + buyer fraud | Account takeover | Fraud platform + identity verification | ~$1M/yr GMV |
| B2B / Invoice | BEC / wire fraud | Fake invoice schemes | Process controls (dual auth, verification), not software | N/A - process, not tools |
How to read this table:
- Primary fraud is what actually hurts you. Most of your losses come from that one column.
- Buy First is the highest-impact move for your business type. Half of them are process changes, not purchases.
- ROI Breakeven is roughly the annual revenue where a dedicated tool starts paying for itself. Below it, your processor's built-in tools are enough.
Service businesses and B2B companies almost never need fraud detection software. Their fraud problems are operational - vague contracts, thin documentation, BEC aimed at accounts payable - and the fixes are process. If customers dispute because they didn't understand the scope of work, a fraud scoring tool is an expensive way to not solve that.
If you're a marketplace: you have the hardest version of this, fighting fraud on both sides of the transaction. Buyer fraud looks like e-commerce fraud. Seller fraud looks like application fraud and synthetic identity. Expect to need a dedicated platform earlier than the table suggests.
The Question That Matters Most
Before you evaluate any vendor, answer this honestly:
Do you actually know why you're getting chargebacks?
Pull your last 50 disputes. Categorize them:
- True fraud (stolen cards)
- Friendly fraud (customer lying)
- Service issues (product problems, shipping delays)
- Billing confusion (didn't recognize the charge)
If most of them aren't true fraud, a fraud tool won't fix your problem. You might need:
- Better billing descriptors (for recognition issues)
- Better customer service (for service issues)
- Better fulfillment (for shipping issues)
- Better product (for quality issues)
Fraud tools solve fraud problems. Make sure you actually have a fraud problem.
Next Steps
SMB evaluating vendors?
- Answer the three questions - Problem, success, failure
- Run the 2-week test - Shadow mode then limited rollout
- Know when to skip vendors - Under $1M may not need
Enterprise running formal evaluation?
- Follow structured process - Rounds 1-4
- Understand POC vs POV vs Pilot - Which to use
- Negotiate contract terms - Performance guarantees, exit
Testing vendor claims?
- Design experiments - Split flagged transactions
- Build feedback loops - Track outcomes
- Ask the right demo questions - Force specifics
Related
- Fraud Vendors Overview for who to evaluate
- Data Enrichment for standalone IP, email, phone APIs
- Chargeback Vendors for dispute-specific tools
- Risk Scoring for understanding what vendors are doing under the hood