Vendor Selection Guide
Most vendor selection advice is written for enterprise procurement teams running formal RFP processes. If you have 2 hours a week for fraud, that is not you.
This page gives you the SMB approach first, then covers the enterprise process for teams that actually need it.
The SMB Approach (Under $10M)
If you are under $10M in annual volume, you do not need a formal evaluation process. You need to answer three questions:
-
What specific problem am I solving? (Not "fraud" in general. Be specific: chargebacks approaching 0.65%? Card testing attacks? High false positive rate?)
-
What does "working" look like? (Chargeback ratio drops below 0.5%? Manual review queue shrinks by half? Specific fraud type stops?)
-
How will I know if this is not working? (Set a deadline. If the tool has not moved your key metric in 30 days, kill it.)
The 2-Week Test
For most SMBs, this is the entire evaluation process:
Week 1:
- Sign up for trial or request demo access
- Connect the tool (most modern tools take hours, not weeks)
- Run in shadow mode (flag but do not block)
- Check: What is it flagging? Do those transactions look risky to you?
Week 2:
- Turn on blocking for one segment (high-risk orders, new customers, orders over $X)
- Watch your false positive rate (customer complaints, support tickets)
- Check: Are you blocking real fraud or annoying good customers?
Decision:
- If it caught fraud you would have missed with acceptable false positives: keep it
- If it is blocking good customers or missing obvious fraud: kill it
- If you cannot tell: you do not have enough volume for this tool yet
Pull your last 50 chargebacks. Show them to the vendor during a demo. Ask: "Which of these would you have caught?" If they cannot answer specifically, they are selling you a black box.
Every Vendor Claims They Stop Fraud
They all have impressive numbers. "95% detection rate." "Blocked $X million in fraud." "Trusted by [big logos]."
None of that tells you if it will work for YOUR business.
The uncomfortable truth: what works for one merchant might destroy another. A tool tuned for high-volume commodity e-commerce will over-decline a luxury brand. A tool trained on US fraud patterns will false-positive international customers. A consortium that flagged someone for disputing a scammy merchant will block them from your legitimate store.
You cannot trust any tool blindly. You have to test it yourself.
Test By NOT Taking Action
The best way to evaluate a fraud tool is counterintuitive: do not act on everything it flags.
Experiment 1: Split your flagged transactions
- 50% of flagged orders: follow the tool's recommendation (decline, void, or refund)
- 50% of flagged orders: let them through anyway
- Wait 60 days. Compare chargeback rates between the two groups.
If the tool is accurate, the "let through" group should have higher chargebacks. If the rates are similar, the tool is generating false positives.
Experiment 2: Vary your thresholds
- Score 80+: Block and measure what happens
- Score 60-79: Let through and track outcomes
- Score 40-59: Let through and track outcomes
You will find where the tool's accuracy actually lives. Often it is much narrower than vendors claim.
Experiment 3: Do nothing on some cases
- Pick a random sample of "high risk" flagged orders
- Do not decline. Do not refund preemptively. Do not void.
- Wait for actual chargebacks (or lack thereof)
This tells you the real false positive rate, not the vendor's claimed rate.
Talk to Your Customers
This is the feedback loop most merchants skip.
When a tool flags a transaction, you have options beyond "block" or "approve":
- Call the customer
- Email asking a clarifying question
- Request additional verification
What you will learn:
"I ordered this for my daughter at college. Different shipping address is her dorm."
"Yes, I used my work VPN. That is why my IP looks weird."
"I disputed that charge at [other merchant] because they never shipped my order. I am not a fraudster."
That last one is important. Consortium data and denylists are full of legitimate customers. Someone who filed a valid dispute against a bad merchant is now flagged in shared databases. Someone whose card was stolen and used by a fraudster is now associated with fraud. Someone who had a billing dispute with their cable company is now "high risk."
If you blindly trust denylist data, you are blocking good customers because of something that happened at a completely different merchant.
Build Feedback Loops
The tool does not know if it was right. You have to tell it.
What to track:
- Every transaction the tool flagged as high risk
- What action you took (blocked, approved, reviewed)
- What actually happened (chargeback, no chargeback, customer complaint)
What to do with that data:
- Calculate the tool's real false positive rate on YOUR traffic
- Identify patterns where the tool is wrong (certain products, customer types, geographies)
- Share outcomes with the vendor so they can tune the model
If you do not build this feedback loop, you are flying blind. You will never know if the tool is helping or hurting.
Different Business Models, Different Results
A "risky" customer for one merchant is a great customer for another.
- International shipping? Risky for some, normal for others.
- High-value first order? Suspicious for commodity goods, expected for luxury.
- Multiple failed payment attempts? Could be fraud, could be a card limit issue.
- New email address? Suspicious for some, but some customers create new emails for every merchant.
The vendor's model was trained on aggregate data. Your business is not aggregate. Test everything against your specific customer base before trusting it.
When a vendor says "this customer is in our fraud consortium," ask: Why? What did they do? At what merchant?
A customer who disputed a legitimate fraud at a sketchy merchant is now "high risk" in shared databases. A customer whose card was compromised is now associated with fraud. A customer who had a billing dispute is now flagged.
Denylists are useful signals, not verdicts. Treat them as one input, not a decision.
What to Ask in a 30-Minute Demo
Skip the slides. Ask these questions:
-
"Show me your dashboard. Walk me through a flagged transaction." (If the dashboard is confusing in 5 minutes, it will be confusing forever.)
-
"What data do you need from me to work?" (Some tools need deep integration. Others work with just a payment processor connection. Know the difference.)
-
"What is your pricing at my volume?" (Get a real number, not "it depends." If they will not give you a number, they are hiding something.)
-
"Can I run in shadow mode first?" (If no, walk away. Any vendor worth buying lets you test without going live.)
-
"What happens if I want to leave?" (Data portability, contract terms, exit process. Ask before you sign.)
Red Flags
Walk away if you see any of these:
- Will not let you test on your actual data
- No customers at your size or in your industry
- Pricing requires a "custom quote" at low volume
- High-pressure tactics ("this pricing expires Friday")
- Cannot explain why a transaction was flagged
- Requires annual contract with no exit clause
- Volume minimums you cannot hit
When to Skip Vendors Entirely
If you are under $1M in annual volume, you probably do not need a dedicated fraud vendor. Your processor's built-in tools (Stripe Radar, Adyen Risk) are almost always enough.
Signs you might actually need a vendor:
- Chargeback ratio approaching 0.65%
- Specific fraud pattern your processor is not catching
- Manual review queue is overwhelming your team
- You have a fraud analyst who needs better tools
Signs you do NOT need a vendor yet:
- "Fraud feels like a problem" but you have not quantified it
- You want to be "proactive" about fraud
- A vendor reached out with a scary pitch
- Your chargeback ratio is under 0.3%
The Enterprise Approach (Over $10M)
If you have a dedicated fraud team, procurement process, and the volume to justify formal evaluation, here is how larger organizations approach vendor selection.
What is an RFP?
RFP stands for Request for Proposal. It is a formal document you send to multiple vendors describing your requirements, asking them to respond with how they would meet those requirements and at what price.
Most SMBs will never issue an RFP. It is a procurement tool for organizations that need to:
- Compare 5+ vendors systematically
- Document the selection process for compliance or audit
- Negotiate enterprise contracts with legal review
- Justify the selection to a board or executive team
If none of those apply to you, skip the RFP and use the SMB approach above.
Structured Evaluation Process
For teams that need formal documentation:
Round 1: Requirements and Shortlist (1-2 weeks)
- Document your specific problem, baseline metrics, and success criteria
- Send requirements to 5-8 vendors
- Shortlist to 3-4 based on responses
Round 2: Demos and Technical Review (2-3 weeks)
- Structured demos against your use cases
- Technical architecture review
- Meet the team you will work with
Round 3: Proof of Value (4-8 weeks)
- Run vendor in shadow mode on live traffic
- Compare vendor decisions against your outcomes
- Measure detection rate and false positive rate
Round 4: Commercial Negotiation (2-4 weeks)
- Finalize pricing based on POV results
- Negotiate contract terms
- Reference checks
POC vs POV vs Pilot
These terms get thrown around. Here is what they actually mean:
| Term | What It Is | When to Use |
|---|---|---|
| POC (Proof of Concept) | Vendor analyzes your historical data offline. Shows what they "would have" caught. | Quick filtering. Low effort but also low signal. |
| POV (Proof of Value) | Vendor runs on live traffic in shadow mode. You compare their decisions to your outcomes. | Validating performance before commitment. Best signal for most evaluations. |
| Pilot | Vendor is live in production, making real decisions. | Final validation. Requires contract negotiation upfront. |
Recommendation: Skip POC when possible. It is too easy for vendors to cherry-pick results from historical data. POV on live traffic gives you real signal.
Evaluation Criteria
| Criterion | Weight | What to Measure |
|---|---|---|
| Detection Accuracy | 25% | What percentage of known fraud did they catch? |
| False Positive Rate | 25% | What percentage of good transactions did they block? |
| Integration Effort | 20% | How long to implement? What resources required? |
| Total Cost | 15% | Per-transaction cost, implementation fees, ongoing support |
| Support Quality | 10% | Responsiveness, expertise, account management |
| Contract Terms | 5% | Exit provisions, data portability, price escalation |
Adjust weights based on your priorities. If integration is your constraint, weight it higher. If cost is secondary to accuracy, weight it lower.
Contract Negotiation Points
Things to negotiate before signing:
- Performance guarantees: Can they commit to a detection rate? What happens if they miss?
- Pilot pricing: Lock in pricing from pilot through production
- Exit terms: What happens to your data? How long to transition out?
- Price escalation: What triggers increases? Cap annual increases.
- SLA credits: Real credits for downtime, not just apologies
Implementation Planning
For enterprise implementations:
- Assign an owner: One person accountable for success
- Phase the rollout: Shadow mode → 10% of traffic → 50% → 100%
- Run parallel: Keep old system running until new system is proven
- Plan for tuning: Launch is the beginning, not the end. Budget time for ongoing optimization.
- Set review cadence: Weekly during implementation, monthly after launch
The Question That Matters Most
Before you evaluate any vendor, answer this honestly:
Do you actually know why you are getting chargebacks?
Pull your last 50 disputes. Categorize them:
- True fraud (stolen cards)
- Friendly fraud (customer lying)
- Service issues (product problems, shipping delays)
- Billing confusion (did not recognize charge)
If most of your chargebacks are not true fraud, a fraud tool will not fix your problem. You might need:
- Better billing descriptors (for recognition issues)
- Better customer service (for service issues)
- Better fulfillment (for shipping issues)
- Better product (for quality issues)
Fraud tools solve fraud problems. Make sure you actually have a fraud problem.
Next Steps
SMB evaluating vendors?
- Answer the three questions - Problem, success, failure
- Run the 2-week test - Shadow mode then limited rollout
- Know when to skip vendors - Under $1M may not need
Enterprise running formal evaluation?
- Follow structured process - Rounds 1-4
- Understand POC vs POV vs Pilot - Which to use
- Negotiate contract terms - Performance guarantees, exit
Testing vendor claims?
- Design experiments - Split flagged transactions
- Build feedback loops - Track outcomes
- Ask the right demo questions - Force specifics
Related
- Fraud Vendors Overview for who to evaluate
- Chargeback Vendors for dispute-specific tools
- Risk Scoring for understanding what vendors are doing under the hood