Vendor evaluation
Choosing a vendor is choosing a dependency you will live with for years and pay
to leave. The evaluation goes wrong when it is run on demos and vibes: the
polished sales walkthrough decides it, criteria get invented to justify the
favorite, and no one asks what it costs to get back out. A disciplined
evaluation tests the vendor on your work and prices the exit before it prices
the entry.
Method
- Set weighted criteria before you see a single demo. List what matters:
functionality, reliability, security posture, support, total cost, and how
hard it is to leave. Assign weights that sum to a fixed total, and write them
down first. Criteria invented after the demo just rationalize the vendor who
demoed best.
- Score against requirements, not against each other. Rate every vendor on
each criterion against your bar, so a weak field cannot make a mediocre
option look strong by comparison. Separate must-haves, which are pass or
fail, from nice-to-haves that earn weighted points.
- Design the proof of concept around your real workload. Feed it your
actual data shapes, your peak volume, your awkward edge cases, and your
integration points. A POC on the vendor's sample data proves their demo
works. Define the success metrics and the time box before you start so the
POC ends in a verdict, not an open-ended trial.
- Cost the exit, not just the entry. Estimate what it takes to migrate off:
data export formats, proprietary interfaces you would rewrite, retraining,
and contract lock-in. High switching cost is a real price even if it never
appears on an invoice. Prefer vendors whose data you can get back in a usable
form.
- Verify security and compliance with evidence, not the questionnaire
alone. Ask for the SOC 2 or ISO report, the data processing terms,
subprocessor list, breach history, and where data is stored. A vendor who
cannot produce these for a serious deal is telling you something.
- Reference-check past the list they gave you. Talk to the references, then
find a customer they did not hand you and ask what breaks at month six:
support response times, hidden costs, the feature that was "on the roadmap."
Sales sells the roadmap; a real customer tells you the product.
Checks
- Were the weighted criteria fixed before demos, or shaped to fit a favorite?
- Did the POC run on your workload and data, or on the vendor's canned scenario?
- Can you state, in dollars and weeks, what leaving this vendor would cost?
Boundaries
This picks the vendor; it does not negotiate the contract terms or run
procurement and legal review, which have their own owners and thresholds.
Approval limits and required security sign-offs are company policy: route spend
and data agreements through the process your organization requires rather than
deciding them here.
1---2name: vendor-evaluation3description: Evaluate vendors against weighted criteria, a proof of concept that tests your real workload, and a clear-eyed accounting of exit costs before you sign. Use when choosing a paid tool or service you will depend on and a wrong pick is expensive to undo.4---56# Vendor evaluation78Choosing a vendor is choosing a dependency you will live with for years and pay9to leave. The evaluation goes wrong when it is run on demos and vibes: the10polished sales walkthrough decides it, criteria get invented to justify the11favorite, and no one asks what it costs to get back out. A disciplined12evaluation tests the vendor on your work and prices the exit before it prices13the entry.1415## Method16171. **Set weighted criteria before you see a single demo.** List what matters:18 functionality, reliability, security posture, support, total cost, and how19 hard it is to leave. Assign weights that sum to a fixed total, and write them20 down first. Criteria invented after the demo just rationalize the vendor who21 demoed best.222. **Score against requirements, not against each other.** Rate every vendor on23 each criterion against your bar, so a weak field cannot make a mediocre24 option look strong by comparison. Separate must-haves, which are pass or25 fail, from nice-to-haves that earn weighted points.263. **Design the proof of concept around your real workload.** Feed it your27 actual data shapes, your peak volume, your awkward edge cases, and your28 integration points. A POC on the vendor's sample data proves their demo29 works. Define the success metrics and the time box before you start so the30 POC ends in a verdict, not an open-ended trial.314. **Cost the exit, not just the entry.** Estimate what it takes to migrate off:32 data export formats, proprietary interfaces you would rewrite, retraining,33 and contract lock-in. High switching cost is a real price even if it never34 appears on an invoice. Prefer vendors whose data you can get back in a usable35 form.365. **Verify security and compliance with evidence, not the questionnaire37 alone.** Ask for the SOC 2 or ISO report, the data processing terms,38 subprocessor list, breach history, and where data is stored. A vendor who39 cannot produce these for a serious deal is telling you something.406. **Reference-check past the list they gave you.** Talk to the references, then41 find a customer they did not hand you and ask what breaks at month six:42 support response times, hidden costs, the feature that was "on the roadmap."43 Sales sells the roadmap; a real customer tells you the product.4445## Checks4647- Were the weighted criteria fixed before demos, or shaped to fit a favorite?48- Did the POC run on your workload and data, or on the vendor's canned scenario?49- Can you state, in dollars and weeks, what leaving this vendor would cost?5051## Boundaries5253This picks the vendor; it does not negotiate the contract terms or run54procurement and legal review, which have their own owners and thresholds.55Approval limits and required security sign-offs are company policy: route spend56and data agreements through the process your organization requires rather than57deciding them here.
Amey-Thakur