Operations Vendor Evaluation

Vendor Evaluation and Scoring

Purpose

Vendor selections are commonly decided by whichever demo landed best and then justified with a matrix built afterwards, whose weightings were chosen to produce the answer already reached. This skill sequences the work so the weights and the scoring anchors are fixed before any vendor contact, applies pass/fail gates that no commercial score can override, and requires evidence per score so the decision survives challenge.

Prerequisites

• Inputs: the problem being solved and the process it supports — ideally a validated map from operations-process-mapping; the budget envelope and the decision deadline; the named decision owner and the evaluation panel.
• Access: security, data protection, legal, and finance reviewers; the users who will operate the tool daily; the incumbent's actual cost and performance data if this is a renewal.
• Required: written requirements before vendor contact. If requirements are written after seeing a demo, they will describe that vendor.

If the requirements or the weightings cannot be agreed before evaluation begins, stop and get them agreed. Everything downstream is decoration otherwise.

Data classification

Confidential. Pricing, scoring, and vendor commercial submissions are restricted to the panel; many are received under non-disclosure terms — check and honour them. Never send personal data, customer records, or production data to a vendor for trial or proof of concept without a completed data protection assessment and a signed processing agreement. Synthetic or properly anonymised data only, until that is in place. If anyone proposes a trial using live customer data before that gate, flag it and stop.

Procedure

1. Write the requirements before contacting anyone. Separate them into mandatory and scored. Mandatory means the absence of it disqualifies, regardless of everything else — write only genuine ones there, because a long mandatory list disqualifies every option and the list then gets quietly relaxed for the preferred vendor.

2. Set weightings and get them signed off, before evaluation. Weights are agreed by the decision owner and the panel and recorded with a date. Changing a weight after scores exist requires re-approval and a written reason; this rule is the main defence against a retrofitted matrix.

3. Apply the pass/fail gates. These are not scored and cannot be traded off:

   Gate	Fails when
   Security posture	No current independent assurance, unresolved material findings, or refusal to complete your security assessment
   Data protection	Processing location or sub-processors incompatible with your obligations; no acceptable processing agreement; inadequate deletion or portability terms
   Regulatory fit	Cannot meet the obligations applying to your sector or data
   Business viability	Cannot evidence the ability to operate for the contract term
   Contractual minimums	Refuses liability, audit, notification, or termination terms you cannot waive
   Exit	No documented, tested route to extract your data in a usable format

   A vendor failing any gate is out. Record it and move on.

4. Write scoring anchors for every scored criterion before scoring. Define in words what a top, middle, and bottom score looks like for that criterion. Numbers without anchors converge on the evaluator's mood and cannot be defended. For example, for a criterion on integration: top = supported natively with documented interfaces we verified; middle = achievable with configuration effort we have sized; bottom = requires custom development or is unsupported.

5. Standardise the evidence-gathering. Every vendor gets the same questions, the same scenario for demonstration, and the same data. A demo the vendor scripted shows what the vendor chose; run your own scenario, drawn from the real process, and have the people who will use it operate it themselves.

6. Score independently, then reconcile. Panel members score alone before discussing. Group scoring converges on the most senior or most confident voice. In reconciliation, discuss only the criteria where scores differ by more than one point, and record why the difference resolved as it did.

7. Record evidence per score. Each score cites what it is based on: a demonstration observed, a document reviewed, a reference call, a test result. Scores citing the vendor's own marketing are marked as unverified and treated as weaker evidence — the substantiation rule applies here as it does to your own claims: a performance or comparative claim from a vendor needs a named, dated source, and an unsourced one is recorded as unsourced rather than accepted.

8. Take references you selected, not only the ones offered. Ask referees about implementation duration versus plan, what went wrong, support responsiveness during a real incident, and what they would do differently. Ask specifically whether anything they were shown in the sales process did not materialise.

9. Build total cost of ownership over the full term, not the licence price: licence or subscription, implementation, integration, data migration, internal effort in person-days, training, ongoing administration, support tier, transaction or overage charges, contractual uplift, and exit and migration cost at the end. The cheapest licence with the largest internal effort is routinely the most expensive option, and the internal effort line is the one usually omitted.

10. Produce the decision record. It contains: the requirements and weights with their approval date, the gate results, the scores with evidence, the total cost comparison, the reference findings, the recommendation, the main risks with mitigations, and the dissenting views. Record dissent — a decision record showing unanimity that did not exist is not usable when the risk materialises.

11. Define the review and exit conditions at selection, while you still have leverage: the service levels that matter, the review points, what would trigger termination, the notice period, and the tested data extraction route. Negotiate exit terms before signature; after signature there is nothing to negotiate with.

12. Hand over to implementation with the assumptions and the internal effort estimate intact, and schedule the first performance review against the criteria that were scored — not against new ones invented later.

Failure modes

• Demo-led selection. Requirements written after the first demo describe that product's feature list.
• Weights tuned to the answer. Prevented only by step 2's sign-off and date.
• Security and data protection consulted last. A gate applied after the business has chosen becomes a negotiation rather than a gate.
• Licence-price comparison. Ignores implementation and internal effort, which frequently exceed licence cost over the term.
• Vendor-supplied references only. They are selected to be positive; ask them harder questions and find at least one referee yourself.
• Incumbent unscored on renewal. The incumbent should be evaluated against the same rubric, with its actual performance data, not carried forward by default.
• No exit plan. Discovered at renewal, when it becomes the vendor's pricing leverage.

Boundaries

• Responding to an RFP as the supplier — use sales-proposal-assembly.
• Documenting the operating procedure once the vendor is live — use operations-sop-authoring.
• An internal technical build-versus-buy or architecture decision — use engineering-decision-record.
• Access provisioning and periodic entitlement review for the selected tool — use it-access-review.

Hand-offs

• Receives from: operations-process-mapping (the process and its real requirements), marketing-campaign-brief (agency or media supplier selection), operations-sop-authoring (a step that should be outsourced or tooled).
• Routes to: it-change-management for implementation, it-access-review for entitlement setup and review, and operations-sop-authoring for the resulting operating procedure.