# Operations Vendor Evaluation

> Runs a defensible vendor or tool selection: requirements weighted and agreed before any demo, a scoring rubric with written anchors, mandatory pass/fail gates for security, data protection and viability, evidence recorded per score, and total cost of ownership including exit. Use when choosing between suppliers, software, or outsourcing options; trigger on 'evaluate these vendors', 'which tool should we buy', 'build a comparison matrix', 'vendor scoring', 'we're renewing this contract, should we switch'. Not for responding to someone else's RFP as the seller (use sales-proposal-assembly), not for documenting the resulting operational process (use operations-sop-authoring), and not for an internal build-versus-buy technical design (use engineering-decision-record).

- Skill: `alihusains/operations-vendor-evaluation` (Agent Skill)
- Install (CLI): `npx skillmds@latest add alihusains/operations-vendor-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/alihusains/operations-vendor-evaluation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- Author: alihusains (https://skillmd.com/u/alihusains)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/alihusains/operations-vendor-evaluation

---


# Vendor Evaluation and Scoring

## Purpose

Vendor selections are commonly decided by whichever demo landed best and then
justified with a matrix built afterwards, whose weightings were chosen to produce
the answer already reached. This skill sequences the work so the weights and the
scoring anchors are fixed before any vendor contact, applies pass/fail gates that
no commercial score can override, and requires evidence per score so the decision
survives challenge.

## Prerequisites

- **Inputs:** the problem being solved and the process it supports — ideally a
  validated map from `operations-process-mapping`; the budget envelope and the
  decision deadline; the named decision owner and the evaluation panel.
- **Access:** security, data protection, legal, and finance reviewers; the users
  who will operate the tool daily; the incumbent's actual cost and performance
  data if this is a renewal.
- **Required:** written requirements before vendor contact. If requirements are
  written after seeing a demo, they will describe that vendor.

If the requirements or the weightings cannot be agreed before evaluation begins,
stop and get them agreed. Everything downstream is decoration otherwise.

## Data classification

**Confidential.** Pricing, scoring, and vendor commercial submissions are
restricted to the panel; many are received under non-disclosure terms — check and
honour them. Never send personal data, customer records, or production data to a
vendor for trial or proof of concept without a completed data protection
assessment and a signed processing agreement. Synthetic or properly anonymised
data only, until that is in place. If anyone proposes a trial using live customer
data before that gate, flag it and stop.

## Procedure

1. **Write the requirements before contacting anyone.** Separate them into
   mandatory and scored. Mandatory means the absence of it disqualifies,
   regardless of everything else — write only genuine ones there, because a long
   mandatory list disqualifies every option and the list then gets quietly
   relaxed for the preferred vendor.

2. **Set weightings and get them signed off, before evaluation.** Weights are
   agreed by the decision owner and the panel and recorded with a date. Changing
   a weight after scores exist requires re-approval and a written reason; this
   rule is the main defence against a retrofitted matrix.

3. **Apply the pass/fail gates.** These are not scored and cannot be traded off:

   | Gate | Fails when |
   | --- | --- |
   | Security posture | No current independent assurance, unresolved material findings, or refusal to complete your security assessment |
   | Data protection | Processing location or sub-processors incompatible with your obligations; no acceptable processing agreement; inadequate deletion or portability terms |
   | Regulatory fit | Cannot meet the obligations applying to your sector or data |
   | Business viability | Cannot evidence the ability to operate for the contract term |
   | Contractual minimums | Refuses liability, audit, notification, or termination terms you cannot waive |
   | Exit | No documented, tested route to extract your data in a usable format |

   A vendor failing any gate is out. Record it and move on.

4. **Write scoring anchors for every scored criterion before scoring.** Define in
   words what a top, middle, and bottom score looks like for that criterion.
   Numbers without anchors converge on the evaluator's mood and cannot be
   defended. For example, for a criterion on integration: top = supported natively
   with documented interfaces we verified; middle = achievable with configuration
   effort we have sized; bottom = requires custom development or is unsupported.

5. **Standardise the evidence-gathering.** Every vendor gets the same questions,
   the same scenario for demonstration, and the same data. A demo the vendor
   scripted shows what the vendor chose; run your own scenario, drawn from the
   real process, and have the people who will use it operate it themselves.

6. **Score independently, then reconcile.** Panel members score alone before
   discussing. Group scoring converges on the most senior or most confident voice.
   In reconciliation, discuss only the criteria where scores differ by more than
   one point, and record why the difference resolved as it did.

7. **Record evidence per score.** Each score cites what it is based on: a
   demonstration observed, a document reviewed, a reference call, a test result.
   Scores citing the vendor's own marketing are marked as unverified and treated
   as weaker evidence — the substantiation rule applies here as it does to your
   own claims: a performance or comparative claim from a vendor needs a named,
   dated source, and an unsourced one is recorded as unsourced rather than
   accepted.

8. **Take references you selected, not only the ones offered.** Ask referees about
   implementation duration versus plan, what went wrong, support responsiveness
   during a real incident, and what they would do differently. Ask specifically
   whether anything they were shown in the sales process did not materialise.

9. **Build total cost of ownership over the full term,** not the licence price:
   licence or subscription, implementation, integration, data migration, internal
   effort in person-days, training, ongoing administration, support tier,
   transaction or overage charges, contractual uplift, and exit and migration cost
   at the end. The cheapest licence with the largest internal effort is routinely
   the most expensive option, and the internal effort line is the one usually
   omitted.

10. **Produce the decision record.** It contains: the requirements and weights
    with their approval date, the gate results, the scores with evidence, the
    total cost comparison, the reference findings, the recommendation, the main
    risks with mitigations, and the dissenting views. Record dissent — a decision
    record showing unanimity that did not exist is not usable when the risk
    materialises.

11. **Define the review and exit conditions at selection,** while you still have
    leverage: the service levels that matter, the review points, what would
    trigger termination, the notice period, and the tested data extraction route.
    Negotiate exit terms before signature; after signature there is nothing to
    negotiate with.

12. **Hand over to implementation** with the assumptions and the internal effort
    estimate intact, and schedule the first performance review against the
    criteria that were scored — not against new ones invented later.

## Failure modes

- **Demo-led selection.** Requirements written after the first demo describe that
  product's feature list.
- **Weights tuned to the answer.** Prevented only by step 2's sign-off and date.
- **Security and data protection consulted last.** A gate applied after the
  business has chosen becomes a negotiation rather than a gate.
- **Licence-price comparison.** Ignores implementation and internal effort, which
  frequently exceed licence cost over the term.
- **Vendor-supplied references only.** They are selected to be positive; ask them
  harder questions and find at least one referee yourself.
- **Incumbent unscored on renewal.** The incumbent should be evaluated against the
  same rubric, with its actual performance data, not carried forward by default.
- **No exit plan.** Discovered at renewal, when it becomes the vendor's pricing
  leverage.

## Boundaries

- Responding to an RFP as the supplier — use `sales-proposal-assembly`.
- Documenting the operating procedure once the vendor is live — use
  `operations-sop-authoring`.
- An internal technical build-versus-buy or architecture decision — use
  `engineering-decision-record`.
- Access provisioning and periodic entitlement review for the selected tool — use
  `it-access-review`.

## Hand-offs

- **Receives from:** `operations-process-mapping` (the process and its real
  requirements), `marketing-campaign-brief` (agency or media supplier selection),
  `operations-sop-authoring` (a step that should be outsourced or tooled).
- **Routes to:** `it-change-management` for implementation, `it-access-review` for
  entitlement setup and review, and `operations-sop-authoring` for the resulting
  operating procedure.