# AI Vendor Evaluation

> Comprehensive framework for evaluating AI vendors and solutions to avoid costly mistakes. Use this skill when assessing AI vendor proposals, conducting due diligence, evaluating contracts, comparing vendors, or making build-vs-buy decisions. Helps identify red flags, assess pricing models, evaluate technical capabilities, and conduct structured vendor comparisons.

- Skill: `diegosouzapw/ai-vendor-evaluation` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add diegosouzapw/ai-vendor-evaluation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/diegosouzapw/ai-vendor-evaluation/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: diegosouzapw (https://skillmd.com/u/diegosouzapw)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/diegosouzapw/ai-vendor-evaluation

---


# AI Vendor Evaluation

**Version 1.0** | October 2025 | Based on $1.2M average AI spend analysis

---

## Overview

This skill provides a systematic framework for evaluating AI vendors and solutions to avoid the costly mistakes that plague 95% of AI projects. Use when conducting vendor due diligence, evaluating proposals, negotiating contracts, or making strategic AI purchasing decisions.

**Key capabilities:**
- Structured evaluation criteria for AI vendors
- Red flag identification in proposals and demos
- Pricing model analysis and fair market rates
- Technical capability assessment
- Contract term evaluation
- Build vs buy decision framework

---

## Quick Decision Tree

**Start here to determine which references to read:**

```
What stage are you in?

├─ Early exploration (multiple vendors being considered)
│  └─ Read: evaluation-criteria.md, use-case-fit.md
│     Use: scorecard-template.xlsx
│
├─ Evaluating specific proposal or demo
│  └─ Read: red-flags.md, technical-assessment.md
│     Check: pricing-models.md for pricing reasonableness
│
├─ Contract negotiation
│  └─ Read: contract-checklist.md, pricing-models.md
│     Reference: red-flags.md for problematic terms
│
├─ Build vs Buy decision
│  └─ Read: build-vs-buy.md, use-case-fit.md
│     Consider: Total cost of ownership from pricing-models.md
│
└─ Post-purchase review or audit
   └─ Read: evaluation-criteria.md, technical-assessment.md
      Assess: Whether vendor is delivering on promises
```

---

## When to Use This Skill

**Trigger scenarios:**
- "Help me evaluate this AI vendor proposal"
- "What should I look for in AI vendor demos?"
- "Is this pricing reasonable for an AI solution?"
- "Should we build or buy this AI capability?"
- "What questions should I ask this AI vendor?"
- "Help me compare these AI vendors"
- "Review this AI contract for red flags"
- "Conduct due diligence on this AI company"

---

## Core Evaluation Framework

### Phase 1: Initial Screening
**Goal**: Eliminate obviously problematic vendors before deep evaluation

**Key questions:**
- Does the vendor have relevant domain experience?
- Are there verifiable customer references?
- Is the technology approach sound?
- Are pricing and terms transparent?

**Read**: `references/red-flags.md` for disqualifying signals  
**Read**: `references/use-case-fit.md` for domain fit assessment

---

### Phase 2: Deep Evaluation
**Goal**: Assess vendor capabilities systematically across all dimensions

**Evaluation dimensions:**
1. **Technical capability** - Can they actually deliver?
2. **Business viability** - Will they still exist in 2 years?
3. **Pricing fairness** - Are costs reasonable for value delivered?
4. **Implementation risk** - How likely is successful deployment?
5. **Contract terms** - Are legal terms acceptable?

**Read**: `references/evaluation-criteria.md` for comprehensive framework  
**Read**: `references/technical-assessment.md` for technical evaluation  
**Read**: `references/pricing-models.md` for pricing analysis  
**Use**: `assets/scorecard-template.xlsx` to score vendors systematically

---

### Phase 3: Contract Negotiation
**Goal**: Secure favorable terms and avoid costly traps

**Critical areas:**
- Performance guarantees and SLAs
- Data ownership and usage rights
- Pricing structure and escalation terms
- Exit clauses and data portability
- Liability and indemnification

**Read**: `references/contract-checklist.md` for essential terms  
**Reference**: `references/red-flags.md` for problematic contract patterns

---

## Common Vendor Patterns

### The Overpromiser
**Characteristics**: Claims to solve everything, vague on technical details, aggressive sales tactics  
**Red flag**: "Our AI can handle any use case"  
**Response**: Demand specific technical explanations and verifiable references

### The Feature Dumper
**Characteristics**: Long feature lists, complex pricing, unclear core value proposition  
**Red flag**: Can't explain what problem they actually solve  
**Response**: Force clarity on primary use case and success metrics

### The Consultant in Disguise
**Characteristics**: Software license + mandatory professional services  
**Red flag**: Professional services cost more than software  
**Response**: Assess true cost of ownership, consider if you're buying software or consulting

### The Model Wrapper
**Characteristics**: Thin layer over OpenAI/Anthropic APIs with high markup  
**Red flag**: No proprietary technology, just API access + UI  
**Response**: Calculate cost of building similar solution in-house

**Full pattern library**: See `references/red-flags.md`

---

## Build vs Buy Decision Framework

**When to read this section**: Before committing to vendor evaluation, determine if building in-house is better option.

**Key factors:**
1. **Capability availability** - Does suitable vendor solution exist?
2. **Time to value** - Buy: weeks-months, Build: months-years
3. **Total cost** - Consider 3-year TCO for both options
4. **Strategic importance** - Core competency? Build. Commodity? Buy.
5. **Team capability** - Do you have talent to build and maintain?

**Read**: `references/build-vs-buy.md` for detailed decision framework

---

## Using the Scorecard Template

The vendor scorecard enables structured comparison across vendors.

**To use**:
1. Open `assets/scorecard-template.xlsx`
2. List vendors to compare (up to 5)
3. Score each vendor on evaluation criteria (1-5 scale)
4. Review weighted scores and vendor comparison chart
5. Document decision rationale

**Customization**: Adjust weights based on priorities for your specific use case.

---

## Reference Documents

### references/evaluation-criteria.md
Comprehensive scoring framework across all vendor evaluation dimensions. Includes specific questions to ask, what constitutes good/bad answers, and how to weight criteria for different use cases.

**Use when**: Conducting systematic vendor evaluation

---

### references/red-flags.md
Catalog of warning signs indicating problematic vendors. Organized by category: technical red flags, business red flags, pricing red flags, contract red flags, and behavioral red flags.

**Use when**: Initial vendor screening or reviewing proposals

---

### references/pricing-models.md
Guide to AI vendor pricing models (per-seat, usage-based, platform fees, etc.), fair market rates, what drives costs, and how to negotiate. Includes pricing red flags and total cost of ownership analysis.

**Use when**: Evaluating vendor pricing or negotiating contracts

---

### references/technical-assessment.md
Framework for assessing technical capabilities: architecture review, model evaluation, integration complexity, scalability, security, and data handling. Includes specific technical questions to ask.

**Use when**: Deep technical evaluation of vendor capabilities

---

### references/contract-checklist.md
Essential contract terms for AI vendor agreements: performance guarantees, data rights, pricing protection, exit terms, liability, and support commitments. Includes negotiation guidance.

**Use when**: Contract review or negotiation

---

### references/use-case-fit.md
Framework for assessing whether vendor solution actually fits your use case. Includes questions to ask yourself, questions to ask vendor, and warning signs of poor fit.

**Use when**: Initial vendor screening or use case definition

---

### references/build-vs-buy.md
Decision framework for whether to build AI capability in-house vs purchasing vendor solution. Includes total cost analysis, capability assessment, and strategic considerations.

**Use when**: Before committing to vendor evaluation process

---

## Assets

### assets/scorecard-template.xlsx
Structured spreadsheet for vendor comparison with:
- Evaluation criteria organized by category
- Scoring system (1-5 scale) with descriptions
- Weighted scoring based on priorities
- Vendor comparison charts
- Decision documentation section

**Customize**: Adjust criteria weights and add company-specific requirements