Evaluating New Technology

Evaluating New Technology

Scope

Covers

• Evaluating a new tool/platform/vendor (including AI products) for adoption
• Emerging tech “should we use this?” decisions
• Build vs buy decisions and tech stack changes
• Running a proof-of-value pilot and capturing evidence
• First-pass risk review (security/privacy/compliance, vendor claims, operational readiness)

When to use

• “Evaluate this new AI tool/vendor for our team.”
• “Should we build this in-house or buy a vendor?”
• “We’re considering changing our analytics/experimentation stack—make a recommendation.”
• “Create a technology evaluation doc with a pilot plan, risks, and decision memo.”

When NOT to use

• You don’t have a real problem/job to solve yet (use problem-definition first).
• You need a full product strategy/roadmap (use ai-product-strategy).
• You’re designing how to build an LLM system (use building-with-llms).
• You need a formal security assessment / penetration testing (engage security; this skill produces a structured first pass).
• You are weighing trade-offs within an existing design or architecture decision (use evaluating-trade-offs).
• You need to manage or pay down existing technical debt rather than adopt something new (use managing-tech-debt).
• You already chose the technology and need an implementation/migration plan (use managing-tech-debt or platform-infrastructure).

Inputs

Minimum required

• Candidate technology (what it is, vendor/build option, links if available)
• Problem/workflow to improve + who it’s for
• Current approach/stack and what’s not working
• Constraints: data sensitivity, privacy/compliance, budget, timeline, regions, deployment model (SaaS/on-prem)
• Decision context: who decides, adoption scope, risk tolerance

Missing-info strategy

• Ask up to 5 questions from references/INTAKE.md (3–5 at a time).
• If still missing, proceed with explicit assumptions and present 2–3 options (e.g., buy vs build vs defer).
• Do not request secrets. If asked to run tools, change production systems, or sign up for vendors, require explicit confirmation.

Outputs (deliverables)

Produce a Technology Evaluation Pack (in chat; or as files if requested), in this order:

1. Evaluation brief (problem, stakeholders, decision, constraints, non-goals, assumptions)
2. Options & criteria matrix (status quo + alternatives, criteria, scoring, notes)
3. Build vs buy analysis (bandwidth/TCO, core competency, opportunity cost, lock-in)
4. Pilot (proof-of-value) plan (hypotheses, scope, metrics, timeline, exit criteria)
5. Risk & guardrails review (security/privacy/compliance, vendor claims, mitigations)
6. Decision memo (recommendation, rationale, trade-offs, adoption/rollback plan)
7. Risks / Open questions / Next steps (always included)

Templates: references/TEMPLATES.md

Workflow (8 steps)

1) Start with the problem (avoid tool bias)

• Inputs: Candidate tech, target workflow/users, current pain.
• Actions: Write a one-sentence problem statement and “who feels it.” List 3–5 symptoms and 3–5 non-goals.
• Outputs: Draft Evaluation brief (problem + non-goals).
• Checks: You can explain the decision without naming the tool.

2) Define “good” and hard constraints

• Inputs: Success metrics, constraints, risk tolerance, decision deadline.
• Actions: Define success metrics (leading + lagging) and must-have constraints (privacy, compliance, security, uptime, latency/cost if relevant). Capture “deal breakers.”
• Outputs: Evaluation brief (success + constraints + deal breakers).
• Checks: A stakeholder can say what would make this a clear “yes” or “no.”

3) Map options and evaluation criteria (workflows → ROI)

• Inputs: Current stack, alternatives, stakeholders.
• Actions: List options: status quo, 1–3 vendors, build, hybrid. Define criteria anchored to workflows enabled and ROI (time saved, revenue impact, risk reduction), not feature checklists.
• Outputs: Options & criteria matrix.
• Checks: Every criterion is measurable or at least falsifiable in a pilot.

4) Fast reality check: integration + data fit

• Inputs: Architecture constraints, data sources, integration points.
• Actions: Identify required integrations (SSO, data pipelines, APIs, logs). Note migration complexity, data ownership, and export/exit path. For PLG/growth tools, sanity-check the stack layers (data hub → analytics → lifecycle).
• Outputs: Notes added to Options & criteria matrix (integration complexity + stack fit).
• Checks: You can describe the end-to-end data/control flow in 5–10 bullets.

5) Build vs buy with “bandwidth” as a first-class cost

• Inputs: Engineering capacity, core competencies, opportunity cost.
• Actions: Compare build vs buy using a bandwidth/TCO ledger (build time, maintenance, on-call, upgrades, vendor management). Prefer building only when it’s a core differentiator or the vendor market is immature/unacceptable.
• Outputs: Build vs buy analysis.
• Checks: The analysis includes opportunity cost and who would maintain the system 12 months from now.

6) Risk & guardrails review (be skeptical of “100% safe” claims)

• Inputs: Data sensitivity, threat model, vendor posture, deployment model.
• Actions: Identify key risks (security, privacy, compliance, reliability, lock-in). For AI vendors: treat “guardrails catch everything” claims as marketing; assume determined attackers exist and design defense-in-depth (permissions, logging, human approval points, eval/red-team).
• Outputs: Risk & guardrails review.
• Checks: Each top risk has an owner and a mitigation or a “blocker” label.

7) Plan a proof-of-value pilot (or document why you can skip it)

• Inputs: Criteria, risks, timeline, stakeholders.
• Actions: Define pilot hypotheses, scope, success metrics, test dataset, and evaluation method. Specify timeline, resourcing, and exit criteria (adopt / iterate / reject). Include rollback and data deletion requirements.
• Outputs: Pilot plan.
• Checks: A team can run the pilot without extra meetings; success/failure is unambiguous.

8) Decide, communicate, and quality-gate

• Inputs: Completed pack drafts.
• Actions: Write the Decision memo with recommendation, trade-offs, and adoption plan. Run references/CHECKLISTS.md and score with references/RUBRIC.md. Always include Risks / Open questions / Next steps.
• Outputs: Final Technology Evaluation Pack.
• Checks: Decision is actionable (owner, date, next actions) and reversible where possible.

Anti-patterns (common failure modes)

1. "Shiny object" framing — Starting from "we should use X" instead of "we need to solve Y." The evaluation becomes a justification exercise rather than a genuine comparison. Always start with the problem statement.
2. Feature-checklist scoring — Evaluating options by counting features rather than mapping to workflows and ROI. Result: the tool with the most checkboxes wins even if it solves the wrong problem.
3. Skipping the status quo option — Failing to include "do nothing / improve current approach" as a baseline. Result: you cannot prove the new tool is actually better than incremental improvement.
4. Vendor demo as evidence — Treating a curated vendor demo as proof of production fitness. Result: the pilot discovers integration, performance, or data-quality issues that the demo hid.
5. No exit plan — Adopting a tool without evaluating lock-in, data export, or migration cost. Result: switching cost grows silently until the team is trapped.

Quality gate (required)

• Use references/CHECKLISTS.md and references/RUBRIC.md.
• Always include: Risks, Open questions, Next steps.

Examples

Example 1 (AI vendor): “Use evaluating-new-technology to evaluate an AI ‘prompt guardrails’ vendor for our support agent. Constraints: SOC2 required, PII present, must support SSO, budget $50k/yr, decision in 3 weeks.”
Expected: evaluation pack that treats guardrail claims skeptically and proposes defense-in-depth + a measurable pilot.

Example 2 (analytics stack): “Use evaluating-new-technology to choose between PostHog and Amplitude for our PLG product. Current stack: Segment + data warehouse; goal is faster iteration on onboarding and activation.”
Expected: options matrix + pilot plan tied to workflows (experiments, funnels, lifecycle triggers) and migration effort.

Boundary example (redirect — no problem defined): “What’s the best new AI tool we should adopt?” Response: Out of scope without a problem/workflow. Ask intake questions and/or propose running problem-definition first to identify the job-to-be-done before evaluating tools.

Boundary example (redirect — neighbor skill): “We already picked Datadog; now help us plan the migration from our current monitoring stack.” Response: This is a migration/tech-debt execution task, not an evaluation. Redirect to managing-tech-debt for a migration plan with milestones, rollback, and decommission steps.