tools-only sf-ai-agentforce-observability Validation Tracking
Overview
This document tracks validation runs for the sf-ai-agentforce-observability skill. Validation ensures the skill's CLI commands and API integrations work correctly against live Salesforce Data 360 environments.
Validation Cycle: Every 30 days or after significant changes
Current Status
| Metric | Value |
|---|---|
| Last Validation | Not yet validated |
| Overall Score | Pending |
| Status | 🟡 PENDING |
| Next Validation Due | After initial run |
Scoring Rubric (100 Points)
| Tier | Category | Points | Description |
|---|---|---|---|
| T1 | Auth & Connectivity | 25 | JWT auth, consumer key resolution, API access |
| T2 | Extraction Commands | 30 | extract, extract-tree, extract-incremental, count |
| T3 | Analysis Commands | 20 | analyze, debug-session, topics, quality-report |
| T4 | Data Model/Schema | 15 | DMO names, field casing, Parquet structure |
| T5 | Negative/Error Cases | 10 | Auth failures, invalid arguments |
Pass Threshold: ≥80 points
Validation History
Run #0 - Initial (Pending)
| Tier | Score | Status | Notes |
|---|---|---|---|
| T1 | -/25 | ⏳ | Pending |
| T2 | -/30 | ⏳ | Pending |
| T3 | -/20 | ⏳ | Pending |
| T4 | -/15 | ⏳ | Pending |
| T5 | -/10 | ⏳ | Pending |
| Total | -/100 | ⏳ | Pending |
Test Mapping to SKILL.md Claims
T1: Auth & Connectivity (25 pts)
| Test | SKILL.md Section | Points |
|---|---|---|
| Key path resolution order | "Key Path Resolution Order" | 5 |
| Consumer key resolution | "Auth Setup" | 5 |
| v65.0 API connectivity | "Prerequisites Checklist" | 10 |
| Error message quality | "Common Issues & Fixes" | 5 |
T2: Extraction Commands (30 pts)
| Test | SKILL.md Section | Points |
|---|---|---|
| extract creates 4 directories | "Output Directory Structure" | 10 |
| Date filtering works | "CLI Quick Reference" | 5 |
| extract-tree for session IDs | "Extract Session Tree" | 5 |
| Incremental watermark behavior | "Incremental Extraction" | 5 |
| count command | "Count Records" | 5 |
T3: Analysis Commands (20 pts)
| Test | SKILL.md Section | Points |
|---|---|---|
| analyze generates stats | "Analysis Examples" | 5 |
| debug-session timeline | "Debug Session Timeline" | 5 |
| topics routing analysis | "Topic Analysis" | 5 |
| quality-report metrics | "Quality Report" | 5 |
T4: Data Model/Schema (15 pts)
| Test | SKILL.md Section | Points |
|---|---|---|
| DMO names correct | "Session Tracing Data Model" | 5 |
| AiAgent field casing | "Key Schema Notes" | 5 |
| Parquet directory structure | "Output Directory Structure" | 5 |
T5: Negative/Error Cases (10 pts)
| Test | SKILL.md Section | Points |
|---|---|---|
| Auth failure handling | "Common Issues & Fixes" | 5 |
| Invalid args rejected | "CLI Quick Reference" | 5 |
Known Issues
None recorded yet.
Remediation Log
| Date | Issue | Resolution | Commit |
|---|---|---|---|
| None yet |
Running Validation
# Quick offline test (T3, T4, T5 only - uses fixtures)
cd sf-ai-agentforce-observability
python3 validation/scripts/run_validation.py --offline
# Full validation against live org
python3 validation/scripts/run_validation.py --org Vivint-DevInt
# Generate report and update this file
python3 validation/scripts/run_validation.py --org Vivint-DevInt --report
# Run specific tier
python3 validation/scripts/run_validation.py --org Vivint-DevInt --tier T1
Integration with test_v65.py
The existing scripts/test_v65.py provides a quick smoke test with 7 progressive levels. The validation framework wraps these and adds:
- Scoring - Quantified pass/fail with points
- Isolation - Offline tests using fixtures
- Coverage - Tests CLI commands, not just API
- Tracking - Historical validation runs
| test_v65.py Level | Validation Tier |
|---|---|
| Level 1-2 (Connectivity) | T1 |
| Level 3-5 (Query API) | T1 |
| Level 6-7 (Extraction) | T2 |
Next Validation Checklist
- Ensure test org has STDM data (Agentforce sessions)
- Verify JWT key exists at
~/.sf/jwt/{org}.key - Verify consumer key exists at
~/.sf/jwt/{org}.consumer-key - Run:
python3 validation/scripts/run_validation.py --org <alias> --report - Update this file with results
- Commit changes