Tpl Situacao Machine Learning Integracao

SITUATION: Machine Learning Model Integration

Skill gerado a partir do pack templates-claude-code. Arquivo de origem: situacao/16-machine-learning-integracao.md. Use como baseline e adapte ao projeto antes de mudancas grandes.

Conteudo do template

CONTEXT

Use this configuration when integrating a trained machine learning model into a production application — whether it's a recommendation engine, fraud classifier, sentiment analyzer, content moderator, or any model served via API or embedded SDK. ML models in production have unique failure modes that differ from traditional software: they can be "right" statistically but wrong for individual cases, they degrade silently as data distributions shift, and they require careful A/B testing to safely update.

OBJECTIVES

• Serve model predictions reliably with graceful fallback when the model is unavailable
• Validate all inputs before inference to prevent nonsensical predictions
• Monitor for data drift and model performance degradation post-launch
• Enable safe A/B testing of model versions without code deploys
• Collect ground truth for continuous model improvement

APPROACH RULES

1. Input validation before inference is mandatory. Never pass raw user input to a model. Validate type, range, cardinality, and format. A model given garbage input produces confident garbage output — this is worse than an error.

2. Always have a fallback. When the model is unavailable (service down, timeout, error), fall back to a reasonable default: most popular items, rule-based logic, or graceful degradation (show without personalization). Never fail the entire request because the model is unavailable.

3. Log every prediction for ground truth collection. Log: input features, prediction output, confidence score, model version, and the user's eventual action. Without ground truth, you cannot measure model accuracy in production.

4. Model versions are deployment artifacts. Like Docker images, model versions must be immutable, tagged, and deployed explicitly. Never load "the latest model" automatically from a shared location in production.

5. A/B test every model change. Updating a model is a behavior change. It must go through the same A/B testing process as a product feature change. Traffic splitting, metric collection, and winner determination process must all apply.

6. Confidence thresholds, not just predictions. For high-stakes decisions (fraud, content moderation), set a confidence threshold below which the model defers to a human or applies a conservative default. A 51% confidence fraud prediction should not automatically block an account.

7. Feature drift is the #1 cause of production ML failures. Monitor input feature distributions. If the distribution of features being sent for inference drifts significantly from training distribution, model accuracy degrades silently.

ROUTING TABLE

Model Serving Architecture

interface ModelPrediction {
  result: string | number | boolean
  confidence: number
  modelVersion: string
  latencyMs: number
}

class RecommendationService {
  constructor(
    private modelClient: ModelClient,
    private fallback: FallbackRecommendations,
    private logger: Logger,
    private metrics: MetricsClient
  ) {}

  async getRecommendations(userId: string, context: UserContext): Promise<Product[]> {
    const start = Date.now()
    
    try {
      // 1. Validate and prepare features
      const features = this.prepareFeatures(userId, context)
      this.validateFeatures(features)
      
      // 2. Call model with timeout
      const prediction = await Promise.race([
        this.modelClient.predict(features),
        timeout(500, new Error('Model timeout'))
      ])
      
      // 3. Apply confidence threshold
      if (prediction.confidence < 0.6) {
        this.logger.warn({ userId, confidence: prediction.confidence }, 'Low confidence prediction')
        return this.fallback.getPopularItems(context.category)
      }
      
      // 4. Log prediction for ground truth
      this.logPrediction(userId, features, prediction)
      
      this.metrics.histogram('model.latency', Date.now() - start)
      this.metrics.increment('model.predictions.served')
      
      return prediction.result as Product[]
      
    } catch (err) {
      this.metrics.increment('model.predictions.fallback')
      this.logger.error({ err, userId }, 'Model inference failed, using fallback')
      return this.fallback.getPopularItems(context.category)
    }
  }

  private logPrediction(userId: string, features: Features, prediction: ModelPrediction): void {
    // Critical: store this for ground truth collection
    this.predictionLog.save({
      timestamp: new Date().toISOString(),
      userId,
      features,
      prediction: prediction.result,
      confidence: prediction.confidence,
      modelVersion: prediction.modelVersion
    })
  }
}

Feature Drift Monitoring

from scipy import stats
import numpy as np

class FeatureDriftMonitor:
    def __init__(self, reference_distribution: dict):
        """
        reference_distribution: {feature_name: np.array of training values}
        """
        self.reference = reference_distribution
        
    def check_drift(self, current_window: dict, threshold: float = 0.05) -> dict:
        """
        Uses KS test to detect distribution drift.
        Returns: {feature_name: {'drifted': bool, 'p_value': float}}
        """
        results = {}
        for feature, current_values in current_window.items():
            if feature not in self.reference:
                continue
            
            ks_stat, p_value = stats.ks_2samp(
                self.reference[feature],
                current_values
            )
            
            drifted = p_value < threshold
            results[feature] = {'drifted': drifted, 'p_value': p_value, 'ks_stat': ks_stat}
            
            if drifted:
                logger.warning({'feature': feature, 'p_value': p_value}, 'Feature drift detected')
                metrics.increment('ml.feature_drift', tags=[f'feature:{feature}'])
        
        return results

DO NOT

• DO NOT deploy a new model version to 100% of traffic without A/B testing
• DO NOT make irreversible decisions (account banning, transaction blocking) without a human review step for borderline cases
• DO NOT skip input validation — garbage in produces confident garbage out
• DO NOT use a single aggregate accuracy metric — slice by user segments and time periods
• DO NOT ignore low-confidence predictions — they often signal distribution shift
• DO NOT load ML models synchronously in web request handlers — pre-load at startup
• DO NOT store raw PII as features in prediction logs — hash/anonymize user identifiers
• DO NOT serve a model that wasn't tested on data representative of production distribution

OUTPUT FORMAT

For each ML integration, deliver:

ML Integration Spec:

## ML Integration: [Model Name]

### Model
- Type: Recommendation / Classification / Regression
- Framework: scikit-learn 1.3 / PyTorch 2.1 / ONNX
- Version: v2.4.1 (immutable artifact SHA: abc123)
- Training data: transactions 2023-01 to 2024-01

### Serving
- Endpoint: https://ml-api.internal/v1/recommend
- Method: REST POST / gRPC / embedded
- SLA: P95 < 200ms
- Timeout: 500ms (then fallback)

### Input Features
| Feature | Type | Range/Values | Handling if Missing |
|---------|------|-------------|--------------------| 
| user_age_days | int | 0-3650 | median imputation (365) |
| purchase_count | int | 0-999+ | clip at 999 |
| category | enum | [electronics, clothing, ...] | "unknown" category |

### Fallback Strategy
- Model unavailable: serve most popular items for the session's category
- Low confidence (< 0.6): serve curated editorial picks
- Feature validation failure: serve bestsellers

### Monitoring
- Latency tracking: P50, P95, P99 histograms
- Error rate: model unavailable, timeout, validation failure
- Feature drift: daily KS test vs training distribution
- Business metric: click-through rate on recommendations

### Ground Truth Collection
- Log: prediction request + features + output + model version
- Ground truth: user click/purchase on predicted item (join after 24h)
- Accuracy report: weekly, sliced by user segment and model version

QUALITY GATES

• Input validation implemented and tested with adversarial inputs
• Fallback path tested independently (works without model available)
• Model version pinned and immutable in production config
• Ground truth collection implemented before first prediction served
• Confidence threshold defined and applied for high-stakes predictions
• Feature drift monitoring in place with alert threshold
• P95 inference latency < SLA with model loaded (test before launch)
• A/B test framework in place before first model update
• No PII stored in raw form in prediction logs
• On-call runbook for: model unavailable, high error rate, detected drift