Results for “attitudinal-metrics”

TRAK: Attributing Model Behavior at Scale

Validates the Emotional Attitude Score (EAS) metric by measuring its consistency with human judgment on word-level sentiment polarity, using the AmbGIMT dataset and pairwise score comparisons.

Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions.

Every metric encodes an opinion about which mistake hurts.

Benchmarks language model agents on the C2C multi-agent negotiation task, reporting win rate across starting positions.

Benchmarks LLM bias by scoring responses to automatically generated open-ended questions across sensitive attributes, producing a composite fitness score from 0 to 5.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Fundamental patterns for effective agentic behavior. Teaches decomposition, tool orchestration, error recovery, context management, quality self-assessment, and knowing when to stop. Model-agnostic principles that make any agent more effective regardless of domain. Activate on: "how should I structure this agent", "agentic workflow", "agent patterns", "multi-step task", "tool orchestration", "/agentic-patterns", "decompose this", "agent best practices", "chain of actions", "when should the agent stop", "agent loop design". NOT for: creating agent infrastructure (use agent-creator), building DAGs (use windags-architect), specific tool implementation.

Slow down interpretation from observation to action. Use when students or adults need to examine assumptions in conflict, dialogue, or inquiry.

Design privacy-aware observability for AI agents using traces, spans, structured events, metrics, cost attribution, dashboards, alerts, and investigation workflows. Use when instrumenting an agent, debugging intermittent tool or model failures, defining service-level objectives, analyzing latency or spend, auditing agent decisions, or preparing production monitoring.

Computes the AUROC metric using torchmetrics, handling binary, multiclass, and multilabel tasks with configurable thresholds and averaging.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session. Use when the user runs /hub:eval or asks to score, compare, or pick a winner among completed AgentHub agents.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Evaluate LLM outputs systematically — benchmarks, automated metrics, human preference, and regression tracking

Benchmarks medical AI agents on synthetic EHR tasks, measuring success rates for data retrieval, temporal aggregation, and threshold-based conditional logic with exact-match scoring.

Evaluate and rank agent results by metric or LLM judge for an AgentHub session.

Optimize prompts via process goals (controllable behavioral instructions) rather than outcome goals (sparse end-result demands). Grounded in sports psychology meta-analysis showing process goals (d=1.36) vastly outperform outcome goals (d=0.09). Use when designing prompts, optimizing LLM steering, implementing CoT/decomposition patterns, or building automatic prompt optimization pipelines. Instantiates surrogate loss paradigm for discrete prompt space.