Results for “bias-evaluation”

Provides production-grade techniques for evaluating LLM outputs using LLMs as judges, covering direct scoring, pairwise comparison, bias mitigation, rubric generation, and confidence calibration.

Validates the Emotional Attitude Score (EAS) metric by measuring its consistency with human judgment on word-level sentiment polarity, using the AmbGIMT dataset and pairwise score comparisons.

Evaluates speech language models on beyond-semantic speech attributes such as dialect comprehension, multi-turn context memory, emotion perception, age-aware response generation, and non-verbal cue handling, reporting accuracy and judge-based scores.

Every metric encodes an opinion about which mistake hurts.

Recommend model families and validation strategy based on data, constraints, and objective. Use when: (1) choosing algorithms, (2) balancing bias/variance, (3) planning benchmark baselines. NOT for: final legal/compliance sign-off.

Evaluate LLM outputs systematically — benchmarks, automated metrics, human preference, and regression tracking

Analyzes prompts for safety, bias, security vulnerabilities, and effectiveness, providing detailed improvement recommendations with frameworks, testing methodologies, and educational content.

Screens resumes with skill matching, experience evaluation, and bias-free candidate ranking

Guides statistical hypothesis testing with assumption checks, effect sizes, power analysis, Bayesian alternatives, and APA-formatted reporting for research data.

Rates an agent's own output on five axes — accuracy, completeness, clarity, actionability, conciseness — producing a structured scorecard with evidence and improvement suggestions.

Evaluates text-to-image models on spatial relationship accuracy using the VISOR metric, separating object detection from spatial correctness to reveal biases like object priority and merging.

Evaluates automated metrics and vision-language models on identifying granular errors in detailed image descriptions and ranking paired descriptions against human judgments, using macro F1, pairwise accuracy, Spearman rank ρ, and Kendall's τ.

Evaluates the correlation between automatic text generation scores and human quality ratings, including robustness to domain and quality drift, using metrics like Kendall's Tau and Pearson correlation.

The learner teaches the concept to the AI, which plays a curious novice peer and identifies gaps through authentic questions. Use when the learner wants to test their understanding — teaching forces a different kind of organisation than studying.

Guides the implementation of ethical AI principles, including fairness auditing, bias mitigation, explainability, accountability, and privacy protection in machine learning systems.

Benchmarks automatic speech translation and recognition on English-French and English-Romanian datasets, reporting BLEU and WER on tokenized outputs.

Evaluates machine learning classifiers on their ability to distinguish signal from background in particle physics simulations, measuring how well algorithms rank signal events above background ones using the AUC metric.

Fix HOLD bias in RL reward function. Trigger when: (1) model learns to always HOLD, (2) trade rate is too low (<10%), (3) slippage penalty exceeds typical price moves.

Dataset analysis and data quality specialist. Use when the user asks to talk to Cypher, requests the data detective, or needs dataset assessment, bias analysis, and benchmark evaluation.

AI/LLM red-team assessment using the OWASP LLM Top 10 (2025) + OWASP AI Testing Guide (AITG v1, Nov 2025) frameworks, plus OWASP MCP Top 10 runtime testing for agentic/MCP targets. Tests prompt injection, jailbreaks, system prompt leakage, sensitive data extraction, excessive agency, improper output handling, model extraction, content bias, evasion, membership inference, MCP token exposure, MCP command injection, and more. Uses four tools in combination: FuzzyAI (single-turn jailbreak fuzzing), PyRIT (multi-turn orchestrated attacks), Garak (probe-based vulnerability scanning), and promptfoo (plugin-based red-team evaluation). Each tool covers different OWASP categories; running them together gives systematic coverage. Includes a conditional MCP reconnaissance phase and a post-access AI infrastructure phase (chained from /post-exploit). Produces: OWASP LLM Top 10 + AITG + MCP coverage matrix, findings per category, architecture diagram of the AI system, PoCs for confirmed exploits. Chains into /gh-export for