Results for “pairwise-ranking”

Evaluates automated metrics and vision-language models on identifying granular errors in detailed image descriptions and ranking paired descriptions against human judgments, using macro F1, pairwise accuracy, Spearman rank ρ, and Kendall's τ.

Benchmarks language model agents on the C2C multi-agent negotiation task, reporting win rate across starting positions.

Quantifies the minimum accuracy gap needed between two models for a sampled micro-benchmark to reliably preserve their ranking, using the MDAD metric from Yauney et al. (2025).

Scores generated image captions against reference captions and source images using the Polos metric, which is trained to align with human judgments and probes hallucination robustness and open-vocabulary evaluation.

Reranking retrieved documents with cross-encoders and LLM rerankers. Cohere Rerank v3, Voyage rerank-2, BGE reranker, ColBERT late interaction, Jina reranker. Cost and latency tradeoffs, top-K in / top-N out strategy. USE WHEN: user mentions "rerank", "reranker", "cross-encoder", "Cohere Rerank", "Voyage rerank", "BGE reranker", "ColBERT", "Jina reranker", "bi-encoder" DO NOT USE FOR: initial retrieval - use `advanced-retrieval` or `hybrid-search`; query rewriting - use `query-transformations`; agent decisions - use `agentic-rag`

Score LLM and agent outputs using LLM-as-judge techniques — direct scoring against rubrics or pairwise comparison between two outputs. Includes built-in bias mitigation for position bias, length bias, and self-enhancement bias. Load when the user asks to score an output, judge a response, evaluate against a rubric, compare two outputs, do direct scoring, run pairwise comparison, or says "rate this", "which response is better", "score this against the rubric", "judge this output", "LLM as judge this". Sub-skill of eval-output orchestrator.

Analyze URLs or text to discover ranked Polymarket, Hyperliquid, and Deribit markets with Quotient edge scores, paid via on-chain x402 settlement.

Correlate findings across OSINT sources—username enumeration, email lookups, social media profiles, domain records, breach databases, and dark-web mentions—into unified intelligence profiles with confidence scoring and link analysis.

Produce displace/team/ghost talk tracks for the incumbent on a recompete pursuit. Use when user wants competitive angles saved to the pursuit vault; optional multi-turn LLM for customer-facing phrasing.

AI 原生的潜在客户情报和外联流水线。用 agent 驱动的信号评分、共同关系人排名、暖场路径发现、来源语音建模和多渠道外联（邮件、LinkedIn、X），替代 Apollo、Clay 和 ZoomInfo。在用户想找到、评估并联系高价值联系人时使用。

Evaluates the robustness and alignment with human judgment of reference-based and reference-free evaluation metrics for machine translation and summarization, particularly under adversarial conditions.

Evaluate LLM outputs systematically — benchmarks, automated metrics, human preference, and regression tracking

Evaluates image captions by converting them into scene graphs and computing an F-score over semantic propositions, measuring how well a generated caption captures the meaning of an image compared to human references.

Trains language models with Direct Preference Optimization using preference pairs, covering DPOTrainer setup, dataset preparation, and beta tuning for stable preference learning without explicit reward models.