Results for “bench”

Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.

Testing and benchmarking LLM agents including behavioral testing, capability assessment, reliability metrics, and production monitoring—where even top agents achieve less than 50% on real-world benchmarks Use when: agent testing, agent evaluation, benchmark agents, agent reliability, test agent.

Benchmarks zero-shot in-context learning on BIG-Bench Hard multiple-choice tasks, comparing self-generated demonstrations against direct prompting and chain-of-thought baselines, and reports accuracy.

Guides the creation, iterative improvement, and evaluation of agent skills, including drafting, testing, benchmarking, and optimizing descriptions.

Benchmarks language model agents on the C2C multi-agent negotiation task, reporting win rate across starting positions.

Benchmarks energy-function-based safe control algorithms on the BIS (Benchmark of Interactive Safety) dataset, scoring safety, efficiency, and hybrid performance in human-robot and robot co-working scenarios.

Benchmarks LLM agents fine-tuned with the Agent Data Protocol across software engineering, web browsing, OS/database tool use, and reasoning tasks, reporting unit test pass rates and task success rates.

Create new skills, modify existing ones, and measure their performance through iterative evaluation and benchmarking.

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality.

Routes prompts to any LLM model across multiple providers via CLI tools or APIs, with auto-discovery of new models and benchmark data.

Run config-driven performance benchmarks against a deployed NVIDIA RAG Blueprint server, including profiling and load testing, with a unified report.

Run autonomous optimization loops that iteratively improve artifacts against evaluators using hypothesis tree refinement, without overfitting.

Optimize AI agents through automated evolution cycles using LLM-driven mutation of prompts, skills, and memory against measurable benchmarks.

Access AI-ready drug discovery datasets, benchmarks, and molecular oracles from Therapeutics Data Commons for therapeutic machine learning and pharmacological prediction.

Evaluates RAG pipelines using a filesystem-based benchmark with corpus/ and train.json, running evaluate_rag.py to tune retrieval and generation flags and interpret RAGAS metrics.

Improves AI skills for activation, clarity, and cross-model reliability through benchmarking, salience tuning, and regression triage.

Benchmarks inference latency and computational runtime of transformer models and MLX operations across Apple Silicon and NVIDIA GPU backends, with configurable input lengths and batch sizes.

MMBench: Is Your Multi-modal Model an All-around Player?

Benchmarks a text-to-text transformer across GLUE, SuperGLUE, CNN/Daily Mail, SQuAD, and WMT, reporting GLUE average, BLEU, ROUGE-2-F, and Exact Match scores.

Run multi-agent research discourse session. Use when the user requests to "start a research party" or "run a journal club".

Control a headless browser to navigate pages, click elements, fill forms, take screenshots, record video, and execute JavaScript using Playwright and inference.sh.

Configure and use Honcho memory with Hermes -- cross-session user modeling, multi-profile peer isolation, observation config, dialectic reasoning, session summaries, and context budget enforcement. Use when setting up Honcho, troubleshooting memory, managing profiles with Honcho peers, or tuning observation, recall, and dialectic settings.

Orchestrates a multi-agent build session, acting as manager to cut goals into units, spawn implementer and tester subagents, and run test-and-review loops with anti-thrash guardrails.

Benchmarks medical AI agents on synthetic EHR tasks, measuring success rates for data retrieval, temporal aggregation, and threshold-based conditional logic with exact-match scoring.

Queries Hugging Face benchmark leaderboards to find the best AI models for a task, filters by device constraints, and returns a ranked comparison table with scores.

Reduce per-token latency on Jetson vLLM servers by appending speculative decoding configuration, with guidance on when to enable and how to benchmark the improvement.

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) using standardized prompts and metrics. Supports HuggingFace, vLLM, and API backends.

Benchmarks LLM bias by scoring responses to automatically generated open-ended questions across sensitive attributes, producing a composite fitness score from 0 to 5.

Evaluates the correlation between a zero-cost NAS metric (epsilon) and actual training accuracy across different neural architecture search spaces, testing the metric's ability to rank architectures without training. It probes whether output dispersion from constant weight initializations can serve as a reliable.

Evaluates social bias in question-answering models using the BBQ benchmark, measuring accuracy and a bias score across ambiguous and disambiguated contexts to reveal reliance on stereotypes.