Best Practices

Best Practices

This file curates blogs (✍️), best practices, architectural guidance, and implementation tips from across all LLM topics.

Contents

• RAG Best Practices
  • The Problem with RAG
  • RAG Solution Design
  • RAG Research
  • RAG Research (Ranked by cite count >=100)
• Agent Best Practices
  • Agent Design Patterns
  • Agent Research
  • Agent Research (Ranked by cite count >=100)
  • Reflection, Tool Use, Planning and Multi-agent collaboration
  • Tool Use: LLM to Master APIs
• Proposals & Glossary

RAG Best Practices

The Problem with RAG

• Seven Failure Points When Engineering a Retrieval Augmented Generation System📑: 1. Missing Content, 2. Missed the Top Ranked Documents, 3. Not in Context, 4. Not Extracted, 5. Wrong Format, 6. Incorrect Specificity, 7. Lack of Thorough Testing [11 Jan 2024]
• Solving the core challenges of Retrieval-Augmented Generation ✍️ [Feb 2024]
• The Problem with RAG
  • A question is not semantically similar to its answers. Cosine similarity may favor semantically similar texts that do not contain the answer.
  • Semantic similarity gets diluted if the document is too long. Cosine similarity may favor short documents with only the relevant information.
  • The information needs to be contained in one or a few documents. Information that requires aggregations by scanning the whole data.

RAG Solution Design

• Advanced RAG with Azure AI Search and LlamaIndex✍️
• Announcing cost-effective RAG at scale with Azure AI Search✍️
• Azure OpenAI chat baseline architecture in an Azure landing zone
• GPT-RAG✨: Enterprise RAG Solution Accelerator [Jun 2023]
• bRAG✨: Everything you need to know to build your own RAG application [Nov 2024]
• Evaluating LLMs and RAG Systems✍️: Best Practices for Evaluating LLMs and RAG Systems [27 Jan 2025]
• From Zero to Hero: Proven Methods to Optimize RAG for Production✍️: ColBERT (Token-level embedding), CoPali(Extends ColBERT's multi-vector retrieval and late interaction from text to vision), RAPTOR, HyDE, Re-Ranking and Fusion [Sep 2025]
• Galileo eBook: 200 pages content. Mastering RAG. 🗄️ [Sep 2024]
• Genie: Uber's Gen AI On-Call Copilot✍️ [10 Oct 2024]
• Introduction to Information Retrieval: The official website for the classic textbook (free to read online) "Introduction to Information Retrieval" by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze.
• Introduction to Large-Scale Similarity Search: HNSW, IVF, LSH✍️ [28 Sep 2024]
• LangChain RAG from scratch✨ 📺 [Jan 2024]
• LLM Twin Course: Building Your Production-Ready AI Replica✨: Learn to Build a Production-Ready LLM & RAG System with LLMOps [Mar 2024]
• LlamIndex Building Performant RAG Applications for Production
• Papers with code: RAG
• RAG at scale✍️: Building a distributed system for synchronizing and ingesting billions of text embeddings [28 Sep 2023]
• RAG context relevancy metric: Ragas, TruLens, DeepEval ✍️ [Jun 2024]
  • Context Relevancy (in Ragas) = S / Total number of sentences in retrieved context
  • Contextual Relevancy (in DeepEval) = Number of Relevant Statements / Total Number of Statements
• RAG-driven Generative AI✨: Retrieval Augmented Generation (RAG) code for Generative AI with LlamaIndex, Deep Lake, and Pinecone [Apr 2024]
• What AI Engineers Should Know about Search [25 Jun 2024]

RAG Research

• A Survey on Retrieval-Augmented Text Generation📑: This paper conducts a survey on retrieval-augmented text generation, highlighting its advantages and state-of-the-art performance in many NLP tasks. These tasks include Dialogue response generation, Machine translation, Summarization, Paraphrase generation, Text style transfer, and Data-to-text generation. [2 Feb 2022]
• Adaptive-RAG📑💡: Learning to adapt RAG via question complexity; dynamic retrieval strategies. [Mar 2024] ✨
• Active Retrieval Augmented Generation📑💡 : FLARE anticipates next sentences; retrieves/regenerates when token confidence is low. [May 2023] ✨
• ARAG📑: Agentic Retrieval Augmented Generation for Personalized Recommendation [27 Jun 2025]
• Astute RAG📑: adaptively extracts essential information from LLMs, consolidates internal and external knowledge with source awareness, and finalizes answers based on reliability. [9 Oct 2024]
• Benchmarking Large Language Models in Retrieval-Augmented Generation📑💡: Retrieval-Augmented Generation Benchmark (RGB) assesses 4 key abilities. [Sep 2023]:
  • 1. Noise robustness (External documents contain noises, struggled with noise above 80%)
    2. Negative rejection (External documents are all noises, Highest rejection rate was only 45%)
    3. Information integration (Difficulty in summarizing across multiple documents, Highest accuracy was 60-67%)
    4. Counterfactual robustness (Failed to detect factual errors in counterfactual external documents.)

1. Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation📑: xMemory — decouples and aggregates agent memories for compact, non-redundant retrieval. [Feb 2026]

• CAG: Cache-Augmented Generation📑: Preloading Information and Pre-computed KV cache for low latency and minimizing retrieval errors [20 Dec 2024] ✨
• CoRAG📑: Chain-of-Retrieval Augmented Generation. RAG: single search -> CoRAG: Iterative search and reasoning [24 Jan 2025]
• Corrective Retrieval Augmented Generation (CRAG)📑: Retrieval Evaluator assesses the retrieved documents and categorizes them as Correct, Ambiguous, or Incorrect. For Ambiguous and Incorrect documents, the method uses Web Search to improve the quality of the information. The refined and distilled documents are then used to generate the final output. [29 Jan 2024] CRAG implementation by LangGraph ✨
• CRAG: Comprehensive RAG Benchmark📑: a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search ✍️ [7 Jun 2024]
• Discuss-RAG📑: Agent-Led Discussions for Better RAG in Medical QA [30 Apr 2025]
• FreshLLMs📑: Fresh Prompt, Google search first, then use results in prompt. Our experiments show that FreshPrompt outperforms both competing search engine-augmented prompting methods such as Self-Ask (Press et al., 2022) as well as commercial systems such as Perplexity.AI. ✨ [5 Oct 2023]
• Graph Retrieval-Augmented Generation: A Survey📑 [15 Aug 2024]
• From Local to Global: GraphRAG📑💡: Entity graph + community summaries to scale QFS over large corpora. [Apr 2024]
• HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation📑: fine-tunes LLMs to learn a three-level hierarchical reasoning process — Filtering: select relevant information; Combination: synthesize across documents; RAG-specific reasoning: infer using retrieved documents plus internal knowledge. [8 Jul 2025]
• HippoRAG📑💡: RAG + Long-Term Memory (Knowledge Graphs) + Personalized PageRank. [23 May 2024]
• Hyde📑: Hypothetical Document Embeddings. zero-shot (generate a hypothetical document) -> embedding -> avg vectors -> retrieval [20 Dec 2022]
• INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning📑: INTERS covers 21 search tasks across three categories: query understanding, document understanding, and query-document relationship understanding. The dataset is designed for instruction tuning, a method that fine-tunes LLMs on natural language instructions. ✨ [12 Jan 2024]
• Medical RAG Benchmark (MIRAGE)📑💡: MedRAG toolkit; RAG improves LLM accuracy up to 18% with mixed corpora/retrievers. [Feb 2024]
• OP-RAG: Order-preserve RAG📑: Unlike traditional RAG, which sorts retrieved chunks by relevance, we keep them in their original order from the text. [3 Sep 2024]
• PlanRAG📑: Decision Making. Decision QA benchmark, DQA. Plan -> Retrieve -> Make a decision (PlanRAG) ✨ [18 Jun 2024]
• RAG Evaluation: RAGAs📑💡: Reference-free metrics to assess context relevance, faithfulness, and answer quality. [Sep 2023]
• RAG for LLMs📑💡: Survey: Naive → Advanced → Modular RAG; practices and benchmarks. [Dec 2023]
• RAG vs Fine-tuning📑: Pipelines, Tradeoffs, and a Case Study on Agriculture. [16 Jan 2024]
• RARE: Retrieval-Augmented Reasoning Modeling 📑: LLM Training from static knowledge storage (memorization) to reasoning-focused. Combining domain knowledge and thinking (Knowledge Remembering & Contextualized Reasoning). 20% increase in accuracy on tasks like PubMedQA and CoVERT. [30 Mar 2025]
• RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval📑: Introduce a novel approach to retrieval-augmented language models by constructing a recursive tree structure from documents. ✨ pip install llama-index-packs-raptor / ✨ [31 Jan 2024]
• RECOMP: Improving Retrieval-Augmented LMs with Compressors📑: 1. We propose RECOMP (Retrieve, Compress, Prepend), an intermediate step which compresses retrieved documents into a textual summary prior to prepending them to improve retrieval-augmented language models (RALMs). 2. We present two compressors – an extractive compressor which selects useful sentences from retrieved documents and an abstractive compressor which generates summaries by synthesizing information from multiple documents. 3. Both compressors are trained. [6 Oct 2023]
• REFRAG: Rethinking RAG based Decoding📑: Meta. Compress (chunk → single embedding) → Sense/Select (RL policy picks relevant chunks) → Expand (Selective expansion). 16× Longer Contexts and Up to 31× Faster RAG Decoding. [1 Sep 2025] ✍️
• Retrieval Augmented Generation (RAG) and Beyond📑:🏆The paper classifies user queries into four levels—explicit, implicit, interpretable rationale, and hidden rationale—and highlights the need for external data integration and fine-tuning LLMs for specialized tasks. [23 Sep 2024]
• Retrieval Augmented Generation or Long-Context LLMs?📑: Long-Context consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage. [23 Jul 2024]
• Retrieval meets Long Context LLMs📑: We demonstrate that retrieval-augmentation significantly improves the performance of 4K context LLMs. Perhaps surprisingly, we find this simple retrieval-augmented baseline can perform comparable to 16K long context LLMs. [4 Oct 2023]
• Retrieval-Augmentation for Long-form Question Answering📑: 1. The order of evidence documents affects the order of generated answers 2. the last sentence of the answer is more likely to be unsupported by evidence. 3. Automatic methods for detecting attribution can achieve reasonable performance, but still lag behind human agreement. Attribution in the paper assesses how well answers are based on provided evidence and avoid creating non-existent information. [18 Oct 2023]
• Searching for Best Practices in Retrieval-Augmented Generation📑: Best Performance Practice: Query Classification, Hybrid with HyDE (retrieval), monoT5 (reranking), Reverse (repacking), Recomp (summarization). Balanced Efficiency Practice: Query Classification, Hybrid (retrieval), TILDEv2 (reranking), Reverse (repacking), Recomp (summarization). [1 Jul 2024]
• Self-RAG💡: Critic tokens (IsREL/IsSUP/IsUse) guide retrieval and generation; adaptive on-demand retrieval. [Oct 2023] ✨
• Sufficient Context: A New Lens on Retrieval Augmented Generation Systems📑: Use Gemini 1.5 Pro (1-shot) as a Sufficient Context AutoRater to verify if enough context is provided. Larger models perform well with sufficient context but often make wrong guesses when information is missing. Accuracy improves by combining Model Self-Rated Confidence with the Sufficient Context AutoRater. [9 Nov 2024]
• The Power of Noise: Redefining Retrieval for RAG Systems📑: No more than 2-5 relevant docs + some amount of random noise to the LLM context maximizes the accuracy of the RAG. [26 Jan 2024]
• Towards AI Search Paradigm📑: Modular 4-agent system (orchestrator, planner, executor, writer) using DAGs for retrieval-intensive search. [Jun 2025]
• UniversalRAG📑: Modality-aware and granularity-aware routing across text/image/video; flexible zero-shot vs trained routing. [Apr 2025]

RAG Research (Ranked by cite count >=100)

1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks📑: Foundational RAG recipe combining seq2seq with dense retrieval; set SOTA on open-domain QA. [May 2020] (Citations: 9,559)
2. Retrieval-Augmented Generation for Large Language Models: A Survey📑: Comprehensive review of Naive/Advanced/Modular RAG paradigms and evaluation frameworks. [Dec 2023] (Citations: 2,570)
3. Self-RAG📑: Trains LMs to retrieve on-demand and generate reflection tokens to control factuality. [Oct 2023] (Citations: 1,203)
4. GraphRAG📑: Builds an entity graph + community summaries to answer global, corpus-level questions. [Apr 2024] (Citations: 835)
5. RAG Meets LLMs: Survey📑: Surveys Retrieval-Augmented LLMs (RA-LLMs), limitations, and future directions. [May 2024] (Citations: 531)
6. Active Retrieval Augmented Generation (FLARE)📑: Iteratively anticipates future content and retrieves when confidence is low for long-form tasks. [May 2023] (Citations: 522)
7. Benchmarking LLMs in RAG (RGB)📑: Benchmarks noise robustness, negative rejection, integration, and counterfactual robustness. [Sep 2023] (Citations: 481)
8. RAGAs📑: Reference-free evaluation metrics for retrieval and generation fidelity in RAG pipelines. [Sep 2023] (Citations: 470)
9. Medical RAG Benchmark (MIRAGE)📑: 7,663 medical QA questions; toolkit shows RAG combos boosting accuracy up to 18%. [Feb 2024] (Citations: 404)
10. Adaptive-RAG📑: Dynamically selects RAG strategies based on query complexity. [Mar 2024] (Citations: 369)
11. RAG for AIGC Survey📑: Surveys RAG foundations and applications for AI-generated content across modalities. [Feb 2024] (Citations: 497)
12. RAFT📑: Fine-tuning with distractor documents improves domain-specific RAG open-book in-domain QA. [Mar 2024] (Citations: 312)
13. The Power of Noise📑: 2-5 relevant docs + random noise maximizes RAG accuracy. [Jan 2024] (Citations: 329)
14. Domain Adaptation for RAG📑: Joint training of retriever and generator for domain-specific knowledge bases. [Oct 2022] (Citations: 258)
15. Graph RAG Survey📑: Comprehensive survey of GraphRAG methodologies and applications. [Aug 2024] (Citations: 306)
16. Evaluation of RAG: A Survey📑: Unifies evaluation metrics and benchmarks for retrieval and generation components. [May 2024] (Citations: 218)
17. MultiHop-RAG📑: Benchmark for multi-hop queries requiring multi-evidence retrieval and reasoning. [Jan 2024] (Citations: 206)
18. RAGTruth📑: Word-level hallucination corpus for RAG responses across domains and tasks. [Jan 2024] (Citations: 209)
19. G-Retriever📑: Prize-Collecting Steiner Tree formulation for textual graph RAG QA. [Feb 2024] (Citations: 202)
20. Seven Failure Points in RAG📑: Engineering failures and lessons learned across domains. [Jan 2024] (Citations: 186)
21. KG RAG for Customer Support📑: Builds KG from tickets to improve retrieval and QA quality. [Apr 2024] (Citations: 152)
22. RankRAG📑: Instruction-tunes one LLM for both context ranking and answer generation. [Jul 2024] (Citations: 196)
23. Chain of Agents (long-context)📑: Multi-agent chain processes long inputs via segmented reading and synthesis. [Jun 2024] (Citations: 180)
24. RAG Privacy📑: Analyzes privacy risks in RAG databases and LLM training data leakage. [Feb 2024] (Citations: 133)
25. RAG vs Fine-tuning📑: Compares RAG and fine-tuning pipelines with agriculture case study. [Jan 2024] (Citations: 148)
26. RQ-RAG📑: Learns to rewrite, decompose, and disambiguate queries to improve retrieval. [Apr 2024] (Citations: 157)
27. Agentic RAG Survey📑: Taxonomy and design patterns for Agentic RAG workflows. [Jan 2025] (Citations: 202)
28. CRAG📑: Retrieval evaluator triggers web search and decompose-recompose filtering. [Jan 2024] (Citations: 158)
29. GNN-RAG📑: Combines GNN reasoning and LLM RAG for KGQA; excels on multi-hop queries. [May 2024] (Citations: 152)
30. eRAG📑: Document-level downstream evaluation correlates better with end-to-end RAG. [Apr 2024] (Citations: 150)
31. FlashRAG📑: Modular toolkit implementing 16 RAG methods and 38 benchmarks. [May 2024] (Citations: 157)
32. HybridRAG📑: Combines GraphRAG + VectorRAG; outperforms both for finance Q&A. [Aug 2024] (Citations: 149)
33. Clinical RAG for Hepatology📑: Structured guideline reformatting + prompt engineering boosts CDSS accuracy. [—] (Citations: 118)
34. Blended RAG📑: Hybrid dense+sparse retrievers with query strategies improve IR and QA. [Apr 2024] (Citations: 115)
35. LightRAG📑: Graph-based indexing and incremental updates for fast, accurate RAG. [Oct 2024] (Citations: 108)
36. GraphRAG Survey (2025)📑: Holistic framework and domain-tailored designs for GraphRAG. [Jan 2025] (Citations: 151)
37. Structured Output RAG📑: RAG reduces hallucinations in structured workflow generation. [Apr 2024] (Citations: 133)
38. HippoRAG📑: KG + Personalized PageRank for efficient long-term memory integration. [May 2024] (Citations: 136)
39. VisRAG📑: Vision-based page-as-image retrieval for multimodal document RAG; outperforms text-only pipelines. [Oct 2024] (Citations: 135)
40. Medical Graph RAG📑: Hierarchical knowledge graph for safe medical LLM RAG on clinical notes. [Aug 2024] (Citations: 122)
41. Best Practices in RAG📑: Query classification + hybrid + HyDE + monoT5 reranking + Recomp summarization. [Jul 2024] (Citations: 120)
42. PoisonedRAG📑: Knowledge corpus poisoning attacks against RAG achieve high attack success rates. [Feb 2024] (Citations: 111)
43. RAG vs Long-Context LLMs📑: Long-context consistently outperforms RAG on average; RAG retains cost advantage. [Jul 2024] (Citations: 110)
44. RAG for NLP Survey📑: Survey of RAG frameworks across NLP tasks: modules, training, and adaptation. [Jul 2024] (Citations: 108)
45. RAG Text Generation Survey📑: Comprehensive survey on retrieval-augmented text generation methods and benchmarks. [Apr 2024] (Citations: 101)
46. ColPali📑: Vision-Language Models for efficient document page retrieval outperform text-only OCR pipelines. [Jul 2024] (Citations: 100)

Agent Best Practices

• AIAgentToolkit.xyz: A curated list of AI agent frameworks, launchpads, tools, tutorials, & resources.
• AI Engineering Guidebook: 💡350+ pages. Daily Dose of Data Science. [26 Dec 2025]
• Agent Leaderboard🤗
• Agent Leaderboard v2✨
• How AI is transforming work at Anthropic✍️: Claude Code covers ~60% of engineering work with ~50% productivity gain. [3 Dec 2025]
• We Got Claude to Fine-Tune an Open Source LLM✍️: 🤗 Hugging Face skill for training an LLM. [4 Dec 2025]

Agent Design Patterns

• 10 Lessons to Get Started Building AI Agents✨: 💡Microsoft. [Nov 2024]
• 5 Agentic AI Design Patterns✍️: Reflection, Tool use, ReAct, Planning, Multi-agent pattern [24 Jan 2025]
• A Practical Guide to Building AI Agents✍️: 💡OpenAI. [11 Mar 2025]
• Agent Factory: The new era of agentic AI—common use cases and design patterns✍️:💡Tool use, Reflection, Planning, Multi-agent, ReAct (Reason + Act) Patterns [13 Aug 2025]
• Agentic AI Architecture Framework for Enterprises ✍️:💡Tier 1: Foundation: Establishing Controlled Intelligence, Tier 2: Workflow, Tier 3: Autonomous (experimental) [11 Jul 2025]
• Agentic Design Patterns✨: Jupyter notebook hands-on guide to building agents. [Sep 2025]
• Agentic Architectures for Retrieval-intensive Applications: Published by Weviate [Mar 2025]
• Agentic Design Patterns✍️: Google Docs. A Hands-On Guide to Building Intelligent Systems. 🗄️ [May 2025]
• Agents:💡Chip Huyen [7 Jan 2025]
• AI agent orchestration patterns:💡Sequential, Concurrent, Group chat, Handoff, Magentic orchestration [17 Jun 2025]
• Azure AI Foundry Agent Ochestration Patterns : Prompt Chaining, Routing, Parallelization, Orchestrator, Evaluator-optimizer
• Beyond the Gang of Four: Practical Design Patterns for Modern AI Systems✍️: Prompting & Context: Few-Shot Prompting, Role Prompting, Chain-of-Thought, RAG; Responsible AI: Output Guardrails, Model Critic; UX: Contextual Guidance, Editable Output, Iterative Exploration; AI-Ops: Metrics-Driven AI-Ops, Prompt-Model-Config Versioning; Optimization: Prompt Caching, Dynamic Batching, Intelligent Model Routing. [15 My 2025]
• Building effective agents✍️:💡Anthrophic. Prompt Chaining, Sequential LLM calls, Routing, Input classification, Parallelization, Concurrent execution, Orchestrator-Workers, Delegation, Evaluator-Optimizer, Feedback loops, Iterative refinement [19 Dec 2024]
• Exploring Generative AI (martinfowler.com): Memos on how LLMs are being used to enhance software delivery practices, including Toochain, Test-Driven Development (TDD) with GitHub Copilot, pair programming, and multi-file editing. [26 Jul 2023 ~ ]
• Generative AI Design Patterns for Agentic AI Systems✨: Design Patterns for Agentic solutions in Azure [May 2023]
• Google AI Agents Whitepaper [12 Nov 2024]
• How Anthropic Built a Multi-Agent Research System✍️: Lead Researcher agent to coordinate specialized subagents, enabling dynamic, parallelized research and citation verification [17 Sep 2025]
• How to write a great agents.md: Lessons from over 2,500 repositories✍️: How to write effective agents.md files for GitHub Copilot with practical tips [19 Nov 2025]
• How we built our multi-agent research system✍️: Anthrophic. [13 Jun 2025]
• Hugging Face Agents Course✨ 🤗 Hugging Face Agents Course. [Jan 2025]
• Language Agent Tree Search Method (LATS)✨: LATS leverages an external environment and an MCTS (Monte Carlo Tree Search)-based search [6 Oct 2023]
• Taxonomy of failure modes in AI agents ✍️: Microsoft AI Red Team (AIRT) has categorized identified failure modes into two types: novel and existing, under the pillars of safety and security. [24 Apr 2025]
• The Different Ochestration Frameworks:💡Orchestration frameworks for LLM applications: Micro-orchestration / Macro-orchestration / Agentic Design Frameworks / Optimizer frameworks [11 Oct 2024]
• The Last Mile Problem: Why Your AI Models Stumble Before the Finish Line: According to Gartner, by 2025, at least 30 percent of GenAI projects will be abandoned after the POC stage. [25 Oct 2024]
• When One AI Agent Isn't Enough - Building Multi-Agent Systems✍️: When Multi-Agent Systems Make Sense: Complex subtasks, diverse expertise, parallel speed, easy scaling, and multi-entity problems. [Jul 14]
• Zero to One: Learning Agentic Patterns:💡Structured workflows follow fixed paths, while agentic patterns allow autonomous, dynamic decision-making. Sample code using Gemini. [5 May 2025]

Agent Research

1. A Practical Approach for Building Production-Grade Conversational Agents with Workflow Graphs📑: DAG-based workflows for e-commerce tasks with response and loss masking. [May 2025]
2. Advances and Challenges in Foundation Agents📑: Brain-inspired to evolutionary, collaborative, and safe agent systems. [Mar 2025]
3. Agent-as-a-Judge📑: Evaluate agents with agents; DevAI benchmark of 55 development tasks. [Oct 2024]
4. AgentBench📑: Evaluates LLM-as-Agent reasoning and decision-making. [Aug 2023]
5. Agentic Context Engineering📑: ACE lets models self-improve by evolving contexts without fine-tuning. [Oct 2025]
6. Agentic Reasoning for Large Language Models📑: Survey of agentic reasoning: planning, self-evolution, and multi-agent collaboration. [Jan 2026]
7. AgentQ📑: Off-policy policy gradient for multi-step agent planning. [13 Aug 2024]
8. Agents Are Not Enough📑: Proposes ecosystem of agents, sims, and assistants. [Dec 2024]
9. AI Agents That Matter📑: Agent evaluations for accuracy and cost optimization. [Jul 2024]
10. Automated Design of Agentic Systems📑: ADAS auto-creates agentic designs with novel building blocks. [Aug 2024]
11. Cognitive Architectures for Language Agents📑: CoALA with procedural, semantic, and episodic memory. [Sep 2023]
12. Darwin Godel Machine📑: Self-modifying agents evolve via SWE-bench/Polyglot validation. [May 2025]
13. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?📑: AGENTS.md often reduces agent success rates and increases cost — keep them minimal. [Feb 2026]
14. AgentCoder📑: Multi-agent code generation with iterative testing. [Dec 2023]
15. LDB📑: LLM debugger verifying runtime execution step by step. [Feb 2024]
16. Generative Agent Simulations of 1,000 People📑: Simulates 1,000 real individuals; 85% General Social Survey accuracy. [Nov 2024]
17. Recursive Language Models (RLMs), a general inference strategy📑: A new general inference strategy that lets large language models process arbitrarily long inputs by treating the entire prompt as an external environment and programmatically decomposing and recursively calling the model on pieces of the input. [Dec 2025]
18. Solving a Million-Step LLM Task with Zero Errors📑: MDAP framework with MAKER decomposition and error correction. [Nov 2025]
19. The Rise and Potential of LLM-Based Agents📑: Survey of LLM-based agents. [Sep 2023]
20. Towards a Science of Scaling Agent Systems📑: Adding agents doesn’t guarantee better performance—multi‑agent AI can drop up to 70% depending on task, coordination, and architecture (Google & MIT, 15,000 trials). [Dec 2025]

Agent Research (Ranked by cite count >=100)

1. AutoGen Framework📑: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework. (Citations: 832)
2. Mind2Web📑: Generalist web agents dataset with 2,000 tasks across 137 sites. [Jun 2023] (Citations: 852)
3. ChatEval📑: Towards Better LLM-based Evaluators through Multi-Agent Debate. [Aug 2023] (Citations: 777)
4. SWE-agent📑: Agent-computer interfaces enable autonomous repo navigation, editing, and testing. [May 2024] (Citations: 569)
5. Multi-Agent Collaboration📑: Framework analyzing Auto-GPT/BabyAGI; collaboration improves task handling. [Jun 2023] (Citations: 369)
6. Character-LLM📑: A Trainable Agent for Role-Playing. [Oct 2023] (Citations: 374)
7. Planning Survey📑: Taxonomy of agent planning: decomposition, selection, external modules, memory. [Feb 2024](Citations: 373)
8. ExpeL📑: Agents learn from experiences without param updates; strong transfer. [Aug 2023](Citations: 381)
9. LATS📑: MCTS-based search unifies reasoning, acting, planning in LM agents. [Oct 2023](Citations: 377)
10. PLMs for Decision-Making📑: LM-initialized policies for sequential decision tasks. [Feb 2022](Citations: 299)
11. CodeAct📑: Executable Python actions unify tool use; improves success on API-Bank. [Feb 2024](Citations: 356)
12. OpenHands📑: Open platform for developer agents with sandboxing and benchmarks. [Jul 2024](Citations: 281)
13. Personal LLM Agents Survey📑: Architecture, capability, efficiency, security of personal agents. [Jan 2024](Citations: 295)
14. Agentless📑: Simple three-phase approach rivals complex SWE agents. [Jul 2024](Citations: 276)
15. Collaboration Mechanisms📑: Social-psychology-inspired multi-agent societies (debate, reflection). [Oct 2023](Citations: 229)
16. RepairAgent📑: An Autonomous, LLM-Based Agent for Program Repair. [Mar 2024] (Citations: 242)
17. Biomedical Agents📑: Agents accelerate biomedical discovery; integration pathways. [Apr 2024] (Citations: 182)
18. ToolEmu Risk Sandbox📑: LM-emulated tool execution for agent risk analysis. [Sep 2023] (Citations: 217)
19. Agents as Models📑: LMs infer communicative intentions; building blocks for intentional agents. [Dec 2022](Citations: 170)
20. MACHIAVELLI📑: Benchmarks trade-offs between reward and ethical behavior. [Apr 2023](Citations: 176)
21. LoCoMo Memory📑: Very long-term conversational memory benchmark (300 turns). [Feb 2024](Citations: 270)
22. AgentPoison📑: Backdoor poisoning of agen

…(truncated)