AI Agents Frameworks - Comparative Learning Repository

AI Agents Frameworks - Comparative Learning Repository

Research Date: 2026-01-31 Source URL: https://github.com/martimfasantos/ai-agents-frameworks GitHub Repository: https://github.com/martimfasantos/ai-agents-frameworks Version at Research: Main branch (pushed 2026-01-24) License: Not specified (no LICENSE file)

Overview

AI Agents Frameworks is a hands-on learning repository that provides practical implementations and comprehensive benchmarks comparing 10 state-of-the-art AI agent frameworks. The repository enables developers to explore, test, and compare modern agent architectures through working examples ranging from simple "hello world" agents to complex multi-agent workflows with RAG and API integrations.

Core Value Proposition: Side-by-side comparison of agent frameworks using consistent benchmarks for response time, token usage, memory effects, and tool utilization accuracy.

Problem Addressed

Key Statistics (as of 2026-01-31)

Frameworks Covered

Key Features

Example Coverage by Framework

Each framework directory contains numbered, progressive examples:

CrewAI Examples (14 examples):

• hello_world, tools, built_in_tools, streaming
• structured_outputs, tasks, callbacks, memory
• reasoning, knowledge, multi_agent_collaboration
• flows, flows_with_agents, crew_simplification

Pydantic-AI Examples (11 examples):

• hello_world, tools_and_metrics, built_in_tools
• streaming, structured_outputs, output_validators
• message_history, agent_delegation
• programmatic_handoff, stateful_graphs, human_in_the_loop

OpenAI Agents SDK Examples (9 examples):

• hello_world, tools_and_metrics, structured_outputs
• parallelization_in_workflow, handoffs_and_streaming
• agents_as_tools, output_guardrails, llm_as_a_judge, tracing

LangGraph/LlamaIndex/Agno: Similar progressive example structure

Unified Benchmark Study

The study-agents-differences/ directory provides:

1. Unified Agent Interfaces: Consistent implementation across Agno, LangGraph, LlamaIndex, OpenAI, Pydantic-AI
2. Performance Metrics: Response time, token usage, tool utilization
3. Configurable Benchmarks: Memory on/off, agent recreation per iteration
4. Streamlit UI: Interactive comparison interface (streamlit run agent-ui.py)

Benchmark Categories

Technical Architecture

Benchmark Infrastructure

study-agents-differences/
├── *_agent.py              # Framework-specific agent implementations
├── *_rag_api_agent.py      # RAG + API integrated variants
├── prompts.py              # Consistent system prompts across agents
├── settings.py             # Shared configuration
├── shared_functions/       # Reusable tool implementations
├── knowledge_base/         # RAG vector store data
├── tests/                  # Benchmark output files
└── agent-ui.py             # Streamlit comparison interface

Benchmark Command Pattern

python llama_index_rag_api_agent.py \
  --mode metrics-loop \
  --iter 30 \
  --create \
  --no-memory \
  --verbose \
  --file tests/test100_llamaindex_rag.txt

Flags:

• --provider [azure|openai|other]: LLM backend
• --mode [None|metrics|metrics-loop]: Execution mode
• --iter [int]: Iteration count for loops
• --no-memory: Disable conversation memory
• --create: Recreate agent instance per iteration
• --file [path]: Output results file

Benchmark Results Summary

Response Time with Memory (100 iterations)

Key Finding: LangGraph shows significant performance degradation as conversation history grows. At iteration 100, performance drops substantially due to memory overhead.

Response Time without Memory (100 iterations)

Key Finding: Without memory, all frameworks perform similarly. LangGraph's memory implementation is the primary performance differentiator.

Token Usage Comparison

Key Finding: Token usage heavily depends on system prompt design. OpenAI and LangGraph are most token-efficient.

RAG Performance (100 iterations)

Key Finding: LlamaIndex achieves best token efficiency for RAG. All frameworks show <5% miss rate for retrieval tasks.

API Integration (100 iterations, multi-tool prompt)

Key Finding: LangGraph excels at multi-tool orchestration with lowest token overhead. LlamaIndex requires prompt tuning for complex tool scenarios.

Installation and Usage

Quick Start

# Clone repository
git clone https://github.com/martimfasantos/ai-agents-frameworks.git
cd ai-agents-frameworks

# Navigate to specific framework
cd crewai  # or langgraph, pydantic-ai, etc.

# Check framework-specific README
cat README.md

# Install dependencies (varies by module)
pip install -r requirements.txt  # or use PDM/uv as specified

Running Benchmarks

cd study-agents-differences

# Create environment
python3 -m venv .venv && source .venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure API keys
cp .env.example .env
# Edit .env with your keys

# Run Streamlit UI for interactive comparison
streamlit run agent-ui.py

Relevance to Claude Code Development

Direct Applications

1. Agent Design Patterns: Working examples of tool integration, memory management, and multi-agent coordination applicable to Claude Code agent development
2. Benchmark Methodology: Reproducible framework for comparing Claude Code sub-agent configurations and delegation strategies
3. Memory Trade-offs: Empirical data on memory impact showing when stateless agents outperform stateful ones
4. Tool Integration Patterns: RAG and API integration examples transferable to MCP tool development

Patterns Worth Adopting

1. Progressive Example Structure: Numbered examples from simple to complex (hello_world -> multi_agent) for skill documentation
2. Consistent Benchmark CLI: Standardized flags (--mode, --iter, --no-memory) for reproducible testing
3. Miss Rate Metric: Tracking tool call failures as reliability indicator for agent evaluation
4. Memory Impact Testing: Explicit with/without memory comparisons to inform delegation decisions
5. Streamlit Comparison UI: Interactive framework for evaluating agent configurations

Integration Opportunities

1. Benchmark Adaptation: Adapt benchmark methodology for Claude Code sub-agent performance testing
2. Memory Guidelines: Use findings to inform when to use stateless vs stateful agents in orchestration
3. Tool Design: Apply token efficiency patterns from RAG/API benchmarks to MCP tool design
4. Example Template: Use progressive example structure for skill documentation

Key Insights for Claude Code

References

1. GitHub Repository: https://github.com/martimfasantos/ai-agents-frameworks (accessed 2026-01-31)
2. Study Methodology: study-agents-differences/README.md (accessed 2026-01-26)
3. Author Profile: Martim Santos, daredata.ai - https://martimfasantos.github.io

Framework Documentation (accessed 2026-01-26)

• AG2: https://docs.ag2.ai/latest/
• Agno: https://docs.agno.com/introduction
• Autogen: https://microsoft.github.io/autogen/stable/index.html
• CrewAI: https://docs.crewai.com/
• Google ADK: https://google.github.io/adk-docs/
• LangGraph: https://langchain-ai.github.io/langgraph/
• LlamaIndex: https://docs.llamaindex.ai/en/stable/
• OpenAI Agents SDK: https://openai.github.io/openai-agents-python/
• Pydantic-AI: https://ai.pydantic.dev/
• smolagents: https://huggingface.co/docs/smolagents/en/index

Freshness Tracking

Change Detection Indicators:

• Monitor GitHub releases and commits for new framework additions
• Check for updated benchmark results
• Review issues for methodology changes
• Track star growth indicating community adoption
• Watch for new framework integrations (Claude SDK, etc.)