Automated Code Reviews with Hodor

Automated Code Reviews with Hodor

Hodor is optimized for automated, READ-ONLY code reviews in CI/CD pipelines. This document explains how to set up and use Hodor for continuous code quality monitoring.

Overview

Hodor provides AI-powered code reviews that:

• ✅ Run automatically on every PR/MR (or on-demand)
• ✅ Use efficient search tools (grep, glob, planning_file_editor)
• ✅ Post review comments directly to GitHub/GitLab
• ✅ Focus on bugs, security, and performance (not style)
• ✅ Support repo-specific guidelines via skills system
• ✅ Work with self-hosted GitLab instances

Key Features for CI/CD

1. Enhanced Tooling for Fast Reviews

Hodor uses specialized tools for efficient code analysis:

These tools are much faster than having the LLM script search operations, reducing review time and token usage.

2. READ-ONLY by Design

Hodor is designed for automated reviews without human intervention:

• Workspace is a fresh clone (agent can't push changes)
• No confirmation policy needed (agent only analyzes, doesn't modify)
• Environment is isolated (temporary directory, cleaned up after)
• Safe for untrusted PRs (no risk of malicious code execution)

3. Repository-Specific Guidelines

Use the skills system to encode your team's standards:

# In your repository root
.cursorrules                    # Simple, single-file project guidelines
agents.md                       # Alternative single-file location
.hodor/skills/*.md              # Modular skills organized by topic

See SKILLS.md for detailed documentation.

Setup for GitHub Actions

Quick Start

1. Add Secrets to your GitHub repository:

   • LLM_API_KEY or ANTHROPIC_API_KEY: Your Claude API key
   • GITHUB_TOKEN is automatically provided by GitHub Actions

2. Copy the workflow file:

   cp .github/workflows/pr-review.yml your-repo/.github/workflows/
   

3. Trigger Options:

   • On-demand: Add label hodor-review to a PR
   • Auto-reviewer: Request hodor-agent as a reviewer
   • Always-on: Modify workflow to trigger on all PRs

Example Workflow

# .github/workflows/hodor-review.yml
name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize]  # Run on every PR

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Hodor
        run: |
          pip install uv
          git clone https://github.com/mr-karan/hodor /tmp/hodor
          cd /tmp/hodor && uv sync

      - name: Run Review
        env:
          LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          cd /tmp/hodor
          PR_URL="${{ github.event.pull_request.html_url }}"
          uv run hodor "$PR_URL" --post --verbose

Setup for GitLab CI

Quick Start

1. Set CI/CD Variables in GitLab Settings > CI/CD > Variables:

   • LLM_API_KEY or ANTHROPIC_API_KEY: Your Claude API key
   • GITLAB_TOKEN: GitLab access token with api scope
   • GITLAB_HOST (if self-hosted): e.g., gitlab.company.com

2. Copy the CI configuration:

   cp .gitlab-ci-example.yml your-repo/.gitlab-ci.yml
   

3. Configure triggers (all PRs, specific labels, etc.)

Example CI Configuration

# .gitlab-ci.yml
hodor-review:
  image: python:3.13-slim
  stage: test

  before_script:
    - apt-get update && apt-get install -y git curl
    - curl -LsSf https://astral.sh/uv/install.sh | sh
    - export PATH="$HOME/.local/bin:$PATH"

  script:
    - git clone https://github.com/mr-karan/hodor /tmp/hodor
    - cd /tmp/hodor && uv sync
    - export MR_URL="${CI_MERGE_REQUEST_PROJECT_URL}/-/merge_requests/${CI_MERGE_REQUEST_IID}"
    - uv run hodor "$MR_URL" --post

  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  allow_failure: true

Configuration Options

Model Selection

Choose the right model for your needs:

# Fast and cost-effective (default)
LLM_MODEL: "anthropic/claude-sonnet-4-5-20250929"

# More thorough reviews
LLM_MODEL: "anthropic/claude-opus-4"

# OpenAI alternative
LLM_MODEL: "openai/gpt-4"

# Reasoning models for complex analysis
LLM_MODEL: "openai/o3-mini"
--reasoning-effort high

Verbose Logging

Enable detailed logging for debugging:

uv run hodor $PR_URL --post --verbose

This shows:

• Which tools are configured
• Real-time command execution (grep, git, file reads)
• Token usage statistics
• Detailed error messages

Custom Prompts

Override the default review prompt:

# Inline prompt
uv run hodor $PR_URL --prompt "Focus on security issues only..."

# From file (create your own custom prompt file)
uv run hodor $PR_URL --prompt-file custom-prompt.txt

Cost Optimization

Hodor is designed to be cost-effective:

Token Usage

Costs based on Claude Sonnet 4.5 pricing ($3/M input, $15/M output)

Optimization Tips

1. Use Efficient Tools:

   • grep/glob are much faster than LLM-scripted searches
   • planning_file_editor is optimized for reading code
   • These tools reduce token usage by 30-50%

2. Focus Reviews:

   • Review only changed files (default behavior)
   • Use custom prompts for specific concerns
   • Skip trivial changes (docs, formatting) with CI rules

3. Smart Triggers:

   • Run on critical PRs only (use labels)
   • Skip draft PRs
   • Run once per PR (not on every commit)

Example GitLab CI rule:

rules:
  - if: $CI_MERGE_REQUEST_LABELS !~ /skip-review/
    when: on_success
  - if: $CI_MERGE_REQUEST_DRAFT == "true"
    when: never

Security Considerations

Safe for Untrusted Code

Hodor's design makes it safe for reviewing untrusted PRs:

• Isolated Environment: Each review runs in a fresh, temporary workspace
• No Write Access: Agent can read code but can't push changes
• No Credentials: Agent doesn't have access to push/deploy credentials
• Automatic Cleanup: Workspace is deleted after review

Protecting Secrets

Best practices for CI/CD:

1. Never log secrets: Hodor doesn't log API keys or tokens
2. Use CI/CD variables: Store sensitive data in GitHub Secrets / GitLab CI Variables
3. Limit token scope: GITLAB_TOKEN only needs api scope (not write_repository)
4. Rotate tokens: Periodically rotate API keys

No Confirmation Policy Needed

Unlike interactive development, automated reviews don't need confirmation:

• Agent is READ-ONLY (can't modify repository)
• Agent can't access external resources (network isolation in CI)
• All actions are logged (audit trail in CI logs)

Troubleshooting

Review Fails with "command too long"

Cause: Large environment variables (DIRENV_DIFF, LS_COLORS) Solution: Hodor automatically uses subprocess terminal (fixed in v1.0+)

"No LLM API key found"

Cause: Missing CI/CD variable Solution: Set LLM_API_KEY or ANTHROPIC_API_KEY in CI/CD settings

"Failed to clone repository"

Cause: Missing or invalid GITLAB_TOKEN Solution: Generate token with api scope in GitLab Settings > Access Tokens

"Review finds no issues on obviously buggy code"

Cause: Model may need more context or different prompt Solution: Try:

1. Use --verbose to see what agent is checking
2. Add repo-specific guidelines in .cursorrules
3. Use more capable model (claude-opus-4)
4. Increase --reasoning-effort high

"glab: command not found" (GitLab CI)

Cause: GitLab CLI not installed Solution: Install in before_script:

curl -fsSL https://gitlab.com/gitlab-org/cli/-/releases/permalink/latest/downloads/glab_linux_amd64.deb -o /tmp/glab.deb
dpkg -i /tmp/glab.deb

Examples

Review with Security Focus

# Custom prompt focusing on security
hodor $PR_URL --prompt "Review for security vulnerabilities: SQL injection, XSS, auth bypasses, secrets in code. Be thorough." --post

Review Specific Files

# Review only backend changes
hodor $PR_URL --prompt "Review only files in src/backend/. Focus on database queries and API security." --post

High-Thoroughness Review

# Use reasoning model for complex PRs
hodor $PR_URL --model anthropic/claude-opus-4 --reasoning-effort high --post

Monitoring and Metrics

Track Review Quality

Monitor review effectiveness:

• True Positive Rate: Issues found that are real bugs
• False Positive Rate: Issues reported that aren't actually problems
• Coverage: Percentage of bugs caught before production

Track Costs

Monitor API usage:

# Verbose mode shows token usage
hodor $PR_URL --verbose --post | grep "Total tokens"

# Example output:
# Total tokens used: 23,456
# Cost estimate: $0.14

Improve Over Time

1. Analyze false positives: Update .cursorrules to reduce noise
2. Track missed bugs: Add patterns to skills system
3. Optimize prompts: Focus on high-value checks
4. Adjust triggers: Run reviews where they provide most value

Best Practices

DO:

• ✅ Run automated reviews on all non-trivial PRs
• ✅ Use .cursorrules for project-specific guidelines
• ✅ Post reviews automatically (use --post flag)
• ✅ Enable verbose logging initially (debug issues)
• ✅ Monitor costs and adjust model based on budget
• ✅ Treat reviews as suggestions (not blockers)

DON'T:

• ❌ Don't block PRs on review results (use as advisory)
• ❌ Don't review trivial changes (docs, typos, formatting)
• ❌ Don't use expensive models for small PRs
• ❌ Don't ignore review feedback (defeats the purpose)
• ❌ Don't commit secrets in .cursorrules

Roadmap

Upcoming features for automated reviews:

• Incremental reviews: Review only new commits in updated PRs
• Batch reviews: Review multiple PRs in parallel
• Learning mode: Improve from human feedback on reviews
• Custom MCP tools: Integrate with your issue tracker, docs, etc.
• Trend analysis: Track code quality over time
• Team metrics: Compare review quality across teams

Support

• Documentation: See README.md and SKILLS.md
• Issues: Report bugs at https://github.com/mr-karan/hodor/issues
• Questions: Start a discussion at https://github.com/mr-karan/hodor/discussions

Conclusion

Hodor provides production-ready, automated code reviews that:

• Catch bugs before they reach production
• Reduce human review burden
• Maintain code quality consistently
• Cost less than $0.50 per typical PR
• Integrate seamlessly with GitHub/GitLab CI

Start with on-demand reviews (label-triggered), measure effectiveness, then graduate to always-on automated reviews once you're confident in the results.

Happy reviewing! 🚪