Autonomous Agent Harness

Autonomous Agent Harness

Turn Gemini Code into a persistent, self-directing agent system using only native features and MCP servers.

Consent and Safety Boundaries

Autonomous operation must be explicitly requested and scoped by the user. Do not create schedules, dispatch remote agents, write persistent memory, use computer control, post externally, modify third-party resources, or act on private communications unless the user has approved that capability and the target workspace for the current setup.

Prefer dry-run plans and local queue files before enabling recurring or event-driven actions. Keep credentials, private workspace exports, personal datasets, and account-specific automations out of reusable EGC artifacts.

When to Activate

• User wants an agent that runs continuously or on a schedule
• Setting up automated workflows that trigger periodically
• Building a personal AI assistant that remembers context across sessions
• User says "run this every day", "check on this regularly", "keep monitoring"
• Wants to replicate functionality from Hermes, AutoGPT, or similar autonomous agent frameworks
• Needs computer use combined with scheduled execution

Architecture

┌──────────────────────────────────────────────────────────────┐
│                    Gemini Code Runtime                        │
│                                                              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌─────────────┐ │
│  │  Crons   │  │ Dispatch │  │ Memory   │  │ Computer    │ │
│  │ Schedule │  │ Remote   │  │ Store    │  │ Use         │ │
│  │ Tasks    │  │ Agents   │  │          │  │             │ │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └──────┬──────┘ │
│       │              │             │                │        │
│       ▼              ▼             ▼                ▼        │
│  ┌──────────────────────────────────────────────────────┐    │
│  │              EGC Skill + Agent Layer                  │    │
│  │                                                      │    │
│  │  skills/     agents/     commands/     hooks/        │    │
│  └──────────────────────────────────────────────────────┘    │
│       │              │             │                │        │
│       ▼              ▼             ▼                ▼        │
│  ┌──────────────────────────────────────────────────────┐    │
│  │              MCP Server Layer                        │    │
│  │                                                      │    │
│  │  memory    github    exa    supabase    browser-use  │    │
│  └──────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘

Core Components

1. Persistent Memory

Use Gemini Code's built-in memory system enhanced with MCP memory server for structured data.

Built-in memory (~/.gemini/projects/*/memory/):

• User preferences, feedback, project context
• Stored as markdown files with frontmatter
• Automatically loaded at session start

MCP memory server (structured knowledge graph):

• Entities, relations, observations
• Queryable graph structure
• Cross-session persistence

Memory patterns:

# Short-term: current session context
Use TodoWrite for in-session task tracking

# Medium-term: project memory files
Write to ~/.gemini/projects/*/memory/ for cross-session recall

# Long-term: MCP knowledge graph
Use mcp__memory__create_entities for permanent structured data
Use mcp__memory__create_relations for relationship mapping
Use mcp__memory__add_observations for new facts about known entities

2. Scheduled Operations (Crons)

Use Gemini Code's scheduled tasks to create recurring agent operations.

Setting up a cron:

# Via MCP tool
mcp__scheduled-tasks__create_scheduled_task({
  name: "daily-pr-review",
  schedule: "0 9 * * 1-5",  # 9 AM weekdays
  prompt: "Review all open PRs in Fmarzochi/EGC. For each: check CI status, review changes, flag issues. Post summary to memory.",
  project_dir: "/path/to/repo"
})

# Via egc -p (programmatic mode)
echo "Review open PRs and summarize" | egc -p --project /path/to/repo

Useful cron patterns:

3. Dispatch / Remote Agents

Trigger Gemini Code agents remotely for event-driven workflows.

Dispatch patterns:

# Trigger from CI/CD
curl -X POST "https://api.anthropic.com/dispatch" \
  -H "Authorization: Bearer $ANTHROPIC_API_KEY" \
  -d '{"prompt": "Build failed on main. Diagnose and fix.", "project": "/repo"}'

# Trigger from webhook
# GitHub webhook → dispatch → Gemini agent → fix → PR

# Trigger from another agent
egc -p "Analyze the output of the security scan and create issues for findings"

4. Computer Use

Leverage Gemini's computer-use MCP for physical world interaction.

Capabilities:

• Browser automation (navigate, click, fill forms, screenshot)
• Desktop control (open apps, type, mouse control)
• File system operations beyond CLI

Use cases within the harness:

• Automated testing of web UIs
• Form filling and data entry
• Screenshot-based monitoring
• Multi-app workflows

5. Task Queue

Manage a persistent queue of tasks that survive session boundaries.

Implementation:

# Task persistence via memory
Write task queue to ~/.gemini/projects/*/memory/task-queue.md

# Task format
---
name: task-queue
type: project
description: Persistent task queue for autonomous operation
---

## Active Tasks
- [ ] PR #123: Review and approve if CI green
- [ ] Monitor deploy: check /health every 30 min for 2 hours
- [ ] Research: Find 5 leads in AI tooling space

## Completed
- [x] Daily standup: reviewed 3 PRs, 2 issues

Replacing Hermes

Setup Guide

Step 1: Configure MCP Servers

Ensure these are in ~/.gemini.json:

{
  "mcpServers": {
    "memory": {
      "command": "npx",
      "args": ["-y", "@anthropic/memory-mcp-server"]
    },
    "scheduled-tasks": {
      "command": "npx",
      "args": ["-y", "@anthropic/scheduled-tasks-mcp-server"]
    },
    "computer-use": {
      "command": "npx",
      "args": ["-y", "@anthropic/computer-use-mcp-server"]
    }
  }
}

Step 2: Create Base Crons

# Daily morning briefing
egc -p "Create a scheduled task: every weekday at 9am, review my GitHub notifications, open PRs, and calendar. Write a morning briefing to memory."

# Continuous learning
egc -p "Create a scheduled task: every Sunday at 8pm, extract patterns from this week's sessions and update the learned skills."

Step 3: Initialize Memory Graph

# Bootstrap your identity and context
egc -p "Create memory entities for: me (user profile), my projects, my key contacts. Add observations about current priorities."

Step 4: Enable Computer Use (Optional)

Grant computer-use MCP the necessary permissions for browser and desktop control.

Example Workflows

Autonomous PR Reviewer

Cron: every 30 min during work hours
1. Check for new PRs on watched repos
2. For each new PR:
   - Pull branch locally
   - Run tests
   - Review changes with code-reviewer agent
   - Post review comments via GitHub MCP
3. Update memory with review status

Personal Research Agent

Cron: daily at 6 AM
1. Check saved search queries in memory
2. Run Exa searches for each query
3. Summarize new findings
4. Compare against yesterday's results
5. Write digest to memory
6. Flag high-priority items for morning review

Meeting Prep Agent

Trigger: 30 min before each calendar event
1. Read calendar event details
2. Search memory for context on attendees
3. Pull recent email/Slack threads with attendees
4. Prepare talking points and agenda suggestions
5. Write prep doc to memory

Constraints

• Cron tasks run in isolated sessions: they don't share context with interactive sessions unless through memory.
• Computer use requires explicit permission grants. Don't assume access.
• Remote dispatch may have rate limits. Design crons with appropriate intervals.
• Memory files should be kept concise. Archive old data rather than letting files grow unbounded.
• Always verify that scheduled tasks completed successfully. Add error handling to cron prompts.

Source: Fmarzochi/everything-gemini — distributed by TomeVault.