Datastore Management Guide

Datastore Management Guide

Last Updated: January 3, 2026 Applies to: All storage backends (MongoDB CE, AWS DocumentDB)

Table of Contents

1. Overview
2. Local Development - MongoDB CE
3. Production - AWS DocumentDB via ECS
4. Common Operations
5. Troubleshooting

Overview

The MCP Gateway Registry uses MongoDB-compatible datastores for storage:

• Local Development: MongoDB Community Edition 8.2 in Docker
• Production: AWS DocumentDB (MongoDB-compatible managed service)

This guide explains how to access and manage datastores in both environments.

Local Development - MongoDB CE

Prerequisites

• MongoDB container running: docker compose ps mongodb shows "healthy"
• No authentication required (configured for local dev simplicity)

Accessing the Datastore (mongosh)

Method 1: Direct Docker Exec (Recommended)

# Connect to MongoDB shell
docker exec -it mcp-mongodb mongosh

# You should see:
# Current Mongosh Log ID: ...
# Connecting to: mongodb://127.0.0.1:27017/?directConnection=true
# ...
# rs0 [direct: primary] test>

Method 2: Connect from Host Machine

If you have mongosh installed locally:

mongosh mongodb://localhost:27017/mcp_registry

Basic Datastore Operations

Once connected to mongosh:

// Switch to the registry database
use mcp_registry

// List all collections
show collections
// Expected output:
//   mcp_agents_default
//   mcp_embeddings_1536_default
//   mcp_federation_config_default
//   mcp_scopes_default
//   mcp_security_scans_default
//   mcp_servers_default

// Check replica set status
rs.status()

// View database statistics
db.stats()

Viewing Collection Contents

List All Servers

// Count total servers
db.mcp_servers_default.countDocuments()

// View all servers (formatted)
db.mcp_servers_default.find().pretty()

// View specific server by path
db.mcp_servers_default.findOne({ path: "/servers/financial-data" })

// List only server names and paths
db.mcp_servers_default.find(
  {},
  { "manifest.serverInfo.name": 1, path: 1, _id: 0 }
)

List All Agents

// Count total agents
db.mcp_agents_default.countDocuments()

// View all agents
db.mcp_agents_default.find().pretty()

// Find agents by tag
db.mcp_agents_default.find({ tags: "finance" }).pretty()

View Vector Embeddings

// Count embeddings
db.mcp_embeddings_1536_default.countDocuments()

// View embedding metadata (without the large vector array)
db.mcp_embeddings_1536_default.find(
  {},
  {
    path: 1,
    entity_type: 1,
    name: 1,
    embedding_metadata: 1,
    indexed_at: 1,
    _id: 0
  }
).pretty()

// Check specific embedding
db.mcp_embeddings_1536_default.findOne({ path: "/servers/financial-data" })

View Scopes

// List all scopes
db.mcp_scopes_default.find().pretty()

// Find server scopes
db.mcp_scopes_default.find({ scope_type: "server_scope" }).pretty()

// Find group mappings
db.mcp_scopes_default.find({ scope_type: "group_mapping" }).pretty()

View Security Scans

// Count security scans
db.mcp_security_scans_default.countDocuments()

// View latest scans
db.mcp_security_scans_default.find().sort({ scan_timestamp: -1 }).limit(5).pretty()

// Find scans for specific server
db.mcp_security_scans_default.find({ server_path: "/servers/financial-data" }).pretty()

Collection Indexes

// View indexes on servers collection
db.mcp_servers_default.getIndexes()

// View indexes on embeddings collection
db.mcp_embeddings_1536_default.getIndexes()

// Check index usage stats
db.mcp_servers_default.aggregate([{ $indexStats: {} }])

Query Performance Analysis

// Explain query execution plan
db.mcp_servers_default.find({ path: "/servers/financial-data" }).explain("executionStats")

// Find slow operations (if profiling enabled)
db.system.profile.find({ millis: { $gt: 100 } }).sort({ ts: -1 }).limit(5).pretty()

Exiting mongosh

// Exit the shell
exit

Or press Ctrl+D

Production - AWS DocumentDB via ECS

Prerequisites

• AWS ECS cluster running with DocumentDB
• ECS exec permissions configured
• manage-documentdb.py script available in registry container

Accessing DocumentDB via ECS Exec

Step 1: SSH into Registry Container

# From your local machine
# Use the ecs-ssh helper script
cd terraform/aws-ecs
./scripts/ecs-ssh.sh registry

# Or manually with AWS CLI
aws ecs execute-command \
  --cluster mcp-gateway-ecs-cluster \
  --task <task-id> \
  --container registry \
  --interactive \
  --command "/bin/bash"

Step 2: Activate Python Virtual Environment

# Inside the ECS container
source .venv/bin/activate

# Verify Python environment
which python
# Should show: /app/.venv/bin/python

Step 3: Run DocumentDB Management Script

The manage-documentdb.py script provides commands for managing collections and querying data.

List All Collections

python scripts/manage-documentdb.py list

Inspect a Collection

# Show collection schema and indexes
python scripts/manage-documentdb.py inspect --collection mcp_servers_default

Count Documents

# Count all documents in collection
python scripts/manage-documentdb.py count --collection mcp_servers_default

Search Documents

# List documents with optional limit
python scripts/manage-documentdb.py search --collection mcp_servers_default --limit 5

# Search specific collection
python scripts/manage-documentdb.py search --collection mcp_agents_default --limit 10

View Sample Document

# Show one sample document from collection
python scripts/manage-documentdb.py sample --collection mcp_servers_default

Query with Filter

# Query with MongoDB filter syntax
python scripts/manage-documentdb.py query \
  --collection mcp_servers_default \
  --filter '{"path": "/servers/financial-data"}'

# Query enabled servers
python scripts/manage-documentdb.py query \
  --collection mcp_servers_default \
  --filter '{"enabled": true}'

# Query by tags
python scripts/manage-documentdb.py query \
  --collection mcp_servers_default \
  --filter '{"tags": "finance"}'

View Embeddings

# Sample embedding document (shows structure without large vector array)
python scripts/manage-documentdb.py sample --collection mcp_embeddings_1536_default

# Count total embeddings
python scripts/manage-documentdb.py count --collection mcp_embeddings_1536_default

Note: The script automatically reads connection parameters from environment variables in the ECS container (DOCUMENTDB_HOST, DOCUMENTDB_USERNAME, DOCUMENTDB_PASSWORD, etc.).

Common Operations

Checking Datastore Health

Local (MongoDB CE)

// In mongosh
db.serverStatus()
db.stats()
rs.status()

Production (DocumentDB)

# Use count command to verify connection and check collections
python scripts/manage-documentdb.py list

# Check specific collection
python scripts/manage-documentdb.py count --collection mcp_servers_default

Searching for Specific Documents

Local (MongoDB CE)

// Search servers by tag
db.mcp_servers_default.find({ tags: "finance" })

// Search by partial name match
db.mcp_servers_default.find({
  "manifest.serverInfo.name": /financial/i
})

// Complex query with multiple conditions
db.mcp_servers_default.find({
  enabled: true,
  tags: { $in: ["finance", "data"] }
})

Production (DocumentDB)

# Search by tags
python scripts/manage-documentdb.py query \
  --collection mcp_servers_default \
  --filter '{"tags": "finance"}'

# Query enabled servers
python scripts/manage-documentdb.py query \
  --collection mcp_servers_default \
  --filter '{"enabled": true}'

Viewing Recent Activity

Local (MongoDB CE)

// Recent server registrations
db.mcp_servers_default.find().sort({ registered_at: -1 }).limit(5)

// Recent embeddings
db.mcp_embeddings_1536_default.find().sort({ indexed_at: -1 }).limit(5)

// Recent security scans
db.mcp_security_scans_default.find().sort({ scan_timestamp: -1 }).limit(5)

Production (DocumentDB)

# View recent servers (sorted by registration)
python scripts/manage-documentdb.py search \
  --collection mcp_servers_default \
  --limit 5

# View recent embeddings
python scripts/manage-documentdb.py search \
  --collection mcp_embeddings_1536_default \
  --limit 5

Backup and Export

Local (MongoDB CE)

Option 1: Binary Backup (mongodump) - Recommended for Full Backups

# Export entire database (BSON format - preserves data types)
docker exec mcp-mongodb mongodump \
  --db=mcp_registry \
  --out=/tmp/mongodb-backup

# Copy backup from container to host
docker cp mcp-mongodb:/tmp/mongodb-backup ./mongodb-backup-$(date +%Y%m%d)

# Restore from backup (if needed)
docker cp ./mongodb-backup-20260103 mcp-mongodb:/tmp/restore-backup
docker exec mcp-mongodb mongorestore \
  --db=mcp_registry \
  /tmp/restore-backup/mcp_registry

Option 2: JSON Export (mongoexport) - Human-Readable, Portable

# Export specific collection to JSON (one document per line)
docker exec mcp-mongodb mongoexport \
  --db=mcp_registry \
  --collection=mcp_servers_default \
  --out=/tmp/servers.json

# Copy to host
docker cp mcp-mongodb:/tmp/servers.json ./servers-backup-$(date +%Y%m%d).json

# Pretty-print JSON (optional, for readability)
docker exec mcp-mongodb mongoexport \
  --db=mcp_registry \
  --collection=mcp_servers_default \
  --jsonArray \
  --pretty \
  --out=/tmp/servers-pretty.json

# Import from JSON (if needed)
docker cp ./servers-backup-20260103.json mcp-mongodb:/tmp/import-servers.json
docker exec mcp-mongodb mongoimport \
  --db=mcp_registry \
  --collection=mcp_servers_default \
  --file=/tmp/import-servers.json

Export All Collections

# Export all collections to JSON
COLLECTIONS="mcp_servers_default mcp_agents_default mcp_scopes_default mcp_embeddings_1536_default mcp_security_scans_default mcp_federation_config_default"

for collection in $COLLECTIONS; do
  echo "Exporting $collection..."
  docker exec mcp-mongodb mongoexport \
    --db=mcp_registry \
    --collection=$collection \
    --out=/tmp/${collection}.json
  docker cp mcp-mongodb:/tmp/${collection}.json ./${collection}-$(date +%Y%m%d).json
done

Production (DocumentDB)

Option 1: AWS Automated Backups (Recommended)

AWS DocumentDB provides automated continuous backups with point-in-time recovery:

# Create manual snapshot (from local machine with AWS CLI)
aws docdb create-db-cluster-snapshot \
  --db-cluster-snapshot-identifier mcp-registry-manual-$(date +%Y%m%d) \
  --db-cluster-identifier mcp-registry-prod

# List available snapshots
aws docdb describe-db-cluster-snapshots \
  --db-cluster-identifier mcp-registry-prod

# Restore from snapshot (creates new cluster)
aws docdb restore-db-cluster-from-snapshot \
  --db-cluster-identifier mcp-registry-restored \
  --snapshot-identifier mcp-registry-manual-20260103 \
  --engine docdb

Option 2: Binary Backup with mongodump (from ECS Container)

# SSH into ECS container
cd terraform/aws-ecs
./scripts/ecs-ssh.sh registry
source .venv/bin/activate

# Export entire database to BSON
mongodump \
  --host=$DOCUMENTDB_HOST \
  --port=27017 \
  --username=$DOCUMENTDB_USERNAME \
  --password=$DOCUMENTDB_PASSWORD \
  --ssl \
  --sslCAFile=/app/global-bundle.pem \
  --db=mcp_registry \
  --out=/tmp/documentdb-backup

# Upload to S3
BACKUP_DATE=$(date +%Y%m%d-%H%M%S)
aws s3 cp /tmp/documentdb-backup \
  s3://mcp-gateway-backups/documentdb-backup-${BACKUP_DATE}/ \
  --recursive

# Cleanup temporary files
rm -rf /tmp/documentdb-backup

echo "Backup uploaded to: s3://mcp-gateway-backups/documentdb-backup-${BACKUP_DATE}/"

Option 3: JSON Export of Specific Collections (from ECS Container)

# SSH into ECS container
cd terraform/aws-ecs
./scripts/ecs-ssh.sh registry
source .venv/bin/activate

# Export specific collection to JSON
mongoexport \
  --host=$DOCUMENTDB_HOST \
  --port=27017 \
  --username=$DOCUMENTDB_USERNAME \
  --password=$DOCUMENTDB_PASSWORD \
  --ssl \
  --sslCAFile=/app/global-bundle.pem \
  --db=mcp_registry \
  --collection=mcp_servers_default \
  --out=/tmp/servers-export.json

# Upload to S3
aws s3 cp /tmp/servers-export.json \
  s3://mcp-gateway-backups/exports/servers-$(date +%Y%m%d).json

# Cleanup
rm /tmp/servers-export.json

Restore from S3 Backup

# SSH into ECS container
cd terraform/aws-ecs
./scripts/ecs-ssh.sh registry
source .venv/bin/activate

# Download backup from S3
aws s3 cp s3://mcp-gateway-backups/documentdb-backup-20260103-120000/ \
  /tmp/restore-backup/ \
  --recursive

# Restore using mongorestore
mongorestore \
  --host=$DOCUMENTDB_HOST \
  --port=27017 \
  --username=$DOCUMENTDB_USERNAME \
  --password=$DOCUMENTDB_PASSWORD \
  --ssl \
  --sslCAFile=/app/global-bundle.pem \
  --db=mcp_registry \
  /tmp/restore-backup/mcp_registry

# Cleanup
rm -rf /tmp/restore-backup

Important Notes:

• mongodump/mongorestore: Binary format (BSON), preserves all data types including binary data and dates
• mongoexport/mongoimport: JSON format, human-readable but may lose type information
• For production: Use AWS automated backups for disaster recovery, manual exports for data migration
• S3 bucket: Replace mcp-gateway-backups with your actual S3 bucket name
• Embeddings: Vector embeddings are large; consider excluding from exports if not needed:

  mongodump --excludeCollection=mcp_embeddings_1536_default ...
  

Troubleshooting

Cannot Connect to MongoDB (Local)

Problem: docker exec -it mcp-mongodb mongosh fails

Solutions:

# Check if container is running
docker compose ps mongodb

# Check container logs
docker compose logs mongodb

# Restart MongoDB
docker compose restart mongodb

# If needed, recreate container
docker compose up -d mongodb

Cannot Connect to DocumentDB (Production)

Problem: manage-documentdb.py commands fail with connection errors

Solutions:

# 1. Verify you're in the ECS container with activated venv
source .venv/bin/activate

# 2. Check environment variables are set
env | grep DOCUMENTDB

# 3. Test connection with simple list command
python scripts/manage-documentdb.py list

# 4. Check security group allows access from ECS tasks (from local machine)
aws ec2 describe-security-groups --group-ids <docdb-sg-id>

# 5. Verify DocumentDB endpoint (from local machine)
aws docdb describe-db-clusters --db-cluster-identifier mcp-registry-prod

Replica Set Not Initialized (Local)

Problem: rs.status() shows "not initialized"

Solutions:

# Re-run initialization
docker compose up mongodb-init

# Or manually initialize
docker exec -it mcp-mongodb mongosh --eval 'rs.initiate({_id: "rs0", members: [{_id: 0, host: "mongodb:27017"}]})'

Collections Not Found

Problem: show collections returns empty

Solutions:

# Verify you're in correct database
# In mongosh:
db.getName()  // Should show "mcp_registry"

# Re-run initialization
docker compose up mongodb-init

# Check if data is in different namespace
db.getCollectionNames()

Slow Queries

Problem: Queries taking too long

Solutions (Local MongoDB CE):

// In mongosh - Check if indexes exist
db.mcp_servers_default.getIndexes()

// Analyze query plan
db.mcp_servers_default.find({ path: "..." }).explain("executionStats")

// Check embeddings indexes
db.mcp_embeddings_1536_default.getIndexes()

Solutions (Production DocumentDB):

# Use inspect command to check indexes
python scripts/manage-documentdb.py inspect --collection mcp_servers_default

# Check embeddings collection indexes
python scripts/manage-documentdb.py inspect --collection mcp_embeddings_1536_default

Quick Reference

Connection Strings

Local MongoDB CE:

mongodb://localhost:27017/mcp_registry

Production DocumentDB:

mongodb://<username>:<password>@<cluster-endpoint>:27017/mcp_registry?tls=true&tlsCAFile=global-bundle.pem&replicaSet=rs0

Common mongosh Commands

Environment Variables Reference

Local (.env):

STORAGE_BACKEND=mongodb-ce
DOCUMENTDB_HOST=mongodb
DOCUMENTDB_PORT=27017
DOCUMENTDB_DATABASE=mcp_registry
DOCUMENTDB_NAMESPACE=default
DOCUMENTDB_USE_TLS=false

Production (ECS Task Definition):

STORAGE_BACKEND=documentdb
DOCUMENTDB_HOST=<cluster-endpoint>
DOCUMENTDB_PORT=27017
DOCUMENTDB_DATABASE=mcp_registry
DOCUMENTDB_NAMESPACE=production
DOCUMENTDB_USERNAME=<from-secrets>
DOCUMENTDB_PASSWORD=<from-secrets>
DOCUMENTDB_USE_TLS=true
DOCUMENTDB_TLS_CA_FILE=/app/global-bundle.pem
DOCUMENTDB_REPLICA_SET=rs0

See Also

• Storage Architecture: MongoDB CE & AWS DocumentDB
• Datastore Schema Design
• Configuration Guide
• MongoDB Documentation
• AWS DocumentDB Documentation