Garage

Garage

Operate Joel's canonical S3-compatible object storage for local self-hosted Convex. This is not generic Garage advice. The canonical path is the Docker/Compose Garage service running on the ASUSTOR NAS three-body.

Custom MinIO is now a rollback/reference path. ASUSTOR MinIO CE is only a smoke-test/reference instance.

When To Use

Use this skill when Joel mentions:

• Garage
• Convex Garage
• Garage S3
• S3-compatible object storage for local Convex
• three-body Garage
• port 39100
• Garage buckets, keys, env wiring, smoke tests, or MinIO-to-Garage migration

If the task is about the full local Convex runtime, also read the local-convex skill. If the task is about general ASUSTOR storage, NFS, SMB, RAID, or NAS health, also read the three-body skill. If the task is explicitly about MinIO rollback or CE cleanup, also read the minio skill.

Current Topology

Host:

• NAS label: three-body
• Tailnet IP: 100.67.156.41
• Prefer the IP in machine config and smoke tests, not the tailnet alias.
• NAS model: ASUSTOR AS6508T
• ADM: 5.1.3.RI81

Canonical Garage:

• S3 API: http://100.67.156.41:39100
• Data: /volume1/joelclaw/s3/garage/data
• Metadata: /volume1/joelclaw/s3/garage/meta
• Metadata snapshots: /volume1/joelclaw/s3/garage/snapshots
• Compose file: /volume1/joelclaw/s3/garage-compose/compose.yaml
• Garage config: /volume1/joelclaw/s3/garage-compose/garage.toml
• Convex S3 env: /volume1/joelclaw/s3/garage-compose/convex-s3.env
• Redacted env: /volume1/joelclaw/s3/garage-compose/convex-s3.redacted.env
• Installer on NAS: /volume1/joelclaw/s3/install-three-body-convex-garage.sh
• Local installer copy: /Users/joel/Documents/Codex/2026-06-17/we-re-setting-up-durable-self/outputs/install-three-body-convex-garage.sh
• Image: dxflrs/garage@sha256:dac0c92add4f1a0b41035e94b41036a270ffbe88a37c7ac9c3f19e6dc5bdccf2

Do not expose Garage publicly. Keep S3 API access LAN/tailnet only.

Canonical Sources

Read these before mutating the setup:

• Project note: /Users/joel/.brain/projects/convex-self-hosting-local.svx
• Local installer: /Users/joel/Documents/Codex/2026-06-17/we-re-setting-up-durable-self/outputs/install-three-body-convex-garage.sh
• NAS compose file: /volume1/joelclaw/s3/garage-compose/compose.yaml
• NAS redacted env: /volume1/joelclaw/s3/garage-compose/convex-s3.redacted.env

If sources disagree, prefer the brain project note, then verify live state with redaction-safe health checks. Do not print raw env files to resolve disagreement.

Safety And Secret Handling

Never print, paste, summarize, commit, or log:

• Garage RPC secret
• Garage admin token
• Garage metrics token
• Convex Garage access key
• Convex Garage secret key
• raw convex-s3.env
• raw .env
• full environment dumps
• shell history containing secrets

It is okay to read and report the redacted env file. It is okay to say whether required keys exist. It is not okay to reveal their values.

Before sharing command output, scan it for:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• GARAGE_RPC_SECRET
• GARAGE_ADMIN_TOKEN
• GARAGE_METRICS_TOKEN
• CONVEX_ACCESS_KEY_ID
• CONVEX_SECRET_ACCESS_KEY
• convex-s3.env

Redact first, then report.

NAS Access Rules

Use one SSH session to three-body at a time. NAS SSH is sensitive to bursts and can reset handshakes after rapid parallel probes.

Default read-only SSH shape:

ssh -o BatchMode=yes -o ConnectTimeout=8 joel@three-body 'uptime; pwd'

Operational rules:

• Back off after kex_exchange_identification, Not allowed at this time, or repeated SSH resets.
• Do not run parallel SSH loops against the NAS.
• Docker operations require sudo or an ADM/admin path.
• Treat ADM as an appliance environment, not a normal Debian server.
• Inspect first. Summarize risk before changing service state, compose files, buckets, or data paths.

Health Checks

Prefer network checks that do not require credentials:

nc -vz -w 5 100.67.156.41 39100
curl -sS -m 5 -o /tmp/garage-root.out -w '%{http_code}\n' http://100.67.156.41:39100/
sed -n '1,3p' /tmp/garage-root.out

Garage does not support anonymous S3 access; 403 AccessDenied from / proves the API is reachable.

Redaction-safe env check:

ssh joel@three-body 'awk -F= "/^(AWS_REGION|S3_ENDPOINT_URL|AWS_S3_FORCE_PATH_STYLE|S3_STORAGE_)/ {print}" /volume1/joelclaw/s3/garage-compose/convex-s3.env'

Presence-only secret check:

ssh joel@three-body 'awk -F= "/^(AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY)=/ {print $1\"=<present>\"}" /volume1/joelclaw/s3/garage-compose/convex-s3.env'

Service checks usually need sudo:

ssh joel@three-body 'sudo docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" | grep -Ei "garage|39100" || true'
ssh joel@three-body 'cd /volume1/joelclaw/s3/garage-compose && sudo docker compose ps'

Do not run logs without a reason. If logs are needed, bound them and redact:

ssh joel@three-body 'cd /volume1/joelclaw/s3/garage-compose && sudo docker compose logs --tail=120 garage'

Convex Buckets

Garage should have these Convex buckets:

• joelclaw-local-usw2-convex-exports
• joelclaw-local-usw2-convex-snapshot-imports
• joelclaw-local-usw2-convex-modules
• joelclaw-local-usw2-convex-files
• joelclaw-local-usw2-convex-search

Bucket rules:

• Perform bucket operations against http://100.67.156.41:39100.
• Use Convex-specific credentials for Convex bucket checks.
• Do not create extra buckets unless the requesting workload names them.
• Do not delete buckets or objects without explicit authorization and a backup/recovery note.
• Do not infer durability from bucket existence alone. Prove upload, read, and delete with the workload credentials.

Convex Env Shape

Convex should point at the Garage IP endpoint:

AWS_REGION=usw2
S3_ENDPOINT_URL=http://100.67.156.41:39100
AWS_S3_FORCE_PATH_STYLE=true
AWS_ACCESS_KEY_ID=<from /volume1/joelclaw/s3/garage-compose/convex-s3.env>
AWS_SECRET_ACCESS_KEY=<from /volume1/joelclaw/s3/garage-compose/convex-s3.env>
S3_STORAGE_EXPORTS_BUCKET=joelclaw-local-usw2-convex-exports
S3_STORAGE_SNAPSHOT_IMPORTS_BUCKET=joelclaw-local-usw2-convex-snapshot-imports
S3_STORAGE_MODULES_BUCKET=joelclaw-local-usw2-convex-modules
S3_STORAGE_FILES_BUCKET=joelclaw-local-usw2-convex-files
S3_STORAGE_SEARCH_BUCKET=joelclaw-local-usw2-convex-search

Validation rules:

• S3_ENDPOINT_URL must be http://100.67.156.41:39100.
• AWS_S3_FORCE_PATH_STYLE must be true.
• Bucket names must match the custom Convex buckets above.
• Do not use the tailnet alias in durable machine config.
• Do not point Convex at MinIO or ASUSTOR MinIO CE unless the task is an explicit rollback/comparison.

Operating Sequence

When wiring or debugging Garage for local Convex:

1. Read the brain project note and this skill.
2. Verify Garage reachability on 39100.
3. Validate the redacted Convex env shape.
4. Confirm local Convex backend runtime env has S3_ENDPOINT_URL=http://100.67.156.41:39100.
5. Run the Convex-level smoke:

cd /Users/joel/Documents/Codex/2026-06-17/we-re-setting-up-durable-self/work/local-convex
./smoke-self-hosted-convex.sh

6. Verify the smoke export ZIP contains custom-garage and http://100.67.156.41:39100.
7. Capture durable discoveries back to the brain project note if facts changed.

Gotchas

• Garage is not MinIO with a different name; it uses its own key/bucket permission model instead of S3 bucket policies.
• Garage does not support anonymous S3 access; 403 AccessDenied from / is expected.
• The Garage S3 API is 39100; custom MinIO is 39000; CE is 29990.
• A green MinIO or CE check does not prove Garage.
• Docker socket access on ADM may require sudo.
• NAS SSH can react badly to rapid probes. Slow down before assuming the service is broken.
• Do not move Convex object storage to a local SSD fallback just to get a smoke test green. The durable target is NAS-backed Garage.