Free Sota Moa

free-sota-moa

This skill installs and configures the FreeLLMAPI + Hermes MoA stack to produce the best currently available LLM output from free-tier pooled quota. It is designed to be repeatable at any future date: it queries the live model catalog rather than hardcoding model names, so the selected models are always the best currently available.

Scope caveat: This stack operates over pooled free-tier provider quotas. Output quality is bounded by what is available and non-exhausted in your catalog at any given moment. "SOTA-seeking" means: best available diverse panel from your current free-tier pool, not guaranteed frontier-model performance. Provider limits, ToS, and model availability all apply.

When to Use

• User says "set up free MoA", "configure MoA", "use the best free models", "set up FreeLLMAPI", or equivalent intent
• User wants to refresh their MoA preset after a model catalog update
• User needs to replicate this setup on a new machine

Architecture Overview

[User / Hermes session]
        │
        ▼
  Hermes Agent  ←──── MoA preset "free-sota"
        │                    │
        │              ┌─────┴──────┐
        │         Reference   Reference
        │         model A     model B
        │         (no tools, (no tools,
        │          advisory)  advisory)
        │              └─────┬──────┘
        │                    ▼
        └──────────── Aggregator model
                       (full tool schema,
                        writes response,
                        emits tool calls)
                             │
                    FreeLLMAPI /v1 router
                     (localhost:3001)
                             │
          ┌──────┬───────────┼──────────┬──────┐
       Groq  OpenRouter  Gemini   DeepSeek  ...
       (free)  (free)    (free)    (free)

Key principle: Reference models provide heterogeneous analysis — no tool schemas, cheap advisory calls. The aggregator synthesizes and acts — it has the full Hermes tool schema and writes the real assistant response. Diversity of model family in references (not raw count or rank) drives quality lift. This matches Hermes MoA's documented agent loop exactly.

Quick Reference

Premortem

Review failure modes before executing any phase.

Failure: FreeLLMAPI port exposed to the internet

• Indicator: lsof -i :3001 shows bind on 0.0.0.0 AND the port is reachable outside your local network or Tailscale.
• Rollback: stop FreeLLMAPI, rebind to 127.0.0.1, rotate FreeLLMAPI unified key from the dashboard, rotate any upstream provider keys that were accessible.

Failure: FreeLLMAPI unified key or upstream provider keys committed to Git or logged

• Indicator: grep -rn "sk-" ~/.hermes/ . finds a key in tracked files, or terminal scrollback/shell history exposes it.
• Rollback: rotate the key immediately from the dashboard, use git filter-repo if committed.

Failure: intelligence_rank field absent from /v1/models response

• Indicator: Phase 2 discovery script prints ? in the Rank column or all ranks are 0.
• Rollback: skip automated ranking; use FreeLLMAPI dashboard → Models tab (sorted by rank) and manually copy the top model IDs instead.

Failure: MoA references all from same model family

• Indicator: Top 3 ranked models are all from the same provider (e.g., three DeepSeek variants).
• Rollback: override rule — pick the best model from each of the three highest-ranked distinct provider families, even if that means accepting lower individual model ranks.

Failure: config.yaml env interpolation not supported by installed Hermes version

• Indicator: hermes doctor fails with "unknown api_key" or literal ${…} appears in error output.
• Rollback: remove interpolation syntax; use the hermes model wizard to register the provider (it stores the key correctly) or paste the literal key value.

Procedure

Phase 1 — Install and Run FreeLLMAPI

Perform this phase once per machine. Skip if FreeLLMAPI is already running and reachable at http://localhost:3001/v1.

1.1 Check prerequisites

node --version   # must be v20.x or v22.x LTS
npm --version    # must be v10+
python3 --version  # must be 3.10+

If Node.js is not installed: https://nodejs.org/en/download or nvm install --lts.

1.2 Clone and configure

git clone https://github.com/tashfeenahmed/freellmapi.git
cd freellmapi
cp .env.example .env

Generate a 32-byte encryption key and insert it — no clipboard exposure:

ENCRYPTION_KEY=$(node -e "console.log(require('crypto').randomBytes(32).toString('hex'))")
sed -i.bak "s/^ENCRYPTION_KEY=.*/ENCRYPTION_KEY=$ENCRYPTION_KEY/" .env
unset ENCRYPTION_KEY   # remove from shell environment
echo "Encryption key written to .env (shell env cleared)"

Security: Store the .env value in a secrets manager (1Password, Bitwarden, macOS Keychain) immediately. A lost key makes stored provider API keys undecryptable. In production, set ENCRYPTION_KEY as a system environment variable (e.g., via systemd EnvironmentFile=) rather than reading from .env.

1.3 Install and start

npm install
npm run dev

Expected: server on port 3001, dashboard (Vite) on port 5173. Catalog sync runs on first boot — wait ~30–60 seconds for models to appear.

1.4 Add provider API keys

Open http://localhost:5173 → Keys tab. Add keys from at least three different provider families to support reference diversity:

All listed providers have free tiers with no payment method required, subject to their own ToS and daily/monthly quota limits.

1.5 Confirm catalog sync

Dashboard → Models tab should populate within 60 seconds. If empty after 2 minutes: check terminal for sync errors and confirm at least one key shows a green checkmark on the Keys tab.

1.6 Record your unified API key (without shell exposure)

Dashboard → Keys → copy the Unified API key value.

Store it in ~/.hermes/.env directly — avoid export to prevent it appearing in shell history or scrollback:

# Write directly to .env without echoing the key to terminal
read -rsp "Paste FreeLLMAPI unified key (input hidden): " FKEY
printf '\nFREELLMAPI_UNIFIED_KEY=%s\n' "$FKEY" >> ~/.hermes/.env
chmod 600 ~/.hermes/.env
unset FKEY
echo "Key stored in ~/.hermes/.env"

Phase 2 — Discover the Best Current Models

Repeat this phase whenever you want to refresh the MoA preset. Never hardcode model names — always query the live catalog.

2.1 Query the catalog with automatic fallback

The script handles two cases: FreeLLMAPI exposes ranking metadata (intelligence_rank, provider), or it does not.

# Source the key if not already set in environment
[ -z "$FREELLMAPI_UNIFIED_KEY" ] && source ~/.hermes/.env

curl -s \
  -H "Authorization: Bearer $FREELLMAPI_UNIFIED_KEY" \
  http://localhost:3001/v1/models \
  | python3 - << 'PYEOF'
import json, sys

raw = json.load(sys.stdin)
models = raw.get('data', raw.get('models', []))

# Filter to chat-capable models only
exclude = ['embed', 'whisper', 'tts', 'image', 'vision-only', 'moderation']
chat = [
    m for m in models
    if m.get('object', 'model') in ('model', 'chat')
    and not any(t in m.get('id', '').lower() for t in exclude)
]

has_rank = any(m.get('intelligence_rank') for m in chat)
has_provider = any(m.get('provider') for m in chat)

if has_rank:
    chat.sort(key=lambda m: m.get('intelligence_rank', 0), reverse=True)
    print('[INFO] Sorted by intelligence_rank from API response.')
else:
    print('[WARN] intelligence_rank not present in /v1/models response.')
    print('[ACTION] Use FreeLLMAPI dashboard → Models tab to rank manually.')
    print('[ACTION] Copy exact model IDs and paste into Phase 3 template.')
    chat.sort(key=lambda m: m.get('id', ''))

print(f'\n{"#":<5} {"Provider":<22} {"Model ID"}')
print('-' * 75)
for i, m in enumerate(chat[:20], 1):
    provider = m.get('provider', m.get('owned_by', '?'))
    rank = f" [rank:{m['intelligence_rank']}]" if has_rank and m.get('intelligence_rank') else ''
    print(f"{i:<5} {provider:<22} {m['id']}{rank}")

if not has_provider:
    print('\n[WARN] Provider family field not present. Group by model name prefix manually.')
PYEOF

2.2 Select models for your MoA preset

From the output, apply these rules:

• Aggregator: single highest-ranked chat-capable model overall
• Reference 1: highest-ranked model from a different provider family than the aggregator
• Reference 2: highest-ranked model from a third provider family

If the script flagged missing fields: open the FreeLLMAPI dashboard → Models tab (sorted by intelligence rank by default in the UI) and identify families manually.

Note the exact Model ID strings — you need them verbatim in Phase 3.

Phase 3 — Configure Hermes

3.1 Register FreeLLMAPI as a Hermes custom provider

Use the interactive wizard. This is the canonical, version-safe method:

hermes model

When prompted:

• Select: Custom endpoint (self-hosted / VLLM / etc.)
• API base URL: http://localhost:3001/v1
• API key: paste your FreeLLMAPI unified key (input is hidden)
• Model name: paste any valid chat model ID from Phase 2

Hermes writes this to ~/.hermes/config.yaml and validates the connection.

Why wizard-first? Hermes docs and the official providers page document literal API key values in config.yaml; env-var interpolation (${VAR}) in YAML is not explicitly confirmed in current Hermes documentation. The wizard handles key storage safely and is version-stable.

Verify after registration:

hermes doctor

Expected: green checkmark on the custom provider. If it fails, confirm FreeLLMAPI is still running: curl http://localhost:3001/v1/models should return JSON.

3.2 Configure the MoA preset (interactive)

hermes moa configure free-sota

This opens an interactive prompt. Supply the model IDs from Phase 2.

Alternatively, append directly to ~/.hermes/config.yaml — replace all <placeholder> values with exact model IDs from Phase 2:

# Append to ~/.hermes/config.yaml
# Do not add this block more than once. If moa: already exists, merge the presets key.
moa:
  default_preset: free-sota
  presets:
    free-sota:
      reference_models:
        - provider: custom
          model: "<reference-model-id-1-from-different-family>"
        - provider: custom
          model: "<reference-model-id-2-from-third-family>"
      aggregator:
        provider: custom
        model: "<highest-ranked-model-id>"
      reference_temperature: 0.7
      aggregator_temperature: 0.3
      max_tokens: 4096
      enabled: true

Verify the preset registered:

hermes moa list
# Expected: free-sota appears in the list

3.3 Set free-sota as the default MoA preset (optional)

hermes moa configure   # update default preset

Phase 4 — Activate and Use

4.1 Session-wide MoA mode

hermes
/model free-sota --provider moa

All turns in this session: aggregator writes the response, references advised silently first. Normal agent loop (tool calls, follow-up iterations, goal mode) all work through MoA as documented.

4.2 One-shot MoA without changing active model

/moa <your hard task here>

Hermes runs one turn through the default MoA preset, then restores your previous model.

4.3 Verify calls route through FreeLLMAPI

FreeLLMAPI dashboard → Requests tab after a /moa call: you should see N+1 entries (one per reference + one aggregator) clustered by timestamp.

Phase 5 — Ongoing Maintenance

hermes moa configure free-sota   # interactive update of the preset

Pitfalls

FreeLLMAPI not reachable: confirm it's running (npm run dev in the freellmapi directory). In persistent deployments use npm run build && npm run start or a process manager (PM2, systemd).

Model ID not found: IDs change when providers rename models. Re-run Phase 2 and update the preset.

Reference call failures: expected when a provider quota is exhausted. FreeLLMAPI reroutes silently; Hermes logs the failure in reference context and continues. Not a bug.

All references from same family: see Premortem. Override by selecting top model from each of three distinct provider families.

MoA increases latency: expected — reference calls run before the aggregator. Use /model <aggregator-model-id> --provider custom directly for latency-sensitive tasks.

Recursive MoA: blocked by Hermes design. Never set the aggregator to another MoA preset.

Verification Checklist

1. hermes doctor — green on custom provider ✓
2. hermes moa list — free-sota shown ✓
3. /moa what is 2+2 — completes without error ✓
4. FreeLLMAPI Requests tab — shows N+1 calls per MoA turn ✓
5. Aggregator model ID in Requests log matches Phase 3 config ✓

Credits

• FreeLLMAPI — Tashfeen Ahmed and contributors https://github.com/tashfeenahmed/freellmapi
• Hermes Agent — Nous Research and contributors https://github.com/NousResearch/hermes-agent
• Mixture-of-Agents research — Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, James Zou (Together AI / Stanford / U. Chicago / Duke) https://arxiv.org/abs/2406.04692
• Agent Skills specification — agentskills.io contributors https://agentskills.io/specification

References

• FreeLLMAPI repository: https://github.com/tashfeenahmed/freellmapi
• Hermes Agent documentation: https://hermes-agent.nousresearch.com/docs
• Hermes MoA feature docs: https://hermes-agent.nousresearch.com/docs/user-guide/features/mixture-of-agents
• Hermes providers docs: https://hermes-agent.nousresearch.com/docs/integrations/providers
• Hermes skills authoring: https://hermes-agent.nousresearch.com/docs/developer-guide/creating-skills
• MoA paper: https://arxiv.org/abs/2406.04692