2558 Rlm Realignment Strategy C8ff8657

RLM Realignment Strategy

Date: 2026-02-05 Owner: Aleph maintainers Status: Proposed execution plan

1) Goal

Realign Aleph with Recursive Language Model (RLM) research on two axes:

1. Presentation: clearly distinguish canonical RLM behavior from Aleph-specific platform features.
2. Ability: strengthen measurable RLM behavior (decompose, recurse, aggregate, verify) with explicit evaluation and regression checks.

2) Research Baseline (What "RLM-aligned" Means)

Using the RLM paper and official implementation as source of truth, canonical RLM behavior is:

1. Treat prompt/context as an external environment object, not as raw prompt tokens.
2. Use a REPL-style symbolic loop: inspect, decompose, execute, observe.
3. Recursively call sub-LMs for decomposition/aggregation when useful.
4. Iterate until a final answer is produced.

Important nuance from research:

1. RLM is an inference framework, not a product surface by itself.
2. RLM can outperform long-context baselines on information-dense tasks, but performance is workload-dependent.
3. Paper limitations explicitly call out open areas: optimal execution strategy, async sub-calls, and recursion-depth policy.

3) Current Repo Audit

3.1 What is already strongly aligned

1. Core closed-loop RLM runtime exists:
   • aleph/core.py implements iterative root loop + REPL + FINAL(...) protocol.
   • aleph/core.py supports sub_query and recursive sub_aleph.
2. Externalized context + symbolic interaction exists:
   • Context is loaded into ctx and explored via helpers/search/code execution.
3. Recursion budget controls exist:
   • Depth, iterations, wall time, and sub-query budgets are enforced.
4. Recursion behavior has test coverage:
   • Nested recursion and sub-query backends are tested in tests/test_double_recursion.py and tests/test_sub_query.py.

3.2 Where presentation currently drifts

1. Mode boundary is under-explicit:
   • Docs position Aleph broadly as RLM-based, but do not consistently separate:
     • canonical "Aleph core loop mode" (aleph run)
     • MCP tool-server orchestration mode (RLM-inspired, host-model-driven).
2. Documentation inconsistency reduces credibility:
   • DEVELOPMENT.md has stale backend priority and stale budget schema examples.
   • docs/CONFIGURATION.md timeout defaults are outdated versus runtime defaults.
3. Some guidance is phrased as universal when it is backend-dependent:
   • Prompt/docs imply fixed sub-query capacity heuristics while runtime enforces configurable truncation limits.

3.3 Where ability currently drifts from research rigor

1. No first-class benchmark harness reproducing paper-style task families in CI.
2. No published "research profile" config that reproduces paper-like control settings for fair comparison.
3. No recurring regression report tracking RLM-specific quality dimensions over time (decomposition quality, recursion efficiency, cost variance).

4) Strategy

Run two workstreams in parallel, then unify with an evaluation gate.

Workstream A: Presentation Realignment

A1. Introduce explicit product taxonomy

Add one canonical section to top-level docs:

1. Core RLM Mode (paper-aligned loop): aleph run, alef, Aleph.complete(...).
2. RLM Infrastructure Mode (MCP external memory server): tool-driven orchestration from host assistants.
3. Platform Extensions (not in canonical RLM): swarm workflows, remote MCP orchestration, workspace/action tooling.

A2. Establish a "Claims Ledger"

Create docs/RLM_CLAIMS_LEDGER.md mapping every major claim to one of:

1. Research-backed (with citation)
2. Code-backed (with repo file reference)
3. Aspirational (roadmap only)

No claim should remain unclassified.

A3. Remove doc drift and ambiguity

Synchronize:

1. README.md
2. docs/CONFIGURATION.md
3. DEVELOPMENT.md
4. docs/prompts/aleph.md

Rules:

1. Defaults must match runtime constants.
2. Backend priority must match aleph/sub_query/__init__.py.
3. Capacity guidance must mention truncation/config dependence.
4. "RLM" label should be used precisely: canonical loop vs inspired tooling.

Workstream B: Ability Realignment

B1. Add an RLM benchmark harness

Create a lightweight benchmark module with deterministic synthetic tasks mirroring paper structure:

1. Constant-information retrieval task (S-NIAH-style)
2. Linear aggregation task (OOLONG-style)
3. Pairwise aggregation task (OOLONG-Pairs-style)
4. Code context reasoning task (CodeQA-style)

Output JSON report fields:

1. accuracy
2. total tokens
3. wall time
4. recursion depth used
5. sub-query count
6. cost estimate (if provider supports it)

B2. Add "research profile" runtime preset

Add a named config profile (example: --profile rlm-paper-like) that pins:

1. recursion depth
2. iteration cap
3. sub-query cap
4. prompt template variant
5. model split policy (root vs sub-model)

Purpose: make "paper-like mode" reproducible and auditable.

B3. Add regression gates for RLM behavior

Extend tests with behavior-level checks:

1. decomposition quality (does model/toolchain split input and aggregate correctly on fixed fixtures)
2. recursion efficiency (budget use stays within expected bounds for fixtures)
3. stability (result variance under repeated runs with controlled temperature)

Workstream C: Evidence and Reporting

C1. Publish periodic evaluation snapshots

Add docs/reports/rlm-eval-YYYY-MM-DD.md containing:

1. benchmark outcomes
2. deltas versus previous run
3. regression explanations
4. confidence and known blind spots

C2. Tie docs to evidence

Any top-level performance statement in README must point to:

1. paper citation, or
2. local benchmark report,

and include run date.

5) Execution Plan (Phased)

Phase 0 (1-2 days): Correctness of messaging

1. Publish taxonomy section in README.md.
2. Fix stale defaults/priority mismatches in DEVELOPMENT.md and docs/CONFIGURATION.md.
3. Add docs/RLM_CLAIMS_LEDGER.md scaffold.

Exit criteria:

1. No contradictory defaults between docs and runtime for sub-query/backend/timeouts.
2. Every major README claim tagged as research-backed, code-backed, or aspirational.

Phase 1 (3-5 days): Measurement baseline

1. Implement benchmark harness and fixture set.
2. Add CI job that runs a reduced benchmark subset nightly.
3. Commit first docs/reports/rlm-eval-*.md.

Exit criteria:

1. Reproducible benchmark report exists.
2. README claims reference concrete report or paper.

Phase 2 (1-2 weeks): Capability hardening

1. Add research profile preset.
2. Add regression tests for decomposition/aggregation and recursion-budget behavior.
3. Improve sub-query policy controls (parallelism policy, truncation observability, retry behavior) behind explicit config.

Exit criteria:

1. RLM-specific regression gates exist in CI.
2. Profile-based reproducibility for paper-like runs is documented.

6) Success Metrics

Presentation metrics:

1. Zero stale-default mismatches between docs and runtime constants.
2. 100% of top-level claims linked to paper or local reports.
3. Clear user understanding of mode boundaries in docs review (qualitative check).

Ability metrics:

1. Benchmark suite committed and reproducible.
2. Stable pass/fail thresholds for synthetic RLM task families.
3. Trendline report shows non-regressing decomposition/aggregation quality.

7) Risks and Mitigations

1. Risk: Overfitting to synthetic benchmarks.
   • Mitigation: include both synthetic structure tests and real-world corpus fixtures.
2. Risk: Benchmark cost/runtime explosion.
   • Mitigation: tiered benchmark modes (smoke/nightly/full).
3. Risk: Docs become stale again.
   • Mitigation: add CI check that validates documented defaults against code constants.

8) Immediate Next Actions

1. Approve this plan as the execution baseline.
2. Start Phase 0 with doc corrections and claims ledger.
3. Open tracking issues for Phase 1 benchmark harness and Phase 2 regression gates.

9) Sources

1. Recursive Language Models paper (arXiv): https://arxiv.org/abs/2512.24601
2. Official RLM codebase: https://github.com/alexzhang13/rlm
3. Author article on RLM framing and limitations: https://towardsdatascience.com/recursive-language-models-new-rules-for-agentic-ai/