# Dmx Local Modeling

> Implementation-oriented local `dmx-learn` fitting skill for coding, fitting, diagnosing, and using models on local or in-memory data once the problem structure is known. Prefer explicit `dmx.stats` estimators for ordinary non-Bayesian work and use `dmx.bstats` for Bayesian, variational, DPM, and automatic mixture workflows. Do not use for Spark, MPI, or other distributed estimation workflows.

- Skill: `llnl/dmx-local-modeling` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add llnl/dmx-local-modeling`
- Raw SKILL.md: https://api.skillmd.com/api/skills/llnl/dmx-local-modeling/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: llnl (https://skillmd.com/u/llnl)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/llnl/dmx-local-modeling

---


# Dmx Local Modeling

Use this skill for implementation-heavy local modeling work after the problem
has already been scoped to a concrete local `dmx-learn` path.

This skill is narrow on purpose:

- it writes explicit estimators, fitting loops, and diagnostics
- it assumes the task is local and in scope for `dmx-learn`
- it does not own broad intake, lightweight EDA, or high-level routing policy

When the problem framing is still ambiguous, hand that work to
[`dmx-expert-orchestrator`](../dmx-expert-orchestrator/SKILL.md) first.

## When To Use This Skill

Use this skill when at least one of these is already known:

- the observation structure is clear enough to code directly
- the model family has already been chosen upstream
- the user wants runnable fitting code, diagnostics, or post-fit usage
- the main question is about implementation details in `dmx.stats`,
  `dmx.bstats`, or `dmx.utils`

Do not use this skill as the main router for vague modeling requests. For:

- intake and lightweight local-data inspection
- structure-first routing across base, composite, mixture, sequence, joint, or
  grouped models
- deciding when to fit one primary model plus one baseline
- deciding whether the task should stay broad and reusable instead of going
  straight to a narrow conditional path

read [`../dmx-expert-orchestrator/SKILL.md`](../dmx-expert-orchestrator/SKILL.md)
and its
[`references/hierarchy-and-data-structure.md`](../dmx-expert-orchestrator/references/hierarchy-and-data-structure.md)
first.

## Default Workflow

### 1. Confirm The Implementation Target

Before coding, restate only the facts needed for implementation:

- what one observation is
- the intended estimator family or estimator tree
- the fitting objective or evaluation target
- any known validation split, restart budget, or scale constraint

If those facts are still uncertain, stop routing locally and defer back to the
orchestrator instead of improvising your own broad routing pass here.

### 2. Choose The Local Modeling Surface

- Prefer explicit `dmx.stats` estimator construction for ordinary
  non-Bayesian estimation when the model family is known.
- Use `dmx.bstats` when the implementation needs explicit priors, expected
  log-density calculations, local variational fitting, a truncated DPM, or an
  automatic estimator that will participate in a Bayesian mixture workflow.
- `dmx.utils.automatic.get_estimator(data, use_bstats=True)` is a valid
  first-class route when automatic Bayesian structure inference is intentional.
  Keep `use_bstats=True` explicit in generated code even though the current
  helper default is `True`.
- `dmx.utils.automatic.get_dpm_mixture(data, ...)` is the first-class local
  automatic DPM route. It fits a truncated `dmx.bstats` DPM variationally,
  removes components below its count threshold, and returns a finite
  `dmx.bstats.MixtureDistribution`.
- Keep the hierarchy chosen upstream intact. Do not flatten a composite,
  sequence, or grouped problem into a simpler estimator only because it is
  quicker to code.
- Read [`references/model-routing.md`](references/model-routing.md) when you
  need the concrete estimator family map after the structure is already known.
- Reuse the orchestrator's structure-first philosophy: hierarchy choice comes
  first, field-level estimators second.

For advanced structure selection, grouped-sharing policy, or joint-model-first
reasoning, go back to the orchestrator reference instead of re-deriving that
logic in this skill.

### 3. Use Repo-Native Local Fitting Helpers

Read [`references/repo-entry-points.md`](references/repo-entry-points.md) for
the concrete repo paths.

Default non-Bayesian implementation path:

1. Build an explicit estimator in `dmx.stats`.
2. Split data with `dmx.utils.estimation.partition_data` when held-out
   validation is needed.
3. Fit with `dmx.utils.estimation.optimize`.
4. Upgrade to `dmx.utils.estimation.best_of` when initialization sensitivity is
   plausible, especially for mixtures or other latent-variable models.
5. Use vectorized post-fit calls such as `seq_encode`, `seq_log_density`,
   `seq_log_density_sum`, or related posterior helpers instead of repeated
   scalar scoring.

Bayesian and automatic implementation paths:

- Build explicit `dmx.bstats` estimators and fit them with local helpers in
  `dmx.bstats.bestimation` when priors or variational behavior must be
  controlled directly.
- Use `get_estimator(..., use_bstats=True)` for automatic Bayesian estimator
  construction, including composite, optional, and sequence structures.
- Use `get_dpm_mixture` for a concise local automatic mixture workflow.
- `prepare_mixture_model` is a first-class automatic mixture route for
  embedding flows; without a supplied model it delegates to the local
  `get_dpm_mixture` path.

Keep the execution boundary explicit: these are local helpers. Do not route to
`dmx.mpi4py.bstats`, `dmx.mpi4py.utils.bestimation`, or
`get_dpm_mixture_mpi`; MPI workflows require separate distributed guidance.

### 4. Keep Shared-Structure Implementations Explicit

This skill should still be useful for coding and fitting advanced local models,
but not for deciding whether they are the right first abstraction.

Implementation defaults:

- for heterogeneous records, build field estimators explicitly and combine them
  with `CompositeEstimator`
- for latent subtypes over heterogeneous records, implement a
  `CompositeEstimator` inside a `MixtureEstimator`
- for Bayesian latent subtypes over heterogeneous records, prefer a
  `dmx.bstats.CompositeEstimator` as the base estimator for a truncated DPM;
  this DPM-over-composite path is the primary high-value `bstats` workflow
- for many-label problems with shared latent structure, use keyed shared
  components explicitly instead of hiding the structure inside many unrelated
  conditional branches

Use [`references/model-routing.md`](references/model-routing.md) and
[`references/repo-entry-points.md`](references/repo-entry-points.md) for the
keyed mixture and composite-mixture patterns. Use the orchestrator references
for the higher-level question of when those patterns should be preferred.

### 5. Code Diagnostics And Post-Fit Usage

Choose the success criterion before writing the final code path:

- held-out log likelihood or likelihood comparisons
- classification metrics or ranking depth
- posterior inspection or cluster structure
- embedding-oriented exploratory views
- parameter sanity and fit stability

Repo-native helpers:

- `src/dmx/utils/metrics.py` for classification and ranking-style evaluation
- `src/dmx/utils/pvalues.py` only for narrow approximate significance or rank
  calculations, not as a generic testing framework
- `src/dmx/utils/htsne.py` and `src/dmx/utils/humap.py` for embedding workflows

For cleaned `bstats` workflow evidence, use:

- `tests/bstats/dpm_test.py` for deterministic DPM initialization, variational
  updates, local optimization, and `get_dpm_mixture` conversion
- `tests/bstats/composite_test.py` for a composite distribution inside finite
  mixture and DPM containers
- `tests/bstats/structural_test.py` and
  `tests/bstats/discrete_primitives_test.py` for
  `get_estimator(..., use_bstats=True)` routing

The MPI import smoke assertion in `tests/bstats/dpm_test.py` is compatibility
coverage only; it does not turn the local fitting recipe into an MPI workflow.

## Example Reuse Policy

- Prefer adapting nearby runnable examples over inventing new scaffolding.
- Start with `examples/stats_examples/gaussian_example.py` for the smallest
  fit-and-score flow.
- Use `examples/utils_examples/detailed_estimation_example.py` for validation and repeated-fit
  loops.
- Use `examples/stats_examples/mixture_example.py` for structured mixtures and
  keyed shared-component implementations.
- Use `examples/stats_examples/spearman_rho_example.py` for ranking workflows.
- For Bayesian and DPM behavior, prefer the focused `tests/bstats` files above;
  the current runnable examples primarily demonstrate `dmx.stats` models.

## Output Expectations

- Restate the implementation assumptions briefly before coding.
- Produce runnable code, not only estimator advice.
- Include fitting, scoring, and model-use snippets when they are relevant.
- Show how to inspect the fitted model or compute the requested diagnostic.
- Cite the exact repo example, utility, or stats module that informed the code.