Working with autoresearch
Autoresearch answers one standing question per model: who in a population will do a target within a horizon?
For example, "which active users will open a report in the next 7 days?".
- The user names a target (an event or a Flind action), a population and a horizon in days.
- A training agent runs in a Flind sandbox. It writes feature SQL and a model, scores each experiment on a holdout set, keeps what improves holdout AUC and discards the rest.
- The server picks the champion from the kept experiments. The agent proposes; the backend decides.
- Every cadence (daily by default, at 02:00 UTC), the champion scores the inference population.
Each scored person gets an
autoresearch_prediction event with their probability.
When the model's write_person_property is on, a person who resolves to a current distinct ID also gets the model's output person property, for example predicted_p_adopt_insights_14d. It is off by default for new models. Turn it on with autoresearch-partial-update before you target people by score in a flag, cohort, workflow or insight. Turning it off removes the property from the people the model scored. Changing the setting needs the person:write scope.
- When a prediction date's horizon has passed, online validation checks the scores against what really happened and records realized AUC.
- Training keeps going on a schedule. On each scoring day the model starts a research run (explore) until a run does not promote a better model, or until 5 research runs have run.
After that it starts one research run every
research_interval_days (maintain, default 7) after the latest training run, so a run started by hand resets the clock.
With research_interval_days null the model trains only when someone asks. A champion that cannot score still gets a short rescue run.
training_phase and next_research_at on posthog:autoresearch-retrieve say where the model is.
In the Flind UI the entity is a model. The API and tools call it a pipeline. Use "model" when you talk to people.
Autoresearch is behind the autoresearch feature flag.
Training runs in a cloud sandbox, so it also needs Flind Desktop access.
If the autoresearch-* tools are missing, the project probably does not have access yet.
Pick the job
| The user wants to |
Start with |
Reference |
| Create a model |
posthog:autoresearch-resolve-template-create or a custom definition, then posthog:autoresearch-validate-create |
defining-a-question.md |
| Know if a model is any good |
posthog:autoresearch-retrieve, then posthog:autoresearch-online-performance-retrieve |
reading-model-quality.md |
| Use the predictions |
The output person property, or posthog:execute-sql on autoresearch_prediction |
using-predictions.md |
| Fix or steer a model |
posthog:autoresearch-training-runs-list, posthog:autoresearch-runs-list, posthog:autoresearch-suggestions-create |
troubleshooting-and-steering.md |
| See what models exist |
posthog:autoresearch-list |
This page |
Create, train, score
The usual path is short. Each tool's description names its prerequisites and refusals, so read it with info before the first call.
- Resolve or define. Templates cover the common questions (activity, inactivity risk, return after first use, feature adoption, repeat behavior). A custom definition works when none fits. See defining-a-question.md.
- Validate.
posthog:autoresearch-validate-create with the same target, horizon, lookback and populations you will create with. It returns the base rate, positive and negative counts and warnings. Fix anything that sets can_proceed to false, and tell the user about the non-blocking warnings.
- Create.
posthog:autoresearch-create. The new model is a draft until training starts.
- Train.
posthog:autoresearch-train-create starts an agent run that takes a while (often tens of minutes). It costs real sandbox and LLM time, so confirm with the user before you start one, and never start a second run while one is live. A 403 means either the call came from inside a sandbox (ask the user to click Start training on the model's page) or the user has no Flind Desktop access (the page will refuse too). Read the error message to tell them apart.
- Wait. Poll
posthog:autoresearch-training-runs-list at a relaxed pace. While the first run is in progress the model is bootstrapping, and a missing champion is normal.
- Score. After a champion is promoted, the next daily sweep scores the population.
posthog:autoresearch-score-create scores today now; call it once, then poll posthog:autoresearch-runs-list. A second manual score for the same date can duplicate predictions.
Pause, resume and delete are in the model's menu in the UI only.
Pausing stops scoring and new training runs. A training run already in progress keeps running, and it can still promote a champion.
Model status
| Status |
Meaning |
draft |
Defined, never trained. |
bootstrapping |
The first training run is in progress. No champion yet. |
running |
A champion exists and scores on the cadence. Retraining can still happen. |
converged |
Treated like running: the champion still scores and is still validated. |
paused |
No scoring and no new training until someone resumes it in the UI. A run already in progress finishes. |
The training run's iteration_count and best_holdout_score are written only when the run completes.
A healthy run can show 0 iterations while its experiments are already recorded, so do not call a run stuck from the counter alone.
How to talk about results
- Holdout AUC is the score on held-back training data. It says how well the model ranked people in the past. Treat it as a promise.
- Realized AUC is the score against real outcomes after the horizon passes. Treat it as the proof. It wins over holdout AUC whenever it exists.
- The UI labels quality Strong at AUC 0.8 and up, Fair at 0.7 and up, and Weak below that, and says whether the label comes from testing only or is confirmed by real outcomes. Use the same words so the agent and the UI agree.
- Lift at 10% is the plain-language headline: "the top 10% are 3.2× more likely to do X than average".
- The score is a probability, but on a rare target it is often best read as a ranking. Say so when the base rate is below a few percent.
reading-model-quality.md covers calibration, likelihood segments, coverage and what to say when the numbers disagree.
Guardrails worth keeping in mind
- Changing the question means a new model. The fields a model was trained on (target, horizon, populations) are frozen once a model exists. Create a new model rather than editing one.
- Autoresearch trains on identified users only. Anonymous traffic does not count toward training data, so raw event volume can overstate what is available. Trust the validation counts.
- Scores are keyed on the person. A model whose scores are almost all the same value usually has an identifier problem, not a weak signal. See troubleshooting-and-steering.md.
- Read the output property name, do not guess it. Take
output_person_property from posthog:autoresearch-retrieve. A derived name includes the target and the horizon (predicted_p_<target>_<horizon>d), and templates use their own prefix, so two models on one target do not overwrite each other.
- The research model sets most of the training cost. A training run uses the model's
agent_model and agent_reasoning_effort. Null uses the default, Claude Opus 5.5 at medium effort, which costs about twice what Claude Sonnet 5.5 costs per run. effective_agent_model on posthog:autoresearch-retrieve shows what the next run uses, and posthog:autoresearch-partial-update changes it. An unknown model or an effort the model does not support returns 400 with the valid choices. posthog:autoresearch-training-runs-list shows the model each run used.
- A model costs money to train and query to score. A very large population scores in rolling batches (about 45,000 people per run), so not everyone gets a fresh score every day. Validation tells you how many runs a full rescore takes.
Bring your own agent
An agent outside Flind (on a laptop, for example) can train a model for an existing pipeline.
It opens a run with posthog:autoresearch-training-runs-create, records each experiment with posthog:autoresearch-training-runs-iterations-create, uploads its bundle with posthog:autoresearch-training-runs-artifacts-upload-create, and finishes with posthog:autoresearch-training-runs-complete-create.
It pulls training data with posthog:execute-sql, because posthog:autoresearch-materialize-features writes only into a Flind sandbox.
Read posthog:autoresearch-training-runs-history first to reuse what earlier runs on the same target learned.
The iteration tool's description is the contract for valid feature SQL: read it in full before the first iteration.
1---2name: working-with-autoresearch3description: >4 Use Flind Autoresearch, agent-driven predictive models: ask "who will do X in the next N days?",5 let an agent train a model, then score people daily into a person property and6 `autoresearch_prediction` events. Use when a user wants to predict churn, activity, return,7 feature adoption, conversion or any event or action per person; create or validate a model from a8 template; start training or score now; read model quality (holdout vs realized AUC, lift,9 calibration, likelihood segments, coverage); steer training with suggestions; act on predictions10 with cohorts, feature flags, workflows or SQL; or diagnose a model that fails to train, scores11 everyone the same, or stops scoring. Trigger on "autoresearch", "predictive model", "propensity12 score", "churn prediction", "likely to convert", "who will do X", "predicted_p_",13 "autoresearch_prediction", "realized AUC", "champion model".14metadata:15 source: https://skillmd.com/skills/thehalfmoon/working-with-autoresearch16---1718# Working with autoresearch1920Autoresearch answers one standing question per model: **who in a population will do a target within a horizon?**21For example, "which active users will open a report in the next 7 days?".22231. The user names a **target** (an event or a Flind action), a **population** and a **horizon** in days.242. A training agent runs in a Flind sandbox. It writes feature SQL and a model, scores each experiment on a holdout set, keeps what improves holdout AUC and discards the rest.253. The server picks the **champion** from the kept experiments. The agent proposes; the backend decides.264. Every cadence (daily by default, at 02:00 UTC), the champion scores the inference population.27 Each scored person gets an `autoresearch_prediction` event with their probability.28 When the model's `write_person_property` is on, a person who resolves to a current distinct ID also gets the model's output person property, for example `predicted_p_adopt_insights_14d`. It is off by default for new models. Turn it on with `autoresearch-partial-update` before you target people by score in a flag, cohort, workflow or insight. Turning it off removes the property from the people the model scored. Changing the setting needs the `person:write` scope.295. When a prediction date's horizon has passed, online validation checks the scores against what really happened and records **realized AUC**.306. Training keeps going on a schedule. On each scoring day the model starts a research run (**explore**) until a run does not promote a better model, or until 5 research runs have run.31 After that it starts one research run every `research_interval_days` (**maintain**, default 7) after the latest training run, so a run started by hand resets the clock.32 With `research_interval_days` null the model trains only when someone asks. A champion that cannot score still gets a short rescue run.33 `training_phase` and `next_research_at` on `posthog:autoresearch-retrieve` say where the model is.3435In the Flind UI the entity is a **model**. The API and tools call it a **pipeline**. Use "model" when you talk to people.3637Autoresearch is behind the `autoresearch` feature flag.38Training runs in a cloud sandbox, so it also needs Flind Desktop access.39If the `autoresearch-*` tools are missing, the project probably does not have access yet.4041## Pick the job4243| The user wants to | Start with | Reference |44| --------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |45| Create a model | `posthog:autoresearch-resolve-template-create` or a custom definition, then `posthog:autoresearch-validate-create` | [defining-a-question.md](references/defining-a-question.md) |46| Know if a model is any good | `posthog:autoresearch-retrieve`, then `posthog:autoresearch-online-performance-retrieve` | [reading-model-quality.md](references/reading-model-quality.md) |47| Use the predictions | The output person property, or `posthog:execute-sql` on `autoresearch_prediction` | [using-predictions.md](references/using-predictions.md) |48| Fix or steer a model | `posthog:autoresearch-training-runs-list`, `posthog:autoresearch-runs-list`, `posthog:autoresearch-suggestions-create` | [troubleshooting-and-steering.md](references/troubleshooting-and-steering.md) |49| See what models exist | `posthog:autoresearch-list` | This page |5051## Create, train, score5253The usual path is short. Each tool's description names its prerequisites and refusals, so read it with `info` before the first call.54551. **Resolve or define.** Templates cover the common questions (activity, inactivity risk, return after first use, feature adoption, repeat behavior). A custom definition works when none fits. See [defining-a-question.md](references/defining-a-question.md).562. **Validate.** `posthog:autoresearch-validate-create` with the same target, horizon, lookback and populations you will create with. It returns the base rate, positive and negative counts and warnings. Fix anything that sets `can_proceed` to false, and tell the user about the non-blocking warnings.573. **Create.** `posthog:autoresearch-create`. The new model is a `draft` until training starts.584. **Train.** `posthog:autoresearch-train-create` starts an agent run that takes a while (often tens of minutes). It costs real sandbox and LLM time, so confirm with the user before you start one, and never start a second run while one is live. A 403 means either the call came from inside a sandbox (ask the user to click **Start training** on the model's page) or the user has no Flind Desktop access (the page will refuse too). Read the error message to tell them apart.595. **Wait.** Poll `posthog:autoresearch-training-runs-list` at a relaxed pace. While the first run is in progress the model is `bootstrapping`, and a missing champion is normal.606. **Score.** After a champion is promoted, the next daily sweep scores the population. `posthog:autoresearch-score-create` scores today now; call it once, then poll `posthog:autoresearch-runs-list`. A second manual score for the same date can duplicate predictions.6162Pause, resume and delete are in the model's menu in the UI only.63Pausing stops scoring and new training runs. A training run already in progress keeps running, and it can still promote a champion.6465## Model status6667| Status | Meaning |68| --------------- | ------------------------------------------------------------------------------------------------------ |69| `draft` | Defined, never trained. |70| `bootstrapping` | The first training run is in progress. No champion yet. |71| `running` | A champion exists and scores on the cadence. Retraining can still happen. |72| `converged` | Treated like `running`: the champion still scores and is still validated. |73| `paused` | No scoring and no new training until someone resumes it in the UI. A run already in progress finishes. |7475The training run's `iteration_count` and `best_holdout_score` are written only when the run completes.76A healthy run can show 0 iterations while its experiments are already recorded, so do not call a run stuck from the counter alone.7778## How to talk about results7980- **Holdout AUC** is the score on held-back training data. It says how well the model ranked people in the past. Treat it as a promise.81- **Realized AUC** is the score against real outcomes after the horizon passes. Treat it as the proof. It wins over holdout AUC whenever it exists.82- The UI labels quality **Strong** at AUC 0.8 and up, **Fair** at 0.7 and up, and **Weak** below that, and says whether the label comes from testing only or is confirmed by real outcomes. Use the same words so the agent and the UI agree.83- **Lift at 10%** is the plain-language headline: "the top 10% are 3.2× more likely to do X than average".84- The score is a probability, but on a rare target it is often best read as a ranking. Say so when the base rate is below a few percent.8586[reading-model-quality.md](references/reading-model-quality.md) covers calibration, likelihood segments, coverage and what to say when the numbers disagree.8788## Guardrails worth keeping in mind8990- **Changing the question means a new model.** The fields a model was trained on (target, horizon, populations) are frozen once a model exists. Create a new model rather than editing one.91- **Autoresearch trains on identified users only.** Anonymous traffic does not count toward training data, so raw event volume can overstate what is available. Trust the validation counts.92- **Scores are keyed on the person.** A model whose scores are almost all the same value usually has an identifier problem, not a weak signal. See [troubleshooting-and-steering.md](references/troubleshooting-and-steering.md).93- **Read the output property name, do not guess it.** Take `output_person_property` from `posthog:autoresearch-retrieve`. A derived name includes the target and the horizon (`predicted_p_<target>_<horizon>d`), and templates use their own prefix, so two models on one target do not overwrite each other.94- **The research model sets most of the training cost.** A training run uses the model's `agent_model` and `agent_reasoning_effort`. Null uses the default, Claude Opus 5.5 at medium effort, which costs about twice what Claude Sonnet 5.5 costs per run. `effective_agent_model` on `posthog:autoresearch-retrieve` shows what the next run uses, and `posthog:autoresearch-partial-update` changes it. An unknown model or an effort the model does not support returns 400 with the valid choices. `posthog:autoresearch-training-runs-list` shows the model each run used.95- **A model costs money to train and query to score.** A very large population scores in rolling batches (about 45,000 people per run), so not everyone gets a fresh score every day. Validation tells you how many runs a full rescore takes.9697## Bring your own agent9899An agent outside Flind (on a laptop, for example) can train a model for an existing pipeline.100It opens a run with `posthog:autoresearch-training-runs-create`, records each experiment with `posthog:autoresearch-training-runs-iterations-create`, uploads its bundle with `posthog:autoresearch-training-runs-artifacts-upload-create`, and finishes with `posthog:autoresearch-training-runs-complete-create`.101It pulls training data with `posthog:execute-sql`, because `posthog:autoresearch-materialize-features` writes only into a Flind sandbox.102Read `posthog:autoresearch-training-runs-history` first to reuse what earlier runs on the same target learned.103The iteration tool's description is the contract for valid feature SQL: read it in full before the first iteration.