Amey-Thakur ML problem framing
Most failed ML projects were framed wrong, not modeled wrong. The frame is: what decision changes, what exactly is predicted, what does each kind of error cost, and would something simpler make the same decision.
Method
- Start from the decision, not the data. Name the action the prediction triggers ("flag order for review", "rank these items", "set this price") and who or what consumes it. A prediction nobody acts on is a dashboard; if the action is unclear, stop here.
- Specify the target variable ruthlessly. "Churn" becomes "no login within 30 days of day X, for accounts active in the prior 60". Definition choices (window, population, exclusions) change the problem more than any model choice will; write them down and get the domain owner to sign (see warehouse-modeling on metric definitions).
- Verify the label's availability and honesty. When does ground truth arrive (instant, 30 days, never)? Is it censored (you only see fraud you caught), biased by the current process (loan defaults observed only for approved loans), or drifting? Label latency bounds your retraining loop; label bias caps what any model can learn.
- Price the errors asymmetrically. False positive cost vs false negative cost, in money or user harm, from the decision owner. This drives the metric (see model-evaluation), the operating threshold, and often the architecture (a cheap model plus human review can beat an expensive autonomous one).
- Establish the non-ML baseline first. The current heuristic, a rules list, a single feature threshold, or "predict the historical mean" (see ml-baselines). The project's real question is the delta over this baseline at the decision, not the model's absolute score; sometimes the baseline wins and the project should become three IF statements.
- Define success upstream and downstream. Offline: metric and threshold on a proper holdout (see train-test-discipline). Online: the business metric the decision should move, measured by experiment (see ab-test-design), plus guardrails (latency budget, fairness slices, override rate). Agree on the kill criteria before building; projects without them run on sunk cost.
Boundaries
- Problems needing 100% correctness, full explainability by law, or where errors are catastrophic and unauditable are rules-and-review territory, not ML.
- Insufficient or unlabelable data is a data-collection project first; committing model timelines before the label pipeline exists reverses the dependency.
- Frame once, revisit at every scope change; silent target drift ("churn now means downgrade too") invalidates every past evaluation.