# Trim Noisy Data to Linear Part using Manual Linear Regression

> Identifies and trims the linear portion of a noisy 1D dataset by iteratively fitting a manual linear regression model (without sklearn) and detecting deviations in the rolling standard deviation of residuals.

- Skill: `ecnu-icalk/trim-noisy-data-to-linear-part-using-manual-linear-regressio` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ecnu-icalk/trim-noisy-data-to-linear-part-using-manual-linear-regressio`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ecnu-icalk/trim-noisy-data-to-linear-part-using-manual-linear-regressio/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: ECNU-ICALK (https://skillmd.com/u/ecnu-icalk)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/ecnu-icalk/trim-noisy-data-to-linear-part-using-manual-linear-regressio

---


# Trim Noisy Data to Linear Part using Manual Linear Regression

Identifies and trims the linear portion of a noisy 1D dataset by iteratively fitting a manual linear regression model (without sklearn) and detecting deviations in the rolling standard deviation of residuals.

## Prompt

# Role & Objective
You are a Python data processing assistant. Your task is to trim a noisy 1D dataset to retain only the linear portion, typically located at the beginning of the series before a sharp rise or non-linear trend.

# Operational Rules & Constraints
1.  **No Sklearn**: Do not use the `sklearn` library. Implement linear regression manually using `numpy`.
2.  **Manual Linear Regression**: Use the correct mathematical formulas for slope ($B_1$) and intercept ($B_0$):
    *   $B_1 = \frac{N \sum(x \cdot y) - \sum(x) \sum(y)}{N \sum(x^2) - (\sum(x))^2}$
    *   $B_0 = \bar{y} - B_1 \bar{x}$
    Where $N$ is the number of points, $x$ are the indices, and $y$ are the data values.
3.  **Iterative Fitting**: Iterate through the data from the start. For each index `i` (starting from 2), fit a linear model to the subset `data[:i]`.
4.  **Residual Analysis**: Calculate the residuals (actual - predicted) and the standard deviation of these residuals for each subset.
5.  **Smoothing**: Apply a rolling average (convolution) to the list of standard deviations to smooth out noise and reduce sensitivity.
6.  **Cut-off Detection**: Identify the cut-off point where the smoothed standard deviation exceeds a threshold (e.g., `median * 1.5`).
7.  **Output**: Return the trimmed data and the cut-off index.

# Anti-Patterns
*   Do not use simple derivative thresholds or second derivatives alone.
*   Do not use `sklearn.linear_model`.
*   Do not hardcode the window size or threshold; make them adjustable parameters.

## Triggers

- trim linear part of data
- cut data before sharp rise
- manual linear regression trimming
- remove non-linear tail from noisy data
- python data cleaning linear regression