# System Design & Architecture

> The text_segmenter is a core utility that depends on core.utils for constants and logging.

- Skill: `tools-only/system-design-and-architecture` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add tools-only/system-design-and-architecture`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/system-design-and-architecture/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-29
- Page: https://skillmd.com/skills/tools-only/system-design-and-architecture

---


# System Design & Architecture

## Architecture Overview
**What is the high-level system structure?**

The `text_segmenter` is a core utility that depends on `core.utils` for constants and logging.

```mermaid
graph TD
    Validator[Validator/Checker] --> Segmenter[core/text_segmenter.py]
    Segmenter --> Utils[core/utils.py]
```

## Data Models
**What data do we need to manage?**

### Chunk Data Class
```python
@dataclass(frozen=True)
class Chunk:
    text: str
    start_index: int
    end_index: int
    is_partial: bool  # True if it's the last, smaller chunk
```

## API Design
**How do components communicate?**

### Primary Interface
- `segment_text(text: str, chunk_size: int = 600, overlap: int = 100) -> List[Chunk]`
- `normalize_whitespace(text: str) -> str`: Helper to ensure single spaces and no trailing whitespace.

## Component Breakdown
**What are the major building blocks?**

- **Normalizer:** Pre-processes the input string.
- **Sliding Window Logic:** Calculates the slices based on size and overlap.
- **Chunk Factory:** Produces `Chunk` objects with correct indices.

## Design Decisions
**Why did we choose this approach?**

- **Character-based slicing:** More predictable than word-based slicing when dealing with potential OCR errors or strange formatting in PDF.
- **Frozen Dataclass:** Ensures chunks are immutable and safe to pass around.
- **Overlap Formula:** `start_index = max(0, previous_end - overlap)`.

## Non-Functional Requirements
**How should the system perform?**

- **Performance:** O(n) complexity.
- **Memory:** Should not create unnecessary copies of long strings during slicing (using Python's efficient string slicing).
- **Correctness:** Must never lose a single character from the original text (except for normalized whitespace).