# Catlx Screen Understanding

> CATLX — OCR & Screen Understanding

- Skill: `samraddhashrivastavatech/catlx-screen-understanding` (Agent Skill)
- Install (CLI): `npx skillmds@latest add samraddhashrivastavatech/catlx-screen-understanding`
- Raw SKILL.md: https://api.skillmd.com/api/skills/samraddhashrivastavatech/catlx-screen-understanding/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: SAMRADDHASHRIVASTAVATECH (https://skillmd.com/u/samraddhashrivastavatech)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/samraddhashrivastavatech/catlx-screen-understanding

---


# CATLX — OCR & Screen Understanding

This skill owns **reading the screen as structured data**: converting screen regions to text and then to a
structured `ScreenModel` of the interactive surface that downstream automation uses. This is a distinct,
tier-scaled capability within the Desktop Control Engine.

> Canonical detail: `../knowledge/references/screen-understanding.md` and
> `../knowledge/references/desktop-control.md` §5.5. Example output: `../examples` (see
> `../examples/screen-model.json`). Load on demand.

---

## Purpose

Produce machine-readable structure from pixels so the Workflow Engine and DCE can click, type, and verify
actions WITHOUT hardcoded coordinates, and can work with legacy apps that lack accessibility trees.

## When to activate

- User asks how CATLX reads/understands the screen, or about OCR and screen models.
- Automating a legacy app (no UIAutomation tree).
- Verifying a UI action by reading screen state.

## What this skill handles

1. **Tier-scaled OCR backends** (from CapabilityMap `ocr_backend`):
   - T0: Tesseract 5.x (`eng+osd`) on CPU.
   - T1: Tesseract + result cache (avoid re-OCR of static regions).
   - T2+: EasyOCR / PaddleOCR with GPU for near-instant full-screen extraction.
2. **Screen Understanding post-processing** — from raw detected text: UI element classification (buttons,
   input fields, labels, links, text blocks), spatial relationship extraction, and element state detection
   (enabled/disabled, checked/unchecked, focused).
3. **`ScreenModel`** — a structured JSON of the screen's interactive surface: element type, bounding box,
   label/accessibility name, state, and spatial relationships. Example in `../examples/screen-model.json`.
4. **Per-display + global merge** — ScreenModels are computed per display by default and merged into a
   global ScreenModel on request (multi-monitor).

## Requirements / constraints

- Environment routing picks the OCR execution context: `LOCAL_PROCESS` (small regions), `LOCAL_SUBPROCESS`
  (T1+ CPU OCR), `LOCAL_CONTAINER` (T2+ GPU EasyOCR/PaddleOCR).
- **R5:** screen read is gated by the permission router (global toggle + per-app exclusions).
- The `ScreenModel` is the contract the Workflow Engine consumes; do not bypass it with raw coordinates.

## Canonical knowledge it reads

`../knowledge/references/screen-understanding.md` · `../knowledge/references/desktop-control.md` ·
`../knowledge/references/hardware-adaptation.md` · `../knowledge/references/security.md`.

## Delegation

- **Which OCR backend per tier** → delegate to `catlx-hardware-adaptation`
  (`skill({ name: "catlx-hardware-adaptation" })`).
- **Where OCR executes + fallback** → delegate to `catlx-capability-routing`
  (`skill({ name: "catlx-capability-routing" })`).
- **Acting on the ScreenModel (click/type)** → delegate to `catlx-desktop-control`
  (`skill({ name: "catlx-desktop-control" })`).
- **Permission for screen capture** → delegate to `catlx-security`
  (`skill({ name: "catlx-security" })`).
- **GPU container (T2+ EasyOCR)** → delegate to `catlx-docker` (`skill({ name: "catlx-docker" })`).

## Edge cases & warnings

- **Static regions:** use the result cache on T1 to avoid wasteful re-OCR.
- **Dynamic content / animation:** re-OCR the affected region; never trust stale ScreenModel state.
- **Full-screen capture cost:** prefer targeted regions; use GPU/container on T2+ for full-screen.
- **Per-app exclusions:** respect the permission router's screen-read exclusions (e.g. password managers).
- Security risk of reading sensitive screen content — audit and gate.

## Component lifecycle policy (reuse → install → adapt → create)

**NEVER create a new component as the default.** Before building/creating anything (a sub-skill, dependency,
reference, workflow, helper, adapter, or template), check, in order:
1. **Reuse** an existing local component (resolve aliases/equivalent capabilities first) — reuse, don't rebuild.
2. **Use** an already-registered component from the registry.
3. **Install** a suitable existing, trusted, supported component → validate → register → connect to the graph → use.
4. **Adapt** an existing compatible component via a small persistent adapter/wrapper instead of re-creating it.
5. **Create only as last resort** — then make it permanent immediately: stable id, canonical location, register,
   add to the capability index + dependency graph, add provenance, use, and allow future reuse.
6. Never reorganise/recreate already-generated components (no `Skill X 2` / `new` / `temp` variants); extend the
   existing one. Never create a second competing knowledge source; connect back to the canonical `knowledge/` layer.
   Promote any reusable artifact out of `/tmp`/scratch into the permanent ecosystem.

> Full policy: `../knowledge/rules/component-lifecycle.md`.

## Source / provenance

- **Source:** PART V §5.5 (OCR — Screen Understanding) as a tier-scaled, post-processed capability; echoed in
  the component tree (PART XVII §17.1: OCR Engine, Screen Understanding Layer).
- **Inferred/adapted:** Windows OCR (Tesseract/EasyOCR/PaddleOCR run on Windows); the ScreenModel structure is
  a faithful representation of the source's elements/state/spatial requirements.