# Text Features

> Turn free-text rows into calibrated numeric features with TypeSafe Jev, then model them on a leakage-safe development split. The main model proposes Noul/Score questions; host.judge answers every row; sklearn in the science extra fits and evaluates. Features keep their source question and measurement error and are not a human gold standard.

- Skill: `pku-yuangroup/text-features` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx get-skill add pku-yuangroup/text-features`
- Raw SKILL.md: https://api.skillmd.com/api/skills/pku-yuangroup/text-features/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: Apache-2.0
- Author: pku-yuangroup (https://skillmd.com/u/pku-yuangroup)
- Updated: 2026-11-11
- Page: https://skillmd.com/skills/pku-yuangroup/text-features

---


# Text features (experimental)

Use this skill when you have many free-text rows and a supervised target, and
you want calibrated numeric features instead of bag-of-words. The main model
proposes operationalizable questions; Jev answers each row; the sidecar fits a
model on a development split and scores the frozen test split once.

This is experimental. Turn on `text_features` under Customize → Experimental
(the master experimental-judgment switch must also be on). Headless: set
`OPENAI4S_EXPERIMENTAL_JUDGMENT=1` and `OPENAI4S_JUDGMENT_TEXT_FEATURES=1`.
When the capability is off, the helpers return `status: "disabled"` and do
not raise.

## When to use it

- Lots of free text (notes, abstracts, reports) plus a label or score.
- You need features a colleague can read: each column is a question, a
  probability or graded expectation, and a spread.
- You can hold out a test set and leave it untouched while questions change.

## When not to use it

- Small samples, or no supervised target — there is nothing to freeze against.
- Sensitive data. Selected row text is sent to `api.typesafe.ai`. The service
  is hosted in the United States. Do not enable this on clinical notes, secrets,
  or anything that must not leave the machine.
- You need a human gold standard. **Jev features are not a human gold
  standard.** They are calibrated model judgments of the text you sent, with
  measurement error. Do not treat a Noul probability as a verified fact.

## Data that leaves the machine

Every `featurize` call sends the user-selected row text in `state.text` to
TypeSafe Jev, together with the question instructions. Identifiers go in
`state.id` so you can audit which rows were judged. Quote location and numeric
modeling stay in this sidecar.

## Import and run

The directory contains a hyphen, so import it with `importlib`:

```python
from importlib import import_module

tf = import_module("text-features.kernel")

questions = tf.propose_questions(
    "Predict whether an abstract reports a significant clinical result.",
    examples,
    n=12,
)
table = tf.featurize(
    rows,
    questions,
    text_field="text",
    id_field="id",
)
study = tf.run_feature_study(
    rows,
    target="label",
    split_by="patient_id",  # or time_col="date"
    rounds=3,
    text_field="text",
    id_field="id",
)
```

`propose_questions` asks `host.llm` for Noul (yes/no facts) and Score
(written-level grades) questions, then validates and deduplicates them.

`featurize` calls `host.judge("features.custom", ...)` once per row. Each Noul
becomes one column, P(yes). Each Score becomes two columns: the expected level
normalized to [0, 1], and the standard deviation of that distribution on the
same scale. Unavailable rows are filled with NaN and counted; they are not
replaced with a default. Every column keeps the question text and template
version.

`run_feature_study` reuses `audit-dataset`, `plan-ml-experiment` (grouped or
chronological split), and `evaluate-model` (metrics and bootstrap 95% CI).
Question edits, feature screening, and thresholds use the development rows
only. The test split is judged once, after the question set is frozen. The
report includes lift versus a constant baseline and a bootstrap interval,
plus the question-set version and per-feature provenance.
Binary targets are encoded in the reported `target_classes` order; predictions
are probabilities of the second class. Rows with missing targets are excluded
from fitting and metrics. Baseline and feature-model lift use the same rows
with available predictions.

Modeling uses numpy / pandas / scikit-learn when the science extra is
installed. They are imported lazily. They are not core dependencies. Without
them the sidecar still featurizes and falls back to a linear least-squares
fit.

## Required output

Name the split, the frozen question-set version, each feature's source
question and template version, unavailable-row counts, cost (requests and
tokens), the baseline, the lift, and the bootstrap interval. Never describe
the features as labels, facts, or a human gold standard.

