majiayu000 Redaction Tool Development Guide
An intelligent document redaction application with two-pass PII detection—regex patterns for standard formats plus AI semantic analysis for contextual data.
Live site: https://redaction-tool.casedev.app/
Architecture
src/
├── app/
│ ├── page.tsx # Main application UI
│ └── api/
│ ├── detect-pii/ # Two-pass PII detection
│ ├── export-pdf/ # PDF generation
│ ├── detect/ # Database-backed detection
│ ├── export/ # Database-backed export
│ ├── jobs/ # Job management
│ └── upload/ # File upload
├── components/
│ ├── redaction/
│ │ ├── PatternSelector.tsx # Redaction type selection
│ │ ├── EntityList.tsx # Detected entities
│ │ └── DocumentPreview.tsx # Preview with highlights
│ ├── upload/
│ │ └── DropZone.tsx # File upload
│ └── ui/ # shadcn components
└── lib/
├── redaction/
│ ├── detector.ts # Two-pass detection logic
│ └── patterns.ts # Regex patterns & presets
├── case-dev/
│ └── client.ts # Case.dev API client
├── db.ts # Database connection
└── utils.ts # Utilities
Core Workflow
Upload Doc → Extract Text → Pass 1: Regex → Pass 2: AI → Review → Export PDF
↓ ↓ ↓ ↓ ↓ ↓
PDF/TXT pdf.js SSN, CC, Names, Toggle Redacted
images extraction phone, email addresses entities document
patterns context
Tech Stack
| Layer | Technology |
|---|---|
| Frontend | Next.js 14, React, Tailwind CSS |
| UI | shadcn/ui |
| PDF Processing | pdf.js (extract), pdf-lib (generate) |
| AI Detection | Case.dev LLM (GPT-4o) |
| Database | PostgreSQL + Prisma (optional) |
Key Features
| Feature | Description |
|---|---|
| Two-Pass Detection | Regex + AI for comprehensive coverage |
| PII Types | SSN, bank accounts, credit cards, names, addresses, phone, email, DOB |
| Presets | Pre-configured redaction profiles |
| Entity Review | Toggle, edit masked values |
| PDF Export | Generate redacted documents |
| Audit Log | Track what was redacted |
Two-Pass Detection
See references/pii-detection.md for patterns and AI prompts.
Pass 1: Regex (Fast, High-Precision)
- SSN:
XXX-XX-XXXXwith validation - Credit cards: Luhn-valid patterns
- Phone: US formats
- Email: Standard format
- Dates: Common formats
Pass 2: AI/LLM (Semantic, Aggressive)
- Non-standard formats ("SSN: one two three...")
- Contextual references ("my social is...")
- Names and addresses
- OCR errors and typos
- Obfuscated data
Redaction Presets
| Preset | Types Included |
|---|---|
| SSNs and Financial | SSN, Account Numbers, Credit Cards |
| All Personal Information | All PII types |
| Contact Information Only | Phone, Email |
| Financial Only | Account Numbers, Credit Cards |
Case.dev Integration
See references/casedev-redaction-api.md for API patterns.
LLM Detection
const aiEntities = await detectWithLLM(text, piiTypes);
OCR for Images
const text = await extractTextFromImage(imageUrl);
Development
Setup
npm install
cp .env.example .env
# Add CASEDEV_API_KEY
npm run dev
Environment
CASEDEV_API_KEY=sk_case_... # Required
DATABASE_URL=postgresql://... # Optional for job persistence
Database (Optional)
npx prisma migrate dev
API Endpoints
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/detect-pii | Two-pass PII detection |
| POST | /api/export-pdf | Generate redacted PDF |
| POST | /api/upload | File upload |
| GET | /api/jobs | List jobs |
| GET | /api/jobs/:id | Get job status |
Common Tasks
Adding a New PII Type
- Add regex pattern to
lib/redaction/patterns.ts - Add to AI prompt in
lib/redaction/detector.ts - Add UI toggle in
PatternSelector.tsx - Add masking function
Improving AI Detection
Modify the LLM prompt to be more/less aggressive or handle specific formats.
Adding Export Format
- Add generation function in
lib/export/ - Add endpoint in
api/export-[format]/ - Add UI option
Troubleshooting
| Issue | Solution |
|---|---|
| Regex missing PII | Check pattern, add variations |
| AI too aggressive | Adjust prompt confidence |
| PDF export fails | Verify pdf-lib compatibility |
| OCR errors | Use higher quality images |