# Self Learning Anonymizer

> Implements self-learning data anonymization systems that automatically discover sensitive field patterns from large document corpora, generate adaptive anonymization rules, and execute comprehensive data masking while maintaining referential integrity.

- Skill: `playerrch/self-learning-anonymizer` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add playerrch/self-learning-anonymizer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/playerrch/self-learning-anonymizer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: playerrch (https://skillmd.com/u/playerrch)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/playerrch/self-learning-anonymizer

---


# Self-Learning Data Anonymization System

This skill implements a complete self-learning anonymization framework that discovers sensitive data patterns from document collections, learns from the data itself rather than relying on predefined rules, and generates adaptive anonymization strategies.

## When to Use This Skill

Use this skill when:
- You need to anonymize large volumes of documents (10,000+ files)
- The data structure is complex and varies across documents
- Predefined rules are insufficient or unknown
- You need to discover what sensitive fields exist in the data
- Data privacy compliance (GDPR, CCPA, etc.) is required
- You want data-driven anonymization rather than manual rule creation

## Core Methodology

### Phase 1: Pattern Discovery (Self-Learning)

**Key Principle**: Let the data tell you what's sensitive, don't assume.

```python
# Pattern Discovery Engine Structure
class PatternDiscoverer:
    FIELD_TRIGGERS = {
        'organization': ['甲方', '乙方', '委托人', '供应商', ...],
        'contact_person': ['联系人', '负责人', '法定代表人', ...],
        'phone': ['电话', '手机', '传真', ...],
        'address': ['地址', '服务地点', '履行地点', ...],
        'bank': ['开户银行', '银行账号', ...],
        'identifier': ['合同编号', '项目编号', ...],
        'id_code': ['统一社会信用代码', '组织机构代码', ...],
        'date': ['签订日期', '履行期限', ...],
        'amount': ['金额', '合同金额', '总价', ...],
        'email': ['邮箱', '电子邮件', ...],
    }
```

**Critical Success Factor**: Process ALL files, not samples.
- Sampling 2,000 from 29,841 → 17,000 patterns discovered
- Processing all 29,841 → 385,201 patterns discovered (22.6x more)

### Phase 2: Statistical Validation

**Confidence Score Calculation**:
```
confidence = frequency_score + diversity_score + consistency_score

frequency_score = min(occurrences / 500, 0.4)        # Up to 0.4
diversity_score = min(unique_samples / 20, 0.3)     # Up to 0.3  
consistency_score = dominant_type_ratio * 0.3         # Up to 0.3
```

**Threshold**: Only patterns with confidence >= 0.6 become rules

### Phase 3: Rule Generation

**Rule Structure**:
```python
@dataclass
class AnonymizationRule:
    name: str                    # e.g., "organization_甲方"
    category: str                # e.g., "organization"
    priority: int                # 50-200, lower = higher priority
    triggers: List[str]          # Field name patterns
    patterns: List[str]          # Regex patterns to match
    replacement_type: str      # Strategy: 'org', 'person', 'phone', etc.
```

**Priority Order** (Critical for correct processing):
1. Organization names (50) - Longest entities first
2. Person names (60) - Avoid partial matching
3. Phone numbers (100)
4. Addresses (110)
5. Bank info (120)
6. Identifiers (150)
7. Dates/Amounts (200)

### Phase 4: Adaptive Replacement

**Strategy by Category**:

| Category | Replacement Strategy | Example |
|----------|-------------------|---------|
| `organization` | Type-aware aliases | 甲方→某政府机构N, 乙方→某供应商N |
| `person` | Keep surname only | 陈玉英→陈xx |
| `phone` | Mask all digits | 15800845608→xxxxxxxxxxx |
| `address` | Keep keywords | 上海沪太路→xx路 |
| `bank` | Generic aliases | 农行上海支行→某银行xx支行 |
| `identifier` | Preserve structure | 11N78361356620251203→xxXxxxxxxxxxxxxxxxxx |
| `date` | Relative descriptions | 2025-09-30→合同约定期限 |
| `amount` | Range descriptions | 14490元→数万元 |
| `email` | Generic mask | user@domain.com→xxx@xxx.com |

**Mapping Table Management**:
- Maintain `org_map` for consistent organization aliasing
- Maintain `id_map` for consistent identifier masking
- Per-file processing but global mapping consistency

## Implementation Guidelines

### Directory Structure

```
project/
├── anonymizer.py              # Main implementation
├── anonymization_kb.pkl         # Knowledge base (auto-generated)
├── anonymization_report.txt     # Learning report (auto-generated)
├── _anonymization_mapping.json  # Mapping tables (auto-generated)
└── data_anonymized/             # Output directory
```

### Key Code Patterns

**1. Full-File Learning (Not Sampling)**
```python
# ✅ CORRECT: Process all files
for file_path in all_files:  # 29,841 files
    discover_patterns(file_path)

# ❌ INCORRECT: Sampling misses patterns
sample_files = random.sample(all_files, 2000)  # Misses 93% of patterns
```

**2. Multi-Pass Processing**
```python
def process(self, content):
    # Pass 1: Collect all entities first
    entities = []
    for rule in self.rules:
        for match in re.finditer(rule.pattern, content):
            entities.append({
                'start': match.start(),
                'end': match.end(),
                'replacement': generate_replacement(match.group())
            })
    
    # Pass 2: Replace from end to start (avoids position shifts)
    entities.sort(key=lambda x: -x['start'])
    for ent in entities:
        content = content[:ent['start']] + ent['replacement'] + content[ent['end']:]
    
    return content
```

**3. Value Type Classification**
```python
def classify_value(value: str) -> str:
    if re.match(r'^[\d\s\-]{7,20}$', value):
        return 'phone'
    if re.match(r'^1[3-9]\d{9}$', re.sub(r'\D', '', value)):
        return 'mobile'
    if re.match(r'^[A-Z0-9\-]{8,25}$', value):
        return 'code'
    if re.search(r'\d{4}[年.\-/]\d{1,2}[月.\-/]', value):
        return 'date'
    # ... more patterns
```

## Expected Outcomes

### Quantitative Metrics
- **Pattern Discovery**: 20-25x more patterns with full vs sampled learning
- **Rule Generation**: 60+ high-confidence rules from 30,000+ documents
- **Coverage**: 99.9% of files processed successfully
- **Mappings**: 30,000-70,000 entity-to-alias mappings generated

### Qualitative Improvements
1. **Completeness**: No sensitive field types missed
2. **Consistency**: Same entity always gets same alias
3. **Adaptability**: Handles domain-specific fields (中标金额, 施工单位)
4. **Quality**: Higher confidence scores through full data exposure

## Lessons Learned

### 1. Data Volume > Algorithm Complexity
Simple algorithm + complete data > Complex algorithm + sampled data

### 2. Full Data Investment Pays Off
- 31 minutes processing 29,841 files
- Discovered 22.6x more patterns than sampling
- Time investment ROI: 1.45x patterns per minute

### 3. Self-Learning > Predefined Rules
Data-driven discovery finds:
- Domain-specific fields
- Unexpected field formats
- Context-dependent patterns

### 4. Statistical Validation is Critical
Confidence scoring prevents:
- False positives
- Over-fitting to common patterns
- Under-fitting to rare but important fields

## Testing & Validation

### Before Production
1. **Sample Verification**: Check 10 random files manually
2. **Pattern Validation**: Verify top 20 patterns have >0.8 confidence
3. **Mapping Consistency**: Ensure entity aliases are consistent
4. **Edge Case Testing**: Test with malformed/edge-case documents

### Quality Checklist
- [ ] All organization names replaced
- [ ] All phone numbers masked
- [ ] All identifiers structure-preserved but masked
- [ ] No original sensitive data in output
- [ ] Referential integrity maintained
- [ ] Mapping table saved for audit

## Advanced Techniques

### Incremental Learning
```python
# Add new documents to existing knowledge base
def incremental_learn(new_files, existing_kb):
    for file in new_files:
        patterns = discover_patterns(file)
        existing_kb.update(patterns)
    existing_kb.regenerate_rules()
```

### Cross-Validation
Split data into training/validation sets to verify pattern generalizability.

### Confidence Threshold Tuning
Adjust threshold (default 0.6) based on:
- False positive tolerance
- Required coverage level
- Data sensitivity

## Common Pitfalls

### ❌ Pitfall 1: Sampling for Speed
Sampling 6.7% of data misses 93.3% of pattern diversity.

### ❌ Pitfall 2: Single-Pass Replacement
Replacing left-to-right causes position shifts and missed matches.

### ❌ Pitfall 3: Fixed Priority Order
Processing short patterns before long ones causes partial matches.

### ❌ Pitfall 4: No Confidence Threshold
Accepting all patterns leads to false positives.

## References

This methodology was validated on 29,841 Chinese government procurement contracts, discovering 385,201 sensitive field patterns and generating 62 adaptive anonymization rules with >99% coverage.

---

**Remember**: In data anonymization, completeness beats efficiency. Invest the time to process all data - the returns are exponential.