aibot88 APS Ingestion Documentation Expert
Specialized skill for generating comprehensive documentation for ingestion layers. Automatically detects and documents connector-specific patterns, authentication methods, rate limiting, and incremental strategies.
When to Use This Skill
Use this skill when:
- Documenting a new data source ingestion workflow
- Creating documentation for REST API connectors (Salesforce, HubSpot, etc.)
- Documenting database ingestion (MySQL, PostgreSQL, BigQuery, etc.)
- Documenting file-based ingestion (S3, GCS, SFTP, etc.)
- Documenting streaming ingestion (Kafka, Kinesis, etc.)
- Creating parent-child documentation for multiple data sources
Example requests:
"Document the Klaviyo ingestion workflow"
"Create documentation for Salesforce API ingestion"
"Document all data sources in the ingestion layer"
"Generate ingestion documentation following this template: [Confluence URL]"
🚨 MANDATORY: Codebase Access Required
WITHOUT codebase access = NO documentation. Period.
If no codebase access provided:
I cannot create technical documentation without codebase access.
Required:
- Directory path to ingestion workflows
- Access to .dig, .yml configuration files
Without access, I cannot extract real table names, connectors, or incremental logic.
Provide path: "Code is in /path/to/ingestion/"
Before proceeding:
- Ask for codebase path if not provided
- Use Glob to verify files exist
- STOP if cannot read files
Documentation MUST contain:
- Real connector names from .dig files
- Actual table names from datasources.yml
- Real incremental fields and schedules
- Working examples from actual configs
NO generic placeholders. Only real, extracted data.
Layer-Specific Intelligence
Auto-Detection Capabilities
This skill automatically detects and documents:
1. Connector Type Detection
REST API Connectors:
Detects from configuration:
- endpoint URLs (https://api.example.com/v1/...)
- HTTP methods (GET, POST, PUT)
- Pagination patterns (offset, cursor, page number)
- Response format (JSON, XML)
Documents:
- API endpoint structure
- Request/response examples
- Pagination strategy
- Response handling
Database Connectors:
Detects from configuration:
- JDBC connection strings
- Query-based ingestion patterns
- Incremental query logic
- Connection parameters
Documents:
- Connection configuration
- Source queries
- Data type mappings
- Isolation levels
File-Based Connectors:
Detects from configuration:
- S3/GCS bucket paths
- File patterns (*.csv, *.json, *.parquet)
- Compression formats (gzip, zip, snappy)
- File naming conventions
Documents:
- Bucket/path structure
- File format specifications
- Decompression logic
- File processing order
Streaming Connectors:
Detects from configuration:
- Kafka topics/consumer groups
- Kinesis streams
- Partition strategies
- Offset management
Documents:
- Topic/stream configuration
- Consumer settings
- Checkpoint mechanisms
- Backpressure handling
2. Authentication Pattern Detection
OAuth 2.0:
Detects:
- Token endpoint URLs
- Client ID references
- Scope definitions
- Token refresh logic
Documents (securely):
- Authentication flow
- Token lifecycle
- Scope requirements
- Refresh strategy
(WITHOUT exposing secrets)
API Key Authentication:
Detects:
- API key header names
- Key rotation patterns
- Rate limit tiers
Documents:
- Header configuration
- Key rotation schedule
- Usage tier limits
Basic Authentication:
Detects:
- Username/password references
- Credential storage patterns
Documents:
- Authentication method
- Credential management
Service Account / JWT:
Detects:
- Service account files
- JWT token generation
- Key expiration
Documents:
- Service account setup
- Token generation process
- Key rotation policy
3. Rate Limiting Strategy Detection
Detects from workflow:
- Request throttling (requests per second/minute)
- Retry backoff strategies (exponential, linear)
- Concurrent request limits
- Circuit breaker patterns
Documents:
- Rate limit thresholds
- Backoff algorithm
- Retry configuration
- Concurrent connection limits
4. Incremental Load Pattern Detection
Timestamp-Based:
Detects:
- updated_at, modified_at, created_at fields
- Timestamp comparison logic
- Watermark tracking
Documents:
- Incremental field name
- Timestamp format
- Watermark storage
- Lookback window
Sequence-Based:
Detects:
- Auto-increment ID fields
- Sequence tracking
- Max ID queries
Documents:
- Sequence field name
- High-water mark logic
- Gap handling
Full Reload:
Detects:
- No incremental field
- Full table scans
- Truncate-and-load patterns
Documents:
- Full reload schedule
- Data volume considerations
- Performance impact
REQUIRED Documentation Template
Follow this EXACT structure (analyzed from production examples):
For Parent Ingestion Page:
## Overview
{Brief description of ingestion layer}
### Project Structure
{Directory tree from actual codebase}
## Main Ingestion Runner
**Workflow File**: ingestion_runner.dig
{Schedule, tasks, parallelization}
## Database Configuration
{Table with databases and purposes}
## Monitoring and Logging
{SQL queries for status checks}
## Individual Source Documentation
{Links to child pages}
For Individual Source (Child Page):
# {Source} Ingestion
## Overview
**Workflow Files:**
- {source}_ingest_inc.dig - Incremental
- {source}_ingest_hist.dig - Historical (if exists)
{Description}
**Data Source Type**: {type}
**Connector**: {connector name}
**Source System**: {system}
**Target Database**: {database}
---
## Configuration Files
{Table with file types and purposes}
---
## Active Tables (Incremental)
{Table with all incremental tables from datasources.yml}
## Active Tables (Historical)
{Table with all historical tables - if exists}
---
## Incremental Workflow Process
### Step 1: Log Ingestion Start
{Code snippet from workflow}
### Step 2: Setup Table and Time
{Explain create table logic + get last time logic}
### Step 3: Load Incremental Data
{Code snippet + query example}
### Step 4: Log Ingestion Success
{Code snippet}
---
## Historical Workflow Process
{Similar steps for historical if exists}
---
## Parallelization
{Explain _parallel settings and concurrency}
---
## Error Handling
{_error block from workflows}
---
## Authentication
{td_authentication_id reference}
---
## Data Flow Diagram
{Simple text diagram showing source → target}
---
## Incremental Logic
{Explain first run vs subsequent runs}
---
## Timestamp Format
{Document actual format from configs}
---
## Monitoring and Troubleshooting
{SQL queries for checking status, errors}
---
## Key Features
{Bullet list of main capabilities}
---
## Adding New Tables
{Step-by-step guide with real examples}
---
## Configuration Reference
{Sample datasource config + load config}
---
## Summary
{Brief recap of workflow capabilities}
Summary
This skill generates production-ready ingestion documentation by:
- Reading actual .dig workflows and .yml configs from codebase
- Following the exact template structure shown above
- Extracting real table names, incremental fields, connectors
- Creating comprehensive, accurate documentation with working examples
Key capability: Transforms codebase into professional Confluence documentation.