micsapp Clay Performance Tuning
Overview
Optimize Clay data enrichment throughput and reduce table processing times. Clay tables process enrichments row-by-row with each enrichment making external API calls (Clearbit, Apollo, Hunter, etc.).
Prerequisites
- Clay account with active tables
- Understanding of enrichment provider rate limits
- API access for programmatic table management
Instructions
Step 1: Optimize Table Structure for Throughput
# Design tables for efficient processing
best_practices:
pre_filter_rows: true
# Remove invalid rows BEFORE enrichment (saves time and credits)
# Filter: valid domains, non-personal emails, non-duplicate entries
order_enrichments_by_speed: true
# Fast enrichments first (email lookup ~100ms)
# Slow enrichments last (company profile ~2-5s)
# Allows fast columns to complete while slow ones process
limit_waterfall_depth: 2
# Each waterfall step adds 1-5s per row
# 5-step waterfall = 5-25s per row
# 2-step waterfall = 2-10s per row
Step 2: Use the Map Endpoint for Parallel Lookups
set -euo pipefail
# Instead of enriching row-by-row via the UI:
# Use the API to submit bulk enrichment requests
curl -X POST "https://api.clay.com/v1/tables/tbl_abc123/enrich-batch" \
-H "Authorization: Bearer $CLAY_API_KEY" \
-d '{
"rows": [
{"domain": "stripe.com"},
{"domain": "linear.app"},
{"domain": "vercel.com"}
],
"enrichments": ["company_profile"],
"async": true
}'
# Batch API processes in parallel internally
Step 3: Pre-Validate Input Data
// Clean data before submitting to Clay to avoid wasted processing time
function preValidateRows(rows: any[]): any[] {
return rows.filter(row => {
// Remove rows with invalid domains
if (!row.domain || !row.domain.includes('.')) return false;
// Remove personal email domains (won't enrich company data)
const personalDomains = ['gmail.com', 'yahoo.com', 'hotmail.com'];
if (personalDomains.includes(row.domain)) return false;
// Remove duplicates
return true;
});
}
// Typical filtering removes 20-40% of rows, directly speeding up processing
Step 4: Schedule Large Tables for Off-Peak Hours
# Clay's enrichment providers have variable response times
# Processing during off-peak hours (US nighttime) often sees faster provider responses
scheduling:
large_tables: # 1000+ rows
preferred_time: "02:00-06:00 UTC"
reason: "Less contention on enrichment provider APIs"
small_tables: # <100 rows
time: "Any time"
reason: "Minimal impact from rate limits"
Step 5: Monitor Processing Progress
set -euo pipefail
# Track table enrichment progress
curl "https://api.clay.com/v1/tables/tbl_abc123/status" \
-H "Authorization: Bearer $CLAY_API_KEY" | \
jq '{
total_rows, rows_enriched, rows_pending, rows_failed,
progress_pct: (.rows_enriched / .total_rows * 100),
estimated_remaining_min: (.rows_pending * 3 / 60) # ~3s per row average
}'
Error Handling
| Issue | Cause | Solution |
|---|---|---|
| Table processing stuck | Enrichment provider rate limit | Wait for rate limit reset, or reduce table concurrency |
| Slow enrichment (>10s/row) | Waterfall trying many providers | Reduce waterfall depth to 2 steps |
| Batch fails halfway | Network timeout | Use async batch endpoint, check status via polling |
| Duplicate results | Same domain enriched multiple times | Deduplicate input data before enrichment |
Examples
Basic usage: Apply clay performance tuning to a standard project setup with default configuration options.
Advanced scenario: Customize clay performance tuning for production environments with multiple constraints and team-specific requirements.
Output
- Configuration files or code changes applied to the project
- Validation report confirming correct implementation
- Summary of changes made and their rationale
Resources
- Official ORM documentation
- Community best practices and patterns
- Related skills in this plugin pack