Data-Normalization

Skill: Data Normalization (Artledger)

Purpose

Standardizes raw incoming data from DB1 before attempting enrichment or entity resolution. Bad normalization causes false identity mismatches; this skill ensures consistent formatting for names, wallets, dates, and identifiers across all data sources (blockchains, marketplaces, auction houses).

System Instructions

You are an OpenClaw agent equipped with the Data Normalization protocol. Adhere to the following rules strictly:

1. Trigger Condition: Activate this skill whenever you are preparing raw DB1 records for enrichment. This runs before entity-resolution and before any DB2 write. It is the first step in the enrichment pipeline.

2. Normalization Logic:

   • Artist Name Normalization:

     1. Strip all leading/trailing whitespace.
     2. Collapse multiple internal spaces to single spaces.
     3. Convert to Title Case for comparison purposes only (preserve original casing in storage).
     4. Remove common suffixes that are not part of the name: "NFT Artist", "Crypto Artist", "Digital Artist", "Official", "Art".
     5. Flag Handles: If name starts with @ or contains underscores/numbers typical of handles, flag as is_handle.
     6. Fallback: If name field is empty/null, check alias, username, and bio fields in that order and use the first non-null value found.
     7. Common Name Flag: If name appears in the top 10,000 most common English names with no other distinguishing signals, flag as common_name (requires extra evidence for entity resolution).

   • Wallet Address Normalization:

     1. Ethereum: Convert to lowercase checksummed format.
     2. Tezos: Preserve tz1/tz2/tz3/KT1 prefix, lowercase the remainder.
     3. Solana: Preserve case exactly (Solana addresses are case sensitive).
     4. Bitcoin Ordinals: Normalize to lowercase.
     5. Cleanup: If a wallet address contains spaces or line breaks, strip them.
     6. Validation: Flag any wallet address that does not match the expected format for its declared chain.
     7. Critical: Never attempt entity resolution on a malformed wallet address — flag and skip.

   • Date & Timestamp Normalization:

     1. Convert all dates to ISO 8601 format: YYYY-MM-DDTHH:MM:SSZ.
     2. If timezone is missing, assume UTC.
     3. If only a date is provided with no time, store as YYYY-MM-DDT00:00:00Z.
     4. Sanity Checks: Flag any date before 2017-01-01 for auction records and before 2020-01-01 for NFT records. Flag any date in the future as a data error.

   • Price & Currency Normalization:

     1. Store all prices in two fields: original_amount + original_currency AND usd_equivalent at time of sale.
     2. Crypto: Record the token amount and token type separately. Never convert crypto to USD yourself — flag for a separate price conversion process.
     3. Auctions: Distinguish between hammer_price and buyers_premium. Store both separately.
     4. Nulls: Null prices are acceptable — do not invent or estimate prices.
     5. Zero: Flag any price that is zero for a completed sale as a potential data error.

   • Social Handle Normalization:

     1. Strip @ symbol from the beginning of handles before storing.
     2. Convert to lowercase for comparison purposes.
     3. Remove trailing slashes from URLs.
     4. Platform Cleanup: Normalize platform URLs to handle only: strip https://twitter.com/ and https://x.com/ leaving just the handle.
     5. Validation: Flag handles that contain spaces as malformed.

   • Chain/Source Identifier Normalization:

     1. Valid Values: ethereum, tezos, solana, bitcoin, avax, artsy, christies, sothebys, phillips.
     2. Mapping: eth → ethereum, tez → tezos, sol → solana, btc → bitcoin.
     3. Unknowns: Any unrecognized chain value must be flagged and not processed until resolved. Never assume a chain — if it is not declared and cannot be inferred from wallet format, flag as unknown.

3. Output Format: For every record processed, produce a normalization report containing:

   • fields_normalized: List of fields that were changed.
   • fields_flagged: List of fields with data quality issues and reason.
   • fields_skipped: List of fields that were null and could not be resolved.
   • normalization_passed: Boolean (true if record is safe to pass to entity-resolution).

4. Guardrails:

   • Failed Normalization: If normalization_passed is false, do not pass the record to entity-resolution. Log it in a normalization_errors table and move on.
   • Non-Destructive: Never discard original raw values — normalization produces cleaned copies, it does not overwrite DB1 source data.
   • Low Quality: If more than 30% of fields in a record are flagged, mark the entire record as low_quality and reduce its weight in any downstream scoring.