Post Connection Data Scope Filtering

Choosing a Data-Scope Filtering Strategy

Customers almost never want every record their connected system has. An HRIS customer might want only active employees in specific departments. An ATS customer might want only candidates past a certain pipeline stage. A CRM customer might want only accounts owned by their team. A ticketing customer might want only tickets from one project or above a priority threshold.

Decide the filtering approach before a live customer forces the decision, because changing it later usually requires a re-sync.

Prerequisites

• Initial sync logic working (sync-implement-webhooks and/or sync-implement-polling)

Before Proceeding

Three pieces of information are needed before writing any filter logic.

If invoked from implementing-post-connection, the first two were answered in Step 1 — use that context. Otherwise, gather them now:

• Categories and endpoints: Which Merge categories and which specific endpoints need filtering? (hris, ats, crm, accounting, ticketing) — filters are defined per endpoint, so /crm/v1/accounts and /crm/v1/contacts accept different sets.
• Existing sync logic location: Where is the sync code (webhook handler / polling job)? Filters apply inside it. Search for files that already call /{category}/v1/{model} or use modified_after.
• Filtering strategy preference: pre-storage (Strategy 1) or post-storage (Strategy 2) — see the two-strategies section below. The recommendation is post-storage unless storage cost is a concern.

Ask the user before continuing:

"Strategy 1 (pre-storage) keeps your DB smaller but requires a full re-sync if filter criteria change later. Strategy 2 (post-storage) is more flexible but stores all records. Which would you like — and which categories should I scope the filtering to?"

Wait for the answer before proceeding.

The two strategies

The same trade-off applies regardless of category. Pick once per integration; document the choice.

Strategy 1: Pre-storage filtering (Merge API filters)

Apply filters inside your sync logic before writing records to your DB. Each record is evaluated at ingestion; non-matching records are discarded.

• Pros: smaller DB, cleaner data, fewer records to query downstream.
• Cons: changing filter criteria requires a full re-sync to backfill previously excluded records; more complex sync logic; you can't show customers "what they would have synced" if they widen filters.

Strategy 2: Post-storage filtering (product filters)

Store ALL records from Merge in a staging table (or with an included flag), and apply filters dynamically at query time or via a separate filtering pass.

• Pros: filter criteria can change without a re-sync; easier to iterate in UI; "preview what changes if I widen this filter" becomes feasible.
• Cons: more data stored; DB queries must always carry filter predicates.

Recommendation

Start with post-storage filtering (Strategy 2) unless storage cost is a meaningful concern. It's easier to tighten filters later than to backfill excluded records.

Filtering by category — Merge API parameters

The Merge query params differ per category. The decision framework above is the same; only the filters themselves change. Use these as starting points; consult Merge's category-specific common-model docs for the full set.

Filters are per-endpoint, not per-category. A parameter that exists on one endpoint is usually absent on its siblings — owner_id works on CRM accounts and opportunities but not on contacts; account_id works on contacts and opportunities but not on accounts. An unrecognized query param is ignored rather than rejected, so a typo returns a full unfiltered page and looks like the filter matched everything. Confirm each one against the endpoint's own API reference page before shipping.

HRIS — GET /hris/v1/employees

Example: GET /hris/v1/employees?employment_status=ACTIVE&expand=employments,manager

⚠️ expand=employment (singular) and expand=locations are not valid tokens. The relation is employments, and locations expand as home_location / work_location.

ATS — GET /ats/v1/candidates and GET /ats/v1/applications

Pipeline-stage and recruiter filters live on applications, not candidates. A candidate has no stage of its own.

Example: GET /ats/v1/applications?current_stage_id={uuid}&job_id={uuid}

⚠️ current_stage, credited_to, and job_id are not parameters on /candidates, and the application-side names end in _id. To scope candidates by stage, filter /applications first and collect candidate IDs, or pull ?expand=applications and post-filter on applications[].current_stage.

CRM — GET /crm/v1/accounts, /contacts, /opportunities

Example: GET /crm/v1/opportunities?stage_id={uuid}&owner_id={uuid}

⚠️ There is no stage parameter — it's stage_id, and only on /opportunities. owner_id is not accepted on /contacts; scope contacts by account_id and resolve ownership from the parent account.

Ticketing — GET /ticketing/v1/tickets

Example: GET /ticketing/v1/tickets?collection_ids={uuid}&priority=HIGH

⚠️ There is no project_id parameter on /tickets. Merge exposes a separate Project model, but tickets are scoped by collection_ids. And priority=HIGH matches only HIGH — to get "HIGH or above" you pass priority=HIGH and priority=URGENT as separate calls, or filter after the fetch.

Accounting — GET /accounting/v1/accounts and /transactions

Example: GET /accounting/v1/accounts?classification=ASSET

⚠️ account_type=ASSET does not work. account_type is an untyped passthrough of the provider's label; the ASSET/LIABILITY/EQUITY/EXPENSE/REVENUE enum is classification. And transaction_date_after is a /transactions parameter — on /accounts it is silently ignored.

If you're building a filter UI for customers

The skeleton is identical regardless of category — substitute "employees", "candidates", "accounts", "tickets", or "transactions" as appropriate.

1. Populate filter options — internal API fetching the available filter values (unique departments, stages, owners, projects, statuses, etc.) from your stored data or Merge's API.
2. Store filter selections — backend table keyed by linked account ID.
3. Save filter selections — internal API endpoint to write selections.
4. Apply filters in sync — either as Merge API query params (Strategy 1) or as predicates on your staging table (Strategy 2).
5. Define filter-change behavior — when a customer updates filters:
   • Re-sync newly included records? (Required if Strategy 1.)
   • Mark previously synced but newly excluded records as inactive / hidden?
   • Notify downstream systems of scope change?

Implementation checklist

• Filtering strategy chosen (pre-storage or post-storage) and documented per integration
• Internal API built to populate filter options in product UI
• Backend table stores filter selections per linked account
• Internal API saves filter selections
• Sync logic incorporates stored filters (in webhook handler and/or polling job)
• Filter-change behavior defined (re-sync? mark inactive? notify?)
• Category-appropriate Merge API filters identified and tested

Troubleshooting

SYMPTOM: Customer changed filter criteria; old records that should now be included are missing CAUSE: Pre-storage filtering was used; previously excluded records were never written to your DB. FIX: Trigger a re-sync that uses the updated filter criteria. Or migrate to post-storage filtering so future filter changes don't require re-syncs.

SYMPTOM: DB queries are slow because every query has filter predicates CAUSE: Post-storage filtering is in use without supporting indexes. FIX: Index the columns used in filter predicates (e.g., employment_status, current_stage, owner_id). For high-cardinality filters, consider materialized views.

SYMPTOM: "Active employees" filter syncs employees that the customer's HRIS shows as terminated CAUSE: Merge's employment_status reflects the provider's status field at sync time; some providers update statuses on a delay or have intermediate states (e.g., PENDING_TERMINATION). FIX: Cross-check Merge's employment_status with termination_date (if present) or a provider-specific custom field, then re-evaluate the filter rule.