Source Management

Search Source Management

Add, configure, monitor, and troubleshoot enterprise search source connectors and indexing pipelines.

Source Management Operations

List Connected Sources

Display all currently configured sources with their status:

# Connected Search Sources

| #  | Source Name       | Type     | Status  | Last Sync          | Items Indexed | Health |
|----|-------------------|----------|---------|--------------------|---------------|--------|
| 1  | [Name]            | [Type]   | Active  | [timestamp]        | [count]       | Good   |
| 2  | [Name]            | [Type]   | Active  | [timestamp]        | [count]       | Warn   |
| 3  | [Name]            | [Type]   | Paused  | [timestamp]        | [count]       | —      |
| 4  | [Name]            | [Type]   | Error   | [timestamp]        | [count]       | Error  |

Legend: Good = syncing normally | Warn = degraded performance | Error = sync failing

Add a New Source

Guided process for connecting a new data source:

Step 1: Identify the Source Type

Step 2: Configure Connection

Required configuration for each source:

Source Configuration:
  name:           [human-readable name for this connection]
  type:           [source type from table above]
  platform:       [specific platform, e.g., "slack"]
  auth_method:    [OAuth, API Key, PAT, etc.]
  credentials:    [securely stored reference — never display in plain text]
  base_url:       [API endpoint if custom]
  
Indexing Configuration:
  sync_frequency: [realtime | 5min | 15min | 1hour | 6hours | daily]
  scope:          [what to index — channels, folders, spaces, projects]
  content_types:  [messages, files, pages, comments, attachments]
  date_range:     [how far back to index — 30d | 90d | 1y | all]
  max_items:      [optional cap on indexed items]

Access Configuration:
  respect_source_permissions: [true/false — recommended: true]
  allowed_users:              [user list or "all"]
  allowed_groups:             [group list or "all"]

Step 3: Set Indexing Scope

Define what content gets indexed:

Step 4: Test Connection

Run a connection test before activating:

Connection Test Results: [Source Name]
  Authentication:  [PASS/FAIL] — [details]
  API Access:      [PASS/FAIL] — [details]
  Permissions:     [PASS/FAIL] — [details]
  Sample Query:    [PASS/FAIL] — returned [N] results
  Index Write:     [PASS/FAIL] — [details]
  
  Overall: [READY / NOT READY]
  [If NOT READY: specific remediation steps]

Remove a Source

Checklist before removal:

• Confirm the source name with the user
• Warn about data that will be removed from the index
• Check if any saved searches or digests depend on this source
• Offer to export indexed metadata before removal
• Remove index data after confirmation
• Update any dependent configurations (digests, alerts, saved searches)

Source Health Monitoring

Health Check Dashboard

# Source Health Report — [Date]

## Summary
Total sources: [N] | Healthy: [N] | Degraded: [N] | Failed: [N]

## Detailed Status

### [Source Name] — [Status Emoji] [Healthy/Degraded/Failed]
  Last successful sync: [timestamp]
  Sync frequency:       [configured frequency]
  Items indexed:        [total] ([+N new] since last sync)
  Average sync time:    [duration]
  Error rate:           [percentage over last 24h]
  Quota usage:          [X/Y API calls used]
  
  Recent issues:
  - [timestamp] [issue description]

Common Health Issues and Remediation

Configure Indexing Settings

Modify settings for an existing source:

# Indexing Configuration: [Source Name]

Current Settings → Proposed Changes:
  sync_frequency: 1hour → [new value]
  scope:          #general,#engineering → [new value]
  content_types:  messages,files → [new value]
  date_range:     90d → [new value]

Impact Assessment:
  Estimated index size change: [+/- N items]
  Estimated sync time change:  [+/- duration]
  API quota impact:            [+/- N calls per day]

Proceed? [Confirm required]

Security and Compliance

Permission Model

Compliance Checklist

• All sources configured with least-privilege access
• Source permissions are respected in search results
• Audit logging is enabled for all queries
• Sensitive content patterns are configured for detection
• Data retention policies align with company policy
• Credentials are stored in a secure vault, never in plain text
• Regular access reviews are scheduled

Edge Cases

• Duplicate sources: Warn if a source of the same type and scope is already connected
• Overlapping scope: If two connectors index the same content, flag potential duplicates
• Large initial sync: For sources with extensive history, recommend starting with a short date range and expanding
• Source deprecation: If a platform announces API deprecation, alert the user proactively
• Partial permissions: If the connector has access to only some channels/folders, clearly document what is and is not indexed
• Free tier limits: Track API quotas and warn before reaching limits

Follow-Up Actions

After any source management operation, offer:

1. "Want me to run a test search to verify the source is working?"
2. "Should I check the health of all other connected sources?"
3. "Want me to update your digest configuration to include this source?"
4. "Should I trigger an initial sync now or wait for the scheduled time?"