Sift Market Research

Sift Market Research

Automate market research using Sift's Market Finder combined with public data sources to produce comprehensive, actionable market analysis for real estate investors.

Mandatory Output Requirements

All data outputs MUST include a properly formatted Excel spreadsheet (.xlsx). Never use plain text files (.txt) or poorly formatted data dumps.

Output Rules:

1. All tabular data MUST be in Excel spreadsheet format
2. Use the template at templates/MarketFinderResearchExample.xlsx as the structural guide
3. Include proper column headers, formatting, and multiple worksheets as needed
4. Pairing a Markdown report with the Excel spreadsheet is encouraged for comprehensive analysis
5. NEVER output raw text files or unformatted data dumps as standalone deliverables

Acceptable Output Combinations:

• Excel spreadsheet only (for quick research)
• Excel spreadsheet + Markdown report (for comprehensive analysis)
• Excel spreadsheet + PDF summary (when requested)

NOT Acceptable:

• Text files (.txt) containing data tables
• Markdown-only outputs with no accompanying spreadsheet
• Unformatted data dumps

Credentials

Use your DataSift account credentials (email and password from app.reisift.io). Never hardcode credentials in skill files, prompts, or shared documents.

Login URL: https://app.reisift.io/

Execution Mode Detection

Before starting, detect the execution environment to determine whether automation is available:

Check 1: Does scripts/extract_market_finder.py exist in this skill's directory? Check 2: Is Playwright available? Run: python -c "from playwright.sync_api import sync_playwright; print('OK')" Check 3: Are credentials set? Check for DATASIFT_EMAIL and DATASIFT_PASSWORD in .env or environment variables.

Automated Mode (Claude Code CLI — all 3 checks pass)

Run the extraction script directly:

# Single county
python scripts/extract_market_finder.py --state "Tennessee" --county "Knox"

# Multiple counties
python scripts/extract_market_finder.py --state "Tennessee" --county "Knox,Blount"

# Headless mode (no visible browser)
python scripts/extract_market_finder.py --state "Tennessee" --county "Knox" --headless

The script handles login, Market Finder navigation, data extraction (ZIP codes + neighborhoods + summary panel), and outputs a JSON file. Then generate the Excel report:

python scripts/generate_market_excel.py --input output/market_finder_Tennessee_Knox_*.json --fred-dom 71

After automated extraction, skip directly to the Excel generation and analysis steps (Step 7). The JSON output contains all data that Steps 2-6 would produce manually.

Manual Mode (Co-Work or no Playwright — any check fails)

Follow Steps 2-6 below manually. Claude will guide you through the DataSift UI step by step. You can still use scripts/generate_market_excel.py for Excel generation if you format your extracted data as JSON.

Workflow Overview

1. Determine analysis type (Quick Research vs. Comprehensive Analysis)
2. Get FRED baseline for Days on Market
3. Login to Sift and extract Market Map data
4. For Comprehensive Analysis: gather public data and blend with Market Map data
5. Generate output (spreadsheet or full market report)

Quick Research? -> Follow Steps 1-7 below for Sift data extraction only Comprehensive Analysis? -> Follow all steps, then see references/ComprehensiveAnalysisFramework.md

Step 1: Get FRED National Baseline

Before analyzing markets, get the national Days on Market baseline from FRED.

CRITICAL: Verify the current value before every analysis.

1. Navigate to https://fred.stlouisfed.org/series/MEDDAYONMARUS
2. Record the most recent monthly value displayed on the chart (look at the rightmost data point)
3. Note the date of the most recent observation (e.g., "Dec 2025: 71 days")
4. Use this exact value as your national baseline -- do NOT use an estimate

Why this matters: The national median DOM fluctuates monthly (it has ranged from 30 days to 90+ days in recent years). Using a stale estimate produces incorrect market comparisons.

• Markets 15-20% below the FRED baseline indicate faster-moving markets
• Markets at or near the baseline are average
• Markets 50%+ above the baseline are slow-moving

Step 2: Login to Sift

1. Navigate to https://app.reisift.io/
2. Enter your DataSift email and password
3. Check the "I agree to the Terms of Use" checkbox
4. Click "Sign In" button
5. Dismiss any notification popups that appear
6. Wait for dashboard to load

Step 3: Navigate to Market Finder

1. From dashboard, click "Market Finder" in the left sidebar navigation
2. The interface shows:
   • Top filters: Select States -> Select Counties -> ZIP Codes/Neighborhoods toggle -> Select ZIP Codes -> Property Type
   • Heat map: Interactive US map with color-coded investor transaction density
   • Legend: Mo. Investor Transactions dropdown (can switch to Homes Sold, Median Sale Price, Homes on Market, Days on Market)
   • Data table: State/County/Zip level data with columns for investor transactions, homes on market, homes sold, days on market, median values
   • Right panel: Summary cards (Median Home Value, Homes on Market, Mo. Investor Transactions, Homes Sold Last Month) plus Home Prices distribution, Homeownership Rate, Market Rent, Gross Rental Yield, Property Types, Bedrooms, Year Built

Navigation Flow:

• Default view shows all US states ranked by investor transactions
• Click "Select States" dropdown -> select target state -> view updates to show counties
• Click "Select Counties" dropdown -> select target county -> view updates to show zip codes
• ZIP Codes toggle switches between ZIP Codes and Neighborhoods view

ZIP Code vs. Neighborhood View: When to Use Each

Decision guide:

• Start with ZIP Code view to identify which ZIP codes have the most investor activity, fastest DOM, and best price points
• Switch to Neighborhood view after you have identified your top 5-10 target ZIPs -- this reveals block-level variation within a ZIP
• Neighborhood view is especially useful for comp analysis (finding which sub-areas within a ZIP have the tightest spreads)
• Some ZIP codes contain neighborhoods with vastly different investor activity levels -- the neighborhood view exposes this

Step 4: State-Level Analysis

The default view shows states sorted by "Total Inv. Trans. for 6 Mo." (investor transactions).

Top States by Investor Activity (as of Jan 2026):

Green Light Criteria:

• Days on Market at or 15-20% below FRED baseline
• 500+ investor transactions in 6 months
• Reasonable inventory levels

Red Flags:

• Days on Market 50%+ above national average
• Less than 10 investor transactions in 6 months

To analyze a state:

1. Click the "Select States" dropdown
2. Select your target state from the alphabetical list
3. The map zooms to show county-level data
4. Data table updates to show counties ranked by investor transactions

Step 5: County-Level Analysis

After selecting a state, the data table shows counties with these columns:

• County: County name
• Total Inv. Trans. for 6 Mo.: 6-month investor transaction count
• Homes on Market: Current inventory
• Homes Sold Last Month: Recent sales volume
• Median Days on Market: Average time to sell
• Median Home Value: Current values
• Median Sale Price: Recent sale prices

Example - Tennessee Counties:

Evaluation Criteria:

Calculate Supply (Months of Inventory):

Supply = Homes on Market / Homes Sold Last Month

Formula explained: This tells you how many months it would take to sell all current inventory at the current sales pace. It is the single most important indicator of whether you are in a buyer's or seller's market.

Example calculation:

• Knox County: 2,700 homes on market / 668 homes sold last month = 4.0 months (balanced market)

To drill down to zip codes:

1. Click the "Select Counties" dropdown
2. Select your target county (format: "County Name STATE")
3. The map zooms to show zip code boundaries
4. Data table updates to show zip codes ranked by investor transactions

Step 6: Zip Code & Neighborhood Data Extraction

After selecting a county, the data table shows either ZIP codes or Neighborhoods (toggle between views).

CRITICAL: Exact Table Column Headers

The Market Finder displays data with these exact column headers. Always capture data using these literal column names:

ZIP Code View Columns (in order):

Neighborhood View Columns (in order):

Pagination and Large Data Sets

IMPORTANT: Some markets have 100+ ZIP codes. The data table may not display all rows at once.

1. After the table loads, scroll to the bottom of the visible rows
2. Wait 1-2 seconds for the next batch to load (DataSift uses lazy loading / infinite scroll)
3. Continue scrolling until no new rows appear
4. Verify your total row count matches the county's known ZIP code count
5. If you see a "Load More" button or similar control, click it and wait for data to load before continuing

How to verify completeness: The right panel's summary statistics (e.g., "Mo. Investor Transactions: 53") should roughly correspond to the sum of the per-ZIP values in your extracted table. Large discrepancies indicate missing rows.

Reference Data: Knox County, TN ZIP Codes

This is the exact data as displayed in Sift Market Finder for Knox County, TN (ZIP Code view). Use this as a reference for data accuracy validation:

Reference Data: Knox County, TN Neighborhoods

This is the exact data as displayed in Sift Market Finder for Knox County, TN (Neighborhood view). Use this as a reference for data accuracy validation:

Data Extraction Rules

1. Capture ALL visible rows - Scroll through the entire table to capture all data (see Pagination section above)
2. Use EXACT column headers - Do not rename or abbreviate columns
3. Preserve data types - Keep currency formatting with $ and commas
4. Capture BOTH views - Extract ZIP Code AND Neighborhood data when available
5. Note the sort order - Default sort is by "TOTAL INV. TRANS. FOR 6 MO." descending

Right Panel Summary Data

Also capture the summary cards from the right panel:

• Median Home Value: e.g., $364.4K
• Homes on Market: e.g., 2.8K
• Mo. Investor Transactions: e.g., 53
• Homes Sold Last Month: e.g., 668
• Market Rent: e.g., $2,107/mo
• Gross Rental Yield: e.g., 7.07%

Property Characteristics Button: Click "Calculate recommended property characteristics" button to get AI-recommended property filters for the selected area.

Step 7: Create Excel Spreadsheet Output

ALWAYS create an Excel workbook (.xlsx) with 7 properly formatted worksheets.

The output spreadsheet should be a complete, self-contained market report that an investor can open and immediately understand the opportunity. The 7-sheet structure mirrors a professional research deliverable: start with the executive summary for quick decision-making, then let the reader drill into ZIP/neighborhood data, economic fundamentals, safety, and actionable recommendations.

Use Python with openpyxl to create the spreadsheet:

import openpyxl
from openpyxl.styles import Font, Alignment, Border, Side, PatternFill
from datetime import datetime

wb = openpyxl.Workbook()

# Style definitions
header_font = Font(bold=True, color="FFFFFF")
header_fill = PatternFill(start_color="4472C4", end_color="4472C4", fill_type="solid")
border = Border(
    left=Side(style='thin'),
    right=Side(style='thin'),
    top=Side(style='thin'),
    bottom=Side(style='thin')
)

Conditional Formatting Rules

Apply these color fills to data cells throughout the workbook (Sheets 2, 3, and 6 especially) for instant visual scanning:

Required Worksheets (7 Sheets)

The workbook must contain exactly these 7 sheets, in this order. Each sheet serves a distinct purpose in telling the market story.

Sheet 1: Executive Summary

This is the first thing the investor sees. It should answer: "Is this market worth my time, and where specifically should I focus?" in under 60 seconds of reading.

Structure:

Row 1: Report title -- [COUNTY], [STATE] - MARKET RESEARCH REPORT (merged across columns A-E) Row 2: Generation date Row 3: Data source attribution -- Data Source: REI Sift Market Finder + Public Data Sources

Section A: County Overview (starts row 5) A table with columns: Metric | Value | Notes

Include these metrics (pulled from Sift right panel + public sources):

Section B: Market Assessment (starts ~row 19) A qualitative ratings table with columns: Category | Rating | Commentary

Rate each category using the criteria defined below:

Additional assessment categories (use STRONG/STABLE/IMPROVING/DECLINING as appropriate):

The commentary column should be a brief, specific explanation -- not generic filler. For example: 53 monthly transactions, strong wholesaling market rather than Good activity level.

Section C: Top 5 ZIP Codes for Wholesaling (starts ~row 28) Columns: Rank | ZIP Code | 6-Mo Inv Trans | Median Home Value | Days on Market

Pull the top 5 ZIPs sorted by investor transactions.

Section D: Top 5 Neighborhoods for Wholesaling (starts ~row 36) Columns: Rank | Neighborhood | 6-Mo Inv Trans | Median Home Value | Days on Market

Pull the top 5 neighborhoods sorted by investor transactions.

Sheet 2: ZIP Code Analysis

This is the deep-dive data sheet. Every ZIP code in the county gets a row, sorted by investor transactions descending. The calculated columns (DOM vs National, Spread %, Supply Months, Wholesaling Score) are what turn raw Sift data into actionable intelligence.

Columns (in order):

Wholesaling Score Methodology:

The star rating is a composite score that answers: "How attractive is this ZIP code for a wholesaler?" It weighs three factors:

1. Investor Activity (primary signal) -- Higher 6-month investor transactions = more proven buyer demand
2. Market Velocity (secondary) -- DOM below national average = properties move quickly, reducing hold risk
3. Price Accessibility (tertiary) -- Lower median home values = smaller capital requirements and broader buyer pool

Scoring rubric:

Use judgment -- these thresholds are guidelines. A ZIP with 35 investor transactions but $600K+ median values might be ★★★☆☆ because the price point limits the wholesale buyer pool, even though volume is strong.

Formatting notes:

• Spread % should show sign: -12.5% or +11.5%
• DOM vs National should show sign: -19 or +34
• Supply Months to 1 decimal: 4.4
• Star ratings as text: ★★★★★, ★★★★☆, etc.
• Apply conditional formatting per the rules table above

Sheet 3: Neighborhood Analysis

Identical structure to Sheet 2, but for neighborhoods instead of ZIP codes. Same columns, same calculated fields, same scoring methodology.

Include ALL neighborhoods from Sift (typically 20-30 rows), sorted by investor transactions descending.

Sheet 4: Economic Indicators

This sheet provides the macroeconomic context that explains WHY the market behaves as it does. An investor needs to know: Is the local economy healthy? Are jobs growing? Is the population stable or growing?

Organize into 3 sections:

Section A: Employment Data (BLS) Pull from BLS (bls.gov/eag) for the county's MSA: Columns: Metric | Value | Trend

Employment by Sector sub-table: Columns: Sector | Jobs (000s) | 12-Mo Change

List all major BLS sectors (Education & Health, Trade/Transport/Utilities, Professional Services, Government, Leisure & Hospitality, Manufacturing, Financial Activities, Mining/Logging/Construction, Other Services, Information). Include the 12-month percentage change for each.

Section B: Demographic Data (Census) Columns: Metric | Value | Notes

Pull from Census QuickFacts (census.gov/quickfacts):

Section C: Housing Market (Redfin) Columns: Metric | Value | YoY Change

Pull from Redfin county page:

Sheet 5: Crime & Safety

Crime data matters for investors because it directly impacts property values, insurance costs, and buyer pool. This sheet should make clear: Is the area getting safer or more dangerous? Which specific areas are higher risk?

Section A: Crime Statistics Columns: Crime Type | Prior Year | Current Year | Change

Pull from local police department or FBI UCR data. Include:

• Murders
• Non-Fatal Shootings
• Robberies
• Motor Vehicle Thefts
• Car Burglaries
• Aggravated Assaults

Format the Change column as percentage: -26%, +2%

Section B: Historical Murder Trend Columns: Year | Murders | Notes

Show 3-4 years of trend data. Murders are the most reliable crime stat (hardest to reclassify) and the strongest signal of neighborhood safety trajectory.

Section C: Safety Assessment by Area Columns: Area | Safety Rating | Notes

Rate major sub-areas within the county (e.g., West Knoxville, North Knoxville, East Knoxville, Downtown, suburban areas) as High, Moderate, or Lower. Include brief context like Lower crime, higher home values or Higher crime, but improving.

Section D: Key Insights for Investors Bullet-point rows (single column, merged) with 4-5 actionable observations connecting crime data to investment decisions. Examples:

• Overall crime trending down significantly - positive for property values
• East Knoxville has higher crime but also higher investor activity
• Consider crime trends when evaluating neighborhood investments

Sheet 6: Investment Recommendations

This is the action sheet -- it translates all the data into specific, prioritized recommendations. An investor should be able to read this sheet alone and know exactly where to focus their marketing.

Section A: Tier 1 - Highest Priority ZIP Codes Columns: ZIP Code | Inv Trans | Median Value | DOM | Supply Months | Rationale

Include 4-6 top ZIPs. The Rationale column should be a concise sentence explaining WHY this ZIP is Tier 1: Highest activity, below-avg DOM, moderate prices

Section B: Tier 1 - Highest Priority Neighborhoods Columns: Neighborhood | Inv Trans | Median Value | DOM | Supply Months | Rationale

Include 4-6 top neighborhoods with specific rationale for each.

Section C: Tier 2 - Secondary Opportunities Columns: ZIP Code | Inv Trans | Median Value | DOM | Supply Months | Rationale

Include 3-5 ZIPs that have merit but with caveats (higher prices, slower DOM, etc.).

Section D: Market Timing Considerations Columns: Factor | Current Status | Implication

This helps investors understand whether NOW is a good time to enter the market:

Section E: Recommended Strategy Numbered rows (single column) with 5-7 specific, actionable steps. These should reference specific ZIP codes, neighborhoods, price ranges, and DOM thresholds from the data. Example:

1. Focus marketing on 37920, 37914, 37917, 37921 ZIP codes
2. Target Colonial Village, Lonsdale, Vestal neighborhoods
3. Look for properties with 60+ DOM for motivated sellers
4. Target homes in $200K-$350K range for best wholesale margins

Sheet 7: Data Sources

Transparency matters. This sheet lets the investor (or their team) verify any number in the report and understand the methodology behind calculated fields.

Section A: Primary Data Sources Columns: Source | Data Type | Date Retrieved | URL/Notes

List every source used:

Section B: Methodology Columns: Analysis Component | Description

Explain each calculated field:

Section C: Disclaimers Bullet-point rows with standard disclaimers:

• Data is current as of retrieval date and subject to change
• REI Sift data represents proprietary investor transaction tracking
• Crime statistics may be preliminary and pending audit
• Investment recommendations are for informational purposes only
• Always conduct independent due diligence before investing

Formatting Requirements

1. Headers: Bold, white text on blue background (#4472C4)
2. Section headers: Bold, larger font, merged across columns where appropriate
3. Borders: Thin borders on all data cells
4. Number Formatting: Currency for prices ($XXX,XXX), 1 decimal for calculations, percentages with sign
5. Column Widths: Auto-fit to content (minimum 12 for data columns, wider for text)
6. Freeze Panes: Freeze header row on data-heavy sheets (Sheets 2, 3, 4)
7. Star ratings: Use Unicode star characters: ★ (filled) and ☆ (empty)
8. Conditional formatting: Apply the color rules from the Conditional Formatting Rules table above

Zip Code Scoring Weights (Source Code Reference)

These weights are used in the SiftStack market_analyzer.py scoring engine. They are verified against source code and should not be modified without updating the codebase.

Grade thresholds (from normalized 0-100 composite score):

• A: Score >= 75
• B: Score >= 55
• C: Score >= 35
• D: Score < 35

Decision Framework

Zip Code Selection Criteria

Tier 1 (Best):

• 10+ monthly investor transactions
• Supply 3-6 months
• Days on Market below national average

Tier 2 (Good):

• 5-10 monthly investor transactions
• Supply 2-7 months
• Days on Market near national average

Tier 3 (Requires Closing Ability):

• Under 5 monthly investor transactions
• Uncontested but requires flip/whole-tail capability

How Many Zip Codes to Target

Output Filename Convention

Required format: [County]_[State]_Market_Research.xlsx

Examples:

• Knox_County_TN_Market_Research.xlsx
• Shelby_County_TN_Market_Research.xlsx
• Fulton_County_GA_Market_Research.xlsx

Comprehensive Market Analysis

For detailed county-level analysis that blends Sift data with public sources, see:

• references/ComprehensiveAnalysisFramework.md - Full analysis structure and methodology

This framework produces:

• Excel Workbook: All quantitative data in properly formatted spreadsheets
• Markdown Report: Narrative analysis covering 12 core indicator categories

Comprehensive Analysis Output Structure:

Public Data Sources to Incorporate:

Reference Files

• references/ComprehensiveAnalysisFramework.md - Detailed analysis structure for blending quantitative data
• references/MarketAnalysisPrompt.pdf - Original methodology document
• references/MarketFinderNavigationAndAnalysisSOP.pdf - Detailed SOP with screenshots
• templates/MarketFinderResearchExample.xlsx - Example output format (use as structural template for spreadsheet outputs)