Collaborative Research

Collaborative Research Workflow

Core Principle: Divide research into Lucky (data gathering) + Jinx (analysis) for maximum efficiency and parallel processing.

When to Use This Skill

✅ Perfect for:

• Market research (competitor analysis, pricing)
• API documentation review
• Trend analysis (Google Trends, marketplaces)
• Technical documentation analysis
• Large-scale content analysis
• Multi-source data comparison

❌ Not suitable for:

• Simple lookups (use direct web_search/web_fetch)
• Real-time data that changes quickly
• Single-page analysis (not worth the overhead)

The 3-Phase Process

Phase 1: Raw Data Gathering (Lucky)

Time: 30-60% of total project time
Focus: Speed and coverage, not precision

1. Set up data directory structure

   mkdir -p /workspace/research/raw-data/YYYY-MM-DD-project
   

2. Use Puppeteer for systematic data collection

   • Navigate to target sites
   • Capture BOTH html and text: { html: document.body.innerHTML, text: document.body.innerText }
   • Save with metadata: URL, timestamp, query/source
   • Don't fight DOM selectors — capture everything

3. Save structured files for Jinx

   METADATA:
   URL: [source_url]
   TIMESTAMP: [iso_timestamp] 
   QUERY: [search_query]
   
   RAW TEXT:
   [page_text_content]
   
   RAW HTML:
   [full_html_content]
   

4. Transfer to Mac Mini SSD

   scp -i ~/.ssh/lucky_to_mac file.html luckyai@100.90.7.148:~/temp/
   ssh -i ~/.ssh/lucky_to_mac luckyai@100.90.7.148 "mv ~/temp/* '/Volumes/Crucial X10/research/raw-data/project/'"
   

Phase 2: Parallel Analysis (Jinx)

Time: 20-40% of total project time
Focus: Pattern extraction and structured output

1. Task Assignment Validation

   • ✅ Analyzing local files (no internet needed)
   • ✅ Structured data processing
   • ✅ Text analysis and extraction

2. Send structured analysis tasks to Jinx

   curl -X POST http://localhost:3001/task -H 'Content-Type: application/json' -d '{
     "prompt": "Analyze files in /Volumes/Crucial X10/research/raw-data/project/. Extract: [specific_data_points]. Output structured JSON with [required_format]. Provide analysis summary with [specific_insights].",
     "priority": "high"
   }'
   

3. Key prompting strategies for Jinx:

   • Be specific about data extraction requirements
   • Request JSON output format
   • Ask for both raw findings AND summary analysis
   • Include comparison requirements if multiple sources

Phase 3: Compilation & Skills Documentation (Lucky)

Time: 10-20% of total project time
Focus: Synthesis and actionable insights

1. Collect Jinx results

   curl -s http://localhost:3001/results/[task-id]
   

2. Compile comprehensive report

   • Executive summary with key findings
   • Structured data tables/comparisons
   • Strategic recommendations
   • Process insights and improvements

3. Document process learnings

   • What worked well / areas for improvement
   • Time saved vs sequential approach
   • Quality of analysis vs manual extraction

Best Practices

Data Gathering (Lucky)

• Capture everything — let Jinx filter, don't pre-filter
• Use consistent file naming — project-source-timestamp.html
• Include rich metadata — helps Jinx understand context
• Work in batches — send first batch to Jinx while gathering more

Analysis Tasks (Jinx)

• Be specific about extraction requirements
• Request execution — ask Jinx to run analysis scripts, not just provide them
• Structure output — JSON format for easy parsing
• Ask for insights — not just data extraction but pattern analysis

Collaboration

• Send tasks early — don't wait for all data before starting analysis
• Check progress regularly — curl status API to monitor queue
• Quality over quantity — better to analyze fewer sources deeply

Time Estimates

Effective time = max(Lucky, Jinx) due to parallelization

Security Considerations

• HTML sanitization — Strip <script> tags before sending to Jinx
• No executable content — Only pass text/HTML data, never code
• Local processing — Jinx has no internet access, data stays secure
• File permissions — Ensure Jinx can read files on SSD

Success Metrics

• Speed: 30-50% time savings vs sequential research
• Coverage: Ability to analyze larger datasets comprehensively
• Quality: Structured, actionable insights vs raw data dumps
• Scalability: Process works for 5 sources or 50 sources

Example Use Cases

1. Market Research: Lucky scrapes Gumroad/Etsy → Jinx extracts pricing/features
2. API Comparison: Lucky gathers docs → Jinx compares capabilities/pricing
3. Trend Analysis: Lucky gets Google Trends → Jinx identifies patterns
4. Competitor Analysis: Lucky browses sites → Jinx structures competitive matrix
5. Content Analysis: Lucky gathers articles → Jinx summarizes themes/insights

Market Research Template

For marketplace/competitor analysis specifically, use this structured approach:

Data Collection Checklist

For each competitor/product found:

## Competitor: [Name]
- Product: [Title]
- Price: $[Amount]
- Bundle Size: [X items]
- Format: [Canva/PSD/AI/etc]
- Sales Indicators: [Reviews/ratings/badges]
- Key Features: [List]
- Customer Complaints: [Common issues from reviews]
- Opportunities: [What they're missing]

Market Analysis Phases

1. Market Mapping — Browse categories on target platforms (Gumroad, Etsy, Creative Market, Redbubble). Screenshot layouts. Document pricing patterns.
2. Competitor Deep Dive — Top performers, pricing intelligence, positioning, visual trends.
3. Customer Intelligence — Mine reviews for pain points, gaps, price sensitivity, feature requests.
4. Trend Analysis — Style evolution, platform preferences, niche saturation, seasonal patterns.
5. Gap Analysis — What customers want but can't find. Underserved niches.

Browser Research Workflow

1. Start browser session
2. Navigate to marketplace, search category
3. Capture screenshots of results
4. Visit top competitor pages
5. Document structured data per template above
6. Save to SSD, feed to Jinx for pattern analysis

Output Deliverables

• Structured competitor profiles
• Pricing analysis with recommendations
• Market gap identification
• Customer pain point summary
• Launch strategy recommendations

Process Evolution

Track and improve:

• Which DOM selectors/sites work best
• Jinx prompt patterns that yield best results
• File transfer automation opportunities
• Quality indicators for different research types

This skill creates a scalable, repeatable process for any research requiring both web access and deep analysis.