Ultimate AI Content Pipeline

Ultimate AI Content Pipeline

Skill by ara.so — Marketing Skills collection.

This project is a complete automated content creation pipeline that transforms keywords into fully-researched, multilingual articles and videos. It crawls recent data from sources like TechCrunch and Twitter, generates content using Claude/OpenAI, and renders videos using Remotion.

What It Does

• Auto-Research: Crawls and analyzes real-time data from news sources and social media
• AI Content Generation: Creates articles in multiple formats (toplist, POV, case study, how-to)
• Multilingual Support: Generates content in English and Vietnamese simultaneously
• Video Rendering: Automatically creates infographics and short-form videos from content
• Multi-Platform Export: Optimized for Reels, TikTok, Shorts

Installation

# Clone the repository
git clone https://github.com/pennydinh/marketing-pineline-share.git
cd marketing-pineline-share

# Install dependencies
npm install
# or
yarn install
# or
pnpm install

Environment Configuration

Create a .env.local file in the project root:

# AI APIs
ANTHROPIC_API_KEY=your_claude_api_key
OPENAI_API_KEY=your_openai_api_key

# Research APIs
RAPIDAPI_KEY=your_rapidapi_key

# Database (if applicable)
DATABASE_URL=your_database_url

# Video Rendering
REMOTION_LICENSE_KEY=your_remotion_license_key

Project Structure

├── src/
│   ├── app/              # Next.js app directory
│   ├── components/       # React components
│   ├── lib/
│   │   ├── ai/          # AI integration (Claude, OpenAI)
│   │   ├── research/    # Web scraping and data collection
│   │   ├── content/     # Content generation logic
│   │   └── video/       # Remotion video rendering
│   └── utils/           # Helper functions
├── remotion/            # Remotion video templates
└── public/              # Static assets

Core Usage Patterns

1. Research & Data Collection

import { autoResearch } from '@/lib/research/auto-scan';

// Crawl recent data on a topic
const researchData = await autoResearch({
  keyword: 'AI automation',
  sources: ['techcrunch', 'twitter', 'linkedin'],
  timeframe: '24h',
  language: 'en'
});

// Returns structured data with insights
console.log(researchData.insights);
console.log(researchData.sources);
console.log(researchData.statistics);

2. AI Content Generation with Claude

import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

async function generateContent(
  topic: string,
  format: 'toplist' | 'pov' | 'case-study' | 'how-to',
  language: 'en' | 'vi'
) {
  const message = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 4000,
    messages: [
      {
        role: 'user',
        content: `Create a ${format} article about ${topic} in ${language}. 
        Include data-backed insights and current trends.`
      }
    ],
  });

  return message.content[0].text;
}

// Generate content
const article = await generateContent('AI Marketing Tools', 'toplist', 'en');

3. Multilingual Content Generation

import { generateMultilingualContent } from '@/lib/content/multilingual';

// Generate content in multiple languages simultaneously
const multilingualContent = await generateMultilingualContent({
  topic: 'Content Automation Trends 2026',
  format: 'how-to',
  languages: ['en', 'vi'],
  tone: 'professional', // or 'friendly', 'humorous'
  researchData: researchData
});

console.log(multilingualContent.en); // English version
console.log(multilingualContent.vi); // Vietnamese version

4. Video Generation with Remotion

import { bundle } from '@remotion/bundler';
import { renderMedia, selectComposition } from '@remotion/renderer';
import { webpackOverride } from './remotion/webpack-override';

async function renderContentVideo(content: any) {
  // Bundle the Remotion project
  const bundleLocation = await bundle({
    entryPoint: './remotion/index.ts',
    webpackOverride,
  });

  // Select composition
  const composition = await selectComposition({
    serveUrl: bundleLocation,
    id: 'ContentVideo',
    inputProps: {
      title: content.title,
      points: content.keyPoints,
      duration: 30, // seconds
    },
  });

  // Render video
  await renderMedia({
    composition,
    serveUrl: bundleLocation,
    codec: 'h264',
    outputLocation: `out/${content.slug}.mp4`,
  });
}

5. Complete Content Pipeline

import { ContentPipeline } from '@/lib/pipeline';

const pipeline = new ContentPipeline({
  aiProvider: 'claude', // or 'openai'
  languages: ['en', 'vi'],
  outputFormats: ['article', 'video', 'infographic']
});

// Run full pipeline
const result = await pipeline.run({
  keyword: 'AI Marketing Automation',
  contentFormat: 'toplist',
  videoAspectRatio: '9:16', // for Reels/TikTok
  autoPublish: false
});

console.log(result.articles);  // Generated articles
console.log(result.videos);    // Rendered video paths
console.log(result.metadata);  // SEO and social metadata

API Integration Examples

Research API Integration

import axios from 'axios';

async function fetchTrendingTopics() {
  const options = {
    method: 'GET',
    url: 'https://trending-topics-api.p.rapidapi.com/topics',
    params: { category: 'technology' },
    headers: {
      'X-RapidAPI-Key': process.env.RAPIDAPI_KEY,
      'X-RapidAPI-Host': 'trending-topics-api.p.rapidapi.com'
    }
  };

  const response = await axios.request(options);
  return response.data;
}

OpenAI Integration Alternative

import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

async function generateWithGPT(prompt: string) {
  const completion = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [
      {
        role: 'system',
        content: 'You are an expert content writer specializing in marketing.'
      },
      {
        role: 'user',
        content: prompt
      }
    ],
    temperature: 0.7,
  });

  return completion.choices[0].message.content;
}

Running the Application

Development Server

# Start Next.js development server
npm run dev

# Access at http://localhost:3000

Video Rendering

# Render a specific composition
npx remotion render remotion/index.ts ContentVideo out/video.mp4

# Preview in Remotion Studio
npx remotion studio

Build for Production

# Build Next.js app
npm run build

# Start production server
npm run start

Common Workflows

Workflow 1: Generate Daily Content

import { scheduleContentGeneration } from '@/lib/scheduler';

// Schedule daily content generation
scheduleContentGeneration({
  cron: '0 9 * * *', // 9 AM daily
  topics: ['AI trends', 'Marketing automation', 'Content strategy'],
  formats: ['toplist', 'how-to'],
  languages: ['en', 'vi'],
  autoRender: true,
  autoPublish: false
});

Workflow 2: Trend-Based Content

async function generateTrendingContent() {
  // 1. Research trending topics
  const trends = await fetchTrendingTopics();
  
  // 2. Generate content for top 3 trends
  const contentPromises = trends.slice(0, 3).map(trend =>
    generateContent(trend.title, 'pov', 'en')
  );
  
  const articles = await Promise.all(contentPromises);
  
  // 3. Render videos for each article
  for (const article of articles) {
    await renderContentVideo(article);
  }
  
  return articles;
}

Workflow 3: Custom Content Format

interface CustomContentConfig {
  topic: string;
  targetAudience: string;
  contentGoal: 'awareness' | 'engagement' | 'conversion';
  includeDataPoints: boolean;
}

async function generateCustomContent(config: CustomContentConfig) {
  const researchData = await autoResearch({ 
    keyword: config.topic 
  });
  
  const prompt = `
    Create content about ${config.topic} for ${config.targetAudience}.
    Goal: ${config.contentGoal}
    ${config.includeDataPoints ? 'Include relevant statistics and data points.' : ''}
    
    Research insights: ${JSON.stringify(researchData.insights)}
  `;
  
  return await generateWithGPT(prompt);
}

Configuration Options

Content Generation Config

interface ContentConfig {
  aiProvider: 'claude' | 'openai';
  model?: string;
  temperature?: number;
  maxTokens?: number;
  tone?: 'professional' | 'friendly' | 'humorous';
  includeImages?: boolean;
  seoOptimized?: boolean;
}

const config: ContentConfig = {
  aiProvider: 'claude',
  model: 'claude-3-5-sonnet-20241022',
  temperature: 0.7,
  maxTokens: 4000,
  tone: 'professional',
  includeImages: true,
  seoOptimized: true
};

Video Rendering Config

interface VideoConfig {
  fps: number;
  width: number;
  height: number;
  codec: 'h264' | 'h265';
  quality: 'low' | 'medium' | 'high';
  aspectRatio: '16:9' | '9:16' | '1:1';
}

const videoConfig: VideoConfig = {
  fps: 30,
  width: 1080,
  height: 1920,
  codec: 'h264',
  quality: 'high',
  aspectRatio: '9:16' // for TikTok/Reels
};

Troubleshooting

API Rate Limits

import { RateLimiter } from '@/lib/utils/rate-limiter';

const limiter = new RateLimiter({
  maxRequests: 50,
  windowMs: 60000, // 1 minute
});

async function callAIWithRateLimit(prompt: string) {
  await limiter.wait();
  return await generateContent(prompt, 'toplist', 'en');
}

Error Handling

async function safeContentGeneration(topic: string) {
  try {
    const content = await generateContent(topic, 'toplist', 'en');
    return { success: true, content };
  } catch (error) {
    if (error.status === 429) {
      console.error('Rate limit exceeded, retry in 60s');
      await new Promise(resolve => setTimeout(resolve, 60000));
      return safeContentGeneration(topic);
    }
    
    if (error.status === 401) {
      console.error('Invalid API key');
      throw new Error('Check your API keys in .env.local');
    }
    
    console.error('Content generation failed:', error);
    return { success: false, error: error.message };
  }
}

Video Rendering Issues

// Check Remotion setup
import { getCompositions } from '@remotion/renderer';

async function debugRemotionSetup() {
  try {
    const bundleLocation = await bundle({
      entryPoint: './remotion/index.ts',
    });
    
    const compositions = await getCompositions(bundleLocation);
    console.log('Available compositions:', compositions);
  } catch (error) {
    console.error('Remotion setup error:', error);
    console.log('Check: 1) remotion/ directory exists, 2) index.ts is valid');
  }
}

Memory Management for Large Content

// Process content in batches
async function batchContentGeneration(topics: string[]) {
  const batchSize = 5;
  const results = [];
  
  for (let i = 0; i < topics.length; i += batchSize) {
    const batch = topics.slice(i, i + batchSize);
    const batchResults = await Promise.all(
      batch.map(topic => generateContent(topic, 'toplist', 'en'))
    );
    results.push(...batchResults);
    
    // Clear memory between batches
    if (global.gc) global.gc();
  }
  
  return results;
}

Best Practices

1. Always validate research data before passing to AI
2. Cache research results to avoid redundant API calls
3. Use streaming for long-form content generation
4. Implement retry logic for API failures
5. Monitor AI costs with usage tracking
6. Version control your prompt templates
7. Test video renders before batch processing