Marketing Pipeline AI Content

Marketing Pipeline AI Content Automation

Skill by ara.so — Marketing Skills collection.

This skill enables AI coding agents to work with the Ultimate AI Content Pipeline - a TypeScript-based content automation system that handles research, scriptwriting, and video generation using AI (Claude 3, OpenAI) and Remotion for video rendering.

What This Project Does

The Marketing Pipeline is an end-to-end content automation system that:

• Auto-crawls research from sources like TechCrunch, a16z, Twitter/X, and LinkedIn
• Generates content in multiple formats (toplists, POV, case studies, how-tos) using Claude/OpenAI
• Creates multilingual content (English & Vietnamese) with customizable tone
• Renders videos automatically using Remotion for social media platforms
• Provides a Next.js interface for managing the entire pipeline

Installation

# Clone the repository
git clone https://github.com/pennydinh/marketing-pineline-share.git
cd marketing-pineline-share

# Install dependencies
npm install
# or
yarn install
# or
pnpm install

Configuration

Create a .env.local file in the root directory:

# AI Services
ANTHROPIC_API_KEY=your_claude_api_key
OPENAI_API_KEY=your_openai_api_key

# Research APIs
RAPIDAPI_KEY=your_rapidapi_key

# Content Settings
DEFAULT_LANGUAGE=en
TONE=professional

Project Structure

marketing-pipeline/
├── src/
│   ├── app/              # Next.js app directory
│   ├── components/       # React components
│   ├── lib/
│   │   ├── research/     # Research crawling modules
│   │   ├── ai/           # AI generation (Claude/OpenAI)
│   │   ├── render/       # Remotion video rendering
│   │   └── utils/        # Helper functions
│   └── types/            # TypeScript type definitions
├── public/               # Static assets
└── remotion/            # Remotion compositions

Core API Usage

1. Research Content Crawling

import { crawlResearch } from '@/lib/research/crawler';

interface ResearchOptions {
  keyword: string;
  sources: ('techcrunch' | 'a16z' | 'twitter' | 'linkedin')[];
  timeframe: '24h' | '7d' | '30d';
}

async function gatherResearch(options: ResearchOptions) {
  const research = await crawlResearch({
    keyword: options.keyword,
    sources: options.sources,
    timeframe: options.timeframe,
  });
  
  return research; // Returns { articles: [], insights: [], data: [] }
}

// Example usage
const data = await gatherResearch({
  keyword: 'AI automation',
  sources: ['techcrunch', 'twitter'],
  timeframe: '24h',
});

2. AI Content Generation

import { generateContent } from '@/lib/ai/generator';
import { Anthropic } from '@anthropic-ai/sdk';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

interface ContentConfig {
  format: 'toplist' | 'pov' | 'case-study' | 'how-to';
  tone: 'professional' | 'friendly' | 'humorous';
  language: 'en' | 'vi';
  research: any;
}

async function createContent(config: ContentConfig) {
  const prompt = `
Based on this research: ${JSON.stringify(config.research)}

Create a ${config.format} article in ${config.language} with a ${config.tone} tone.
Include data-backed insights and real examples.
  `;

  const message = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 4096,
    messages: [{
      role: 'user',
      content: prompt,
    }],
  });

  return message.content[0].text;
}

3. OpenAI Alternative

import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

async function generateWithOpenAI(prompt: string) {
  const completion = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [
      {
        role: 'system',
        content: 'You are an expert content creator specializing in marketing and social media.',
      },
      {
        role: 'user',
        content: prompt,
      },
    ],
    temperature: 0.7,
    max_tokens: 3000,
  });

  return completion.choices[0].message.content;
}

4. Video Rendering with Remotion

import { bundle } from '@remotion/bundler';
import { renderMedia, selectComposition } from '@remotion/renderer';
import path from 'path';

interface VideoConfig {
  title: string;
  content: string;
  duration: number;
  format: 'reels' | 'tiktok' | 'shorts';
}

async function renderContentVideo(config: VideoConfig) {
  const bundleLocation = await bundle({
    entryPoint: path.join(process.cwd(), 'remotion/index.ts'),
    webpackOverride: (config) => config,
  });

  const composition = await selectComposition({
    serveUrl: bundleLocation,
    id: 'ContentVideo',
    inputProps: {
      title: config.title,
      content: config.content,
    },
  });

  const dimensions = {
    reels: { width: 1080, height: 1920 },
    tiktok: { width: 1080, height: 1920 },
    shorts: { width: 1080, height: 1920 },
  };

  await renderMedia({
    composition,
    serveUrl: bundleLocation,
    codec: 'h264',
    outputLocation: `out/${config.title}.mp4`,
    ...dimensions[config.format],
  });
}

Common Patterns

Complete Content Pipeline

import { crawlResearch } from '@/lib/research/crawler';
import { generateContent } from '@/lib/ai/generator';
import { renderContentVideo } from '@/lib/render/video';

async function runContentPipeline(keyword: string) {
  try {
    // Step 1: Research
    console.log('Starting research...');
    const research = await crawlResearch({
      keyword,
      sources: ['techcrunch', 'twitter'],
      timeframe: '24h',
    });

    // Step 2: Generate content
    console.log('Generating content...');
    const content = await createContent({
      format: 'toplist',
      tone: 'professional',
      language: 'en',
      research,
    });

    // Step 3: Render video
    console.log('Rendering video...');
    await renderContentVideo({
      title: keyword,
      content,
      duration: 30,
      format: 'reels',
    });

    return {
      success: true,
      content,
      videoPath: `out/${keyword}.mp4`,
    };
  } catch (error) {
    console.error('Pipeline error:', error);
    throw error;
  }
}

Bilingual Content Generation

async function generateBilingualContent(research: any) {
  const [englishContent, vietnameseContent] = await Promise.all([
    createContent({
      format: 'pov',
      tone: 'professional',
      language: 'en',
      research,
    }),
    createContent({
      format: 'pov',
      tone: 'professional',
      language: 'vi',
      research,
    }),
  ]);

  return {
    en: englishContent,
    vi: vietnameseContent,
  };
}

Batch Content Creation

async function createMultipleFormats(keyword: string) {
  const research = await crawlResearch({
    keyword,
    sources: ['techcrunch', 'a16z'],
    timeframe: '7d',
  });

  const formats: Array<'toplist' | 'pov' | 'case-study' | 'how-to'> = [
    'toplist',
    'pov',
    'case-study',
    'how-to',
  ];

  const contents = await Promise.all(
    formats.map((format) =>
      createContent({
        format,
        tone: 'professional',
        language: 'en',
        research,
      })
    )
  );

  return formats.reduce((acc, format, index) => {
    acc[format] = contents[index];
    return acc;
  }, {} as Record<string, string>);
}

Running the Application

Development Server

npm run dev
# or
yarn dev
# or
pnpm dev

Visit http://localhost:3000 to access the Next.js interface.

Build for Production

npm run build
npm start

Render Videos Only

# If the project has a dedicated video rendering script
npm run render

API Routes (Next.js)

// app/api/generate/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { runContentPipeline } from '@/lib/pipeline';

export async function POST(request: NextRequest) {
  const { keyword, format, language } = await request.json();

  try {
    const result = await runContentPipeline(keyword);
    
    return NextResponse.json({
      success: true,
      data: result,
    });
  } catch (error) {
    return NextResponse.json(
      { success: false, error: error.message },
      { status: 500 }
    );
  }
}

Troubleshooting

API Rate Limits

// Implement retry logic with exponential backoff
async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  maxRetries = 3
): Promise<T> {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (i === maxRetries - 1) throw error;
      await new Promise(resolve => setTimeout(resolve, Math.pow(2, i) * 1000));
    }
  }
  throw new Error('Max retries exceeded');
}

Video Rendering Memory Issues

// Use smaller compositions or split rendering
const composition = await selectComposition({
  serveUrl: bundleLocation,
  id: 'ContentVideo',
  inputProps: {
    title: config.title,
    content: config.content.slice(0, 500), // Limit content length
  },
});

Claude/OpenAI Token Limits

function truncateContent(text: string, maxTokens = 3000): string {
  // Rough estimate: 1 token ≈ 4 characters
  const maxChars = maxTokens * 4;
  return text.length > maxChars ? text.slice(0, maxChars) : text;
}

Environment Variables Reference

Best Practices

1. Always validate research data before passing to AI generators
2. Cache research results to avoid redundant API calls
3. Use environment-specific configs for development vs production
4. Monitor API usage to stay within rate limits
5. Test video renders locally before batch processing
6. Implement proper error logging for production debugging