marketing-pipeline-automation

name	marketing-pipeline-automation
description	Automated AI content pipeline from research to video generation using Claude, OpenAI, and Remotion
triggers	["how do I automate content creation with AI","help me set up the marketing pipeline project","generate video content from text automatically","crawl news and create content with Claude","automate research to video workflow","build AI content generation pipeline","create automated marketing content system","set up remotion video rendering for content"]

Marketing Pipeline Automation Skill

Skill by ara.so — Marketing Skills collection.

This skill enables AI coding agents to help developers use the Ultimate AI Content Pipeline - an automated system that takes a keyword and produces complete content pieces including research, written articles, and rendered videos. The pipeline integrates Claude 3, OpenAI, web scraping, and Remotion for video generation.

What This Project Does

The Marketing Pipeline automates the entire content creation workflow:

1. Auto-Research: Crawls recent news from TechCrunch, a16z, Twitter/X, LinkedIn (last 24h)
2. AI Content Generation: Creates articles in multiple formats (toplist, POV, case study, how-to) using Claude/OpenAI
3. Multi-language: Generates content in English and Vietnamese simultaneously
4. Video Rendering: Automatically creates infographics and short videos using Remotion
5. Platform Optimization: Exports videos for Reels, TikTok, Shorts

Installation

Prerequisites

# Node.js 18+ required
node --version

# Clone the repository
git clone https://github.com/pennydinh/marketing-pineline-share.git
cd marketing-pineline-share

# Install dependencies
npm install
# or
yarn install

Environment Configuration

Create a .env.local file in the project root:

# AI APIs
OPENAI_API_KEY=
ANTHROPIC_API_KEY=

# Web Scraping
RAPIDAPI_KEY=

# Database (optional)
DATABASE_URL=

# Remotion (for video rendering)
REMOTION_LICENSE_KEY=

# Application
NEXT_PUBLIC_APP_URL=http://localhost:3000

Development Server

// Start the Next.js development server
npm run dev
// or
yarn dev

// Access at http://localhost:3000

Core Architecture

Project Structure

marketing-pineline-share/
├── app/                    # Next.js app directory
├── components/             # React components
├── lib/                    # Core utilities
│   ├── ai/                # AI integrations
│   ├── scraper/           # Web scraping modules
│   └── video/             # Remotion video generation
├── remotion/              # Video templates
└── public/                # Static assets

Key Components and Usage

1. Content Research Module

// lib/scraper/news-crawler.ts
import { fetchNewsFromSources } from '@/lib/scraper/news-crawler';

interface NewsSource {
  name: string;
  url: string;
  selector: string;
}

async function gatherResearch(keyword: string) {
  const sources: NewsSource[] = [
    { name: 'TechCrunch', url: 'https://techcrunch.com', selector: '.post-block' },
    { name: 'a16z', url: 'https://a16z.com/posts', selector: '.article' }
  ];
  
  const results = await fetchNewsFromSources(keyword, sources, {
    timeRange: '24h',
    maxResults: 20
  });
  
  return results;
}

// Usage
const research = await gatherResearch('AI automation');
console.log(research.articles); // Array of crawled articles

2. AI Content Generation

// lib/ai/content-generator.ts
import Anthropic from '@anthropic-ai/sdk';
import OpenAI from 'openai';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

interface ContentRequest {
  keyword: string;
  format: 'toplist' | 'pov' | 'case-study' | 'how-to';
  language: 'en' | 'vi';
  tone: 'expert' | 'friendly' | 'humorous';
  researchData: any[];
}

async function generateContentWithClaude(request: ContentRequest) {
  const prompt = buildPrompt(request);
  
  const message = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 4096,
    messages: [{
      role: 'user',
      content: prompt
    }]
  });
  
  return message.content[0].text;
}

async function generateContentWithOpenAI(request: ContentRequest) {
  const prompt = buildPrompt(request);
  
  const completion = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [
      { role: 'system', content: 'You are an expert content creator.' },
      { role: 'user', content: prompt }
    ],
    max_tokens: 4096
  });
  
  return completion.choices[0].message.content;
}

function buildPrompt(request: ContentRequest): string {
  const { keyword, format, language, tone, researchData } = request;
  
  return `
Create a ${format} article about "${keyword}" in ${language}.
Tone: ${tone}
Based on this research data: ${JSON.stringify(researchData)}

Requirements:
- Include data-backed insights
- Use recent statistics (last 24h)
- Format for ${language === 'vi' ? 'Vietnamese' : 'English'} audience
- Optimize for SEO
`;
}

3. Video Generation with Remotion

// lib/video/render-video.ts
import { bundle } from '@remotion/bundler';
import { renderMedia, selectComposition } from '@remotion/renderer';
import path from 'path';

interface VideoConfig {
  title: string;
  content: string[];
  style: 'infographic' | 'text-animation' | 'slideshow';
  platform: 'reels' | 'tiktok' | 'shorts';
}

async function renderContentVideo(config: VideoConfig) {
  const { title, content, style, platform } = config;
  
  // Platform-specific dimensions
  const dimensions = {
    reels: { width: 1080, height: 1920 },
    tiktok: { width: 1080, height: 1920 },
    shorts: { width: 1080, height: 1920 }
  };
  
  const { width, height } = dimensions[platform];
  
  // Bundle Remotion project
  const bundleLocation = await bundle({
    entryPoint: path.join(process.cwd(), 'remotion/index.ts'),
    webpackOverride: (config) => config,
  });
  
  // Select composition
  const composition = await selectComposition({
    serveUrl: bundleLocation,
    id: style,
    inputProps: {
      title,
      content,
      width,
      height
    }
  });
  
  // Render video
  const outputPath = path.join(process.cwd(), 'public/videos', `${Date.now()}.mp4`);
  
  await renderMedia({
    composition,
    serveUrl: bundleLocation,
    codec: 'h264',
    outputLocation: outputPath,
    inputProps: {
      title,
      content
    }
  });
  
  return outputPath;
}

4. Remotion Video Template

// remotion/compositions/Infographic.tsx
import { AbsoluteFill, useCurrentFrame, useVideoConfig, interpolate } from 'remotion';

interface InfographicProps {
  title: string;
  content: string[];
}

export const Infographic: React.FC<InfographicProps> = ({ title, content }) => {
  const frame = useCurrentFrame();
  const { fps, durationInFrames } = useVideoConfig();
  
  const titleOpacity = interpolate(
    frame,
    [0, 30],
    [0, 1],
    { extrapolateRight: 'clamp' }
  );
  
  const titleScale = interpolate(
    frame,
    [0, 30],
    [0.8, 1],
    { extrapolateRight: 'clamp' }
  );
  
  return (
    <AbsoluteFill style={{ backgroundColor: '#1a1a1a' }}>
      <div style={{
        display: 'flex',
        flexDirection: 'column',
        justifyContent: 'center',
        alignItems: 'center',
        padding: '40px',
        height: '100%'
      }}>
        <h1 style={{
          color: 'white',
          fontSize: '72px',
          fontWeight: 'bold',
          textAlign: 'center',
          opacity: titleOpacity,
          transform: `scale(${titleScale})`
        }}>
          {title}
        </h1>
        
        <div style={{ marginTop: '60px' }}>
          {content.map((item, index) => {
            const itemFrame = 40 + (index * 20);
            const itemOpacity = interpolate(
              frame,
              [itemFrame, itemFrame + 15],
              [0, 1],
              { extrapolateRight: 'clamp' }
            );
            
            return (
              <p key={index} style={{
                color: '#ffffff',
                fontSize: '36px',
                marginBottom: '30px',
                opacity: itemOpacity
              }}>
                {item}
              </p>
            );
          })}
        </div>
      </div>
    </AbsoluteFill>
  );
};

5. Complete Pipeline Example

// app/api/generate-content/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { gatherResearch } from '@/lib/scraper/news-crawler';
import { generateContentWithClaude } from '@/lib/ai/content-generator';
import { renderContentVideo } from '@/lib/video/render-video';

export async function POST(request: NextRequest) {
  try {
    const { keyword, format, language, platform } = await request.json();
    
    // Step 1: Research
    console.log('🔍 Gathering research...');
    const research = await gatherResearch(keyword);
    
    // Step 2: Generate content
    console.log('✍️ Generating content...');
    const content = await generateContentWithClaude({
      keyword,
      format,
      language,
      tone: 'expert',
      researchData: research.articles
    });
    
    // Parse content into video-friendly format
    const contentLines = content.split('\n').filter(line => line.trim());
    
    // Step 3: Render video
    console.log('🎬 Rendering video...');
    const videoPath = await renderContentVideo({
      title: keyword,
      content: contentLines.slice(0, 5), // First 5 key points
      style: 'infographic',
      platform
    });
    
    return NextResponse.json({
      success: true,
      data: {
        content,
        videoUrl: videoPath.replace(process.cwd() + '/public', '')
      }
    });
    
  } catch (error) {
    console.error('Pipeline error:', error);
    return NextResponse.json(
      { success: false, error: error.message },
      { status: 500 }
    );
  }
}

6. Frontend Component

// components/ContentPipeline.tsx
'use client';

import { useState } from 'react';

export function ContentPipeline() {
  const [keyword, setKeyword] = useState('');
  const [loading, setLoading] = useState(false);
  const [result, setResult] = useState<any>(null);
  
  const handleGenerate = async () => {
    setLoading(true);
    
    try {
      const response = await fetch('/api/generate-content', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          keyword,
          format: 'toplist',
          language: 'vi',
          platform: 'reels'
        })
      });
      
      const data = await response.json();
      setResult(data);
    } catch (error) {
      console.error('Error:', error);
    } finally {
      setLoading(false);
    }
  };
  
  return (
    <div className="max-w-4xl mx-auto p-6">
      <h1 className="text-3xl font-bold mb-6">AI Content Pipeline</h1>
      
      <div className="space-y-4">
        <input
          type="text"
          value={keyword}
          onChange={(e) => setKeyword(e.target.value)}
          placeholder="Enter keyword (e.g., 'AI automation')"
          className="w-full px-4 py-2 border rounded"
        />
        
        <button
          onClick={handleGenerate}
          disabled={loading || !keyword}
          className="px-6 py-2 bg-blue-600 text-white rounded disabled:opacity-50"
        >
          {loading ? 'Generating...' : 'Generate Content'}
        </button>
        
        {result && (
          <div className="mt-6 space-y-4">
            <div className="p-4 bg-gray-50 rounded">
              <h3 className="font-bold mb-2">Generated Content:</h3>
              <p className="whitespace-pre-wrap">{result.data.content}</p>
            </div>
            
            {result.data.videoUrl && (
              <div>
                <h3 className="font-bold mb-2">Generated Video:</h3>
                <video src={result.data.videoUrl} controls className="w-full" />
              </div>
            )}
          </div>
        )}
      </div>
    </div>
  );
}

Configuration Options

AI Model Selection

// config/ai-config.ts
export const AI_CONFIG = {
  primaryModel: 'claude', // or 'openai'
  models: {
    claude: {
      model: 'claude-3-5-sonnet-20241022',
      maxTokens: 4096
    },
    openai: {
      model: 'gpt-4-turbo-preview',
      maxTokens: 4096
    }
  }
};

Scraper Configuration

// config/scraper-config.ts
export const SCRAPER_CONFIG = {
  sources: [
    { name: 'TechCrunch', url: 'https://techcrunch.com', enabled: true },
    { name: 'a16z', url: 'https://a16z.com/posts', enabled: true },
    { name: 'Twitter', url: 'https://twitter.com', enabled: false } // Requires auth
  ],
  timeRange: '24h',
  maxArticles: 20,
  timeout: 30000
};

Video Configuration

// config/video-config.ts
export const VIDEO_CONFIG = {
  defaultStyle: 'infographic',
  fps: 30,
  platforms: {
    reels: { width: 1080, height: 1920, duration: 30 },
    tiktok: { width: 1080, height: 1920, duration: 60 },
    shorts: { width: 1080, height: 1920, duration: 60 }
  }
};

Common Patterns

Batch Content Generation

async function generateBatchContent(keywords: string[]) {
  const results = await Promise.all(
    keywords.map(async (keyword) => {
      const research = await gatherResearch(keyword);
      const content = await generateContentWithClaude({
        keyword,
        format: 'toplist',
        language: 'vi',
        tone: 'expert',
        researchData: research.articles
      });
      
      return { keyword, content };
    })
  );
  
  return results;
}

Multi-language Generation

async function generateMultiLanguage(keyword: string) {
  const research = await gatherResearch(keyword);
  
  const [english, vietnamese] = await Promise.all([
    generateContentWithClaude({
      keyword,
      format: 'toplist',
      language: 'en',
      tone: 'expert',
      researchData: research.articles
    }),
    generateContentWithClaude({
      keyword,
      format: 'toplist',
      language: 'vi',
      tone: 'expert',
      researchData: research.articles
    })
  ]);
  
  return { english, vietnamese };
}

Troubleshooting

API Rate Limits

// lib/utils/rate-limiter.ts
class RateLimiter {
  private queue: Promise<any> = Promise.resolve();
  
  async throttle<T>(fn: () => Promise<T>, delay: number = 1000): Promise<T> {
    this.queue = this.queue.then(() => 
      new Promise(resolve => setTimeout(resolve, delay))
    );
    
    await this.queue;
    return fn();
  }
}

const limiter = new RateLimiter();

// Usage
const content = await limiter.throttle(() => 
  generateContentWithClaude(request)
);

Video Rendering Memory Issues

# Increase Node.js memory limit
NODE_OPTIONS="--max-old-space-size=4096" npm run dev

Scraper Blocked

// Add user agent rotation
const USER_AGENTS = [
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
];

function getRandomUserAgent() {
  return USER_AGENTS[Math.floor(Math.random() * USER_AGENTS.length)];
}

Environment Variables Not Loading

// Check .env.local exists
if (!process.env.OPENAI_API_KEY) {
  throw new Error('OPENAI_API_KEY not found in environment variables');
}

CLI Commands

# Development
npm run dev          # Start dev server
npm run build        # Build for production
npm run start        # Start production server

# Remotion
npm run remotion     # Open Remotion studio
npm run render       # Render video compositions

# Testing
npm run test         # Run tests
npm run lint         # Run linter