marketing-pipeline-auto-content

name	marketing-pipeline-auto-content
description	Automated AI content pipeline for research, scriptwriting, auto-posting and video generation using Claude, OpenAI and Remotion
triggers	["how do I set up the AI content automation pipeline","automate content creation from research to video","use Claude and OpenAI for content generation","create automated marketing content pipeline","generate videos from content automatically with Remotion","crawl news sources and create content posts","build an AI content factory for social media","set up auto-posting content workflow"]

Marketing Pipeline Auto Content

Skill by ara.so — Marketing Skills collection.

This skill enables AI coding agents to work with the Ultimate AI Content Pipeline - a complete automated content creation system that handles research (crawling news sources), content generation (using Claude/OpenAI), and video rendering (via Remotion). The pipeline transforms keywords into ready-to-publish content across multiple formats and languages.

What This Project Does

The Marketing Pipeline is an all-in-one content automation system that:

• Auto-scans research sources: Crawls TechCrunch, a16z, Twitter/X, LinkedIn for fresh data within 24 hours
• Generates multi-format content: Creates toplist, POV, case studies, how-to articles using Claude 3 or OpenAI
• Supports bilingual output: Produces Vietnamese and English content simultaneously
• Renders videos automatically: Uses Remotion to transform written content into Reels/TikTok/Shorts videos
• Optimizes for platforms: Exports videos in proper aspect ratios for different social platforms

Installation

Prerequisites

# Node.js 18+ and npm/yarn required
node --version  # Should be 18.x or higher

Clone and Install

git clone https://github.com/pennydinh/marketing-pineline-share.git
cd marketing-pineline-share

# Install dependencies
npm install
# or
yarn install

Environment Configuration

Create a .env.local file in the project root:

# AI Provider Keys
ANTHROPIC_API_KEY=your_claude_api_key
OPENAI_API_KEY=your_openai_api_key

# Crawler/Research APIs
RAPIDAPI_KEY=your_rapidapi_key

# Remotion (Video Rendering)
REMOTION_LICENSE_KEY=your_remotion_license

# Next.js Configuration
NEXT_PUBLIC_API_URL=http://localhost:3000

Start Development Server

npm run dev
# or
yarn dev

Access the application at http://localhost:3000

Project Structure

marketing-pineline-share/
├── app/                    # Next.js app directory
│   ├── api/               # API routes
│   │   ├── research/      # Content research endpoints
│   │   ├── generate/      # Content generation endpoints
│   │   └── render/        # Video rendering endpoints
│   ├── components/        # React components
│   └── page.tsx          # Main page
├── lib/                   # Core utilities
│   ├── ai/               # AI provider integrations
│   │   ├── claude.ts     # Claude API wrapper
│   │   └── openai.ts     # OpenAI API wrapper
│   ├── crawler/          # Web scraping modules
│   ├── content/          # Content generation logic
│   └── video/            # Remotion video templates
├── remotion/             # Remotion video configurations
└── public/               # Static assets

Key API Endpoints

Research Endpoint

// app/api/research/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { crawlSources } from '@/lib/crawler';

export async function POST(req: NextRequest) {
  const { keyword, sources } = await req.json();
  
  // Crawl multiple news sources
  const results = await crawlSources({
    keyword,
    sources: sources || ['techcrunch', 'a16z', 'twitter'],
    timeframe: '24h'
  });
  
  return NextResponse.json({ data: results });
}

Content Generation Endpoint

// app/api/generate/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { generateContent } from '@/lib/content/generator';

export async function POST(req: NextRequest) {
  const { research, format, language, tone, aiProvider } = await req.json();
  
  const content = await generateContent({
    researchData: research,
    format: format || 'toplist', // toplist, pov, casestudy, howto
    language: language || 'vi', // vi, en, both
    tone: tone || 'professional', // professional, friendly, humorous
    provider: aiProvider || 'claude' // claude, openai
  });
  
  return NextResponse.json({ content });
}

Video Rendering Endpoint

// app/api/render/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { renderVideo } from '@/lib/video/renderer';

export async function POST(req: NextRequest) {
  const { content, platform, template } = await req.json();
  
  const video = await renderVideo({
    content,
    platform: platform || 'reels', // reels, tiktok, shorts
    template: template || 'infographic',
    aspectRatio: platform === 'reels' ? '9:16' : '1:1'
  });
  
  return NextResponse.json({ videoUrl: video.url });
}

Core Modules Usage

AI Content Generation with Claude

// lib/ai/claude.ts
import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

export async function generateWithClaude(prompt: string, systemPrompt?: string) {
  const message = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 4096,
    system: systemPrompt || 'You are an expert content writer.',
    messages: [
      {
        role: 'user',
        content: prompt
      }
    ]
  });
  
  return message.content[0].text;
}

// Usage example
export async function createTopListArticle(research: any, language: string) {
  const prompt = `
Based on this research data: ${JSON.stringify(research)}

Create a toplist article in  with:
- Engaging headline
- 5-7 items with data-backed insights
- Each item with title, description, and key metrics
- Conclusion with actionable takeaways
`;

    (prompt);
}

AI Content Generation with OpenAI

// lib/ai/openai.ts
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export async function generateWithOpenAI(prompt: string, systemPrompt?: string) {
  const completion = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [
      {
        role: 'system',
        content: systemPrompt || 'You are an expert content writer.'
      },
      {
        role: 'user',
        content: prompt
      }
    ],
    temperature: 0.7,
    max_tokens: 4096
  });
  
  return completion.choices[0].message.content;
}

Web Crawler for Research

// lib/crawler/index.ts
import axios from 'axios';

interface CrawlOptions {
  keyword: string;
  sources: string[];
  timeframe: string;
}

export async function crawlSources(options: CrawlOptions) {
  const { keyword, sources, timeframe } = options;
  const results = [];
  
  for (const source of sources) {
    try {
      let data;
      
      if (source === 'techcrunch') {
        data = await crawlTechCrunch(keyword, timeframe);
      } else if (source === 'a16z') {
        data = await crawlA16Z(keyword, timeframe);
      } else if (source === 'twitter') {
        data = await crawlTwitter(keyword, timeframe);
      }
      
      results.push({
        source,
        data,
        crawledAt: new Date().toISOString()
      });
    } catch (error) {
      console.error(, error);
    }
  }
  
   results;
}

  () {
  
   response =  axios.(, {
    : { : keyword, timeframe },
    : {
      : process..,
      : 
    }
  });
  
   response.;
}

  () {
  
   response =  axios.(, {
    : { 
      : keyword,
      : ,
      : 
    },
    : {
      : 
    }
  });
  
   response.;
}

Content Generator

// lib/content/generator.ts
import { generateWithClaude } from '@/lib/ai/claude';
import { generateWithOpenAI } from '@/lib/ai/openai';

interface GenerateContentOptions {
  researchData: any;
  format: 'toplist' | 'pov' | 'casestudy' | 'howto';
  language: 'vi' | 'en' | 'both';
  tone: 'professional' | 'friendly' | 'humorous';
  provider: 'claude' | 'openai';
}

export async function generateContent(options: GenerateContentOptions) {
  const { researchData, format, language, tone, provider } = options;
  
  const systemPrompt = buildSystemPrompt(format, tone);
  const userPrompt = buildUserPrompt(researchData, format, language);
  
  let content;
  if (provider === 'claude') {
    content = await generateWithClaude(userPrompt, systemPrompt);
  } else {
    content = await generateWithOpenAI(userPrompt, systemPrompt);
  }
  
  // Parse and structure the content
   structured = (content, format);
  
  
   (language === ) {
     otherLang =  (structured);
     { : structured, : otherLang };
  }
  
   structured;
}

 ():  {
   toneDescriptions = {
    : ,
    : ,
    : 
  };
  
   ;
}

 ():  {
   formatInstructions = {
    : ,
    : ,
    : ,
    : 
  };
  
   ;
}

 () {
   {
     .(content);
  }  {
    
     {
      : (content),
      : (content),
      : (content, format),
      : (content),
      : (content)
    };
  }
}

Video Rendering with Remotion

// lib/video/renderer.ts
import { bundle } from '@remotion/bundler';
import { renderMedia, selectComposition } from '@remotion/renderer';
import path from 'path';

interface RenderOptions {
  content: any;
  platform: 'reels' | 'tiktok' | 'shorts';
  template: string;
  aspectRatio: string;
}

export async function renderVideo(options: RenderOptions) {
  const { content, platform, template, aspectRatio } = options;
  
  // Bundle the Remotion project
  const bundleLocation = await bundle({
    entryPoint: path.join(process.cwd(), 'remotion/index.ts'),
    webpackOverride: (config) => config,
  });
  
  // Select composition
  const composition = await selectComposition({
    serveUrl: bundleLocation,
    id: template,
    inputProps: {
      content,
      platform,
      aspectRatio
    },
  });
  
  // Render video
   outputLocation = path.(
    process.(),
    ,
    
  );
  
   ({
    composition,
    : bundleLocation,
    : ,
    outputLocation,
    : {
      content,
      platform,
      aspectRatio
    },
  });
  
   {
    : outputLocation.(path.(process.(), ), ),
    : composition.,
    : composition.,
    : composition. / composition.
  };
}

Remotion Video Template

// remotion/compositions/Infographic.tsx
import { AbsoluteFill, Sequence, useCurrentFrame, useVideoConfig } from 'remotion';
import React from 'react';

interface InfographicProps {
  content: {
    headline: string;
    body: Array<{ title: string; description: string }>;
  };
  platform: string;
  aspectRatio: string;
}

export const Infographic: React.FC<InfographicProps> = ({ content, platform }) => {
  const frame = useCurrentFrame();
  const { fps } = useVideoConfig();
  
  return (
    <AbsoluteFill style={{ backgroundColor: '#1a1a1a' }}>
      {/* Intro sequence */}
      <Sequence from={0} durationInFrames={fps * 2}>
        <AbsoluteFill =
           '',
           '',
            / ( * )
        }}>
          
            {content.headline}
          
        
      
      
      {/* Body items */}
      {content.body.map((item, index) => (
        
          
            
              
                {index + 1}. {item.title}
              
              
                {item.description}
              
            
          
        
      ))}
    
  );
};

Frontend Component Example

// app/components/ContentPipeline.tsx
'use client';

import { useState } from 'react';

export default function ContentPipeline() {
  const [keyword, setKeyword] = useState('');
  const [loading, setLoading] = useState(false);
  const [result, setResult] = useState<any>(null);

  const runPipeline = async () => {
    setLoading(true);
    
    try {
      // Step 1: Research
      const researchRes = await fetch('/api/research', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          keyword,
          sources: ['techcrunch', 'twitter']
        })
      });
      const research = await researchRes.json();
      
      // Step 2: Generate Content
      const contentRes = await fetch(, {
        : ,
        : { :  },
        : .({
          : research.,
          : ,
          : ,
          : ,
          : 
        })
      });
       content =  contentRes.();
      
      
       videoRes =  (, {
        : ,
        : { :  },
        : .({
          : content.,
          : ,
          : 
        })
      });
       video =  videoRes.();
      
      ({ research, content, video });
    }  (error) {
      .(, error);
    }  {
      ();
    }
  };

   (
    
  );
}

Common Patterns

Full Pipeline Automation

// lib/pipeline/automation.ts
export async function runFullPipeline(keyword: string, config: any) {
  // 1. Research Phase
  const research = await crawlSources({
    keyword,
    sources: config.sources || ['techcrunch', 'twitter'],
    timeframe: '24h'
  });
  
  // 2. Content Generation Phase
  const content = await generateContent({
    researchData: research,
    format: config.format || 'toplist',
    language: config.language || 'both',
    tone: config.tone || 'professional',
    provider: config.aiProvider || 'claude'
  });
  
  // 3. Video Rendering Phase
  const video = await renderVideo({
    content,
    platform: config.platform || 'reels',
    template: config.template || 'infographic',
    aspectRatio: '9:16'
  });
  
  // 4. Optional: Auto-post to platforms
   (config.) {
     ({
      content,
      video,
      : config. || [, ]
    });
  }
  
   { research, content, video };
}

Batch Processing

// Process multiple keywords
async function batchProcess(keywords: string[]) {
  const results = await Promise.all(
    keywords.map(keyword => 
      runFullPipeline(keyword, {
        format: 'toplist',
        language: 'vi',
        platform: 'reels'
      })
    )
  );
  
  return results;
}

Troubleshooting

API Rate Limits

// lib/utils/rateLimit.ts
import pLimit from 'p-limit';

const limit = pLimit(3); // Max 3 concurrent requests

export async function batchWithRateLimit<T>(
  items: T[],
  fn: (item: T) => Promise<any>
) {
  return Promise.all(
    items.map(item => limit(() => fn(item)))
  );
}

// Usage
const results = await batchWithRateLimit(keywords, async (keyword) => {
  return await crawlSources({ keyword, sources: ['techcrunch'], timeframe: '24h' });
});

Video Rendering Memory Issues

If video rendering fails with memory errors:

// Increase Node.js heap size
// package.json
{
  "scripts": {
    "render": "NODE_OPTIONS='--max-old-space-size=4096' node render.js"
  }
}

Content Quality Improvement

// Add validation and retry logic
async function generateWithRetry(options: any, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    const content = await generateContent(options);
    
    if (validateContent(content)) {
      return content;
    }
    
    console.log(`Retry ${i + 1}/${maxRetries}`);
  }
  
  throw new Error('Failed to generate valid content');
}

function validateContent(content: any): boolean {
  return (
    content.headline?.length > 10 &&
    content.body?.length >= 3 &&
    content.conclusion?.length > 50
  );
}

Environment Variables Missing

// lib/utils/config.ts
export function validateEnv() {
  const required = [
    'ANTHROPIC_API_KEY',
    'OPENAI_API_KEY',
    'RAPIDAPI_KEY'
  ];
  
  const missing = required.filter(key => !process.env[key]);
  
  if (missing.length > 0) {
    throw new Error(`Missing environment variables: ${missing.join(', ')}`);
  }
}

// Call at app startup
validateEnv();

This pipeline system enables complete content automation from research to publication, significantly reducing content creation time while maintaining quality and consistency.