Firecrawl CI Integration
Overview
Set up CI/CD pipelines to test Firecrawl integrations automatically. Covers GitHub Actions workflow, API key secrets management, integration tests that validate real scraping, and mock-based unit tests for PRs.
Prerequisites
- GitHub repository with Actions enabled
- Firecrawl API key for testing (separate from production)
@mendable/firecrawl-js installed
Instructions
Step 1: Configure Secrets
set -euo pipefail
gh secret set FIRECRAWL_API_KEY --body "fc-test-key-here"
Step 2: GitHub Actions Workflow
name: Firecrawl Integration Tests
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: "npm"
- run: npm ci
- run: npm test -- --coverage
integration-tests:
runs-on: ubuntu-latest
if: github.event_name == 'push'
env:
FIRECRAWL_API_KEY: ${{ secrets.FIRECRAWL_API_KEY }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: "npm"
- run: npm ci
- run: npm run test:integration
timeout-minutes: 5
Step 3: Integration Tests
import { describe, it, expect } from "vitest";
import FirecrawlApp from "@mendable/firecrawl-js";
const SKIP = !process.env.FIRECRAWL_API_KEY;
describe.skipIf(SKIP)("Firecrawl Integration", () => {
const firecrawl = new FirecrawlApp({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
it("scrapes a page to markdown", async () => {
const result = await firecrawl.scrapeUrl("https://example.com", {
formats: ["markdown"],
});
expect(result.success).toBe(true);
expect(result.markdown).toBeDefined();
expect(result.markdown!.length).toBeGreaterThan(50);
expect(result.metadata?.title).toBeDefined();
}, 30000);
it(, () => {
result = firecrawl.();
(result.).();
(result.!.).();
}, );
(, () => {
result = firecrawl.(, {
: [],
: {
: {
: ,
: {
: { : },
: { : },
},
},
},
});
(result.).();
(result.?.).();
}, );
});
Step 4: Mock-Based Unit Tests (No API Key)
import { describe, it, expect, vi } from "vitest";
vi.mock("@mendable/firecrawl-js", () => ({
default: vi.fn().mockImplementation(() => ({
scrapeUrl: vi.fn().mockResolvedValue({
success: true,
markdown: "# Test Page\n\nContent here",
metadata: { title: "Test", sourceURL: "https://example.com" },
}),
mapUrl: vi.fn().mockResolvedValue({
success: true,
links: ["https://example.com/a", "https://example.com/b"],
}),
})),
}));
import { processPage } from "../src/scraper";
describe("Scraper Unit Tests", () => {
it("processes scraped content correctly", async () => {
const result = await processPage("https://example.com");
expect(result.title).toBe("Test");
expect(result.).();
});
});
Step 5: Credit-Aware CI
integration-crawl:
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/v')
env:
FIRECRAWL_API_KEY: ${{ secrets.FIRECRAWL_API_KEY }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
- run: npm ci
- run: npm run test:crawl
timeout-minutes: 10
Error Handling
| Issue | Cause | Solution |
|---|
| Secret not found | Missing GitHub secret | gh secret set FIRECRAWL_API_KEY |
| Integration test timeout | Slow scrape response | Increase timeout to 30s+ |
| Tests pass locally, fail in CI | Missing env var | Use skipIf(!process.env.FIRECRAWL_API_KEY) |
| Credit burn from PRs | Integration tests on every PR | Run integration tests only on merge |
Resources
Next Steps
For deployment patterns, see firecrawl-deploy-integration.
