cursor-plugin-convex-rule-use-pagination-for-large-datasets

name	cursor-plugin-convex-rule-use-pagination-for-large-datasets
description	Use cursor-based pagination for large datasets instead of .collect(). Prevents performance issues and provides smooth infinite scroll.
metadata	{"version":"0.1.0"}

Use Pagination for Large Datasets

Never use .collect() on large or unbounded queries. Use Convex's cursor-based pagination instead.

The Problem with .collect()

❌ Bad: Loading Everything

export const getAllTasks = query({
  handler: async (ctx) => {
    // ❌ Loads ALL tasks - slow and breaks with large data
    return await ctx.db.query("tasks").collect();
  },
});

Problems:

• Slow for large datasets (100s-1000s of items)
• Wastes bandwidth
• Poor user experience (long loading)
• Can hit memory limits
• Doesn't scale

✅ Good: Pagination

export const getTasks = query({
  args: {
    paginationOpts: paginationOptsValidator,
  },
  handler: async (ctx, args) => {
    return await ctx.db
      .query("tasks")
      .order("desc")
      .paginate(args.paginationOpts);
  },
});

Benefits:

• Fast (only loads what's needed)
• Scales to millions of items
• Smooth infinite scroll
• Reactive updates
• Better UX

When to Paginate

✅ Always Paginate:

• User-generated content (posts, comments, messages)
• Activity feeds and timelines
• Search results
• Notification lists
• Any list that could grow unbounded
• Lists with > 100 items

⚠️ Maybe Don't Paginate:

• Small, bounded lists (user's 3 favorite colors)
• Configuration options (< 20 items)
• Dropdown menus with limited options
• Tags or categories (if truly limited)

Rule of thumb: If it could grow to 100+ items, paginate!

Basic Pagination

Backend: Query with Pagination

import { query } from "./_generated/server";
import { paginationOptsValidator } from "convex/server";

export const listTasks = query({
  args: {
    paginationOpts: paginationOptsValidator,
  },
  handler: async (ctx, args) => {
    const user = await getCurrentUser(ctx);

    return await ctx.db
      .query("tasks")
      .withIndex("by_user", q => q.eq("userId", user._id))
      .order("desc")
      .paginate(args.paginationOpts);
  },
});

Returns:

{
  page: Doc<"tasks">[],        // Current page of results
  continueCursor: string,      // Cursor for next page
  isDone: boolean,             // True if no more pages
}

Frontend: usePaginatedQuery

import { usePaginatedQuery } from "convex/react";
import { api } from "../convex/_generated/api";

function TaskList() {
  const { results, status, loadMore } = usePaginatedQuery(
    api.tasks.listTasks,
    {},
    { initialNumItems: 20 }
  );

  return (
    <div>
      {results?.map(task => (
        <TaskItem key={task._id} task={task} />
      ))}

      {status === "CanLoadMore" && (
        <button onClick={() => loadMore(20)}>Load More</button>
      )}

      {status === "LoadingMore" && <div>Loading...</div>}
    </div>
  );
}

Infinite Scroll Pattern

import { useEffect, useRef } from "react";

function InfiniteTaskList() {
  const { results, status, loadMore } = usePaginatedQuery(
    api.tasks.listTasks,
    {},
    { initialNumItems: 20 }
  );

  const observerRef = useRef<IntersectionObserver>();
  const loadMoreRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    if (observerRef.current) observerRef.current.disconnect();

    observerRef.current = new IntersectionObserver((entries) => {
      if (entries[0].isIntersecting && status === "CanLoadMore") {
        loadMore(20);
      }
    });

    if (loadMoreRef.current) {
      observerRef.current.observe(loadMoreRef.current);
    }

    return () => observerRef.current?.disconnect();
  }, [status, loadMore]);

  return (
    <div>
      {results?.map(task => (
        <TaskItem = = />
      ))}

      {status === "CanLoadMore" && (
        
          Loading more...
        
      )}
    
  );
}

Pagination with Filters

export const listTasks = query({
  args: {
    status: v.optional(v.union(
      v.literal("todo"),
      v.literal("done")
    )),
    paginationOpts: paginationOptsValidator,
  },
  handler: async (ctx, args) => {
    const user = await getCurrentUser(ctx);

    let query = ctx.db
      .query("tasks")
      .withIndex("by_user", q => q.eq("userId", user._id));

    // Filter in TypeScript after query
    const results = await query.order("desc").paginate(args.paginationOpts);

    if (args.status) {
      return {
        ...results,
        page: results.page.filter(task => task.status === args.status),
      };
    }

    return results;
  },
});

Note: For better performance, use compound indexes:

// schema.ts
tasks: defineTable({
  userId: v.id("users"),
  status: v.string(),
}).index("by_user_and_status", ["userId", "status"])

// Query with index
export const listTasks = query({
  args: {
    status: v.union(v.literal("todo"), v.literal("done")),
    paginationOpts: paginationOptsValidator,
  },
  handler: async (ctx, args) => {
    const user = await getCurrentUser(ctx);

    return await ctx.db
      .query("tasks")
      .withIndex("by_user_and_status", q =>
        q.eq("userId", user._id).eq("status", args.status)
      )
      .order("desc")
      .paginate(args.paginationOpts);
  },
});

Reactive Pagination

Convex pagination is fully reactive! When data changes, pages update automatically.

The Challenge (Other Systems)

With offset-based pagination, insertions/deletions cause:

• Duplicate items across pages
• Missing items
• Inconsistent pagination

Convex Solution

Convex uses cursor-based pagination with automatic tracking:

// Items are inserted/deleted while user scrolls
// Convex automatically handles this!

const { results } = usePaginatedQuery(api.tasks.listTasks, {}, { initialNumItems: 20 });

// Results stay consistent even as data changes
// No duplicates, no missing items

How it works:

• Cursors track positions, not offsets
• Pages automatically adjust to data changes
• Seamless reactive updates

Advanced: Custom Pagination

For complex cases, use getPage from convex-helpers:

import { getPage } from "convex-helpers/server/pagination";

export const customPagination = query({
  args: {
    category: v.string(),
    paginationOpts: paginationOptsValidator,
  },
  handler: async (ctx, args) => {
    const user = await getCurrentUser(ctx);

    // Custom query logic
    const results = await ctx.db
      .query("posts")
      .withIndex("by_user", q => q.eq("userId", user._id))
      .filter(q => q.eq(q.field("category"), args.category))
      .collect();

    // Apply custom pagination
    return getPage(results, args.paginationOpts);
  },
});

Search with Pagination

export const searchTasks = query({
  args: {
    searchTerm: v.string(),
    paginationOpts: paginationOptsValidator,
  },
  handler: async (ctx, args) => {
    const user = await getCurrentUser(ctx);

    const allTasks = await ctx.db
      .query("tasks")
      .withIndex("by_user", q => q.eq("userId", user._id))
      .collect();

    // Filter by search term
    const filtered = allTasks.filter(task =>
      task.title.toLowerCase().includes(args.searchTerm.toLowerCase())
    );

    // Paginate filtered results
    return getPage(filtered, args.paginationOpts);
  },
});

Better: Use Convex's built-in text search when available.

Pagination Options

interface PaginationOptions {
  numItems: number;      // How many items per page
  cursor: string | null; // Cursor from previous page (null for first page)
  id?: string;          // Optional: Specific ID to paginate from
}

// Usage
const page1 = await query.paginate({ numItems: 20, cursor: null });
const page2 = await query.paginate({ numItems: 20, cursor: page1.continueCursor });

Common Patterns

Pattern 1: Load More Button

const { results, status, loadMore } = usePaginatedQuery(
  api.tasks.list,
  {},
  { initialNumItems: 20 }
);

<button
  onClick={() => loadMore(20)}
  disabled={status !== "CanLoadMore"}
>
  {status === "LoadingMore" ? "Loading..." : "Load More"}
</button>

Pattern 2: Show/Hide Older Items

const [showAll, setShowAll] = useState(false);

const { results } = usePaginatedQuery(
  api.tasks.list,
  {},
  { initialNumItems: showAll ? 100 : 10 }
);

<button onClick={() => setShowAll(!showAll)}>
  {showAll ? "Show Less" : "Show All"}
</button>

Pattern 3: Paginated Tabs

const [activeTab, setActiveTab] = useState<"todo" | "done">("todo");

const { results } = usePaginatedQuery(
  api.tasks.listByStatus,
  { status: activeTab },
  { initialNumItems: 20 }
);

// Separate pagination per tab

Performance Tips

1. Use Indexes

   // ✅ Fast
   .withIndex("by_user", q => q.eq("userId", userId))
   .paginate(opts)
   
   // ❌ Slow
   .filter(q => q.eq(q.field("userId"), userId))
   .paginate(opts)
   

2. Reasonable Page Sizes

   // ✅ Good: 10-50 items per page
   { initialNumItems: 20 }
   
   // ❌ Too small: Too many requests
   { initialNumItems: 5 }
   
   // ❌ Too large: Slow loading
   { initialNumItems: 500 }
   

3. Order Matters

   // ✅ Use .order() for consistent pagination
   .query("tasks")
   .withIndex("by_created")
   .order("desc")
   .paginate(opts)
   

Debugging Pagination

Check Total Items (Dev Only)

export const debugCount = query({
  handler: async (ctx) => {
    const count = (await ctx.db.query("tasks").collect()).length;
    console.log(`Total tasks: ${count}`);
    return count;
  },
});

Log Pagination State

const { results, status } = usePaginatedQuery(api.tasks.list, {}, { initialNumItems: 20 });

console.log({
  itemsLoaded: results?.length,
  status,
  canLoadMore: status === "CanLoadMore"
});

Checklist

• Never use .collect() on unbounded queries
• Use .paginate() for lists that could grow large
• Set reasonable initialNumItems (10-50)
• Use usePaginatedQuery on frontend
• Provide "Load More" or infinite scroll UX
• Use indexes for paginated queries
• Add .order() for consistent results
• Test with large datasets (100+ items)
• Consider compound indexes for filtered pagination

Learn More

• Convex Pagination Docs
• Take Control of Pagination
• Fully Reactive Pagination
• Queries that Scale
• convex-helpers pagination