Pagination Handling in n8n API Requests

I learned this the hard way when a “working” workflow quietly stopped after the first 100 records and left a U.S. CRM sync incomplete for hours. Pagination Handling in n8n API Requests is how you keep pulling every page (safely) without duplicates, missing records, or runaway executions.

Why pagination fails in real automations (and what it costs)

When an API paginates results, it’s telling you: “You only get a slice—come back for the next slice.” If your workflow doesn’t handle that contract correctly, you typically see one of these failures:

• Silent truncation: you sync only the first page (common defaults: 50, 100, 250 items).
• Duplicates: offset/page logic re-reads items while new records are inserted upstream.
• Gaps: records fall between pages when sorting isn’t stable or filtering changes mid-run.
• Infinite loops: “next page” conditions never resolve because tokens/offsets aren’t updated correctly.
• Rate-limit pressure: fast pagination spikes request volume and triggers throttling at scale.

In high-value English-speaking markets (think U.S. CRMs, commerce, billing, and lead pipelines), pagination mistakes don’t just “lose data”—they break attribution, revenue reporting, fulfillment, and compliance workflows.

Identify the pagination type before you build the loop

APIs paginate in a few standard ways. You’ll build the cleanest n8n design when you match the loop to the API’s native pattern:

Pagination style	What you receive	How you request the next page	Typical risk
Page-based	page, totalPages, or similar	Increment page=2,3,4…	Duplicates/gaps if data changes
Offset/limit	limit and an implied offset	offset += limit	Most fragile under inserts/deletes
Cursor-based	next cursor/token	Send cursor=NEXT_TOKEN	Token mishandling causes stalls
Link-header (RFC style)	HTTP Link header with rel="next"	Follow the next URL exactly	Parsing mistakes, partial headers
GraphQL connections	edges + pageInfo.endCursor	Send after=endCursor	Wrong query shape inflates payload

Your first practical move: run one request in n8n and inspect both the JSON body and the response headers. Pagination signals often live in headers (especially Link-based pagination), not just in the body.

Core pattern: a controlled pagination loop (without infinite runs)

The most reliable pagination loop in n8n has four properties:

• Stable ordering: the API sorts results consistently (createdAt asc, id asc, updatedAt asc with tie-breakers).
• Deterministic “next”: a clear next token/URL or a page counter you control.
• Hard stop guardrails: max pages, max total items, and “no progress” detection.
• Idempotency strategy: you can re-run safely (dedupe keys, upsert, or checkpointing).

In n8n, you typically implement this with a loop that:

1. Requests a page (HTTP Request node).
2. Emits items (normalize output into an array of records).
3. Computes the next page token/URL (Set/Code node).
4. Stops when there is no next token/URL or when guardrails trip.

n8n makes it fast to build this loop, but here’s the real weakness you must plan around: when your loop logic is “implicit” (spread across multiple nodes with small expressions), it’s easy to ship a workflow that looks correct but fails under edge cases (empty pages, token expiry, partial retries). The fix is to centralize the pagination state (token/page/attempt count) and add explicit stop conditions you can audit.

Cursor-based pagination in n8n (the most production-friendly option)

Cursor pagination is the best fit for large U.S. datasets (CRMs, orders, transactions) because it’s resilient to inserts and deletes while you’re paging. Your workflow should treat the cursor as the source of truth and never “invent” the next page.

Use this approach:

• Store cursor in a single state field (for example: pagination.nextCursor).
• Send the cursor only if it exists; otherwise send the initial request without it.
• Stop immediately when the API returns no next cursor (or returns an empty string/null).

Code node (JS): extract next cursor safely and add guardrails

/**

 * Incoming item contains API response body in: $json.body

 * Adjust paths to match your API.

 */

const body = $json.body || {};

const records = Array.isArray(body.results) ? body.results : (Array.isArray(body.data) ? body.data : []);

const nextCursor = body.paging?.next?.after ?? body.next_cursor ?? body.nextCursor ?? null;

const pageInfo = {

  count: records.length,

  nextCursor: nextCursor,

  // Guardrails: stop if the API returns a cursor but no records (often a loop bug)

  shouldStop: records.length === 0 || !nextCursor,

};

return [

  {

    json: {

      records,

      pagination: pageInfo,

    },

  },

];

Common failure: you keep requesting the same cursor because the “nextCursor” path is wrong (or because you’re reading the cursor from the previous item). Fix: always compute nextCursor from the current response, and set a “no-progress” stop: if the cursor doesn’t change for two iterations, stop and alert.

Page-based and offset/limit pagination (use extra safety)

Some APIs still use page numbers or offset/limit. You can make them work reliably, but only if you take stability seriously:

• Request with a stable sort (if supported) and include a tie-breaker field.
• Prefer “updatedSince + stable sort” patterns when supported, rather than offset-only scans.
• Checkpoint the last processed record key (id, updatedAt) so retries don’t duplicate.

Offset/limit is the most fragile style when new records are being inserted (typical in U.S. ecommerce and lead pipelines). Your best workaround is to reduce reliance on offsets by adding a deterministic filter window (for example, updatedAt range) and paging within that range.

Link-header pagination (follow the URL exactly)

Some APIs return pagination links in the HTTP Link header (often used in developer-first platforms). In this case, you should not reconstruct URLs manually. Parse the header and request the “next” URL as-is.

Code node (JS): parse Link header and get rel="next" URL

/**

 * Link header example:

 * <https://api.example.com/items?cursor=abc>; rel="next", <...>; rel="prev"

 */

const link = $json.headers?.link || $json.headers?.Link || "";

let nextUrl = null;

if (link) {

  const parts = link.split(",");

  for (const p of parts) {

    const matchUrl = p.match(/<([^>]+)>/);

    const matchRel = p.match(/rel="([^"]+)"/);

    if (matchUrl && matchRel && matchRel[1] === "next") {

      nextUrl = matchUrl[1];

      break;

    }

  }

}

return [{ json: { nextUrl } }];

Common failure: the Link header is truncated because you’re not capturing response headers in your HTTP Request node configuration. Fix: enable response headers output and verify the header key casing (some environments expose it as Link vs link).

Real-world API examples you’ll meet in U.S. stacks

HubSpot CRM exports and backfills

HubSpot API pagination is widely used for contacts, companies, deals, and engagements. The biggest weakness you’ll run into is that large backfills can collide with rate limits and internal processing delays when you page aggressively. The practical fix is to pace pagination (small concurrency, short waits between page pulls) and to checkpoint your cursor so you can resume without restarting the entire export.

Shopify orders and fulfillment operations

Shopify Admin API is a classic case where pagination is only half the battle—payload size and field selection matter. A common weakness is fetching “full” order objects across many pages and hitting response size/latency issues that cause timeouts. The fix is to request only the fields you truly need for your downstream system, then enrich selectively (for example, fetch line items only when an order qualifies for a follow-up step).

Salesforce data pulls for reporting and sync

Salesforce REST API workflows often combine pagination with query semantics. The weakness here is not just paging—it’s inconsistent results when your query’s ordering isn’t stable or when your integration user lacks access to certain records mid-run. The fix is to keep queries deterministic (ordered by a stable field), handle partial visibility as a first-class outcome, and write sync logic that upserts by Salesforce Id so retries stay safe.

Dedupe and consistency: the part most workflows skip

If you’re pushing paginated data into a U.S.-centric system (CRM, billing, fulfillment, analytics), assume retries will happen. Build for it:

• Use an upsert strategy: write by a stable external ID (or compute a dedupe key).
• Keep a checkpoint: store last cursor/page processed in a durable place (DB, KV, or your target system).
• Detect anomalies: if a page returns fewer items than expected repeatedly, alert before you trust the sync.
• Stop on “no progress”: if next cursor/page doesn’t change, fail fast instead of looping.

A clean approach is to emit each record with a computed key and let your destination enforce uniqueness. When that’s not possible, build a lightweight dedupe step (for example, track seen IDs within a run and skip duplicates).

Common pagination mistakes in n8n (and the fixes that actually stick)

• Mistake: Building the next URL manually and accidentally dropping query parameters.
  Fix: Prefer cursor/token fields or Link headers; if you must build URLs, build them from a single “state” object.
• Mistake: Looping until “items length > 0” without checking next token.
  Fix: Stop when next token is missing; treat empty pages as an error unless the API explicitly documents them.
• Mistake: Paging with offset while data is changing (new leads/orders arriving).
  Fix: Use cursor pagination when available, or apply a stable time window + ordering.
• Mistake: No upper bounds on total pages/items.
  Fix: Add guardrails: max pages per run and max total records before you fail.

FAQ

How do you stop an n8n pagination loop from running forever?

Stop on the absence of a next cursor/URL, add a max-pages cap, and add a “no progress” rule: if the next cursor/page value repeats (or doesn’t change) across iterations, stop and alert instead of continuing.

What’s the safest pagination strategy for fast-changing datasets like leads and orders?

Use cursor-based pagination whenever the API supports it. If you’re forced into offset/page-based pagination, enforce stable ordering and page within a fixed filter window (such as a bounded updatedAt range) to reduce shifting results.

How do you handle pagination when an API uses a Link header instead of JSON fields?

Capture response headers in your HTTP Request node and parse the Link header for rel="next". Request that URL exactly rather than reconstructing it, and stop when no rel="next" URL exists.

How do you avoid duplicates when a workflow retries halfway through pagination?

Upsert into your destination using a stable unique identifier (like a CRM record ID), or compute a dedupe key and enforce uniqueness downstream. Also checkpoint pagination state so you can resume from the last known cursor/page without restarting from page one.

When should you split paginated results into batches inside n8n?

Batch after retrieval when the destination has strict request limits (CRMs, email platforms, billing APIs). Pull pages safely first, then process records in controlled batches so a destination failure doesn’t force you to re-run the entire pagination sequence.

What’s the fastest way to validate your pagination logic before you run it on production data?

Run the first three iterations only, log the cursor/page progression, and verify record uniqueness across those iterations. If you see repeated IDs or an unchanged next cursor, fix the state logic before scaling up.

Read also: Idempotency in n8n: Prevent Duplicate Actions

Closing note

Once you treat pagination as state management (not just “repeat the request”), your n8n workflows become predictable under load and safe to retry. Lock down the pagination style, add explicit stop conditions, checkpoint progress, and your API pulls will stay reliable even when U.S.-scale datasets and real-time inserts push the edges.