Why a CRM Re-Engagement Sync Needs Three Layers of Pagination, Not One

One Loop Isn't the Same Problem at Every Scale

A pagination loop that walks a list of records a page at a time works fine when the list is small enough to finish in a handful of requests. Point that same loop at a CRM with 50,000+ contacts and run it every day, and the assumptions baked into "just page through it" start to break. Most CRM and API pagination is designed around one dataset at a time — a page of contacts, a page of activities — but a re-engagement job that reasons about idle contacts, pulls their history, and sends follow-ups is really several data-fetching problems stacked on top of each other, and each one paginates independently.

More Than One Thing Can Need Paging

A job shaped like this — walking a CRM, building context per contact, and tracking what's been sent — generally has to page through more than one kind of data, whether or not an implementation names them as separate layers. There's the contact list itself: finding everyone who has gone idle for 60-90 days means paging through the CRM's full contact set, not just the first page an API hands back by default. There's the data hanging off each contact: building a follow-up worth sending means reading enough of that person's history to have real context, and a contact with a long history can itself return that data across multiple pages. And there's the state of the run itself: which contacts have already been evaluated, drafted for, or sent to, so a job that gets interrupted or rate-limited partway through doesn't have to guess where it left off.

Treat any one of these as a single flat loop and it usually still works — until the dataset grows past whatever page size an API defaults to, at which point records get silently skipped or, worse, re-processed and re-sent.

Why This Has to Be a Day-One Decision

None of this is difficult in isolation. What's easy to miss is that these pagination concerns interact: incomplete pagination on the contact list compounds with incomplete pagination on a contact's history, because the job may now be reasoning about a person it hasn't fully iterated in the first place. And a job with no persisted state for what it has already done has no way to resume cleanly — it either reprocesses records it already handled, which risks a duplicate send, or it restarts from scratch, which stops scaling as the dataset grows.

Retrofitting pagination discipline after a daily job has started missing or duplicating records is a much harder fix than designing for it up front, because by then the failure mode is already live against real data every day. For any sync job that touches a growing dataset — not just this one — the design-time question isn't whether the API paginates. It's how many separate things in the pipeline need their own pagination, and whether the job can resume each of them independently.