Reading a Paginated API Without Holding the Whole Thing in Memory

Your API hands out 50 records at a time across 400 pages. You need all of them. You do not need them all at once. Here's a very familiar situation that shows up constantly on the backend. Some API returns data in pages, 50 or 100 records at a time, and you need to walk every page: sync them to your database, export them to a file, run a report. The endpoint gives you a cursor or a page number and you keep asking until there's nothing left. The way most of us write it the first time looks like this: async function getAllRecords () { const all = []; let cursor = 0 ; while ( cursor !== null ) { const { records , nextCursor } = await fetchPage ( cursor ); all . push (... records ); cursor = nextCursor ; } return all ; } const everything = await getAllRecords (); for ( const record of everything ) { process ( record ); } It works. At four hundred records it's fine. The trouble starts when the dataset grows, and it has three separate problems hiding in it. It holds the entire dataset in memory before you touch a single record. It's all or nothing: if page 380 fails, you've thrown away the 19,000 records you already fetched . And it's eager. You can't start processing record one until the very last page has landed , even if all you wanted was the first ten. There's a shape in JavaScript built for exactly this, and if you read the first two posts in this series you already have both halves of it. Two ideas you've already seen In the CSV post , we pulled rows out of a huge file one at a time with a generator, so the file never fully loaded into memory. Lazy. Pull-based. You ask for the next row, you get the next row, nothing more. In the async/await post , we saw that a generator can pause at a yield and resume later.A generator can hold its place across an asynchronous gap. Put those together. A generator that pulls data lazily, and can pause to await something between pulls. That's an async generator, and it's the natural tool for walking a paginated API. You pull records