Web Scraping Pagination: URLs, Buttons, and Infinite Scroll

1.Identify the pagination model before writing the loop

Open page 2 and inspect both the address bar and the browser's Network panel. If the URL changes to something like `?page=2` or `/page/2/`, the next document is probably fetchable directly. If the page contains an `<a>` element with a next relation, follow its actual href instead of guessing a sequence. The HTTP Link header can also advertise `next`, `prev`, `first`, and `last` relationships without visible links.

When the address does not change, inspect the requests triggered by the control. A JSON request containing `cursor`, `after`, `offset`, or `pageInfo` often reveals a paginated API. If the state cannot be reproduced with an authorized request, keep the work in a browser session.

• Numbered URL: request each discovered URL and stop when the target's terminal condition is confirmed.
• Next link: parse the current document and resolve its next href relative to the current URL.
• Cursor API: send the opaque cursor returned by the previous response; never manufacture cursor values.
• Load-more button: click while the control exists and each action adds new item identities.
• Infinite scroll: scroll a meaningful sentinel into view and wait for a measurable content change.

2.Prefer discovered links and URLs over browser clicks

Direct page requests are easier to cache, retry, and resume than a stateful browser. Treat the link destination as authoritative: its query may preserve category, sort order, locale, or a continuation token that a guessed page URL would lose.

The Python example below uses the public Quotes to Scrape practice site also used by Scrapy's official link-following tutorial. It asks OmniScrape for each page, resolves the next link with `urljoin`, and stores a stable item key. A visited-URL set catches navigation loops, while a no-new-records check catches pages that repeat under different URLs.

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960import os
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

API_URL = "https://api.omniscrape.io/v1/scrape"
START_URL = "https://quotes.toscrape.com/page/1/"
MAX_PAGES = 50


def fetch_html(url: str) -> str:
    response = requests.post(
        API_URL,
        headers={"X-API-Key": os.environ["OMNISCRAPE_KEY"]},
        json={"url": url, "mode": "auto", "output_format": "html"},
        timeout=90,
    )
    response.raise_for_status()
    body = response.json()
    if not body.get("success"):
        raise RuntimeError(body.get("error", "scrape failed"))
    return body["data"]["content"]


def scrape_all_pages() -> tuple[list[dict], str]:
    url = START_URL
    visited_urls: set[str] = set()
    seen_items: set[tuple[str, str]] = set()
    records: list[dict] = []

    for _ in range(MAX_PAGES):
        if url in visited_urls:
            return records, "repeated URL"
        visited_urls.add(url)

        soup = BeautifulSoup(fetch_html(url), "html.parser")
        new_on_page = 0
        for card in soup.select(".quote"):
            text = card.select_one(".text").get_text(strip=True)
            author = card.select_one(".author").get_text(strip=True)
            key = (text, author)
            if key not in seen_items:
                seen_items.add(key)
                records.append({"text": text, "author": author, "source": url})
                new_on_page += 1

        if new_on_page == 0:
            return records, "no new records"

        next_link = soup.select_one("li.next a[href]")
        if next_link is None:
            return records, "next link absent"
        url = urljoin(url, next_link["href"])

    return records, f"safety limit reached ({MAX_PAGES} pages)"


items, stop_reason = scrape_all_pages()
print({"unique_records": len(items), "stop_reason": stop_reason})

3.Use explicit stopping conditions, not an empty-page guess

A robust loop has a natural terminal signal and a safety limit. Natural signals include an absent next link, null cursor, disabled control, or `has_next_page: false`. Safety limits include maximum pages, records, elapsed time, and repeated-state detection. Persist the final signal so truncated runs are distinguishable from complete ones.

Do not assume HTTP 404 means the end. Some sites redirect out-of-range pages to page 1, repeat the last page, or return an empty HTTP 200 template. Stop and flag a page that adds no new stable item identities.

• Natural stop: next link missing, cursor null, button absent, or an explicit final-page field.
• Progress check: each successful iteration must add at least one previously unseen record identity.
• Loop check: reject a repeated normalized URL, cursor, or response fingerprint.
• Safety cap: bound pages, records, elapsed time, and retry attempts independently.
• Audit field: persist stop reason, last URL or cursor, page count, and unique-record count.

4.Treat cursor pagination as a sequential chain

Cursor APIs return an opaque token that points after the current result window. Send that token unchanged in the next authorized request and stop when the response omits it or declares that no next page exists. Do not increment, decode, or predict cursors: they may embed ordering state, filters, expiry, or a signature. Because page N+1 depends on page N's response, one cursor chain is usually sequential even when separate categories can run in parallel.

Checkpoint the last fully persisted cursor with its query, sort, and filters. Resume only when the provider guarantees a stable traversal; otherwise restart and use record-level upserts. In a changing catalog, stable deduplication matters more than the nominal page count.

123456789101112131415161718cursor = None
seen_cursors = set()

while True:
    payload = {"limit": 100, "after": cursor}
    page = authorized_api_request(payload)
    upsert_by_stable_id(page["items"])

    next_cursor = page.get("page_info", {}).get("end_cursor")
    has_next = page.get("page_info", {}).get("has_next_page", False)
    if not has_next or not next_cursor:
        break
    if next_cursor in seen_cursors:
        raise RuntimeError("Pagination cursor repeated")

    seen_cursors.add(next_cursor)
    cursor = next_cursor
    save_checkpoint(cursor)

5.Wait for content growth after load more or infinite scroll

Use a browser only when pagination requires stateful interaction. Playwright locators wait for actionability before a click, as its actionability guide documents, but a click still does not prove new data arrived. Measure the item count before the action and wait for growth. On infinite lists, scroll the final item or sentinel into view rather than moving an arbitrary number of pixels; see Playwright's locator API.

This Browser-as-a-Service pattern exits when the button disappears, content stops growing twice, or the cap is reached. Replace its selectors with stable attributes from an authorized target. Avoid fixed sleeps, which waste time on fast responses and still race slow ones.

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647import asyncio
import os
from playwright.async_api import async_playwright, TimeoutError as PlaywrightTimeout

WS_URL = (
    "wss://browser.omniscrape.io"
    f"?apikey={os.environ['OMNISCRAPE_KEY']}&render_media=false"
)


async def collect_paginated_cards(url: str) -> list[str]:
    async with async_playwright() as p:
        browser = await p.chromium.connect_over_cdp(WS_URL)
        try:
            page = await browser.contexts[0].new_page()
            page.set_default_timeout(15_000)
            await page.goto(url, wait_until="domcontentloaded", timeout=30_000)

            cards = page.locator("[data-testid='result-card']")
            stagnant_attempts = 0

            for _ in range(40):
                before = await cards.count()
                button = page.get_by_role("button", name="Load more")

                if await button.count() == 0 or not await button.is_visible():
                    break
                await button.click()

                try:
                    await page.wait_for_function(
                        "([selector, count]) => document.querySelectorAll(selector).length > count",
                        ["[data-testid='result-card']", before],
                        timeout=10_000,
                    )
                    stagnant_attempts = 0
                except PlaywrightTimeout:
                    stagnant_attempts += 1
                    if stagnant_attempts >= 2:
                        break

            return list(dict.fromkeys(await cards.all_inner_texts()))
        finally:
            await browser.close()


results = asyncio.run(collect_paginated_cards("https://public.example/catalog"))

6.Match the retrieval method to the pagination state

Use OmniScrape Web Unlocker for independent URLs that need proxy routing, rendering, or challenge handling. Its documented actions include `scroll_depth`, `load_more`, and `next_page`. When you must inspect intermediate state or retain cookies across actions, Browser-as-a-Service over CDP gives Playwright full control. Start with the least stateful method that returns validated, complete data.

• Independent numbered or next-link pages: one Web Unlocker request per discovered URL.
• Authorized JSON cursor endpoint: direct HTTP requests with sequential cursor checkpointing.
• One bounded scroll or load-more action: Web Unlocker JavaScript actions may be sufficient.
• Multi-step, stateful interaction: Playwright or Puppeteer through Browser-as-a-Service.

7.Deduplicate records, not just page URLs

Pagination boundaries are not transactions. Insertions can push an older item onto the next page, while deletions can pull an unseen item backward. Upsert by a target ID or canonical detail URL; use a text hash only when no stable identity exists. Keep sort, filters, locale, and relevant session state constant. For rapidly changing sources, overlap a small window on incremental runs and deduplicate downstream.

8.Validate pagination completeness in production

Log the URL or redacted cursor, status, extracted and new-unique counts, duration, retries, and next-state presence per page. Alert when duplicate ratio rises, items per page changes sharply, totals disagree, or jobs increasingly hit a safety cap. Save a redacted response or screenshot for failed validations. The web scraping data pipeline guide covers durable checkpoints, while web scraping timeouts covers bounded deadlines.

• Assert required fields and a plausible record count on every page.
• Compare reported totals with cumulative unique records when the site exposes a total.
• Persist the last completed page or cursor only after its records are durably written.
• Separate transport success, extraction success, and pagination completeness metrics.
• Test first, middle, and final pages after any selector or target layout change.

9.Web scraping pagination checklist

A production loop should finish with evidence: unique records, pages attempted, a durable checkpoint, and a stop reason. For independent pages that need protected retrieval or rendering, start with OmniScrape Web Unlocker. Collect only authorized data and respect applicable law, terms, access controls, privacy obligations, robots directives where applicable, and rate limits. Pagination increases load quickly, so every job needs a safety cap.

• Classify the target as URL, next-link, cursor, load-more, or infinite-scroll pagination.
• Follow discovered hrefs or returned cursors instead of guessing the next state.
• Use stable record identities and idempotent upserts across page boundaries.
• Define natural, progress, loop, and safety stopping conditions.
• Checkpoint only after records are persisted and log the final stop reason.
• Use browser automation only when stateful interaction is genuinely required.
• Validate content and totals; HTTP 200 alone does not prove completeness.