Web Scraping Rate Limits: Handle HTTP 429 Correctly

1.HTTP 429 identifies a rate limit, not its cause

RFC 6585 defines 429 Too Many Requests but leaves the counting method to the server. A limit may be keyed by IP address, account, API key, cookie, route, or a combination. It may measure a time window or concurrent work, so a new IP does not necessarily reset it.

Do not group 429 with every access failure. HTTP 403 often represents authorization or policy enforcement; 503 indicates temporary service unavailability; a connection timeout means no response completed. These conditions need different controls. Log the status, response body category, relevant headers, target hostname, credential or tenant identifier, and current in-flight count before selecting a remedy.

• 429 from the target: reduce traffic to that target and honor its published policy.
• 429 from a scraping API: reduce account-wide request pressure or concurrent jobs.
• 403 or login redirect: investigate authorization, access policy, or expired state; do not retry as a rate limit.
• 503 with Retry-After: pause for service recovery, but track it separately from client rate limiting.

2.Find the layer that returned the 429

A managed scraper has at least two HTTP conversations: your client calls the scraping service, and the service calls the target. With OmniScrape, an outer HTTP 429 is an API-layer signal. When the API returns a JSON response, inspect `data.status_code` for the target's HTTP status as documented in the scrape API reference. This distinction tells you whether to slow all OmniScrape workers or only jobs for one target domain.

Use the current OmniScrape rate-limit guide and error reference for platform behavior. Keep account capacity in configuration instead of copying a possibly stale plan limit into code.

• All domains receive outer 429s: inspect the shared API account budget and total in-flight work.
• One domain returns target status 429: apply a domain-specific cooldown and lower its request rate.
• One credential or tenant fails: isolate its queue; do not pause unrelated tenants automatically.
• 429s begin only after scaling workers: the limit is probably shared, while each worker assumes it owns the full budget.

3.Treat Retry-After as the minimum cooldown

The `Retry-After` field can contain either delay-seconds or an HTTP date. Both forms are defined by RFC 9110. Parse both, clamp past dates to zero, and account for clock skew when a server sends a date. If the field is absent or malformed, fall back to capped exponential backoff with random jitter.

A fixed sleep in each worker is not enough. If 100 workers receive 429 together and sleep for the same duration, they wake together and create another burst. Publish the longest active cooldown to shared state and add jitter to unscheduled retries. `Retry-After` is a lower bound: local policy may wait longer when recent attempts are still failing.

12345678910111213141516171819from datetime import datetime, timezone
from email.utils import parsedate_to_datetime


def retry_after_seconds(value: str | None) -> float | None:
    if not value:
        return None

    value = value.strip()
    if value.isdigit():
        return float(value)

    try:
        retry_at = parsedate_to_datetime(value)
        if retry_at.tzinfo is None:
            retry_at = retry_at.replace(tzinfo=timezone.utc)
        return max(0.0, (retry_at - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError, OverflowError):
        return None

4.Combine bounded concurrency with a shared cooldown

A semaphore limits simultaneous requests, but it is not a rate limiter. Ten short requests can pass through one slot much faster than ten long requests. Use the semaphore to protect connection and job capacity, then add a time-based budget when the server publishes a request rate. Python's official `asyncio.Semaphore` documentation describes the in-process concurrency primitive.

The example checks both status layers and shares a cooldown in one process. In a multi-process deployment, move `cooldown_until` to Redis or another atomic store. Its configuration values are examples; derive yours from current policy and measured workload.

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980import asyncio
import os
import random
import time

import httpx

API_URL = "https://api.omniscrape.io/v1/scrape"


class RateAwareScraper:
    def __init__(self, max_in_flight: int = 4) -> None:
        self.semaphore = asyncio.Semaphore(max_in_flight)
        self.cooldown_until = 0.0
        self.cooldown_lock = asyncio.Lock()
        self.client = httpx.AsyncClient(
            headers={"X-API-Key": os.environ["OMNISCRAPE_KEY"]},
            timeout=90,
            limits=httpx.Limits(max_connections=max_in_flight),
        )

    async def wait_for_cooldown(self) -> None:
        delay = self.cooldown_until - time.monotonic()
        if delay > 0:
            await asyncio.sleep(delay)

    async def extend_cooldown(self, delay: float) -> None:
        async with self.cooldown_lock:
            self.cooldown_until = max(
                self.cooldown_until,
                time.monotonic() + delay,
            )

    async def scrape(self, url: str) -> dict:
        for attempt in range(5):
            await self.wait_for_cooldown()

            async with self.semaphore:
                # Recheck after waiting for a slot; another task may have
                # extended the cooldown while this task was queued.
                await self.wait_for_cooldown()
                response = await self.client.post(
                    API_URL,
                    json={"url": url, "mode": "auto", "output_format": "html"},
                )

            try:
                body = response.json() if response.content else {}
            except ValueError:
                body = {}
            target_status = body.get("data", {}).get("status_code")
            rate_limited = response.status_code == 429 or target_status == 429

            if not rate_limited:
                response.raise_for_status()
                if not body.get("success"):
                    raise RuntimeError(body.get("error", "scrape failed"))
                return body

            header_delay = retry_after_seconds(response.headers.get("Retry-After"))
            jitter_delay = random.uniform(0, min(60, 2 ** attempt))
            delay = max(header_delay or 0, jitter_delay)
            await self.extend_cooldown(delay)

        raise RuntimeError("Rate-limit retry budget exhausted")

    async def close(self) -> None:
        await self.client.aclose()


async def main() -> None:
    scraper = RateAwareScraper(max_in_flight=4)
    try:
        result = await scraper.scrape("https://example.com/")
        print(result.get("success"))
    finally:
        await scraper.close()


asyncio.run(main())

5.Enforce one budget across every worker

A per-process limiter multiplies capacity by the process count. Put the authoritative token bucket, next-allowed timestamp, or leased concurrency counter in shared storage. Key target budgets by normalized hostname and account budgets by credential or tenant.

Backpressure must also reach the queue. During a cooldown, delay eligible jobs instead of letting workers repeatedly dequeue and requeue them. Preserve the original attempt count and deadline, and use idempotent writes because a response can complete just before a client disconnects. The web scraping data pipeline guide covers queue leases, retries, and dead-letter handling in more depth.

• Global limiter: protects the scraping account and shared infrastructure.
• Per-domain limiter: prevents one source from consuming the entire global budget.
• Per-tenant limiter: contains one customer's burst without blocking every customer.
• Delayed queue: makes cooldowns schedulable without occupying worker connections.

6.Recover throughput gradually after a 429

After the cooldown, do not release the whole backlog. Send a small probe, validate the response, then increase traffic in measured steps. A new 429 should reduce the affected budget again. This adapts to policy changes without embedding a guessed permanent limit.

Keep retry traffic inside the same budget as first attempts. Reserving unlimited capacity for retries creates a second request stream exactly when the service is overloaded. Cap attempts and total elapsed time; when the budget is exhausted, delay or dead-letter the job instead of looping forever. The web scraping timeout guide explains how retry time fits inside a job deadline.

7.HTTP 429 troubleshooting matrix

Use the smallest corrective action that matches the evidence. A rate limit is a control signal, not proof that a target requires a browser or a different proxy.

• 429 with Retry-After — pause at least that long, publish the cooldown, and resume gradually.
• 429 without Retry-After — use capped backoff with jitter and lower the shared request budget.
• 429s across unrelated targets — inspect the API account limit, credential scope, and total in-flight jobs.
• 429s on one target only — reduce that hostname's rate and check its official API or published crawling policy.
• Latency rises before 429 — treat queueing delay as an early warning and reduce concurrency before retries accumulate.
• 429 rate grows after adding workers — replace per-worker limits with one distributed limiter.
• 403 or challenge page — leave the 429 path; diagnose access policy or protected-page retrieval separately.

8.Measure accepted records, not request volume

Track 429 count by layer, domain, tenant, and credential; parsed Retry-After duration; active cooldowns; queue age; in-flight work; retry attempts; and records that pass validation. Requests started per second is a poor success metric when retries and empty responses inflate the count. The useful outcome is fresh, valid records delivered within the job deadline.

Alert on sustained limits and jobs reaching the attempt cap. Keep headers and a redacted body sample for diagnosis, but never log API keys, authorization headers, session cookies, or personal data. Separate platform and target 429s so the on-call action is clear.

9.Use Web Unlocker without hiding rate-control decisions

OmniScrape Web Unlocker can handle the retrieval layer for independent authorized pages that need managed routing, rendering, or structured extraction. Your application still owns queue scheduling, account-wide concurrency, target-specific cooldowns, validation, and idempotent storage. A managed fetch does not remove the obligation to respect a target's rate limits.

Start with a representative URL set, log both API and target statuses, and tune from evidence. Use Browser-as-a-Service only when the workflow requires stateful browser interaction; a 429 by itself is not a reason to open a browser. Collect only data you are authorized to use and respect applicable law, website terms, robots directives where applicable, privacy obligations, and published limits.

• Identify the limiting layer before changing proxies, modes, or worker count.
• Parse both Retry-After formats and share the longest active cooldown.
• Bound concurrency and request rate separately.
• Coordinate budgets across processes, tenants, and domains.
• Resume with probes, validate content, and increase traffic gradually.
• Cap retry attempts and route exhausted jobs to a delayed or dead-letter queue.