How to Bypass Cloudflare in Python

1.Why the User-Agent never mattered

A TLS handshake exposes an ordered fingerprint before any HTTP is exchanged: the cipher suites offered and their order, the extensions present and their order, supported groups, signature algorithms, and — for Chrome — the GREASE values it injects deliberately. Summarised as a JA3 or JA4 hash, this identifies the client stack with high confidence.

requests sits on urllib3 and OpenSSL as configured by your Python build. That combination produces a fingerprint that is stable, well known, and unmistakably not a browser. Cloudflare can therefore reject the connection while your carefully ordered headers are still queued behind the handshake, which is why header work produces no measurable improvement.

httpx changes the API and adds HTTP/2 and async, both genuinely useful, but it does not change what is underneath. Its fingerprint is a Python fingerprint. Migrating from requests to httpx to fix blocks is a common and entirely wasted refactor.

2.What each option actually solves

The libraries people reach for divide cleanly into two groups, and the division is the thing worth internalising: some fix the fingerprint but cannot execute JavaScript, and some execute JavaScript but carry an automation surface that is itself detectable. Very little fixes both cheaply.

curl_cffi is the strongest option in the first group. It binds to a curl-impersonate build and reproduces a real browser's TLS and HTTP/2 characteristics faithfully, at close to zero CPU cost. Against zones that block purely on network-layer fingerprinting it works well. It cannot help you against a JavaScript challenge, because there is no JavaScript engine in the picture.

In the second group, undetected-chromedriver and patched Playwright builds run a real engine and can therefore clear JavaScript challenges. The cost is real RAM per instance, meaningfully slower requests, and an ongoing arms race — the patches address known detection vectors, and new ones appear. Neither is a set-and-forget dependency.

• requests / httpx — Python TLS fingerprint, no JS. Blocked at the handshake on protected zones.
• curl_cffi — browser-grade TLS and HTTP/2, no JS engine. Good for fingerprint-only blocks.
• cloudscraper / cfscrape — reimplement an old challenge that no longer ships. Effectively dead.
• undetected-chromedriver — real engine, real RAM cost, detection surface that shifts over time.
• Playwright with stealth patches — same trade-off, better API, same arms race.

3.Using curl_cffi where it fits

When the target blocks on fingerprint alone and serves server-rendered HTML, curl_cffi is the cheapest thing that works. The impersonate argument selects a browser profile, and the resulting handshake matches that browser closely enough to pass network-layer scoring.

Validate the body rather than the status code. A challenge page is frequently served with a 200, so the honest test is whether the markup you need is present — not whether the request completed.

Keep the profile current. Browser fingerprints change with browser releases, and an impersonation profile pinned to an old Chrome version gradually becomes its own anomaly: a fingerprint that matches a build almost nobody still runs.

1234567891011121314from curl_cffi import requests as cffi_requests

resp = cffi_requests.get(
    "https://target.example.com/product/8821",
    impersonate="chrome",   # keep this profile current with real browser releases
    timeout=30,
)

body = resp.text
# Status alone is not a success signal: challenge pages ship with 200.
if "challenge-platform" in body or len(body) < 15_000:
    raise RuntimeError("got an interstitial, not the page")

print(len(body))

4.The cost that shows up in month three

Every option in the browser group needs upkeep. Chrome ships a new stable version regularly, detection vectors are found and patched on both sides, and the stack that worked in January quietly degrades by April. The failure is rarely loud — success rate drifts down while your pipeline reports the same number of completed requests.

Running browsers at volume also has an infrastructure shape that surprises people who prototyped on a laptop. Each instance holds real memory, needs process supervision, leaks under load if not recycled, and requires enough headroom that a traffic spike does not take the pool down. That is a service to operate, not a library to import.

This is the trade worth stating plainly: the libraries are free and the time is not. If bypass is incidental to what you are building, delegating it is usually cheaper than the engineer-weeks. If bypass is your product, owning the stack may be the right call — but own it deliberately, with monitoring on success rate rather than on completion count.

5.Calling OmniScrape from Python

The API is a single POST, so there is no SDK to install and nothing to keep in sync. Send the target URL with mode auto and enable_solver true, and the routing decision — fast HTTP first, browser escalation only when challenge signals appear — happens server-side. You are billed for successful unlocks, so the escalation is not something you have to price manually per request.

Keep your key in the environment rather than the source. The examples below read OMNISCRAPE_KEY, which is also the convention used throughout the API documentation.

Validate content in the same place you validate the response envelope. success tells you the unlock worked; data.status_code tells you what the origin said afterwards; the body tells you whether the page is the one you wanted. All three matter, and the third is the one pipelines usually skip.

12345678910111213141516171819202122232425262728import os
import requests

def scrape(url: str, **overrides) -> dict:
    payload = {
        "url": url,
        "mode": "auto",
        "enable_solver": True,
        "output_format": "html",
        **overrides,
    }
    resp = requests.post(
        "https://api.omniscrape.io/v1/scrape",
        headers={"X-API-Key": os.environ["OMNISCRAPE_KEY"]},
        json=payload,
        timeout=120,
    )
    resp.raise_for_status()
    body = resp.json()
    if not body.get("success"):
        raise RuntimeError(body.get("error", "unlock failed"))
    return body

result = scrape(
    "https://cf-protected-shop.com/product/8821",
    proxy="residential:us",
)
print(result["metadata"]["challenge_solved"], result["data"]["status_code"])

1234567891011121314151617181920212223242526272829303132import asyncio
import os
import httpx

CONCURRENCY = 5          # match your plan's concurrency limit
RETRY_STATUS = {429, 500, 502, 503, 504}

async def scrape_one(client: httpx.AsyncClient, url: str, sem: asyncio.Semaphore) -> dict | None:
    async with sem:
        for attempt in range(4):
            resp = await client.post(
                "https://api.omniscrape.io/v1/scrape",
                headers={"X-API-Key": os.environ["OMNISCRAPE_KEY"]},
                json={"url": url, "mode": "auto", "enable_solver": True},
                timeout=120,
            )
            if resp.status_code in RETRY_STATUS:
                # 429 carries Retry-After; fall back to exponential backoff.
                delay = float(resp.headers.get("Retry-After", 2 ** attempt))
                await asyncio.sleep(delay)
                continue
            resp.raise_for_status()
            return resp.json()
    return None

async def main(urls: list[str]) -> list[dict]:
    sem = asyncio.Semaphore(CONCURRENCY)
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*(scrape_one(client, u, sem) for u in urls))
    return [r for r in results if r]

asyncio.run(main(["https://cf-protected-shop.com/product/8821"]))