How to Scrape a Website: A Step-by-Step Guide for 2026

1.What web scraping actually is

Web scraping is the automated extraction of data from web pages. Instead of copying values by hand, a program fetches the page's HTML and pulls structured data out of it. prices, listings, reviews, contact details, articles. into a format you can store and analyze, like CSV, JSON, or a database.

Every scraper, no matter how sophisticated, is really just two steps repeated: fetch (get the page) and parse (extract the data). Everything else. proxies, browsers, challenge solving, retries. exists to keep the fetch step working when a site does not want to be scraped. Understanding this two-step core keeps the rest of the process simple in your head.

2.Step 1: Inspect the page structure

Before writing any code, open the target page in your browser and press F12 to open DevTools. Right-click the value you want. a price, a title. and choose Inspect. This shows you the exact HTML element and the CSS classes or IDs that identify it. Those selectors are what your scraper will use to find the data.

Check two things while you are here. First, is the data present in the initial HTML, or does it appear only after the page loads (a sign of JavaScript rendering)? View the page source (Ctrl+U) and search for a known value. if it is not in the raw source, the page needs a browser. Second, look at how pages are numbered or linked so you can plan pagination.

3.Step 2: Install your tools

For a basic scraper you need one library to fetch and one to parse. In Python that is requests and Beautiful Soup. Add lxml as a fast parser backend.

1pip install requests beautifulsoup4 lxml

4.Step 3: Fetch the HTML

The fetch step downloads the raw HTML. This works on sites without bot protection. small business sites, open-data portals, scraper-friendly demo sites. Send a realistic User-Agent header so you look like a browser rather than a default library client.

If this returns the content you expected, you are ready to parse. If you get a 403, a CAPTCHA page, or a "Checking your browser" screen, the site has anti-bot protection. jump to the section on getting blocked.

12345678import requests

url = "https://books.toscrape.com/catalogue/page-1.html"
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}

response = requests.get(url, headers=headers, timeout=30)
response.raise_for_status()
print(f"Fetched {len(response.text):,} bytes, status {response.status_code}")

5.Step 4: Parse the data you want

Now turn HTML into structured rows using the selectors you found in DevTools. Beautiful Soup queries the DOM with the same CSS selectors you would use in the browser. Always confirm each field is non-empty before saving. an empty value usually means the selector is wrong or the layout changed.

12345678910111213from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, "lxml")

items = []
for card in soup.select("article.product_pod"):
    title = card.select_one("h3 a")["title"]
    price = card.select_one(".price_color").get_text(strip=True)
    items.append({"title": title, "price": price})

print(f"Extracted {len(items)} items")
for row in items[:3]:
    print(row)

6.Step 5: Handle pagination

Most useful data spans many pages. The pattern is always fetch, parse, find the next page, repeat. Stop when there is no next link or the server returns a 404. Add a short delay between requests when scraping directly. two seconds is polite and reduces your chance of being rate-limited.

1234567891011121314151617181920import time

all_items = []
page = 1
while True:
    url = f"https://books.toscrape.com/catalogue/page-{page}.html"
    r = requests.get(url, headers=headers, timeout=30)
    if r.status_code == 404:
        break
    soup = BeautifulSoup(r.text, "lxml")
    cards = soup.select("article.product_pod")
    if not cards:
        break
    for card in cards:
        all_items.append({"title": card.select_one("h3 a")["title"]})
    print(f"Page {page}: {len(cards)} items")
    page += 1
    time.sleep(2)

print(f"Total: {len(all_items)} items")

7.Step 6: Save the results

Write your rows to a file or database. CSV is fine for one-off jobs; JSON is better when fields are nested; a database is right when you scrape on a schedule and need to deduplicate or query.

123456import csv

with open("items.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price"])
    writer.writeheader()
    writer.writerows(all_items)

8.Step 7: When the page needs JavaScript

Many modern sites ship an almost-empty HTML shell and load the real content with JavaScript after the page opens. requests only sees the shell. You have two options: run a real browser (Playwright or Selenium) that executes the JavaScript, or send the URL to a scraping API that renders it for you.

Running your own browser works but is heavy. one Chromium process per page, memory-hungry, and still blockable. For a deeper look at this specific problem, see scraping JavaScript-rendered pages and headless browser scraping.

9.Step 8: What to do when you get blocked

Sooner or later you will hit Cloudflare, DataDome, Akamai, or a plain rate limit. Header tweaks and a single proxy buy you a little time, but protection vendors update constantly and you end up maintaining TLS fingerprints and proxy pools instead of shipping features.

This is the point most teams switch to a scraping API. You keep your parsing code and replace only the fetch step. The API rotates residential IPs, mimics real browser fingerprints, and solves challenges, returning the real HTML. Read web scraping without getting blocked for the full set of techniques.

• Rotate residential IPs so no single address gets rate-limited
• Match a real browser's TLS and header fingerprint, not a library default
• Render JavaScript in a real browser when the page needs it
• Solve anti-bot challenges (Cloudflare, DataDome, Akamai) automatically

10.Step 9: Scale with a scraping API

Here is the same scraper, but the fetch step now goes through OmniScrape. Your parsing logic is unchanged. mode:auto tries a fast HTTP path first and only opens a browser if the page needs it, so you pay for a browser only when necessary.

You can also skip parsing entirely by asking the API to extract fields with CSS selectors. the response comes back as structured JSON. That is the most robust option for production because a broken selector shows up as an empty field, not a silent bad row.

123456789101112131415161718import os, requests

resp = requests.post(
    "https://api.omniscrape.io/v1/scrape",
    headers={"X-API-Key": os.environ["OMNISCRAPE_KEY"]},
    json={
        "url": "https://protected-shop.com/product/123",
        "mode": "auto",
        "output_format": "css_extractor",
        "css_selectors": {"title": "h1.product-name", "price": ".price-current"},
    },
    timeout=120,
)
data = resp.json()
if data["success"]:
    print(data["data"]["css_extracted"])
else:
    print("Failed:", data.get("error"))

11.Scrape responsibly

Being able to scrape a site does not always mean you should. Review the target's terms of service and robots.txt, avoid collecting personal data you are not authorized to use, rate-limit yourself so you do not degrade the site, and cache results instead of re-fetching. Responsible scraping keeps your access. and your project. sustainable.