1.When this comparison is worth your time
This is a live comparison if you are running somewhere between a few thousand and a few million requests a month against a mixed list — some hard retail targets behind a WAF, plenty of ordinary pages, and a long tail of things you have not classified. At that shape, small structural differences compound into meaningfully different bills and meaningfully different on-call load.
It is not a live comparison if your targets are unprotected. Both products are more machinery than an open data portal or a public API requires, and a plain HTTP client with sensible retries will serve you better and cheaper than either.
It is also not the right comparison if you need managed datasets, contracted SLAs, or procurement documentation. Neither vendor is positioned there, and evaluating both against that requirement will just tell you what you already know.
2.Reliability is the only claim worth testing
Every vendor in this category claims high success rates, and the number is close to meaningless without knowing what counts as success. A response delivered is not a page delivered. If the body is a Cloudflare interstitial served with a 200, most measurement harnesses record it as a win, and the vendor's dashboard agrees.
This is why reliability has to be measured on extracted output rather than on transport outcomes. The question is not what fraction of requests returned, it is what fraction produced a record with the fields you needed. Those two numbers can differ enormously on hard targets, and the gap is exactly where vendor differences live.
OmniScrape exposes the state that makes this checkable. metadata.challenge_solved reports whether a challenge was actually cleared, metadata.method_used reports which lane ran, and data.status_code reports what the origin said after the unlock rather than what the API wrapper said. Whatever vendor you compare against, insist on equivalent visibility — a product that only returns a body and a success boolean cannot be audited.
3.What happens when a request fails
This is the single most consequential structural difference between vendors in this bracket, and it is easy to miss during evaluation because it only bites on the hard tail of your URL list.
OmniScrape does not bill failed unlocks. billing.charged is 0.00 on a failure, and the per-request price is $0.0035. The consequence is that retry behaviour does not change your unit economics: a target that needs three attempts costs the same as one that succeeds immediately, so your cost tracks records extracted rather than attempts made.
Where a vendor bills per attempt, or bills in credits with a multiplier for browser rendering, your effective cost on hard targets is your list price multiplied by your average attempt count — a number most teams never compute and which is worst precisely on the targets you care most about. When comparing, ask directly what is billed on a failed unlock, and verify it against an invoice rather than a documentation page.
- Is a failed unlock billed, or only a successful one?
- Is browser rendering priced at a multiplier, and if so, what is it?
- Is the engine chosen per request, or fixed per account or per plan?
- Does a failed request still consume concurrency for its full timeout?
4.Per-request routing versus a plan-level setting
On a mixed URL list, the difference between choosing an engine per request and choosing one per account is the difference between paying browser rates on the pages that need them and paying browser rates on everything.
OmniScrape's auto mode attempts a fast HTTP fetch first and escalates to a browser context only when the response carries challenge signals. You can override this per request with mode set to fast or js_rendering when you already know a target's shape, but the default requires no classification work from you.
The measurable consequence is that metadata.method_used tells you, per URL, whether a browser was actually needed. Aggregated over a real crawl, that field is a map of your list's difficulty — and often the first time a team discovers that most of what they were paying premium rates for is static HTML.
1234567891011curl -X POST https://api.omniscrape.io/v1/scrape \
-H "Content-Type: application/json" \
-H "X-API-Key: ${OMNISCRAPE_KEY}" \
-d '{
"url": "https://mixed-difficulty-target.com/p/9182",
"mode": "auto",
"enable_solver": true,
"output_format": "html"
}'
# metadata.method_used comes back as "fast" or "js_rendering".
# Aggregate that field across a crawl to see what you are really paying for.
5.Session and IP continuity
Anti-bot cookies are validated against the IP that earned them. Any job that spans more than one request against the same target — pagination, a detail page reached from a listing, an authenticated flow — needs to hold one identity across those requests or it pays for a fresh challenge solve on each one.
OmniScrape handles this with session_id, which pins the same IP and cookie jar across requests, combined with a proxy mode of sticky, rotate, or smart. The pattern that follows is simple: one session_id per logical unit of work, rotation between units rather than within them.
When comparing vendors, test this explicitly rather than reading the documentation. Run a ten-page paginated walk and count how many pages triggered a solve. A product without real session pinning will solve repeatedly, and on a per-attempt billing model that is where the invoice goes wrong.
6.A bake-off design that produces a defensible answer
Use your own URLs, sampled across your real difficulty distribution. A curated set of hard targets flatters whichever vendor happens to handle those specific sites, and a curated easy set tells you nothing at all. Take a random sample of production URLs, several thousand, and keep the sample fixed across both vendors.
Run both vendors against the identical sample in the same window, and key the results by URL so records can be diffed field by field. Record four columns per request: whether a usable record was extracted, the amount actually charged, the origin status code, and total elapsed time. Cost per successfully extracted record falls out of the first two.
Cross a weekend and a weekday peak. Anti-bot posture tightens under load on many targets, and a bake-off run over a quiet period will overstate both products. Two weeks is usually enough; two days is not.
Frequently asked questions
What actually separates vendors in this bracket?
Not the feature grid, which converges. What separates them is whether failed unlocks are billed, whether the engine is chosen per request or per account, and whether a multi-request job can hold one IP and cookie jar. Those three decide the invoice and the on-call load.
How should success rate be measured?
On extracted records, not on responses returned. A challenge interstitial served with a 200 counts as success in most harnesses. Compare the fraction of requests that produced a record with the fields you needed, which is a very different number on hard targets.
Does OmniScrape bill failed requests?
No. billing.charged is 0.00 on a failed unlock, and a successful request is $0.0035. This means retries do not change your unit economics — cost tracks records extracted rather than attempts made.
How do I keep one identity across a paginated crawl?
Pass the same session_id across the requests in that unit of work, with a proxy mode of sticky. That pins the IP and cookie jar so one challenge solve covers the walk. Rotate between units, not within them.
How long should a bake-off run?
Long enough to cross a weekend and a weekday peak — roughly two weeks. Anti-bot posture is not constant, and a comparison run over a quiet period overstates both vendors.
Related guides