06 / 07

SERP Engine

Google search results, AI Overviews included, fetched and turned into clean JSON.

1 / 4

What it is

The product, and the problem it exists to solve.

Send a search, get Google's results back as structured data: organic links, AI Overview, People also ask and related searches.
Built for a web data company that wants its own Google scraper running on its own servers.
Google fights scrapers, so the real work is staying unblocked, and saying clearly when it can't.
A live dashboard shows how each mode is doing, how often Google pushes back, and how many searches were answered from cache.

My part in it, feature by feature.

Three fetch modes, cheapest first: Google's old feature-phone page, a plain HTTP request with Chrome's network fingerprint, then a real browser.
Searches that want the AI Overview go straight to the browser, since the overview only appears once JavaScript runs.
A parser that leans on page structure that has held for years, not on class names Google renames every few weeks.
Per-IP pacing and an hourly cap, with any IP that hits a captcha benched for a cooldown that doubles on each repeat block and survives restarts.
A cache that answers repeat searches without asking Google, and falls back to a saved result, marked stale, when Google can't be reached.
FastAPI service, CLI and dashboard, with offline tests against saved result pages that flag when Google changes its layout.

I take products from data model to interface, and the infrastructure underneath. Tell me what you’re building.