A Year of Fighting Scrapers: Defending a 1.5 Million-Page Website
If you run a website with any meaningful amount of original content, you are already in a low-grade war. On the other side of that war is a swarm of automated bots — some belonging to search engines you actually want, many belonging to AI training crawlers, SEO scrapers, price-monitoring services, and outright content thieves who will republish your work under someone else's byline within hours.
The scale of the problem has changed. A blog with a few thousand pages might see occasional scraping. A site with 1.5 million pages — think a long-running news archive, a large e-commerce catalogue, or a mature directory — becomes a permanent target. The bots never sleep, they rotate IPs, they lie about their user agents, and they can quietly consume more bandwidth than your paying human readers.
Drawing on lessons publishers and independent site operators have shared this year, including the kind of forensic work described in Pocketables' recent write-up on using AI to interrogate server logs, here is a practical, evergreen guide to protecting a large site from scrapers.
Step one: know what is actually hitting your server
You cannot defend against traffic you cannot see. The first mistake most site owners make is relying on analytics products like Google Analytics or Plausible to understand who is on their site. Those tools depend on JavaScript execution, and scrapers overwhelmingly do not execute JavaScript. To a JS-based analytics tool, a bot pulling a million pages a day is invisible.
The truth lives in your raw server logs — the access logs from Nginx, Apache, Caddy or whatever sits in front of your application. Those logs record every request, including the ones that never touch your analytics pixel. The problem is that a busy site produces gigabytes of log data a day, and most of it is noise.
This is where the recent shift toward AI-assisted log analysis, of the kind Pocketables describes, has genuinely changed the game for solo operators and small teams. Instead of writing brittle grep and awk pipelines, you can feed a chunk of anonymised log data to a large language model and ask questions in plain English: which user agents are hitting me hardest, which IP ranges are behaving suspiciously, which endpoints are being hammered out of proportion to their real traffic. It is not magic, and it will not replace a proper SIEM, but as a rapid triage tool for someone who suspects they are being scraped, it is surprisingly effective.
Step two: separate the good bots from the bad
Not every bot is your enemy. Googlebot, Bingbot, and — depending on your view — bots from Applebot and DuckDuckGo are the reason organic traffic exists. The trouble is that anyone can put Googlebot in a user-agent string.
The defence here is verification. Legitimate search engine crawlers publish the IP ranges they use, and most of them support reverse-DNS verification: you do a reverse DNS lookup on the requesting IP, then a forward lookup on the resulting hostname, and check that it matches. If a request claims to be Googlebot but its IP resolves to a residential ISP in another country, it is lying.
Maintain three lists:
- Allow: verified crawlers you want (major search engines, your own uptime monitors).
- Deny: known-bad user agents, AI training crawlers you have opted out of, and abusive IP ranges.
- Watch: everything else, subject to rate limits and behavioural checks.
Step three: rate limit like you mean it
Blanket blocking is a blunt tool that will eventually catch a real user on a shared IP. Rate limiting is more surgical. The pattern that works well for large content sites:
- A generous limit per IP for HTML page views — say, one request every second or two, bursting to a handful — which no genuine reader will ever hit.
- Tighter limits on expensive endpoints: search, faceted filters, sitemap indexes.
- Per-ASN limits, not just per-IP. Scrapers routinely rotate through hundreds of residential proxies, but they usually come from a handful of hosting ASNs.
Nginx's limit_req module, Cloudflare's rate-limiting rules, and Fastly's edge logic all support this. On a 1.5 million-page site, per-ASN limits catch the industrial scrapers that per-IP limits miss entirely.
Step four: make scraping expensive
You are unlikely to stop a determined adversary. What you can do is make your site the least attractive target. A few techniques that raise the cost of scraping without harming real readers:
- Server-side rendering with meaningful HTML. Counterintuitively, sites that ship an empty shell and hydrate with JavaScript are easier to scrape today than they were five years ago, because scrapers now use headless Chrome. But sites that require complex client-side session state or that lazy-load key content on scroll are still noticeably harder.
- Signed, short-lived URLs for images, downloads and API responses. A scraper cannot cheaply mirror your media if every URL expires in ten minutes.
- Honeypot links — invisible links to URLs no human would ever click, which auto-ban any IP that follows them. Cheap, effective, and they cost real users nothing.
- Slightly randomised HTML. Vary class names, insert zero-width characters, shuffle attribute order. This breaks brittle scrapers that rely on exact selectors.
Step five: know your legal position (especially in Australia)
For Australian operators, the legal environment is worth understanding. Original written content is protected by copyright automatically under the Copyright Act 1968 — no registration required. Scraping and republishing your articles is straightforwardly infringement. Enforcement, however, is a different question: most scrapers are offshore, judgment-proof, and hidden behind shell companies.
Practical remedies that actually work:
- DMCA and equivalent takedowns to the hosting provider and CDN of the offending site. Cloudflare, in particular, will forward abuse complaints.
- Google's copyright removal tool, which can de-list infringing pages from search results. This is often more painful to the scraper than a lawsuit.
- Ad network complaints. Most scraper sites run on AdSense or similar. Reporting them to the ad network is frequently what actually kills the operation.
The Office of the Australian Information Commissioner and the ACCC have both flagged concerns about mass data harvesting; if scraping involves personal information (comments, user profiles), Privacy Act considerations may also apply to the scraper.
Step six: accept that this is now permanent
The single biggest mental shift, after a year of this, is accepting that scraper defence is not a project you finish. It is a maintenance task, like backups or security patching. The bots evolve. Every few months a new AI company launches a crawler, a new proxy service becomes cheap enough for scrapers to adopt, or a new headless-browser framework makes fingerprinting harder.
The sites that cope best treat this as ongoing gardening: weekly log reviews (increasingly LLM-assisted), monthly reviews of blocked ASNs, quarterly audits of which AI companies they want to allow into their robots.txt. It is not glamorous work. But if your business or reputation depends on a large body of original content, it is now part of the job.
The good news is that the tools have caught up. Between managed edge providers, cheap log storage, and LLMs that can read a million log lines for you, a single competent operator can now defend a site that would have required a small SOC team a few years ago. The bots have scale. So, finally, do we.