Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages

Help for owners

My website is full of bots and someone is copying my content

Short answer

Not all automated traffic is harmful: search crawlers are bots too, so blocking by guesswork can remove you from search results. First find out what the traffic is and what it hits, then slow or challenge the bad part. Verify a crawler's claimed identity against its source address, never against its name alone.

  • Written for business owners
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Does this sound like your situation?

  • My visitor numbers jumped but sales and enquiries did not.
  • The site is slow or the host sends usage warnings, and the traffic looks odd.
  • Someone has copied my product pages or prices onto another site.
  • The logs show thousands of requests to one page, search or login from a few addresses.
  • Analytics shows visitors from countries I do not sell to.

Why does this happen?

  1. Bots are a normal part of the web, good and bad

    Cloudflare's documentation describes bots as either beneficial, such as search engine crawlers and chatbots, or harmful, such as those used for inventory hoarding and credential stuffing. A public site will always get both.

  2. Bots now differ by purpose, not by one label

    Cloudflare classifies AI bots by behaviour: search (indexing content so questions can be answered), agent (acting in real time for a person) and training (crawling to train a model). One bot can do more than one. Each has a different effect on your site, and you may want to allow some and not others.

  3. A bot can lie about who it is

    Google notes that its crawler can be verified when you are worried about spammers claiming to be from Google. The check is a reverse DNS lookup of the visiting address, confirming a Google domain, then a forward lookup returning the same address. Google also publishes lists of its address ranges. The name in the request is not proof.

  4. Scrapers want your content, not your visitors

    They fetch pages in bulk to copy prices, listings or text. Pages that build expensive database queries, search boxes and unlimited product filters cost the most when hit thousands of times, so the site slows for real customers too.

What can I do about it today?

  1. Do not block countries or addresses in a hurryA broad block can lock out real customers, your payment provider's callbacks or search engines. Write down what you see first.
  2. Look at what is being requestedAsk your host for a short summary of the busiest pages and addresses. A pattern, such as one search page or the login, tells you what is being targeted.
  3. Check whether the traffic is hurting youCompare against orders, enquiries and server load. If real visitors see no slowdown and nothing is leaking, you may only need to filter your reports.
  4. Look for a bot setting in tools you already useYour host, CDN or firewall provider may have a bot setting. Turn on a challenge or monitor mode first, before a hard block.
  5. Keep search engines and payment callbacks workingBefore any rule goes live, confirm real search crawlers and payment or shipping notifications are still allowed.
  6. Save examples of copied pagesIf your text or photos are copied, keep screenshots, links and dates. A copyright complaint to the host of the other site needs them.

When should I bring someone in?

  • The site slows down or goes offline during the bot waves.
  • Logins are being tried again and again, or accounts are being created that you did not expect.
  • You are not sure a rule is safe, and a mistake would block customers or search engines.

How do I handle it?

  1. Read the logsI sort the traffic by address, page, speed and what it asks for, to separate crawlers, scrapers, attack attempts and real people.
  2. Verify the crawlers you wantSearch engine crawlers are checked against the engine's published method, so rules do not rely on a name that anyone can fake.
  3. Slow, then challenge, then blockRate limits and challenges go in first, on the pages that are being abused. Hard blocks are for clear cases.
  4. Make the expensive pages cheaperCaching, limits on search and filters and removing open endpoints reduce what a scraper can cost you.
  5. Watch for side effectsAfter each rule I check real orders, forms, payment callbacks and search crawling, and I can remove the rule again.

The price depends on what I find, so I quote in writing after a short discovery call. Tell me what is happening and I reply within 24 hours.

What do owners ask about this?

Can I stop all scraping?

No. Anything a browser can see can be fetched by a script. You can make it slower, more expensive and easier to detect.

Will blocking bots hurt my search ranking?

It can if a search crawler is blocked by mistake. That is why I verify crawlers by their published method before writing any rule.

Should I block AI crawlers?

That is a business choice. Cloudflare lets customers manage search, agent and training behaviours separately, so you can treat them differently.

Is a CAPTCHA the answer?

It helps on forms and logins. It does little against scrapers that simply read public pages.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Found a mistake? Tell me.

Next step

Want this off your desk?

Send me a few lines on where things stand. I reply within 24 hours.