AI Crawler/bot protection?

Smokey's Avatar
Smokey · in Support
2 replies 210 views

What sort of safeguards are in place to prevent or combat AI crawler/scraper bots from bombarding the servers at FS?

One of the reasons i'm planning on moving my site over is because it's being bombarded by crawler/AI scrape bots and using up all my hosting's resources.

Last edited by Smokey

Something.....something......dark side.....something something something.....complete....

0

He can use AI Labyrinth from Cloudflare

0

ForumSpark is under Cloudflare protection.

Content-Signal: search=yes, ai-train=no

That translates to:

Search engines can:

Index pages

Show links + snippets

Not allowed:

AI training (model training / fine-tuning)

Bots that collect data for training should NOT use your content

Cloudflare-managed blocks:

These are fully disallowed (cannot crawl anything)

  • Amazonbot

  • Applebot-Extended

  • Bytespider (TikTok)

  • CCBot

  • ClaudeBot

  • CloudflareBrowserRenderingCrawler

  • Google-Extended (Google AI training bot)

  • GPTBot (OpenAI crawler)

  • meta-externalagent (Meta/Facebook AI)

  • PetalBot (Huawei)

  • AspiegelBot

  • AhrefsBot

  • SemrushBot / SemRush

  • DotBot

  • MauiBot

  • MJ12bot

  • ClaudeBot

All other bots not listed above, including:

Googlebot (normal search), Bingbot, DuckDuckBot, etc. can crawl everything and index content.

You can view that using your links and adding robots.txt after the link. I do want to note, I continuously monitor this, have specific notifications going in case something is being flooded, or we are under DDoS attack, and this is also being steered when needed.

👍 1
Theme: FS 2.0
Create Your Community with ForumSpark

ForumSpark is a modern community platform built on custom software. It combines the familiar feel of classic forums with a fast, simple experience designed for today’s communities.