AI Crawler Bots Are Overwhelming Servers: How to Detect and Block Them

If a server that used to run comfortably is suddenly under load with no real change in visitor numbers, AI crawlers are a likely cause before you look anywhere else. Automated requests now make up more than half of all web traffic, and AI crawling specifically has grown sharply year over year. Unlike a search engine bot that indexes a page occasionally and respects crawl delay, an AI crawler can hit the same set of pages repeatedly, ignore caching signals, and pull from parts of a site a normal visitor never touches, all of which turns into real CPU, database and bandwidth load on the server underneath.

Why this hits smaller and mid-sized servers hardest

A large platform with a CDN and generous compute absorbs this kind of traffic without much trouble. A shared cPanel or Plesk account, a single VPS running WordPress, or a modestly sized dedicated server does not have that headroom. Every AI crawler request that reaches PHP-FPM, queries the database, or triggers a dynamic page render competes directly with your actual visitors and legitimate search engine crawling for the same limited CPU and memory. On a server already close to its resource ceiling, a crawl spike from one or two aggressive bots is enough to push load average up, slow down every site on the box, or trigger the same symptoms as a small DDoS.

Confirm it is actually AI bot traffic before changing anything

Do not block on a guess. Check access logs first and look at request volume by user agent:

grep -i "bot" /var/log/apache2/access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -rn | head -20

On an Nginx or Litespeed box, adjust the log path accordingly. Look specifically for user agents such as GPTBot, ClaudeBot, PerplexityBot, Bytespider, Amazonbot, CCBot and Meta-ExternalAgent. A handful of these generating thousands of requests an hour, especially against pages with no caching or against search and filter URLs that generate a fresh database query every time, is the pattern to look for. Cross-check the spike against your monitoring: a load average or PHP-FPM worker count that climbs in step with a specific user agent's request volume confirms the cause before you touch server configuration.

Decide which bots to block and which to allow

Not every AI crawler is worth blocking outright, and the right answer depends on the bot. Some send meaningful referral traffic back when their tools cite your content; others do not.

A reasonable default for most support and infrastructure sites:

• Block entirely: Bytespider, CCBot, and any bot generating heavy load with no real referral or citation value.
• Allow but rate limit: GPTBot, ClaudeBot, PerplexityBot, since these have some citation and referral upside but should not be allowed to hit a server at an unrestrained rate.
• Leave alone: standard search engine crawlers such as Googlebot and Bingbot, since blocking those directly damages search visibility.

Start with robots.txt, but do not rely on it alone

robots.txt is the correct first step because well-behaved crawlers respect it, and it costs nothing to add:

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: GPTBot
Crawl-delay: 10

User-agent: ClaudeBot
Crawl-delay: 10

The limitation is that robots.txt is voluntary. It stops compliant crawlers, but it does nothing against a bot that ignores it, and it will not help at all during an active load spike, since the crawler has to request the file and choose to honor it. Treat it as a courtesy signal, not a control.

Enforce it at the web server, where it actually works

Blocking or rate limiting at the web server level works regardless of whether the bot respects robots.txt. On Nginx:

map $http_user_agent $blocked_bot {
    default 0;
    ~*Bytespider 1;
    ~*CCBot 1;
}

server {
    if ($blocked_bot) {
        return 403;
    }

    limit_req_zone $binary_remote_addr zone=ai_bots:10m rate=6r/m;
    location / {
        limit_req zone=ai_bots burst=10 nodelay;
    }
}

On Apache, the same idea works through mod_rewrite:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Bytespider|CCBot) [NC]
RewriteRule ^ - [F,L]

On a cPanel server, this can go in the domain's Apache include via WHM, or be applied through ModSecurity with a custom rule targeting the user agent. On a Plesk server, the equivalent lives in the domain's additional Nginx directives or Apache directives under the Hosting Settings.

Use fail2ban for bots that ignore both signals

Some crawlers rotate IP ranges or spoof user agents to get around simple matching. For those, a fail2ban jail that watches request rate per IP regardless of what the user agent claims to be catches the behavior instead of the label:

[ai-bot-flood]
enabled = true
filter = ai-bot-flood
logpath = /var/log/nginx/access.log
maxretry = 120
findtime = 60
bantime = 3600

A filter that flags any single IP making more than roughly two requests per second sustained for a minute is aggressive enough to catch a scraping burst without touching normal human browsing patterns.

Put a CDN or reverse proxy in front if the load keeps recurring

If AI bot traffic is a recurring problem rather than a one-time spike, handling it at the origin server every time is treating the symptom. A CDN or reverse proxy in front of the server, such as Cloudflare, can identify and manage verified AI bot traffic before it ever reaches your infrastructure, and it absorbs the request volume on infrastructure built for exactly this kind of load rather than your production server's CPU.

Watch for the second-order effect: cache and database load

The most expensive AI crawler requests are the ones that bypass page caching, typically search results, filtered listings, or any URL with a query string. If your caching layer, whether that is a page cache plugin, Varnish, or Nginx's own cache, is not configured to serve cached responses to these bot user agents the same way it does for regular visitors, every crawl hit falls through to PHP and the database. Confirm your cache is actually caching for these user agents rather than assuming it does, since some cache configurations deliberately exclude anything that looks like a bot.

Putting it together

AI crawler traffic is not going away, and treating every bot the same way, either allowing everything or blocking everything, is the wrong response either direction. Confirm the load is actually coming from AI crawlers using your access logs, decide which bots carry enough upside to allow at a limited rate, block the ones that do not, enforce that decision at the web server rather than relying on robots.txt alone, and make sure your caching layer is not quietly exempting bot traffic from the caching that protects everything else.

iServerSupport provides server security services including access log analysis, bot traffic mitigation and web server hardening, so AI crawler load gets identified and controlled before it degrades a production server.