A significant share of traffic to any public site is automated, and the instinct on discovering that is to block it. The instinct needs tempering: some of that traffic is Googlebot, without which you do not appear in search results, and some is monitoring, link previews and legitimate partner integrations. The task is discrimination rather than exclusion.
Start by categorising rather than counting. Search engine crawlers are welcome and should be verified rather than trusted - user-agent strings are trivially forged, and the correct check is a reverse DNS lookup on the source address confirming it belongs to the crawler's stated domain. Beneficial bots such as uptime monitors and social preview fetchers are fine. Scrapers taking your content or pricing are a business decision. Credential-stuffing and vulnerability scanning traffic is unambiguously hostile.
For the hostile category, the highest-value protection is on authentication endpoints. Credential stuffing works by trying leaked username and password pairs across many sites, at low rate per address, spread across thousands of addresses - which means per-IP rate limiting alone will not see it. Limit per account as well, and watch the aggregate failure rate across the whole login endpoint, because a sudden rise in failed logins across many accounts is the signature no per-user check will catch.
For scrapers, be honest about what is achievable. Content you serve publicly can be taken; determined scraping cannot be prevented, only made more expensive. Rate limiting by address, requiring a session for expensive endpoints, and watching for the behavioural signature - perfectly regular intervals, no static assets fetched, pages requested in exact sequence - raises the cost meaningfully. Fingerprinting and challenge services raise it further, at the price of occasionally inconveniencing real users, which is a trade to make consciously.
Do the free things properly first. A robots.txt that states your rules gives you a documented position, even though it is advisory. A sitemap helps the crawlers you want. Serving static assets from a CDN removes most automated load from your own servers before it reaches them. And blocking obviously malicious paths - probes for administrative panels and known vulnerable endpoints - at the edge is cheap and removes a great deal of noise from your logs.
Whatever you deploy, measure the false positives, because the failure mode nobody notices is blocking real users. A customer behind a corporate proxy shared with a thousand colleagues looks exactly like a bot by request volume. If you cannot see how often your protection rejects legitimate traffic, you cannot tell whether it is helping or quietly costing you business - and the complaints reach support long before they reach engineering.
