Создадим файл bot.conf с содержимым по адресу /etc/nginx/vhosts-includes/
if ($bad_bot = 1) {
return 444;
}Создадим второй файл bot.conf с содержимым по адресу /etc/nginx/conf.d/ с перечислением User agent ботов
map $http_user_agent $bad_bot {
default 0;
~(?i)(GeedoProductSearch|keys-so-bot|185.22.153.240|Bytespider|MegaIndex.ru|SeznamBot|SeekportBot|MJ12bot|SEOkicks|archive.org|wp_is_mobile|Barkrowler|ahrefsbot|LinkpadBot|DataForSeoBot|amazonbot|ApacheBench|PetalBot|Adsbot/3.1|serpstatbot|MauiBot|AspiegelBot|Aboundex|360Spider|Cogentbot|Alexibot|asterias|BackDoorBot|Bandit|BatchFTP|Bigfoot|Black.Hole|BlackWidow|BlowFish|BotALot|Buddy|BuiltBotTough|Bullseye|BunnySlippers|Cegbfeieh|CheeseBot|CherryPicker|ChinaClaw|Collector|Copier|CopyRightCheck|cosmos|Crescent|Custo|AIBOT|DIIbot|DittoSpyder|dragonfly|Drip|eCatch|EasyDL|EirGrabber|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|Exabot|Extractor|EyeNetIE|Foobot|flunky|FrontPage|Go-Ahead-Got-It|gotit|GrabNet|Grafula|Harvest|hloader|HMView|HTTrack|humanlinks|IlseBot|InfoNavibot|InfoTekies|Intelliseek|InterGET|Iria|Jakarta|JennyBot|JetCar|JOC|JustView|Jyxobot|Kenjin.Spider|Keyword.Density|larbin|LexiBot|libWeb/clsHTTP|likse|LinkextractorPro|LinkScan/8.1a.Unix|LNSpiderguy|LinkWalker|lwp-trivial|LWP::Simple|Mag-Net|MarkWatch|Mata.Hari|MIDown|MIIxpc|Missigua|moget|NAMEPROTECT|Navroad|NearSite|NetAnts|Netcraft|NetMechanic|NetSpider|NetZIP|NextGenSearchBot|NICErsPRO|niki-bot|NimbleCrawler|Ninja|NPbot|Octopus|Openfind|OutfoxBot|PageGrabber|pavuk|Pockey|ProPowerBot/2.14|ProWebWalker|psbot|Pump|QueryN.Metasearch|RealDownload|Reaper|Recorder|ReGet|RepoMonkey|Siphon|SiteSnagger|SlySearch|SmartDownload|Snake|Snapbot|Snoopy|sogou|SpaceBison|SpankBot|spanner|Sqworm|Stripper|Sucker|SuperBot|SuperHTTP|Surfbot|suzuran|Szukacz/1.4|tAkeOut|Teleport|Telesoft|TurnitinBot/1.5|The.Intraformant|TheNomad|TightTwatBot|Titan|True_bot|turingos|TurnitinBot|URLy.Warning|Vacuum|VoidEYE|WebAuto|WebBandit|Webclipping.com|WebCopier|WebEnhancer|WebFetch|GrapeshotCrawler|serpstatbot|commoncrawl|Web.Image.Collector|WebLeacher|WebmasterWorldForumBot|WebReaper|WebSauger|WebStripper|WebWhacker|WebZIP|Whacker|Widow|WISENutbot|WWWOFFLE|WWW-Collector-E|Xaldon|Zeus|ZmEu|Zyborg|Acunetix|FHscan|BLEXBot|Nimbostratus|SemrushBot|Slurp|DotBot) 1;
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0" 1;
}Перезагрузим NGINX
systemctl restart nginx
Теперь боты будут получать 444
