検索エンジン対策

Apache2 httpd.conf

http://fuga.jp/~densuke/hiki/hiki.cgi?Apache参照。
Googlebotは通すけど、GoogleBotは弾くわよ。
GoogleBotは某NaverBOTの詐称なの。

SetEnvIf User-Agent "pompos.html" BadRobot nolog
SetEnvIf User-Agent "WWWC" BadRobot nolog
SetEnvIf User-Agent "WebFetch" BadRobot nolog
SetEnvIf User-Agent "htdig" BadRobot nolog
SetEnvIf User-Agent "ia_archiver" BadRobot nolog
SetEnvIf User-Agent "Openfind" BadRobot nolog
SetEnvIf User-Agent "TurnitinBot" BadRobot nolog
SetEnvIf User-Agent "dloader" BadRobot nolog
SetEnvIf User-Agent "FAST-WebCrawler" BadRobot nolog
SetEnvIf User-Agent "Sidewinder" BadRobot nolog
SetEnvIf User-Agent "Mercator" BadRobot nolog
SetEnvIf User-Agent "BaiDuSpider" BadRobot nolog
SetEnvIf User-Agent "WISEnutbot" BadRobot nolog
SetEnvIf User-Agent "mogimogi" BadRobot nolog
SetEnvIf User-Agent "Rumours-Agent" BadRobot nolog
SetEnvIf User-Agent "Openbot" BadRobot nolog
SetEnvIf User-Agent "Scooter" BadRobot nolog
SetEnvIf User-Agent "SideWinder" BadRobot nolog
SetEnvIf User-Agent "CyberSpider" BadRobot nolog
SetEnvIf User-Agent "Gaisbot" BadRobot nolog
SetEnvIf User-Agent "InternetLinkAgent" BadRobot nolog
SetEnvIf User-Agent "slurp@inktomi.com" BadRobot nolog
SetEnvIf User-Agent "NPBot" BadRobot nolog
SetEnvIf User-Agent "searchhippo" BadRobot nolog
SetEnvIf User-Agent "grub-client" BadRobot nolog
SetEnvIf User-Agent "MSNBOT" BadRobot nolog
SetEnvIf User-Agent "www.kokotoi.org" BadRobot nolog
SetEnvIf User-Agent "Tkensaku" BadRobot nolog
SetEnvIf User-Agent "HenryTheMiragoRobot" BadRobot nolog
SetEnvIf User-Agent "GoogleBot" BadRobot nolog

Deny from env=BadRobot

CustomLog logs/access.log combined env=!nolog
CustomLog logs/badrobot.log combined env=BadRobbot