So I have a wonderfully large stats website and bing likes to "attack" it while scraping its contents for its search engine. I don't want to block it though. I would rather have it be like google, where is just searches at a nice content rate, and I was wondering if anyone here has a good idea on how to do that?
user-agent:bingbot
crawl-delay:1
For robots.txt, but it doesn't change the rate of the lookup, just makes it happen at a very very very slow rate. Any ideas?