Crawlers eating your bandwidth?

The Internet is a active landscape with something going on all the time. Often times you hope the visitors are those seeking what your site offers, but sometimes it's the search engine ‘bots’ crawling your site.  These bots can and do consume quite a bit of your website hosting bandwidth and CPU power resources. Bot's without some rules will  hammer your site mercilessly in an attempt to fully index it. Controlling them and limiting their ability to hammer away at your site is handled through a special file that resides in the root of your website called…

0 Comments