Outtage earlier today

[John]

Seemingly a process didn't terminate when it should have and cascaded into using all of the RAM on the server. That's why we shut down. I'm working with the ISP to figure out what exactly went wrong. Thanks.

 

[John]

There was another outtage today. And now I know the cause. It's yahoo's crawlers. They're swamping the site in terms of the resources they use. I'm banning yahoo's crawlers one by one as a result.

 

[i_love_necturus]

Sorry but whats a Yahoo crawler? Is it someone who found the sight by using the Yahoo search engine?

 

[Abrahm]

Sorry but whats a Yahoo crawler? Is it someone who found the sight by using the Yahoo search engine?

I believe that Yahoo! Crawlers (and Google spiders) are tiny programs that access webpages so that search engines find them. I'm not really sure how they work but they help index and catalog words and terms and whatnot so that when you run a search you find webpages related to what you were searching for.

 

[i_love_necturus]

Oh I see. That makes sense, thanks!

 

[peter81965]

yahoo crawlers

John, can I feed my paddletails on live yahoo crawlers?

 

[John]

I wish you could - I wish there was a way like that to get rid of them.

 

[i_love_necturus]

Darn crawlers!

Man they sound like real b****ers! Is there any software (I doubt it) specializing in "killing" these crawlers and spiders available?

 

[Abrahm]

John have you contacted Yahoo! about this or seen this help page on stopping crawlers from accessing your site? I don't know how helpful it is as I have no idea about servers or complicated web design but it seems to imply you just need a special configuration file warding off robots. Might be easier than banning each little robot.

 

[John]

That b word you're both using is somewhat offensive to people with sensibilities in the UK/Ireland so I bleeped it. It's a throw-away word here in the US but not where I'm from.

I've banned yahoo for now to see how the site behaves without them. I'll re-enable it after a while with a large crawler interval in the robots.txt for yahoo. Yahoo is a has-been search engine any how, and we don't receive many visitors through them.

 

[i_love_necturus]

Sorry about the "b" word, never knew it ment anything bad.