Outtage earlier today

John

John

Founder
Staff member
Joined
Feb 6, 2001
Messages
8,173
Reaction score
429
Points
83
Age
46
Location
USA
Country
Ireland
Display Name
John Clare
Seemingly a process didn't terminate when it should have and cascaded into using all of the RAM on the server. That's why we shut down. I'm working with the ISP to figure out what exactly went wrong. Thanks.
 
There was another outtage today. And now I know the cause. It's yahoo's crawlers. They're swamping the site in terms of the resources they use. I'm banning yahoo's crawlers one by one as a result.
 
Sorry but whats a Yahoo crawler? Is it someone who found the sight by using the Yahoo search engine?
 
i_love_necturus said:
Sorry but whats a Yahoo crawler? Is it someone who found the sight by using the Yahoo search engine?
Click to expand...

I believe that Yahoo! Crawlers (and Google spiders) are tiny programs that access webpages so that search engines find them. I'm not really sure how they work but they help index and catalog words and terms and whatnot so that when you run a search you find webpages related to what you were searching for.
 
I wish you could - I wish there was a way like that to get rid of them.
 
Darn crawlers!

Man they sound like real b****ers!:mad: Is there any software (I doubt it) specializing in "killing" these crawlers and spiders available?
 
Last edited by a moderator:
John have you contacted Yahoo! about this or seen this help page on stopping crawlers from accessing your site? I don't know how helpful it is as I have no idea about servers or complicated web design but it seems to imply you just need a special configuration file warding off robots. Might be easier than banning each little robot.
 
Last edited:
That b word you're both using is somewhat offensive to people with sensibilities in the UK/Ireland so I bleeped it. It's a throw-away word here in the US but not where I'm from.

I've banned yahoo for now to see how the site behaves without them. I'll re-enable it after a while with a large crawler interval in the robots.txt for yahoo. Yahoo is a has-been search engine any how, and we don't receive many visitors through them.
 
You must log in or register to reply here.
General chit-chat
Help Users
  • No one is chatting at the moment.
  • Shane douglas:
    with axolotls would I basically have to keep buying and buying new axolotls to prevent inbred breeding which costs a lot of money??
     +1
    Unlike
  • Thorninmyside:
    Not necessarily but if you’re wanting to continue to grow your breeding capacity then yes. Breeding axolotls isn’t a cheap hobby nor is it a get rich quick scheme. It costs a lot of money and time and deditcation
     +1
    Unlike
  • stanleyc:
    @Thorninmyside, I Lauren chen
     +1
    Unlike
  • Clareclare:
    Would Chinese fire belly newts be more or less inclined towards an aquatic eft set up versus Japanese . I'm raising them and have abandoned the terrarium at about 5 months old and switched to the aquatic setups you describe. I'm wondering if I could do this as soon as they morph?
     +1
    Unlike
    Clareclare: Would Chinese fire belly newts be more or less inclined towards an aquatic eft set up versus... +1
    Back
    Top