Options

Google is picking up the forums

Alexan DriteAlexan Drite Registered User regular
I remember reading a few years back that Alpha had marked the forums in some way such that it would disable search engines from picking them up. I remember because I tried to access the web.archives.com and it said the forums were marked specifically in such a way to not allow web crawlers to pick it up.
I believe, the term is "The Robots Exclusion Protocol", though I am not a web master so I wouldn't know.

Apparently this no longer is working, as google searches of penny-arcade.com are picking up forum topics. Just a heads up in case it's not supposed to be that way. Update your robots.txt file or something.

Alexan Drite on

Posts

  • Options
    Seattle ThreadSeattle Thread Registered User regular
    edited February 2006
    Hmm, I've tried doing just that and it's working OK for me... only by searching "Penny Arcade Forums" does Google return something forum-related, and then it's only the Index.

    Seattle Thread on
    kofz2amsvqm3.png
  • Options
    ÄlphämönkëyÄlphämönkëy Registered User regular
    edited February 2006
    Yeah, I turned that off now that it isn't a major load on the servers. Thanks for keeping an eye out though.

    Älphämönkëy on
Sign In or Register to comment.