search engines and forums

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

Post Reply
User avatar
Forum Regular
Posts: 574
Joined: Fri Sep 05, 2008 3:34 pm
Location: Victoria, BC

search engines and forums

Post by Vegan »

I spent much time over the last several weeks checking robots.txt for social media and forums. No surprise is that most block search engines from crawling them.

This forum is crawled by Google at al so as a result any solutions here are easily found and it cuts down on the same old post surfacing over and over again. While forums etc have local search the lack of skill makes it hard to leverage.

Not many forums allow search but some do. I suggest forums based on phpBB use a careful robots.txt that blocks user profiles but allow reading posts.

I noticed it took Google more than 90 days to discover my posts here. This suggests it may be some time before this forum comes back to life. There is a lot of good content so hopefully it bounces back.

Bing has been better at discovering backlinks but yet content crawling is really slow. After 90 days it has not even discovered 300 pages of some 6000 or so on my site.
Hardcore Games™ Legendary is the Only Way to Play™
My site is powered by LAMP
Post Reply