robots.txt rus

Date: Tue, 25 Feb 1997 17:00:33 -0400
From: "David L. Potter" <ab934@chebucto.ns.ca>
To: CSuite Development List <csuite-dev@chebucto.ns.ca>,

next message in archive
no next message in thread
previous message in archive
previous message in thread
Index of Subjects

Index of Subjects

On Tue, 25 Feb 1997, Edward Dyer wrote:

> I think this is an issue that should see discussion on CSuite-Dev mailing 
> list, so I have copied it there.

I hadn't mentally prepared myself for discussion so this will be fairly 
short...

With the acknowledgement that any site is free to change or delete _at 
will_ the robots.txt included in the distribution...

1) we would prefer that any indexing of CSuite documents be done on the 
most recent docs and our assumption is that those will reside on chebucto...

2) as a frequent user of search engines I'm fed up with the reams of 
garbage and would philosophically limit off-site indexing to areas that 
can be expected to return the highest signal to noise...

As a result I started with the assumption that I would exclude anything I 
didn't see exceptional value in... hence excluding ip.html and jumps.html
I have a link to my homepage in Help/...;-)

3) When I suggested that Services/ and Policy/ would get picked up under  
Help I was thinking that a user would find a reference to the topic in 
Help/* and could follow the link themselves... 

4) Anything under recent/ will get picked up elsewhere... to me it just 
looks like adding unnecessary volume which to me is part of theproblem.

jmo

---------------------------------------------------------------------
David Potter                 http://chebucto.ns.ca/CSuite/CSuite.html
Documentation Team                             Chebucto Community Net
============== CSuite - Community Network Software ==================


next message in archive
no next message in thread
previous message in archive
previous message in thread
Index of Subjects