Search Engine Indexing

> I‘ve always found the spidering of one‘s own site to be highly > ineffective and prone to error.  Some of the problems I‘ve seen (and > I‘ve done 10 page to 2+ million page sites):>   - navigation/copyright/boilerplate text frequently ends up > prominent > in the result or used in query evaluation ranking (such > content should > be excluded)Better search engines give you a way around this. For example, Ultraseek hasstop/start tags you can place around such content so it doesn‘t index it.Heck, even Fluid Dynamics Search Engine, a $40 Perl app, does that. Somebodywho‘s serious about search needs to spend some time looking at thearchitecture of their individual documents to ensure users get to just theinformation they need. This goes from excluding boiler plate to making suredocuments have meaningful titles and descriptions (how many sites have yougone to where a search results in 400 documents all of which seem to havethe same title and generic description?).>   - heavy load needs to be managed tightly so as to not conflict with > customers of the site (how many customers pull every page daily?);Solution: Run the search engine on a separate box. CPUs are cheap ...>   - spiders generally cannot tell when a page is no longer > reachable via > the site navigation (forgotten content still searchable);Then maybe this content should be pulled off the site, in which case thesearch engine would drop it after X number of attempts.> > You could put an event driven indexer into your content publishing > system.  Base it on the Verity, Autonomy, or Lucene engines (or > something else), and construct the ‘document‘ that they see > to contain > the specific content minus all of the wrapper.  You will only be > indexing/removing the content that has changed moments after > it has been > changed.This sort of leads into the old dynamic/static publishing question - thismodel only works if all your content is dynamic. Although we‘ve never playedwith it, Ultraseek does have an API that, theoretically, would let your CMSnotify the search engine of a new document as soon as it‘s published (ratherthan waiting for, say, your nightly crawl).> > Think if it the same way as publishing your site.  Do you > publish _all_ > the contents every night (and _only_ every night) throwing > out the old > and completely replacing it?  Or, do you publish only the > documents that > have changed whenever you need to?Good point. How you implement search should probably be dependent on how youpublish your pages. But Ultraseek doesn‘t re-index every page every night -it only grabs files with a new or changed timestamp.No, I don‘t work for Verity - I just like my Ultraseek :-).Adam GaffinExecutive Editor, Network World Fusion[EMAIL-REMOVED] / [phone] / http://www.nwfusion.com "I programmed my robotic dog to bite the guy who delivers the electronicmail." -- Kibo