search script

[james438]

For a while now I have been working on a program that will search my site. The program has evolved a lot and I think has come a long way since I started. I also think that it is fairly fast too, but then again there are only 300+ files to search through in my database.

recently I came across an indexing program called sphider that does something called full text indexing, which is a new term for me. I am trying to figure out whether indexing is better and/or faster than what I am currently doing. As far as I can tell it should be pretty similar to doing a simple sql query.

I guess the question is whether indexing should be the next step in making my search program better.

thanks

[djr33]

indexing makes a lot of sense.

Just like google, you would create a set of data that represents the content of the searched pages.
This content could be the exact data from the page (saving the need to actually include it from the page), or it could be a list of keywords generated for the page, either automatically by searching/interpreting the page or manually with your input.

This is then searched, rather than the page itself, directly, and any matches are used to generate a link to that page, etc.

Basically, it might be harder than a direct search, but by indexing, you will be able to search faster in most cases. Indexing could be slow, but it would only be once, not each time it is searched. (Once in the sense that it need be done only once for every change; you would want to keep the index up to data.)

[james438]

interesting. It sounds like learning how to do full text indexing will be the next step. Two questions: do you hava a suggestion for where I can learn how to do text indexing? and
What if a page is updated or expanded? does the program have to index the site all over again?

[djr33]

Manual indexing would not be updated, then. Automatic would.
Automatic is harder to program, though.

I'm not really sure the best route to go about that. You might just want to embed the google search tool into your site.

[james438]

embedding a google search engine into my site is too easy and doesn't operate the way I want and I don't learn much that way ;) However, if I do give up I will install some sort of search program. That actually might not be that bad of an idea really. Why mess with what works? For now I kinda want to get my hands dirty and construct a program of my own that operates exactly how I want.

First, I need to learn how they do it though. I do have one rather large program that I can study. It is just a bit large to go through.

Any ideas where a good starting point might be? I know it is a bit of a vague question.

[alexjewell]

You could probably find something here: http://code.google.com/oss.html