Laurent: several relational databases provvide full text serach capabilities.

You can send a SELECT statement with a MATCH() function to the server and get
the result of search back (in case of MySQL). I think OpenDBX
library has to do nothing special here: it depends on which query you write.


�MySQL and PostgreSQL server has support for full text search (aka.
tsearch in PostgreSQL I think).


cheers

mariano

On Fri, Jan 28, 2011 at 3:57 AM, Norbert Hartl <norbert@hartl.name> wrote:

On 27.01.2011, at 20:48, laurent laffont wrote:

> Hi,
>
> I need to have full text indexing and searching over a 3-5 million documents. Actually the products I work on are written in PHP - MySQL.
>
> What would you use (open source - mandatory - our products are open source) in Smalltalk world �to do this ?
>
That topic troubled me some time in the last years. The first approach I took was postgresql+tsearch2 because I used Glorp back then. AFAIK tsearch2 has been incorporated into postgresql in the meantime so this is no extra weight. Stemming libraries for postgresql are quite good and available for some languages. If you would use Glorp you can even extend the query language to support it out of the box.
I changed the setup as I moved from postgresql to gemstone. With the new setup I would have used postgresql only for the fulltext retrieval which I didn't like. I used SOLR [1] instead. It is a search server build upon lucene which supports faceted search and a lot of nifty stuff. I used magritte to generate the xml that I've put into SOLR. I was satisfied with this solution. But all solutions that need to cross the smalltalk boundary start to become cumbersome at some stage.
If I find time I would take another approach. I found a smalltalk stemming library while observing the moose mailing list. If you have good stemming support it may be a reasonable effort to produce forward and inverted indexes of your objects with search capabilities. If you use magritte (or pragmas) then you could add the fields to index there. The rest would be automatic: Taking all fields that should be searchable. Tokenize and canonicalize (using stemming) the words and build forward and inverted index. The only thing that would be needed is a change notification system that updates the index but I think this is more of an easy task. Well, that is the sketch what I would try to do myself but don't find time right now. I personally think that the power you gain by search smalltalk objects directly outweighs all the external servers by some extent.
If you use a special external database anyway that supports it, well, then this would be the easiest start.

hope this helps,

Norbert