Compass' upcoming version will support native integration of Lucene with DataGrid vendors GigaSpaces and Oracle Coherence allowing native storage of Lucene index data within a fast, distributed, probably partitioned, Data Grid. This integration allows Lucene based frameworks such as Compass, Hibernate Search, and Solr, as well as pure Lucene applications, to change the index storage mechanism to a distributed one with a few lines of code changes (thanks to Lucene's Directory abstraction). On top of this, Compass is integrated with Coherence and GigaSpaces "Write Behind" support (GigaSpaces mirror and Coherence CacheStore) to automatically index a DataGrid. The implementation will, in a reliable asynchronous (up to the vendor) manner, reflect any changes done to the DataGrid to be mirrored to the search engine allowing to create a "Searchable DataGrid" very simply. More information, including some simple code examples, can be found here. What do you think? Does having the ability to store Lucene data in a distributed shared memory system make sense to all of you Lucene users? Do you think having google-like search capabilities on top of a DataGrid is something you need?

I think I need to get him to do ObjectGrid also :)

I think I need to get him to do ObjectGrid also :)

Next milestone, definitely!

I've implemented an Apache Lucene Directory store for IBM WebSphere eXtreme Scale now. It's available with source code at http://www.github.com/bnewport

I'm not sure if this will work with Compass though.

Hey, way to go Shay! This is very cool stuff.

I have been playing around with a prototype for a JBoss Cache plugin for Lucene as well. We'll hopefully release + blog about this shortly; something you might find useful to include with Compass.

I have been playing around with a prototype for a JBoss Cache plugin for Lucene as well.

We'll hopefully release + blog about this shortly; something you might find useful to include with Compass.

Does JBoss Cache support partitioning? Based on their FAQ they do not. It makes little sense to store the Lucene index in a DataGrid without partitioning...

Perhaps a combination of Hibernate Search(Written for Lucene) and Hibernate Shards(written to partition Hibernate stores) can bridge the gap between these technologies?

Perhaps a combination of Hibernate Search(Written for Lucene) and Hibernate Shards(written to partition Hibernate stores) can bridge the gap between these technologies?

Zip I believe that the difference with this approach is that it is purely in-memory and enable to partition the entire lucene index in-memory. From what i understood any application that is using Lucene could benefit from this integration and that can should include Hibernate Search. Nati S.

Does JBoss Cache support partitioning? Based on their FAQ they do not. It makes little sense to store the Lucene index in a DataGrid without partitioning...

JBoss Cache does support partitioning when used with GridGain Affinity Load Balancing. You basically can load the whole database into JBoss Cache partitioned on a cluster of nodes (each partition belongs to a certain node) and then send your computations exactly to the node where your data is. It's just a matter of configuration and you should not have to change your application logic. Best, Dmitriy Setrakyan GridGain - Grid Computing Made Simple

.. Java as a platform is definitely continuing to push the envelope. Nice job, Shay, and maybe in a few years we'll see some standardized APIs for all of this, once we all sort out which functionality is obvious and common. Peace, Cameron Purdy Oracle Coherence: The Java Data Grid

What do you think? Does having the ability to store Lucene data in a distributed shared memory system make sense to all of you Lucene users? Do you think having google-like search capabilities on top of a DataGrid is something you need?

Yes. Most definitely. This is great news.

I for one think that searching collections in Java is just ridiculous and well thought through search (w/o a DB) is something sorely needed in the JDK. I do want to understand the differences between our existing Lucene module for Terracotta first. We take the memory store and cluster that transparently, but Compass is different. I am looking forward to working together on a Compass / Terracotta module. --Ari

I for one think that searching collections in Java is just ridiculous and well thought through search (w/o a DB) is something sorely needed in the JDK.

I do want to understand the differences between our existing Lucene module for Terracotta first. We take the memory store and cluster that transparently, but Compass is different.

I am looking forward to working together on a Compass / Terracotta module.

--Ari

The part where you can store a Lucene index with Terractota can be utilized by Compass as well. The interesting part is to have DSO automatically indexable and searchable by Compass. A Compass module is certainly in place.

Shay Sorry for jumping in late.. The fact that i can "google" my Java application (Domain Model) and for that matter any application is a Killer Application (and indeed a good candidate to win our Killer application contest. Shay one thing to note that maybe wasn't clear from your post is that this integration enable to scale out lucene even if you don't use Compass on top of it. Freedom of choice is another important value, unlike other solutions that support lucene the fact that your solution can integrate with different In Memory Data Grids is an important value, something that i was missing from other solutions. Nati S. GigaSpaces