Comments on: HAvroBase: a searchable, evolvable entity store on top of HBase and Solr

By: spullara

spullara — Thu, 08 Jul 2010 08:54:12 +0000

This is using Solr completely independently of HBase and no indexes (except primary) are stored in HBase.

By: Sergio Bossa

Sergio Bossa — Sat, 03 Jul 2010 15:14:19 +0000

Thanks for your kind words Sam.
Solr merits over ElasticSearch (or vice-versa) apart, coding a Solr integration for Terrastore is a matter of hours, thanks to Terrastore event system.

Anyways, good luck with your system

Sergio B.

By: spullara

spullara — Sat, 03 Jul 2010 02:22:37 +0000

Terrastore looks promising but the choice of Elastic Store over Solr doesn't seem well thought out. The advanced search and indexing capabilities of Solr far outweigh its deployment difficulties.

By: Sergio Bossa

Sergio Bossa — Fri, 02 Jul 2010 21:04:00 +0000

Also, Terrastore (http://code.google.com/p/terrastore/) provides all requested features: key ranges, atomic updates, evolvable schemas through custom serializers/deserializers by the client side, lucene integration via ElasticSearch … you may want to take a look

Cheers,

Sergio B.

By: spullara

spullara — Thu, 01 Jul 2010 23:50:24 +0000

If MongoDB has those hooks, which I did not investigate, it seems like it could be a reasonable solution. Here is a reference to a few other libraries that do something similar for other NoSQL stores + Solr:

http://nosql.mypopescu.com/post/383437318/integ...

By: spullara

spullara — Thu, 01 Jul 2010 23:48:54 +0000

Right now doing this in the client. Sounds like you are a little further down the “implement guarantees around indexing” track than I am right now.

By: spullara

spullara — Thu, 01 Jul 2010 23:48:15 +0000

There are a couple strategies that I am thinking about. HBase is adding coprocessors and that might give me the right hooks. Simplest would be writing to HBase twice for each update, once to insert the update and again when the index is complete with a dirty bit and periodically scanning for inconsistencies. Kind of a hack but for my use case could be ok.

By: spullara

spullara — Thu, 01 Jul 2010 23:45:26 +0000

You could use any key value store that has all the properties listed above. I chose HBase but you could choose MySQL or Memcache as long as it has the right properties. For example, I want to be able to scan by key, so that leaves out Memcache. I want to be able to scale across many machines easily, that leaves out MySQL. I want atomic compare and set, so that leaves out the current version of Cassandra.

By: Otis Gospodnetic

Otis Gospodnetic — Thu, 01 Jul 2010 21:26:37 +0000

Hm, but I was trying to point out/ask whether HAvroBase is actually using a different approach than HBasene all together.

I believe HBasene is essentially saying "instead of storing Lucene indices in the raw FS, we'll model the indices in various HBase column families, so the actual index will live in a HBase database".

vs. how I understand HAvroBase from your post, which is:

"I want to store data in a DB (HBase), but I also want it searchable, so let me create a component that both puts data in HBase and also indexes it in regular, external Lucene indices (i.e. they still end up living in the raw/local FS)".

Maybe I misunderstood how HAvroBase uses Lucene? Does HAvroBase also model Lucene indices inside HBase and when one uses HAvroBase there are no Lucene indices in the local FS?

Thanks.

By: Otis Gospodnetic

Otis Gospodnetic — Thu, 01 Jul 2010 21:22:14 +0000

Doesn't MongoDB have some hooks to index everything that goes in it into an external Solr instance? Or maybe in external Lucene indices?