Performance and scalability: Using in-memory database for data processing
Dear Experts, I've searched for information on what seems to be a simple question but cannot find an answer. My question is, would it be a good idea to use an in-memory database, such as HSQLDB or H2, for data processing, instead of creating complicated structures of Maps and Lists? In other words, what do you think of the idea of using an in-memory db, just for the duration of your data processing, to store your calculations and provide an easy way to query the results? Just throwing the idea out there, I'd like to know what you think, if you've ever tried something like this.. let me know if you need more details.. Thanks in advance Frederic
- Posted by: Freddy Daoud
- Posted on: April 26 2007 16:47 EDT
- Re: Using in-memory database for data processing by Eugene Krivosheyev on May 09 2007 19:15 EDT
- Re: Using in-memory database for data processing by Cameron Purdy on May 21 2007 15:06 EDT
- Re: Using in-memory database for data processing by N B on May 23 2007 01:14 EDT
Hi, Frederic. It's good idea, we use such approach in some apps. But u need an architecture analisys and stress testing in every case to understand what approach to use.
My question is, would it be a good idea to use an in-memory database, such as HSQLDB or H2, for data processing, instead of creating complicated structures of Maps and Lists?Depends on if you are wanting to use SQL or Java to work with the data. If SQL, then check out TimesTen. If Java, and you are clustering, then check out Coherence. Peace, Cameron Purdy Tangosol Coherence: The Java Data Grid
Thanks Cameron and everyone for your responses. I ended up trying with H2, which worked out nicely, and JoSQL, which worked out even nicer and is my current solution. Cheers, Frederic
Hi Frederic, Although I'm new to this domain and can't say am expert..... However feel this is good idea..we have used this for certain complex logics in project... I think , it makes amendments/changes and bug rectification easy specially after deployment.. But not sure how it will affect performance if used for all kinds of data processing...probably a database experts can have their say on it..