Support Forums: Message List - Article: Spring Batch Overview http://www.theserverside.com Most recent forum messages en Jive Forums Silver 5.5.30 (www.jivesoftware.com) Sat, 18 May 2013 10:33:20 -0400 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 Now, it's absolutely possible that there is little to nothing that can be done until the end of the month. In that case, the event-based solution doesn't really provide any benefits and it may be the appropriate choice to not use...]]> Tue, 20 Nov 2007 04:41:11 -0500 Tue, 20 Nov 2007 04:41:11 -0500 Tue, 20 Nov 2007 04:41:11 -0500 Nov 20, 2007 John Brand 0 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 Ok, then I understand.

Just to explain my point of view: 2*queue put=2*10ms, plus time for conversion (20 ms?), plus time for real work (40ms) = 80ms. This would move my typical/prototype month-end batch job from 9-12 hours to 48+...]]> Fri, 16 Nov 2007 11:58:49 -0500 Fri, 16 Nov 2007 11:58:49 -0500 Fri, 16 Nov 2007 11:58:49 -0500 Nov 16, 2007 James Watson 1 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 We are using file triggers (not SQL triggers) and writing the new and old row data to the queue in raw form. That is, we are writing basically an array of bytes to the queue. This executes fairly quickly. It's not completely insignificant....]]> Fri, 16 Nov 2007 04:21:46 -0500 Fri, 16 Nov 2007 04:21:46 -0500 Fri, 16 Nov 2007 04:21:46 -0500 Nov 16, 2007 John Brand 2 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506
Pushing the raw bytes from the record to a queue lets us capture the state without blocking the execution of the updates.


Well, I still dont know the details of your problem, and I am sure you have picked the...]]> Thu, 15 Nov 2007 09:53:41 -0500 Thu, 15 Nov 2007 09:53:41 -0500 Thu, 15 Nov 2007 09:53:41 -0500 Nov 15, 2007 James Watson 3 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 Pushing the raw bytes from the record to a queue lets us capture the state without blocking the execution of the updates....]]> Thu, 15 Nov 2007 04:19:13 -0500 Thu, 15 Nov 2007 04:19:13 -0500 Thu, 15 Nov 2007 04:19:13 -0500 Nov 15, 2007 John Brand 4 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 True, but I think that a more conventional design of the original batch would be a mix between event and batch, i.e. changes to the data would mark it as changed (using a trigger or other means) which would make it easy and efficient to pick...]]> Wed, 14 Nov 2007 14:48:39 -0500 Wed, 14 Nov 2007 14:48:39 -0500 Wed, 14 Nov 2007 14:48:39 -0500 Nov 14, 2007 James Watson 5 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 The example I give above is a case in point. Searching though millions of records for changes is much less efficient than capturing the change event for a few thousand records and sending a message to a queue. The search can take several...]]> Wed, 14 Nov 2007 14:07:17 -0500 Wed, 14 Nov 2007 14:07:17 -0500 Wed, 14 Nov 2007 14:07:17 -0500 Nov 14, 2007 John Brand 6 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 Its tempting to try to resolve these issues by throwing out the batch-model (no batches means no batch window, right...), replacing it with either an event based model or by putting the work in online transactions. Both approaches are...]]> Wed, 14 Nov 2007 10:22:41 -0500 Wed, 14 Nov 2007 10:22:41 -0500 Wed, 14 Nov 2007 10:22:41 -0500 Nov 14, 2007 James Watson 7 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 I think we have a few conclusions:

1. No single application model fits all circumstances - specifically, some problems are well-suited to queue based models; some to batch; some to other (e.g. request/response)

2. Care should be...]]> Wed, 14 Nov 2007 10:06:05 -0500 Wed, 14 Nov 2007 10:06:05 -0500 Wed, 14 Nov 2007 10:06:05 -0500 Nov 14, 2007 John Brand 8 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 Ok James, so this has been an interesting discussion....]]> Wed, 14 Nov 2007 09:33:41 -0500 Wed, 14 Nov 2007 09:33:41 -0500 Wed, 14 Nov 2007 09:33:41 -0500 Nov 14, 2007 James Watson 0 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 ...]]> Tue, 13 Nov 2007 22:10:33 -0500 Tue, 13 Nov 2007 22:10:33 -0500 Tue, 13 Nov 2007 22:10:33 -0500 Nov 13, 2007 Chris Vignola 10 Re: the batch playing field http://www.theserverside.com/discussions/thread.tss?thread_id=47506
And on a more general level, where I work now, I think we have little or no need for batching but more everything is done as a batch. The batch never completes on time causing a lot of headaches. We end up spending more on...]]> Tue, 13 Nov 2007 14:04:52 -0500 Tue, 13 Nov 2007 14:04:52 -0500 Tue, 13 Nov 2007 14:04:52 -0500 Nov 13, 2007 James Watson 0 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506

The batch approach is not only slow but results in data integrity issues. We end up with stale and incorrect data.




Batch patterns are not inherently slow, nor do they inherently compromise data...]]> Tue, 13 Nov 2007 13:48:19 -0500 Tue, 13 Nov 2007 13:48:19 -0500 Tue, 13 Nov 2007 13:48:19 -0500 Nov 13, 2007 James Watson 11 Re: the batch playing field http://www.theserverside.com/discussions/thread.tss?thread_id=47506 And on a more general level, where I work now, I think we have little or no need for batching but more everything is done as a batch. The batch never completes on time causing a lot of headaches. We end up spending more on infrastructure to...]]> Tue, 13 Nov 2007 13:04:25 -0500 Tue, 13 Nov 2007 13:04:25 -0500 Tue, 13 Nov 2007 13:04:25 -0500 Nov 13, 2007 Gary P 1 Re: Can someone enlighten me? http://www.theserverside.com/discussions/thread.tss?thread_id=47506 Tue, 13 Nov 2007 12:39:36 -0500 Tue, 13 Nov 2007 12:39:36 -0500 Tue, 13 Nov 2007 12:39:36 -0500 Nov 13, 2007 Chris Vignola 12