We have legacy system running with data uploads in place, which is basically doing in memory operations. Since the upload of records involve so many other tasks like triggering workflow and so, it will take lot of time. Causing memory issues and slowing down the web operations as well.
1) As a first thing we are planning to remove the in-memory operation and planning insert data to temp tables and process from there in batches.
2) Making web upload requests as a background job and to facilitate webservice requests as well.
we are Planning to run the batch jobs from a different server in order to avoid hitting the heap space of the jvm where web applications are running.
Background jobs can use some temporary database like derby to store the temp table data, so that we can avoid maintaining too many tables/avoid storing too many xml data in the main database.
Please suggest any better ways to implement the above points.
Note: All the contents we are dealing with is dynamic. Data is been processed entirely in the form of xmls and we transform xml data to database tables before inserting. This BPM application is an engine which serves dozens of applications. Since there are too many others dependency spring batch cant be implemented.