I have a requirement to create a web service client to upload 150,000+ records from our database. My main question is whether others are creating web services to process bulk data and how they handle memory concerns.
I began with the standard API, populating the jax-b annotated beans created from the wsimport. I used JConsole to analyze memory usage. I noticed that there is a sharp spike in the plot of the heap usage (from 200MB to 800MB) right after the invoke method call.
The problem is that the max heap size on our app server is set to 1GB and since this application shares it with other applications I keep getting a out of memory error.
I was asked to look for alternatives, so I tried using the Dispatch API creating the payload by sub-tree marshaling to StAX. The issue I then ran into is that I have a outputstream and the dispatch api is expecting an inputstream. I tried the options found here http://ostermiller.org/convert_java_outputstream_inputstream.html
However after using jconsole to analyze again it appears the writing to a temp file or using a byte array increases the amount of memory being used to almost equivalent of using the standard API. I tried using piped streams in separate threads and the single threaded circular buffer methods, but the application seems to hang after the invoke method call.
Any help would be appreciated.