Discussions

General J2EE: File downloads and memory

  1. File downloads and memory (4 messages)

    Has anybody had experience with file download issues in a J2EE application? Our users are given the ability to upload and download files. We store these files in the db as blobs. At the moment when a user requests a file we get the attachment content into the ORM corresponding object AttachmentContent, convert it into a value object e.g. AttachmentContentDVO, fire the AttachmentContentDVO over the wire to the web-tier and stream it out to the browser. For large files this quickly exhausts our allocated memory resources. We are running 1.5G for the WLS app server but still we are getting java.lang.OutOfMemory exceptions. I have not been able to prove 100% that the abuse of file upload/download (repeated clicks for files etc) is the issue but I think it is a big contributing factor.

    So my question is has anybody had a similar experience? At the moment I am thinking we should break our design and go straight to the db for the blob from the web tier, avoiding the EJB tier (big model objects in memory, dvos, the marshalling etc).

    Regards.
    J

    Threaded Messages (4)

  2. File downloads and memory[ Go to top ]

    Hi,
    Instead of passing around the full blob content, try to pass a refference to the input stream obtained from the blob. The download servlet reads from the blob input stream and writes into the client(browser) output stream. You have to be carefull to close the blob output stream (+ result set + statement) when the download is over or when an IO exception occurs.
    Best regards, Mircea
  3. File downloads and memory[ Go to top ]

    Can you send across the implementation,It will be easy to find the actual cause of the problem.

    From your problem, I think you are reading the whole blod in the buffer which is consuming all the memory.

    As suggested you pass the BlobInputStream to ServletOutputStrem or you can use the above class.
    import java.io.*;
    import java.net.*;

    public class IOMapper
    {
      public static void map( InputStream in,OutputStream out) throws IOException
      {
    byte b[] =new byte[1024*4];
    int inumber = 0;
    while( true)
    {
    inumber = in.read(b);
    if( inumber <= 0)
    break;
    out.write(b,0,inumber);
    out.flush();
    }

    }
    public static void main(String [] argv) throws Exception
    {
    IOMapper.map(System.in,System.out);
    }
    }
  4. File downloads and memory[ Go to top ]

    Thank you for your responses. Yes it is true that the entire blob is in memory thus the impact on memory usage.

    We are doing something similar to what the person who posted
    http://www.theserverside.com/discussions/thread.tss?thread_id=23125 and we are getting similar results in a similar environment (WLS 704 on Solaris)

    I am aware of how to stream the content via the ServletOutputStream to the browser. My problem is our current J2EE architecture which involves us having to pass a value object over the network to the EJB for upload and to the WEB tier for download. As a result I cannot stream the blob content directly from the database to the browser.

    Current Process:
    &#61623; User clicks on the link to file they want to download.
    &#61623; Application goes to database and retrieves attachment content based on the attachment ID.
    &#61623; On the EJB tier a new persistent object called AttachmentContent with the appropriate content type (PDF, MS Word etc) is created in memory.
    &#61623; Once AttachmentContent is constructed, an AttachmentContentDVO object is constructed with the values of AttachmentContent for passing the contents of AttachmentContent to the WEB tier of the application.
    &#61623; The AttachmentContentDVO is sent over the network to the WEB tier. This involves the marshalling and demarshalling of the serializable AttachmentContentDVO.
    &#61623; The WEB tier gets the AttachmentContentDVO object, opens a Servlet output stream to the browser and flushes the AttachmentContentDVO content to the user.

    Current Problems:
    &#61623; Memory usage and the JVM (Java Virtual Machine or Java runtime environment). The creation of AttachmentContent and the AttachmentContentDVO objects means the amount of memory needed to send an Attachment to the browser is doubled. It also puts more strain on the JVM to properly garbage collect large amounts of memory used by Attachment content. It has been proven that concurrent download requests by multiple users can rapidly exhaust server memory resources leading to more frequent garbage collections by the JVM. Since large garbage collections by the JVM can greatly affect the next point - performance.
    &#61623; Performance. The creation of AttachmentContent and the AttachmentContentDVO objects means the application spends time creating two memory objects with the same values. The marshalling and demarshalling of the AttachmentContentDVO object adds to the time taken to send the Attachment to the user.


    Proposed Solution:
    Access the database straight from the web tier for attachment upload and downloads. This way we can stream the content straight from the db to the client browser.

    Would like to hear anyone's thoughts.
  5. File downloads and memory[ Go to top ]

    Yup!!! You are right, I didn't gave a thought about the Bean and then servlet Stream sorry :-).

    So again to repeat you still have 2 option.If you are not running in the same JVM
    1) Create a temp file from Bean and passing the file name to the Servlet to output the content.( You will require a common share of harddisk in this case).

    2) Read the BLob as a RandomAccessFile in term of the byte array and then you can output in form of small chunk.

    3) Directly access Database from Web interface.

    Regards,
    Hanuman