Hi , I need advice how return to client large amount of data

Discussions

EJB design: Hi , I need advice how return to client large amount of data

  1. Hi . how return to client large amount of data ???
    I need build report at client side,
    I call Session bean that execute query to database
    and need to return data. but the result of query may produse
    a large amount of data, and trying send it to client cause
    error out of memory. As I understood all the data serialized and only
    after that sending to client. but i needn`t this, i only need pipe between
    data and client.

    how to obtain this not use large amount of memory on server ????
  2. Andrey[ Go to top ]

    Andrey,

    Personally, I don't have practical experience of working with such large amounts of data that overflow your memory.

    But I am sure that you have to retrieve your data in portions. This could be done in the following ways.

    1. Use of InputStream.

    If I were you, I would probably look in the direction of org.omg.CORBA.portable.InputStream class and javax.rmi.CORBA package. You should get your SQL data as an OutputStream and than try to associate it with CORBA InputStream (don't ask me how, it's just a theory).

    2. Return data from the bean in discrete portions - the bean has to be Stateful. You don't close your statement (and connection) and iterate through ResultSet (RowSets will not work here as long as they are serialized) with each client's request and close connection only after you finish with your data (i.e. reach the end of your ResultSet). I assume that you include the following method into your stateful bean:

    Object getDataChunk (int chunkNumber);

    This method should get the current ResultSet being iterated from bean's variable scope (actually, just a pointer) and retrieve the next N records from the database (you could fiddle a bit with prefetch parameters of your JDBC driver to synchronize). This should work in practice - the only drawback that you will have pretty long connections with your database.

    3. Install more memory to your server. The most simple way.

    Hope it helps.
    Alex
  3. First, make sure your client is running with sufficient memory. By default, the JVM will only use 64M of memory, and after that, it throws OutOfMemoryErrors. Depending on the client configuration, you should increase this with the -Xm flag (see docs for more info).

    If you are at your limit, or cannot increase memory any more, then you have several options. The previous poster had some good suggestions. YOu can also redesign your session bean to only return a portion of the data to you at a time. If your server has a lot of memory, you could use a stateful session bean to hold ALL the data, and send a subset of it to the client at a time.

    Of course, on the client side, you may still have to deal with large amounts of data, and for that, you will probably have to use secondary storage (i.e. disk) to cache the data. If you are generating a report, you can write the report to disk in parts, so you never have the entire report in memory at once.
  4. If you are absolutely positive that you want to use Session Beans to do this, then they need to be statefull.

    Another question you need to answer is , do you want forward reverse scrolling functionality? I am going to assume this is a given,


    Your session bean interface could probably look like this.

    VO<Collection> getReportData ( int page , int numRows ) ;
    int getReportTotalRows () ;
    int getReprtTotalPages ( int numRowsPerPage ) ;

    Here the basic idea is that the bean returns a collection of Value Objects representing the data to be reported. The client tracks the page number and num of rows to be displayed in each page ( its nice to have this configurable ) and passes updated values of these everytime it wants data.

    The getReportTotalRows tells you how many rows are there in the report.

    Client side invocation will look something like this -

    VO<Collection> coll = bean.getReportData( 0 , 100 ) ; // Page number start with 0 and give me the first hundred rows


    VO<Collection> coll = bean.getReportData( 1 , 100 ) ; // Page number start with 0 and give me the first hundred rows

    and so on. You can go back too with this interface. I guess you get the drift on the Interface.


    From an implementation standpoint, you need to create a Scrollable result set. Look at the JDBC documentation on ResuleSet to find out how. YOu need to make sure you store the result set for retreival the next time, ( Well technically you dont , but then you'll have to create it evertime which is epensive ). Calculate your absolute position based on the page number and numRowsToDisplay and scroll through the set to get to the location and generate your ValueObjects.


    You can acheive the same results through straight Servlet code by caching the ResultSet in your HTTPSession on the server side too.

    Hope this helps. Let me know if you need further information.
  5. Oh I forgot to mention , make you you set your fetch size on your statement and result set appropriately. It impacts performance in a large way ( try it out its fun to see ) and ofcourse as the others stated make sure you bump up your servers memory too.
  6. hi, thank for reply,,,,[ Go to top ]

    If you are absolutely positive that you want to use Session Beans to do this, > then they need to be statefull.


    may be i needn`t session bean

    I have server(j-boss) with has connection thru jdbc to database,
    i need build report client side, for this I need data as result of query to database

    so i see it as a pipe between client and database
    i`ll only want get whole result of query ( only moving forvard while geting data ) and build report using xslt

    i think about, may be i need somethink like httpsession.....
  7. hi, thank for reply,,,,[ Go to top ]

    i think about, may be i need somethink like httpsession.....

    You seems to me mixed several notions together. HTTP session is a session btw. a browser and a web tier though SessionBeans (stateless or stateful) are onse that live on the serverside completely (let's say btw. web and application/business tier). If your data is so huge that it flows out of the memory (server's memory?) then the whole picture looks worse to me. If a server can't manage such amount of data how a pure clien will do it? And how much time it would take just to transfer it to the client?
    I guess the single approach can be, as people already have said, to use paging. To chunk you data on pages and to pass it separately. Hence you can carry out aditional operations over each chunk (like server side XSL preprocessing) and every chunck in case of failure maybe gotten again easy. For such an approach it's not necessarily to use stateful beans if in every request you specify a starting record number and a number of records to featch. It's cheaper (from resources' point of view) and more flexible.
    Using streams to get the data through a pipe resolves just the problem of memory but not the rest but brings another problems into consideration (closing streams, loosing connections and so on).
  8. i didn`t know if i need session or some another

    i know only that resultset.size may be very large
    i need only transfer this dat to client (all data from result set)
    and client may be want straight save it to disk
    so at my opinoin using serialization in this situation
    is not good solution, (even by part)

    i now try examine all variants
  9. You can acheive the same results through straight

    > Servlet code by caching the ResultSet in your
    > HTTPSession on the server side too.
    > Hope this helps. Let me know if you need further information.

    Yes. I very beginner in j2ee and client/server technology so
    I need father infomation. ^)