Discussions

General J2EE: Sockets, threads and JDBC programming

  1. Sockets, threads and JDBC programming (2 messages)

    Hello,

    I am currently in the architecture analysis and design phase for a project which may very likely involve pure socket, thread and JDBC programming.

    In this project, thousands of client devices may periodically send measurement data that has to be stored into a database system. The communication with the client devices would be over TCP/IP sockets.

    These sockets would probably be long-lived because the server may have to collect information from the client devices in particular scenarios.

    This data has to be decompressed/transformed into a format for storing into the database.

    As bulk amounts of data are expected, this solution has to be highly performant and scalable.

    I have a few questions here-
    1. For each socket opened between the client and server, would it be advisable to dedicate a thread for reading the socket? Is there any limit to the threads and sockets that can be kept open on the server? If yes, it would probably be better to reuse the threads across various open sockets. How long can these sockets be kept alive?

    2. Each request's data has to be processed (probably transformed) and inserted into database tables. Is it better to acquire a connection from a connection pool for each thread and insert the data into the database table or would a better architecture be to dedicate a thread for batch insertion of database tables over a single connection and make the receiver threads store data in a queue wherefrom this data would be picked up by this thread and inserted into the database in bulk. Again, how many threads for database interaction are recommended.

    3. If I need to test this out, what would be the right approach/tools to use?


    Thanks and Regards,
    Archanaa

    Threaded Messages (2)

  2. Sockets, threads and JDBC programming[ Go to top ]

    1. You should spawn a new thread for each client connection.

    Just my 2 cents.
  3. Here is my solution.[ Go to top ]

    Very interesting problem.

    From the description it seems that client does not need any response from server.
    I hope that I am correct in my assumption.

    In my opinion you can have your design based on following pointers.

    You will need three components.

    1. Socket reader – Reads data from socket.
    2. Data Bucket – Pipe between reader and writer.
    3. Data Formatter and Writer – Formats the data and dumps it to database.

    Socket Reader: System will deploy pool of reader threads which will keep reading data from socket sent by devices.

    Data Bucket: Once the data is read completely, the reader will push that data to Data Bucket and will return back to free pool.

    Data Formatter & Writer: It is a pool of threads which keep pinging on Data bucket. For every data, a dedicate thread will carry out formatting and dumping of data to DB.

    Advantages:

    The system can be scaled simply by creating larger thread pool. The reader will not be blocked even if the DB is slower. Hence the system would respond very fast to incoming data.

    Limitation on threads:
    It depends on your database configuration and hardware profile.

    Disadvantages:
    None ;)
    {Just kidding … please comment and let me know.}

    cheers,
    shrini