I am currently in the architecture analysis and design phase for a project which may very likely involve pure socket, thread and JDBC programming.
In this project, thousands of client devices may periodically send measurement data that has to be stored into a database system. The communication with the client devices would be over TCP/IP sockets.
These sockets would probably be long-lived because the server may have to collect information from the client devices in particular scenarios.
This data has to be decompressed/transformed into a format for storing into the database.
As bulk amounts of data are expected, this solution has to be highly performant and scalable.
I have a few questions here-
1. For each socket opened between the client and server, would it be advisable to dedicate a thread for reading the socket? Is there any limit to the threads and sockets that can be kept open on the server? If yes, it would probably be better to reuse the threads across various open sockets. How long can these sockets be kept alive?
2. Each request's data has to be processed (probably transformed) and inserted into database tables. Is it better to acquire a connection from a connection pool for each thread and insert the data into the database table or would a better architecture be to dedicate a thread for batch insertion of database tables over a single connection and make the receiver threads store data in a queue wherefrom this data would be picked up by this thread and inserted into the database in bulk. Again, how many threads for database interaction are recommended.
3. If I need to test this out, what would be the right approach/tools to use?
Thanks and Regards,