Performance and scalability: Odd Performance Problem
We have a Servlet/JSP application deployed on iPlanet Web Server 6.0 SP5 (Now known as Sun Java System Web Server, which is basically Sun One Application Server 7 sans the EJB container, but I digress).
We have iWS deployed on a quad processor box (Sun UltraSPARC running Solaris 8 using lwp threads), and we're having an odd problem where iWS will only scale to maybe 50% of the machine before it peaks off. At this point, new connections to the web address connect straigt away and requests for static content (gifs, etc) server very quickly, but any servlet/JSP requests just hang. It's almost like there aren't enough available connections to the servlet engine.
Is anyone familiar with iWS and could point me to a good resource on tuning iWS to scale onto multiple CPU's. I've been doing a lot of looking through Sun's somewhat lacking documentation, and can't find anything. We've tried everything from adding multiple processes to tweaking almost any relevent parameters, but nothing works... all we can do to make it scale higher is have 4 instances of iPlanet (one for each CPU) and then load balance requests onto these instances.
Thanks in advance,
- Sounds like a threading issue? by Mario C on August 23 2004 09:39 EDT
- Odd Performance Problem by Cameron Purdy on August 25 2004 13:57 EDT
- Odd Performance Problem by Jose Ramon Huerga Ayuso on August 29 2004 16:33 EDT
- Odd Performance Problem by Kirk Pepperdine on September 29 2004 03:58 EDT
Sounds like the Servlet Container is out of threads. How long do your transactions take to complete? Do you have any monitoring/profiling tools to identify long running methods?
Take a look at JView 2004 @ http://www.devstream.com, a J2EE Performance Monitor and Profiler. You will be able to find your culprit.
Use native threads and allocate maybe 50 up front. See if that will saturate it.
Coherence: Shared Memories for J2EE Clusters
By Native Threads, you mean KernelThreads?
Am running this on Solaris.. and the documentation makes much mention of how native threads and kernel threads are a Win32 only thing.
What am I missing on Solaris as far as tuning the threads that the JVM is using? I tried creating a thread pool and assigning the NSServletPlugin to that pool, but that just makes it worse. :(
Native threads, as in "not green threads."
The point is, if you have lots of threads allocated for requests and your CPUs are staying really low, then you have a problem elsewhere. So I suggested lots of threads to see if that were the case.
It could be a synchronization problem (everyone sync'ing on the same object) or it could be a high-latency I/O operation (e.g. messaging or database system) causing the threads to fail to soak the CPUs.
Coherence: Shared Memories for J2EE Clusters
... we're having an odd problem where iWS will only scale to maybe 50% of the machine before it peaks off. At this point ... any servlet/JSP requests just hang. It's almost like there aren't enough available connections to the servlet engine.... We've tried everything from adding multiple processes to tweaking almost any relevent parameters, but nothing works... all we can do to make it scale higher is have 4 instances of iPlanet (one for each CPU) and then load balance requests onto these instances.You should increase the number of threads of the web container. Before trying to increase this value, you should take a look to the thread dumps (kill -3) before trying to increase the number of threads. If, for example, you perform several thread dumps, and you get that the threads are waiting for the database to response, maybe you have database deadlock problems (SELECT FOR UPDATE to the same record).
If the trhead dump doesn't show a database problem, then maybe you can see problems with portions of the source code with the 'synchornized' keyword. You may only increase the number of threads of the web container if all the threads are really working, and not working for other processes.
Hope this helps.
Jose Ramon Huerga
all we can do to make it scale higher is have 4 instances of iPlanet (one for each CPU) and then load balance requests onto these instances.Thanks in advance,DylanI find is interesting that you've actually understood the problem enough to suggest the most likely cause. Then you state this solution (it being the more complex one) which works but for all the wrong reasons and all because of a focus on CPU.
In most applications, the only reason that it may not go as fast as is needed is because of contention on some resource, these are CPU, I/O (network and disk) or locks (programatic). You explination perfectly eliminated CPU and disk I/O. I could guess that network I/O is also not an issue based on the static content delivery comment. That leaves lock contention of which not having enough threads avalible would result in a request being blocked on a lock.
I know that Cameron has already given you his summary of going through this process of elimination, I just thought it might be worth while to provide the process as this is not the first time that I've seen a preocupation on CPU time divert attention from the real problem. Too bad that Sun documentation doesn't point this out more clearly. If it did, we'd miss out on this fun interaction ;)