Discussions

Performance and scalability: Threading in JBoss - anomoly

  1. Threading in JBoss - anomoly (3 messages)

    Hello everyone,

    I am currently looking at the effect of the size of the thread pool for the clients of an EJB application.

    I am referring to the setting maxThreads in server.xml under the following directory:
    %JBOSS%\server\default\deploy\jbossweb-tomcat50.sar\
    (see below for excerpt)

    I get the following figures for throughput (TP) and response time(RT):

    1 thread: TP=144, RT=11.421
    10 threads: TP=142, RT=11.615
    100 threads TP=66, RT=26.874

    The number of users in the system is 10 so extra threads are redundant but why do spare threads have such a detrimental effect on RT (more than doubled) and TP (less than half)?

    I am running a simple stateless session bean-only application that simulates a load on CPU and Memory so there should be no real resource contention (Memory consumption is well below memory availability) and I am using JMeter to control the load and return the RT and TP statistics.

    Any ideas of what might cause this would be very much appreciated.
    Thanks a lot,
    Dave

    PS Changing jboss-service.xml in %JBOSS%\server\default\conf has no real effect on TP or RT

    Excerpt from server.xml file:

       <Service name="jboss.web"
          className="org.jboss.web.tomcat.tc5.StandardService">

          <!-- A HTTP/1.1 Connector on port 8080 -->
          <Connector port="8080" address="${jboss.bind.address}"
             maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
             enableLookups="false" redirectPort="8443" acceptCount="100"
             connectionTimeout="20000" disableUploadTimeout="true"/> ...
  2. why do spare threads have such a detrimental effect on RT (more than doubled) and TP (less than half)?

    This is a typical issue in the field of performance tuning of application servers. You must start with a very low number of threads (for example, two threads per CPU in the system) and perform load testing. If you don't reach the 90% of activity in the CPUs, you must increase in 50% the number of threads and repeat testing until you get to 90% of activity in the CPUs.

    It is very typical to get less transactions per secons when you have a lot of threads, because the system is all the time doing context switching instead of real work.


    Jose Ramon Huerga
    http://www.terra.es/personal/jrhuerga
  3. Hi again,

    Thanks for the reply. I have run some more tests. for these I have set the memory consumption of the stateless session beans to zero and I increased the number of users to 100 (all running the same single method transaction). There are also 100 bean instances in the pool. What I would expect to find is that as we increase from 1 thread up to somewhere near 100, we should get increasing throughput and decreasing response times, but quite the opposite is happening (see below). Regarding context switching, surely JBoss should only switch between threads when there is something active on the thread and should not switch to threads that are not in use (and so unused threads should not cause extra runtime CPU usage (or a very minimal amount to check the extra threads for usage???)

    Thanks again for any insight provided,

    Regards,
    David

    RESULTS
    ====================
    Threads RT TP
    1 87092 132
    5 84231 142
    10 95694 111
    25 90989 129
    50 101541 116
    100 132768 99
    =====================
  4. Hi David,

    I think that the issue is pretty clear now: you must set the number of threads to 5, because that is the value that allow you to get the best response time, and - much more important - the higher number of transactions per second.

    The number of threads must be aligned with the processing power of the machine you are using. If, for example, you've got a 4-way SMP machine, the number of threads should be at least 8 (depending of the desing of the application) but if you are using a single-CPU machine, a high number of threads tents to be counterproductive.

    Regarding context switching, you must understand that the JVM can switch between threads in any moment, not just when they are waiting for I/O. Of course, if you set 100 threads, the JVM (and in the end the CPU) must do a lot of context switching and this is a very bad thing... ;D


    Jose Ramon Huerga
    http://www.terra.es/personal/jrhuerga