Memory Leak in WebLogic (or in my brain?)

Discussions

EJB programming & troubleshooting: Memory Leak in WebLogic (or in my brain?)

  1. Memory Leak in WebLogic (or in my brain?) (9 messages)

    I am not terribly experienced with WebLogic, so I apologize in advance if it is my brain's problem, and not WebLogic's.

    Details

    In its simplest form, the problem is reproduced as follows.

    I created the most trivial entity bean imaginable:
    * This entity bean uses bean-managed persistence.
    * Its primary key is a string.
    * The finder method of this bean always returns the primary key, which means that the bean can be "found" by any primary key at all.
    * The bean has one business method (called execute) that takes no arguments and returns a string containing run-time memory diagnostics.

    This is not a practically useful bean, but it is the simplest way to reproduce the problem. Below I will explain how this problem maps to real-life scenarios.
     
    The client Java application looks up the home interface of this entity bean, and then, in a loop, repeatedly performs the following actions:

    * Look up the entity bean by a primary key that is never the same.
    * Call execute.
    * Forget the reference to the bean and do it again.

    Executing this test causes Weblogic to run out of Java heap in a time that is inverse-proportional to the length of the keys. For example, I initially was using keys like this: a1, a2, a3, ... This caused WebLogic to run out of 64 MB of memory in about 10 hours, after performing a couple million of requests. When the length of the key is increased to 4096 characters, WebLogic runs out of 96 MB or memory in about one minute after executing just over 6600 requests.

    Observations

    * Killing the client virtual machine and restarting it before WebLogic runs out of memory cancels the effect of the leak caused by the killed session of client, until enough memory leaks again.
    * Using keys from a small fixed set of strings eliminates the leak.
    * Changing the amount of memory needed to allocate the entity bean object itself does not affect the number of requests that it takes to fill WebLogic's memory, so the bean object itself is probably not leaking.
    * Increasing the primary key size directly increases the speed at which Weblogic runs out of memory.
    * Changing transaction attributes of the bean does not help.
    * Setting up a facade stateless session bean whose single business method looks up the entity bean and calls execute, and calling the session bean from the client instead of calling the entity bean directly does not help.
    * Tweaking cache parameters does not help.

    Why this is a real problem

    Although the entity bean has "strange properties" in that it is always found and does not load itself from any source, it represents a real scenario. It models a very large database, with the client potentially accessing arbitrary parts of it.
     
    A more interesting question is, how feasible it is to have a client that accesses thousands of beans from the same VM session. A very likely scenario is that this client is a Web servlet engine. I have not investigated whether the WebLogic servlet engine has any "under the covers" way of communicating with the EJB container. But if we are to use a third-party servlet engine, such as Apache Tomcat, I can't imagine why the servlets would appear to WebLogic any different than the test client of my application. This means that WebLogic will keep track of all primary keys that were looked up by the servlet engine, not its clients. And, given a big enough database, this will cause WebLogic to eventually run out of memory or at least lose a lot of performance (due to increased frequency of garbage collection, at least under JDK 1.2).

    Source

    If anyone is interested, the entire source can be downloaded form my site:

    http://www.florintsev.com/stuff/leak.zip

    Contact

    If anyone is interested in any more details, my e-mail is gene at florintsev dot com

    Regards!
  2. I have a few questions..

    1) Are your GC settings ok?
    2) Have you considered using a tool to investigate the memory leak and trace down the source?
    3) What are the pool settings for the bean?
    4) It sounds like you are doing strange things with the entity bean returning the same PK each time. Is this really an entity bean? Or is it more of a session bean?
  3. Dear Mr. Roman:

    1. I run the WebLogic server in 96M of Java heap. I am not sure what other settings I should check/tweak. The test client runs in 8M of memory with no problem.

    2. This is a very good idea. However, I am not sure what that will actually give me (probably, because of my ignorance). My server-side code has no "persistent" data collections to produce a memory leak, so it must be in the app server. Where in the WebLogic code the keys (allegedly) are being accumulated, I do not know, and I am not sure how I would use this information if I had it. Your suggestions are welcome.

    3. I tried 10, 100 and 1000 without observing any real effect. I mostly played with 10. Of course, I have thought about simply not having enough memory to accomodate the cache, but that does not seem to be the case.

    4. I probably did not explain this clearly. I am returning not "the same PK each time", but I am returning the PK that was passed INTO findByPrimaryKey (). This mimics the behavior of a real BMP entity (in the case when the PK actually exists), less the actual query. I am trying to model the behavior of a bean that is "looking" into a very large database (where there are many, many different PKs) without actually setting up a connection with any real DB. Do you see any problem with this approach?

    5. I reported this problem to a BEA consultant that is working with me on the same project, and I am not sure so far what he has done about this. I posted this problem in the hopes that somebody will tell me what I am doing wrong. This is just too darn trivial not to work!

    - Gene Florintsev
  4. Another question for you.. have you sent the source to BEA and had them repro the leak and confirm it is a BEA problem?
  5. Hi Gene,

       Sounds like Weblogic is continually creating more and more references to entity beans, and since you are doing this in a loop, it sounds like the VM simply doesn't have the time to run garbage collection. I don't think this is a bug with WL, its just an extreme case, of which the following solutions could help:

    1) Throw on more memory and increase your heap size to something more reflective of a real deployment (at least 196megs).
    2) Set your entity bean deployment descriptors to have a very short timeout period for your beans (to cause quick passivation).
    3) Run your app. in a profiler, to make sure that your not doing something silly in your code (I had a similar problem, and I ran Optimize It and quickly found out that the problem was with my code.

    hope this helps,

    Floyd
  6. Floyd,

    hmmm... I have been working with Java for a while, but I have never heard that it "doesn't have the time to run garbage collection"... In my experience, when it fails to allocate memory it will first run garbage collection, no matter whether it thinks it has the time, or it doesn't. I am not quite clear on your idea here. You must know something that I don't. Is this a new JDK 1.2 behavior?

    Given the above, I do not see how throwing in more memory will help, except it will prolong WebLogic's death proportionally to the amount of memory increase. But this is a simple experiment, which I will perform, because I may be wrong.

    Tweaking the timeout is something I have not done, but I did see the beans become passivated - in the WL console. What I think happens, is that the primary keys are being kept in memory for some reason, but, again, I will try this.

    As far as running my app in the profiler to see if I am doing something strange - the source of my so-called "app" is 6KBytes, all of it, including client code and deployment descriptors. It is absolutely trivial and does exactly what I described, and nothing else. It is not an app, really, just a trivial load test.

    I have a question, though. What is supposed to happen with a bean that has been accessed by a client, after the transaction ends? Can the bean be forgotten and sent back to the pool? Sure, it is cached until the cache overflows, but what happens when the entity bean is evicted from the cache? I am assuming that it should be stripped of its "identity" and sent back to the pool. If not, then that could be the problem with my test, right there.
  7. Well, I am no VM expert, so perhaps there is no such thing as "not having time to run GC", I thought that it might be possible to run out of heap space before GS has time to do anything about it.

    >I have a question, though. What is supposed to happen with
    >a bean that has been accessed by a client, after the
    >transaction ends?
        It will remain in cache until it "times out", after which it will be sent to the pool (I believe).

        Just a thought, are you saving your PK class as a member variable in your entity beans? If you are then make sure you are setting it to NULL (or any other member variables for that matter) in ejbPassivate().

    let me know what happens,

    Floyd
  8. BEA has acknoledged this as a defect. They are not releasing information in the lock table when beans are passivated. Thanks everyone for your help with this.

    I hope they fix this soon, I think it is a major bug when the DB becomes really huge.

    - Gene

  9. Has weblogic resolved this problem?
  10. Memory Leak in WebLogic, NOT my brain.[ Go to top ]

    I had a problem where we were running out of memory with WLS5.1 and it turned out to be a leaky JVM from sun. I can get you the versions if you like (t_milburn at hotmail dot com) but we upgraded and said goodbye to the leak, cya