EJB design: Issue with Weblogic Server 8.1 (

  1. Issue with Weblogic Server 8.1 ( (7 messages)

    Hi All,

    I am facing an issue with Weblogic server(8.1). One of the managed servers stops abruptly. Server run for some days. After 5 or 6 days it stops without giving any log message. Admin(mgt server) and this managed server are running in the same box(AIX Environment). Below is the only log I got. Any idea why it is happening?

    According to the BEA Message it may be due to network problem. Since both the servers are running in same box, i don't think it is network issue.

    Can anyone please help me on this? - thanks in advance

    Log Trace Message:
    <AF[196]: Allocation Failure. need 528 bytes, 67173345 ms since last AF>
    <AF[196]: managing allocation failure, action=1 (0/1040390840) (32611304/33283912)>
      <GC(197): freeing class jsp_servlet._ofr._invoice._shipper._jsp.__exceptioninvoicelist(302453c8)>
      <GC(197): freeing class jsp_servlet._lcom._common._jsp.__page_navigation(3024ecc8)>
    <Dec 21, 2005 10:47:18 PM KST> <Warning> <Management> <BEA-141138> <Managed Server ofrserver is disconnected from the ad
    min server. This may be either due to a managed server getting temporarily partitioned or the managed server process exi
      <GC(197): freeing class jsp_servlet._ofr._filter._jsp.__filterdetails(30252718)>
      <GC(197): freeing class jsp_servlet._lcom._admin._jsp.__devtools(3040f898)>
      <GC(197): freeing class jsp_servlet._lcom._admin._util._jsp.__devtoolslogin(3040f260)>
      <GC(197): freeing class jsp_servlet._lcom._admin._util._jsp.__processdevtoolslogin(30411870)>
      <GC(197): freeing class jsp_servlet._ofr._admin._util._jsp.__xmlcommmanagerimport(30413e58)>
      <GC(197): freeing class jsp_servlet._ofr._admin._util._jsp.__processxmlcommmanagerimport(30414a88)>
      <GC(197): unloaded and freed 8 classes>
      <GC(197): GC cycle started Wed Dec 21 22:47:23 2005
      <GC(197): freed 721499624 bytes, 70% free (754110928/1073674752), in 263157 ms>
      <GC(197): mark: 248258 ms, sweep: 1857 ms, compact: 13042 ms>
      <GC(197): refs: soft 0 (age >= 32), weak 0, final 27363, phantom 0>
      <GC(197): moved 0 objects, 0 bytes, IC reason=14>
      <GC(197): stop threads time: 1356, start threads time: 19>
    <AF[196]: completed in 282883 ms>
    Shanmuga perumal

    Threaded Messages (7)

  2. Issue with Weblogic Server 8.1 ([ Go to top ]

    There's a huge hint in there:

    completed in 282883 ms

    Your GC cycle took several minutes! That's pretty much fatal for a managed server.

    You can look forward to having lots of fun tuning the garbage collection algorithm. First, I would recommend getting on as modern a JDK 1.4.2 as you can. There have been a number of bugfixes (assuming you are on Sun's JDK, and not JRocket).

    Then, first you need to make sure you are passing -server as a command-line argument to java, not -client. Then you need to look at tuning the way Java allocates its heap, as well as which model you are using. This article will be very helpful:


    Here's a command line that works pretty well for one particular application:

    -server -Xms512m -Xmx512m -XX:NewSize=128m -XX:MaxNewSize=128m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=2 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

    Each one of these arguments was chosen after quite a bit of experimentation, so, while this exact command line MIGHT be useful for you, it's more intended to give you an indication of the kinds of decisions you need to be making. Most of these settings were chosen by a perl script which explored the space of settings, and ran a performance test against a server for each particular setting. It took quite awhile to get it right, but doing so had a huge (positive) impact on its sustained performance.
  3. As Eric suggested, it appears it is a GC issue.

    It actually appears you are using IBM's JDK? If you are on IBM 1.3, then you are very limited on what you can do. About the only really useful setting I could find while tuning IBM's JDK: -Xgcpolicy:optavgpause (versus optthroughput)

    *** However, since it sounds like it was a process that ran for a while and then cratered after a few days, no matter what settings you use, you will eventually hit this issue. ***

    Best bet is to use a profiler such as OptimizeIt or JProbe to see what objects are leaking. In the meantime, can you graceful do a rotating bounce of your cluster every night to clear the heap?

    If you can use 1.4 or even Sun's JDK, then you can use some more advanced options as Eric points out. Most importantly, how many CPU's does your server have and how much available ram. Looks like you have a 1gb heap right now. These params will help determine the GC settings. If you have 10 cpu's, then you will want to bump the number of parallel threads up, for sure.

  4. Thanks a lot - Eric and Dustin.

    Sorry for the late reply. I was bit confused about the options to fix this.

    More about My Environment:
    1. We are using Weblogic(8.1 SP4) runs on AIX and it uses IBM JDK(1.4.2 ClassicVM)
    2. The box is having 18 CPUs.

    The server cannot be bounced to clear the heap and not sure about whether IBM will support the below options
    XX:PermSize=128m -XX:MaxPermSize=512m -XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=2 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

    Since i am using IBM JDK, i couldn't introduce more command line options. Earlier I was having below options

    -Xms1024m -Xmx1024m -verbose:gc

    Now, my plan is to decrease the pause time. For this i have introduced the option "-Xgcpolicy:optavgpause". This is will reduce the pause time. Now the JVM parameters will be

    -Xms1024m -Xmx1024m -verbose:gc -Xgcpolicy:optavgpause

    If you look into the log which says "67173345 ms since last AF" - i.e Time interval between the two GCs is around 18.7 hours. Is there any way to increase the number of collections(Full GC) in limited interval, so that it can decrease the pause time in IBM JDK(1.4.2 ClassicVM)?

    Thanks again,
    Shanmuga perumal
  5. No problem, this is "fun" stuff to figure out.. :)

    The real issue, however, is that it sure appears you have a memory leak. Therefore, no matter what you try or do, you will always have the GC issue and your server will eventually become unresponsive.

    If you are clustered, then you could possibly perform a rotating bounce. If not, then your hands are probably tied.

    You CAN tune the paramaters which trigger a garbage collection. In 1.4.x, you have other parameters too. Unfortunately, I have no experience on IBMs 1.4 JDK.

    Check out http://www-128.ibm.com/developerworks/eserver/library/es-JavaVirtualMachinePerformance.html for a start.

    Good luck!
  6. Similar Issue[ Go to top ]

    we are also facing a similar issue.We use WLS 8.1 on Sun Solaris with Sun JDK.
  7. Similar Issue[ Go to top ]

    Can you please provide the JVM Parameters and logs what you are getting?

    You may need add the JVM parameters like "-XX:MaxPermSize=128m".

    Shanmuga perumal
  8. Re: Similar Issue[ Go to top ]

    I have the same proble using Weblogic 8.1 SP3 and jdk142_03. Arguments settings are: MEM_ARGS=" -Xms1024m -Xmx1024m -XX:MaxPermSize=128m -verbose:gc" The error log is the following: [Full GC 1013631K->1012505K(1013632K), 16.8171342 secs] [Full GC 1012505K->1012499K(1013632K), 16.6594067 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    - IOException - java.io.FileNotFoundException: /app/bea81/conf/cmbrick/msite_request.dtd (Too many open files) - :: 5 : Internal Xml-Error - :: 5 : Internal Xml-Error [Full GC 1013631K->1012542K(1013632K), 16.9346198 secs] <Server subsystem "weblogic.health.HealthMonitorService@1cb31f4" failed> [Full GC 1013631K->1012523K(1013632K), 16.7488253 secs] [Full GC[Unloading class sun.reflect.GeneratedConstructorAccessor769] 1012523K->1012496K(1013632K), 16.6873396 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC 1013630K->1012488K(1013632K), 16.7440355 secs] [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9870] 1012488K->1012465K(1013632K), 16.8176581 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC 1013626K->1012493K(1013632K), 16.8698099 secs] [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9875] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9876] 1012493K->1012476K(1013632K), 16.7270695 secs] <[ServletContext(id=12793391,name=bea_wls_internal,context-path=/bea_wls_internal)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9869] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9868] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9867] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9866] 1013631K->1012492K(1013632K), 18.1155455 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC 1013626K->1012476K(1013632K), 16.7598492 secs] [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9882] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9872] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9873] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9874] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9871] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9877] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9879] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9878] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9881] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9880] 1012476K->1012420K(1013632K), 16.7287337 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    <[ServletContext(id=12793391,name=bea_wls_internal,context-path=/bea_wls_internal)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC 1013625K->1012427K(1013632K), 16.7342053 secs] [Full GC 1012427K->1012418K(1013632K), 23.3552224 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    <Server subsystem "weblogic.health.HealthMonitorService@1cb31f4" failed> [Full GC 1013631K->1012464K(1013632K), 16.7308591 secs] [Full GC 1012464K->1012453K(1013632K), 16.6501778 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC 1013631K->1012492K(1013632K), 16.8152297 secs] [Full GC 1012492K->1012480K(1013632K), 16.7022836 secs] [Full GC 1013631K->1012491K(1013632K), 22.6652430 secs] [Full GC 1012491K->1012482K(1013632K), 16.6803936 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    - IOException - java.io.FileNotFoundException: /app/bea81/conf/cmbrick/msite_request.dtd (Too many open files) - :: 5 : Internal Xml-Error - :: 5 : Internal Xml-Error [Full GC 1013631K->1012485K(1013632K), 16.7493764 secs] [Full GC 1012485K->1012474K(1013632K), 17.4933843 secs] - :: 0 : Request executed Successfully <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    - IOException - java.io.FileNotFoundException: /app/bea81/conf/cmbrick/msite_request.dtd (Too many open files) - :: 5 : Internal Xml-Error - :: 5 : Internal Xml-Error [Full GC 1013631K->1012447K(1013632K), 16.8169442 secs] [Full GC 1012447K->1012420K(1013632K), 16.7391379 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    <[ServletContext(id=12793391,name=bea_wls_internal,context-path=/bea_wls_internal)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC 1013631K->1012419K(1013632K), 16.7773449 secs] [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9888] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9891] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9885] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9887] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9883] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9886] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9889] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9890] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9884] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9892] 1012419K->1012390K(1013632K), 16.6897260 secs] <Server subsystem "weblogic.health.HealthMonitorService@1cb31f4" failed> <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC 1013631K->1012681K(1013632K), 17.5122557 secs] [Full GC 1012681K->1012660K(1013632K), 17.2029832 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC 1013631K->1012634K(1013632K), 16.7684683 secs] [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9897] 1012634K->1012617K(1013632K), 16.7161761 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC 1013626K->1012658K(1013632K), 16.9211296 secs] [Full GC 1012658K->1012643K(1013632K), 16.7027264 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError
    <[ServletContext(id=12793391,name=bea_wls_internal,context-path=/bea_wls_internal)] Root cause of ServletException. java.lang.OutOfMemoryError
    [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9893] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9895] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9894] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9896] 1013631K->1012639K(1013632K), 16.7697661 secs] [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9898] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9899] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9902] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9901] [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9900] 1012639K->1012617K(1013632K), 16.7574503 secs] <[ServletContext(id=11047935,name=msitegtw,context-path=/msitegtw)] Root cause of ServletException. java.lang.OutOfMemoryError