Jean-Francois Arcand, Tomcat5 developer, talks about Tomcat5 development, internal architecture, and embedding it into other processes. Tomcat's migration to JMX is also discussed, as well as its impact on performance.

Mr. Arcand brings up Tomcat's valve architecture and how it affects Tomcat, and how developers might leverage it in their own projects.

View Jean-Francois Arcand on Tomcat5

Congradulations to Tomcat team.

I am particularly happy that Tomcat can now be embedded. I do not know whether it was possible in previous versions.

I tried to integrate Jetty and OpenEJB in embedded mode, but enountered lots of problems, mainly due to JNDI and not to mention lack of documentation. I will try to see whether it would be easier with the combination of Tomcat and OpenEJB in "both" embedded mode. Has anyone experience with this?

The best thing with Tomcat5 is that it starts in two seconds on my machine!

That’s all I ask..

Why not use something like Orion, which is a full J2EE container - and still manages to start in two seconds or so?

I worked in JBoss and other commercial application servers. It takes more time to start and load the application.

Tomcat is really faster in startup even if you have many web-apps in one box.

I am using Eclipse with tomcat plugin for devlopment. It saves atleat one or two minutes in each development cycle.

Everything is open source here.

Why not use something like Orion, which is a full J2EE container - and still manages to start in two seconds or so?

Well, as much as I liked Orion in former days, it isn't a really viable choice anymore. It hasn't moved forward for (literally) years. How compelling is a J2EE 1.3 server these days, in particular one that had significant compatibility issues until quite recently?

Tomcat has grown into a viable production platform over the past few years, and it is still compelling as development platform too. If all you need is a decent, up-to-date J2EE web container, Tomcat is hard to beat these days. In particular with a decent application framework on top of it, caring for all the rest :-)

BTW, don't forget Resin, which has proven to be a very nice and up-to-date J2EE web application server for years, and still is a compelling choice for both development and production. It starts up very fast, and is generally hard to beat in terms of performance.

Juergen

Why not use something like Orion, which is a full J2EE container - and still manages to start in two seconds or so?

Well, as much as I liked Orion in former days, it isn't a really viable choice anymore. It hasn't moved forward for (literally) years. How compelling is a J2EE 1.3 server these days, in particular one that had significant compatibility issues until quite recently?

Which significant compatibility issues?

Besides, Orion *has* moved forward... just not as fast as I'd have liked.

As far as 1.3 being compelling... well, I'd like to see more recent versions of the specs, too, but then again, I've had no problems with J2EE 1.3 either. Maybe it's just me.

Which significant compatibility issues? Besides, Orion *has* moved forward... just not as fast as I'd have liked. As far as 1.3 being compelling... well, I'd like to see more recent versions of the specs, too, but then again, I've had no problems with J2EE 1.3 either. Maybe it's just me.

Regarding compatibility issues: have a look at the following issue - a severe problem in the JSP container, making the affected Orion versions effectively unusable for custom tags that export nested scripting variables.

http://bugs.orionserver.com/issue/view.jsp?id=1081

Look at how long it took them to fix this basic JSP compliance issue: reported on July 5th 2003, fixed on March 28th 2004 (!) This is not acceptable for such severe issues.

I haven't had problems with J2EE 1.3 either. Well, JSP 2.0 of course has the built-in EL scripting and tag files as major new features. But in general, I wouldn't mind working with a J2EE 1.3 server. My main issue with Orion is the lack of responsiveness regarding J2EE 1.3 compliance issues.

That said, I still appreciate Orion and do not intend to discredit it in general. But unfortunately, the 2001 fork of their codebase that became Oracle OC4J is significantly more compliant and more up-to-date in the meantime. A pity, but that's how things seem to have turned out to me.

Juergen

Hmm, something's really funky there. I've used Orion for a long time, and I've exported scripting variables successfully; something's going on somewhere for your case, and I have no idea where it would be.

I know at least two servlet 2.3 compliant engines/web servers with startup time less than one second: bj Http Web Server and BEJY Java server.
The authors claim these servers beat Tomcat (and several other web servers) in performance, but these servers provide not so many extra features.
I would not recommend these tiny servers in production (cause I've no such experience), but I've found them useful in development.

Regards,
Theodore Kupolov

... which makes them suitable for the people who want J2EE 1.3, but if you actually want up to date software, you choose something else ;)

I know at least two servlet 2.3 compliant engines/web servers with startup time less than one second: bj Http Web Server and BEJY Java server.The authors claim these servers beat Tomcat (and several other web servers) in performance, but these servers provide not so many extra features.I would not recommend these tiny servers in production (cause I've no such experience), but I've found them useful in development.Regards,Theodore Kupolov

I seriously doubt either of those two webservers can beat Tomcat. If you look at the results in the spreadsheet I posted, tomcat is already hitting the network IO bottleneck. For static files larger than 1K, tomcat is even with Apache 1.3 and 2.0. Both of them saturate the network IO.

Unfortunately, I don't have gigabit ethernet, so I can't say conclusively that tomcat or apache don't have performance bottlenecks. But here's the thing, how many ISP give you a full 100mbit of bandwidth? Keeping in mind a 100mbit link to their router does not constitute 100mbit of bandwidth. Even when I had servers co-located at Level3's facility, we hit a network bottleneck around 10mbit. Back then, the San Diego facility had an OC128 or something close to that.

The only people capable of handing sustained 100mbit of bandwidth are carriers or gigantic sites like yahoo, or google. Even then, they have thousands of servers co-located across several sites for redundancy. The biggest factor in performance isn't the webserver, but people still worry about it. oh well, that's life.

peter

Why not use something like Orion, which is a full J2EE container - and still manages to start in two seconds or so?

To my knowledgment, there are a couple of issues with Orion. Firstly, Orion does not expose a ready-to-use API. Secondly, I am not sure it has been designed to be embedded. I asked the same question in Orion discussion forum, but never got an answer.

What does "expose a ready-to-use API" mean?

I haven't seen anything on embedding it, true.

As far as J2EE 1.3, well... I suppose it depends on what you need. Plus, 1.4 is on its way. :)

For those curious about how Tomcat compares against HTTPD 1.3 and 2.0, here is a recent benchmark I ran. The raw charts are also available for those who want to see.

http://jakarta.apache.org/tomcat/articles/benchmark_summary.pdf
http://people.apache.org/~woolfel/tc_results.html

peter

May be I missed somethink but it looks like the results for large files reach 10,000 -12,000 Kbyte/s or 80-90Mbit/sec - pretty close to theoretical throughput of 100Mb/s network he wos using

I need to run the tests on a system with gigabit ethernet or something with dual or quad ethernet. tomcat is pretty efficient now. I did compare it to jetty for static files and tomcat was faster. though honestly, both are plenty fast. More likely than not, the bandwidth to the internet will be clogged long before either Jetty or Tomcat die.

peter

The HUGE sites have always been the exception and not the rule.

The modern Java servers are more than adequate for a vast majority of applications barring some specific application requirements.

For those that are running their applications off of a single box, the difference is essentially irrelevant. When the application starts slowing down to be noticable, you're not going to be happy with 10-15% improvement in performance, so you may as well get another box and deal with that problem rather than swapping in the other server, because the head room if gives you just isn't enough to matter, save for the most short term of solutions. And if you're that desperate, you're going to be needing a new box soon anyway.

Use the devil you know and master it.

The Geronimo team has been working on embedding Tomcat into the Geronimo application server:

http://svn.apache.org/viewcvs.cgi/geronimo/trunk/modules/tomcat/

Jetty support is there too:

http://svn.apache.org/viewcvs.cgi/geronimo/trunk/modules/jetty/

It would be nice if you could load JSP's from something other than the filesystem...

The Geronimo team has been working on embedding Tomcat into the Geronimo application server:http://svn.apache.org/viewcvs.cgi/geronimo/trunk/modules/tomcat/Jetty support is there too:http://svn.apache.org/viewcvs.cgi/geronimo/trunk/modules/jetty/

Correct me if I am wrong, but in Geronimo, the communication between various containers goes through some sort of proxy.
However, I would very much like to directly call each server.
Have I misunderstood something here?

I have not used "embedded" feature of Tomcat but the impression, from the interview, is that this just means there is an interface which allows you to control Tomcat from your application.

"Embedded" in the case of Jetty means much more - embedded Jetty is just one JAR and you can use it as one Jar, it is very small, in size, so you can really embed.

So, unless I misunderstood something, for the embedding (e.g. for testing purposes, using as a Stub) Jetty, still, seems much more convenient.

And, for me personally, it was alerting to hear that Jean-Francois has no idea about the embedding features in Jetty. That may not mean much but is odd. People who have tendency towards the re-invention of the wheel, do not usually invent good wheels. If you begin work on embedding feature and there is a product that has that figured-out, why would not you go and see how they did it? Maybe you won't do it the same way (most probably not) but at least - you should look into it, should not you?

Also:

I was really surprised by one comment from the interview. The question was if people should use Apache in front of Tomcat or run it as-is. Jean-Francois made it sound like it is just a matter of performance and almost suggested that Tomcat without Apache is just as good.

NOT TRUE.

Yes, Tomcat has gone long way in performance but that just proves the point that: putting Apache in front of Tomcat is not a matter of performance (performance may be the same) but _security_.

To make Tomcat listen to port 80 you will need to launch it under root user, which means your application code will also run under root. That's _evil_. User applications should never run with root privileges.

When you put Apache in front of Tomcat (or any other servlet engine/J2EE container, for that matter) your application can run under an non-privileged user. That's much more secure.

Thats partly true, but you could just use iptables to redirect 80 -> 8080 which does sound like a hack, but works quite well in practice.

A more compelling reason to use apache (or squid as a reverse proxy) is that it can buffer slow connections, because of the single threaded nature of requests in tomcat, slow connections can leave threads running for longer than normal. This could cause the server to run out of available threads for subsequent requests. By putting apache or squid in front of tomcat, tomcat can perform at full speed and the other daemon will take care of the buffer.

I've written more about using squid here.

Peter: I *will* get that benchmarking done and send the results over :$

I have not used "embedded" feature of Tomcat but the impression, from the interview, is that this just means there is an interface which allows you to control Tomcat from your application."Embedded" in the case of Jetty means much more - embedded Jetty is just one JAR and you can use it as one Jar, it is very small, in size, so you can really embed.

I second that. Tomcat indeed has a nice API for embedding, yet you need a lot of jars and there's a lot of classloader trickery going on that seems pretty brittle to me. In our case, the presence of xercesImpl.jar on the application class path would cause some commons-modeler to break. I don't want that component anyway ...

javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found
        at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:99)
        at org.apache.commons.modeler.util.DomUtil.readXml(DomUtil.java:284)
...
        at org.apache.catalina.startup.Embedded.start(Embedded.java:846)

Using Jetty was no problem at all.

To make Tomcat listen to port 80 you will need to launch it under root user

Well, privileged ports for root only are so outdated. A modern OS as Solaris 10 allows you to grant that privilege to "normal" users:

# usermod -K defaultpriv=basic,net_privaddr bob

Now bob's Tomcat will be able to bind to port 80.

Matthias

Among other inaccuracies in your post, this one sticks out:

>To make Tomcat listen to port 80 you will need to launch it >under root user, which means your application code will also >run under root. That's _evil_. User applications should never >run with root privileges.

Not true. Tomcat comes with a service utility which allows you to run it on port 80 not as the root user. It uses the Jakarta Commons Daemon component for this, as documented at http://jakarta.apache.org/tomcat/tomcat-5.5-doc/setup.html.

Also, Jean-Francois mentioned several options for embedding Tomcat. There's the interface you mentioned, but also JMX and others. It's flexible and solid, including Commons-Modeler.

Yoav,

This is exactly from the documentation:
[blockquote]Tomcat can be run as a daemon using the jsvc tool from the commons-daemon project. Source tarballs for jsvc are included with the Tomcat binaries, and need to be compiled. Building jsvc requires a C ANSI compiler (such as GCC), GNU Autoconf, and a JDK.[/blockquote]

I do not see much different between bridging using C-code jsvc and Apache through an AJP connector. None of them look to me like "pure-Tomcat". I would rather use Apache, though, because I trust it more, with all due respect.

I much rather accept the "analog boy"'s reply that one can do port-mapping.

Generally, I do not understand why Tomcat team wastes time on reimplementing, in Java, functionality that Apache has been providing for years perfectly. AJP connectors are quite solid and that seems sufficient to me.

Regarding the embedding: apparently, we have different views on what convenient "embedding" means: simple and lighweight. Tomcat embedding is neither, IMHO. I think, Mathias made a very true comment on that.

cheers

Yoav, This is exactly from the documentation:[blockquote]Tomcat can be run as a daemon using the jsvc tool from the commons-daemon project. Source tarballs for jsvc are included with the Tomcat binaries, and need to be compiled. Building jsvc requires a C ANSI compiler (such as GCC), GNU Autoconf, and a JDK.[/blockquote]I do not see much different between bridging using C-code jsvc and Apache through an AJP connector. None of them look to me like "pure-Tomcat". I would rather use Apache, though, because I trust it more, with all due respect.I much rather accept the "analog boy"'s reply that one can do port-mapping.Generally, I do not understand why Tomcat team wastes time on reimplementing, in Java, functionality that Apache has been providing for years perfectly. AJP connectors are quite solid and that seems sufficient to me.Regarding the embedding: apparently, we have different views on what convenient "embedding" means: simple and lighweight. Tomcat embedding is neither, IMHO. I think, Mathias made a very true comment on that.cheers

Apache httpd and tomcat are both rock solid servers, but I don't understand the Apache front approach to be blunt. On heavy duty sites, we setup Cisco to forward port 80 requests to 8080. That is far better in my bias opinion because it generally performs better. If you're servers are handling 10million+ page views a day, do you really want to spend all that extra time forwarding from Apache to tomcat using AJP? Doing it at the load balancer with routing filters is definitely more efficient in terms of performance. my bias .2 cents.

peter

Peter. Sometimes I love you!

For numerous reasons, often times, you may want to set up several application servers (e.g. Tomcat) on one physical machine.

I know how to do it using Apache+AJP connector and it will provide me load-balancing, failover, sticky sessions, too. It's quite easy, actually.

I would appreciate, if somebody could educate me, if and how that can be achieved using port mapping?

thank you

For numerous reasons, often times, you may want to set up several application servers (e.g. Tomcat) on one physical machine.I know how to do it using Apache+AJP connector and it will provide me load-balancing, failover, sticky sessions, too. It's quite easy, actually.I would appreciate, if somebody could educate me, if and how that can be achieved using port mapping? thank you

if you're using something like localDirector or some other hardware load balancer, it provides several ways to provide sticky sessions and port forwarding. Generally speaking, Cisco's local director can redirect using url pattern, port or hostname. For example, you can setup http://cluster.mydomain.com/ to load balance across all tomcat instances behind the load balancer. of course, if you can't afford a hardware load balancer this approach won't work for you. I won't bother copy/pasting product info from Cisco. Just go to Cisco's page for localDirector and see for yourself.

peter

For numerous reasons, often times, you may want to set up several application servers (e.g. Tomcat) on one physical machine.I know how to do it using Apache+AJP connector and it will provide me load-balancing, failover, sticky sessions, too. It's quite easy, actually.I would appreciate, if somebody could educate me, if and how that can be achieved using port mapping? thank you

if you're using something like localDirector or some other hardware load balancer, it provides several ways to provide sticky sessions and port forwarding. Generally speaking, Cisco's local director can redirect using url pattern, port or hostname. For example, you can setup http://cluster.mydomain.com/ to load balance across all tomcat instances behind the load balancer. of course, if you can't afford a hardware load balancer this approach won't work for you. I won't bother copy/pasting product info from Cisco. Just go to Cisco's page for localDirector and see for yourself.peter

JBoss Inc., through Remy Maucherat and Mladen Turk, is funding a lot of JNDI work on Tomcat to beef up performance so that an Apache HTTPD front does not need to be there.

Bill

For numerous reasons, often times, you may want to set up several application servers (e.g. Tomcat) on one physical machine.I know how to do it using Apache+AJP connector and it will provide me load-balancing, failover, sticky sessions, too. It's quite easy, actually.I would appreciate, if somebody could educate me, if and how that can be achieved using port mapping? thank you

if you're using something like localDirector or some other hardware load balancer, it provides several ways to provide sticky sessions and port forwarding. Generally speaking, Cisco's local director can redirect using url pattern, port or hostname. For example, you can setup http://cluster.mydomain.com/ to load balance across all tomcat instances behind the load balancer. of course, if you can't afford a hardware load balancer this approach won't work for you. I won't bother copy/pasting product info from Cisco. Just go to Cisco's page for localDirector and see for yourself.peter

JBoss Inc., through Remy Maucherat and Mladen Turk, is funding a lot of JNDI work on Tomcat to beef up performance so that an Apache HTTPD front does not need to be there. Bill

Sorry, that should say "is funding a lot of JNI" work. In other words, Remy/Mladen are adding a lot of native hooks to get rid of the advantage Apache HTTPD has.

Sorry, that should say "is funding a lot of JNI" work. In other words, Remy/Mladen are adding a lot of native hooks to get rid of the advantage Apache HTTPD has.

In my bias opinion, most people aren't going to benefit from the work :) I've been keeping an eye on the work Mladen's been doing. High end applications that have to support a large number of concurrent requests or higher throughput will benefit. The typical webapp isn't really going to get much considering all the improvements in TC5.

It's still great to have those features using apache APR + JNI. Hopefully when that work is done, I can run a new set of benchmarks with high concurrency to see how it really compares.

peter

you couldn't do that with port mapping, you could only forward from one port to another. You could use squid to proxy the connections which could forward to different ports depending on the url requested, but you'd be better off using apache in your case.

in a high-volume environment you wouldn't use multiple servers anyway, you'd have dedicated machines. if you aren't dealing with huge volumes then don't worry about your apache config.

in a high-volume environment you wouldn't use multiple servers anyway, you'd have dedicated machines.

I don't understand this comment. Please explain?

(I assume by high volume, you're talking at least 10 million page views a day.)

Peace,

Cameron Purdy
Tangosol, Inc.
Coherence: Cluster your POJOs!

soz, what i meant it that you are less likely to be running a variety of applications on machines that are having to cope with a lot of use, you would probably have an application server installation per physical machine.

if the server isn't under large load, then just do the simplest thing to make it work because you don't have to obsess yourself with through-put. in the case of running multiple application servers on a single host, apache and mod_jk is probably an easy way forward for most admins.

analog boy,

Imagine such scenario: You have dual (or quad) processor servers, a cluster of them. Each machine has 4 Gigs of RAM. However, they are 32bit CPU servers. As you know, 32bit JVM has the limit on RAM it can use ~1.6GB.

Let's assume, that is fine because your average in-process memory consumption (including cache) is under 1GB. Now, What you can do is - you can install 4 Tomcats (or whatever) and utilize your RAM better.

Now, why would one not just buy 1 GB RAM (ignoring RAM needed for OS) per machine, instead? Why would one buy more RAM than it can use?

The thing is, whether we like it or not, the stability of an OS (e.g. Linux) is much higher than that of an application server with our app deployed into it. So, there is much more probability that Tomcat will go down than the machine will go down. So, deploying several Tomcats on one machine your are cost-effectively increasing your stability by statistically increasing fail-over support.

More App Servers in the cluster - more fail-over, hence more secure your application as a whole, right? Now, what is cheaper - to buy 20 machines, each with 2 GB RAM or to buy 10 of them with 4GB RAM and deploy two instances of Tomcat in each of them?

If you have 20 Tomcats in cluster you are more secure than when you have 10 of them because even if 10 of them will go down (site is down in the latter case) you will have other 10 to stay up.

Makes sense?

Also, I really do not know to start a flame here and Tomcat guys throwing some figures and numbers of their benchmarks but here's what was the the empirical experience we had: our JBoss (with embedded Tomcat5) would crush if the number of concurrent connections to it would exceed some number near 200. I do not know whether it is a Tomcat problem, JBoss problem (why would it be? JBoss is not processing requests) or the problem of integration, but that's a fact from our own, sole experience.

In a case like this, having several (N number) Tomcats behind one Apache really helps performance because you will be able to serve N times more users.

All in all, IMHO even in high-load environments you may need to have several app servers on one physical machine for the reasons of stability and performance.

analog boy,Imagine such scenario: You have dual (or quad) processor servers, a cluster of them. Each machine has 4 Gigs of RAM. However, they are 32bit CPU servers. As you know, 32bit JVM has the limit on RAM it can use ~1.6GB.Let's assume, that is fine because your average in-process memory consumption (including cache) is under 1GB. Now, What you can do is - you can install 4 Tomcats (or whatever) and utilize your RAM better.Now, why would one not just buy 1 GB RAM (ignoring RAM needed for OS) per machine, instead? Why would one buy more RAM than it can use?The thing is, whether we like it or not, the stability of an OS (e.g. Linux) is much higher than that of an application server with our app deployed into it. So, there is much more probability that Tomcat will go down than the machine will go down. So, deploying several Tomcats on one machine your are cost-effectively increasing your stability by statistically increasing fail-over support.More App Servers in the cluster - more fail-over, hence more secure your application as a whole, right? Now, what is cheaper - to buy 20 machines, each with 2 GB RAM or to buy 10 of them with 4GB RAM and deploy two instances of Tomcat in each of them?If you have 20 Tomcats in cluster you are more secure than when you have 10 of them because even if 10 of them will go down (site is down in the latter case) you will have other 10 to stay up.Makes sense?Also, I really do not know to start a flame here and Tomcat guys throwing some figures and numbers of their benchmarks but here's what was the the empirical experience we had: our JBoss (with embedded Tomcat5) would crush if the number of concurrent connections to it would exceed some number near 200. I do not know whether it is a Tomcat problem, JBoss problem (why would it be? JBoss is not processing requests) or the problem of integration, but that's a fact from our own, sole experience.In a case like this, having several (N number) Tomcats behind one Apache really helps performance because you will be able to serve N times more users.All in all, IMHO even in high-load environments you may need to have several app servers on one physical machine for the reasons of stability and performance.

I don't know which version of jboss + tomcat you were using, but I've looked at Tomcat thoroughly using benchmarks and optimizeIt. 200 concurrent connections for static files are trivial for tomcat5.5.x code base.

keep in mind the benchmark I performed measured response time and resource utilization under varying load. Notice that I ran over 200 benchmarks to establish how tomcat performs by varying file size and concurrent threads. This wasn't just 1 benchmark for each file size. All the data is there for you to see. The test plans I wrote for the benchmarks are also available http://cvs.apache.org/~woolfel/testplans.zip.

This series of tests provide a foundation of general performance for static files. I seriously doubt Tomcat is at fault, since it is able to saturate the network IO quite easily while keeping resource usage under 20%. If the EJB is slow, the problem isn't tomcat. I would like to point out Tomcat performed the same in both client and server mode. Coyote connector and the mapper are all very efficient now. If you're having performance issues, then I would suggest reporting it to Jboss or tomcat-user.

When ever I come across performance issues, I will at minimum deploy my application on another container and quantify the difference. If I can reliably reproduce the behavior, I write down the exact sequence of events and then report it. In 95% of the cases, the performance issues were in my own application and not tomcat. I'm curious what kind of behavior you were seeing?

peter lin

Anyone have a good Tomcat 5.5.7 book to suggest? I went to the library, errr, bookstores last night and I only found one currently on the shelves. I tried using the online Tomcat docs but ... .

Anyone have a good Tomcat 5.5.7 book to suggest? I went to the library, errr, bookstores last night and I only found one currently on the shelves. I tried using the online Tomcat docs but ... .

I assume you've already checked the resource page? http://jakarta.apache.org/tomcat/resources.html. I don't know of a good book off hand, since I prefer to dig in to tomcat's code. This question is asked quite a bit on tomcat-user, so you should be able to find some recommendations in the archive? I know it's a bit of a pain and there really should be more docs, but that's always going to be true.

peter

Anyone have a good Tomcat 5.5.7 book to suggest? I went to the library, errr, bookstores last night and I only found one currently on the shelves. I tried using the online Tomcat docs but ... .

I assume you've already checked the resource page? http://jakarta.apache.org/tomcat/resources.html. I don't know of a good book off hand, since I prefer to dig in to tomcat's code. This question is asked quite a bit on tomcat-user, so you should be able to find some recommendations in the archive? I know it's a bit of a pain and there really should be more docs, but that's always going to be true.peter

Sorry Peter, my post was more a follow up to your post and not so directly to you (but thanks for your response).
I was looking for someone's recommendation on a book (something that was helpful to them). The resources on the tomcat are way out of date (I can use Amazon to find book titles). Yes, I can and do find answers on the website/forums - but having it in one place in a not-disjointed fashion is sometimes helpful. I'll dig if/when I have to. After a day of [click link] what?, [click link] what?, [click link] what? I decided to spend a few bucks.

Peter,

thank you for the information.

The behavior we were seeing was that when number of concurrent requests to JBoss 3.2.5 with integrated Tomcat 5, would reach some number near 200 JBoss would crash irrecoverably.

As I said, I do not know if it was a problem of Tomcat, per se, JBoss, of their integration. Yes, there's always a chance that the actual application might have caused problems but it is hard, for me, to understand how could it have had such problems that it would crash the container. IMHO, container should be resistant to that.

I am not trying to bash Tomcat, we use Tomcat (under JBoss) in our production servers and that's, probably, the best proof that I very much favor Tomcat.

What I am trying to say is that, in my limited experience, it may be quite beneficial to be able to run several app servers on one physical server and I was very glad that Apache + AJP was able to support that so easily and smoothly.

Basically, I am trying to defend Apache as frontend, not - give hard time to Tomcat :)

You are right, one solution would have been to compare Tomcat with other servers but our aim was not benchmarking and we did not have much desire to get off JBoss+Tomcat, so what we ended up doing was - increased the number of Tomcat installations + fine-tuned our application performance so that requests would stay for less time and less concurrent requests would accumulate.

Peter,thank you for the information.The behavior we were seeing was that when number of concurrent requests to JBoss 3.2.5 with integrated Tomcat 5, would reach some number near 200 JBoss would crash irrecoverably. As I said, I do not know if it was a problem of Tomcat, per se, JBoss, of their integration. Yes, there's always a chance that the actual application might have caused problems but it is hard, for me, to understand how could it have had such problems that it would crash the container. IMHO, container should be resistant to that.I am not trying to bash Tomcat, we use Tomcat (under JBoss) in our production servers and that's, probably, the best proof that I very much favor Tomcat.What I am trying to say is that, in my limited experience, it may be quite beneficial to be able to run several app servers on one physical server and I was very glad that Apache + AJP was able to support that so easily and smoothly.Basically, I am trying to defend Apache as frontend, not - give hard time to Tomcat :)You are right, one solution would have been to compare Tomcat with other servers but our aim was not benchmarking and we did not have much desire to get off JBoss+Tomcat, so what we ended up doing was - increased the number of Tomcat installations + fine-tuned our application performance so that requests would stay for less time and less concurrent requests would accumulate.

I totally understand. It's not always possible to go the extra step of running the same app in another container. My interest is finding out the limitations of a given application, so whenever an application breaks I get very curious. I don't know about others, but I generally try to break servers and applications to see how it breaks. Seeing how something breaks is very revealing and provides clues into potential limitations and/or flaws. I feel much more comfortable when I know the devil.

You may not have permission to disclose more information, but were the response times over 30 seconds or was the app using lots of XML? In my experience, the types of things that can crash tomcat are OutOfMemoryException or JDBC drivers.

peter

My interest is finding out the limitations of a given application, so whenever an application breaks I get very curious. I don't know about others, but I generally try to break servers and applications to see how it breaks. Seeing how something breaks is very revealing and provides clues into potential limitations and/or flaws.

+1
So very true.

In my experience, the types of things that can crash tomcat are OutOfMemoryException or JDBC drivers. peter

Yes, DB queries were not too optimized. I think (hope LOL) that it was not as bad as 30 seconds but there was space for optimization. That was the first thing we did right off.

I would not expect JDBC driver could crash the whole server, tho :-( If you are saying that's something known, then, probably, that was the problem. I just thought it was much more probable that number of concurrent connections would crash the server than JDBC. However, as the user load increases the load on the JDBC part increases, too so you are, probably, right.

My interest is finding out the limitations of a given application, so whenever an application breaks I get very curious. I don't know about others, but I generally try to break servers and applications to see how it breaks. Seeing how something breaks is very revealing and provides clues into potential limitations and/or flaws.

+1So very true.

In my experience, the types of things that can crash tomcat are OutOfMemoryException or JDBC drivers. peter

Yes, DB queries were not too optimized. I think (hope LOL) that it was not as bad as 30 seconds but there was space for optimization. That was the first thing we did right off.I would not expect JDBC driver could crash the whole server, tho :-( If you are saying that's something known, then, probably, that was the problem. I just thought it was much more probable that number of concurrent connections would crash the server than JDBC. However, as the user load increases the load on the JDBC part increases, too so you are, probably, right.

In my experience, the only time I've seen a JVM do a core dump is using native db drivers like Oracle or DB2. I've seen Tomcat lock-up under sudden spike in concurrent requests with Tomcat4, but with tomcat5.5.x it should just deny connections. These are relatively simple scenarios without database calls. Other common scenarios that can cause slowness include jsp tags or using large XML documents.

back in 2002 Remy and I ran over 1K benchmarks combined on tomcat 4.0 and 4.1. those results are also posted on tomcat's resource page. By any chance are you using Sql Server?

peter lin

We are using Oracle 9i

From past experience with Oracle8i, sounds like the semiphore settings need to be modified. If you did the default installation of Oracle8 or 9, the semiphore settings assume low concurrency. 200 concurrent queries isn't high, but it's definitely not low either. The semiphore settings could easily be the cause of the slow down. just a wild guess, since I'm assuming you've already looked into that.

peter

Peter,

thanks.

According to our DBA, the settings were fine-tuned but there were some real bad SQL queries and, probably, JDBC was misbehaving.

Although I wish that's never happen to me, I'd have to admit I've done that. Why is probably why I get very annoyed when I'm told by the product manager to "skip stress testing, because it's not important" :)

peter

i think a lot here depends on your installation, i'm working for an on ultrasparc IIIs so the ram limitation doesn't mean much. also depends on whether you see the cost of the project being the up-front cost of the hardware or the maintenance cost (it can be cheaper to have more low spec'd boxes than a few high spec'd ones).

ultimately it depends on your installation, for development i don't use apache, on live we do for php and cgi support, but on some servers we use squid because we only want to improve the tomcat performance without making the install any more complex.

i think it was a bad question to ask, there is no single answer to the problem because it is *absolutely* specific to your platform and your intended users. it is important to see beyond the FUD tho, tomcat can suffice as a webserver for in many instances, but personally, i don't think you should use it unless you know that all your clients will have fast access (DSL externally or on an intranet).

I really wish I had the resources to test Tomcat with 3-4 dozen dial up connections to measure the real impact of slow connections. It's just me, but I prefer to see real data showing a slow down. Hopefully the work on native sockets will help Tomcat in that area.

peter

Yoav, This is exactly from the documentation:[blockquote]Tomcat can be run as a daemon using the jsvc tool from the commons-daemon project. Source tarballs for jsvc are included with the Tomcat binaries, and need to be compiled. Building jsvc requires a C ANSI compiler (such as GCC), GNU Autoconf, and a JDK.[/blockquote]

I do not see much different between bridging using C-code jsvc and Apache through an AJP connector. None of them look to me like "pure-Tomcat". I would rather use Apache, though, because I trust it more, with all due respect.

I much rather accept the "analog boy"'s reply that one can do port-mapping.

Thanks for the doc quote, I'm fairly familiar with it ;) Whether you see much "different" or not is up to you. There are enough people using the feature that it's certainly not a waste of time from our perspective. You don't like it, don't use it. But don't assume a lot of people share your point of view ;)

Generally, I do not understand why Tomcat team wastes time on reimplementing, in Java, functionality that Apache has been providing for years perfectly. AJP connectors are quite solid and that seems sufficient to me.

There's a significant and increasing trends towards usage of Tomcat standalone in heavily trafficked sites. Again, as someone who monitors the Tomcat user list closely, I know we're far from wasting time. These features highly useful and highly in demand. But if AJP works for you, great.

Regarding the embedding: apparently, we have different views on what convenient "embedding" means: simple and lighweight. Tomcat embedding is neither, IMHO. I think, Mathias made a very true comment on that.cheers

Different views indeed ;)

As a reminder, if you have a better way to embed Tomcat, or for that matter any other opinions, suggestions, or ideas, we would love to hear them at tomcat-dev at jakarta dot apache dot org.