Andrig Miller discusses the steps to improve the JVM throughout on 64-bit Unix OS's supporting large page memory.

When using a 64-bit OS, in my case Fedora 8 and RHEL 5.1, I wanted to investigate the usage of large page memory support, or HugeTLB as its referred to within the Linux kernel. What I found was very scarce documentation around using this, and that that documentation was incomplete to actually make it work. What I also found, is it makes a huge difference in overall throughput and response times of an application, when using heap sizes above 2GB. So, without further ado, let's dive into how to set this up. These instructions are for Linux, specifically for Fedora 8 and RHEL 5.1, but the results should be generally applicable to any 64-bit OS and 64-bit JVM that supports large page memory (which all the proprietary UNIX's do, and I found an MSDN article describing how to use this on 64-bit Windows). * Step by step instructions omitted, see blog link at the end * Well, in my case, I was able to achieve an over 3x improvement in my EJB 3 application, of which fully 60 to 70% of that was due to using large page memory with a 3.5GB heap. Now, a 3.5GB heap without the large memory pages didn't provide any benefit over smaller heaps without large pages. Besides the throughput improvements, I also noticed that GC frequency was cut down by two-thirds, and GC time was also cut down by a similar percentage (each individual GC event was much shorter in duration). Of course, your mileage will vary, but this one optimization is worth looking at for any high throughput application.

Read Andrig Miller post on performance tuning http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html

This is a less common configuration, but not that difficult IMHO. We (eg BEA) and the other major JVM vendors describe it in our respective reference manuals. I jumped when I read the claim of a 60-70% boost; that must surely include other configuration as well. Simply enabling large pages only avoids the TLB cache miss bottleneck, and I've never seen it yield more than 10-20% of performance. Also, it is not always a good choice. For example; on some Opteron systems you get much fewer TLB page entries when large pages are enabled, which can result in worse performance for some workloads. There are also other caveats, such as not being able to swap out the memory (can be good or bad) and sometimes a longer startup time. That said, we generally recommend large pages for systems that are used almost exclusively for Java - it is a reasonably simple way to get a "free" performance boost. -- Henrik, BEA JRockit team