In the first place, it seems that 64-bit would be the best solution whenever you are able to use it. More address space available must be worth it. But you should also be aware of the problems introduced.
The downside of the 64-bit architecture is that the same data structures consume more memory. A lot more. Our measurements show that depending on the JVM version and the operating system version along with hardware architecture you end up using 30-50% more heap than on 32-bit. Larger heap can also introduce longer GC pauses affecting application latency – running a full GC on a 4.5GB heap is definitely going to take longer than on a 3GB one. So it will not be correct to jump on the 64-bit bandwagon just because 64 is bigger than 32.
Read further on why you can actually use only 1.5GB heap on 32-bit Windows and what are the workarounds on large heaps on 32-bit platforms from the blog post available here.