Firstly, do transactions have this kind of flexibility? And do I have to embrace full J2EE to use the transaction APIs, or can I wrap normal calls from a Servlet or a JSP in transaction blocks?
I'm not sure exactly what you are asking here... If you mean, can you perform transactions that span more than one database, the answer is yes, but you will need to a distributed transaction coordinator to do this. Read up on JTA and play around with JBOSS's implementation to see how it works.
When you ask if you need to embrace J2EE to use the transaction API's, the answer is, yes, sort of... If you are using JTA and JSP or Servlets, then you are using J2EE. I think you may have meant to ask if you must embrace EJB. If that is the case, the answer to that is an emphatic no. You do not need EJB's to perform distributed transactions. Check out the Spring framework for a pretty cool way to do declarative global transactions using POJO's and your choice of data access libraries (you still need JTA though).
Secondly, what are the real world tradeoffs that are made when choosing between JSPs and Servlets? I know I can go and do some reading here, so feel free to ignore this question!
You do not have to choose on or the other. In fact, all web applications built on Struts and Spring use both. Typically, Servlets are used as "Controllers" or "Dispatchers". They receive an HTTP request and invoke code, but they don't really generate HTML. JSP's are generally used as views, they don't make application control decisions... they just take data and render it into HTML. Creating Servlets that generate HTML is a pain. I don't recommend it.
Thirdly, I hear a lot about messaging. What are the benefits of sending messages between servers rather than using RMI? The messaging stuff in a pure Java solution doesn't seem to bring any benefits.
By messaging I'm assuming you mean Web-services a la SOAP over HTTP. Technically, messaging and RMI are fundamentally the same. They are both ways to handle inter-process communication over a network. However, when people say "messaging" they usually mean asynchronous communication. RMI, ususally means synchronous communication. The synchronous vs asynchronous debate is much more interesting than messaging vs RMI.
My two cents is, avoid a lot of synchronous network communcation. Avoid lots of fine grained EJB's that make lots of remote method calls. When you communicate between process over the networks, the communication should be infrequent and large-grained. I definitely favor asynchronous message-style interractions. Note, that this can be accomplished with both web-serivices and RMI.
Finally, any clues on load balancing? I guess the main servlet could allocate new incoming requests to another Servlet, but do we have to roll our own load balancing (maintaining a cookie->servlet mapping in the main servlet), or can we get it from free from an application server or library? Am I talking complete rubbish?!!!
Do not attempt to roll your own load-balancer, it is a pretty hard problem to solve and plenty of them already exist. I don't know this for sure, but I think JBOSS supports clustering. You should deffinitely look for free/cheap load balancers before attempting to build one.
Anyway, hope that helps.