Discussions

News: Threadless AJAX polling with Jetty 6.0 Continuations

  1. Jetty 6.0 servlet server has a new alpha release with support for Continuations, which will allow scalable AJAX applications to be built with threadless waiting for asynchronous events.

    Web applications built with AJAX can place a significant burden on a java server. In order to receive asynchronous event, the AJAX client must poll the server. To avoid latency or busy loops, the server often holds onto a request waiting for an event or a timeout, before sending the response. Due to the synchronous nature of the servlet API, this normally requires a thread per user, which is not a solution that will scale to large numbers of users.

    Jetty 6 introduces Continuations, a mechanism that allows a servlet request to effectively be suspended and it's thread returned to the thread pool. The request is restarted when an asynchronous event has resumed the Continuation or once timeout has occurred.

    Java AJAX servers now have a mechanism to send asynchronous messages to the client with minimal latency and minimal thread usage.

    The mechanism is described in MBlog and a chat room demo is included in the jetty-6.0.0alpha3 release.

    If Continuations work as well as hoped, they may be part of a Servlet 3.0 JSR.

    Threaded Messages (16)

  2. Plain HTML client[ Go to top ]

    As I understand from the blog comments, the TCP connections are alive over the conversation period. What about POH [Plain Old HTML :)) ] clients ? Can they do continuations (as RIFE can AFAIK)?

    Regards,
    Horia Muntean
  3. Plain HTML client[ Go to top ]

    I'll be writing a short article this week to talk a bit about 'continuations' and 'continuations'. Imho, this Jetty 6 feature aren't 'continuations'. They don't capture the local variable state and the program counter, they don't restore it later and once they're used, they can't be re-used. After reading Jetty's sources it looked to me that active the thread was merely put into a wait state to free up the pool of request threads, later it is simply resumed. This only manifests one of the benefits of continuations: pause the execution and resume it later. However, no state is stored outside of the thread that allows continuations to offer all their compelling features for web development. While Jetty 6 offers a performance optimization for certain Ajax approaches, it doesn't seem to offer anything else.
    Continuations seem to be 'hot' and it feels to me like the name is being used for features that only contain certain parts (I find it hard to see people call a car without wheels or an engine, a car). I'll detail this more in a few days.
  4. Better Name?[ Go to top ]

    Gert,

    I agree that naming is difficult and that Continuations in other languages may be able to offer more features than are available in java servlets.

    But as far as java and servlets go, the Jetty Continuations
    come pretty close to the Wikipedea definition.

    But Servlets are stateless request handlers and the state they use is contained in the passed request object, the session, the associated attributes and any state extablished by the container (security, class loader, JNDI context etc. etc.).

    Thus when jetty reestablishes the request and it's context and replays it through the stateless filters/servlets, it is resuming the state of the request.

    So while some cars have ABS, air bags, trip computers, aircon, etc., it doesn't mean that my 1991 fiat panda is any less a car without them.
  5. Better Name?[ Go to top ]

    But as far as java and servlets go, the Jetty Continuationscome pretty close to the Wikipedea definition. But Servlets are stateless request handlers and the state they use is contained in the passed request object, the session, the associated attributes and any state extablished by the container (security, class loader, JNDI context etc. etc.). Thus when jetty reestablishes the request and it's context and replays it through the stateless filters/servlets, it is resuming the state of the request.
    Greg,

    the wikipedia definition says "a continuation is a representation of an execution point". Jetty's continuation implementation contains no information about an executing point at all (scroll down to WaitingContinuation). It merely waits for a resource to be available and resumes execution of a paused thread when the resource is set. This doesn't represent an execution point, otherwise any wait/notify Java code would 'support continuations'. More even, since the request actually stalls completely, you don't even resume a continuation after interaction from a client, you just resume the same request in exactly the same context.

    As far as Java and servlets go, there has been a lot of research to achieve proper continuations support natively in Java. One of the most interesting ones was probably Brakes, it inspired both RIFE and Cocoon to offer proper native continuations to Java web applications with true local variable stack and program counter capturing, pausing and resuming.

    The stateless nature of HTTP has got nothing to do with continuations and the fact that you manually have to do work to create the state that is passed around (even though it is reproducible) clearly shows that there's no 'support' for continuations. You do get one of the benefits (state that's coupled with a logic flow step), though REST is based on exactly the same principle. It is however nowhere close to the granularity of native language continuations where each local variable is automatically captured and restored.

    I'm sure that this new feature that you've added is handy and benefits quite some people that are doing Ajax, but I can't see any resemblance with continuations nor with any languages or frameworks that natively support them.
  6. Better Name?[ Go to top ]

    Hi Greg,

    sorry, I didn't see the RetryContinuation inner class of SelectChannelConnector until this morning when I had time to play with the Ajax Chat demo.

    However, it seems even further from continuations than your WaitingContinuation implementation. You never resume the execution from inside the ChatFilter code, is that correct? It seems that when you detect an active 'Continuation' for a request, you reuse that request instance with its fields but re-execute the handler completely.

    I suppose that you will now argue that this is the same as continuations but on a HTTP level, since you store the execution point on the level of a Request. However, HTTP is not a language but a protocol and each GET URL request in itself can then be seen as a 'continuation' too since it also represents an 'execution point'.

    I suggest that you rename your feature to 'connection freezing', or 'request parking', or something like that. You would do everyone a favour: your users will not be disappointed to see that Jetty's continuations are totally not what they expect, your efforts aren't wasted because people that come looking for it were actually looking for something else, and the IT community isn't more confused due to an alternative shifted meaning of 'continuations'.

    I do think that the feature itself is interesting and that the servlet spec could benefit from it.
  7. Better Name?[ Go to top ]

    MMMmmmmmm...... I agree the current implementations are less than pure formal Continuations. But they are just implementations. The intent and purpose of the interface is for Continuations and that is how the programmer will use it.

    Originally I named the mechanism after the implementation - it was called RequestRetry. But when I explained it to others they found it difficult to understand. Eventually when they did understand the concept they said - "Oh you mean a Continuation". Hence the name change.

    Besides the Continuation that I'm using is an interface and can have multiple implementations. One is the simple waiting one, another is the RetryException (an ExitContinuation) and the reply request. Eventually we may be able to do a fancy byte code manipulation based implementation that truely does restore the context rather than replay the request.

    So I think that Continuation is the correct name for the interface - No? I'll work on the javadoc to avoid any confusion about the available implementations.
  8. Better Name?[ Go to top ]

    From what I understand, your purpose is not to provide continuations, but to provide a mechanisms to respond to requests that are waiting for events without blocking resources (threads). So I think that you should reflect that intent in the name of the interface. Real continuations might in the future be one implementation that provides this, the current implementations provide this too, but aren't continuations. I kinda liked my suggested of 'ParkedRequest', but it's not my call ;-)
    Maybe you should ask your mailing list for suggestions?
  9. Plain HTML client[ Go to top ]

    I'll be writing a short article this week to talk a bit about 'continuations' and 'continuations'. Imho, this Jetty 6 feature aren't 'continuations'. They don't capture the local variable state and the program counter, they don't restore it later and once they're used, they can't be re-used. After reading Jetty's sources it looked to me that active the thread was merely put into a wait state to free up the pool of request threads, later it is simply resumed. This only manifests one of the benefits of continuations: pause the execution and resume it later. However, no state is stored outside of the thread that allows continuations to offer all their compelling features for web development. While Jetty 6 offers a performance optimization for certain Ajax approaches, it doesn't seem to offer anything else.Continuations seem to be 'hot' and it feels to me like the name is being used for features that only contain certain parts (I find it hard to see people call a car without wheels or an engine, a car). I'll detail this more in a few days.

    I'll be interested in your article to know if this stuff (and actually continuations in general) could bring some benefits. I'm very skeptical, as keeping all the saved requests/responses could be quite expensive. Also, the feature might not work that well if the application is complex (lots of request dispatcher usage). Wouldn't it be just more efficient / easier to just issue separate HTTP requests (the overhead of request parsing is very slim compared to the IO involved, and it still would use the same keptalive connection), given this doesn't have the benefits of actual continuations ?

    As for naming the feature, I'd say something along the lines of "IO poller" or something would be better.
  10. Wouldnt the browser timeout while waiting for the container to resume processing?

    Also, I would think that the more efficient use of the thread pool is just one of the concerns regarding scalability. If we are saying that the browser will retain its connection to the server and wait for asynchronous events then the server will need to maintain, potentially thousands of long-lived connections. How will this effect scalability.

    One last question...

    How long are we going to try to turn webapps into regular client/server applications? We are trying to force the browser to move from 'simple markup renderer' into 'Generic application client'.... square block; round hole?
  11. Browser timeouts[ Go to top ]

    Wesly,

    The timeout on the server has to be less than the time most browser will wait for a response. I find 20 or 30 seconds works fine and is a lot less load than an AJAX client that polls continously and a lot more responsive that a client that polls ever few seconds.

    ... and hey, I don't control the way developers write client side applications. Jetty 6 is just following the trend here and trying to support the load presented to it as best it can.

    Besides, why is a program written in javascript that manipulates DOM any less a valid client side solution than one in C# that manipulates pixels? Also it is not just browsers that have dom rendering engines in them - geckos get everywhere!
  12. If the problem is holding connections on the server, shouldn't the state be completely stripped from the server the asynchronicity be managed by the AJAX client?

    AJAX already starts magnifying processing burdens on the web client side, so one should be able to do AJAX async: put request to server to queue up some result, and then the web client polls the server to see if the server had produced the desired information yet. After all, the state is on the client, why duplicate it on the server? Obviously it's easier to program, but when you run into performance problems, tough. Gotta ditch the server-side connection state here.

    The web client can then handle the "waiting..." information, and not block them if necessary.

    Suspending presumably stateful connections - sounds ugly, seems like it will result in unreliable results.

    But what the heck do I know.
  13. If the problem is holding connections on the server, shouldn't the state be completely stripped from the server the asynchronicity be managed by the AJAX client?AJAX already starts magnifying processing burdens on the web client side, so one should be able to do AJAX async: put request to server to queue up some result, and then the web client polls the server to see if the server had produced the desired information yet. After all, the state is on the client, why duplicate it on the server? Obviously it's easier to program, but when you run into performance problems, tough. Gotta ditch the server-side connection state here. The web client can then handle the "waiting..." information, and not block them if necessary.Suspending presumably stateful connections - sounds ugly, seems like it will result in unreliable results. But what the heck do I know.

    The change in Jetty basically means that if your Ajax client is requesting some data (such as pulling some data from a messaging system), then Jetty can - if your servlet decides - suspend the current Servlet thread and resume it later when there is data available. This basically stops the Servlet engine using up a thread for every Ajax client which is connected, waiting for some data.

    Now if every time your Ajax client wants some data, its available, then you don't need this change. However if you have 1000s of Ajax clients consuming from a JMS topic or queue, where data might not always be available (the servlet might need to wait a while to get data) - you probably don't want 1000s of servlet threads blocked just waiting for inbound messages.

    So this change basically means in those kinds of circumstances, you can tell Jetty that there is not an immediate response available. The Servlet is suspended (like a mini-continuation) - then the servlet can be resumed later on when a message actually arrives for the client.

    To put that another way - if you are using lots of asynchronous messaging with Ajax, then this change allows you to support 1000s of clients, to use the Servlet API on the server side and not have 1000s of blocked threads.

    James
    LogicBlaze
  14. shouldn't the AJAX client do the polling?[ Go to top ]

    If the problem is holding if you are using lots of asynchronous messaging with Ajax, then this change allows you to support 1000s of clients, to use the Servlet API on the server side and not have 1000s of blocked threads.JamesLogicBlaze
    James,

    This sounds quite incredible; if this is true I wouldn't mind rewriting a lot of our GUI in some other AJAX framework . Have you guys hooked up your benchmark with Jetty to see if this theory is actually true ?
  15. shouldn't the AJAX client do the polling?[ Go to top ]

    If the problem is holding if you are using lots of asynchronous messaging with Ajax, then this change allows you to support 1000s of clients, to use the Servlet API on the server side and not have 1000s of blocked threads.JamesLogicBlaze
    James,This sounds quite incredible; if this is true I wouldn't mind rewriting a lot of our GUI in some other AJAX framework . Have you guys hooked up your benchmark with Jetty to see if this theory is actually true ?

    Incredible is maybe a little over the top :). It purely means you can recycle threads - which coupled with NIO means you can handle 1000s of clients easily.

    Though it depends on your OS & JVM. e.g. this change only helps avoid using unnecessary threads - you still have to support 1000s of sockets, which is often an OS limitation. e.g. on linux you can usually recompile your kernel to support a massive number of concurrent sockets.

    We've not added Jetty to our benchmark yet; its on the list of things to do.

    James
    LogicBlaze
  16. http://simpleweb.sourceforge.net/design/concurrency.php
  17. As an ex team mate of the aimexpress.aol.com team (pure dhtml client for aim) which used similar technology, we noticed that this is a proven scalable architecture for certain clients with special needs and can be also generalized for all the other types of clients. About three years back, We built servlet extensions to our proprietary servlet container in aimexpress team to support request parking/passivation and activation so that the valueble threads are properly used. We had an api similar to the above continuation api where in the servlet is resumed when either any i/o event happens on the socket or some backend thread resumes the continuation explicitly. The goal was to have each process support atleast 10k connections. The native servlet container used C++ and jni to acheive this polling on linux platform. Unfortunately, we didnt have javanio at that time. The product is currently in production supporting from stats about 8 - 10k active connections of aim users per servlet container process. The timeout of browser aspect is dealt with having the connection be max alive only for 10 minutes and then the reconnection establishes session at a different process which gets the state from the previous process thru migration.

    Although programming this kind of solution is tricky, it is well worth the effort for certain class of apps as the jetty team pointed out. I feel happy that somebody else built similar infrastructure to support similar needs confirming that the concept does makes sense.

    The only question i have is - whether enterprises allow a pure java server to be an http server instead of apache fronting - due to lots of dos attack scenarios.