Discussions

News: Esper 1.0: Event Stream Processing and Correlation in Java

  1. Esper 1.0, an event stream processing (ESP) and event correlation engine (CEP) written in Java, has been released. Esper performs grouping, aggregation, sorting, filtering, merging and joining of event streams using a tailored SQL-like query language. Applications can define time-based, interval-based, length-based and sorted windows and perform inner-joins and outer joins of an unlimited number of windows and event streams. Support for time windows and other statistical computation is provided, along with queries. Esper also performs complex event pattern matching, allowing logical and temporal event correlation. In systems relying on classical SQL database, events are not stored and are later polled in a repository. With Esper, as the flow of events come matching registered event conditions, Esper triggers customized registered actions (POJO). Esper is open-source software available under the LPGL license. Do you think ESP/CEP technology is useful to your projects? Are you involved with any other technologies targeting low latency decisions on high volume streams of data? How does Esper compare?

    Threaded Messages (10)

  2. Thank you[ Go to top ]

    That's original.
  3. Quite interesting. It is even more interesting to see an OSS project cover something that does not even have more then one or two commercial offerings.
  4. Not thread safe?[ Go to top ]

    Since an engine instance is not thread safe (aside from the special case outlined in the FAQ), I wonder how much utility this has for the wider audience. For example, I have a multi-threaded application servicing requests with a thread-per-request model. It would be nice to be able to post events from these request threads. Maybe I'm not thinking about the it the right way, but this seems like a major limitation. Question... Would it be safe to have multiple threads that synchronize when posting events or is there ThreadLocal data in the engine which would get corrupted?
  5. Re: Not thread safe?[ Go to top ]

    Thanks for raising this good question about thread-safety. The FAQ text is indeed unspecific and we'll make sure to update it. I hope this answer makes it clearer. Currently, as stated in the FAQ, Esper supports multiple independent engines per Java VM. Each engine instance itself is NOT multithread-safe. In other words, for a given engine instance we do not currently guarantee the engine instance behaves the same in a multithreaded environment as in a single-threaded environment. Applications using Esper can synchronize around the sendEvent() method. No ThreadLocal data representing engine state remains after the sendEvent() method completes. Thus multiple threads that synchronize posting events to the same engine instance is supported, with the following limitations. - statement creation through the EPAdministrator interface, and statement start/stop must be avoided or synchronized with the posting of events - same for iterating (pulling) of statement results via iterator() We plan to improve this in upcoming versions. regards - Tom, of the Esper team bernhardttom@yahoo.com
  6. This sounds pretty interesting and I can see an event processing query language amd framework like this being useful in a variety of use cases beyond financial or high volume. Imagine using it to consume thousands of RSS feeds and scan for patterns and do useful things with that info, or perhaps scanning incoming web requests in memory and detecting denial of service attacks or simply upsurges in requests/minute and being able to respond to that in smart ways. See also Alexandre Vasseur interviewed about Esper.
  7. Esper - J2EE integration[ Go to top ]

    Hi Tom, is it possible to integrate Esper in a J2EE compliant application (inside the EJB-container)?
  8. Re: Esper - J2EE integration[ Go to top ]

    Very good question Roger. It's one of the key value add provided by Esper: it's very easy to integrate into a J2EE or webapp container. As of today we are not leveraging any J2EE thingy but we are planning to make use of J2EE Timers and possibly CommonJ work managers (such as those in WebLogic 9), as well as simplifying MDB integration. If you have specific requirements on this chapter or wish some samples please let us know (here or better join the user list). Alex
  9. Yes we want to make Esper as easy as possible to integrate into J2EE containers. Your feedback is really appreciated. Version 1.0 is able to run in a J2EE container with limitations as below, but we have not tested this well enough just yet. Esper is light-weight in terms of threads: each engine instances starts 1 timer thread. However the engine can also be externally timed and the timer thread disabled. Thus Esper engine threading should not interfere with J2EE container threading. Please also see the threading discussion above. There is a limitation on the number of threads per engine instance and the need for external synchronization. This is relevant since the J2EE container will be supplying multiple threads. The sendEvent method will need to be synchronized on by your application. Esper engine instances are not clusterable. That is, engine state cannot be replicated within a J2EE application server cluster if your server supports this configuration. If you are running inside an EJB container, using stateless session EJBs, pooled EJB instances would need to share the same Esper engine instance. The mechanism for sharing an engine instance between EJBs, I think, would have to be via static member variable, Singleton or possibly JNDI. For servlets, the servlet context would seem to suit. On the roadmap for future versions are threading improvements and also a native JMS adapter. cheers - Tom
  10. Esper - J2EE integration[ Go to top ]

    Hi Tom, hi Alex,
    If you have specific requirements on this chapter or wish >some samples please let us know.
    Yes, if you have some examples of Esper running in an EJB container I would be very interested. Our use case is the following: We have an J2EE-based self-service terminal managing system, which gets a lot of events from connected terminals (500 per second; e.g. ´paper low´, ´terminal out of order´) and we are correlating these events in the server to build up higher-level events. So for us it would be nice to use Esper in conjunction with our framework which is based mostly on stateless session beans and JMS messaging.
    Esper is light-weight in terms of threads: each engine >instances starts 1 timer thread. However the engine can >also be externally timed and the timer thread disabled. >Thus Esper >engine threading should not interfere with J2EE >container threading.
    Sounds perfect. I think it is very important that Esper is fully J2EE compliant and not specific to any AppServer, so that it can be used in many scenarios and big applications.
    Esper engine instances are not clusterable. That is, engine >state cannot be replicated within a J2EE application server >cluster if your server supports this configuration.
    No problem in a stateless application using JMS persistence.
    If you are running inside an EJB container, using stateless >session EJBs, pooled EJB instances would need to share the >same Esper engine instance. The mechanism for sharing an >engine instance between EJBs, I think, would have to be via >static member variable, Singleton or possibly JNDI. For >servlets, the servlet context would seem to suit.
    An example of SLSBs using the Esper engine instance would be nice. Best Regards, Roger
  11. Re: Esper - J2EE integration[ Go to top ]

    Yep we shall create an example. Sorry for the very late reply. An enhancement request has been created in our issue tracking tool at http://jira.codehaus.org/browse/ESPER-64