Discussions

News: Yet another rule engine

  1. Yet another rule engine (13 messages)

    Hammurapi Event Bus is a lightweight embedded rule engine implemented in Java. It is an evolution of Hammurapi Rules. Key characteristics:

    • Caller-thread and multi-threaded inference.
    • Pure Java API and annotations - no config files, no reflection (with compiled bindings), no runtime code generation, no specialized rule language.
    • Runs on Java SE 6 and Android 2.0+
    • Small footprint (500 kb runtime with compiled bindings)

    Resources:

    Threaded Messages (13)

  2. Inefficient design or NIH[ Go to top ]

    I took a look at how predicates are shared and it ignores 30 years of prior art.

    http://doc.hammurapi.com/dokuwiki/doku.php?id=products:event_bus:start#predicate_chaining

    There are dozens of papers on query plan optimization and pattern matching optimization. On the same page, it describes inference loops. The problem is the rules are invalid and really should advice the user they are making a mistake. Inference loops is actually a subset of challenges related to refraction. It's quite sad so many people that write rule engines ignore 30 years of prior art and reinvent a square wheel.

    It's sad to see so much great research and prior art in this field ignored because someone is too lazy to put the time in to understand the domain.

  3. I've been waiting for you to reply :-)[ Go to top ]

    Hi again Peter,

    I knew you wouldn't miss my post :-) I'm amazed how quick you are to cast judgment! Thank you, though, for looking at the docs!

    I remember our discussion about Hammurapi Rules a couple of years ago. I'd like to remind you that after that discussion you had to admit that some of your initial statements were wrong. However, you also brought up some good points that time!

    Event Bus, as well as Hammurapi Rules originally, was built to dispatch language elements to inspectors in Hammurapi Code review system. And it successfully does its job for millions of language elements and hundreds of inspectors. So it is an event dispatching engine, which also happens to support forward chaining, because handlers can post events and the bus routes them to other handlers. It was needed it to chain inspectors. One of design goals was to enable Java developers to write inspectors (rules) without extensive training. This is why there is no specialized language - hanlders (rules) are Java methods and conditions are expressed through annotations.

    Event Bus is for Java developers who need rules functionality in their applications and who are not necessarily lazy, they just don't have time to study 30 years of rules art because they have  deliverables to ship.

    Event Bus doesn't follow Forgy's and other papers letter by letter and it never meant to (although some of them were taken into account). It is a small and simple rule engine which works. It even works on Android! I know this is a sever-side site so the Android point is a weak one ;-)

    I usually prefer simple things which meet my requirements and move to more complex stuff only when the simple thing doesn't meet my requirements anymore. I do it not because I'm lazy but because I value my time.

     

     

     

     

     

  4. Thank you for the idea![ Go to top ]

    Peter,

    This is in addition to my previous post. I'd like to thank you for giving me a good idea! As you know in the event bus inference filters are pluggable - i.e. it is possible to use simple filters (e.g inference loop filter) for simple cases and more complex filters for more complex cases where there are more refraction challenges. Complexity usually has a performance price tag, so it's not wise to use the most complex filter in all situations.

    Your note about query plans and pattern matching optimization gave me an idea to make the dispatch network builder also pluggable for the same reason - for simple cases (like I described - hundreds of handlers, millions of events, relatively short inference chains) the out-of-the box builder should be enough. For cases as you described in our discussion a couple of years ago (thousands of handlers, long infrence chains) a different, more advanced, dispatch network builder might be a better fit. It's like a query plan in relational databases - it is important to build an optimized plan for large data sets but for small data sets it may take more time to build an optimized plan than to execute a query with a simple plan.

    The beauty of pluggable builders and infrence filters is that the client interface will stay intact!

    So, thanks again, I appreciate your feedback!

     

     

     

     

  5. Thank you for the idea![ Go to top ]

    Peter,

    This is in addition to my previous post. I'd like to thank you for giving me a good idea! As you know in the event bus inference filters are pluggable - i.e. it is possible to use simple filters (e.g inference loop filter) for simple cases and more complex filters for more complex cases where there are more refraction challenges. Complexity usually has a performance price tag, so it's not wise to use the most complex filter in all situations.

    Your note about query plans and pattern matching optimization gave me an idea to make the dispatch network builder also pluggable for the same reason - for simple cases (like I described - hundreds of handlers, millions of events, relatively short inference chains) the out-of-the box builder should be enough. For cases as you described in our discussion a couple of years ago (thousands of handlers, long infrence chains) a different, more advanced, dispatch network builder might be a better fit. It's like a query plan in relational databases - it is important to build an optimized plan for large data sets but for small data sets it may take more time to build an optimized plan than to execute a query with a simple plan.

    The beauty of pluggable builders and infrence filters is that the client interface will stay intact!

    So, thanks again, I appreciate your feedback!

    I know I'm very critical when it comes to rule engines and studying prior. Glad you were able to look beyond my blunt comments and do additional research. I have a bad habit of saying what ever is on my mind.

     

     

  6. Pluggable matchers have been implemented in 2011/01/20 release of Hammurapi Event Bus.

  7. Yet another rule engine[ Go to top ]

    I've tried Hamurappi about 2 years ago to build a payroll-like application. This hasn't worked out quite well, and I've encountered a few bugs along the way (including several race conditions). So I've abandoned Hamurappi and wrote my own custom code (which also hasn't worked out that well).

    This year I've tried Drools, but got burned by its inadequate language and flaky Truth Maintanance System which was not tested by anything more complex than a textbook sample. Drools might be good for a lot of simple rules, but I found myself spending more time trying to express rules in DRL then eveyrthing else combined.

    Besides, Drools are slow. All the examples I've seen so far are "add an item into working memory and apply rules to it" while I want "load a million of items into working memory and let the engine sort it out".

    Can Hamurappi allow to quicly build my own engine, probably reusing some of Hamurappi's infrastructure?

  8. Yet another rule engine[ Go to top ]

    I've tried Hamurappi about 2 years ago to build a payroll-like application. This hasn't worked out quite well, and I've encountered a few bugs along the way (including several race conditions). So I've abandoned Hamurappi and wrote my own custom code (which also hasn't worked out that well).

    This year I've tried Drools, but got burned by its inadequate language and flaky Truth Maintanance System which was not tested by anything more complex than a textbook sample. Drools might be good for a lot of simple rules, but I found myself spending more time trying to express rules in DRL then eveyrthing else combined.

    Besides, Drools are slow. All the examples I've seen so far are "add an item into working memory and apply rules to it" while I want "load a million of items into working memory and let the engine sort it out".

    Can Hamurappi allow to quicly build my own engine, probably reusing some of Hamurappi's infrastructure?

    You're applying the wrong tool to solve the problem. If you really have a million or millions of facts/objects into a rule engine, it's going to take time. Your options are using a query driven approach like backward chaining engine, translate rules to queries and use data driven rules.

    In theory, backward chaining is "query-like" and doesn't materialize all partial matches. The downside is you have to understand how to write effective rules. Translating rules to SQL queries isn't that hard, but it does take time to do. There's prior art in this area dating back to late 90's. I know someone that did their Phd on that topic.

    Using data driven rules, you can selectively load data as needed, rather than load all million objects up front. Having built systems that had to reason over 1 million facts for an order management system, you have to be smart about loading data. Don't try to load everything up front and use the rules to only load what you need.

    The final option is to buy a license of OPSJ, which can load 1 million facts "way faster" than any other rule engine out there. The bottom line is, take time to study production rule engines, forward chaining, backward chaining, pattern matching and logic programming. Once you have a good understanding of the domain, it will be easier to solve challenging issues like reasoning over 1 million objects. The downside is it takes years to acquire depth in these areas of study. If you have a specific question, feel free to email me woolfel AT gmail DOT com.

  9. Yet another rule engine[ Go to top ]

    I've tried Hamurappi about 2 years ago to build a payroll-like application. This hasn't worked out quite well, and I've encountered a few bugs along the way (including several race conditions). So I've abandoned Hamurappi and wrote my own custom code (which also hasn't worked out that well).

    This year I've tried Drools, but got burned by its inadequate language and flaky Truth Maintanance System which was not tested by anything more complex than a textbook sample. Drools might be good for a lot of simple rules, but I found myself spending more time trying to express rules in DRL then eveyrthing else combined.

    Besides, Drools are slow. All the examples I've seen so far are "add an item into working memory and apply rules to it" while I want "load a million of items into working memory and let the engine sort it out".

    Can Hamurappi allow to quicly build my own engine, probably reusing some of Hamurappi's infrastructure?

    I forgot another option that "may" be a viable solution. If you use Oracle or DB2, you can use java stored procs to call java libraries directly. This means you can compile the rules to pure java and load them in Oracle. Then you just have to trigger the java stored proc to execute the rules within Oracle. The downside is you could impact your database, which could bring the wrath of the DBA.

  10. Yes it can[ Go to top ]

    Alex,

    Hammurapi Event Bus has modular structure. You can provide your own implementation of event store, executor service, and inference filter. In the next release it will also be possible to provide a custom implementation of the dispatch network builder. So, you can start with out-of-the-box components and if any of them falls short you can either enhance it or replace with your own implementation, which may be domain-specific and be more efficient than generic implementations.

  11. Yes it can[ Go to top ]

    Alex,

    Hammurapi Event Bus has modular structure. You can provide your own implementation of event store, executor service, and inference filter. In the next release it will also be possible to provide a custom implementation of the dispatch network builder. So, you can start with out-of-the-box components and if any of them falls short you can either enhance it or replace with your own implementation, which may be domain-specific and be more efficient than generic implementations.

    Honestly, this has nothing to do with event bus or even pluggable design. The problem is more fundamental than that. To put it simply, "how can we reason over millions of data elements efficiently?" We can do divide and conquer for some classes of problems, by breaking the data into partitions and reason over it in parallel. We can intelligently load the data we need, which reduces CPU, Memory and IO load. We can buy really expensive systems and hope the extra hardware speeds up the time. We can buy a license of OPSJ and hope the dataset doesn't grow by an order of magnitude over the next 12/24 months.

    The best option in my mind is intelligently loading the data we need, which ultimately should scale better and be "more future proof". This technique is quite old and has been around for over 2 decades. Gary Riley's book Expert systems and programming talks about it. So does a few other AI text books. Those that have been working with production rule engines for a decade or more on real projects know first hand. There's always going to be "more data" to reason over, so a short sighted solution may work today, but it may not work tomorrow or next year.

  12. To be more clear[ Go to top ]

    The first three approaches I listed can be done with or without a rule engine. Ultimately, solving this class of problems isn't about what rule engine to use, but how to do work more efficiently. Doing a brute force attack only goes so far.

  13. Consumability[ Go to top ]

    Peter,

    I agree with you that doing work more efficiently is the key. For me "more efficiently" means delivering software, which meets requirements, faster. Alex made a good point that many rule languages are foreign for Java developers. I believe that many Java developers, like myself, prefer to think in Java and write in Java to strengthen their Java muscle rather to learn a new language only to forget it after they are done with the current project. I also believe that reasoning is a great concept which many Java developers can leverage in their applications. This is why Event Bus is built as "pure Java" - to make it easier for Java developers to get started with it.

    During my years in IT I saw many problems that could be very nicely solved using a rule engine and rule sets with between a couple of dozens and a couple of hundreds of rules. These problems weren't big enough to justify training of developers and maintenance staff in rule engines and rule languages. So, they were solved by hard-coding the logic, which was sub-optimal, inflexible, but yielded shorter "time to market".

     

  14. Consumability[ Go to top ]

    Peter,

    I agree with you that doing work more efficiently is the key. For me "more efficiently" means delivering software, which meets requirements, faster. Alex made a good point that many rule languages are foreign for Java developers. I believe that many Java developers, like myself, prefer to think in Java and write in Java to strengthen their Java muscle rather to learn a new language only to forget it after they are done with the current project. I also believe that reasoning is a great concept which many Java developers can leverage in their applications. This is why Event Bus is built as "pure Java" - to make it easier for Java developers to get started with it.

    During my years in IT I saw many problems that could be very nicely solved using a rule engine and rule sets with between a couple of dozens and a couple of hundreds of rules. These problems weren't big enough to justify training of developers and maintenance staff in rule engines and rule languages. So, they were solved by hard-coding the logic, which was sub-optimal, inflexible, but yielded shorter "time to market".

    The real problem with using java syntax or "java like" syntax is it tends to lead to procedural thinking. In my experience, the problem with writing lots of rules that are procedural is that it's no different than writing java. For me, the key to effectively using rules is applying logic programming techniques to identify logic issues. By that I mean extracting the logic from the domain and taking a hard look. Often what people "think" is the business problem isn't. It's after you've identified the real problem that you get the benefit of using business rule engines or production rule engines. The other benefit of applying logic programming techniques is that you can employ proof theory to validate the logic is valid and correct. Again, there's a lot of literature on this topic.

    Hard-coding does lead to faster time to market without a doubt, unless we're talking about reasoning over 1 million facts as the core of the business. An example of that is 1940 act, FSA and other financial regulations. Apply a bunch of procedural rules to a simple problem with tiny datasets under 100,000 facts is completely different than reasoning over millions of positions in hundreds of thousands of portfolios.

    It's important to keep in mind what the problem is, before choosing a design that will never work.