Drools 2.0 Released

Discussions

News: Drools 2.0 Released

  1. Drools 2.0 Released (55 messages)

    The Drools development team is proud to announce the release of Drools 2.0 final. Drools is a Rete-based Rules Engine written in Java, but able to run on Java and .Net.

    Drools is designed to allow pluggeable language implementations. Currently rules can be written in Java, Python and Groovy. Drools also enables Domain Specific Languages (DSL) via XML using a Schema defined for your problem domain. DSLs consist of XML elements and attributes that represent the problem domain. An XML Authoring tool provides a semi-rapid development environment with a drag and drop type interface based on the provided Schema.

    The Project Downloads page contains full information on obtaining Drools.

    Support, consultancy and training are available from Iterion.

    Do you know if your application could use rules engines? How would you approach rules-based design?

    Threaded Messages (55)

  2. slight correction[ Go to top ]

    The current release is a forward chaining rule engine and isn't strictly RETE, since it's missing Alpha Node, alpha memory and indexes. Work on full RETE is actively under development, so eventually Drools will be full RETE.

    peter
  3. Drools 2.0 Released[ Go to top ]

    I'm still trying to decide between Drools and Mandarax. I need something that has friendly UI, etc. for those entering the rules.
  4. Drools and Mandarax[ Go to top ]

    They each do completely separate jobs. One is forward chaining the other is backward chaining. Drools is a forward chaining system this means its reactive to data that is asserted into the system, its event based. Mandarax is backward chaining, like prologue, you ask it questions and it tries to give you the answer. For instance in Drools you would assert todays date, and if there is a matching rule it will fire telling you its your birthday. In backward chaining system you would ask it "is it my birthday" and it would search the available goals to determine the answer.

    For an excellent explanation of forward and backward chaining read Charles Forgey's recent articles at http://rulespower.com/ - Forward and Backward Chaining:
    Parts 1, 2 and 3.

    If you want something with a nice user interface, have a look at the Excel integration for decision tables.
  5. Drools and Mandarax[ Go to top ]

    They each do completely separate jobs. One is forward chaining the other is backward chaining. Drools is a forward chaining system this means its reactive to data that is asserted into the system, its event based. Mandarax is backward chaining, like prologue, you ask it questions and it tries to give you the answer. For instance in Drools you would assert todays date, and if there is a matching rule it will fire telling you its your birthday. In backward chaining system you would ask it "is it my birthday" and it would search the available goals to determine the answer.For an excellent explanation of forward and backward chaining read Charles Forgey's recent articles at http://rulespower.com/ - Forward and Backward Chaining:Parts 1, 2 and 3.
    If you want something with a nice user interface, have a look at the Excel integration for decision tables.
    I had seen that. Not sure why they used Excel. :( Once I saw that it was Excel, I didn't go any further.
  6. Drools and Mandarax[ Go to top ]

    They each do completely separate jobs. One is forward chaining the other is backward chaining. Drools is a forward chaining system this means its reactive to data that is asserted into the system, its event based. Mandarax is backward chaining, like prologue, you ask it questions and it tries to give you the answer. For instance in Drools you would assert todays date, and if there is a matching rule it will fire telling you its your birthday. In backward chaining system you would ask it "is it my birthday" and it would search the available goals to determine the answer.For an excellent explanation of forward and backward chaining read Charles Forgey's recent articles at http://rulespower.com/ - Forward and Backward Chaining:Parts 1, 2 and 3.
    If you want something with a nice user interface, have a look at the Excel integration for decision tables.
    I had seen that. Not sure why they used Excel. :( Once I saw that it was Excel, I didn't go any further.

    Take a look through the examples. Drools allows a variety of inputs; XML markup for DSLs Java, Groovy. It also supports POJOs with Spring integration planned. Future versions will also look at creating a none-xml based native language, leaving XML for DSLs and legacy support.
  7. Spreadsheets - can use OpenOffice[ Go to top ]

    The spreadsheet feature will work with OpenOffice, or NeoOffice (java port of open office that my wife uses on mac).

    I wrote the spreadsheet feature, but it is still early days.

    Michael
  8. Drools and Mandarax[ Go to top ]

    It's also worth mentioning that Drools enables XML based Domain Specific Languages. Drools allows XML elements and attributes to be used to represent business rules, this may be combined with a Schema (XSD) and a good XML authoring tool to provide semi-RAD like environments.

    Java based rule:
    <java:condition>
      room.getName( ).equals( "lounge" )
    <java:condition>
    <java:condition>
      convertToCelsius( room.getTemperature() ) > 20
    <java:condition>

    DSL based rule:
    <house:condition>
      <house:room name="lounge">
        <house:temperature>
          <house:greater-than scale="C">20</house:greater-than>
        </house:temperature>
      </house:room>
    </house:condition>
  9. Drools and Mandarax[ Go to top ]

    Have you seen the Oryx UI for Mandarax? That is what I am looking for. If we go with Drools, I might have to write it. :)

    Is there a good, real world, tutorial on Drools? Everything I've seen (been to the Drools site and downloaded stuff) has kind of left me feeling something was missing - Like, how do I actually use it in a project.
  10. In time[ Go to top ]

    Have you seen the Oryx UI for Mandarax? That is what I am looking for. If we go with Drools, I might have to write it. :)Is there a good, real world, tutorial on Drools? Everything I've seen (been to the Drools site and downloaded stuff) has kind of left me feeling something was missing - Like, how do I actually use it in a project.

    A really great IDE takes a long time to build. Once the full RETE is done, which will take a year or so, I hope to start working on an rule IDE. Of course, that's what I would like to do assuming I have time.

    In terms of using Mandarax, and backward chaining rule engines, a backward chaining rule engine is much more forgiving when the developer has no clue or write procedural rules. Taking time to learn how to write rules to take advantage of rule chaining is difficult, but in the end, it produces much better results. Assuming you give the same problem to two rule experts: 1 RETE and 1 backward chaining, the RETE approach will be faster. Backward chaining is query driven, so there really isn't an option to match the static facts in advance. Or if some facts don't change very often, it's a good idea to match on those facts in advance.

    If a backward chaining rule engine implemented the ability to do some of the work before the process begins, it should be able to match RETE. What that achieves is bi-directional chaining. From my experience, it's much easier to implement bi-directional rules using RETE rule engine. It's rather simple to write some beans to execute a query and assert those facts to the rule engine. Not many people use the technique, but it is very powerful.

    peter
  11. IDEs[ Go to top ]

    Depends whom the audience is. IDEs are for developers - not BAs/users (and developers don't like another ide - they want plug ins !).

    End users like the normal office tools (which is why I started using spreadsheets - as that was the language my users were already talking to me in).
  12. IDEs[ Go to top ]

    Depends whom the audience is. IDEs are for developers - not BAs/users (and developers don't like another ide - they want plug ins !).End users like the normal office tools (which is why I started using spreadsheets - as that was the language my users were already talking to me in).

    (Not to put down your work - and sorry if I already set that tone, but ... )
    True. But those apps usually migrate to something more maintainable, etc (they want IT to start maintaining it). I've done plenty of Excel VBA. End users like using Office, but they don't really know what is involved in maintaining something that someone else will use, especially in a tool that wasn't really meant for it.

    When Peter says IDE, I think he means some sort of user interface specifically designed for the task. Nowadays the line between IDE and plugins is very blurred. Is RAD(post-WSAD) an IDE or a runtime engine with a BUNCH of plugins? :)
  13. IDEs[ Go to top ]

    I know what you mean !

    Yeah if it is an IDE or a set of plugins, then yes there is
    definately a need for some sort of tooling as you describe. I think there is a VAR who does something called "DRLAssistant" or something like that (Eclipse plug in).

    I guess it depends on what you are using rules for. Rules are both another way to program (for technical audience) as well as "business" rules - which is a different kettle of fish ! They are both valid uses, and in both cases the tooling would be different (but even for business rules, there still needs to be technical tooling).

    I am personally interested in the business part of the interface (dunno why, am more of a technical loving guy myself, I must just have a soft spot for the rest of the world !).

    I was dis-satisfied with commercial "IDEs" that I saw, as they want to do everything, and consequently want you to do it their way - neither developers NOR business users like to work like that (and developers complain louder).

    I am interested to see all this new value added stuff around drools - certainly a lot happening.
  14. clarification[ Go to top ]

    Depends whom the audience is. IDEs are for developers - not BAs/users (and developers don't like another ide - they want plug ins !).End users like the normal office tools (which is why I started using spreadsheets - as that was the language my users were already talking to me in).
    (Not to put down your work - and sorry if I already set that tone, but ... )True. But those apps usually migrate to something more maintainable, etc (they want IT to start maintaining it). I've done plenty of Excel VBA. End users like using Office, but they don't really know what is involved in maintaining something that someone else will use, especially in a tool that wasn't really meant for it.

    When Peter says IDE, I think he means some sort of user interface specifically designed for the task. Nowadays the line between IDE and plugins is very blurred. Is RAD(post-WSAD) an IDE or a runtime engine with a BUNCH of plugins? :)

    When I say IDE, I am talking about a rule development environment that can provide both DSL and natural language support. I've written rule IDE's 4 times already and have built frameworks to facilitate rule transformation and translation. I consider transformation converting a rule from some sort of programming language to either Natural language or something similar. Rule translation in my mind is more for computer and systems. Like translating rules from RuleML to clips/jrules/blaze format at load time.

    I've mainly focused on two domains over the last 4 years with rule engines: wirless and compliance. In the wireless scenario, the user would use a dedicated rule editor to write rules easily. Most of the scenarios revolve around user profiles and preferences. In the compliance scenario, there's a DSL using the semantics of the domain.

    Assuming one uses a markup like RuleML, it's straight forward to externalize the DSL mapping for translation and grammar mapping for NL transformation. Once the framework is in place, changing from English to French is a matter of changing the grammar mapping. In olde rrule editors, the transformation is hardcoded. Changing from English to German for example would be painful because german grammar is very different. for example, a simple sentence like "I would like to buy a cup of tea" becomes "Ich mochte eine tasse tea kaufen". The verb goes at the end, so hardcoding the translation often means rewriting 2/3 of a rule editor. When one consider translating from English to chinese, the grammar is simple, but the context is critical.

    peter
  15. clarification[ Go to top ]

    When I say IDE, I am talking about a rule development environment that can provide both DSL and natural language support.
    That is what I thought. I just used smaller words (so people like Rolf could understand).
    I've written rule IDE's 4 times already
    So when are you gonna get it right? :) ;)


    But what you described IS what I am looking for. Something that is not programming, per se, but is not free-form either. Most of the rules will be created/entered by those who need "rules" (a tool that constrains them).
  16. You're guess is as good as any[ Go to top ]

    When I say IDE, I am talking about a rule development environment that can provide both DSL and natural language support.
    That is what I thought. I just used smaller words (so people like Rolf could understand).
    I've written rule IDE's 4 times already
    So when are you gonna get it right? :) ;)But what you described IS what I am looking for. Something that is not programming, per se, but is not free-form either. Most of the rules will be created/entered by those who need "rules" (a tool that constrains them).

    I couldn't tell you when I'll get it right. I have refined my approach with each version and learned some very valuable lessons. IE I made lots of mistakes and learned why I was dead wrong. If it was easy, the current crop of rule editors would be much better than they are today. Some rule editors are just god aweful. Luckily, I'm not alone in this effort and others are still searching for a good solution. whether my approach is good or better is very hard to say, since I haven't look at the code for other rule editors beyond the free clips editor for eclipse.

    hopefully in another 10 years i'll be close to getting it right. if such a thing exists.

    peter
  17. Drools 2.0 Released[ Go to top ]

    If you are looking for a good way of entering rules then look at http://www.softlawcorporation.com/ RuleBurst is a backward and forward chaining commercial rulesengine that runs on Java and .NET. The rules capture process is done direct from Word in (almost) natural language. I've found it amazingly productive, especially for business users. Worth a look if you've got the money.
  18. interesting claims[ Go to top ]

    If you are looking for a good way of entering rules then look at http://www.softlawcorporation.com/ RuleBurst is a backward and forward chaining commercial rulesengine that runs on Java and .NET. The rules capture process is done direct from Word in (almost) natural language. I've found it amazingly productive, especially for business users. Worth a look if you've got the money.

    I took a look at softlaw and the claims are interesting. in their benchmark white paper, the claim the following:
    The RuleBurst Engine utilises Linear Inferencing, SoftLaw’s ultra-fast inferencing algorithm, which provides better performance than the Rete algorithm on today’s CPU architectures and is much better suited to batch processing.

    That claim is flat out wrong. What they are comparing is poorly written rules for RETE to linear (ie procedural) inferencing. The link to their white paper shows their rule engine can handle 250 jobs/second. I have no idea what they consider a "job", but having built high performance apps with JESS, on equivalent hardware JESS can run faster than that. Assuming of course the develper takes time to write the rules to take advantage of rule chaining. I'm guessing these rules do not modify any facts, which means a well optimized linear approach would be easier from a development perspective.

    there are many cases where the overhead for RETE rule engines is too great and a business doesn't have the staff required for the project. In those cases, it makes sense from a business perspective to choose that approach, but to claim an algorithm is considerably faster than RETE indicates the author of the white paper hasn't got a clue what RETE does.

    peter
  19. interesting claims[ Go to top ]

    Hi Peter

    Well now I'm in trouble ;-) The CTO contacted me and justified himself thusly:

    - the claim is not wrong – rete is old and is predicated on out-of-date assumptions about inferencing processing scenarios and it doesn’t properly exploit processor caching

    - there are some job metrics provided; basically jobs are large chunks of forward inferencing and the 250 job/sec figure isn’t representative of inferencing performance because there are significant data IO bottlenecks (unless I’m reading the graphs wrong, there seems to be up to 2MB of data IO processed per second as well)

    - benchmarks like 1,000,000 premise evaluations and 250,000 inferences per second per CPU on commodity hardware are more relevant; this is bloody good performance and represents the result of a lot of hard work

    - the inferencing is not procedural; it’s fully declarative; there’s an optimization process that orders the rules to maximise the linearity of processing the rules; there is one very fast linear sweep of the rules, irrespective of how many data items are set, to perform all inferencing rather than the "rule network walking" that rete involves

    - the optimization process means that a developer doesn’t have to take the "time to write the rules to take advantage of rule chaining" which is the whole point of our approach – ie. a tool for non-technical users

    - our rules modify facts

    - the linearity of our approach scales very well for very large rulebases

    - "author of the white paper hasn't got a clue what RETE does" – well that’s a bit strong; softlaw has been doing this for nearly 2 decades and it’s previous implementations of inferencing were rete based, going right back to prolog based implementations; the paper could use a bit more detail, perhaps, but it’s essentially a marketing paper; it’s wrong to claim that we don’t have a clue about this stuff

    Hope that's useful.
  20. interesting claims[ Go to top ]

    first off, apologies for being a bit harsh, but the marketing paper should be revised, so it is clearer.
    Hi Peter
    Well now I'm in trouble ;-) The CTO contacted me and justified himself thusly:

    - the claim is not wrong. rete is old and is predicated on out-of-date assumptions about inferencing processing scenarios and it doesn't properly exploit processor caching

    the above statement is not true of java implementations of RETE. The vm will effectively handle that, especially in a case where multiple engines are running on a SMP machine. It might be true of old prolog rule engines, but that is a result of the underlying prolog engine, not RETE. ART's engine was written in C++ and was very efficient at maximizing performance.
    - there are some job metrics provided; basically jobs are large chunks of forward inferencing and the 250 job/sec figure isn’t representative of inferencing performance because there are significant data IO bottlenecks (unless I'm reading the graphs wrong, there seems to be up to 2MB of data IO processed per second as well)

    those metrics could mean anything honestly. a better measurement would be to run Manners with 64 and 128 guests and see how their rule engine performs.
    - benchmarks like 1,000,000 premise evaluations and 250,000 inferences per second per CPU on commodity hardware are more relevant; this is bloody good performance and represents the result of a lot of hard work

    If you're talking about evaluations, a procedural rule engine can easily do 1 million evaluations, but that is a poor measurement of performance. Manners and Waltz benchmark have been around for almost 2 decades. since they've used prolog, it's rather simple to run the benchmark.
    - the inferencing is not procedural; it's fully declarative; there's an optimization process that orders the rules to maximise the linearity of processing the rules; there is one very fast linear sweep of the rules, irrespective of how many data items are set, to perform all inferencing rather than the "rule network walking" that rete involves

    I can think of two ways of executing rules in a semi-linear fashion. one is to run the rules one at a time (ie procedurally). two is to isolate the rules into modules. the statement "the optimization that orders the rules to maximize the linearity of processing" doesn't make sense to me. that is equivalent of sequencing the rules in a decision tree or table so that the most restrictive constraints are processed first, therefore avoiding the bulk of the rules. That approach works and is effective, but it is only as good as the rule compiler. If the rule compiler sequences the rules in a suboptimal order, then you get no benefit. Keeping mind that inferencing is not strictly limited to RETE. Inferencing from an user perspective is when the rule engine arrives at a conclusion based on partial facts. Inferencing from a low level technical perspective is the assertion of new facts or modification of facts during the reasoning cycle.
    - the optimization process means that a developer doesn't have to take the "time to write the rules to take advantage of rule chaining" which is the whole point of our approach; ie. a tool for non-technical users

    having worked on rule applications, the bulk of the problem is the learning curve. it would seem that ruleburst approach is to aggressively profile and simply the rules so that it produces an optimal decision tree. their approach seems similar to Corticon in that both have correctly identified the fact that most business don't "really" need inferencing in the classical AI sense. Instead, what they need is to use the rules to filter facts and then execute an aggregate query (to some external source), thereby improving overall performance and reducing memory requirements. These techniques are rather common.
    - our rules modify facts - the linearity of our approach scales very well for very large rulebases

    assuming the tools generate an optimal decision tree, i would say that is true. LEAPS algorithm pioneered some of these techniques, so it's likely they are using the techniques proposed by LEAPS algorithm.
    - "author of the white paper hasn't got a clue what RETE does"; well that's a bit strong; softlaw has been doing this for nearly 2 decades and it's previous implementations of inferencing were rete based, going right back to prolog based implementations; the paper could use a bit more detail, perhaps, but it's essentially a marketing paper; it's wrong to claim that we don't have a clue about this stuff Hope that's useful.


    again, i sincerely apologize to the author. I've had similar discussion with Corticon and the thing is. If the tools/IDE can generate an optimal sequence of rules or a decision table, it should be straight forward for a rule engineer to generate either forward or backward chaining rules. Corticon takes a backward chaining approach.

    I'm definitely not an expert, but from my 5 years of experience writing rule compilers, rule editors and working on rule engines, to fault RETE in this cases is wrong. RETE is not the cause of the performance issues people see. It is the tools, approach, rules and implementation. The actual RETE algorithm describes the optimal method of pattern matching. Whether that is used for inferencing or optimizing rule evaluation is a separate issue. The primary limitation of sequencing rules is that if you need to add a new rule, the entire rulebase has to be re-evaluated and re-optimized. that in itself is fine, but where a linear approach fails is the manners and waltz cases. From my experience, a linear evaluation approach to manners and waltz will perform an order of magnitude worse than RETE.

    peter
  21. interesting claims[ Go to top ]

    Great. Read Peter's post and now I have a headache. :)

    Good stuff.
  22. that's all easy stuff[ Go to top ]

    Great. Read Peter's post and now I have a headache. :)

    Good stuff.

    thanks for the laugh. compared to some of the people I know, I've only scratched the surface of rule development. once you get into other areas like Knowledge base/case base reasoning, there a lot more that i barely know.

    peter
  23. that's all easy stuff[ Go to top ]

    Great. Read Peter's post and now I have a headache. :)Good stuff.
    thanks for the laugh. compared to some of the people I know, I've only scratched the surface of rule development. once you get into other areas like Knowledge base/case base reasoning, there a lot more that i barely know.peter
    Know what you mean. I've gathered tons of stuff in my head (I've been in aircraft maint, Satellite System Maint/Computer Hardware and now software) and am far ahead of many. But then I talk to people like you and Cameron and I can't believe how much there still is to learn.

    I need to get a hold of some of those "study pills" or "homework pills" or whatever the kids are calling them.
  24. interesting claims[ Go to top ]

    thanks for responding to my post and please feel free to pass on my apologies to the CTO of Softlaw. I decided to do a bit of research on "linear inferencing" and came across this paper on ACMQueue Implementing deductive databases by mixed integer programming.

    Although I've come across these techniques in the past, my understanding of these techniques are shallow. The abstract is interesting:
    Existing and past generations of Prolog compilers have left deduction to run-time and this may account for the poor run-time performance of existing Prolog systems. Our work tries to minimize run-time deduction by shifting the deductive process to compile-time. In addition, we offer an alternative inferencing procedure based on translating logic to mixed integer programming. This makes available for research and implementation in deductive databases, all the theorems, algorithms, and software packages developed by the operations research community over the past 50 years. The method keeps the same query language as for disjunctive deductive databases, only the inferencing procedure changes. The language is purely declarative, independent of the order of rules in the program, and independent of the order in which literals occur in clause bodies. The technique avoids Prolog's problem of infinite looping. It saves run-time by doing primary inferencing at compile-time. Furthermore, it is incremental in nature.

    If I understand that abstract correctly, my initial guess at Softlaw's approach should be fairly close. by improving the rule compilation, and removing some of the inferencing, a rule engine can do less work. Even more interesting is the review of the paper at the bottom of the page.
    The authors propose a new inferencing procedure for deductive databases that allows the deduction to be done at compile time rather than runtime. They also offer an alternative inferencing procedure, based on mixed integer programming, that is intended to make available to deductive database systems all of the algorithms and packages that have been developed by the “operations research community over the past 50 years.” Their approach is based on the fact that linear programming can replace logical deduction if the logical conditions are represented as Boolean inequalities of the form c1x1+&ldots;+cnxn ≥a , where a,c1,&ldots;,cn represent integers. The authors extend this approach so that they “solve a symbolic logic program by treating it as a numerical problem.” Thus, a deductive database is translated into a set of linear constraints. The least models for such a database can then be computed using a linear programming algorithm. This work has much in common with the work that has been done on finding efficient ways to provide materialized views in relational databases. The authors provide a method for translating deductive databases into sets of linear constraints, show their correspondence with minimal models, and provide a technique for computing the minimal models. Although much of their work is theoretical, and they admit to not knowing an efficient way to accomplish some of the computations, they have successfully demonstrated their approach with a prototype compiler.

    I've discussed these techniques with friends on numerous occasions. I'm just guessing, but from what I know, the statement "The language is purely declarative, independent of the order of rules in the program, and independent of the order in which literals occur in clause bodies." isn't really achievable for the general case. On the otherhand, if you sequence the rules in optimal order, it should be feasible to gain a significant performance boost. of course, the trade off is you loose flexibility. One of the founders of RuleML has experience in this area ABE. Horn logic is a related topic in this domain. I also came across another article in the same domain On linear potential functions for approximating Bayesian computations.

    I think rather finding fault with RETE, Softlaw's white paper would be more direct and clearly describe their approach as rule compilation optimization.

    peter
  25. doh typos galore[ Go to top ]

    What I meant to say was this.

    If Softlaw's white paper describe their approach as rule compilation optimization, it would be much more effective. Incorrectly blaming RETE for poor prolog implementation only serves to discredit their argument and achievements. RETE has been tested in numerous scenarios and has been proven to be the most efficient pattern matching algorithm for most cases. For edge cases, algorithms like Treat, Leaps and others can provide better performance. The trick is understanding when to RETE, Treat, or leaps.

    peter
  26. doh typos galore[ Go to top ]

    Thanks for the extensive replies Peter, with a pretty impressive bibliography shaping up as well ;-)

    I too am off to the pharmacy for some "study pills".

    I'm no expert on the theoretical or practical implementation of rules engines so will shut my trap on that subject. However the rules capture process is why I orginally mentioned Softlaw - in response to somebody asking abut good IDEs.

    I take the point above that Word may be a proprietary environment - however that would be true of any "IDE". And change tracking does provide one way of managing change - if you consider Word as potentially a colloborative environment. However there are issues about how you can integrate this with versioning systems.

    In common with a previous poster I have also implemented rules capture using Excel and OpenOffice Calc. On one project when I realised our system testers had all their test cases running in Excel before we'd started development, I looked at the feasibility of just writing an "Excel -> ILOG compiler" - hence neatly sidestepping the implementation phase ;-) Needless to say it wasn't quite that simple - but would have been quite a coup - truly test driven development!

    Anyway the spreadsheet format seems a perfect fit for a while - until the number of rules and the level of complexity hits a certain level then it rapidly becomes an albatross.

    However in my experience this isn't the case with the RuleBurst Word-centric approach, and an investigation is currently under way about the feasibility of targeting the compiled rule output from RuleBurst for other rules platforms such as ILOG, RuleML etc.

    Luke
  27. doh typos galore[ Go to top ]

    Thanks for the extensive replies Peter, with a pretty impressive bibliography shaping up as well ;-)I too am off to the pharmacy for some "study pills".I'm no expert on the theoretical or practical implementation of rules engines so will shut my trap on that subject. However the rules capture process is why I orginally mentioned Softlaw - in response to somebody asking abut good IDEs.I take the point above that Word may be a proprietary environment - however that would be true of any "IDE". And change tracking does provide one way of managing change - if you consider Word as potentially a colloborative environment. However there are issues about how you can integrate this with versioning systems.In common with a previous poster I have also implemented rules capture using Excel and OpenOffice Calc. On one project when I realised our system testers had all their test cases running in Excel before we'd started development, I looked at the feasibility of just writing an "Excel -> ILOG compiler" - hence neatly sidestepping the implementation phase ;-) Needless to say it wasn't quite that simple - but would have been quite a coup - truly test driven development!Anyway the spreadsheet format seems a perfect fit for a while - until the number of rules and the level of complexity hits a certain level then it rapidly becomes an albatross. However in my experience this isn't the case with the RuleBurst Word-centric approach, and an investigation is currently under way about the feasibility of targeting the compiled rule output from RuleBurst for other rules platforms such as ILOG, RuleML etc.

    Luke

    Targetting the output to other formats should be straight forward. Take for example the excel case. Assuming a business user creates a grid of the atomic constraints and the possible states, the process can be broken down into several steps.

    1. create a mapping from the columns (assuming conditions are the columns) to object.field

    2. first sort the columns by count from lowest to highest.

    3. sort the states from in ascending order based on the number of constraints that state has. The excel spreadsheet should look like a decision tree.

    At this point, one can do several things to optimize the rules. Any condition that all states must have can be discarded, since it doesn't differentiate the states. If a condition is used by all rules, but the object is needed for the right-hand side (ie the action/result) of the rule, it's best to use forward chaining to evaluate that specific condition. The reason for this is so the fact is evaluated once for all rules, rather than once for each rule.

    The next thing is to see which conditions are used by a large percentage of the rules. If the condition is used by more than 30% (this is arbitrary) of the rules, you can analyze it and determine if the condition can be queried at runtime. If it can, it's easy to write a bean to execute the query based on the conditions that do match. If the condition isn't an aggregate, the next thing I would do is take the subset of states with that condition and count which conditions are shared within the subset.

    As you repeat this process, you end up with a priority list. by priority list, I mean ordering the conditions from more important to less important. Once you have the priority list, it's straight forward to determine if a condition should use chaining or not. A simple way of making the priority list is to calculate the weight of each condition from 1 to 100.

    There are other tricks and techniques, but these are some of the more common ones.

    peter
  28. More techniques[ Go to top ]

    Thanks for the extensive replies Peter, with a pretty impressive bibliography shaping up as well ;-)

    I too am off to the pharmacy for some "study pills".

    I'm no expert on the theoretical or practical implementation of rules engines so will shut my trap on that subject. However the rules capture process is why I orginally mentioned Softlaw - in response to somebody asking abut good IDEs.
    I take the point above that Word may be a proprietary environment - however that would be true of any "IDE". And change tracking does provide one way of managing change - if you consider Word as potentially a colloborative environment. However there are issues about how you can integrate this with versioning systems.In common with a previous poster I have also implemented rules capture using Excel and OpenOffice Calc. On one project when I realised our system testers had all their test cases running in Excel before we'd started development, I looked at the feasibility of just writing an "Excel -> ILOG compiler" - hence neatly sidestepping the implementation phase ;-) Needless to say it wasn't quite that simple - but would have been quite a coup - truly test driven development!Anyway the spreadsheet format seems a perfect fit for a while - until the number of rules and the level of complexity hits a certain level then it rapidly becomes an albatross. However in my experience this isn't the case with the RuleBurst Word-centric approach, and an investigation is currently under way about the feasibility of targeting the compiled rule output from RuleBurst for other rules platforms such as ILOG, RuleML etc.

    Luke

    I'm probably going to bore everyone, but I thought it would might be useful to point out some of the benefits of going with a pure RETE approach. The process I described in the previous post provides some tips for generating rules for non-Rete rule engines. In procedural approaches, the rules have to be in optimal order to get good performance. Calculating the priority of the conditions being one way of figuring which rules are more important.

    If I had to take the same decision table and generate rules for a RETE engine, I can simple map the conditions to the objects and create a map of the joins between objects. I don't need to sort the conditions, states and evaluate the "importance" of each condition. This makes the rule compiler much simpler, because RETE will automatically share the evaluation of each condition between the rules. Since procedural rule engines evaluate a given condition n times for n rules, the sequence of the rules is absolutely critical.

    leaf node evaluations = c (condition) x r (rules) x facts

    Forgy's RETE algorithm really is an elegant and powerful solution. A good RETE rule engine that implements node sharing reduces the number evaluations.

    leaf node evaulations = c (condition) x facts
    number of conditions = uc (unique condition) + sc (shared condition)

    When one looks at a decision table, often many states share the same conditions. In this case, a picture is worth a thousand words.

    peter
  29. More techniques[ Go to top ]

    I'm probably going to bore everyone, ...

    (I forgot to post this)
    Thanks for the Sunday afternoon nap! :)
  30. Good work[ Go to top ]

    I think the guys over there deserve a good pat on the back for the hard work. An open source, fully RETE based, forward chaining rules engine, with support for DSLs is what the industry needs at this point.

    I like Drools because its simple (inspite of being RETE-OO based, which is quite difficult to assimilate at first). In fact, I learnt RETE and RETE-OO from/with Drools.

    I am hoping that at least there will be more widespread adoption of business rules in typical small business applications (internal automation and the likes) with an open source alternative like Drools. If nothing else, there will be more and more developers writing rules for Drools, if it remains open source. Other commercial rules engines (and I mean forward chaining) don't even have a decent trial download.

    After taking a look at the DSL support and semantic module framework of Drools, building an impressive Eclipse plugin or a full fledged IDE should be fairly trivial, methinks!

    All the folks out there comparing Drools with Mandarax and Oryx... its incorrect to compare forward-chaining with backward-chaining... its like comparing apples and oranges.

    Lastly, Decision Table is one of the most effective ways of writing business rules. I have been working on a business rules implementation for a large scale government project ... the only tool their policy staff can work with is Excel and the only language they seem to understand is Yes/No ;-)
  31. It's not full RETE[ Go to top ]

    I think the guys over there deserve a good pat on the back for the hard work. An open source, fully RETE based, forward chaining rules engine, with support for DSLs is what the industry needs at this point. I like Drools because its simple (inspite of being RETE-OO based, which is quite difficult to assimilate at first). In fact, I learnt RETE and RETE-OO from/with Drools. I am hoping that at least there will be more widespread adoption of business rules in typical small business applications (internal automation and the likes) with an open source alternative like Drools. If nothing else, there will be more and more developers writing rules for Drools, if it remains open source. Other commercial rules engines (and I mean forward chaining) don't even have a decent trial download.After taking a look at the DSL support and semantic module framework of Drools, building an impressive Eclipse plugin or a full fledged IDE should be fairly trivial, methinks!All the folks out there comparing Drools with Mandarax and Oryx... its incorrect to compare forward-chaining with backward-chaining... its like comparing apples and oranges.Lastly, Decision Table is one of the most effective ways of writing business rules. I have been working on a business rules implementation for a large scale government project ... the only tool their policy staff can work with is Excel and the only language they seem to understand is Yes/No ;-)

    I already mentioned this in an earlier post, but the current release is only forward chaining and not full RETE. There's a very important distinction between just forward chaining and full RETE. Forward chaining can be implemented in any number of ways and the joins can be arbitrary. Most non-RETE forward chaining rule engines see a significant performance drop when the ruleset size increases. In other words, it fails to meet the primary benefit of RETE. One easy way to see if a rule engine really implements RETE is to load 10, 100, and 1000 rules and look at the response time to fire a rule. A rule engine that does not really implement RETE will usually see exponential increase in reasoning time.

    peter
  32. It's not full RETE[ Go to top ]

    I know. I was going by what you said in your earlier post about Drools becoming fully RETE-OO based later, and assuming that it remains open-source!
  33. yup... still apache license[ Go to top ]

    The guys are currently merging the new implementation I worked on over the last 3 months, but I'm not officially a committer. More like I'm just creating more work for the others. The work is progressing and the basic implementation of alpha, beta node and the required memory is there. The addition of "NOT" exist test node is done, but that will be merged in later. There's also CLIPS test node, which is basically done, but merging it will be more challenging. CLIPS was the inspiration for the new implementation and I tried to stay as close to CLIPS as practical. Some of the features I am working on include Just-In-Time optimization of the alphanodes and optimization of the Not exist node. Whether I get around to it will depend on how much time I have, since I contribute to several other OSS projects.

    In time, the implementation will be solid, but it's a boat load of work.

    peter
  34. Good work[ Go to top ]

    All the folks out there comparing Drools with Mandarax and Oryx... its incorrect to compare forward-chaining with backward-chaining... its like comparing apples and oranges.
    Well I was not really trying to compare them here. I was saying, check out the Oryx tool. That is what I would like for Drools. Not a copy cause I know they work different, but the concept.
    Lastly, Decision Table is one of the most effective ways of writing business rules. I have been working on a business rules implementation for a large scale government project ... the only tool their policy staff can work with is Excel and the only language they seem to understand is Yes/No ;-)
    Does it have to be in Excel? Can is just be a tool that has a grid?
  35. sure, if you build it :)[ Go to top ]

    All the folks out there comparing Drools with Mandarax and Oryx... its incorrect to compare forward-chaining with backward-chaining... its like comparing apples and oranges.
    Well I was not really trying to compare them here. I was saying, check out the Oryx tool. That is what I would like for Drools. Not a copy cause I know they work different, but the concept.
    Lastly, Decision Table is one of the most effective ways of writing business rules. I have been working on a business rules implementation for a large scale government project ... the only tool their policy staff can work with is Excel and the only language they seem to understand is Yes/No ;-)
    Does it have to be in Excel? Can is just be a tool that has a grid?

    it doesn't have to be excel or openoffice. you could use SWT/Swing and just make a table. that's what I've done in the past. some just to write it :)

    peter
  36. sure, if you build it :)[ Go to top ]

    All the folks out there comparing Drools with Mandarax and Oryx... its incorrect to compare forward-chaining with backward-chaining... its like comparing apples and oranges.
    Well I was not really trying to compare them here. I was saying, check out the Oryx tool. That is what I would like for Drools. Not a copy cause I know they work different, but the concept.
    Lastly, Decision Table is one of the most effective ways of writing business rules. I have been working on a business rules implementation for a large scale government project ... the only tool their policy staff can work with is Excel and the only language they seem to understand is Yes/No ;-)
    Does it have to be in Excel? Can is just be a tool that has a grid?
    it doesn't have to be excel or openoffice. you could use SWT/Swing and just make a table. that's what I've done in the past. some just to write it :)peter
    That was what I was getting at. :) Users say "I want a spreadsheet." What they mean is the grid format. Happened just last week.

    Well, if I use Drools, I may have to build it. No problem (with having to do it).
  37. sure, if you build it :)[ Go to top ]

    All the folks out there comparing Drools with Mandarax and Oryx... its incorrect to compare forward-chaining with backward-chaining... its like comparing apples and oranges.
    Well I was not really trying to compare them here. I was saying, check out the Oryx tool. That is what I would like for Drools. Not a copy cause I know they work different, but the concept.
    Lastly, Decision Table is one of the most effective ways of writing business rules. I have been working on a business rules implementation for a large scale government project ... the only tool their policy staff can work with is Excel and the only language they seem to understand is Yes/No ;-)
    Does it have to be in Excel? Can is just be a tool that has a grid?
    it doesn't have to be excel or openoffice. you could use SWT/Swing and just make a table. that's what I've done in the past. some just to write it :)peter
    That was what I was getting at. :) Users say "I want a spreadsheet." What they mean is the grid format. Happened just last week. Well, if I use Drools, I may have to build it. No problem (with having to do it).

    Why build it, when its already there are pretty much ubiquitous (c'mon, all large coporations have excel... for those who don't open-office spreadsheet tool works!). God bless Apache POI.
  38. sure, if you build it :)[ Go to top ]

    Depends. I have not tried the suggested tool yet. But I built and used other tools imlemented in such a way. If it works like I need it and I don't have to touch the code - then ok.

    But I don't see how what Peter is describing can be done in Excel.
  39. sometimes dumb is good[ Go to top ]

    Depends. I have not tried the suggested tool yet. But I built and used other tools imlemented in such a way. If it works like I need it and I don't have to touch the code - then ok. But I don't see how what Peter is describing can be done in Excel.

    even though an advanced editor is nice, sometimes you just want a dumb editor for a dumb user. there are plenty of dumb users, who would mess up with a fancy editor. anyone in the trenches of rule development has come across atleast 1 case where a reasonably intelligent person becomes a total moron when give a nice editor. a dumb user in thise case may not necessary be a dumb person. they just go blank and start behaving like mad man.

    peter
  40. All these rule engines that do natural language I think are bound to fail because of the issue that you raised. Rulesharp uses regular expressions and wildcards and you can build your own condition processor. No translation needed. Everything is in database and the rule format has a simple xml-schema. hope it will help you with what you are trying to do. it is at: www.rulesharp.com randy
  41. OMG Production Ruleml[ Go to top ]

    Related to the topic of rules, currently there is an initiative with OMG for production ruleml. I believe the goal is to get a draft specification out by the end of the year and a final spec out some time next year. If that happens, it's likely that more modeling programs supporting UML, XMI will support rule modeling. Obviously this is all in the future and there's a ton of work to get to a good product, but hopefully the OMG effort will help make rule engineering easier.

    Part of the OMG effort will produce an object model for the major platforms like J2EE and .NET. I just posted a release candidate of Dingo that can generate either C# or Java code from xml schema.

    peter
  42. ruleml[ Go to top ]

    Yeah that would make the rule knowledge portable, which would be nice. Of course with rules, you need to have some sort of expression language, and that always has a limit. Getting to the data in an object model is only part of the problem. The other part is the functions you need in the rule engine itself to manipulate it (some would say none - you should do all the "work" in the underlying language - but that is less flexible).

    It has been a long time coming, so it may be still a long time yet.
  43. Biz users 'writting' rules[ Go to top ]

    I have a general business rules process question for the people out there that have a system in production. The company I work for is implimenting a third party processing system that has a rules engine in it. One of the selling points that was made was that business users can write business rules on their own. I think what they should have said was 'business users can define rules on their own'. I imagine most companies have the business users define the rules using a template/form type process and a developer person actually bangs them into the system?

    Also, what process are people using for changing the rules in production? One selling poing of a rules engine is you can 'change stuff on the fly', but the thought of changing stuff without full QA cycles makes me kind of uneasy (the old 'I just need to change this small snippet of HTML' problem).
    Any thoughts?

    Thanks
    Mike
  44. Biz users 'writting' rules[ Go to top ]

    I have a general business rules process question for the people out there that have a system in production. The company I work for is implimenting a third party processing system that has a rules engine in it. One of the selling points that was made was that business users can write business rules on their own. I think what they should have said was 'business users can define rules on their own'. I imagine most companies have the business users define the rules using a template/form type process and a developer person actually bangs them into the system? Also, what process are people using for changing the rules in production? One selling poing of a rules engine is you can 'change stuff on the fly', but the thought of changing stuff without full QA cycles makes me kind of uneasy (the old 'I just need to change this small snippet of HTML' problem). Any thoughts?

    Thanks
    Mike

    Good questions. The answer to that depends on how the rule editor works and what kind of automated validation the system employs. for example, a wizard driven rule editor can be used by end users like business analysts, but what i would do is have a biz user create a new template. the biz user should be part of defining the new template, but the developer should be responsible for creating the code to back that up and run through all the required steps to insure the new template is good.

    only time i would say bypass all these steps is when performance, scalability or reliability doesn't matter. if the business expects the server to crash and perform unreliably, then sure biz users can write rules and deploy them immediately.

    the biz rule marketing guys have the sales pitch down and over sell. so it results in a backlash when the customer realizes you still need qa and run through a smoke test. why people fall for this is beyond me.

    peter
  45. Biz users 'writting' rules[ Go to top ]

    Good questions. The answer to that depends on how the rule editor works and what kind of automated validation the system employs. for example, a wizard driven rule editor can be used by end users like business analysts, but what i would do is have a biz user create a new template. the biz user should be part of defining the new template, but the developer should be responsible for creating the code to back that up and run through all the required steps to insure the new template is good.only time i would say bypass all these steps is when performance, scalability or reliability doesn't matter. if the business expects the server to crash and perform unreliably, then sure biz users can write rules and deploy them immediately.the biz rule marketing guys have the sales pitch down and over sell. so it results in a backlash when the customer realizes you still need qa and run through a smoke test. why people fall for this is beyond me.peter

    Well, when don't salespeople oversell something :) I think in this instance it's a case of the product does allow it, but it doesn't mean it's a good idea.

    I think the other big sales pitch for rules engines is 'anyone can write rules'. Joe in accouting can replace your IT department....
  46. Biz users 'writting' rules[ Go to top ]

    I think the other big sales pitch for rules engines is 'anyone can write rules'. Joe in accouting can replace your IT department....
    Sounds like the MDA tool pitch.
  47. Biz users 'writting' rules[ Go to top ]

    Well, when don't salespeople oversell something :) I think in this instance it's a case of the product does allow it, but it doesn't mean it's a good idea. I think the other big sales pitch for rules engines is 'anyone can write rules'. Joe in accouting can replace your IT department....

    sales people are a necessary part of life. if it wasn't for sales people, where would most IT people be? I think just about every single rule provider today claims "joe in accounting can write rules without the developer."

    of course when joe blows up the production server, who is responsible for staying all night to fix the problem? maybe in another 20 years rule ide's and DSL will have matured enough to make it less hazardeous for joe accounting to write a rule, click "validate rule" and schedule for deployment.

    peter
  48. Biz users 'writting' rules[ Go to top ]

    I have seen several rules engine implementations into production, and there is often that lingering requirement that biz users be able to edit the rules. I've often seen rules expressed in an active way through Excel - but they quickly become hard to follow and maintain. The most effective way I have come accross is using natural language (or as close as possible - softlaw's CEDF constaint explicit definition format) which for RuleBurst (from Softlaw) is expressed in Word by applying styles. From a management point of view this is great - you don't need a separate IDE and you can always turn on Word change tracking to manage changes to the rulebase!

    My personal opinion is that one rules engine is pretty much like another at runtime. The real trick is in how you capture and manage the actual rules - this is where the ROI happens.


    (Disclaimer: I am currently working on a contract with Softlaw, but have to say that having worked on several projects with (and without) them I am really impressed with their rules capture process)
  49. Biz users 'writting' rules[ Go to top ]

    The most effective way I have come accross is using natural language
    I would think so too. That is what I liked about the tool I mentioned.

    (or as close as possible - softlaw's CEDF constaint explicit definition format) which for RuleBurst (from Softlaw) is expressed in Word by applying styles. From a management point of view this is great - you don't need a separate IDE and you can always turn on Word change tracking to manage changes to the rulebase!HE-larious. Ok not really. When you say management do you mean managing or the bosses?

    While it may work, I can't see how using Word (a VERY proprietary tool) and its change tracking is a good idea. Especially if you need to version the rules with your codebase.
  50. but why fight the biz guys[ Go to top ]

    The most effective way I have come accross is using natural language
    I would think so too. That is what I liked about the tool I mentioned.(or as close as possible - softlaw's CEDF constaint explicit definition format) which for RuleBurst (from Softlaw) is expressed in Word by applying styles. From a management point of view this is great - you don't need a separate IDE and you can always turn on Word change tracking to manage changes to the rulebase!
    HE-larious. Ok not really. When you say management do you mean managing or the bosses?While it may work, I can't see how using Word (a VERY proprietary tool) and its change tracking is a good idea. Especially if you need to version the rules with your codebase.
    sometimes fighting the biz guys is a loosing battle. if they really want to use word, then might as well use word.

    peter
  51. but why fight the biz guys[ Go to top ]

    sometimes fighting the biz guys is a loosing battle. if they really want to use word, then might as well use word.peter
    That is why they shouldn't be making technology decisions. It is our job to help them see why they need to choose the right tools for the job. That has little to do with dislikes\likes.

    But you are right, sometimes you can't use the right tool for the job.
  52. Drools 2.0 Released[ Go to top ]

    Looks like I will be needing to develop some sort of UI to develop drl files.

    I looked at what was available and for some of the things I will be doing, they will work. I think DRLAssistant will be good for visualizing the relationships but for the actual development, I think more is needed.

    One of the things I will be doing is developing rules for questionaires. They can be quite large and one question can affect a lot of things. Like if you answer yes to one thing, some new questions are added and some others are removed.

    Maybe I am thinking it is more complicated than it is. But I showed everything to someone else and he thought the same.

    I would like to maybe see if we can get an OSS project going. Peter? What do you think? I probably will be doing a DSL within Drools
  53. Drools 2.0 Released[ Go to top ]

    Looks like I will be needing to develop some sort of UI to develop drl files. I looked at what was available and for some of the things I will be doing, they will work.

    I think DRLAssistant will be good for visualizing the relationships but for the actual development, I think more is needed.

    One of the things I will be doing is developing rules for questionaires. They can be quite large and one question can affect a lot of things. Like if you answer yes to one thing, some new questions are added and some others are removed.

    sounds like you're planning on using classic KB (knowledge base) techniques. The animal classification example in CLIPS and JESS come to mind. I'm not an expert in this field or remotely close, but the basic technique is to encode the knowledge as rules. I know some systems have used the KB approach to build an interactive Help/support application. If you use the CLIPS example as a starting point, building a simple UI shouldn't take more than 6-8 months :) A prototype shouldn't take more than a few weeks, but a complete UI should take about 8 months.
    Maybe I am thinking it is more complicated than it is. But I showed everything to someone else and he thought the same. I would like to maybe see if we can get an OSS project going. Peter? What do you think? I probably will be doing a DSL within Drools

    I wish I had more time, but right now I'm focused on the the RETE rewrite.

    peter
  54. Drools 2.0 Released[ Go to top ]

    If you use the CLIPS example as a starting point, building a simple UI shouldn't take more than 6-8 months :) A prototype shouldn't take more than a few weeks, but a complete UI should take about 8 months.
    Only 8 months? Thought it would be much more than that. :)
  55. Drools 2.0 Released[ Go to top ]

    If you use the CLIPS example as a starting point, building a simple UI shouldn't take more than 6-8 months :) A prototype shouldn't take more than a few weeks, but a complete UI should take about 8 months.
    Only 8 months? Thought it would be much more than that. :)

    I guess since i've done it a few times, my estimate may be "a bit unrealistic" and not true for the general case. Ask me again next year and maybe i'll have enough time to work on a rule editor. though I doubt you really need my help.

    peter
  56. Drools 2.0 Released[ Go to top ]

    Just had to mess with you. :)

    Thanks for your input.