Jetbrains Releases White Paper on Domain-Specific Languages

Home

News: Jetbrains Releases White Paper on Domain-Specific Languages

  1. Sergey Dmitriev of Jetbrains has released a new whitepaper titled 'Language Oriented Programming: The Next Programmign Paradign'. It outlines Jetbrains' forthcoming MPS product, which operates in the same space as Microsoft's Software Factories.

    Sergey promises that IDEA users will soon be able to experiment with his take on domain-specific languages:

    There is already a prototype plugin for IntelliJ IDEA which will allow you to include MPS concept models in your project. The models will automatically be translated into Java source code in the background as you edit them. So, you will be able to write part of your Java applications using MPS, as much or as little as you want. This means that you get all the power of MPS, such as the ability to create and use specialized DSLs, to make whatever language extensions you want, as well as to use customizable editors with code completion, error highlighting, refactoring, etc. The plugin will be tightly integrated with IDEA, allowing you to embed Java code in your MPS models, navigate to embedded or generated Java code, and even perform concept-level debugging similar to the JSP debugging support already available in IDEA.

    Read Language-Oriented Programming

    Threaded Messages (67)

  2. Plugin: where?[ Go to top ]

    Anyone know where to get the mentioned IDEA plugin?

    I'd like to try and create a language that generates pointcut definitions for AOP.
  3. Interesting[ Go to top ]

    Very Interesting


    You can find the old version here http://www.sergeydmitriev.com/mps/

    No idea where the new one is have to check the Intellij forums

    Cheers,
    Parthi
  4. Re:Interesting[ Go to top ]

    We'll open an early access to MPS later, in about 1-2 months. You can subscribe to RSS feed at http://www.jetbrains.com/mps/ to be notified about releases.
  5. what happened to IP?[ Go to top ]

    The article refers to Charles Simonyi's Intentional Programming. The site on IP at Microsoft Research disappeared. Does anybody know what happened to it?
  6. what happened to IP?[ Go to top ]

    Charles Simonyi has started his own company called Intentional Software. They are pursuing IP based on the work Charles did at Microsoft (they have an agreement of some kind with Microsoft).
  7. what happened to IP?[ Go to top ]

    Yes, I know. I did not state my question clearly.
    What I'm wondering is: what happened with it at Microsoft. Was it tranformed into a product? If not: why not? Does Whitehorse incorporate Intentional Programming?
  8. what happened to IP?[ Go to top ]

    Someone mentioned somewhere that Microsoft has a project called "Software Factories". I don't know much about it. I do know that Microsoft is still interested in the general concept of IP, but they are probably going to do it differently than Simonyi would, so I don't think that IP itself is alive at Microsoft, even though some related projects are.
  9. what happened to IP?[ Go to top ]

    Charles Simonyi's 1995 paper on Intentional Programming, 'The Death Of Computer Languages, The Birth of Intentional Programming' appears to be available here. Interesting stuff.
  10. slight problem I see[ Go to top ]

    From experience, it's not easy for an average programmer to design a language. In fact, i would say it's actually rather hard to design a domain specific language which is structurally sound, expressive and translatable. Just look at the history of natural language inspired languages. Many attempts to create a more expressive domain specific language fail. One of the common goals in AI and expert systems is creating a "natural language" like language, which business users can use. I've seen a couple different attempts. The failed attempts start with the assumption of a limited set of natural language elements, which then map to specific structures. In practice, this approach to langauge design doesn't work for expert systems. Approaches that work start out with a sound structure based on first order logic or other rigorous mathematic principles.

    Once the structure is sound, the domain specific language can then map to one or more natural language syntax. There are numerous examples of this in compiler books and natural language parser theory. For LOP to work, I would think the person designing the language has to be an expert in language design and the target domain. A person with language design experience, but no domain expertise won't be able to design a good DSL. An expert in a domain has to have experience designing languages to create a DSL that is feasible.

    Looking at the screenshot of the editor on page 7, it's nice to have a GUI for designing languages, but defining the structured language isn't easy. having worked on rule engines and rule editors the last 4 years, the reality of language design isn't that simple. In fact, I would argue that some domains are sufficiently complex that only a hand full of individuals have the experience and skills.
  11. Re:slight problem I see[ Go to top ]

    Yes, it is not so easy task to design new domain-specific language. But that is exactly what we do every time when designing a new class library - in this case our task is even more difficult, because we design a language, by representing its concepts by only one way - as classes (sets of fields and methods). It makes things even worse, because such representation also defines fixed runtime behaviour of designed language - the only way to manipulate with language concepts defined this way - is to call some method or instantiate some object.
      LOP suggests to do all these things (that we need to do anyway) much easier: we'll have an ability to separate definition of runtime behaviour (by creating generator)from definition of language, and we also can customize the way we edit programs written in this language. We also should be able to reuse or extend languages, previously designed in LOP.
  12. interesting perspective[ Go to top ]

    Yes, it is not so easy task to design new domain-specific language. But that is exactly what we do every time when designing a new class library - in this case our task is even more difficult, because we design a language, by representing its concepts by only one way - as classes (sets of fields and methods). It makes things even worse, because such representation also defines fixed runtime behaviour of designed language - the only way to manipulate with language concepts defined this way - is to call some method or instantiate some object.  LOP suggests to do all these things (that we need to do anyway) much easier: we'll have an ability to separate definition of runtime behaviour (by creating generator)from definition of language, and we also can customize the way we edit programs written in this language. We also should be able to reuse or extend languages, previously designed in LOP.

    In the proper hands, I agree that creating libraries is similar to designing languages. Where it breaks down in my mind is when a domain isn't sufficiently understood. This is my own bias, but having worked in crazy startup environments where the domain was in constant flux and everyone is grasping for an understanding, it's easier to write small libraries. Trying to distill an emerging domain into a DSL feels rather daunting to me.

    Back in 99/2000 I worked at a startup building a wireless portal. Back then, hdml, wml, and WAP were still going through refinement. So many pieces were in flux that we were building new prototypes every week and rolling out alpha version twice a month. In those kind of conditions, the overhead of a DSL would be counter productive in my mind. It was hard enough to define personalization within a mobile context that trying to build a high level abstraction would have been practically impossible.
  13. re: interesting perspective[ Go to top ]

    The detail is that there are two levels when it comes to designing pretty much anything.

    The first is the technical requirements, and then there's the usability.

    A big rock can be an adequate hammer, and may well meet the technical spec, but usability is somewhat in question.

    Classes and libraries are the same way, but many times we're just happy that the technical spec is met and work around the unusable parts.

    Much of this can be handled simply by refactoring, but sometimes you need to get beyond the library, and thus we end up with supporting "Util" classes to manage this for us.

    All of these changes are designed to abstract away as much as possible. When we can concentrate at this higher level, we are more productive.

    DSL's let us abstract not just functionality, but syntax and semantics.

    But all of these designs need to be organic so that they can survive first contact with the users of the specification. Use of the design brings out a lot more issues than what may have been considered in the design.

    A classic example of this happened to me. We have our own DSL that lets us code against our central data model. For this it works really well. But even though I wrote it, I never USED it (it was a spec plopped on my desk -- "Implement this!").

    Then came the day when I had to actually use it, and implement a bunch of business logic.

    Because of the nature of the logic spec, and the interrelationships among the different aspects, I wrote YET ANOTHER DSL that compiled down into the first one, but it maintained higher level relationships that the original one was never designed to. This allowed me to work at an even higher level, and because of this, I was able to respond to the changes in the logic spec (of which there were legion) very quickly, and more important, accurately.

    This is the kind of environment where the more extensible languages fair much better by not having to rely upon outside tools to generate these DSLs, but rather leveraging the language itself to make tiny DSLs on the fly as the application is built.
  14. Reading your comments, it reminds me of RuleML and its sub-languages. In the case of something like RuleML, the founders are well grounded in formal logic and have a clear understanding first order logic, monotonic/non-monotonic, weak negation, strong negation and other language related principles. Who knows, maybe LOP can work for some cases. I'm open to learning/unlearning how to write/develop software.
  15. Tim Sheard[ Go to top ]

    Peter, why not introduce yourself to the work of Tim Sheard?
    http://www.cs.pdx.edu/~sheard/
    Start with where he is going with the language Omega. It is being designed for implementing DSLs.
  16. more to read[ Go to top ]

    Peter, why not introduce yourself to the work of Tim Sheard?http://www.cs.pdx.edu/~sheard/Start with where he is going with the language Omega. It is being designed for implementing DSLs.

    thanks for the link. at first I thought maybe Omega was related to Microsoft research's COmega, but after a bit more searching I found http://www.cse.ogi.edu/~sheard/cse506Sp2004/lecturenotes/.

    Man i really need to figure out how to get by on 3hrs sleep instead of 5hrs sleep. that is, if I don't kill myself in the process of filling my two remaining brain cells with additional knowledge. the discussion has been fun.
  17. interesting perspective[ Go to top ]

    In the proper hands, I agree that creating libraries is similar to designing languages. Where it breaks down in my mind is when a domain isn't sufficiently understood. This is my own bias, but having worked in crazy startup environments where the domain was in constant flux and everyone is grasping for an understanding, it's easier to write small libraries. Trying to distill an emerging domain into a DSL feels rather daunting to me.

    Hi Peter,

    You are assuming that creating and modifying languages will have a lot of overhead (let's ignore the design difficulty for a moment). Just imagine for a moment that creating and modifying languages was exactly as difficult as creating and modifying class libraries. Again, ignore the design difficulty aspect (it's easy to mis-design class libraries too).

    If the 'overhead of a DSL' didn't exist (in comparison with class libraries), then would you still have the same opinion that creating DSLs wouldn't be worth it?

    I would argue that if the overhead was identical, then DSLs would be the better choice, because of expressiveness and environment support (editor, debugger, compiler error reporting, etc.). Even a badly designed DSL would be better than a similarly badly designed class library.

    LOP promises to make creating and modifying DSLs easier than creating and modifying class libraries because it applies LOP to itself (you have special languages and editors for creating languages).

    Now, returning to the difficulty of designing good languages. I think this is a valid issue. There is a lot of research and education on OOP. Design, patterns, well-known refactorings, best practices, theory, etc. Many of these things are lacking in LOP. This is the essence of the paradigm shift. People will need to explore and research and experiment to build up a body-of-knowledge about LOP, which will include how to do good language design. But it will happen. It will be similar to the shift from procedural to OOP. I don't think it will be terribly difficult to learn, though, just as OOP isn't terribly difficult to learn.

    I think LOP will be used by every programmer (well, most, let's say), not just 'language designers'. Designing languages will become synonymous with the word 'programming', just as writing classes and methods and class libraries is synonymous with programming today. It will become a standard part of the curriculum; a chapter in an introductory programming textbook; a full course or three in a degree program.

    In this context, going back to your wireless portal example, the work that you did every day -- dealing with flux in the wml/wap space -- you probably used OOP or maybe just procedural techniques to do this work. Adding LOP to the mix would mean that you could have replaced some of that work with a little language or two (or many) to help tame the fluctuation. If some low-level technology changes (like WAP), you don't modify the high-level design, you just modify the transformation from high-level to low-level. If you do any refactoring today, you are already doing these kinds of transformations. I would argue that you would have had less work, and it would have been easier, if you had a tool like MPS and a method like LOP.

    Regards,
    Rob Harwood
  18. interesting perspective[ Go to top ]

    first off, it's my own limitations, so any perceived differences by me could be a lack of sleep :)
    Hi Peter,You are assuming that creating and modifying languages will have a lot of overhead (let's ignore the design difficulty for a moment). Just imagine for a moment that creating and modifying languages was exactly as difficult as creating and modifying class libraries. Again, ignore the design difficulty aspect (it's easy to mis-design class libraries too).If the 'overhead of a DSL' didn't exist (in comparison with class libraries), then would you still have the same opinion that creating DSLs wouldn't be worth it?I would argue that if the overhead was identical, then DSLs would be the better choice, because of expressiveness and environment support (editor, debugger, compiler error reporting, etc.). Even a badly designed DSL would be better than a similarly badly designed class library.LOP promises to make creating and modifying DSLs easier than creating and modifying class libraries because it applies LOP to itself (you have special languages and editors for creating languages).

    If the "overhead of DSL" is just a figment of my imagination, then I would agree following a formalized method such as LOP would be better. The idea of being able to replace the translation to gain better code generation/performance is very appealing to me.

    Now, returning to the difficulty of designing good languages. I think this is a valid issue. There is a lot of research and education on OOP. Design, patterns, well-known refactorings, best practices, theory, etc. Many of these things are lacking in LOP. This is the essence of the paradigm shift. People will need to explore and research and experiment to build up a body-of-knowledge about LOP, which will include how to do good language design. But it will happen. It will be similar to the shift from procedural to OOP. I don't think it will be terribly difficult to learn, though, just as OOP isn't terribly difficult to learn.

    I'm totally biased on the relative "ease" of changing modes of thinking. having spent the last two years trying to teach VB programmers declarative programming ala LISP, it's rather hard. I finally gave up and quit that job. On the other hand, if we think in terms of kids going into universities the next 2 decades than I think there's great hope. Well atleast I have great hope for the future generation. They have less baggage and bad habits to change. I see it as a challenge of changing human nature and not really a technical issue.
    In this context, going back to your wireless portal example, the work that you did every day -- dealing with flux in the wml/wap space -- you probably used OOP or maybe just procedural techniques to do this work. Adding LOP to the mix would mean that you could have replaced some of that work with a little language or two (or many) to help tame the fluctuation. If some low-level technology changes (like WAP), you don't modify the high-level design, you just modify the transformation from high-level to low-level. If you do any refactoring today, you are already doing these kinds of transformations. I would argue that you would have had less work, and it would have been easier, if you had a tool like MPS and a method like LOP.Regards,Rob Harwood

    In the wireless example from past experience, the high level design and business logic was very unstable. Everyone in the industry was struggling to make sense of what a mobile enterprise application means. There are very few successful niches in the mobile application space that I know of. Probably the only one that could be classified as clear success is mobile applications for shipping companies. The wireless game sector is really starting to pick up, but back in 99 there was a lot of confusion. Actually, the developers wanted to write games for phones, but all the managers said there's no money in that. it wasn't until Nokia came out with their game tk in 2000 for phones that any of the wireless companies took it seriously.

    maybe if there was a tool like LOP in 99/00, but it's hard to say if it really would have helped. our team was small, only 5-7 guys for the first 10 months, so spending time to write our own DSL and translator definitely wouldn't have been an option. This is my own bias and lack of experience with LOP, but it's just easier to use what I already know. Especially in a case where I'm given the task of designing a prototype from a foggy description of a potential application. it was kind of liking playing musical chairs in a dark room with blind folds on. I like that people are actively trying to figure out better ways to build software, even if it's overhead that's fine. Well most things are beyond my abilities. I do see some parallels between software factories/DSL and the current trend in the W3C with domain specific markup languages.
  19. interesting perspective[ Go to top ]

    yes, interesting perspective. I was just wondering can we call XML schema language as domain language,the Domain of O/R modelling.

    I say this because a single programming language,XML schema language, can be used for both Object and Relational modelling.
    XML schema language gives enough language constructs to express our Types in Model, in either Object or Relational form , or both...."AND MORE" :)(the last gimmick of marketing)

    Regards
    Surajeet
  20. slight problem I see[ Go to top ]

    From experience, it's not easy for an average programmer to design a language.

    It is true not only for designing languages, but also for designing any class libraries. And what happens here: a few talented developers create such class libraries (collections libraries, web frameworks, IOC containers etc.) and the rest of us just use it. So in case of DSL I suspect same thin happen. There will be good DSLs developed by few professionals and available to download / purchase. And the rest of us will just happily use/extend them.
  21. The link given in this post goes only to the PDF version of article - for printing it. If you want to read it from screen you can go to
    http://www.onboard.jetbrains.com/articles/04/10/lop/
    There is also discussion of this article at http://www.intellij.net/forums/forum.jsp?forum=48
    This article is published in the first release of JetBrains' online magazine "onBoard".
    You can find there also 3 other interesting articles about JetBrains' technologies.
  22. Interesting indeed[ Go to top ]

    That's very interesting indeed. Unfortunatelly not all people needs it (which is oblivious from the discussions in your livejournal) but sometimes it's really makes sense.
    However it's slighly similar to Eclipse EMF (also about language model description and code generation, especially non-textual representation) and even MDA.
    Could you, Sergey, compare your proposal with existing solutions?
  23. More thoughts[ Go to top ]

    Dear Sergey,

    First of all, thank you for a good paper that
    shows up a new perspective and furthers the
    thinking about tools for language design and
    support. I'd like to make a few comments on some
    aspects of the paper.

    Modeling and programming languages should be
    appropriate for the problem they're applied to. I
    believe, we share this idea. One of the benefits
    of OO as I see it was that it started to blur the
    boundaries between language and library
    design. Blaming libraries for their growing
    complexity alone doesn't help. Most of those
    libraries tend to grow complex because they have
    to solve complex problems. Assuming that not all
    library / framework designers are unable to do
    their job right, this goes to show that a domain's
    complexity is always hard to hide. I don't believe
    that this complexity we observe is only introduced
    by the OO paradigm in itself. My suspicion is that
    DSLs develop the same tendency that libraries
    do. In addition to that, libraries written in a
    standard programming / modeling language have one
    important advantage over corresponding DSLs: they
    fit within a programming (meta-)model that has
    tons of support for it: debugging, analysis,
    quality metrics, pre-defined refactoring support
    and many more. A DSL will have to provide similar
    capabilities on its own which has to repeat the
    investments of a large community on a much smaller
    base. This is not to say DSLs won't ever help;
    it's just shifting the tradeoff towards lowering
    the desirable number of different
    languages. Making the creation and evolution of
    modeling / programming languages easy is a great
    thing. However, a proliferation of languages
    causes a new class of problems. Language features
    have to be well-defined, documented and learned by
    people having to use or read them (BTW, just as
    library functionality has). An abundance of
    languages and language features aggrevates these
    problems.

    The need to specify nitty-gritty detail brings one
    of two possible things to the surface: either the
    libraries lack reasonable defaults, or the detail
    actually is required to characterize the
    solution. The former is a library design problem
    that can typically be solved while the latter
    won't go away by designing more abstract DSLs. It
    will just allow for a separation of roles in the
    development process: domain experts may not have
    to be bothered with the details while at a more
    fine-grained level a technical developer will
    still have to provide the details. While I believe
    in the usefulness of this separation, I also know
    that it requires substantial progess in the
    development of tools that help us with
    synchronizing the more abstract models with the
    more detailed models and tracing the mapping and
    refinements between them.

    I believe there is a need for at least three
    different kinds of languages, but languages of
    each of these kinds should be standardized as far
    as possible. Standardization will allow
    communities and vendors to provide reuable models,
    tools and mappings based on these languages. The
    kinds I envison are languages for capturing domain
    knowledge (business process languages, DSLs for
    particular vertical domains), languages for
    technical paradigms (e.g., UML for the OO paradigm
    or OCL for expressions over objects), and
    languages for technical platforms (language for
    J2EE; language for .NET).

    The concepts defined in the Structure Language
    sound pretty much like the MOF core. I'm missing
    some reasoning why a proprietary approach has been
    chosen, given the tremendous existing support for
    MOF (XMI for import/export, JMI as Java language
    binding, to name just a few).

    On the template issue you may want to take a look
    at state-of-the-art generator tools. Even though I
    no longer work for iO, I recommend looking at
    their ArcStyler product and its code generation
    principles. It's underlying architecture (CARAT)
    encodes several years of experience in the
    construction of large-scale reusable and
    customizable generators producing text-based
    artifacts in all kinds of target languages.

    On language extension, you may want to look at the
    MOF concept of package import, combination and
    merge. It allows you to extend and modify existing
    modeling languages in a non-intrusive way,
    providing a good means to reuse language
    definitions.

    Summarizing, I believe there is great value in
    easing the creation and support for programming /
    modeling languages, as have attempted many
    meta-case tools. However, being a big fan of
    IntelliJ IDEA, I'm doubtful that we'll see the
    same amount of sophisticated, meticulous support
    that IDEA provides for Java provided by tools for
    our own, proprietary languages. From experience I
    know how tricky the business of model
    transformation and code generation can be,
    including reasonable tool support that spans a
    flexible set of model transformations.
    Traceability through the mappings becomes an
    issue, not only in debugging but also in
    refactoring. And then add the option to mix-in
    manual refinements in the intermediate languages
    being used. I believe languages have to be
    designed with great care, and there should be at
    most a few if not only one for each domain /
    paradigm / platform. Only then will we be able to
    make our programs long-lived, valuable assets that
    don't go away only because the next version of our
    favorite IDE didn't understand our self-designed
    language. Also, I believe we need standardization
    on the language definition level, as provided for
    example by the MOF standard at least for the
    structural part (we're still missing a standard
    for the definition of language representations).
    I'm glad to see a new twist to the design of the
    programming models we've been used to. Let's
    continue this discussion, so we can learn how we
    can create software more efficiently with higher
    quality, less learning effort and lower
    maintenance costs in the future.

    Best regards,
    -- Axel Uhl
  24. More thoughts[ Go to top ]

    Axel,

    I agree with many of your comments. I think one reason why MPS didn't embrace the MOF standard is because it is an example of premature standardization. What if MPS needs some feature that MOF can't provide? We won't know this until MPS matures.

    MPS is a project which explores LOP. I think it is fair to say that LOP is a 'new' thing, which has not been adequately explored. There are a lot of new ideas popping up daily. Before we standardize, we need to see just what's needed. You can't draw a map of someplace you've never been. Well, you *could* draw a map, but don't be surprised when you find out that it's wrong.

    When the vision becomes clear, and there are a few implementations, then I think it is time to standardize.

    Regards,
    Rob Harwood
  25. Active Libraries and Universal Languages[ Go to top ]

    People interested in what is possible in support of the DSL approach may wish to acquaint themselves with the ideas in Veldhuizen's thesis "Active Libraries and Universal Languages".
    http://www.cs.chalmers.se/~tveldhui/papers/2004/dissertation.pdf
  26. Ewige Wiederkehr[ Go to top ]

    It seems that programming is coming back to its roots. Maybe in a few years LISP will be back in fashion - every nontrivial LISP program is a domain specific language. And with LISP macros, we'd have no need for code generation templates...
  27. Ewige Wiederkehr[ Go to top ]

    It seems that programming is coming back to its roots. Maybe in a few years LISP will be back in fashion - every nontrivial LISP program is a domain specific language. And with LISP macros, we'd have no need for code generation templates...

    Exactly my thoughts. In fact it makes me think if DSL is such a great approach. That is not if it can work at all - it will probably to a degree - but if it is bound to be successful on a larger scale. Lisp could work extremely well for you as well and its overall impact was fairly limited. In fact thinking of it could probably do much much more powerful things, like modifying the language itself during execution....
  28. Ewige Wiederkehr[ Go to top ]

    It seems that programming is coming back to its roots. Maybe in a few years LISP will be back in fashion - every nontrivial LISP program is a domain specific language. And with LISP macros, we'd have no need for code generation templates...
    Exactly my thoughts. In fact it makes me think if DSL is such a great approach. That is not if it can work at all - it will probably to a degree - but if it is bound to be successful on a larger scale. Lisp could work extremely well for you as well and its overall impact was fairly limited. In fact thinking of it could probably do much much more powerful things, like modifying the language itself during execution....

    Many of the mechanisms required to make Language-Driven Development/Language-Oriented Programming are well established both in research and real languages such as Lisp and Smalltalk. The bigger challenge is to make these mechanisms palatable to everyday users' such that the paradigms become mainstream.

    Just a few thoughts:

    While there is clearly cost to developing whole DSLs from scratch, it is unlikely that for many purposes you would start from scratch. Instead an existing language might be "customised", this might be defining a new concrete syntax for that language, or adding a new semantic construct and appropriate syntaxes (when is a language a new language or a dialect of an existing language?!).

    The price of a DSL is a loss of generality. The better it addresses specific requirement (the closer the abstractions of the language map to those in the requirements domain) the less appropriate it is for requirements outside of that specific domain. Therefore in order to develop a non-trivial system, with a diverse number of requirements across horizontal and vertical domains, many DSLs would be required. The issue then is not only defining the DSLs but also integrating the DSLs so that they make collective "sense".

    I encourage anyone interested in this area to take a look at the book "Applied Metamodelling: A Foundation for Language-Driven Development" which I co-authored.

    James
  29. Ewige Wiederkehr[ Go to top ]

    The book I mentioned can be downloaded from http://www.xactium.com.
    It seems that programming is coming back to its roots. Maybe in a few years LISP will be back in fashion - every nontrivial LISP program is a domain specific language. And with LISP macros, we'd have no need for code generation templates...
    Exactly my thoughts. In fact it makes me think if DSL is such a great approach. That is not if it can work at all - it will probably to a degree - but if it is bound to be successful on a larger scale. Lisp could work extremely well for you as well and its overall impact was fairly limited. In fact thinking of it could probably do much much more powerful things, like modifying the language itself during execution....
    Many of the mechanisms required to make Language-Driven Development/Language-Oriented Programming are well established both in research and real languages such as Lisp and Smalltalk. The bigger challenge is to make these mechanisms palatable to everyday users' such that the paradigms become mainstream.Just a few thoughts:While there is clearly cost to developing whole DSLs from scratch, it is unlikely that for many purposes you would start from scratch. Instead an existing language might be "customised", this might be defining a new concrete syntax for that language, or adding a new semantic construct and appropriate syntaxes (when is a language a new language or a dialect of an existing language?!).The price of a DSL is a loss of generality. The better it addresses specific requirement (the closer the abstractions of the language map to those in the requirements domain) the less appropriate it is for requirements outside of that specific domain. Therefore in order to develop a non-trivial system, with a diverse number of requirements across horizontal and vertical domains, many DSLs would be required. The issue then is not only defining the DSLs but also integrating the DSLs so that they make collective "sense".I encourage anyone interested in this area to take a look at the book "Applied Metamodelling: A Foundation for Language-Driven Development" which I co-authored.James
  30. Reminds me of video game engines[ Go to top ]

    The practice is quite common; I would say a high percentage, perhaps the majority of video games written today involve writing the language(s) (i.e. "game engine") and then writing the game in that language.

    Taking the idea to its logical conclusion, leveraging really clever tool support (on the level that Jetbrains is capable of) would certainly be a new wrinkle in that process. I agree with you 100% on the notion of abandoning text as a canonical representation; many of us know that is long, long overdue.

    Obviously we already have high-level, even non-text-centric systems for solving many problems, but I can imagine this making the competition between high level languages richer, and extending coverage to underserved problem areas. And theoretically with better interoperation...

    For LOP developers starting from scratch, I could say something general like "I think this will work well, but only for projects of a certain size," but then we have the more fundamental questions.

    You are always going to have that tension between expressiveness and concision versus ambiguity and informality, to the point of unpredictability. A DSL-framework needs to be flexible to be effective, but flexibility makes each DSL more difficult to write, learn, or reuse. Perhaps there is a good gain a solid middle-ground.

    Ultimately, though, you will hit a wall with DSLs well before the "ultra-high-level" holy grail you allude to, because adding layers can only hide complexity, not reduce it. Your ability to almost effortlessly communicate a design to another person (actually, another software developer) is based on the listeners ability to _independently synthesize the solution_; your communication only represents "hints" along the way.

    If you want "such domain-specific representations [to be] the program," you will have to teach the computer the same trick. Which is not to say that the exercise of LOP is bad, in fact I expect great things from a spirited and original implementation of the idea. I look forward to hearing more.
  31. Reminds me of video game engines[ Go to top ]

    Ultimately, though, you will hit a wall with DSLs well before the "ultra-high-level" holy grail you allude to, because adding layers can only hide complexity, not reduce it.
    The Veldhuizen reference above is, amongst other things, about the generation of optimal programs. An old dream made possible through his active library concept. The complexity is automatically handled for you. The level of the DSL can be a high as you like, with no abstraction penalty paid.
  32. Reminds me of video game engines[ Go to top ]

    Emphasizing - the generated programs are not merely proven correct, they are proven optimal.
  33. Reminds me of video game engines[ Go to top ]

    Ultimately, though, you will hit a wall with DSLs well before the "ultra-high-level" holy grail you allude to, because adding layers can only hide complexity, not reduce it.
    The Veldhuizen reference above is, amongst other things, about the generation of optimal programs. An old dream made possible through his active library concept. The complexity is automatically handled for you. The level of the DSL can be a high as you like, with no abstraction penalty paid.

    Your reference is fascinating. I apologize for my vague metaphor. If I understand you, you are referring to runtime efficiency, where I am talking about developer efficiency.

    Unless I have totally misunderstood the purpose of the Active Library, it is not a radical advance in the field of AI; it does not foster some form of natural language programming, which is basically what "ultra-high-level" languages are in this context.

    LOP could never give you the tools to describe your problems to the computer the way you would to a person, despite an implied claim to the contrary (excepting cases where people already communicate in perfect formalisms of course).
  34. Reminds me of video game engines[ Go to top ]

    If I understand you, you are referring to runtime efficiency, where I am talking about developer efficiency.
    The automatic generation of optimal code is very much in the interests of developer efficiency.
    Unless I have totally misunderstood the purpose of the Active Library,
    You do, but larger than that, you misunderstand the purpose of DSLs.
    it is not a radical advance in the field of AI;
    It has nothing to do with AI (and neither do DSLs). It has to do with program logic and proof.
    it does not foster some form of natural language programming,

    DSLs aren't particularly about natural language. Thinking they are seems to be the basis of your misunderstanding.
    which is basically what "ultra-high-level" languages are in this context.
    False. There is no particular connection between ultra-high-level language and natural language.







     LOP could never give you the tools to describe your problems to the computer the way you would to a person, despite an implied claim to the contrary (excepting cases where people already communicate in perfect formalisms of course).
  35. Reminds me of video game engines[ Go to top ]

    It has nothing to do with AI (and neither do DSLs). It has to do with program logic and proof.

    my knowledge of proof systems is very limited, but I thought there's still a lot of debate in the academic world about proof vs theoroms from the perspective of mathematics and implementation. In fact, there was a rather active debate in RDF rules group on the topic. Some of the people were in favor of RDF's monotonic approach, because it was "provable" in their minds. In reality, that wasn't the case and the RDF draft was revised to exclude a claim that monotonic approach is more "provable" than non-monotonic and first order logic.

    Is my understanding/interpretation correct? If the goal is to design DSL's that are provable, I think that would be aiming far too high. I would argue that only in very narrow domains and cases would a thorough proof be possible. I'm not an expert, so there's likely confusion on my part about writing proof systems.
  36. Reminds me of video game engines[ Go to top ]

    If the goal is to design DSL's that are provable, I think that would be aiming far too high. I would argue that only in very narrow domains and cases would a thorough proof be possible. I'm not an expert, so there's likely confusion on my part about writing proof systems.
    The key point there is that there is a difference between the properties of the DSL and the proofs relating to its implementation (e.g. proof of correctness, proof of optimality). The research into DSL implementation does not seek to limit the DSLs that can be implemented. The approach to proofs about DSL implementation is applicable to all possible DSLs. Limiting DSLs to monotonic logic languages is not a goal.
  37. guess I'll have to try when it's released[ Go to top ]

    The key point there is that there is a difference between the properties of the DSL and the proofs relating to its implementation (e.g. proof of correctness, proof of optimality). The research into DSL implementation does not seek to limit the DSLs that can be implemented. The approach to proofs about DSL implementation is applicable to all possible DSLs. Limiting DSLs to monotonic logic languages is not a goal.

    gotta say I'm out of my league here. I look forward to trying it when the first release comes out. On a related note, how pluggable will the API be? I ask because I can think of a couple scenarios. Say someone writes a high level DSL for regulator compliance. Better yet, say the SEC defines one and all trading systems have to translate from that DSL to executable code.
  38. guess I'll have to try when it's released[ Go to top ]

    On a related note, how pluggable will the API be?
    With Sheard's Omega effort, the target language for generation can be Java. With Veldhuizen's universal language effort, you could call that universal language from Java, though I suspect you would write the whole application or suite of applications in the universal language.
    I ask because I can think of a couple scenarios. Say someone writes a high level DSL for regulator compliance.
    Better yet, say the SEC defines one and all trading systems have to translate from that DSL to executable code.
    Interesting. The application area you mention could be very switched on (can't see it as likely to happen though). One point - the semantics of the translation (i.e. the language) would be provided to all parties so that there was no work to do in that regard. Don't forget that unlike XML, DSLs aren't just syntax.
  39. guess I'll have to try when it's released[ Go to top ]

    I ask because I can think of a couple scenarios. Say someone writes a high level DSL for regulator compliance. Better yet, say the SEC defines one and all trading systems have to translate from that DSL to executable code.
    Interesting. The application area you mention could be very switched on (can't see it as likely to happen though). One point - the semantics of the translation (i.e. the language) would be provided to all parties so that there was no work to do in that regard. Don't forget that unlike XML, DSLs aren't just syntax.

    that's correct. for there to be a standard SEC DSL for regulatory compliance, it would have to describe 3 parts: a schema model representing the critical concepts, a generic rule structure for writing rules and a set of predefined actions for process flow.

    There are rules in 2A7 which say something like, "if 3/4 of the securities of an account exceed 15%, an board meeting must be schedule and the board must meet within 7 days."

    obviously, without a model translation would be much harder. Some actions like "schedule a board meeting" are real requirements and it must be logged. The SEC expects firms to file reports and show they fixed any compliance violations. In this specific case, each firm already has a compliance system they wrote in house or license from a third party. The compliance system wouldn't be able to prove to the SEC their system meets the requirements, if the DSL didn't have all three components. Well that's not really true. Today, it's done manually and takes a lot of human effort to make sure violations are logged and reported correctly.
  40. Reminds me of video game engines[ Go to top ]

    Is my understanding/interpretation correct? If the goal is to design DSL's that are provable, I think that would be aiming far too high. I would argue that only in very narrow domains and cases would a thorough proof be possible. I'm not an expert, so there's likely confusion on my part about writing proof systems.

    I think there will be some cross-paradigm confusion. For example, it appears that some people are talking about DSLs from one perspective, with a certain amount of baggage, and others are talking about them from a different perspective, with different baggage.

    DSLs are a tool that can be used in many different ways. There is no single goal or pupose for using DSLs. To be more specific, in LOP, the primary goal of using DSLs is to increase programmer productivity, not for producing efficient runtime implementations or for proving programs correct. Sure, we might take advantage of that, but it's not the main goal. In other contexts, efficiency or correctness might be the primary goal, but not in LOP.
  41. I'm totally lost then[ Go to top ]

    Is my understanding/interpretation correct? If the goal is to design DSL's that are provable, I think that would be aiming far too high. I would argue that only in very narrow domains and cases would a thorough proof be possible. I'm not an expert, so there's likely confusion on my part about writing proof systems.
    I think there will be some cross-paradigm confusion. For example, it appears that some people are talking about DSLs from one perspective, with a certain amount of baggage, and others are talking about them from a different perspective, with different baggage.DSLs are a tool that can be used in many different ways. There is no single goal or pupose for using DSLs. To be more specific, in LOP, the primary goal of using DSLs is to increase programmer productivity, not for producing efficient runtime implementations or for proving programs correct. Sure, we might take advantage of that, but it's not the main goal. In other contexts, efficiency or correctness might be the primary goal, but not in LOP.

    for the last 4 years I've been working with RuleML and other forms of DSL. What I see first hand is that often developers are more confused by DSL than by low level languages. If we consider Schema as a type of DSL, the problems that I see is this.

    When all the developers are in the room with the project manager, some people will discuss their particular component in terms of an abstract domain. Other developers think at the lower level and see it as tables or classes. Problems like these where one person is talking about the architecture at a high level and someone else is talking about technical limitations, it results in more confusion than clarity.

    This isn't to say LOP won't work. Just that I see human problems getting in the way of progress. I use LISP and find it rather elegant, but many people disagree and think it's aweful. I'm definitely bias here, but the solution to many development problems is learning to communicate more effectively. From my limited experience over the last 10 years, poor code quality on large projects is often the result of bad communication. It's great to have nice tools, but I'll gladly trade it for open and constructive communication between the developers and business guys any day.
  42. I'm totally lost then[ Go to top ]

    My hunch is that the DSLs you've experienced had confusing or verbose syntax, as well as unobvious semantics. For example, you mentioned RuleML. Here's some RuleML:

    <imp>
      <_head>
        <atom>
          <_opr><rel>discount</rel></_opr>
          <var>customer</var>
          <var>product</var>
          <ind>5.0 percent</ind>
        </atom>
      </_head>
      <_body>
        <and>
          <atom>
            <_opr><rel>premium</rel></_opr>
            <var>customer</var>
          </atom>
          <atom>
            <_opr><rel>regular</rel></_opr>
            <var>product</var>
          </atom>
        </and>
      </_body>
    </imp>

    What the heck does that do? I have no idea except that it has something to do with a discount, but the syntax is so confusing that all meaning is lost unless you know the syntax. What is 'imp'? Why is there an underscore at the beginning of '_body'? What does <var>customer</var> relate to?

    This is a perfect example of why we need MPS. On the surface of it, I would say that RuleML is a lame DSL. Maybe the implementation is great and once you understand it, it is very powerful; I don't know, I've never used it. However, the use of XML as a concrete syntax kills any power the language has, from the perspective of a newcomer to the language.

    In MPS, the above code would have a simple syntax something like this (but better, since this is merely text):

    Rule: Discount regular products for premium customers by 5.0 percent.

    The variable parts and boiler-plate parts would be syntax-highlighted like in IDEA. The editor would then allow you to find all usages of the rule (if that makes sense), jump to the definition of 'customer', find all usages of 'product', extract '5.0 percent' into a constant, or parameterize the 5.0 into a parameter, etc. No underscores, angle brackets, or any of that crap which is only there to help a parser (or due to HTML legacy problems like <head> vs. <_head>).

    Heck, if you check out the comment in the code I stole that example from (http://www.ruleml.org/exa/0.8/discount.ruleml), it basically says the same thing:

    <!-- Imp 1
    The discount for a customer buying a product is 5.0 percent
    if the customer is premium and the product is regular.

    Remember in the article, the part about having to map your conceptual model into a complicated language? What this example is telling you is "Here's an easy concept like 'The discount is ...', and here's how you map it into RuleML: <blech><_snort><puke></puke></_snort></blech>." Why not make the easy concept easy to program? That is the kind of productivity-enhancing DSL I'm talking about.
  43. That's because Rule systems aren't simple[ Go to top ]

    I have to say you're not the first person to ask why certain nodes are prefixed with an "_" underscore in RuleML. The story is rather long, but it is there for a reason. It's not something that can be answered in a simple post, but I'll just say it is influenced by case base reasoning (CBR) and rule chaining.

    The field of knowledge base and case base reasoning has certain core concepts like lefthand side, righthand side, truth maintenance, implications and inference.

    Future versions of RuleML may get rid of the "_" underscore prefix. In terms of structure, RuleML is influenced by Lisp and inherits many of the grammatical structures from it. That's just the tip of the iceberg. In expert systems like CLIPS, OPS5, JRules, JESS, and blaze advisor, rules can be chained in forward, backward and bi-directional ways.

    the problem with your simple MPS example, it's actually not sufficient. Many of the existing systems handle it similar to what you described, but it ends up being more work than simply encoding the rule in a low level language. I won't bother going into that topic either, since it's rather long and ugly. The founders of RuleML each have 20yrs of experience, so the design and structure is sound.

    This is why I ask how applicable LOP is to these classes of problems. It really isn't as simple as representing class objects as a DSL. The real power of ruleml is you can easily use XSLT to translate rules to CLIPS, Sql or any other format.
  44. That's because Rule systems aren't simple[ Go to top ]

    I guess you'll just have to wait and see. I don't agree with any of your points, but I suspect that it's because we are talking from different paradigms. You believe that rule systems are complicated because they are complicated in mainstream programming. But a rule is just a rule in LOP, and most of the 'complexity' you are describing is what Fred Brooks calls Accidental Difficulty.

    Examples of our paradigm communication difficulties:

    To you, the underscore has a reason for being. To me, it has none. The underscore is not due to some esoteric reason like "it is influenced by case base reasoning (CBR) and rule chaining." It's a text-based syntax kludge pure and simple.

    You say that the MPS example is not sufficient. Technically, you are right because, as I admitted, it is written in text, and not in MPS. However this isn't much of a problem in my view because if it was written using MPS, it would be sufficient (i.e. there would be a syntax for writing rules the way I wrote it).

    You say that "the real power of ruleml is you can easily use XSLT to translate rules to CLIPS, Sql or any other format." In my view, this is actually a weakness of RuleML, because in MPS, I would have special editor support for translating rules into CLIPS, SQL, or any other language, and I definitely wouldn't want to use XSLT, which is another lame text-based DSL that has no editor support and is awful to read and program in.
  45. That's because Rule systems aren't simple[ Go to top ]

    I guess you'll just have to wait and see. I don't agree with any of your points, but I suspect that it's because we are talking from different paradigms. You believe that rule systems are complicated because they are complicated in mainstream programming. But a rule is just a rule in LOP, and most of the 'complexity' you are describing is what Fred Brooks calls Accidental Difficulty.

    not quite. the reason it's complex is because the regulations are written by lawyers. They are purposely written to allow multiple interpretations. the laws and rules derived from them are complex because the requirement was convoluted to begin with. if you consider lawyers an accident of nature, then absolutely it's accidental.
    Examples of our paradigm communication difficulties:To you, the underscore has a reason for being. To me, it has none. The underscore is not due to some esoteric reason like "it is influenced by case base reasoning (CBR) and rule chaining." It's a text-based syntax kludge pure and simple.You say that the MPS example is not sufficient. Technically, you are right because, as I admitted, it is written in text, and not in MPS. However this isn't much of a problem in my view because if it was written using MPS, it would be sufficient (i.e. there would be a syntax for writing rules the way I wrote it).You say that "the real power of ruleml is you can easily use XSLT to translate rules to CLIPS, Sql or any other format." In my view, this is actually a weakness of RuleML, because in MPS, I would have special editor support for translating rules into CLIPS, SQL, or any other language, and I definitely wouldn't want to use XSLT, which is another lame text-based DSL that has no editor support and is awful to read and program in.

    I'm definitely not getting MPS. I don't mean to claim MPS or LOP are flawed. I'm simply attempting to express my own doubts and thoughts. Granted I'm doing a terrible job of understanding what MPS has to provide, but I am trying. RuleML is far from perfect, but it is better than the other rule markup languages out there. That's my opinion and not a fact.

    You may consider it a kludge, but those in the field do understand it. Many people disagree with RuleML, so it's not like there's universal acceptance of the syntax. For better or worse, there are historical reasons for the underscore. Just because you're not in the field of CBR/KBR, doesn't mean some really smart people didn't spend 20 years thinking about these issues. The founders of RuleML are respected in the field and are considered domain experts.

    Let me give you an example of how convoluted and fuzzy regulations can be. A fixed income security like a bond can have one or more entities backing it. In some cases, the regulation limits the exposure of a money market fund to any single entity to a set limit, like 15%. The problem is the entity backing the bond can be considered both partial and layered backers. Some rules make it so you have to calculate using both methods. In other words, bond A may have 2 agencies backing it 100%, while 3 others may back it 33/33/34. The first two entities back the bond 100% in case the other cannot. If the second entity cannot back it 100% for some reason, the remaining three entities may back it up to the full percentage, or they may alternatively divide it 25% each way. It get more complex than this, but I'm not a lawyer and honestly don't understand all the subtleties.

    How the in world would MDA, LOP or MPS model this type of relationship? I've never read Fred Brooks book, but the complexity is due to the actual law. Not due to technical design. Most companies consult law firms specializing in regulatory law and pick one way of implementing it. But across the industry as a whole, it varies dramatically. Even if I choose to create my own DSL, it is not portable to other compliance systems and most likely would not match what the SEC uses.

    So in this specific case, I just think it's a human problem and not a technology problem. I would love for LOP, MDA or MPS to solve this problem. Perhaps I'll never understand it, but that has nothing to do with how effective MPS can be in practice. I apologize if my comment appeared dismissive, that wasn't my intent. It just looks similar to approaches that I am aware of in compliance. In all likelihood, it could just be the same principle poorly applied. I don't have any answers honestly, but I do enjoy the discussion and exploring the topic. thank you for taking time to respond.
  46. just wanted to know your opinion[ Go to top ]

    You mentioned CLIPS, OPS5, JRules, JESS, and Blaze. What about Rulesharp, will it be a good engine on your opinion. can be found at www.rulesharp.com You also mentioned XSLT translation from RuleML to SQL. Rulesharp uses SQL. Do you have any reference to this translation? thank you, randy
  47. Reminds me of video game engines[ Go to top ]

    To be more specific, in LOP, the primary goal of using DSLs is to increase programmer productivity, not for producing efficient runtime implementations or for proving programs correct. Sure, we might take advantage of that, but it's not the main goal. In other contexts, efficiency or correctness might be the primary goal, but not in LOP.
    The benefits of the DSL/LOP appraoch have long been known. The two big stumbling blocks for DSL/LOP have been the cost of developing the DSL implementation (you are after all developing a compiler for the DSL), and the execution costs of the compiled program. In the context of the DSL/LOP approach being successful, these are the primary concerns. If Jetbrains has a story on these concerns beyond editor support, it would do well to tell this story. If Jetbrains doesn't have a story on these concerns, it would do well to get one in a hurry. I suggest drawing on the work of others. The marketplace will be quite merciless on you if you attempt to avoid addressing these concerns. Also, the DSL/LOP approach's first very public foray will end in that approach's repute being decremented.
  48. Reminds me of video game engines[ Go to top ]

    The benefits of the DSL/LOP appraoch have long been known. The two big stumbling blocks for DSL/LOP have been the cost of developing the DSL implementation (you are after all developing a compiler for the DSL), and the execution costs of the compiled program. In the context of the DSL/LOP approach being successful, these are the primary concerns. If Jetbrains has a story on these concerns beyond editor support, it would do well to tell this story. If Jetbrains doesn't have a story on these concerns, it would do well to get one in a hurry. I suggest drawing on the work of others. The marketplace will be quite merciless on you if you attempt to avoid addressing these concerns. Also, the DSL/LOP approach's first very public foray will end in that approach's repute being decremented.

    you're questions are stated far better than my sorry attempts. I went back and looked at the paper again. If I understand the screen shots correctly, the DSL is backed by concrete implementation.

    So would it be fair to say that one way of looking at LOP is it changes how one designs an application. A programming using Java today would likely think of interfaces, classes and methods. Would the equivalent in LOP be closer to writing a sentence using the DSL? by that I mean, using the LOP editor to chain together semantic elements to form a sentence. though not really a sentence in the natural language sense. A sentence in the sense of a complete phrase based on the structure of the DSL.

    Is that a fair summary, or am I totally off?
  49. Reminds me of video game engines[ Go to top ]


    If I understand the screen shots correctly, the DSL is backed by concrete implementation.
    The point being ... ?
    So would it be fair to say that one way of looking at LOP is it changes how one designs an application. A programming using Java today would likely think of interfaces, classes and methods. Would the equivalent in LOP be closer to writing a sentence using the DSL? by that I mean, using the LOP editor to chain together semantic elements to form a sentence. though not really a sentence in the natural language sense. A sentence in the sense of a complete phrase based on the structure of the DSL.Is that a fair summary, or am I totally off?
    You seem to have it there (though I would replace the word "semantic" in the above context with "language").
    A point to be made perhaps is that there still needs to be machinery to execute a sentence in the language, and that's where it get interesting. I have to say that Jetbrains tranformation rules DSL to that appears quite naive from a performance perspective. That that see that the DSL approach is the way of the future means they have some bright people, and I look to those bright people to fix the hole in their argument. The easiest and possibly best fix would be for them to acknowledge that they do not have the whole DSL/LOP story here.

    Incidentally Peter, a popular use of DSLs will be to define the interface to an application. The application is a compiler for the DSL. Where today we receive an XML document at a web service, tomorrow that XML document will conform to a DSL and be executed as a sentence in that language. The application will be built using tools used to build a compiler. The parts of the application will be the parts of a compiler. Research is underway to find truly modular ways of building compilers, so that new language features may be added without altering the rest of the compiler.
  50. Reminds me of video game engines[ Go to top ]

    If I understand the screen shots correctly, the DSL is backed by concrete implementation.
    The point being ... ?

    Just wanted to make sure I have a few of the basic facts clear for myself.
    You seem to have it there (though I would replace the word "semantic" in the above context with "language"). A point to be made perhaps is that there still needs to be machinery to execute a sentence in the language, and that's where it get interesting. I have to say that Jetbrains tranformation rules DSL to that appears quite naive from a performance perspective. That that see that the DSL approach is the way of the future means they have some bright people, and I look to those bright people to fix the hole in their argument. The easiest and possibly best fix would be for them to acknowledge that they do not have the whole DSL/LOP story here.Incidentally Peter, a popular use of DSLs will be to define the interface to an application. The application is a compiler for the DSL. Where today we receive an XML document at a web service, tomorrow that XML document will conform to a DSL and be executed as a sentence in that language. The application will be built using tools used to build a compiler. The parts of the application will be the parts of a compiler. Research is underway to find truly modular ways of building compilers, so that new language features may be added without altering the rest of the compiler.

    I'm still not convinced I have any kind of understanding of LOP. For the sake of exploration, I'll pretend I understand the basic idea, minus all the interesting and important details.

    So according to the paper, the structure of the DSL roughly encapsulates the target domain. The elements of the DSL are backed by concrete implementation. Writing a new application can then be done with a LOP editor using just the DSL.

    Instead of a developer writing a shopping cart from beginning to end, he could instead model the various functions of a shopping cart. In today's Java world, I might create an object named com.mydomain.web.shopping.Cart. The class may have some generic methods.

    public interface Cart {

    int getItemCount();
    int getItem(int);
    boolean removeItem(int);
    boolean addItem(com.mydomain.products.Item);
    boolean removeItem(com.mydomain.products.Item);
    ... etc ...
    }

    What would the equivalent LOP embodiment look like? It would be great if JetBrains can take a concrete example and explain it in greater detail.
  51. Reminds me of video game engines[ Go to top ]

    So according to the paper, the structure of the DSL roughly encapsulates the target domain.
    The DSL precisely encapsulates the domain of interest.
    The elements of the DSL are backed by concrete implementation.
    The DSL is translatable to a language for which an execution implementation exists.
    Writing a new application can then be done with a LOP editor using just the DSL.
    No. The DSL implementation (translation to an executable language) must also be created. However, once this implementation is created, it is just a matter of using the editor for the DSL to create expressions in that DSL, and smart editor support for this could be quite sweet.
  52. Reminds me of video game engines[ Go to top ]

    First thing, thanks for responding.
    So according to the paper, the structure of the DSL roughly encapsulates the target domain.
    The DSL precisely encapsulates the domain of interest.

    Assuming the given domain can be precisely encapsulated in a DSL I would agree with that. Domains that cannot be precisely encapsulated is another story. I look forward to JetBrains' ideas on how to handle these kinds of cases.
    Writing a new application can then be done with a LOP editor using just the DSL.
    No. The DSL implementation (translation to an executable language) must also be created. However, once this implementation is created, it is just a matter of using the editor for the DSL to create expressions in that DSL, and smart editor support for this could be quite sweet.

    So this is where I kind of get lost.

    1. the screenshot is meant to show an intermediate language, or the product of the translation component?
    2. if there is an intermediate language, does it mean the higher level DSL is a subset of what can be expressed in the intermediate language?
    3. if there is an intermediate language, that would imply translators only work with the intermediate language to produce executable code. Is that method by which a single DSL may be translated to one or more executable languages like C/C++/Java/C#/Perl?
    4. would there be cases where the DSL is a superset of what the intermediate language can express?
    5. If there is no intermediate language and the screenshot is meant to represent concrete code in the target language, how would one provide flexibility across systems or languages?
    6. How would LOP handle cases where company X wants to create a high level DSL which end users can use directory to create custom components, like macros for example?
  53. Reminds me of video game engines[ Go to top ]

    Assuming the given domain can be precisely encapsulated in a DSL I would agree with that. Domains that cannot be precisely encapsulated is another story.
    If the domain has a precise model, a precise DSL can be had. If the domain does not have a precise model, there is much that cannot be done in any approach, DSL or otherwise.
    if there is an intermediate language, does it mean the higher level DSL is a subset of what can be expressed in the intermediate language?
    There are three languages, the DSL, the target language and the metaprogramming language that takes expressions in the DSL and produces expressions in the target language. The target language is more general purpose than the DSL.
    Sheard's Omega is a metaprogramming language. The DSL and target language are of your choosing.
    I will look at your question a bit further later - have to run now.
  54. Reminds me of video game engines[ Go to top ]

    So we have a metaprogramming language program that takes as input a DSL program and outputs a target language program. If you like, the three languages are the language input to the compiler (the DSL), the language in which the compiler is written (the metaprogramming language), and the language which is output by the compiler (the target language).

    Because the DSL approach requires that a compiler be created by application developers, a better approach to assuring correctness than mere testing is especially desirable. The DSL compilers will not see extensive testing by an army of intending users. It is highly desirable that a DSL compiler be proven to generate target code that is equivalent to the input DSL expression. As a DSL compiler is likely to get less developer time on optimising the code it generates, it is similarly desirable that the compiler provably generate optimal code. The central relevance of proof to the DSL approach can perhaps now be seen.
  55. Reminds me of video game engines[ Go to top ]

    So we have a metaprogramming language program that takes as input a DSL program and outputs a target language program. If you like, the three languages are the language input to the compiler (the DSL), the language in which the compiler is written (the metaprogramming language), and the language which is output by the compiler (the target language).

    thanks for taking time to respond. So as a developer using LOP tools, ideally a programmer would write compilers using the metaprogramming language. That sounds good to me, since I enjoy writing compilers. I just noticed a typo in my previous post. "directory" should have been "directly".

    So the challenge I see with writing compilers from first hand experience is this.

    1. many programmers do not understand how to write compilers
    2. writing good compilers is hard and takes lot of experience

    Over the last four years, I've been writing rule compilers and these are the kinds of problems I've seen first hand. In my mind the approach is feasible and I would agree it is better. The real catch is finding a good developer for a project to write the compiler. Obviously not everyone on the team would be writing compilers and it probably should be written by someone with real experience.
    Because the DSL approach requires that a compiler be created by application developers, a better approach to assuring correctness than mere testing is especially desirable. The DSL compilers will not see extensive testing by an army of intending users. It is highly desirable that a DSL compiler be proven to generate target code that is equivalent to the input DSL expression. As a DSL compiler is likely to get less developer time on optimising the code it generates, it is similarly desirable that the compiler provably generate optimal code. The central relevance of proof to the DSL approach can perhaps now be seen.

    the primary challenge I see first hand writing rule compilers is this.

    1. the ease of understanding the DSL
    2. the ease of using DSL
    3. the flexibility of the DSL and the ease of extending it
    4. insuring extensions to the DSL do not create structural/logical flaws

    many people who work in the rule engine industry translate from some higher level language (often natural language like) to the rule engine's language, which is then compiled by the rule engine to RETE network. The difficulty is the higher level language. Most end users in these cases are business analysts, who do not think in logical terms. In fact, most of the times they write things that make no sense. by that I mean this. the analyst writes a statement, which conforms to the DSL's structure, but practically it is paradoxical or non-sensical. A common example from compliance is this.

    "if the security's country is equal to US, Canada and France."

    Although the sentence may be structuraly correct, it makes no sense. In 99% of the cases, a security can only have 1 primary issuer. What the user means in this case is this.

    "if the security's country is in list (us, canada, france)"

    these kinds of issues from my experience are not technological. it's mostly user error or misunderstanding.
  56. Reminds me of video game engines[ Go to top ]

    So the challenge I see with writing compilers from first hand experience is this.1. many programmers do not understand how to write compilers2. writing good compilers is hard and takes lot of experience Over the last four years, I've been writing rule compilers and these are the kinds of problems I've seen first hand. In my mind the approach is feasible and I would agree it is better. The real catch is finding a good developer for a project to write the compiler. Obviously not everyone on the team would be writing compilers and it probably should be written by someone with real experience.
    Yes, that has been the problem that has stopped the DSL/LOP approach from becoming popular. The fix is to radically reduce the cost of developing compilers, hence the focus on the work of Sheard and Veldhuizen.
     
    the primary challenge I see first hand writing rule compilers is this.1. the ease of understanding the DSL2. the ease of using DSL3. the flexibility of the DSL
    I would have said these properties of a DSL were largely separate from the challenges of building a DSL compiler, other than the more complex the DSL, possibly the more complex the compiler.
    The difficulty is the higher level language. Most end users in these cases are business analysts, who do not think in logical terms. In fact, most of the times they write things that make no sense. by that I mean this. the analyst writes a statement, which conforms to the DSL's structure, but practically it is paradoxical or non-sensical. A common example from compliance is this."if the security's country is equal to US, Canada and France."Although the sentence may be structuraly correct, it makes no sense. In 99% of the cases, a security can only have 1 primary issuer. What the user means in this case is this."if the security's country is in list (us, canada, france)"these kinds of issues from my experience are not technological. it's mostly user error or misunderstanding.
    That a user can make a statement that makes no sense but passes checking by the compiler has to mean that the semantics of the DSL are insufficiently well defined.
  57. Reminds me of video game engines[ Go to top ]

    The difficulty is the higher level language. Most end users in these cases are business analysts, who do not think in logical terms. In fact, most of the times they write things that make no sense. by that I mean this. the analyst writes a statement, which conforms to the DSL's structure, but practically it is paradoxical or non-sensical. A common example from compliance is this."if the security's country is equal to US, Canada and France." Although the sentence may be structuraly correct, it makes no sense. In 99% of the cases, a security can only have 1 primary issuer. What the user means in this case is this." if the security's country is in list (us, canada, france)" These kinds of issues from my experience are not technological. it's mostly user error or misunderstanding.
    That a user can make a statement that makes no sense but passes checking by the compiler has to mean that the semantics of the DSL are insufficiently well defined.

    I guess that's one way to put it. Another way to look at it is that regulatory compliance is largely driven by law. Law often makes no sense :) If I were to ask a lawyer specializing in financial regulations, they would disagree with me. To them it's not that laws are irrational, it's meant to be open to interpretation.

    That obviously doesn't help from a technology and software development perspective. If the requirements (in this case laws) are not logical, it's impossible to define a DSL so that it is sufficiently well defined. What is worse is regulations actually change frequently, but most companies make changes once or twice a year. What many companies do is spend 8-10 months to update their rule library to reflect the latest changes and roll it out at the end of the year.

    I suppose the definition of "sufficient" is the key here. I've worked with people with 20 years of compliance experience and no one has found an elegant or good solution in the last 30 years :) Then again, I doubt it is even possible based on the 2A7, 1940Act and FSA laws I've read.

    thanks again for take time to reply and explain. I'm beginning to see how LOP can be effective in more focused domains where defining a DSL sufficiently is feasible. For the whacky world of regulations, nothing helps. Just about every methodology fails to make it easier. the ony way to make building compliance systems easier to hire an expert with 10 years of experience.
  58. Reminds me of video game engines[ Go to top ]

    the ony way to make building compliance systems easier to hire an expert with 10 years of experience.
    Sure, but such an expert comes up with an unambiguous model. That's the point of getting an expert. Software development under any approach is constrained to what can be unambigously modeled.
  59. Reminds me of video game engines[ Go to top ]

    the ony way to make building compliance systems easier to hire an expert with 10 years of experience.
    Sure, but such an expert comes up with an unambiguous model. That's the point of getting an expert. Software development under any approach is constrained to what can be unambigously modeled.

    yeah, the hard part is figuring what is noise and what is feasible. Though having worked with a guy that built compliance systems for 20 years, somethings are just very hard to figure out even for him. The worse part is convincing the government a particular law is non-sense isn't easy and changing it is even harder. Lobying the government to "not enforce" a particular section of a regulation is very hard, but in the meantime most people end up not implementing that particular rule.

    The reason I keep mentioning compliance is the industry and the government are looking for a solution that is DSL like. Since LOP attempts to solve some of the problems using DSL's, I'm hoping to learn something that may be of use. Most of the attempts at DSL in this field has been ad-hoc and proprietary. Most of them fail because they are either too restrictive to grow, or ambiguous (ie not feasible). The sad thing is, with 2-3 years of hardcore experience, I'm just realizing how complex financial compliance is. Just modeling the domain is hard enough that creating a comprehensive generalized domain is nearly impossible. Many people just aim for 80-90% coverage. Even then it's really quite frustrating and hard to resolve issues like treating a security's issuers as primary/secondary and tiered/layered within the same system.
  60. just for kicks[ Go to top ]

    Here is an example of a few paragraphs of one section of 2A7 regulation.
    Share Price Calculations. The current price per share, for purposes of distribution, redemption and repurchase, of any redeemable security issued by any registered investment company ("money market fund" or "fund"), notwithstanding the requirements of section 2(a)(41) of the Act and of Rule 2a-4 and Rule 22c-1 thereunder, may be computed by use of the Amortized Cost Method or the Penny-Rounding Method; Provided, however, that:

       1. Board Findings. The board of directors of the money market fund shall determine, in good faith, that it is in the best interests of the fund and its shareholders to maintain a stable net asset value per share or stable price per share, by virtue of either the Amortized Cost Method or the Penny- Rounding Method, and that the money market fund will continue to use such method only so long as the board of directors believes that it fairly reflects the market-based net asset value per share.

       2. Portfolio Maturity. The money market fund shall maintain a dollar- weighted average portfolio maturity appropriate to its objective of maintaining a stable net asset value per share or price per share; Provided, however, that the money market fund will not:

             1. Except as provided in paragraph (c)(2)(ii) of this section, Acquire any instrument with a remaining maturity of greater than 397 calendar days; or

             2. In the case of a money market fund not using the Amortized Cost Method, Acquire a Government Security with a remaining maturity of greater than 762 calendar days; or

             3. Maintain a dollar-weighted average portfolio maturity that exceeds ninety days.

    For someone to create a high level DSL for this specific domain, first the individual would have to know how to read legal jargon, followed by meeting with a reg lawyer and then about 50 other steps before there's any kind of clear functional requirements.
  61. Concrete example[ Go to top ]

    What would the equivalent LOP embodiment look like? It would be great if JetBrains can take a concrete example and explain it in greater detail.

    Good suggestion, it appears that this is quite important. We are already working on this. It will be one of our upcoming articles. The very next article will probably be an FAQ, in which we gather all the feedback we've received and respond to the common questions in one document.
  62. Reminds me of video game engines[ Go to top ]

    The automatic generation of optimal code is very much in the interests of developer efficiency.

    Let me back up. What I'm referring to is talk like this:

    "To explain how to lay outa GUI form, I could just draw the form, for example. If this drawing has enough detail, then the drawin itself represents the solution... Such domain-specific representations should be the program. In other words, there should be a method that allows me to use this representation as an actual program, not just as a way of communicating with other developers."

    This is a particularly telling example.

    Can I use LOP to just sketch a GUI and create a working application, or even a major part of it? No, it has to be "detailed enough." That's a pretty big gloss. The devil is in the details.

    DSLs are great - we've been using them for years, and better tools to make them are welcome. But they do not make the complexity of formally specifying solutions disappear.

    This kind of talk is overstating what DSLs can do, I think. There's no AI in there.
    You do, but larger than that, you misunderstand the purpose of DSLs.

    Actually now I think it's you who misunderstands me.
    DSLs aren't particularly about natural language. Thinking they are seems to be the basis of your misunderstanding.

    No, definitely not.
    which is basically what "ultra-high-level" languages are in this context.
    False. There is no particular connection between ultra-high-level language and natural language.
    As I have pointed out, the paper implies that connection, and it is what I object to.
  63. Reminds me of video game engines[ Go to top ]

    This is a particularly telling example. Can I use LOP to just sketch a GUI and create a working application, or even a major part of it? No, it has to be "detailed enough." That's a pretty big gloss. The devil is in the details.

    DSLs are great - we've been using them for years, and better tools to make them are welcome. But they do not make the complexity of formally specifying solutions disappear.

    It is true that the devil is in the details, and this is a good criticism. It is also true that the complexity won't simply disappear. However, what I would like to point out is that, even if we imagine that all the details must still be supplied, using a DSL will still be easier because a) the concepts of the DSL will be closer to your own mental concepts, and b) the tools provided with the DSL (such as intelligent editors) will make providing the right details in the right places much easier. You only have to look at IntelliJ IDEA to see what editor support can do for a programmer.

    (Disclosure: I work for JetBrains)
  64. Reminds me of video game engines[ Go to top ]

    What I'm referring to is talk like this:"To explain how to lay outa GUI form, I could just draw the form, for example. If this drawing has enough detail, then the drawin itself represents the solution... Such domain-specific representations should be the program.
    A domain specific expression (in conjunction with the rules of its domain specific language) gives an unambiguous account of behaviour or effect. I wonder why you think your GUI form is an expression in a DSL?

    Secondly, and perhaps more contentiously, you are more likely to be generating forms with the assistance of an already defined DSL.
    There's no AI in there.
    Correct. There isn't claimed to be any, and as above there doesn't need to be. What's all this about AI?
    There is no particular connection between ultra-high-level language and natural language.
    As I have pointed out, the paper implies that connection, and it is what I object to.
    Sergei will have noted that implying such a connection is not useful.
  65. The coolest thing: extended IDE support[ Go to top ]

    I've always wanted a language for which I could extend it's syntax to easily express my domain language. Now if that's topped with the possibility to represent it in some arbitrary form other than text, and create my own editor to handle that form, and all this integrated/supported in my favorit IDE - well, that's more than I wanted.

    I salute a framework which allows the definition/distribution of new languages and seamlessly integrates them through one IDE. Plugging in a new language looks to me like the ultimate plugin ;-)

    I'm an IntelliJ user for several years now, and I think it's one of the top Java IDEs around. I'm looking forward for trying out the MPS plugin - even if it's a non-complete prototype.

    Have you (IntelliJ guys) considered to open up the development of these language extensions (cause that's what they are) to the open source community ? I think it would make more sense, develop faster, become more robust, and I'm sure it would boost the sales of the supporting IDE once it proves a hit. Not to mention that it would make it possible to create plugins for existing languages... ever wanted to REALLY compete with eclipse ?

    Cheers,
    Csaba.
  66. How about version control ?[ Go to top ]

    Almost forgot about this. Any nontrivial development system needs version control. Currently the only good version control systems are text-based, so how would the non-text language programs fit in the version control ? Will there be some IDE support for it ? I can only think about using some text form for version control, and then map the changes to the structures from the language model to present higher level changes.

    Cheers,
    Csaba.
  67. How about version control ?[ Go to top ]

    Almost forgot about this. Any nontrivial development system needs version control. Currently the only good version control systems are text-based, so how would the non-text language programs fit in the version control ? Will there be some IDE support for it ? I can only think about using some text form for version control, and then map the changes to the structures from the language model to present higher level changes.

    Again, definitely. However, we are currently able to use text-based version control with XML files, so this is a bit lower priority. At some point in the future we will add direct support for version control on concept models, with much smarter features than text-based allows. I imagine that the EAP will only have text-based version control to start with.
  68. The coolest thing: extended IDE support[ Go to top ]

    Have you (IntelliJ guys) considered to open up the development of these language extensions (cause that's what they are) to the open source community ?

    Definitely, development of languages by anybody is almost the whole point of it!