XJ: Extensible Java (a proposal)


News: XJ: Extensible Java (a proposal)

  1. XJ: Extensible Java (a proposal) (20 messages)

    Java is a fixed language. In Java 5 annotations were introduced in recognition that the Java language need to accomodate customisation towards specific use cases such as Enterprise systems. Annotations can be seen as an approach to defining Domain Specific Languages (DSLs) based on the core Java language. Annotations are essentially name-value bindings and are limited in what they can express. They cannot define a new language constructs such as a "select" from collection statement. The fact that annotations exists suggests a requirement that Java needs DSL capabilities but without the richness to realise fully blown DSLs. Here we make a proposal that extends Java to support user defined syntax. The extension is conservative in the sense that it will not conflict with any existing language features and will preserve backward compatibility. The proposal extends classes with syntax defnitions to produce new modular language constructs, called syntax-classes, that can easily be distributed along with an application in the usual way. The proposal is called XJ (eXtensible Java). XJ introduces the idea of a syntax-class into Java. A syntax-class is a normal Java class that defines a language grammar. When the Java parser encounters an occurrence of a language feature delimited by a syntax-class, the class's grammar is used to process the input. If the parse succeeds then the grammar synthesizes an Java abstract syntax tree (AST). An object of type AST has a standard interface that is used by the Java compiler when it processes the syntax. New types of AST can be defined providing that they implement the appropriate interface. Consider a simple language construct in Java that selects an element from a collection based on some predicate. An example of using the new construct is shown below:import language mylang.Select; public Person getChild(Vector people) { @Select Person p from people where p.age < 18 { return p; } else { return null; } }The new language construct is called Select which is defined by prefixing a reference to the syntax-class with the @-character. The value p is selected from the vector people providing that the age of the person is less than 18. If a value can be selected then it is returned otherwise null is returned. The use of Select is equivalent to the following definition:public Person getChild(Vector people) { for(int i = 0; i < people.size(); i++) { Person p = people.elementAt(i); if(p.age < 18) return p; } return null; }A new language construct is introduced in XJ by defining a syntax-class. A syntax-class contains a grammar which is used by the Java parser to process the concrete program syntax and to return an abstract syntax tree (AST). Once a syntax-class has been defined, it can be used in program code by referencing the class after the syntax escape character '@'. The definition for the select construct looks like the following:package mylang; import language java.syntax.Grammar; import java.syntax.AST; import java.syntax.Block; import java.syntax.Context; import java.syntax.Statement; import java.syntax.Sugar; import java.syntax.Type; import java.syntax.Var; public class Select extends Sugar { private Type type; private Var var; private AST collection; private AST test; private Block body; private Block otherwise; public Select(Type T,String n,AST c,AST t,Block b,Block o) { type = T; var = new Var(n); collection = c; test = t; body = b; otherwise = o; } // Select Grammar definition @Grammar extends Statement { Select ::= T = Type n = Name 'from' c = Exp 'when' t = Exp b = Block o = ('else' Block | { return new Block(); }) { return new Select(T,n,c,t,b,o); }. } // Desugar to produce an abstract syntax tree public AST desugar(Context context) { Class cType = context.getType(collection); if(isVector(cType)) return desugarVector(cType,contect); else // More cases... } public AST desugarVector(Class cType,Context context) { Var done = context.newVar(); Var coll = context.newVar(); return [| boolean = false; coll = ; for(int i = 0; i < .size(); i++) { = .elementAt(i); if() { = true; ; } } if(!) ; |]; } }The Select grammar rule specifies that a well-formed statement is a type followed by a name, the keyword 'from' followed by an expression, the keyword 'when' followed by an expression and then a block which is the body of the select. After the body there may be an optional else keyword preceding a block. In each case within the Select rule, the parse elements produce a value that may optionally be associated with names. For example, the type is associated with the name T. In addition, a parse rule can contain Java statements that return a value. These are enclosed in { and }, and may reference any of the names that have been defined to the left of the statement. The final value returned by the Select rule is an instance of the class Select. The value synthesized and returned by a grammar must be an instance of java.syntax.AST. If the return value is an instance of one of the standard Java AST classes then no special action needs to be taken by the syntax-class. If the return value is an instance of a user-defined syntax-class then that class must implement the AST interface which is used by the compiler to translate the source code into Java VM code. To make this process easier, a user defined syntax-class can extend java.syntax.Sugar which implements the AST interface through a method called desugar. The desugar method is responsible for translating the receiver into an AST for which the interface is defined (typically desugaring into standard Java code). A common way of desugaring syntax is to use quasi-quotes to express abstract syntax in terms of existing concrete syntax. Quasi-quotes are used in the desugarVector method in the Select definition, to introduce these here is a simple quasi-quoted AST:[| x + |]which is equivalent to the Java expression:new java.syntax.BinExp( new java.syntax.Var("x"), "+", new java,syntax.Int(1))The delimiters [| and |] transform the enclosing concrete syntax into the corresponding abstract syntax. Within [| and |] the delimiters < and > can be used to 'unquote' the syntax in order to drop in some abstract syntax. The two forms of delimiters can be arbitrarily nested. Quasi-quotes are an easy way to create code templates in XJ. In this short article we have introduced an extension to Java called XJ which allows new Java syntax to be defined thus allowing extensions to the Java language. Although XJ has not yet been implemented in Java, it is one of the key features of the XMF language that has been used in commercial tools (XMF-Mosaic) and has been made open-source in 2008. More details of this proposal and extended examples (including the application of XJ to Enterprise Java) can be found in the paper "Beyond Annotations: A Proposal for Extensible Java (XJ)."

    Threaded Messages (20)

  2. That's all cool, but how do you relate errors (especially type errors, especially generic type errors) to the DSL code? What about IDE support? Autocompletion? I could come up with quite a few things more... Isn't it better to just improve the language to the point where you can define embedded DSLs that make use of the existing tools, like Ruby or Haskell? I for one would be scared if Java would have such a "feature".
  3. I like the idea, and the example was pretty straightforward up to the 'desugar' definition. I'm sure there's a good way to do the syntax, but hopefully it's not by adding everywhere. That's just ugly and makes it a bear to read. Besides that, it's an interesting idea. I wonder how it compares to C# 3.0's new features that enables LINQ and all those associated goodies. This seems like a different approach (I don't think those features in C# do anything at the AST level).
  4. (I don't think those features in C# do anything at the AST level).
    Not directly, but in C# you have access to the AST as an in-memory datastructure representing the parsed code at runtime, which is a quite cute feature.
  5. Begs the question: why? W H Y? This is all cool stuff and reminds me "Look, ma, no hands!" type of technology. What is so wrong with plain JDBC, or iBatis, or JPA? What advantage does it bring? Will it be dramatically simpler, safer or more productive to use? These questions really need to be answered in details before considering something like that. Regards, Nikita Ivanov. GridGain - Grid Computing Made Simple
  6. What is so wrong with plain JDBC, or iBatis, or JPA? What advantage does it bring?
    Well JPQL/JDOQL(/SQL) is String-based, has no type checking and no auto-refactoring if a class is refactored - have to manually update the queries. Having a type-safe refactorable query language available for persistence solution makes a heck of a lot of sense. Doesn't mean that this is the best way to provide it though.
  7. Raul, Refactor-safe SQL is a good argument. I'll buy that. Thanks, Nikita Ivanov. GridGain - Grid Computing Made Simple
  8. -1 Java have a definite syntax. It is clear, what is written does everywhere the same. This enables the great tooling possibility. We should not litter this with unnecessary complexity, in order to be able to write something what is already possible another way. I think, this is not where the productivity is gone, for most of the project... Cheers, Tamas
  9. Re: XJ: Extensible Java (a proposal)[ Go to top ]

    Who needs this stuff anyways? This cr*p is from the same guys who are promoting "super languages" unfortunately for which there are no takers...
  10. Hmm... It reminds me of SQLJ (and how ugly it is...) If you really had to provide an extension, why can't you think of something that be Object Oriented? HQL could be a good start. Or perhaps a rule based approach ? I am writing a DSL to provide a declarative (rule based approach) processing engine. I would rather use Hibernate to populate my objects. -Nagraj TeleCommunication Systems,Inc
  11. Nice as an add-on[ Go to top ]

    Looking at the example I think that does a rather decent job at what it does. I don't want it in the Java language itself though. And add me to the list of objectors to the delimiter for escaped code, man that is ugly!
  12. In the wrong hands..[ Go to top ]

    My problem with these types of extensions are the same as my problem with AspectJ: I see the benefit, I see how it can increase productivity and clarity, but I know that someday, somewhere, I will have to maintain a system that was developed by someone who had no clue how to use these things properly. With the Java language, at least there is an upper limit on what the syntax in front of me means, with arbitrary extensions I simply have no clue.
  13. Macros[ Go to top ]

    The idea is good but the implementation is too complex, IMHO. The end result is something like C++ macros but that operate on the AST. Better IMO would be to start smaller and just have compile-time typesafe macros that produce standard source code. Kit
  14. This is a good idea, I have suggested it several times, but not for the DSL reason that is mentioned here. Basically what is being done here, IMHO, is not so much an extention of the language as well a stand way of extenting the compiler. And being able to extend the compiler is very interesting because certain frames, like AspectJ or JPA postprocessing, can finally be done in a way so that you can combine them. Has anyone ever tried to use a JPA agent and at the same time AspectJ? AFAIK these two conflict because they both need to do some processing on the classes. But it is very logical to use AspectJ on the business model... (Yes yes, preprocessing, but you'll always have that chance of missing an aspect.) To the point of DSL... I do not see the need to express domain spefic stuff in the syntax of a programming language. Please lets not mix apples and spaghetti. Domain specific info has to be modeled, not embedded, in a programming language. If you fancy a DSL, then IMHO it should be bolted on "on top" not "inside". Call it "separation of concern", if you please. The only thing that is interesting for this specific example are closures, so we can write that select more easily.
  15. Bad idea[ Go to top ]

    -10 Annotations should never be used for interpretation of the language. They should be used for attaching information to the code. And don't try to "fix" Java by adding all kinds of extensions to it. Use a language that is well thought out from the start, like Scala or Haskell. It would be trivial to implement your example in Scala (or Haskell) without the need for AST manipulation and it would be readable by anyone who knows the Scala syntax.
  16. Re: Bad idea[ Go to top ]

    Annotations should never be used for interpretation of the language. They should be used for attaching information to the code.
  17. Re: Bad idea[ Go to top ]

    Annotations should never be used for interpretation of the language. They should be used for attaching information to the code.

    +! from me on this as well. Anyway back to the example, if you want rather similar behavior then use groovy and closures you would be amazed about the result you would get.
  18. Never seen anything more dangerous since long time. I agree with almost every negative comment. Looks like smoothing the highway to hell. In any case, I find more straightforward a traditional approach like: class CollectionSelector public Object select(Collection c, Predicate p) { .... } even if I understand that in a throw-away prototype the proposed approach might (read again, might) speed-up implementation. Unfortunately, the real system implementation never starts throwing away the prototype. Guido
  19. I've said this before, but a (IMO) better solution already exists -- Write it in Groovy :) Not all of your code. Just the parts that need the certain level of expressiveness that you've described there. Take for example: http://grails.org/doc/1.0.x/ref/Domain%20Classes/findAllWhere.html If you need expressiveness, use a dynamic language that integrates seamlessly with Java. Where you need raw performance, keep it in Java. Done.
  20. I'm all for DSLs but trying to bend java to support new kinds of constructs will result in people waiving their backward compatibility flags, and there by we ending up with a shitty implementation (type erasure anyone?). I'd rather this be a new extensible language on top of the JVM . Developers can decide if they want to use it or ignore it. Dushyanth Blogging at: Listen to Me!!!
  21. Bad proposal[ Go to top ]

    In order to decide whether it's good or bad I'd propose to have a look at origins of Java. Java was invented as simple language in contrary to C++ (no multiple inheritance, no overloading of ops, all functions are virtual, etc). From this point of view XJ brings unnecessary complexity.