Vincent Massol discusses "intelligent builds"

Discussions

News: Vincent Massol discusses "intelligent builds"

  1. Vincent Massol, in "IntelliB," discusses an imaginary product that is able to build a project without build files or POMs. There are some interesting challenges in this area, and Vincent is looking for potential solutions or problem areas he hasn't seen.
    Source locations It is possible to guess where sources are by looking for *.java files (for Java projects - The same applies for other project types). Now we still need to differentiate main sources from test sources but that's also relatively easy to do. We can check for classes extending JUnit's TestCase for example or the TestNG equivalent, or any other well-known testing framework. Note: An interesting thing here is that to be intelligent we'd need the help of the community to add new rules to the discovery process. For example imagine that a new testing framework appears; we'd need to add it to the Test Discovery Rules. Thus, this type of intelligent build system would need to rely a lot on the community and thus would need to get its data from an online repository that could be edited by the community. Dependencies How do we detect project dependencies? One relatively way is to parse the sources that we have found above and find all external imports. Then query ibiblio to find matching package names (this information is present in Maven POMs on ibiblio). Now for guessing the version, there's no easy magic. A first approach would be to get the latest released version of the dependencies we've found. Project type Project types can easily be guessed by looking at some files. For example if a web.xml file is present then it's a WAR project, if an application.xml one is found then it's an EAR project, if a jnlp file is found then it's a JNLP project, etc. SCM SCM can easily be guessed by looking for special files on the filesystem of the project. For example we would look for .cvs directories for SCV and for .svn files for Subversion, etc. Developers Once we got the SCM URL we can then query the SCM to get the list of all developers. Project name The project name could be the name of the top level directory and the version could be set arbitrarily to 1.0. Actually we could even check ibiblio to see if the project is already on ibiblio, get the latest version there and increase the minor number by one as a first order guess. Another strategy would be to query the SCM and look for tags and deduce existing versions by parsing those tags (there are some usual conventions for naming tags so it should be possible to make a good guess). Additional information Of course, the information found above are just guesses. In most cases they could be correct but of course we would need to offer a way for the user to edit them and to add any missing information.
    What do you think?

    Threaded Messages (29)

  2. Dependencies How will you find dependencies which are setup through reflection (Class.forName() etc...) If you actually get the tool to build you will have to generate a POM or else your approach of asking ibiblio will sooner or later run dry since new projects won't add the needed POM information to the repository. ProjectTypes A lot of projects may have both a web.xml and a ejb-jar.xml lying around as well as a jnpl. How will the tool handle this? Just a few thinks you will have to consider... Regards Stefan
  3. As you say, there's no easy magic in guessing the version. If you take JUnit 4.1 as an example, I was amazed to see that AWT and Swing runners simply disappeared, not even deprecated. That means that if you compile your code with the latest version and it doesn't work, you need to use the previous version and so on and so forth until it compiles... and this is just for compilation, not to mention runtime.
  4. It would be difficult to bootstrap a project with such a tool. For example, if there is no code yet, compile time dependancies cannot be discovered immediately. Plus, there is a paradigm shift that could have many drawbacks for few benefits: the Maven POM creates a structural constraint for the project team that benefits the overall predictibility and stability of development. Conversely, with this "intelligent" way of guessing what to do, build process is literally driven by development, which can lead to a major risk in a multi developer project. Event interesting aspects at first, like guessing production code from test code, may lead to create unstability on where to put java files for each category (plus, you can have test code that use (test only) helper class that don't belong to well known framework hierarchy). I must admit that, while I think targeting simplification of project configuration is the way to go, I am rather skeptical on this "no configuration file" build engine idea.
  5. It would be difficult to bootstrap a project with such a tool. For example, if there is no code yet, compile time dependancies cannot be discovered immediately.

    Plus, there is a paradigm shift that could have many drawbacks for few benefits: the Maven POM creates a structural constraint for the project team that benefits the overall predictibility and stability of development. Conversely, with this "intelligent" way of guessing what to do, build process is literally driven by development, which can lead to a major risk in a multi developer project.

    Event interesting aspects at first, like guessing production code from test code, may lead to create unstability on where to put java files for each category (plus, you can have test code that use (test only) helper class that don't belong to well known framework hierarchy).

    I must admit that, while I think targeting simplification of project configuration is the way to go, I am rather skeptical on this "no configuration file" build engine idea.
    +1 Very well said.
  6. Just to be clear, I've been a Maven developer since 2002 and I love Maven (I use it every day). I'm not suggesting at all that there is a better way of doing a build. This is just a thought experiment and as I say in my post one value this could have is to guess out some initial value for creating a POM for people migrating to Maven. All that said I remembered people saying that they'd rather that the build tool adapts to their project structure rather than the other way around and I thought "Could that be possible?". Hence this thought experiment. I'm very much aware of the difficulties around this idea and the fact that it is not deterministic. I'm just curious to see how far it could guess stuff...
  7. "All that said I remembered people saying that they'd rather that the build tool adapts to their project structure rather than the other way around and I thought "Could that be possible?"." Yes, but Maven stinks. In Maven you are giving up control in order to get a free, not terrible build process. It's cool, but that's the tradeoff. All of this stuff is definately possible. Eclipse does a have decent job of making many of these guesses with just a little prompting. But even if you make this happen, you still have maven. But I think the question is, if there was such a tool, would I want to use it? For me, no. Writing an ant build script just isn't that bad. Let's look at what I lose. Dependencies: Getting log4j from ibiblio is the most boring dependency example I can think of. Why do I care if log4j is updated? And if I just want version X, I'm going to download it and check it in. The only advantage is if I'm sending my project off to have someone else build it and I want send a small amount of data and make the downloader get the rest of the project from ibiblio. The real dependecy problem I have is that my project depends on the stuff from the guys down the hall who are working on a common library for me and some other team. Oh, and there are a dozen of those dependencies with other teams. That's an interesting problem, and one that I don't think can be addresseed by this scheme. Obfuscation: What if I'm shipping this software and need to obfuscate? This just can't really be guessed. Interesting divisions: One project I'm working on has a server and client piece built from the same codebase. Ant rules assemble the different jars from the common set of compiled code. There's no way to guess this. Testing: Over beers the other night, I was talking to a friend about his process. He has a good suite of tests that run very quickly and another suite that is much slower and is more functional than unit. For him, automatically running everything testable on every build would destroy CI. He needs to run the quick stuff everytime and then circle back and run the slow stuff nightly. That approach isn't exactly uncommon. And so on. The more magic the solution, the less control I have when I want to do something interesting. Putting the source directory where I want isn't very interesting and maven should definately handle that. But other things are interesting and you're going to have to write some sort of logic to control that at some point.
  8. "All that said I remembered people saying that they'd rather that the build tool adapts to their project structure rather than the other way around and I thought "Could that be possible?"."


    Yes, but Maven stinks.
    +1 I've yet to be able to get an external Maven project to build by simply checking it out. Internal to your enterprise or team I don't care if you use Maven or not - if it works for you, great. I think it is kind of arrogant to expose that dependency if you are publishing an open-source tool. For example I wanted to download and build X-Fire's examples. Well, of course the common courtesy of a simple build.xml file isn't there - nope, I have to use Maven. So I download Maven and point it to my JDK 1.5. The first thing that aggravates me is that it goes out and tries to download every jar - even though all those dependency jars are part of the X-Fire download. They are just one folder away. For some reason - maybe bad luck or maybe because I tend to consult at larger corporations - Maven can never download all the jars. Either it can't find it or errors out or something. To get the XFire build to work I had to hand copy all of the jars from the xfire download into the specific directories in the common Maven library area. Well I finally got the stuff to sort of compile but then there was an ant-task error. At that point I gave up and renewed my opinion that Maven is a hacky piece of software that I'd never recommend. I get tricked about once every year or two to try building a Maven project. Hasn't worked yet. For some reason all open source projects I've downloaded that provided an ant build.xml file seem just to work. Maybe if Maven had a task to generate a basic ant file for publishing your code for external use? I like this author's thought experiment though. It seems that you'd never be able to get rid of configuration entirely but a lot of it could be done by convention. Not suggesting this but throwing it out there: What about the possibility of annotating your import statements for dependencies? It would be a pain to do by hand of course but who hand types import statements anyway. Seems like an IDE could figure out what version of a library you are compiling and insert the annotation. // annotate that logging needs commons logging 1.1 or greater [org.apache.commons.logging 1.1+] import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; This would help the no-config build machine that we are thinking about but I'm not sure if it is much better than specifying the libraries manually in a common area. BTW: Back to Maven bashing - what is the point of not just storing the libraries with the project? Saving disk space? ______________ George Coller DevilElephant

  9. "BTW: Back to Maven bashing - what is the point of not just storing the libraries with the project? Saving disk space?
    I think it only makes sense if the libraries you depend on are changing and you need to stay up to date with those changes. That's usually not important for log4j or junit but is important when you depend on the project from the team down the hall.
  10. I think it only makes sense if the libraries you depend on are changing and you need to stay up to date with those changes. That's usually not important for log4j or junit but is important when you depend on the project from the team down the hall.
    It is a very common case, of course but I am not sure why I need Maven to do it its way, when I can easily automate the same in Ant. And I mean ALL of it. Unfortunately, many tools that were supposed to be helpful ended up being "control freaks". That's why I love IntelliJ so much, as opposed to Eclipse. In IntelliJ, the IDE helps you in a highly non-intrusive way and it feels good. Eclipse does not help you that much and yet it is very intrusive. Maven wants to be IntelliJ, maybe but so far, my impression is that it is more like Eclipse - control freak and not flexible enough.
  11. Hi George, It seems you have had some bad experience with maven. That's sad. It is a bit off topic for this thread to discuss Maven but if you're interested in helping us out to make a better product it would be real nice if you could send your experience on the maven list (maybe without using any harsh words in order to have a constructive discussion ;-)). I'm pretty sure you'd also learn quite a lot about Maven in the process. I know you won't believe it but there are lots of people using Maven happily every day :-) OTOH the best we could do to help drive Maven adoption is talk to people who have had issues with it. Thanks -Vincent
  12. Hi George,

    It seems you have had some bad experience with maven. That's sad. It is a bit off topic for this thread to discuss Maven but if you're interested in helping us out to make a better product it would be real nice if you could send your experience on the maven list (maybe without using any harsh words in order to have a constructive discussion ;-)). I'm pretty sure you'd also learn quite a lot about Maven in the process. I know you won't believe it but there are lots of people using Maven happily every day :-) OTOH the best we could do to help drive Maven adoption is talk to people who have had issues with it.

    Thanks
    -Vincent
    Post the link to the list and I'll try to do an even-handed write-up of my experiences. It isn't that I don't believe that Maven works for some teams - I'm sure it does. What I don't like is forcing me to have it installed and configured just to try out an open-source project. I believe that a Maven project should degrade gracefully down to a standard ant build that can be published for public use.
  13. Hi George,

    It seems you have had some bad experience with maven. That's sad. It is a bit off topic for this thread to discuss Maven but if you're interested in helping us out to make a better product it would be real nice if you could send your experience on the maven list (maybe without using any harsh words in order to have a constructive discussion ;-)). I'm pretty sure you'd also learn quite a lot about Maven in the process. I know you won't believe it but there are lots of people using Maven happily every day :-) OTOH the best we could do to help drive Maven adoption is talk to people who have had issues with it.

    Thanks
    -Vincent


    Post the link to the list and I'll try to do an even-handed write-up of my experiences. It isn't that I don't believe that Maven works for some teams - I'm sure it does. What I don't like is forcing me to have it installed and configured just to try out an open-source project. I believe that a Maven project should degrade gracefully down to a standard ant build that can be published for public use.
    But you also need Ant if you want to build an Ant project no? If you only want binaries, Maven can produce archives just like Ant can.
  14. But, sometimes Maven builds don't work "out of the box." You have to manually download and install stuff like the servlet API or the Java Persistence API. That stuff can be a real hassle and a lot of folks have complained about it. I have several open-source projects out there and my "customers" often complain that they wish I would just check the dependencies into SVN and let them download a complet, buildable project.
  15. But, sometimes Maven builds don't work "out of the box." You have to manually download and install stuff like the servlet API or the Java Persistence API. That stuff can be a real hassle and a lot of folks have complained about it. I have several open-source projects out there and my "customers" often complain that they wish I would just check the dependencies into SVN and let them download a complete, buildable project.
    Yes thanks, exactly what I'm trying to say. Maybe I was too hard on Maven (maybe not hard enough ;-) but let me put it this way: I'm not an idiot but trying to build someone else's Maven project often makes me feel like one. If that is the feeling I get from your open-source project then how do you think that will affect adoption of your product? Truthfully, if I can't download and build with a VERY minimal amount of setup I'm going to pass your project by and look elsewhere. I can't be the only one. ______________ George Coller DevilElephant
  16. But you also need Ant if you want to build an Ant project no? If you only want binaries, Maven can produce archives just like Ant can.
    LOL. I'm chuckling because when I wrote my last post I knew that someone was going to make this argument. My post was too long as it was so I didn't cover my butt. I hear you but come on, I haven't met a Java developer in years who didn't know how to run an ant build - even if some of them only could do it through their IDE. Marketshare: Ant: pretty dang close to 100% Maven: I've got no idea. Maybe 3%? Is the Maven dependency I'm lamenting about a secret ploy to get more Maven installs out there? Pretty sneaky sis! Again, I'm bashing Maven only from the standpoint that it shouldn't be forced on the potential consumers of your code. Maybe this isn't a Maven issue but a Maven-user's issue. ______________ George Coller DevilElephant
  17. Is the Maven dependency I'm lamenting about a secret ploy to get more Maven installs out there? Pretty sneaky sis!
    Maven dependencies management is really really bad. It is better to use Ivy for dependencies management and stick with Ant fro builds. Ranges are especially bad: they cause build unpredictability and non repeatability because they make build to depend on server repo content. Other complaints about dependencies management: no way to specify Java5 binaries with debug info for dev time build and nodebug obfuscated binaries for production build. No way to override version definition from command line (just to try new one)
  18. I hear you but come on, I haven't met a Java developer in years who didn't know how to run an ant build - even if some of them only could do it through their IDE.
    Go back five years and you have the exact same situation with Ant/Make as you have with Maven/Ant now. People didn't want to install Ant when they had Make lying around. I can hear you saying "Everybody wanted to use Ant because it was so much easier", but it had its fair share of problems just as Maven does. But many don't realize almost all of the problem people face with Maven are, in fact, social problems. We have the techincal means to make things easy but when project metadata is not correct it means Maven doesn't function very well. This is one of trade-offs when using Maven as it is a community-oriented tool with which people care share information about build infrastructures.
  19. We have the techincal means to make things easy ....
    Not quite IMO. The whole Maven dependency management idea that promotes build time POMs to the repository is flawed IMO because it ties versions of the dependencies to the version of the main artifact (when ranges are not used ). What is missed here is the concept of runtime bundle. Let me explain what I mean on the example: lets consider struts-a.jar that has the corresponding POM struts-a.pom that points to fileuploads-1.1.jar as dependency. Everything is fine till a security fix version of fileupload.jar 1.2 is released. Now we want to start migrating to fileupload-1.2.jar as dependency. How can it be accomplished to preserve build repeatability and predictability? I think that all transitive dependencies should be handled via bundles. I mean that there should be separation of dependency descriptions: one dependency description is for building an artifact and another to expose the artifact and its dependencies. Therefore the dependencies descriptions could have independent evolution, if runtime users want to start using fileupload-1.2 or 3.10 as a dependency with struts then ‘bundle’ definition can be updated and they will get the fileupload fix if they will start pointing to a new bundle version. To simplify maintenance runtime bundle definition could inherit from build time pom and just have overrides or blockers.
  20. Not quite IMO.
    The whole Maven dependency management idea that promotes build time POMs to the repository is flawed IMO because it ties versions of the dependencies to the version of the main artifact (when ranges are not used ). What is missed here is the concept of runtime bundle.
    Let me explain what I mean on the example: lets consider struts-a.jar that has the corresponding POM struts-a.pom that points to fileuploads-1.1.jar as dependency. Everything is fine till a security fix version of fileupload.jar 1.2 is released. Now we want to start migrating to fileupload-1.2.jar as dependency. How can it be accomplished to preserve build repeatability and predictability
    It is already like this in Maven. Maven 2 always takes the "nearest definition" first when it come across 2 or more similar dependencies. So if you want to be sure to get fileuploads-1.2, just specify the version in your project pom. This way Maven is going to ignore what version Struts specify and always take the version 1.2 since you have already specified which version to use in your project pom.
  21. It is already like this in Maven. Maven 2 always takes the "nearest definition" first when it come across 2 or more similar dependencies. So if you want to be sure to get fileuploads-1.2, just specify the version in your project pom. This way Maven is going to ignore what version Struts specify and always take the version 1.2 since you have already specified which version to use in your project pom.
    When every single project has to do its own tracking of version updates it is not quite helpful. And it can and should be eliminated (well, minimized is more appropriate word).

  22. But you also need Ant if you want to build an Ant project no? If you only want binaries, Maven can produce archives just like Ant can.


    LOL. I'm chuckling because when I wrote my last post I knew that someone was going to make this argument. My post was too long as it was so I didn't cover my butt.

    I hear you but come on, I haven't met a Java developer in years who didn't know how to run an ant build - even if some of them only could do it through their IDE.

    Marketshare:
    Ant: pretty dang close to 100%
    Maven: I've got no idea. Maybe 3%?

    Is the Maven dependency I'm lamenting about a secret ploy to get more Maven installs out there? Pretty sneaky sis!

    Again, I'm bashing Maven only from the standpoint that it shouldn't be forced on the potential consumers of your code. Maybe this isn't a Maven issue but a Maven-user's issue.

    ______________
    George Coller
    DevilElephant
    You know I have students doing internship where I work and they complain when they download some source code and it comes with an Ant build. They are more use to Maven (since this is what we use) and prefer it. So this argument can go both way with new Java developers coming out of schools.

  23. Maybe if Maven had a task to generate a basic ant file for publishing your code for external use?
    http://maven.apache.org/plugins/maven-ant-plugin/
  24. "All that said I remembered people saying that they'd rather that the build tool adapts to their project structure rather than the other way around and I thought "Could that be possible?"."


    Yes, but Maven stinks. In Maven you are giving up control in order to get a free, not terrible build process. It's cool, but that's the tradeoff. All of this stuff is definately possible. Eclipse does a have decent job of making many of these guesses with just a little prompting. But even if you make this happen, you still have maven.

    But I think the question is, if there was such a tool, would I want to use it? For me, no. Writing an ant build script just isn't that bad.

    Let's look at what I lose.

    Dependencies: Getting log4j from ibiblio is the most boring dependency example I can think of. Why do I care if log4j is updated? And if I just want version X, I'm going to download it and check it in. The only advantage is if I'm sending my project off to have someone else build it and I want send a small amount of data and make the downloader get the rest of the project from ibiblio.

    The real dependecy problem I have is that my project depends on the stuff from the guys down the hall who are working on a common library for me and some other team. Oh, and there are a dozen of those dependencies with other teams. That's an interesting problem, and one that I don't think can be addresseed by this scheme.

    Maven 2 never updates your dependency when you specify an official version. It only does it if you specify a snapshot version so you can work in team. Have you actually tried it?

    Testing: Over beers the other night, I was talking to a friend about his process. He has a good suite of tests that run very quickly and another suite that is much slower and is more functional than unit. For him, automatically running everything testable on every build would destroy CI. He needs to run the quick stuff everytime and then circle back and run the slow stuff nightly. That approach isn't exactly uncommon.
    If you are still talking about Maven, it is really easy to have 2 sets of test using profiles.
  25. When I do ant builds I end up using templating a lot so that I can make builds for different environments by changing properties in one place. Usually this ends up being only a handful of properties but it is nice to consolidate them in one configuration file and have Ant find and replace as it copies from source to distribution directories. This makes config-management on multi-component projects much simpler than a big hierarchy of property files. Again - great thought experiment. ______________ George Coller DevilElephant
  26. Interesting. I see the notion of "convention over configuration" is starting to make it's power felt in all sorts of areas. I'll just have to applaud this notion, and start thinking really hard about how to make it work.
  27. Autmatic Build Maker[ Go to top ]

    Absolutely right i think it would be easy enough to scrap off clues from the source and other files that make a build to get the necessary information to create a POM for example. However automation at such a level in my opinion would still require some amount of human intervention , namely in dependencies parsing the source files for imports would indeed reveal required artifacts but then we dont really know which version is required. Imagine the amount of parsing that would be needed to retrieve just the dependencies :P ...thats a lot. Then the proper branch for the source and version number for the build would still need to be typed in. Of course am sure the automation would also at times induce "automated errors" and the build master would have to digg into the generated-POM to correct any bugs. So I think that indeed an IDE could be used to create around 80% of the content of a POM for example but the rest would have to be added manually.
  28. Re: Autmatic Build Maker[ Go to top ]

    You cannot download the Sun JARs from Ibiblio because it is techincally illegal. It is also illegal to check them into your SCM. This is not something Sun is likely to chase you down for but the reason they are not in the central repository is a legal one. This is changing as free implementations are becoming available from projects like Geronimo and GlassFish. So if you have the Java Mail, Activation, or any other Sun Binary License JAR in your SCM, according to the SBL you are techincally in violation. You are only supposed to distribute them. I've been trying since day one of Maven to make this easier but Sun has never done anything about it. So this is not something we did to intentionally try to make developers' lives' difficult.
  29. I think building a Maven 2 project is really easy except for two major problems at the moment (which are being adressed right now) : 1) Sun jars which can't be put online. 2) There is no way to correct a wrong pom without releasing a new version. This has burned me couple of times. Also, I would like the possibility to let me turn off Maven dependency management features for a given build and let me point to local dependencies without ressorting to install them in my local repository if I need too. Very practical when you run offline and you don't have time to install locally every dependencies. System dependencies already make this possible but I would like a way to do it myself from the command line without changing the pom file (which isn't mine most of the time). But note that Maven works better when you setup an internal repository where you can put all the missing proprietary stuff that isn't on ibiblio. I think the hard part of Maven for many users come from it's dependency management features. There should be a way to turn it off if you don't want to use it but still want to enjoys every other Maven features. Other than that I just love Maven 2, saves me a lot of time and make team collaboration very easy.
  30. I think a step towards your goal may be some kind of 'wizard' that will scan a project and generate build and pom files. This way the developer can use it on-demand. The list of plugins you could add to such a wizard can grow very long...