Discussions

News: AntFlow: Hotfolder Driven Workflow and Automation based on Ant

  1. AntFlow builds upon Apache Ant to provide a new approach to simplifying system automation that uses pipelines of hot folders chained together to perform a given task. Using XML, AntFlow associates an automated task, such as data transfer, encryption, or XML processing with a directory on the local system. Whenever a file is copied or written into the hot folder, the associated task is executed and the file is moved to the next hot folder in the pipeline for further processing. AntFlow enhances Ant in the following ways:

     * Hot Folder Triggers - AntFlow allows users to automate processes based on a file being added or changed within a folder on disk. This provides an intuitive and user-friendly way for end-users to interact with applications and can greatly simplify integration between applications that can output and injest files.

     * Powerful Task Scheduling - Through the use of the embedded Quartz scheduler, AntFlow allows repeating tasks such as backup, file transfer, or web service calls to be scheduled according to a flexible set of rules.

     * Workflow and Error Handling: AntFlow adds the capability to create workflows by chaining together hot folders, allowing complex manual processes to be automated. Error handling workflows can also be set up to send out e-mails or pager notifications when critical problems occur.

    Visit the AntFlow home page.

    Threaded Messages (19)

  2. Quick Start[ Go to top ]

    Hey guys, from using AntFlow internally for over a year, it never stops amazing me how easy it is to create hotfolder based file processing apps just by writing some Ant build.xml files.

    The challenge we have is actually explaining this stuff to people, so we've whipped up a simple quick start, but I don't know if it goes far enough. So any feedback on AntFlow or the intro doc would be most appreciated.
  3. What license have you chosen?
  4. We've chosen the Apache license. We could just have easily used any BSD-like license, but out of respect for Ant, which provides a significant portion of AntFlows functionality, we thought it made the most sense to use Apache.
  5. AntFlow website[ Go to top ]

    Curious that the website is built by Maven!
  6. why so?[ Go to top ]

    I'm the guy running that we site (and handling the OS release in general) and maven seemed a good fit. AntFlow itself is more about automating file based data transformation tasks. It would do a fine job of initiating xdoc->html transforms and subsequent file transfer to the website if you wanted them auto-triggered and unattended, but site:deploy is more the sort of thing one wants handled in an on-demand fashion and maven's a better fit for that one.
  7. AntFlow website[ Go to top ]

    Yeah, Maven is pretty cool. We're big supporters of any tools based on XML scripting or templating. When we first started on AntFlow we were actually debating between building the technology on top of Jelly or on top of vanilla Ant.

    While I'm sure Jelly would have been a great choice, we finally settled on vanilla Ant because it already does a great job with tasks related to file processing and Ant is very easy for novice programmers to learn. The cool thing is, once we get support for Ant 1.6.2 within AntFlow, people will be able to use the new Script task to insert arbitrary snippets of Javascript, Python, Beanshell, etc into their Ant code.

    So for 95% of AntFlow tasks, you should be able to use standard Ant XML, but for those cases where you need more powerful language constructs, you can do some more complex scripting.
  8. Congratulation Justin! One thing I'm interested in is how you have used this tool in your organization. Do you have any example usages you could share?

    -dain
  9. AntFlow Use Cases[ Go to top ]

    Hey Dain! We've already deployed some pretty complicated applications internally based on AntFlow including a content delivery network for genomics data, database-driven file delivery via satellite, etc. Our next big task is to start documenting and sharing these use cases.

    It'll take some time, but we'll be posting more tutorials and use cases and try to do a better job of explaining this concept of "fault tolerant scripting" that AntFlow provides.

    Hopefully this process will generate some new use cases for AntFlow and be able to use them to continue to evolve the architecture.
  10. Interesting Stuff[ Go to top ]

    One of my tasks is to deploy new builds to multiple production machines. I wrote a little application which lets me do this but I am considering changing to AntFlow. The idea is very cool and I'm going to look into it more in the near future. Good job!
  11. Interesting Stuff[ Go to top ]

    Cool. Feel free to lean on us as much as you need via the mailing lists or forums to get it working how you want it. We've discovered a number of AntFlow "design patterns" through trial and error so we can probably recommend some approaches you can take.
  12. AntFlow why use it?[ Go to top ]

    What i would like to know is why to use AntFlow and some situations where one would opt to adopt it. If i understand well AntFlow is just about having some set of instructions executed on a file once it is placed inside a "hot" folder.

    I believe that the flow of instructions is already implemented in your ant build script so why this idea of hot folder?

    Any examples to show why this would be helpful?
  13. Re: AntFlow why use it?[ Go to top ]

    What i would like to know is why to use AntFlow and some situations where one would opt to adopt it. If i understand well AntFlow is just about having some set of instructions executed on a file once it is placed inside a "hot" folder.I believe that the flow of instructions is already implemented in your ant build script so why this idea of hot folder?Any examples to show why this would be helpful?

    Well, you can use AntFlow, for example, in order to implement a file transfer solution like a distributed solution. A MyInternet for file transfer.

    I have seen recently 2 ESB vendor presentations. Both of them have described their clients' ESB solution evolution as the following :
    - first, a file transfer solution,
    - then, a next-generation transfer solution on top of JMS,
    - and after that, an improved solution with XML transformation (between nodes) and business rules.

    Both of them have described the first solution as an ESB solution (a first generation ESB).

    So, I guess AntFlow could be used for developping a low-cost first-generation ESB. First-generation but with sophisticated features, with transfer priority, supervision, data aggregation on event and so, on. JDK 1.5 provides enhancements that enables to implement faster such solution (using the thread pool management of JDK 1.5 for example).

    Working a little on a file transfer mechanism months ago, I have thought about a ANT-based solution, but I had no time to influence the solution design unfortunately. I am happy to see that AntFlow has emerged.

    Dominique
  14. Re: AntFlow why use it?[ Go to top ]

    What i would like to know is why to use AntFlow and some situations where one would opt to adopt it. If i understand well AntFlow is just about having some set of instructions executed on a file once it is placed inside a "hot" folder.I believe that the flow of instructions is already implemented in your ant build script so why this idea of hot folder?Any examples to show why this would be helpful?

    Well, you can use AntFlow, for example, in order to implement a file transfer solution like a distributed solution. A MyInternet for file transfer.

    I have seen recently 2 ESB vendor presentations. Both of them have described their clients' ESB solution evolution as the following :
    - first, a file transfer solution,
    - then, a next-generation transfer solution on top of JMS,
    - and after that, an improved solution with XML transformation (between nodes) and business rules.

    AntFlow allows hot folder triggers.

    Does AntFlow allow other types of observable targets ? Like JMS ?

    Do you plan to use Spring ?

    I wonder if Spring makes sense here to separate the workflow logic (in ANT scripts) and the assembly of business components (in Spring configuration !?). Such business components might be available in ANT scripts and already configured through Spring.

    My 2 cents.

    Thanks.

    Dominique
  15. Re: AntFlow why use it?[ Go to top ]

    Currently AntFlow only uses the hot folder triggers, but your suggestion makes perfect sense that that concept be generalized to support any type of trigger.

    A change that we're actually looking at making in the very near future is with our Quartz scheduling. We'll make Quarts a top-level trigger just like the hot folders, so there will be no difference between a task that is triggered due to a hot folder and one that is triggered based on time of day.

    I personally don't know much about Spring, but maybe my other guys do. We'll definitely look into it.

    AntFlow allows hot folder triggers.

    Does AntFlow allow other types of observable targets ? Like JMS ?

    Do you plan to use Spring ?

    I wonder if Spring makes sense here to separate the workflow logic (in ANT scripts) and the assembly of business components (in Spring configuration !?). Such business components might be available in ANT scripts and already configured through Spring.
  16. Re: AntFlow why use it?[ Go to top ]

    So, I guess AntFlow could be used for developping a low-cost first-generation ESB.

    That sounds much like GridAnt. With Ant's evolutionary momentum in that direction, I'm increasingly considering Ant and BPEL as either potential competitors or subject to some kind of synergistic convergence.
  17. AntFlow why use it?[ Go to top ]

    With the current version of AntFlow, there are two main reasons to use it: Integrating with the End User, Integration with other appplications.

    When building any complex application, one of the key issues is how to provide a user-friendly user interface that the average PC user can interact with. Conventional wisdom tells us that we can either develop a web-based appliction, or invest a lot of time and money in developing a rich GUI application.

    For certain applications, which tend to be file-centric, I think the "hotfolder" approach represents an extremely user-friendly approach to user-interface. Interacting with the application is as simple as drag-and-drop, or "save as". One example that is trivial with AntFlow is file format conversion or XSLT processing. If a user wants to convert a word document to a PDF, you would simply create a task that does word2PDF conversion, set up the AntFlow folder, and when the user drops the file into the "Word2PDF" folder, it will automatically be transformed. Another trivial example is PGP-style encryption.

    The other use case for AntFlow is application integration. There are thousands of applications out there today that can participate in a file-centric workflow by the simple virtue that they can injest and create files. For example, perhaps you are writing an application for a bank that needs to have check images injested into a SQL database. You can set up a simple AntFlow setup where your incoming hotfolder is monitoring an FTP or WebDAV upload directory. From there perhaps you decrypt a bundle of images using GPG using a simple AntFlow task, then perhaps move the files to a directory where they are expected to reside with a certain naming scheme, perhaps using timestamps on the file names. Then you can execute an SQL query from Ant and store the metadata about the images in the database (or store the images themselves). If there are any problems at any point in the execution, such as running out of disk space, or the database being down, you can fork off an e-mail file to the e-mail hot folder to notify the administrator that there is a problem.
  18. Looks pretty cool[ Go to top ]

    I'm looking forward to trying this out. Seems like one of those great ideas that are lying around waiting for someone to think of them and implement.

    Basically just marrying a file/directory monitor to ant tasks. Good idea. Looks like you threw quartz scheduling in there too.
  19. Comparison to Babledoc?[ Go to top ]

    This looks really interesting. I was wondering if someone could comment on how this project compares to Babeldoc? (http://babeldoc.com/).
  20. Just visited the site and it's been untouched since last November. Is this an active project or has it been left to die? It's still RC1 not final.