AspectWerkz team releases AWbench: AOP benchmark tool

Discussions

News: AspectWerkz team releases AWbench: AOP benchmark tool

  1. The AspectWerkz team has released AWbench, a microbenchmark project to compare performance of different AOP frameworks and proxy approaches.

    The first results include AspectJ, AspectWerkz, JBoss AOP, Spring, DynAOP and CGLib.

    The report illustrates how the results can vary from one framework to another and one constructs to another, with situations where the slowest is more than 50 times behind the fastest.

    Read more about AWbench

    Threaded Messages (20)

  2. not clear[ Go to top ]

    in the result table I see AspectWerkz twice. Does the first column represent the 2.0RC? or some other version? Some captions would be nice and make it easier to read.
  3. not clear[ Go to top ]

    The bench includes AspectWerkz 2 times
    - AspectWerkz 2.0.RC2-snapshot
    - AspectWerkz 1.0 final
    since the underlying architecture is completely different.

    Moreover, the 3 "ext:..." labels in the results for "ext:aspectj", "ext:spring" and
    "ext:aopalliance" have been obtained by running aspectj aspects, spring aspects and aopalliance aspects (in term of programming model) within the AspectWerkz extensible container that has been presented here
    http://www.theserverside.com/articles/article.tss?l=AspectWerkzP1

    The resulting performance of "ext:.." is thus roughly similar to regular AspectWerkz 2.x aspects, since the underlying runtime is the same. A main difference occurs when one framework semantic provides only reflective access to the advised target, since this "feature" is tied to the programming model (the way we write aspects and advice).

    Alex
  4. I am not sure, but it looks like Spring and DynAOP uses CGlib as proxy implementation library and results must be same.
    CGlib never was AOP framework, have you implemented custom AOP framework on top of it (it does nothing without user code) ?
  5. CGlib does not allow to chain interceptors (callback), hence both Spring and DynAOP bring their own glue to allow that.

    That s right, CGlib is not AOP. To some extend some other implementations are not neither, so I left this debate to others. It is intereting to have CGlib here since it allows to measure the overhead brought by upper layers that are using it.

    Complete code is available from public CVS and can be easily run thru Ant. All measurements do the same from a user perspective f.e. the most simple before advice increment a static int counter. This can be off course be achieved in both CGlib and more rich AOP implementations.

    Alex
  6. time to reconsider[ Go to top ]

    well after being more than pleased with Spring's ease of use/integration with Hibernate in the financial web application i am currently working on, i was also looking into using it for its AOP capabilities. but now after reviewing these benchmarks as listed on the site.....i may have to reconsider. if i am reading it correctly Spring performed HORRIBLY compared to the rest of the AOP frameworks. Of course, all this extra overhead time implies that the Spring framework is doing some sort of extra work that may prove more beneficial to the developer but which the benchmarks do not reflect....

    anyone have any input on other criterion to take into consideration other than these statistics?
  7. time to reconsider[ Go to top ]

    In our implementation of the bench ALL of the frameworks does EXACTLY the same thing in each single comparison. Noone is doing any extra work that you might benefit from later.

    It simply shows the execution time of different aspects, taking different kinds of context ('this', 'target', 'arguments etc.) while doing exactly the same thing (in this case increment a counter).

    But there are of course many more things to take into consideration when choosing your tool than a simple benchmark. Such as programming model, easy of use, coherent semantics, code stability, etc. etc., but this is out of scope for this little benchmark project.
  8. time to reconsider[ Go to top ]

    First of all, great performance numbers from AspectWerkz 2.0 RC - congratulations to the team! However, the benchmark exercise might be quite misleading to readers. In response to Ben, in particular: Spring certainly does't perform "horribly" here. Don't reconsider the use of Spring but rather reconsider the relevance of the performance numbers to the typical use of AOP in your applications.

    Firstly, consider all of this with a large grain of salt. Like all such benchmarks, this is highly artificial: It just considers the invocation overhead rather than real-life operations. If your method implementation does anything meaningful (like database access) and you're just applying AOP to coarse-grained services, the invocation overhead will usually be completely negligible - with any AOP framework.

    Secondly, the highlighted lines in the results are not the most important AOP use cases but rather the ones where AspectWerkz 2.0 RC shines brightest. For example, transaction management usually implies an around advice: The difference between AspectWerkz 2.0 RC and the others is not quite as significant there (though still impressive). Again, we're just talking about the invocation overhead here - what's really relevant for overall performance is your target object implementation!

    Thirdly, Spring's AOP framework almost exactly performs at the level of AspectWerkz 1.x. So if Spring's performance had to be considered horrible, AspectWerkz 1.x's performance would have to be considered horrible too. I rather see that AspectWerkz 1.x's performance is quite good already, and so is Spring's. It's just that AspectWerkz 2.0 aims to improve significantly, which is good - but as I said, the difference is usually irrelevant if all you want to do is use AOP for coarse-grained services.

    Juergen
  9. time to reconsider[ Go to top ]

    AWbench does not aim at pushing AspectWerkz 2.x as a rocking things. AspectWerkz 2.x is still a ReleaseCandidate.

    We have simply observed that since more than 6 months, for every AOP talk and every AOP related announcement (be it AspectWerkz or JBoss or AOP in general), they were at least one question about performance (no matter the context).
    We thus felt important to propose an open source effort in that sense, allowing both framework developpers (you and I etc) to measure and improve their layers, and users interested in that to measure by themselves.

    It would be even more interesting to include memory measurements for all of the frameworks in different scenarios. A simple -verbose:gc says a lot about object allocation going on under the hood, even when the "interceptor"/"advice" is only a singleton instance method incrementing one static int counter. This heap consumption is more difficult to measure effectively so is not yet part of the report.

    More rich instantiation models (perTarget, perCflow etc), and cflow pointcut would be valuable as well.
    If there are enough volunteers, I'll be pleased to move this project to a neutral place (java.net f.e.) and welcome one representant of each framework.

    Alex
  10. time to reconsider[ Go to top ]

    Alex,

    I think it is a great idea to continue this test suite and I am certainly happy to get involved on the Spring side of things. As I am sure you are aware there is a lot of tweaking that can be done to the Spring configuration to improve the performance, and as of the next release of Spring there will be even more in the way of options for tuning.

    Rob
  11. time not to reconsider[ Go to top ]

    [...] i was also looking into using [Spring] for its AOP capabilities. but now after reviewing these benchmarks as listed on the site.....i may have to reconsider. if i am reading it correctly Spring performed HORRIBLY compared to the rest of the AOP frameworks.
    No it didn't. Of the list, only AspectJ and AspectWerkz 2 are lots faster. They better be, because both (especially AspectJ) address a totally different use case. Whereas Spring AOP intends to be used for coarse-grained advice against business objects to implement aspects such as transactions and security, AspectJ and AspectWerkz target a completely different development style where AOP is used in a pervasive, fine-grained way. Their performance has to be an order of magnitude better. If that is what you want to do, Spring AOP, DynAOP, JBossAOP and so on are simply not for you.

    If, on the other hand, you want to use AOP for transactions and so on then, well, the slowest of them (AspectWerkz 1) takes 470ns for around advice. The code being advised will take at least 50,000 times that if it hits the database, or a few hundred times that if it hits a cache. From a performance point of view, the choice of AOP framework is totally and utterly immaterial.

    Finally, if your team or your managers are of a conservative disposition, it might be relevant that the best-performing frameworks necessarily use bytecode manipulation rather than plain old JDK 1.3 proxies.

     - Peter
  12. time to consider carefully[ Go to top ]

    [...] i was also looking into using [Spring] for its AOP capabilities. but now after reviewing these benchmarks as listed on the site.....i may have to reconsider.
    No it didn't. Of the list, only AspectJ and AspectWerkz 2 are lots faster. They better be, because both (especially AspectJ) address a totally different use case.
    AspectJ and AspectWerkz are intended for the full range from micro-fine to coarse-grained. So its not so much a different use case as a broader use case.
    If, on the other hand, you want to use AOP for transactions and so on then, well, the slowest of them (AspectWerkz 1) takes 470ns for around advice. The code being advised will take at least 50,000 times that if it hits the database, or a few hundred times that if it hits a cache. From a performance point of view, the choice of AOP framework is totally and utterly immaterial.

    I don't think this is right, because time on the server is a zero sum game. Any work one task forces the server to do impacts the overall performance of the server. So performance of the framework is critical on the server.

    I'm not trying to slam Spring, JBoss etc. We can expect their performance to improve in the future, and other technical issues may be more important at this stage of adoption. Note that even for AspectJ, the recent focus has been on speed of weaving more than flat-out runtime efficiency.

    But we do need to be careful about the performance arguments we make. Down the road, you aren't going to want to be using an AOP framework that doesn't have efficiency comparable (or better than) the best that AspectJ and AspectWerkz currently have to offer. Exactly when each kind of application needs to get that underlying AOP efficiency depends on a number of factors though.
  13. time to consider carefully[ Go to top ]

    They better be [faster], because both (especially AspectJ) address a totally different use case.
    AspectJ and AspectWerkz are intended for the full range from micro-fine to coarse-grained. So its not so much a different use case as a broader use case.
    With all due respect, get real. If all you need is coarse-grained aspects against business objects then only an absolute madman would invent or adopt a whole new language.
    From a performance point of view, the choice of AOP framework is totally and utterly immaterial.
    I don't think this is right, because time on the server is a zero sum game [...] performance of the framework is critical on the server.
    That's a purely academic and frankly irrelevant argument. In the real world, there's only so much time available to learn the technologies and implement a system. If you look at the figures above, time spent by Ben Possolo to learn and implement AspectWerkz 2 just for things like transactional and security aspects - and I'm not even talking about AspectJ - is never going to pay off over a dozen other things he could do with that time, such as checking his algorithms, tuning the database a little bit better or reducing object churn in his code. The importance of AOP in this particular zero sum game is miniscule, and his worries about these synthetic benchmarks are totally misplaced. It is pure Hoarean premature optimisation.

    Now don't get me wrong. AspectJ and AspectWerkz are hugely exciting and important, and where our usage of AOP becomes more sophisticated and pervasive their performance will be very relevant indeed. But this is a modus operandi that most of the others are simply not trying to address.

     - Peter
  14. time to consider carefully[ Go to top ]

    That's a purely academic and frankly irrelevant argument. In the real world [blah blah blah]
    I just realised that this looks like an attempt to dismiss you as "an academic". My apologies. I do not. I just feel that the impact of AOP on this particular zero-sum game is an academic issue for the developer: there are many far more important terms in the sum to worry about.

     - Peter
  15. time to consider carefully[ Go to top ]

    Peter.

    Have you ever used AspectWerkz?
    It does not extend the Java syntax, it is annotation and or xml defined plain Java.
    It is not in any way more complex and harder to integrate into J2EE environments than Spring AOP. It takes perhaps a little bit more time to learn (since there is more to learn), but when you know the API you can write your code both faster (much less code to write) and safer (no reflection but statically compiled) than using Spring AOP.

    /Jonas
  16. time to consider carefully[ Go to top ]

    No matter what you may think, the Spring team is already working on optimization and tunning options, and we will include that in AWbench and publish new results based on their input. You can already check for preliminary figures on Spring mailing lists.

    Aside, it is worth knowing that JBoss AOP evolved from a reflective based model to a statically compiled model before shipping their 1.0 final, and this is interesting when we know that their EJB 3 developper early preview is build on top of it.

    We could probably check on Geronimo and ask if they are using JDK proxy or CGlib proxy instead for their EJB 2 services, and ask why.

    It seems that everyone aims at minimizing the overhead wherever possible, even when ones have to use / introduce some new dependancies.
    Aside, once again, check the heap usage and garbage collection of a reflective based approach - or an approach wrapping arguments in object arrays every time when an advice gets executed vs a more rich approach with straitght access to the arguments (thru direct reference - see AspectJ args(..), this(..) and target(..) supported in AspectWerkz 2.x as well).
    And as Gregor said, those little differences in CPU or heap usage here and there will have an effect on the whole JVM.
  17. time to consider carefully[ Go to top ]

    or an approach wrapping arguments in object arrays every time when an advice gets executed vs a more rich approach with straitght access to the arguments (thru direct reference

    I have yet to write an aspect (probably same with Spring as well) where direct straight access to arguments is needed. All the aspects in our aspect library are generic and are meant to be used anywhere. Most of them don't need arguments, and the one that does (caching through field interception) needs the arguments untyped and wrapped as java.lang.Object or Object[].

    Our current EJB 2.1 architecture uses the reflective approach, and I have yet to see any benchmark of any application where performance was affected by these tiny memory allocations. J2EE applications tend to have bottlenecks in synchronization or with the database.

    Although I appreciated the benchmarks you've done, I think the AspectWerkz team needs to write a few more real-world aspects before they annoint which features of AOP are "most-used".

    Bill
  18. time to consider carefully[ Go to top ]

    Peter. Have you ever used AspectWerkz? It does not extend the Java syntax [...]
    I have - I was thinking of AspectJ there. AspectWerkz does not have anywhere near that impact. It's still a bit more hassle to set up, certainly if you're already using Spring. The simplicity and transparency of proxies is hard to beat.

    If your use cases are limited to some simple advice against business objects, like transaction management, I doubt that AspectWerkz is really much less code or much safer. Get more serious than that and the picture changes considerably: I don't think there's any disagreement there.

     - Peter
  19. time to consider carefully[ Go to top ]

    The simplicity and transparency of proxies is hard to beat.

    Proxy based approach is a fine one which is why the next version of JBoss AOP will have a proxy framework in place. Its mostly implemented and our current EJB3 implementation in CVS HEAD is actually built on top of it. EJB itself is a proxy-based framework so a proxy-based framework is needed to implement it. WE'll have full proxies, with the power of a poincut language behind it.

    Bill
  20. time to consider carefully[ Go to top ]

    Last reply. Promise :)
    But we do need to be careful about the performance arguments we make. Down the road, you aren't going to want to be using an AOP framework that doesn't have efficiency comparable (or better than) the best that AspectJ and AspectWerkz currently have to offer.
    This is true, if only because down the road we will all use AOP for so much more than just managing transactions. But until that time, if a "slow" AOP framework is just a little bit easier to learn or integrate, that will be enough to give it the edge over its rivals.

    Now if your point is that completely different dynamics apply to the Spring, JBoss etc development teams, that's of course absolutely right. They should, and do, spend significant time optimising the performance of the AOP framework and documenting the tweaks you can apply to it. But even so, the different use case addressed compared to, say, AspectJ, gives different priorities: for example, the choice to use JDK 1.3 proxies wherever possible has not exactly been made on the basis of their superior speed.

     - Peter
  21. time to reconsider[ Go to top ]

    well after being more than pleased with Spring's ease of use/integration with Hibernate in the financial web application i am currently working on, i was also looking into using it for its AOP capabilities. but now after reviewing these benchmarks as listed on the site.....i may have to reconsider. if i am reading it correctly Spring performed HORRIBLY compared to the rest of the AOP frameworks.

    Those measurements are in nanos.

    yotta zetta exa peta tera giga mega kilo hecto deca deci centi milli micro nano pico femto atto zepto yocto

    I would suggest that Spring AOP is more than fast enough for 99.9% of scenarios out there .. in fact, I was quite surprised to see all of the AOP implementations so close in performance.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Shared Memories for J2EE Clusters