JXInsight 5.7.27 Released – 1 Billion Operations Per Second

Discussions

News: JXInsight 5.7.27 Released – 1 Billion Operations Per Second

  1. The twenty-seventh update to JXInsight 5.7 has been published. In this release we have made a prodigious gain in reducing the runtime overhead incurred in the execution of probe instrumented methods that are deemed (dynamically at runtime) to not be hotspots. With our new set of complete coverage dynamic probe extensions the overhead per method call has dropped down to 1 nanosecond. Effectively we can now execute 1 billion instrumented fine grain (no-op) method calls in one second (once warmed up). Most importantly what is deemed a hotspot (and not a hotspot) is entirely under your control with the ability to choose which meter (including custom meters), which metering statistic, what threshold and at what point in time of a method’s workload should the assessment be made.

    Threaded Messages (21)

  2. The link: http://blog.jinspired.com/?p=655
  3. Are you only going to give us information about how great JInspired is, or are you also going to tell us how you did it? :) Atm I do quite a lot with instrumentation for Multiverse ( http://multiverse.googlecode.com ), a Java based STM implementation. And one of the things I need in the future is transaction level statistics (performance, failure rates, conflicts etc). So more information about a low overhead measuring technique is always welcome.
  4. Our approach to continuously improving the performance of the product is based on the same principle (in fact technology and tooling) that we offer in the product - custom resource metering. http://williamlouth.wordpress.com/2009/05/04/execution-profiling-counting-kpis/ Whereas most applications can be optimized for performance and scalability using standard resource meters like cpu.time and clock.time we use the following: field.read.count field.write.count field.volatile.read.count field.volatile.write.count call.count execution.count and we aggregate them into a single custom cost meter which applies weighting (a standard cost table) to each of the base primitive meters (counters) which are tracked at the thread level and correlated with each (chained) activity fired and metered. Obviously we cannot use a native time based meter to profile our own code because the cost of actually getting the underlying counter is greater than 100 nanoseconds (two paired calls) on most standard platforms. We then meter the various execution paths (activities) performed by our runtime effectively (metering the metering) looking for ways to re-arrange the cost structure and using heuristics (extremely important). In our case the cost structure equates close enough to the field access & method call behavior and logic (ordering). How we have managed to make the substantial gains in performance lately? We have coupled this (obsessive and all consuming) approach with that which is described in the On Intelligence book introducing bi-directional communication between the instrumentation layer and the underlying metering (hotspot strategy) layer - short circuiting the flow as early as possible and using predictive cache strategies.
  5. This is really, really impressive. Very cool.
  6. Thanks Joe. I never felt we could go lower than 10 nanoseconds on standard hardware but I was pushed in a recent thread on TSS to "come up with the goods" even though the product had already surpassed every other product in the field by a huge margin. I worked flat out for a number of days & nights (dream mode) getting as close as possible to 10 ns and then I had a flash of brilliance which got us to here though there were some moments between the brilliance and the final solution that proved equally as challenging but that is what makes what we do so special and rewarding. William
  7. Ah, thanks for reminding me how meaningless vendor benchmarks are.
  8. Meaningless? You mean just like all the other vendors in this space claiming to have zero overhead when in fact any method interception will result in anywhere between 660 ns and 6,000 ns. Oh they forgot to mention that this is only applicable if your application is dog slow (which is highly likely if you are talking to them) and that you only apply their instrumentation at coarse grain execution boundaries such as a http request entry point and a database/sql exit point. We have consistently stated our metering overhead for non-hotspots and hotspots and each figure has dropped substantially release after release and in every comparison that has been made out performs by them all by huge margin. Benchmarks do in fact serve a useful purpose to the community and those that discount them [so flippantly] fail to see that with every published data point there is something to learn and something to gain. Benchmarks encourage the improvement in each of the implementations as trickery can only really get you so far in the race. We have not lost a single benchmark shout-off in a proof-of-concept so there must be some element of truth. Does everyone need this extreme low level of overhead. Unfortunately no as the performance & scalability behavior of most business applications [created by companies not in the business of creating software] is dismal and plagued with low hanging fruit which is why some people have managed to improve performance by simply request a thread stack dump. Fortunately there are some large financial companies that do require extreme low latency in requesting processing were time is money. In addition the wave of the future, cloud computing, is going to see a marriage of performance management and cost management at the micro (payment) level.
  9. And other products?[ Go to top ]

    How does JXInsight compare to other products? - to 'lowlevel' profilers like JProfiler (I use it for Multiverse and it rocks). - to high level performance monitoring environments like Dynatrace, Wiley Intrascope etc. And another question about the performance: to track performance you need to publish data. How is this data published without causing too much memory bus traffic? If you use AtomicLongs for example to store information, you won't get far. I have a 8 processor system (2 xeon 5500 cpu's) and the maximum increase rate I get on my machine is in the tens of millions and not in the billions.
  10. Re: And other products?[ Go to top ]

    I am not sure I could really do a feature-by-feature comparison in a single posting. But I will say that our product technologies (probes, traces, transact, diagnostics, insight, metrics) cover the complete spectrum (below and above those listed as well as in the middle) and all are extremely extensible in terms of what and how things are measured. To be frank any new feature that has appeared in these products lately and claimed to be "revolutionary" or "innovative" has been in JXInsight (or JDBInsight) already for years. We were the first to do distributed tracing, contextual (annotated) tracing, transactions paths, wait/blocks/gc/alloc method level stats, ...... We were also the first to create an extensible multi-resource chained strategy metering & costing model & runtime based on activity based costing principles that is applicable in and outside of the cloud and across languages and programming constructs. Not to mention none of these has every come close to matching the look and feel of our product and the quality of our information visualizations. But of course I would say that but you are more than welcome to prove me wrong. William
  11. Re: And other products?[ Go to top ]

    How is this data published without causing too much memory bus traffic? Read the announcement again. Hotspot and Non-hotspot. For any performance management & monitoring runtime the goal should be to only measure what is relevant short circuiting any future firing & metering of a particular named probe as the cost of accessing any underlying counter is huge (in comparison) never mind updating data structures. But how does determine what is hot and not (as it is contextual to the execution behavior) and how does this information get to the appropriate in the execution that ensures the smallest of costs.
  12. Re: And other products?[ Go to top ]

    Standard data collection approaches used by performance monitoring solutions 1. [Standard] Purchase a product that comes with a set of instrumentation templates for technologies such as JDBC, JMS, Servlets,.... that generally exhibit high overhead in most enterprise applications. If the problem is elsewhere your in trouble. If the implementation of the technology is somewhat different (i.e. in-memory) the vendor is in trouble. Note: We ship with 700+ technology specific extension jars (deployment config module) for probes, traces, transact, diagnostics, insight, metrics.... 2. [Blindly] Use existing support within a framework to measure and collect data: (listeners, interceptors, filters) in a rather ad-hoc and unstructured manner. This fails in practice because the developer generally has insufficient knowledge of the production environment, software execution model, workload characteristics, execution costs, and performance penalties of any instrumentation applied. 3. [Measure] Continuously refine the instrumentation and overhead based on previous [configurable and dynamic] hotspot analysis. Instrument->Measure->Refine->Repeat http://williamlouth.wordpress.com/2009/04/06/instrument-measure-refine-repeat/ The only problem is that you might have missed (failed to generate) a hotspot in your performance load/stress test environment. In addition changes in production will invalidate to some degree the non-hotspot and hotspot probes. 4. [Dynamic] Reduce the overhead down to levels (1 ns) that allow as much coverage in production as possible ensuring that any new hotspots (again this is configurable) that arise due to changes in the application code base, system and services, or workload characteristics. Our product allows you to do all 4 (together) via our extensive set of extension libraries, AspectJ aspects library SDK, Open API (for each technology), tooling (management console) and runtime (dynamic & chained metering strategies).
  13. Hotspot definition[ Go to top ]

    Hi William, you did a nice work with JInspired! As I understand you "mark" instrumented methods that you categorize not to be hotspot methods and then decrease the overhead to 1ns for these methods. This allows the user to do a "shotgun instrumentation" and JInspired will automatically reduce the overhead of these methods at runtime (probably after a special amount of calls?), so that only "interesting" methods get measured at runtime which then generates approx. 600ns overhead per method. Did I understand this right? This would be a nice feature as my experience is that refining the instrumentation is very time consuming and often difficult because even with dynamic bytecode instrumentation it can take a while to reinstrument the classes. A short statement to your comments: Some years ago I attended a partner event of a big Monitoring/Diagnosis tool vedor (who has been bought by a bigger company for some hundred million $ later on) and they said, that we should never offend competitors, by claiming that theit tool has less overhead, better features etc. Why? Because it puts you in a bad position. It makes you vulnerable and in some cases you even give the customer the idea to contact the competitor to verify if your statement is correct - this gives the sales every chance to counter your statement. And in reality it is very hard to measure overhead etc. in a shoot out... Just think about it. Mirko
  14. Re: Hotspot definition[ Go to top ]

    Hi Mirko, Yes in part though the devils in the details as our probes can represent any abstract execution construct and not just a Java method as named probes are basically hierarchical cost/metering structures. Because our runtime is no way limited in what constitutes an actual resource meter there is no straight forward answer to what is a hotspot or not (again it need not be a method) and the user has a lot of flexibility in defining this especially as they can register their own resources (meters), create composite chained metering strategies which alter the population of a one or more metering statistics, and install various probe provider extensions which focus the firing and metering on particular (probe) execution behavioral patterns including concurrency levels, re-entrants, baseline deviations,..... With regard to your last statement you should probably stop hanging around with your partners (our competitors) who are more focused on selling expensive & poor imitations of our product and technologies. If it was all just about the money which is largely the case for your friends then I personally would be doing something entirely different. Who honestly likes to sell to IT people? I like the challenge and its always rewarding knowing that no matter how big the competition is or gets every day you out-perform and out-class them in terms of creativity, innovation, design and development. Not to forget that the product is used and extended daily by ourselves (which is why it is so different than others) in providing "war council" (troubleshooting) services to many of the top companies in the world when all the experts have failed to identify and classify a problem in production. Nothing is more rewarding than flying into a customer site and identifying the underlying issue(s) within a matter of hours if not minutes (in some cases) with our technology. I am sure there will be some marked improvements in other offerings but any sort of cloning this time around will cost your partners. It is funny that you talk about getting sales involved because a large part of the sales team at your present partner came from another (acquired) company were they stated they had less than 5% overhead only to later restate that to be 50% when they joined your partner were they stated the new technology had an overhead of less than 5% (which we both know to be nonsense). Sales people will say anything and if someone believes them without testing it then they probably have more money and sense and best to be avoided (unless of course you just want to make money).
  15. Partner[ Go to top ]

    Hi William, thank you for your reply and the answer to my question - I think this is really a very useful feature for troubleshooting and performance analysis. We have a lot of partners - starting with Quest, CA/Wily, dynaTrace and ej-technologies (JProfiler). And even our partner SpringSource is now in the Performance market by aquiring Hyperic. All of these tools have their strength and weaknesses. We are an indipendent consultant which we proof by not being a reseller and by having best of breed partners. Our main focus is also on troubleshooting and I can just say that I also enjoy the moment when you identify performance problems at customer site. I am constantly following the development of you product and I can just say that you are doing a great job and that you are technically a very clever guy and in the performance area for sure one of the best in the world. What I cannot undestand is that such a clever guy does statements like "If it was all just about the money which is largely the case for your friends" or "who are more focused on selling expensive & poor imitations of our product and technologies" than you a) think that this is great marketing b) you really think this is true. Probably it is b) because it is your personal product. But the reality is somewhat different. There are a lot of tools out there which are much older than your tool and much more innovative like: - Wily invented a lot of stuff around JVMTI (JSR-163) which you are using extremely in your product for sure. - Monitoring Specification by IBM (JSR-174) which is included in Java 5 and you will probably use in your product. - Where is your name on JSR-326? Which will be an important standard for future trouble shooting of memory leaks in production. - Sitraka had distributed tracing long before JDbInsight existed -> you know PerformaSure? If not you shouldn't say that you have been the first having it... - etc. etc. etc. By the way how many people has your company? How many engineers do you have? How many support guys for your tool? Show me the documentation of your tool. No, no not the blog please :-) So, I don't want to attack you, as I really like your ideas and your tool. I hope you once hire a marketing/communication guy and maybe a sales guy and you focus on engineering and troubleshooting - this would be great for your tool and competition in this area. I would really like to have you at our site to present your ideas to our performance specialists at codecentric (we are just 250km away from you). We have also our "meet the experts" (http://www.meettheexperts.de/) where you could proof your statement to more than 60 customers. Last time Kirk Pepperdine showed how to fix performance problems with just JVM internal tools :-) Mirko
  16. Re: Partner[ Go to top ]

    Mirko you are wrong. Here is what I said:
    We were the first to do distributed tracing, contextual (annotated) tracing, transactions paths, wait/blocks/gc/alloc method level stats, ...... We were also the first to create an extensible multi-resource chained strategy metering & costing model & runtime based on activity based costing principles...
    We had distributed trace analysis [in java] before PerformaSure as I wrote it for Borland VisiBroker while working as a technology advisor to the Borland & Inprise & Borland. I know the history as Sitraka made a play to buy our before we even shipped the first release (prior to their acquisition by Quest). Our trace analysis also did not need a central management server and the trace was "contextual" and stack based. Sitraka used a simple tag based approached whereas JDBInsight's agents could be completely disconnected from each other but still render the contextual trace stack from the client as we have a distributed and partition database technology under our snapshot facility. Maybe I should have been clearer "distributed contextual tracing" but I always thought this was the only meaningful way. By the way Sitraka initially only supported HTTP[browser]-Java tagging. JDBInsight was the first JDBC profiling solution. The first (and still the only one) to do true resource transaction path analysis. Wily introduced its primitive SQL Agent add-on after we released our product. It is still a pretty dismal product in this regard. JDBInsight was the first to publish monitoring waiting/blocking times at the contextual trace and transaction levels as well as gc time, alloc count, alloc bytes and clock adjusted (less gc, waiting, blocking). This was 4-5 years before dynaTrace made its "revolutionary" announcement about such metrics. JXInsight is the only solution on the market with "an extensible multi-resource chained strategy metering & costing model & runtime based on activity based costing principles...". Regarding JSR 326: Steve Poole actually contacted me a year before this specification was made public. We discussed it at length [in person] and with the rest of his JVM support team whilst I was trying to push for IBM to support our Probes Open API as a JSR. We had a difference of opinion. I felt our Diagnostics Open API solution was the right way to approach the problem and would replace the use of JMX in a number of cases especially with regard to runtime object graph state inspection. Steve coming from a support background was more focused on after death analysis whereas I was more focused on state inspection in real-time. I published a short blog entry yesterday on this particular aspect of our product. Dumping OSGi – SpringSource dm server http://williamlouth.wordpress.com/2009/07/30/dumping-osgi-springsource-dm-server/ Maybe you should go and work full-time for your partners because you seem to be doing their work for free. If you want to see the documentation all you need do is send me a purchase order. Please next time can you at least state your interests (before mouthing off your dribble).
  17. Re: Partner[ Go to top ]

    [JSR-163] Robert Field the specification lead refused our entry after I stated in an email that I would like the specification to look at integrating resource metering and billing into the runtime via contextual cost center tagging. Yes before even cloud computing was being talked about I was already pitching for this to be a standard. The JRockit team can vouch for this as in 2003 [6 years ago] they flew me over to Stockholm to discuss the possibility of leading the tools team. I went in with my grand vision of the future of grid computing based on collective JVM's with costing/billing & automatic provision inherently supported. I completely scared the team off but eventually when the dust settled a few years later they announced Liquid VM which largely failed. William
  18. My interests[ Go to top ]

    Hi William, just to clarify my interests: I am just interested in Java performance topics. Actually we have transformed into a company that does more software development based on agile methodologies. Troubleshooting is still a service of us, but as I stated above: We are not a reseller of any partner and we've never sold any performance tool directly to a customer. We are using the tools of our partners in our services and in many cases (because we have a 100% success rate) they will buy the tools later on - but this has nothing to do with our business model... So my comments are not in the name of any of our partners nor in the name of my company. This are just my personal thoughts. And as this is an open and free community I am happy that I can reply to your thoughts. And as I said: I would really like to deepen our discussion in a personal way.
  19. Re: Partner[ Go to top ]

    Mirko you are wrong [...]
    Maybe you should go and work full-time for your partners because you seem to be doing their work for free.

    If you want to see the documentation all you need do is send me a purchase order.

    Please next time can you at least state your interests (before mouthing off your dribble).
    You represent the kind of company I would never do business with. *MEGAPLONK*
  20. I worked flat out for a number of days & nights (dream mode) getting as close as possible to 10 ns and then I had a flash of brilliance which got us to here though there were some moments between the brilliance and the final solution that proved equally as challenging but that is what makes what we do so special and rewarding.
    I'll attest that you are definitely stubborn and crazy, and more than a little bit brilliant and cantankerous ;-) It's good to see you continuing to push the technology boundaries with JXInsight. It is hard to understand that single-minded (insane) pursuit of something so intangible unless you've been bitten by the "do the impossible" bug at some point in the past. Congratulations on the release. Peace, Cameron Purdy | Oracle Coherence http://coherence.oracle.com/
  21. Getting back on track the importance here is not just one of speed it is of coverage. Most other application performance management solutions introduce a limited amount of instrumentation into an application's code base because of their overhead both in terms of instrumentation & measurement. Typically they target well known technology bottleneck execution points such as database, messaging, and remoting interactions. Unfortunately this means that a large amount of the actual application code base is not covered so if you have a problem (performance or scalability) out side of the these bottleneck points your are out of luck or likely to have a prolonged problem resolution phase. With JXInsight you can now just about instrumented every aspect of an application runtimes execution including new technologies and still have low overhead whilst ensuring you have adequate risk cover because you are now delegating the performance bottleneck discovery to JXInsight's activity-based resource metering runtime with its hotspot detection strategy which can be augmented with additional strategies. This is very similar to what is performed by the Sun hotspot JVM.
  22. I have published a blog entry showing how the same technology performs across dynamic languages executing within a JVM runtime. This also demonstrates our de-coupling from any particular execution construct which is not supported by any other commercial Java product I know of. The Fastest Ruby Profiler is a Java Profiler http://williamlouth.wordpress.com/2009/08/05/the-fastest-ruby-profiler-is-a-java-profiler/