Nine Language Performance Round-up: Benchmarking Math & File IO

Discussions

News: Nine Language Performance Round-up: Benchmarking Math & File IO

Bookmark and Share Get thread feed

  1. Nine Language Performance Round-up: Benchmarking Math & File IO (35 messages)

    This article discusses a small-scale benchmark test run on nine modern computer languages or variants: Java 1.3.1, Java 1.4.2, C compiled with gcc 3.3.1, Python 2.3.2, and the languages supported by Visual Studio .NET 2003. The benchmark tests arithmetic and trigonometric functions using a variety of data types, and also tests simple file I/O.

    Java (at least, in the 1.4.2 version) performed very well on most benchmark components when compared to the .NET 2003 languages. If we exclude the trigonometry component, Java performed virtually identically to Visual C++, the fastest of Microsoft's languages.

    Read more about the benchmark: Nine Language Performance Round-up: Benchmarking Math & File I/O
    Reply to this

    Threaded Messages (35)

  2. Who cares?[ Go to top ]

    Does anyone really care about these type of benchmarks anymore? I've never seen a case where one language was chosen over another because of how fast the arithmetic functions were, or how efficient the I/O libs were. If this was really a concern, we'd all be programming in C or (shudder) Assembly.

    What's really important is how efficiently I can create and maintain code. That means tools and frameworks are important. I guess because you can't put these real concerns in quantitative terms and create neat graphs, we keep seeing "benchmarks" like this.

    -Nathan
    Reply to this Reply to original

  3. Who cares? - Some people do![ Go to top ]

    I can give you plenty of situations where languages are chosen for their performance, obviously not on the back of a small benchmark like this but one of the main arguements for most banks not using Java in their pricing and risk engines is performance. I know this isn't true but that's the arguement I hear when proposing Java based technologies (Jini et al).

    -John-
    Reply to this Reply to original

  4. Who cares? - Some people do![ Go to top ]

    I can give you plenty of situations where languages are chosen for their performance, obviously not on the back of a small benchmark like this but one of the main arguements for most banks not using Java in their pricing and risk engines is performance. I know this isn't true but that's the arguement I hear when proposing Java based technologies (Jini et al).

    >
    > -John-

    I just don't buy that. In the case of financial institutions, they were scared about the reliability of interpreted code (or partially-interpreted in Java's case) over compiled code. They also had lots of legacy code to contend with.

    Now, they do care about pushing through thousands of transactions a second reliably. But, this is a factor of the database and the application framework around processing these transactions, _not_ the underlying programming language.

    -Nathan
    Reply to this Reply to original

  5. Performance matters[ Go to top ]

    <
    I just don't buy that. In the case of financial institutions, they were scared about the reliability of interpreted code (or partially-interpreted in Java's case) over compiled code. They also had lots of legacy code to contend with.

    Now, they do care about pushing through thousands of transactions a second reliably. But, this is a factor of the database and the application framework around processing these transactions, _not_ the underlying programming language.
    />

    Transaction processing as you describe is only one aspect of a bank's IT needs and I won't disagree with you as regards such systems. However there are also systems that need to calculate a bank's financial risks. Typically these systems are very computationally intensive, and not particularly transactional in nature.

    The users of such systems commonly include maths Ph.D.s who typically write C, and have little or no experience of Java. They worry a lot about code speed and usually think Java is slow (whatever the facts may be). Since they know more about IT than your typical user, their opinion on a system's design carries a lot of weight amongst the other users. If they don't agree with your design, you probably won't get the business to pay for it. In my experience having to make a case about Java's perfromance relative to C/C++ is quite common. News that the maths functions are so slow is not good as far as I'm concerned. Why is it so much slower than 1.3?
    Reply to this Reply to original

  6. Performance matters[ Go to top ]

    In my experience having to make a case about Java's perfromance relative to C/C++ is quite common. News that the maths functions are so slow is not good as far as I'm concerned.

    Totally. I can't even convince coworkers to accept Java somewhere in the rendering pipeline, let alone actual scientific processing upstream. The culture amongst so many of the e-scientists seems to be that since gcc and g77 emit fast code, there's no need for further technology consideration. My team lead and I get clobbered anytime we mention Java Native Interface as a way to bestow J2EE features upon legacy science libraries. It doesn't seem to matter to these folks that it took C++ 10 years to standardize a string class, or that Fortran 2000 is yet unfinalized in 2004. All that seems to matter to some subject matter experts is FLOPs, and Moore's Law be damned. Not even JavaGrande.org has changed this culture.
    Reply to this Reply to original

  7. Performance matters[ Go to top ]

    "doesn't seem to matter to these folks that it took C++ 10 years to standardize a string class"

    A small nit to pick, Java is not standardized by any "folks" and it doesn't look like it will ever be.
    Reply to this Reply to original

  8. Performance matters[ Go to top ]

    |
    |A small nit to pick, Java is not standardized by any "folks" and it doesn't
    |look like it will ever be.
    |

    Except for these folks. But who are they to matter...

    -Nick
    Reply to this Reply to original

  9. Performance matters[ Go to top ]

    Sun is a standards body a la ISO, ANSI, etc? That's news to me.
    Reply to this Reply to original

  10. Performance matters[ Go to top ]

    |
    |Sun is a standards body a la ISO, ANSI, etc? That's news to me.
    |

    Standard : "something established by authority, custom, or general consent"
    It doesnt have to come from a "standards body" to be a standard...

    -Nick
    Reply to this Reply to original

  11. Java float flawed?[ Go to top ]

    Java floating point arithmetic seriously flawed. Does anybody know if this still holds? I would not use Java in a mars probe until this has been corrected..

    //ras
    Reply to this Reply to original

  12. Java float flawed?[ Go to top ]

    Java floating point arithmetic seriously flawed.

    yop until JDK1.3 and introdcution of strictfp keyword.
     
    > Does anybody know if this still holds? I would not use Java in a mars probe until this has been corrected..

    see http://math.nist.gov/javanumerics/reports/pejfps.txt

    >
    > //ras

    remi
    Reply to this Reply to original

  13. Performance matters - so use Java[ Go to top ]

    The users of such systems commonly include maths Ph.D.s who typically write C, and have little or no experience of Java.

    Andy,
       You seem to have worked in the same banks as I. I agree totally, the main problem is the attitude of the PhD risk "guru" writing parallelised curve analysis or VaR calculations etc. They frequently use Fortran or C++ and their only knowledge of Java is often a slow running Applet they once saw in the 90s.

    Very few of the High Performance Computing (HPC) projects we worked on in banks have needed a database other than for static data. Typical engines have over 100 CPUs, we have worked in several of the world's largest banks (US and Europe) and we are finally starting to see an up take in Java. It's easier to manage, easier to program, more reliable than C++ due to better memory management, easier to maintain etc. etc. There are a lot of serious advantages to using Java in HPC.

    I am very curious about the trig results, anyone got any ideas yet?

    -John-
    Reply to this Reply to original

  14. Who cares? - Some people do![ Go to top ]

    A pricing engine doesn't necessarily deal with database connectivity. It's a number-cruncher, so performance outside of IO and dbs is important.
    Reply to this Reply to original

  15. Who cares? - Some people do![ Go to top ]

    I can give you plenty of situations where languages are chosen for their performance, obviously not on the back of a small benchmark like this but one of the main arguements for most banks not using Java in their pricing and risk engines is performance. I know this isn't true but that's the arguement I hear when proposing Java based technologies (Jini et al).


    It would be unfortunate if decision-makers used this kind of data in isolation. A language's performance is more than the time to execute a mathematical instruction --- even for math-intensive software, there are still optimisations that are facilitated by a higher-level language which allows more sane reasoning --- to quote Donald Knuth, "Premature optimization is the root of all evil".

    And don't forget Moore's law --- performance continues to double every eighteen months. In the study, Java is roughly half the speed of Visual C++ (by total). Even if you naievely take that completely at face value, it's a difference of eighteen months by prevailing Moore's law reasoning. If you could tolerate C++ performance in mid-2002 (and for typical applications, you probably could), then you can tolerate Java performance early 2004.
    Reply to this Reply to original

  16. Who cares? - Some people do![ Go to top ]

    I have a bond yield calculator that I use for benchmarking "real" math algorithm performance between Java and C. The algorithms are exactly the same in both languages. Even with the latest Java from Sun, it is actually 5 times slower than compiled C.

    In the financial industry, this is particularly important when running thousands of simulations on complex asset structures. It allows us to (1) get products to market quicker or (2) run more simulations in a finite time frame to provide better assessments.
    Reply to this Reply to original

  17. Who cares?[ Go to top ]

    Performance still plays a role, especially when dealing with encryption, SSL etc. and nice to know that java runs comparably with the C++. It wood be much worse if it doesn’t.
    Reply to this Reply to original

  18. Who cares?[ Go to top ]

    I've never seen a case where one language was chosen over another because of how fast the arithmetic functions were...

    You obviously don't work in e-science, as I do. My colleagues mostly laugh at Java. But wide area networking is becoming more important to e-science, so scientists are gradually warming to Java.
    Reply to this Reply to original

  19. Who cares?[ Go to top ]

    I've never seen a case where one language was chosen over another because of how fast the arithmetic functions were...

    >
    > You obviously don't work in e-science, as I do. My colleagues mostly laugh at Java. But wide area networking is becoming more important to e-science, so scientists are gradually warming to Java.

    No, I don't. But, that's a tiny fraction of the entire software developement industry.

    Anyway, if you're working in a field like this, you're probably not going to look at these type of benchmarks anyway... you're going to run test cases for the specific code you need to optimize.

    -Nathan
    Reply to this Reply to original

  20. Who cares?[ Go to top ]

    No, I don't. But, that's a tiny fraction of the entire software developement industry.


    Actually, there are a lot of financial applications(asset allocation,modelling,analytics,...) which require heavy computations and are a large part of financial and insurance applications. Number crunching is becoming more and more important as systems go towards real-time processing.

    I agree with you on benchmarks, for the most part they are worthless. However, it is interesting to see that Java is comparable in computing primitive operations with the more "native" C++/C# but is terrible with trig functions. This seems more like a poor implementation of the trig library versus java itself.

    The real problem with java is with precision. The language needs to adopt a long double datatype to address alot of precision computing issues. This is probably more of an issue for the e-Science problems mentioned, but I'm seeing it crop up in the financials also.

    - Frank Bolander
    Reply to this Reply to original

  21. Nine Language Performance Round-up: Benchmarking Math & File IO[ Go to top ]

    Much that I hate benchmarks this didn't try to draw any earth shattering conclusions from the results. It's a shame the author didn't use his knowledge of languages like Ruby and Perl to extend the benchmark to those too, for fun of course.

    I'm rather intrigued about Java's trig results given such excellent performance in the rest of the field. It would be interesting to try with the new (alpha) JDK 1.5 but of course we couldn't publish the results yet.

    -John-
    Reply to this Reply to original

  22. What about XML processing?[ Go to top ]

    My question is, how does Java compare with C/C++ when it comes to processing large XML files? It would seem to me that if you are doing heavy-duty XML processing it might make sense to use C/C++ routines to do the heavy lifting in order to enhance performance.
    Reply to this Reply to original

  23. What about XML processing?[ Go to top ]

    "It would seem to me that if you are doing heavy-duty XML processing it might make sense to use C/C++ routines to do the heavy lifting in order to enhance performance."

    What, are you mad? :-) It might make a slightly faster SAX parser but what about JAXB etc. You really have to be on a different planet to use something like C++ for XML parsing, even c# would be better and I'm a Java fan. Java based XML binding has got to be the way to go.

    (Don't take it personally please, there's a smiley in there)

    -John-
    Reply to this Reply to original

  24. Spread the word.[ Go to top ]

    Nice article if you want to convince someone that Java is fast. Just show him the results and tell him that his application doesn't use trigonometry at all (most of the time this will be applicable). Of course that is not the whole truth, but who cares. The other side is cheating, too. Everything about benchmark results (especially micro benchmark results) has been said already in this forum.
    Reply to this Reply to original

  25. Why is the Trig functions so slow in 1.4.2??[ Go to top ]

    Ok, so does anyone have any idea why the author got the excessively slow results for the trig functions?
    Reply to this Reply to original

  26. why is it so slow?[ Go to top ]

    I had a quick test of how fast the Java test would be using native methods for the trig functions. It was a quick hack and I didn't do any error trapping in the code, so I'm not sure my results need a lot of caveats. The trig trest ran on my PC (jdk 1.4.2_02) in 80 secs. Using the native code reduced that to 6 secs.
    Reply to this Reply to original

  27. slow trig bench[ Go to top ]

    The trig section of the benchmark is slow because of what seems to be slow implementaion of the trigonometric functions in the Hotspot VM. IBM's VM seems to have similar perf characteristics. Bea's VM performs a lot better on the trig stuff.

    As other people mentioned, this is a rather silly microbenchmark. What you can see though is that languages compiled to bytecodes (or similar) are on par with traditional compiled C/C++, regardless if you prefer VB, C# or Java.

    /F
    Reply to this Reply to original

  28. Only tangent is slow[ Go to top ]

    Here are the results broken down by function for 10M iterations

    1516ms sin
    1468ms sin
    28984sec tan....
    4375ms log
    250ms sqrt
    Reply to this Reply to original

  29. Platform Independent Results[ Go to top ]

    An article at JavaWorld on JNI several months ago raised the issue of math performance using JSDK 1.4. Author Jeff S. Smith noted that Sun now uses a new math package that gurantees identical math results across all platforms.

    Smith's article can be found at:

    http://www.javaworld.com/javatips/jw-javatip141.html

    This probably answers the question as to why trigonometric performance degraded from JSDK 1.3 to 1.4.
    Reply to this Reply to original

  30. this is what the result says[ Go to top ]

    C# 65.3, Java 103.1 when Java is allocated 3-4 times as much memory to run.

    That is good enough for me but what is even more interesting is what is the differences in speed when both languages has the same amount of memory.

    Regards
    Rolf Tollerud
    Reply to this Reply to original

  31. Fast Java Matrix Operations?[ Go to top ]

    These benchmarks give me hope. However, what I'd really like is some benchmarks of floating point matrix operations. We currently have a prototype system doing heavy matrix operations in Java, but it's slower than we'd like. Pointers from other readers on fast Java sparse matrix libraries would help us avoid converting this portion of our code to C++.

    Thanks in advance.
    Reply to this Reply to original

  32. this doesnt deserve to be on TSS[ Go to top ]

    Benchmarking is such a big thing where in a small parameter can cause a huge difference in the final results.
    This research is so immature and the author doesnt address so many details that "This thread should just be removed from TSS"
    Reply to this Reply to original

  33. Nine Language Performance Round-up: Benchmarking Math & File IO[ Go to top ]

    Trig Results have more than doubled from Java1.3 to 1.4. I dont remember anything drastically changed in Math Libs from 1.3 to 1.4 or may be this is some joke
    Reply to this Reply to original

  34. Nine Language Performance Round-up: Benchmarking Math & File IO[ Go to top ]

    Didn't the last set nanobenchmarks posted on slashdot also show 1.4 trig was much slower than 1.3? 1.3 is buggy on x86 as trig functions return incorrect results for large values. The slowness come from software normalisation of numbers bigger than pi. There shouldn't be a problem on Sparcs, PPC, etc. I didn't find any particular problem with the tan performance on my lowly PII.

    They were a pretty meaningless bunch of benchmarks. Most of them loop over:

    x = (x+1)*(i+2)/(i+3);
    i += 4;

    Mostly that's an integer divide then. Many moons ago someone posted to comp.compilers a list of the frequency basic operations were performed by general software. Integer divide was a tad infrequent.

    The file I/O benchmark was a bit poor. The C version was just reading and writing byte arrays, while Java was doing character set encoding/decoding. Not that Java blocking I/O is particularly fast.
    Reply to this Reply to original

  35. Nine Language Performance Round-up: Benchmarking Math & File IO[ Go to top ]

    Many moons ago someone posted to comp.compilers a list of the frequency basic operations were performed by general software. Integer divide was a tad infrequent.

    No one is disputing that Java performs well for general purpose. We're pondering Java's ability to crunch numbers in e-science and finance. This matters if heterogeneous grids are to succeed. It also matters if scientists are ever to be free of the perils of C++ debugging.
    Reply to this Reply to original

  36. Explanation from Sun[ Go to top ]

    A decent explanation of the cause of the 1.4 StrictMath performance "degradation" can be found in the Bug Database:

    http://developer.java.sun.com/developer/bugParade/bugs/4857011.html
    Reply to this Reply to original