This article discusses a small-scale benchmark test run on nine modern computer languages or variants: Java 1.3.1, Java 1.4.2, C compiled with gcc 3.3.1, Python 2.3.2, and the languages supported by Visual Studio .NET 2003. The benchmark tests arithmetic and trigonometric functions using a variety of data types, and also tests simple file I/O.
Java (at least, in the 1.4.2 version) performed very well on most benchmark components when compared to the .NET 2003 languages. If we exclude the trigonometry component, Java performed virtually identically to Visual C++, the fastest of Microsoft's languages.
Read more about the benchmark: Nine Language Performance Round-up: Benchmarking Math & File I/O
-
Nine Language Performance Round-up: Benchmarking Math & File IO (35 messages)
- Posted by: Jamie Schiner
- Posted on: January 08 2004 23:12 EST
Threaded Messages (35)
- Who cares? by Nathan Zumwalt on January 09 2004 10:32 EST
- Who cares? - Some people do! by John Davies on January 09 2004 10:40 EST
-
Who cares? - Some people do! by Nathan Zumwalt on January 09 2004 12:03 EST
-
Performance matters by Andy Pearson on January 09 2004 01:48 EST
-
Performance matters by Brian Miller on January 09 2004 03:13 EST
-
Performance matters by Todd Murray on January 10 2004 11:51 EST
-
Performance matters by Nick Minutello on January 11 2004 10:22 EST
-
Performance matters by Todd Murray on January 12 2004 02:32 EST
- Performance matters by Nick Minutello on January 13 2004 08:53 EST
-
Performance matters by Todd Murray on January 12 2004 02:32 EST
-
Performance matters by Nick Minutello on January 11 2004 10:22 EST
-
Java float flawed? by Test Dummy on January 12 2004 07:35 EST
- Java float flawed? by Remi Forax on January 12 2004 02:18 EST
-
Performance matters by Todd Murray on January 10 2004 11:51 EST
- Performance matters - so use Java by John Davies on January 09 2004 08:04 EST
-
Performance matters by Brian Miller on January 09 2004 03:13 EST
- Who cares? - Some people do! by Drew McAuliffe on January 09 2004 03:27 EST
-
Performance matters by Andy Pearson on January 09 2004 01:48 EST
-
Who cares? - Some people do! by Michael Mahemoff on January 09 2004 02:02 EST
- Who cares? - Some people do! by John Walker on January 14 2004 02:13 EST
-
Who cares? - Some people do! by Nathan Zumwalt on January 09 2004 12:03 EST
- Who cares? by Vladimir Goncharov on January 09 2004 10:48 EST
- Who cares? by Brian Miller on January 09 2004 11:46 EST
-
Who cares? by Nathan Zumwalt on January 09 2004 11:54 EST
- Who cares? by Frank Bolander on January 09 2004 12:26 EST
-
Who cares? by Nathan Zumwalt on January 09 2004 11:54 EST
- Who cares? - Some people do! by John Davies on January 09 2004 10:40 EST
- Nine Language Performance Round-up: Benchmarking Math & File IO by John Davies on January 09 2004 10:37 EST
- What about XML processing? by George Lawniczak on January 09 2004 11:16 EST
- What about XML processing? by John Davies on January 09 2004 11:23 EST
- Spread the word. by Michael Mayr on January 09 2004 11:22 EST
- Why is the Trig functions so slow in 1.4.2?? by Martin Devlin on January 09 2004 13:12 EST
- why is it so slow? by Andy Pearson on January 09 2004 13:54 EST
- slow trig bench by Fredrik Sandberg on January 09 2004 17:11 EST
- Only tangent is slow by damian penney on January 09 2004 06:36 EST
- Platform Independent Results by Joseph Haller on January 12 2004 02:36 EST
- this is what the result says by Rolf Tollerud on January 09 2004 13:31 EST
- Fast Java Matrix Operations? by Dan Greening on January 09 2004 14:45 EST
- this doesnt deserve to be on TSS by sean decor on January 09 2004 14:45 EST
- Nine Language Performance Round-up: Benchmarking Math & File IO by hthjf fgfgfg on January 10 2004 12:38 EST
- Nine Language Performance Round-up: Benchmarking Math & File IO by Tom Hawtin on January 11 2004 14:59 EST
- Nine Language Performance Round-up: Benchmarking Math & File IO by Brian Miller on January 12 2004 12:39 EST
- Explanation from Sun by Jim Cakalic on January 14 2004 03:57 EST
- Nine Language Performance Round-up: Benchmarking Math & File IO by Tom Hawtin on January 11 2004 14:59 EST
-
Who cares?[ Go to top ]
- Posted by: Nathan Zumwalt
- Posted on: January 09 2004 10:32 EST
- in response to Jamie Schiner
Does anyone really care about these type of benchmarks anymore? I've never seen a case where one language was chosen over another because of how fast the arithmetic functions were, or how efficient the I/O libs were. If this was really a concern, we'd all be programming in C or (shudder) Assembly.
What's really important is how efficiently I can create and maintain code. That means tools and frameworks are important. I guess because you can't put these real concerns in quantitative terms and create neat graphs, we keep seeing "benchmarks" like this.
-Nathan -
Who cares? - Some people do![ Go to top ]
- Posted by: John Davies
- Posted on: January 09 2004 10:40 EST
- in response to Nathan Zumwalt
I can give you plenty of situations where languages are chosen for their performance, obviously not on the back of a small benchmark like this but one of the main arguements for most banks not using Java in their pricing and risk engines is performance. I know this isn't true but that's the arguement I hear when proposing Java based technologies (Jini et al).
-John- -
Who cares? - Some people do![ Go to top ]
- Posted by: Nathan Zumwalt
- Posted on: January 09 2004 12:03 EST
- in response to John Davies
I can give you plenty of situations where languages are chosen for their performance, obviously not on the back of a small benchmark like this but one of the main arguements for most banks not using Java in their pricing and risk engines is performance. I know this isn't true but that's the arguement I hear when proposing Java based technologies (Jini et al).
>
> -John-
I just don't buy that. In the case of financial institutions, they were scared about the reliability of interpreted code (or partially-interpreted in Java's case) over compiled code. They also had lots of legacy code to contend with.
Now, they do care about pushing through thousands of transactions a second reliably. But, this is a factor of the database and the application framework around processing these transactions, _not_ the underlying programming language.
-Nathan -
Performance matters[ Go to top ]
- Posted by: Andy Pearson
- Posted on: January 09 2004 13:48 EST
- in response to Nathan Zumwalt
<
I just don't buy that. In the case of financial institutions, they were scared about the reliability of interpreted code (or partially-interpreted in Java's case) over compiled code. They also had lots of legacy code to contend with.
Now, they do care about pushing through thousands of transactions a second reliably. But, this is a factor of the database and the application framework around processing these transactions, _not_ the underlying programming language.
/>
Transaction processing as you describe is only one aspect of a bank's IT needs and I won't disagree with you as regards such systems. However there are also systems that need to calculate a bank's financial risks. Typically these systems are very computationally intensive, and not particularly transactional in nature.
The users of such systems commonly include maths Ph.D.s who typically write C, and have little or no experience of Java. They worry a lot about code speed and usually think Java is slow (whatever the facts may be). Since they know more about IT than your typical user, their opinion on a system's design carries a lot of weight amongst the other users. If they don't agree with your design, you probably won't get the business to pay for it. In my experience having to make a case about Java's perfromance relative to C/C++ is quite common. News that the maths functions are so slow is not good as far as I'm concerned. Why is it so much slower than 1.3? -
Performance matters[ Go to top ]
- Posted by: Brian Miller
- Posted on: January 09 2004 15:13 EST
- in response to Andy Pearson
In my experience having to make a case about Java's perfromance relative to C/C++ is quite common. News that the maths functions are so slow is not good as far as I'm concerned.
Totally. I can't even convince coworkers to accept Java somewhere in the rendering pipeline, let alone actual scientific processing upstream. The culture amongst so many of the e-scientists seems to be that since gcc and g77 emit fast code, there's no need for further technology consideration. My team lead and I get clobbered anytime we mention Java Native Interface as a way to bestow J2EE features upon legacy science libraries. It doesn't seem to matter to these folks that it took C++ 10 years to standardize a string class, or that Fortran 2000 is yet unfinalized in 2004. All that seems to matter to some subject matter experts is FLOPs, and Moore's Law be damned. Not even JavaGrande.org has changed this culture. -
Performance matters[ Go to top ]
- Posted by: Todd Murray
- Posted on: January 10 2004 11:51 EST
- in response to Brian Miller
"doesn't seem to matter to these folks that it took C++ 10 years to standardize a string class"
A small nit to pick, Java is not standardized by any "folks" and it doesn't look like it will ever be. -
Performance matters[ Go to top ]
- Posted by: Nick Minutello
- Posted on: January 11 2004 22:22 EST
- in response to Todd Murray
|
|A small nit to pick, Java is not standardized by any "folks" and it doesn't
|look like it will ever be.
|
Except for these folks. But who are they to matter...
-Nick -
Performance matters[ Go to top ]
- Posted by: Todd Murray
- Posted on: January 12 2004 14:32 EST
- in response to Nick Minutello
Sun is a standards body a la ISO, ANSI, etc? That's news to me. -
Performance matters[ Go to top ]
- Posted by: Nick Minutello
- Posted on: January 13 2004 08:53 EST
- in response to Todd Murray
|
|Sun is a standards body a la ISO, ANSI, etc? That's news to me.
|
Standard : "something established by authority, custom, or general consent"
It doesnt have to come from a "standards body" to be a standard...
-Nick -
Java float flawed?[ Go to top ]
- Posted by: Test Dummy
- Posted on: January 12 2004 07:35 EST
- in response to Brian Miller
Java floating point arithmetic seriously flawed. Does anybody know if this still holds? I would not use Java in a mars probe until this has been corrected..
//ras -
Java float flawed?[ Go to top ]
- Posted by: Remi Forax
- Posted on: January 12 2004 14:18 EST
- in response to Test Dummy
Java floating point arithmetic seriously flawed.
yop until JDK1.3 and introdcution of strictfp keyword.
> Does anybody know if this still holds? I would not use Java in a mars probe until this has been corrected..
see http://math.nist.gov/javanumerics/reports/pejfps.txt
>
> //ras
remi -
Performance matters - so use Java[ Go to top ]
- Posted by: John Davies
- Posted on: January 09 2004 20:04 EST
- in response to Andy Pearson
The users of such systems commonly include maths Ph.D.s who typically write C, and have little or no experience of Java.
Andy,
You seem to have worked in the same banks as I. I agree totally, the main problem is the attitude of the PhD risk "guru" writing parallelised curve analysis or VaR calculations etc. They frequently use Fortran or C++ and their only knowledge of Java is often a slow running Applet they once saw in the 90s.
Very few of the High Performance Computing (HPC) projects we worked on in banks have needed a database other than for static data. Typical engines have over 100 CPUs, we have worked in several of the world's largest banks (US and Europe) and we are finally starting to see an up take in Java. It's easier to manage, easier to program, more reliable than C++ due to better memory management, easier to maintain etc. etc. There are a lot of serious advantages to using Java in HPC.
I am very curious about the trig results, anyone got any ideas yet?
-John- -
Who cares? - Some people do![ Go to top ]
- Posted by: Drew McAuliffe
- Posted on: January 09 2004 15:27 EST
- in response to Nathan Zumwalt
A pricing engine doesn't necessarily deal with database connectivity. It's a number-cruncher, so performance outside of IO and dbs is important. -
Who cares? - Some people do![ Go to top ]
- Posted by: Michael Mahemoff
- Posted on: January 09 2004 14:02 EST
- in response to John Davies
I can give you plenty of situations where languages are chosen for their performance, obviously not on the back of a small benchmark like this but one of the main arguements for most banks not using Java in their pricing and risk engines is performance. I know this isn't true but that's the arguement I hear when proposing Java based technologies (Jini et al).
It would be unfortunate if decision-makers used this kind of data in isolation. A language's performance is more than the time to execute a mathematical instruction --- even for math-intensive software, there are still optimisations that are facilitated by a higher-level language which allows more sane reasoning --- to quote Donald Knuth, "Premature optimization is the root of all evil".
And don't forget Moore's law --- performance continues to double every eighteen months. In the study, Java is roughly half the speed of Visual C++ (by total). Even if you naievely take that completely at face value, it's a difference of eighteen months by prevailing Moore's law reasoning. If you could tolerate C++ performance in mid-2002 (and for typical applications, you probably could), then you can tolerate Java performance early 2004. -
Who cares? - Some people do![ Go to top ]
- Posted by: John Walker
- Posted on: January 14 2004 14:13 EST
- in response to Michael Mahemoff
I have a bond yield calculator that I use for benchmarking "real" math algorithm performance between Java and C. The algorithms are exactly the same in both languages. Even with the latest Java from Sun, it is actually 5 times slower than compiled C.
In the financial industry, this is particularly important when running thousands of simulations on complex asset structures. It allows us to (1) get products to market quicker or (2) run more simulations in a finite time frame to provide better assessments. -
Who cares?[ Go to top ]
- Posted by: Vladimir Goncharov
- Posted on: January 09 2004 10:48 EST
- in response to Nathan Zumwalt
Performance still plays a role, especially when dealing with encryption, SSL etc. and nice to know that java runs comparably with the C++. It wood be much worse if it doesnt. -
Who cares?[ Go to top ]
- Posted by: Brian Miller
- Posted on: January 09 2004 11:46 EST
- in response to Nathan Zumwalt
I've never seen a case where one language was chosen over another because of how fast the arithmetic functions were...
You obviously don't work in e-science, as I do. My colleagues mostly laugh at Java. But wide area networking is becoming more important to e-science, so scientists are gradually warming to Java. -
Who cares?[ Go to top ]
- Posted by: Nathan Zumwalt
- Posted on: January 09 2004 11:54 EST
- in response to Brian Miller
I've never seen a case where one language was chosen over another because of how fast the arithmetic functions were...
>
> You obviously don't work in e-science, as I do. My colleagues mostly laugh at Java. But wide area networking is becoming more important to e-science, so scientists are gradually warming to Java.
No, I don't. But, that's a tiny fraction of the entire software developement industry.
Anyway, if you're working in a field like this, you're probably not going to look at these type of benchmarks anyway... you're going to run test cases for the specific code you need to optimize.
-Nathan -
Who cares?[ Go to top ]
- Posted by: Frank Bolander
- Posted on: January 09 2004 12:26 EST
- in response to Nathan Zumwalt
No, I don't. But, that's a tiny fraction of the entire software developement industry.
Actually, there are a lot of financial applications(asset allocation,modelling,analytics,...) which require heavy computations and are a large part of financial and insurance applications. Number crunching is becoming more and more important as systems go towards real-time processing.
I agree with you on benchmarks, for the most part they are worthless. However, it is interesting to see that Java is comparable in computing primitive operations with the more "native" C++/C# but is terrible with trig functions. This seems more like a poor implementation of the trig library versus java itself.
The real problem with java is with precision. The language needs to adopt a long double datatype to address alot of precision computing issues. This is probably more of an issue for the e-Science problems mentioned, but I'm seeing it crop up in the financials also.
- Frank Bolander -
Nine Language Performance Round-up: Benchmarking Math & File IO[ Go to top ]
- Posted by: John Davies
- Posted on: January 09 2004 10:37 EST
- in response to Jamie Schiner
Much that I hate benchmarks this didn't try to draw any earth shattering conclusions from the results. It's a shame the author didn't use his knowledge of languages like Ruby and Perl to extend the benchmark to those too, for fun of course.
I'm rather intrigued about Java's trig results given such excellent performance in the rest of the field. It would be interesting to try with the new (alpha) JDK 1.5 but of course we couldn't publish the results yet.
-John- -
What about XML processing?[ Go to top ]
- Posted by: George Lawniczak
- Posted on: January 09 2004 11:16 EST
- in response to Jamie Schiner
My question is, how does Java compare with C/C++ when it comes to processing large XML files? It would seem to me that if you are doing heavy-duty XML processing it might make sense to use C/C++ routines to do the heavy lifting in order to enhance performance. -
What about XML processing?[ Go to top ]
- Posted by: John Davies
- Posted on: January 09 2004 11:23 EST
- in response to George Lawniczak
"It would seem to me that if you are doing heavy-duty XML processing it might make sense to use C/C++ routines to do the heavy lifting in order to enhance performance."
What, are you mad? :-) It might make a slightly faster SAX parser but what about JAXB etc. You really have to be on a different planet to use something like C++ for XML parsing, even c# would be better and I'm a Java fan. Java based XML binding has got to be the way to go.
(Don't take it personally please, there's a smiley in there)
-John- -
Spread the word.[ Go to top ]
- Posted by: Michael Mayr
- Posted on: January 09 2004 11:22 EST
- in response to Jamie Schiner
Nice article if you want to convince someone that Java is fast. Just show him the results and tell him that his application doesn't use trigonometry at all (most of the time this will be applicable). Of course that is not the whole truth, but who cares. The other side is cheating, too. Everything about benchmark results (especially micro benchmark results) has been said already in this forum. -
Why is the Trig functions so slow in 1.4.2??[ Go to top ]
- Posted by: Martin Devlin
- Posted on: January 09 2004 13:12 EST
- in response to Jamie Schiner
Ok, so does anyone have any idea why the author got the excessively slow results for the trig functions? -
why is it so slow?[ Go to top ]
- Posted by: Andy Pearson
- Posted on: January 09 2004 13:54 EST
- in response to Martin Devlin
I had a quick test of how fast the Java test would be using native methods for the trig functions. It was a quick hack and I didn't do any error trapping in the code, so I'm not sure my results need a lot of caveats. The trig trest ran on my PC (jdk 1.4.2_02) in 80 secs. Using the native code reduced that to 6 secs. -
slow trig bench[ Go to top ]
- Posted by: Fredrik Sandberg
- Posted on: January 09 2004 17:11 EST
- in response to Martin Devlin
The trig section of the benchmark is slow because of what seems to be slow implementaion of the trigonometric functions in the Hotspot VM. IBM's VM seems to have similar perf characteristics. Bea's VM performs a lot better on the trig stuff.
As other people mentioned, this is a rather silly microbenchmark. What you can see though is that languages compiled to bytecodes (or similar) are on par with traditional compiled C/C++, regardless if you prefer VB, C# or Java.
/F -
Only tangent is slow[ Go to top ]
- Posted by: damian penney
- Posted on: January 09 2004 18:36 EST
- in response to Fredrik Sandberg
Here are the results broken down by function for 10M iterations
1516ms sin
1468ms sin
28984sec tan....
4375ms log
250ms sqrt -
Platform Independent Results[ Go to top ]
- Posted by: Joseph Haller
- Posted on: January 12 2004 14:36 EST
- in response to Fredrik Sandberg
An article at JavaWorld on JNI several months ago raised the issue of math performance using JSDK 1.4. Author Jeff S. Smith noted that Sun now uses a new math package that gurantees identical math results across all platforms.
Smith's article can be found at:
http://www.javaworld.com/javatips/jw-javatip141.html
This probably answers the question as to why trigonometric performance degraded from JSDK 1.3 to 1.4. -
this is what the result says[ Go to top ]
- Posted by: Rolf Tollerud
- Posted on: January 09 2004 13:31 EST
- in response to Jamie Schiner
C# 65.3, Java 103.1 when Java is allocated 3-4 times as much memory to run.
That is good enough for me but what is even more interesting is what is the differences in speed when both languages has the same amount of memory.
Regards
Rolf Tollerud -
Fast Java Matrix Operations?[ Go to top ]
- Posted by: Dan Greening
- Posted on: January 09 2004 14:45 EST
- in response to Jamie Schiner
These benchmarks give me hope. However, what I'd really like is some benchmarks of floating point matrix operations. We currently have a prototype system doing heavy matrix operations in Java, but it's slower than we'd like. Pointers from other readers on fast Java sparse matrix libraries would help us avoid converting this portion of our code to C++.
Thanks in advance. -
this doesnt deserve to be on TSS[ Go to top ]
- Posted by: sean decor
- Posted on: January 09 2004 14:45 EST
- in response to Jamie Schiner
Benchmarking is such a big thing where in a small parameter can cause a huge difference in the final results.
This research is so immature and the author doesnt address so many details that "This thread should just be removed from TSS" -
Nine Language Performance Round-up: Benchmarking Math & File IO[ Go to top ]
- Posted by: hthjf fgfgfg
- Posted on: January 10 2004 12:38 EST
- in response to Jamie Schiner
Trig Results have more than doubled from Java1.3 to 1.4. I dont remember anything drastically changed in Math Libs from 1.3 to 1.4 or may be this is some joke -
Nine Language Performance Round-up: Benchmarking Math & File IO[ Go to top ]
- Posted by: Tom Hawtin
- Posted on: January 11 2004 14:59 EST
- in response to hthjf fgfgfg
Didn't the last set nanobenchmarks posted on slashdot also show 1.4 trig was much slower than 1.3? 1.3 is buggy on x86 as trig functions return incorrect results for large values. The slowness come from software normalisation of numbers bigger than pi. There shouldn't be a problem on Sparcs, PPC, etc. I didn't find any particular problem with the tan performance on my lowly PII.
They were a pretty meaningless bunch of benchmarks. Most of them loop over:
x = (x+1)*(i+2)/(i+3);
i += 4;
Mostly that's an integer divide then. Many moons ago someone posted to comp.compilers a list of the frequency basic operations were performed by general software. Integer divide was a tad infrequent.
The file I/O benchmark was a bit poor. The C version was just reading and writing byte arrays, while Java was doing character set encoding/decoding. Not that Java blocking I/O is particularly fast. -
Nine Language Performance Round-up: Benchmarking Math & File IO[ Go to top ]
- Posted by: Brian Miller
- Posted on: January 12 2004 12:39 EST
- in response to Tom Hawtin
Many moons ago someone posted to comp.compilers a list of the frequency basic operations were performed by general software. Integer divide was a tad infrequent.
No one is disputing that Java performs well for general purpose. We're pondering Java's ability to crunch numbers in e-science and finance. This matters if heterogeneous grids are to succeed. It also matters if scientists are ever to be free of the perils of C++ debugging. -
Explanation from Sun[ Go to top ]
- Posted by: Jim Cakalic
- Posted on: January 14 2004 15:57 EST
- in response to Tom Hawtin
A decent explanation of the cause of the 1.4 StrictMath performance "degradation" can be found in the Bug Database:
http://developer.java.sun.com/developer/bugParade/bugs/4857011.html