[JRuby] JRuby perf questions answered : RUBY

(Sorry if this is a duplicate post...the original seems to have disappeared)

Many folks still seem to be in the dark about JRuby performance relative
to Ruby 1.8.x. I figured a short post with some basic info could start a
discussion for anyone wanting to know more.

Long story short, JRuby 1.1 is now generally faster than Ruby 1.8.6, and
for most execution benchmarks it is the fastest 1.8-compatible Ruby
implementation available. This is based on the YARV benchmarks, the
Alioth benchmarks, a number of benchmarks we created for JRuby, and a
few other external benchmarks we run from time to time. Many (most?) of
these are microbenchmarks, but a number are nontrivial code.

For example, M. Ed. Borasky's MatrixBenchmark:

JRuby:
Hilbert matrix of dimension 64 times its inverse = identity? true
18.784000 0.000000 18.784000 ( 18.784000)

Ruby 1.8.6:
Hilbert matrix of dimension 64 times its inverse = identity? true
32.560000 0.110000 32.670000 ( 32.776915)

Or the YARV "pentomino" benchmark:

~/NetBeansProjects/jruby $ time jruby -J-server
test/bench/yarv/bm_app_pentomino.rb

real 1m31.770s
user 1m32.016s
sys 0m1.278s
~/NetBeansProjects/jruby $ time ruby test/bench/yarv/bm_app_pentomino.rb

real 1m48.100s
user 1m47.489s
sys 0m0.287s

Ola Bini of the JRuby team recently ran all the YARV benchmarks and
reported the results here:

http://ola-bini.blogspot.com/2007/10...rformance.html

And I periodically post about performance progress on my blog here:

http://headius.blogspot.com

I expect we'll see Alioth updated soon after the 1.1 or 1.1 beta 1 releases.

What about Rails?

One of the biggest uses for JRuby recently has been Rails. Rails
performance is a more complicated thing to measure; there's a lot of
non-execution bottlenecks that get in the way. But both Ola and Nick
Sieger (also of JRuby team) have found that JRuby on Rails performance
is either very near or exceeding Ruby 1.8.x on Rails:

http://ola-bini.blogspot.com/2007/10...pressions.html
(toward the bottom of the post are updated Rails numbers)

http://blog.nicksieger.com/articles/...ls-fast-enough

Rails is probably the ultimate real-world benchmark, and we're excited
to see that JRuby's starting to pass Ruby 1.8.x performance here as well.

I won't drag this on any longer, but I'd encourage you to try out your
own Ruby code and let us know how it performs. We've spent a lot of time
on performance issues for the 1.1 release, and have many future plans to
continue increasing performance. We want your input.

Many people believed we'd never be faster than the C implementation, and
many still think we're slower. Now that I've set that record straight,
any questions?

- Charlie

> any questions?

Startup time and comparison with 1.9?
--
Posted via http://www.ruby-forum.com/.

Roger Pack wrote:
>> any questions?
>
> Startup time and comparison with 1.9?

Also memory use comparison compared to normal C version. Does the GC
throw as often, too?
--
Posted via http://www.ruby-forum.com/.

Roger Pack wrote:
> Roger Pack wrote:
>>> any questions?
>> Startup time and comparison with 1.9?
>
> Also memory use comparison compared to normal C version. Does the GC
> throw as often, too?

In profiling most apps I've tried, even pretty brutal ones generating a
lot of garbage, the GC cost is almost invisible. A tiny blip once in a
while.

I don't have any really good concrete numbers on overall memory usage.
Tom's done a comparison of a minimal/bare Rails app here:

http://www.bloglines.com/blog/ThomasEEnebo?id=38

The general story is that per JRuby on Rails instance (usually you have
many in a single JVM), we use less memory than an equivalent Mongrel +
Ruby + Rails process.

A quick check of "jirb" compared to "irb" gives me:

jirb: 27MB
irb: 2.8MB

This is on OS X, Java 6 preview version, so it's safe to say that any
JRuby process will have an initial cost of about 25MB to start up the JVM.

We've also done a lot in 1.1 to reduce memory consumption, and there's
more work coming up.

- Charlie

There was a recent thread which asked how fast ruby ran for each user.

http://www.ruby-forum.com/topic/125344#new

Josip Gracin wrote:
> Here's something interesting. The processor is Intel T7200 @ 2.00GHz,
> Linux Ubuntu.
>
> 1. With ruby 1.8.5. which comes packaged with Ubuntu:
>
> Ruby 1.8.5 on i486-linux
> It took 20.346013 seconds to run. 49149 iterations per
> second.
>
> 2. With ruby 1.9.0 (2006-06-08) which comes packaged in Ubuntu:
>
> Ruby 1.9.0 on i486-linux
> It took 23.875774 seconds to run. 41883 iterations per
> second.
>
>
> 3. With ruby 1.8.6 compiled from sources on the machine:
>
> Ruby 1.8.6 on i686-linux
> It took 7.107875 seconds to run. 140689 iterations per
> second.
>
>
> Obviously, the target instruction set matters quite a lot. I think I'll
> be compiling my own ruby from now on. :-)

Greg Halsey wrote:
> windows xp. intel core2 1.86GHz
>
> JRUBY:
>
> C:\jruby\jruby-trunk\bin>jruby calculate.rb
> 55
>
> It took 41.125 seconds to run. 24316 iterations per
> second.
>
> Not so good. Let's disable ObjectSpace:
> C:\jruby\jruby-trunk\bin>jruby -O calculate.rb
> 55
>
> It took 11.109 seconds to run. 90017 iterations per
> second.
>
> That was good. Now let's give it some more juice:
> C:\jruby\jruby-trunk\bin>jruby -J-server -O calculate.rb
> 55
>
> It took 5.75 seconds to run. 173913 iterations per
> second.
> --------------------------------------------
> MRI:One-Click
> ruby calculate.rb
> 55
>
> Ruby 1.8.6 patch 0 on i386-mswin32
> It took 15.828 seconds to run. 63179 iterations per
> second.

Now, I am thinking that jruby might work well if we know how to find the
sweet spot. Why the pre compiled version is so very much slower than a
home compiled version makes me wonder why they do not compile it better.
As things can be so radically different in jruby if only you know how to
tweak it perfectly is another thing that makes me think.

So, do you find your own benchmarks and work your compiler to make it
run at its fastest or is performance going to be "roll your own" for
those with insider knowledge?
--
Posted via http://www.ruby-forum.com/.

Roger Pack wrote:
>> any questions?
>
> Startup time and comparison with 1.9?

Startup time varies widely between platforms. OS X is probably one of
the fastest:

~/NetBeansProjects/jruby $ time ruby -e "puts 'hello'"
hello

real 0m0.015s
user 0m0.007s
sys 0m0.008s
~/NetBeansProjects/jruby $ time jruby -e "puts 'hello'"
hello

real 0m1.815s
user 0m1.709s
sys 0m0.183s

There's been reports that on server-class machines the startup time can
be as high as 5-10 seconds. The slow startup time is partially due to
the JVM and partially our fault (JRuby's complicated, yo).

As for Ruby 1.9...it's hard to make a comparison right now. Most of the
benchmarks we use to measure Ruby 1.9 (like those I described
previously) are usually a bit faster in Ruby 1.9. And of course almost
all of Ruby 1.9's benchmarks are fastest in Ruby 1.9.

Part of the difficulty comparing is that Ruby 1.9 contains a bunch of
changes to Ruby language semantics that we have not yet put in JRuby,
since we've focused on 1.8 compatibility. Things like:

- fast math operations, that speed up +, -, etc, but you can't override them
- constant-time case/when when literal values are used, but you can't
override === for those literal types

Here's A few numbers, for your edification. Some things are faster in
Ruby 1.9:

JRuby:
~/NetBeansProjects/jruby $ jruby -J-server
test/bench/bench_fib_recursive.rb
0.869000 0.000000 0.869000 ( 0.869000)
0.677000 0.000000 0.677000 ( 0.677000)
0.652000 0.000000 0.652000 ( 0.652000)
0.648000 0.000000 0.648000 ( 0.648000)

Ruby 1.9:
~/NetBeansProjects/jruby $ ../ruby1.9/ruby -I ../ruby1.9/lib
test/bench/bench_fib_recursive.rb
0.400000 0.000000 0.400000 ( 0.405639)
0.400000 0.000000 0.400000 ( 0.406707)
0.400000 0.000000 0.400000 ( 0.400150)
0.400000 0.000000 0.400000 ( 0.398065)

And some things are slower:

JRuby:
1m loops yielding three fixnums 10 times to block splatting and
accessing them
3.387000 0.000000 3.387000 ( 3.387000)
3.314000 0.000000 3.314000 ( 3.315000)
3.235000 0.000000 3.235000 ( 3.235000)

Ruby 1.9:
1m loops yielding three fixnums 10 times to block splatting and
accessing them
4.980000 0.020000 5.000000 ( 5.055369)
4.980000 0.020000 5.000000 ( 5.073821)
5.010000 0.020000 5.030000 ( 5.088681)

And so on. However, for Rails, I've only seen one comparison that showed
Ruby 1.9 was either slightly faster or slower than Ruby 1.8.x:

http://eigenclass.org/hiki/non-synth...marks-for-yarv

I'm not sure if there's been a more recent comparison. I believe we'll
be able to match Ruby 1.9 performance once we start making the same
optimizations.

- Charlie

Lloyd Linklater wrote:
> There was a recent thread which asked how fast ruby ran for each user.
>
> http://www.ruby-forum.com/topic/125344#new
>
> Josip Gracin wrote:
>> Here's something interesting. The processor is Intel T7200 @ 2.00GHz,
>> Linux Ubuntu.
>>
>> 1. With ruby 1.8.5. which comes packaged with Ubuntu:
>>
>> Ruby 1.8.5 on i486-linux
>> It took 20.346013 seconds to run. 49149 iterations per
>> second.
>>
>> 2. With ruby 1.9.0 (2006-06-08) which comes packaged in Ubuntu:
>>
>> Ruby 1.9.0 on i486-linux
>> It took 23.875774 seconds to run. 41883 iterations per
>> second.
>>
>>
>> 3. With ruby 1.8.6 compiled from sources on the machine:
>>
>> Ruby 1.8.6 on i686-linux
>> It took 7.107875 seconds to run. 140689 iterations per
>> second.
>>
>>
>> Obviously, the target instruction set matters quite a lot. I think I'll
>> be compiling my own ruby from now on. :-)
>
> Greg Halsey wrote:
>> windows xp. intel core2 1.86GHz
>>
>> JRUBY:
>>
>> C:\jruby\jruby-trunk\bin>jruby calculate.rb
>> 55
>>
>> It took 41.125 seconds to run. 24316 iterations per
>> second.
>>
>> Not so good. Let's disable ObjectSpace:
>> C:\jruby\jruby-trunk\bin>jruby -O calculate.rb
>> 55
>>
>> It took 11.109 seconds to run. 90017 iterations per
>> second.
>>
>> That was good. Now let's give it some more juice:
>> C:\jruby\jruby-trunk\bin>jruby -J-server -O calculate.rb
>> 55
>>
>> It took 5.75 seconds to run. 173913 iterations per
>> second.
>> --------------------------------------------
>> MRI:One-Click
>> ruby calculate.rb
>> 55
>>
>> Ruby 1.8.6 patch 0 on i386-mswin32
>> It took 15.828 seconds to run. 63179 iterations per
>> second.
>
> Now, I am thinking that jruby might work well if we know how to find the
> sweet spot. Why the pre compiled version is so very much slower than a
> home compiled version makes me wonder why they do not compile it better.
> As things can be so radically different in jruby if only you know how to
> tweak it perfectly is another thing that makes me think.
>
> So, do you find your own benchmarks and work your compiler to make it
> run at its fastest or is performance going to be "roll your own" for
> those with insider knowledge?

We have been running whatever benchmarks we can, and they're all pretty
generic. For all the numbers I've reported (and I believe most of the
numbers others have reported for JRuby), it's been JRuby trunk (soon to
be 1.1) with the Java 6 "server" VM. There's really no more tweaking
than that required to get the same performance.

As for the compiler...there's no additional user tweaking necessary to
make it run best; it should run its best all the time. The additional
tweaks you may want to apply would be using a different Java version,
tweaking the JVM or GC settings, and so on. Normal server tuning most
people won't need.

The reality with the JVM, and especially with JRuby, is that short tests
aren't going to show the real long-term performance. For most execution,
JRuby starts out interpreted, and the interpreter is slower than Ruby
1.8.x. But after running for a while, code will eventually compile to
Java bytecode (or you can precompile to skip that time). Then there's an
additional delay while the JVM itself eventually compiles that bytecode
down to platform-native code.

We expect that for JRuby 1.1, the main sweet spot will be on
longer-running apps e.g. for server-side apps or for heavy data processing.

Charles Oliver Nutter wrote:
> JRuby 1.1 is now generally faster than Ruby 1.8.6, and
> for most execution benchmarks it is the fastest 1.8-compatible Ruby
> implementation available.

I look forward to getting my hands on JRuby 1.1 and running some Rails
apps. Keep up the good work!
--
Posted via http://www.ruby-forum.com/.