Benchmarking thin vs unicorn

Aug 26, 2010
Bb177d24e627be2b5a858cf233a8053e

Hi All!, I have been buzzed lately about Unicorn (one of the newest Ruby web application servers) and I've been asked if we have already tested it, I answered that no, we haven't. We're really happy using nginx + haproxy + thin + akamai.

But, I have to admit that after to have a short read about unicorn and having some free time, I started to dig at some already performed benchmarking comparisons between thin and unicorn my friend google showed me some, but all I could find were using really small basic scripts. I needed more realistic numbers, so with some beers in my fridge and no plans with my saturday, I decided to take our 5gb database with ~130 tables, a huge rails project and try to do my own research.

Let's start with my nginx.conf, it looks like this nginx.conf:

I'm going to use rvm to create isolated environments, since I am familiar with thin, I'll start with it.

ecruz@ecruz-mbp:ecomm (prod)$ rvm gemset create thin_test

info: Gemset 'thin_test' created.
ecruz@ecruz-mbp:ecomm (prod)$ rvm gemset use thin_test

info: Now using gemset 'thin_test'
ecruz@ecruz-mbp:ecomm (prod)$ gem install --no-rdoc --no-ri bundler
Successfully installed bundler-0.9.26
1 gem installed
ecruz@ecruz-mbp:ecomm (prod)$ bundle install
Fetching source index from http://rubygems.org/
Installing RedCloth (4.2.3) from rubygems repository at http://rubygems.org/ with native extensions 
Installing active_presenter (1.2.1) from rubygems repository at http://rubygems.org/ 
Installing crack (0.1.8) from rubygems repository at http://rubygems.org/ 
Installing httparty (0.6.1) from rubygems repository at http://rubygems.org/ 
Installing bitly (0.5.3) from rubygems repository at http://rubygems.org/ 
Installing cgi_multipart_eof_fix (2.5.0) from rubygems repository at http://rubygems.org/ 
Installing columnize (0.3.1) from rubygems repository at http://rubygems.org/ 
Installing curb (0.7.7.1) from rubygems repository at http://rubygems.org/ with native extensions 
Installing daemons (1.1.0) from rubygems repository at http://rubygems.org/ 
Installing delayed_job (2.0.3) from rubygems repository at http://rubygems.org/ 
Installing fastercsv (1.5.3) from rubygems repository at http://rubygems.org/ 
Installing fastthread (1.0.7) from rubygems repository at http://rubygems.org/ with native extensions 
Installing gem_plugin (0.2.3) from rubygems repository at http://rubygems.org/ 
Installing handsoap (1.1.7) from rubygems repository at http://rubygems.org/ 
Installing json (1.4.6) from rubygems repository at http://rubygems.org/ with native extensions 
Installing linecache (0.43) from rubygems repository at http://rubygems.org/ with native extensions 
Installing mash (0.1.1) from rubygems repository at http://rubygems.org/ 
Installing mime-types (1.16) from rubygems repository at http://rubygems.org/ 
Installing mysql (2.8.1) from rubygems repository at http://rubygems.org/ with native extensions 
Installing nokogiri (1.4.3.1) from rubygems repository at http://rubygems.org/ with native extensions 
Installing oauth (0.4.2) from rubygems repository at http://rubygems.org/ 
Installing packet (0.1.15) from rubygems repository at http://rubygems.org/ 
Installing rmagick (2.13.1) from rubygems repository at http://rubygems.org/ with native extensions 
Installing ruby-debug-base (0.10.3) from rubygems repository at http://rubygems.org/ with native extensions 
Installing ruby-debug (0.10.3) from rubygems repository at http://rubygems.org/ 
Installing twitter (0.6.3) from rubygems repository at http://rubygems.org/ 
Installing twitter_oauth (0.4.3) from rubygems repository at http://rubygems.org/ 
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

ecruz@ecruz-mbp:ecomm (prod)$ gem install --no-rdoc --no-ri thin
Building native extensions.  This could take a while...
Building native extensions.  This could take a while...
Successfully installed rack-1.2.1
Successfully installed eventmachine-0.12.10
Successfully installed thin-1.2.7
3 gems installed

Good! I'm happy to have bundler, I might've spent all my afternoon installing required gems by hand. But, we're here to see numbers:

ecruz@ecruz-mbp:ecomm (prod)$ ruby -v
ruby 1.8.7 (2010-06-23 patchlevel 299) [i686-darwin10.3.1]

ecruz@ecruz-mbp:ecomm (prod)$ thin -e production -p 3000 -d start
ecruz@ecruz-mbp:ecomm (prod)$

It's ready, my nginx is listening two virtual hosts and I've added to my /etc/hosts these virtual names. Ok, since we're starting, let's use small numbers

ecruz@ecruz-mbp:blog.crowdint.com (edwin.cruz)$ ab -n 10 -c 10 http://ecomm_thin/store/Apparel

Server Software:        nginx/0.7.64
Server Hostname:        ecomm_thin
Server Port:            80

Document Path:          /store/Apparel
Document Length:        81385 bytes

Concurrency Level:      10
Time taken for tests:   0.713 seconds
Complete requests:      10
Failed requests:        0
Write errors:           0
Total transferred:      817362 bytes
HTML transferred:       813850 bytes
Requests per second:    14.02 [#/sec] (mean)
Time per request:       713.256 [ms] (mean)
Time per request:       71.326 [ms] (mean, across all concurrent requests)
Transfer rate:          1119.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:   711  712   0.5    713     713
Waiting:      710  711   0.6    711     712
Total:        711  713   0.4    713     713

Percentage of the requests served within a certain time (ms)
  50%    713
  66%    713
  75%    713
  80%    713
  90%    713
  95%    713
  98%    713
  99%    713
 100%    713 (longest request)

Mhm, 713 ms per request, I've seen better numbers in our production servers, but well, I'm doing this in my MBP, without memcached and mysql query cached disabled, so, I'd say that: cool!

Now let's work with Unicorn:

ecruz@ecruz-mbp:ecomm (prod)$ rvm gemset create unicorn_test

info: Gemset 'unicorn_test' created.
ecruz@ecruz-mbp:ecomm (prod)$ rvm gemset use unicorn_test

info: Now using gemset 'unicorn_test'
gecruz@ecruz-mbp:ecomm (prod)$ gem install --no-rdoc --no-ri bundler
Successfully installed bundler-0.9.26
1 gem installed
ecruz@ecruz-mbp:ecomm (prod)$ bundle install
Fetching source index from http://rubygems.org/
   #same gems

Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
ecruz@ecruz-mbp:ecomm (prod)$ gem install --no-rdoc --no-ri unicorn
Building native extensions.  This could take a while...
Successfully installed rack-1.2.1
Successfully installed unicorn-1.1.2
2 gems installed
ecruz@ecruz-mbp:ecomm (prod)$ unicorn_rails -p 8000 -E production

Done... I love bundler, I really do, OK, let's continue with our tests:

ecruz@ecruz-mbp:ecomm (prod)$ unicorn_rails -p 8000 -E production

ecruz@ecruz-mbp:~ $ ab -n 10 -c 10 http://ecomm_unicorn/store/Apparel

Server Software:        nginx/0.7.64
Server Hostname:        ecomm_unicorn
Server Port:            80

Document Path:          /store/Apparel
Document Length:        81391 bytes

Concurrency Level:      10
Time taken for tests:   0.737 seconds
Complete requests:      10
Failed requests:        0
Write errors:           0
Total transferred:      817581 bytes
HTML transferred:       813910 bytes
Requests per second:    13.56 [#/sec] (mean)
Time per request:       737.212 [ms] (mean)
Time per request:       73.721 [ms] (mean, across all concurrent requests)
Transfer rate:          1083.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:    47  332 227.5    406     736
Waiting:       47  332 227.5    406     736
Total:         48  332 227.4    406     737

Percentage of the requests served within a certain time (ms)
  50%    406
  66%    436
  75%    480
  80%    510
  90%    737
  95%    737
  98%    737
  99%    737
 100%    737 (longest request)
ecruz@ecruz-mbp:blog.crowdint.com (edwin.cruz)$ 

Ok, decent numbers with small difference and knowing that we used a single app server for both, let's simulate a little bit more a production config:

I've adjusted my nginx.conf to support multiples thins

upstream balancer_thin {
  server 127.0.0.1:3000;
  server 127.0.0.1:3001;
  server 127.0.0.1:3002;
  server 127.0.0.1:3003;
  server 127.0.0.1:3004;
}

And I started 5 thins listening in ports 3000-3004

ecruz@ecruz-mbp:ecomm (prod)$ thin -e production -p 3000 -d -s 5 start
Starting server on 0.0.0.0:3000 ... 
Starting server on 0.0.0.0:3001 ... 
Starting server on 0.0.0.0:3002 ... 
Starting server on 0.0.0.0:3003 ... 
Starting server on 0.0.0.0:3004 ... 
ecruz@ecruz-mbp:ecomm (prod)$ 

I'll use 100 requests with 15 concurrent users(remember that I'm using my MBP).

ecruz@ecruz-mbp:blog.crowdint.com (edwin.cruz)$ ab -n 100 -c 15 http://ecomm/store/Apparel

Server Software:        nginx/0.7.64
Server Hostname:        ecomm
Server Port:            80

Document Path:          /store/Apparel
Document Length:        81385 bytes

Concurrency Level:      15
Time taken for tests:   2.665 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      8173616 bytes
HTML transferred:       8138500 bytes
Requests per second:    37.53 [#/sec] (mean)
Time per request:       399.706 [ms] (mean)
Time per request:       26.647 [ms] (mean, across all concurrent requests)
Transfer rate:          2995.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    54  374 227.0    324     918
Waiting:       51  271 167.3    232     708
Total:         54  375 227.0    325     919

Percentage of the requests served within a certain time (ms)
  50%    325
  66%    494
  75%    528
  80%    594
  90%    656
  95%    858
  98%    919
  99%    919
 100%    919 (longest request)

Whohoo 399ms, no bad for 15 concurrent users and 100 requests, it's time to see how unicorn performs, I tried to read unicorn documentation and I saw I can adjust unicorn behavior via custom configurator.rb, but since I'm lazy engineer I've modified unicorn gem directly to start with 5 workers:

ecruz@ecruz-mbp:ecomm (prod)$ mate /Users/ecruz/.rvm/gems/ruby-1.8.7-p299@unicorn_test/gems/unicorn-1.1.2/lib/unicorn/configurator.rb

ecruz@ecruz-mbp:ecomm (prod)$ unicorn_rails -p 8000 -E production
I, [2010-08-14T20:52:48.684992 #40603]  INFO -- : listening on addr=0.0.0.0:8000 fd=3
I, [2010-08-14T20:52:48.685496 #40603]  INFO -- : worker=0 spawning...
I, [2010-08-14T20:52:48.686348 #40603]  INFO -- : worker=1 spawning...
I, [2010-08-14T20:52:48.687000 #40603]  INFO -- : worker=2 spawning...
I, [2010-08-14T20:52:48.687786 #40603]  INFO -- : worker=3 spawning...
I, [2010-08-14T20:52:48.689268 #40603]  INFO -- : worker=4 spawning...

Done, I have unicorn running 5 workers and listening at port 8000, let's send it the same traffic: ecruz@ecruz-mbp:blog.crowdint.com (edwin.cruz)$ ab -n 100 -c 15 http://ecomm_unicorn/store/Apparel

Server Software:        nginx/0.7.64
Server Hostname:        ecomm_unicorn
Server Port:            80

Document Path:          /store/Apparel
Document Length:        81391 bytes

Concurrency Level:      15
Time taken for tests:   2.672 seconds
Complete requests:      100
Failed requests:        0
Write errors:           0
Total transferred:      8175813 bytes
HTML transferred:       8139100 bytes
Requests per second:    37.42 [#/sec] (mean)
Time per request:       400.860 [ms] (mean)
Time per request:       26.724 [ms] (mean, across all concurrent requests)
Transfer rate:          2987.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    69  368 165.3    334     642
Waiting:       69  368 165.3    334     641
Total:         69  368 165.3    334     642

Percentage of the requests served within a certain time (ms)
  50%    334
  66%    504
  75%    523
  80%    543
  90%    587
  95%    614
  98%    622
  99%    642
 100%    642 (longest request)
ecruz@ecruz-mbp:blog.crowdint.com (edwin.cruz)$ 

Look at that!!! the difference is minimal, this is a really huge surprise.

Ok, I'll run tests using extreme over loading against the same 5 thins and unicorn with 5 workers (For each test I restarted my machine):

ecruz@ecruz-mbp:blog.crowdint.com (edwin.cruz)$ ab -n 1000 -c 100 http://ecomm_thin/store/Apparel

Server Software:        nginx/0.7.64
Server Hostname:        ecomm_thin
Server Port:            80

Document Path:          /store/Apparel
Document Length:        81385 bytes

Concurrency Level:      100
Time taken for tests:   31.030 seconds
Complete requests:      1000
Failed requests:        4
   (Connect: 0, Receive: 0, Length: 4, Exceptions: 0)
Write errors:           0
Total transferred:      81736483 bytes
HTML transferred:       81385312 bytes
Requests per second:    32.23 [#/sec] (mean)
Time per request:       3102.981 [ms] (mean)
Time per request:       31.030 [ms] (mean, across all concurrent requests)
Transfer rate:          2572.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.0      0      10
Processing:    55 3035 1467.6   2880    7831
Waiting:       54 2433 1276.4   2202    7369
Total:         55 3036 1468.1   2880    7833

Percentage of the requests served within a certain time (ms)
  50%   2880
  66%   3314
  75%   3584
  80%   3794
  90%   4485
  95%   6825
  98%   7073
  99%   7369
 100%   7833 (longest request)
ecruz@ecruz-mbp:blog.crowdint.com (edwin.cruz)$ 

Wow! my machine survived...

ecruz@ecruz-mbp:blog.crowdint.com (edwin.cruz)$ ab -n 1000 -c 100 http://ecomm_unicorn/store/Apparel

Server Software:        nginx/0.7.64
Server Hostname:        ecomm_unicorn
Server Port:            80

Document Path:          /store/Apparel
Document Length:        81391 bytes

Concurrency Level:      100
Time taken for tests:   27.692 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      81758141 bytes
HTML transferred:       81391000 bytes
Requests per second:    36.11 [#/sec] (mean)
Time per request:       2769.227 [ms] (mean)
Time per request:       27.692 [ms] (mean, across all concurrent requests)
Transfer rate:          2883.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.8      0       4
Processing:    81 2639 441.8   2671    3178
Waiting:       81 2638 441.8   2670    3172
Total:         86 2639 441.1   2671    3179

Percentage of the requests served within a certain time (ms)
  50%   2671
  66%   2759
  75%   2868
  80%   2923
  90%   2976
  95%   3008
  98%   3048
  99%   3099
 100%   3179 (longest request)
ecruz@ecruz-mbp:blog.crowdint.com (edwin.cruz)$

Wow! look at that numbers, specifically: Failed Requests, Requests per second, TIme per request and Transfer rate, it looks like unicorn performs better on heavy loading, I'll post another performance numbers with more production like environment running in a dedicated server hosted in rackspace with more advanced features for each configurations: caching, fail over, haproxy between nginx and thins, cdn, etc. Wait for it, I'll try to include some NewRelic numbers and use different pages/modules.

Ok, there're the numbers, think about if it's worth to change any current infrastructure, monitoring tools, deployment scripts, etc to switch between each other. We've taken ours x).

Machine used:

Processor: 2.4 GHz Intel Core i5
Memory: 4GB 1067 MHz DDR3
Mac OS X: 10.6.3
blog comments powered byDisqus