– we create awesome web applications

We usually use dragonfly to handle user generated assets in almost all the projects. But sometimes dragonfly with ImageMagick doesn't play nicely in a limited environments like heroku.

We were getting tons of R14 - Memory quota exceeded errors after analyzing even small images using ImageMagick's identify command.

Here is how we solved it.

First of all the context.

We use direct S3 upload on the client side in order to reduce the heroku servers load. Client goes to the Rails server with a sign request and gets back a policy, a signature and a key (aka path) of the resource to be uploaded to S3. There are more details about jQuery-File-Upload flow here.

Once the file is uploaded client goes to ImagesController and creates a record for the image.

class My::ImagesController < InheritedResources::Base
  before_filter :authenticate_user!
  actions :create, #...
  respond_to :json

  # ...
end

"create" action receives the only parameter original_image_uid which is passed to the model.

class Image < ActiveRecord::Base
  belongs_to :user
  dragonfly_accessor :original_image

  def original_uid=(value)
    self.original_image_uid = value
    self.original_image_width = original_image.analyse(:width)
    self.original_image_height = original_image.analyse(:height)
    self.original_image_size = original_image.file.size
  end
end

This is where all the (image-) magic happens. Before the model is saved we analyze the image width, height and the file size in order to use later according to the application business needs.

First call to the original_image will download the file from S3, files can be upto a few megabytes, so it takes about a second in the production environment. Than original_image.analyse calls the ImageMagick's identify command and cache its results.

So everything is quite straightforward. But, we started to get R14 errors on heroku after the images#create requests. We were under impression that some huge memory leak eats up all the memory, but it turned out that it was not garbage collected memory bloat that happens right after the identify command returns.

It looks like ImageMagick's identify tries to get as much memory as possible with no particular reason from my perspective. So we had to fight these bloats in a few different ways.

First is to run garbage collection. Check out gctools, the only thing we had to do is to add these lines to config.ru

require 'gctools/oobgc'
if defined?(Unicorn::HttpRequest)
  use GC::OOB::UnicornMiddleware
end

It works with unicorn running on ruby 2.1. Learn more about it in the Aman Gupta's blog.

And everything got back to normal, no R14 any more because the memory was cleaned up properly after each request.

But, why should we allow identify to take so much memory at the first place? And here comes the solution: passing limits to the identify command.

Another line of code added to initializers/dragonfly.rb

Dragonfly.app.configure do
  plugin :imagemagick, identify_command: "identify -limit memory 0 -limit map 0"
  #...
end

So, the ImageMagick doesn't eat so much memory any more, and even if it does the bloat will be garbage collected after the request.

Stop using strftime. Seriously. At least if you are using Rails that is.

Rails, or rather, its I18N dependency, has a much better alternative I18n.l. The great thing about it is that you provide the name/kind of the format that you want separately of the format itself, so that you can, for example, change it completely for the whole application, or for a different locale.

The usage is quite simple. Instead of

Time.now.strftime('%Y-%m-%d %H:%M:%S')

You can do instead:

I18n.l Time.now, format: :myformat

with the myformat format defined in a locale file, say config/locale/time_formats.en.yml:

en:
  time:
    formats:
      myformat: '%Y-%m-%d %H:%M;%s'

The format string supports all the same format options as strftime, so conversion of your existing strftime code should be completely trivial.

It is important to pass a Symbol to the :format option of I18n.l, or it will try to interpret it as the format itself, and not its 'name' in the localization file.

Note: I18n.l has an alias I18n.localize, feel free to use it if you like to type.

When you are inside of a Rails view, you have another shortcut:

= l Time.now, format: :myformat

This is not all, yet...

It works for dates to:

I18n.l Date.today, format: :myformat

Event though it uses the same format name, it will use a different localisation key:

en:
  date:
    formats:
      myformat: '%Y-%m-%d'

Of course myformat is not such a great name for a format name ;). In a real application I would use something like compact, full, connfig, etc.

A couple of formats,:short, and :long are already provided, but I wouldn't rely on them and I suggest you define by yourself any time/date format that you intend to use inside of your application.

Lately, my prefered Postgres distribution of choice for OSX is Postgres.app.

Its very easy to download and install (just drag-n-drrop, like most other OSX Apps), no need to wait for it to compile, and it comes with a nice menubar icon for control.

In its default configuration it only listens on a port, but not on a unix socket. The problem is, Rails recipes with postgres create a config/database.yml file that assumes a socket presence.

Of example:

✗ rails new test1 -d postgresql
      create
      create  README.rdoc
      ...
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
✗ cd test1
✗ rake db:create
could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?

This problem was bothering me for a while as I tried to use rails_apps_composer to bootstrap new applications and it was failing on those database errors.

I didn't want to mess with Postgres.app configs, as I suspected they'd be overwriten on each version upgrade, so at first I tried to somehow trick it to stop and let me replace the config/database.yml file in time. The solution turned out to be much simpler though.

The Rails Postgres driver recognizes standard Postgres environment variables, one of which is PGHOST. When set to localhost Rails will go for the port instead of the Unix socket, even if there is no host: localhost setting in the YAML file.

✗ PGHOST=localhost rake db:create
✗ _

So just add export PGHOST=localhost to your shell start file, e.g. ~/.zshrc and you are set.

Im actually went further with it, and now I have shell aliases to reset postgres env config, configure it for localhost, or read current application Heroku configs and pre-set Postgres environment for a direct access to Heroku db, but that is a topic for another blog post.

I recently had an interesting bug that I want to share.

We had an age method in the User model, that was implemented like this:

def age
  return unless birthday
  now = Time.now.utc.to_date
  now.year - birthday.year - (birthday.to_date.change(year: now.year) > now ? 1 : 0)
end

And the following test for it:

describe :age do
  it 'should calculate age on exact date' do
    user = record(birthday: '2000-10-10')
    Timecop.freeze(Date.parse('2010-10-10')) do
      user.age.should == 10
    end
  end

  it 'should calculate age on next date' do
    user = record(birthday: '2000-10-10')
    Timecop.freeze(Date.parse('2010-10-11')) do
      user.age.should == 10
    end
  end

  it 'should calculate age on prev date' do
    user = record(birthday: '2000-10-10')
    Timecop.freeze(Date.parse('2010-10-9')) do
      user.age.should == 9
    end
  end
end

Suddenly, on 2013-07-14, I received a Circleci email that my last commit broke the specs. While investigating I found out that it failed in a place completely unrelated to my latest changes. It failed with an ArgumentError: invalid date. WTF?!

Investigating I found that we had a typo in one of our fixtures, that went like this:

triton:
  name: Triton
  birthday: <%= 501.days.ago %>
  ...

Notice the days instead of years that were ment to be used. And 501 days before 2013-07-14 is 2012-02-29, a leap year extra day, oops ;)

The age implementation tried to do .change(year: 1) to 2012-02-29 which produced an invalid date 2013-02-29. Apparently Date#change wasn't smart enough to take care of that:

> d = Date.parse('2012-02-29')
=> Wed, 29 Feb 2012
> d.change(year: 1)
ArgumentError: invalid date

I changed tirton's age to 501 years, and added the following test:

it 'should not fail when birthdate is on feb-29' do
  user = record(birthday: '2012-02-29')
  Timecop.freeze(Date.parse('2013-05-01')) do
    user.age.should == 1
  end
end

and fixed the implementation to be like so:

def age
  return unless birthday
  now = Time.now.utc.to_date
  diff = now.year - birthday.year
  diff - (diff.years.since(birthday) > now ? 1 : 0)
end

ActiveSupport's Date#since is smarter then #change and handles invalid dates properly:

> d = Date.parse('2012-02-29')
=> Wed, 29 Feb 2012
> 1.year.since(d)
=> Thu, 28 Feb 2013

Also note that there will be no problem with user fixtures on Feb-29 of the next leap year 2016-02-29. The x.years.ago that is used for birthdays will work just fine.

The are multiple ways of configuring your Rails application for different environments (e.g. staging, production, etc). One of the popular ones is through environment variables. For example Heroku uses this type of configuration extensively.

One of the benefits of it is that configuration values are never stored in the source control system, which improves security (for sensitive configuration parameters) and also makes it easier to try different configuration setups w/o changing the sources or re-deploying the application.

On the other hand writing (ENV['PRIMARY_DOMAIN'] || "myapp.com") every time you need your domain string becomes cumbersome pretty fast, not to mention duplication and having the default repeated all over the place.

A competent programmer will of course only do this once, and re-use the value everywhere. Something like this:

PRIMARY_DOMAIN = ENV['PRIMARY_DOMAIN'].presence || 'myapp.com'
S3_BUCKET = ENV['S3_BUCKET'] || raise 'missing S3_BUCKET'
ORDER_EXPIRATION_DAYS = (ENV['ORDER_EXPIRATION_DAYS'].presence || 1).to_i

But it quickly becomes complicated, and again, quite a bit of similarly looking code that begs to be refactored out.

constfig is something I extracted from a couple of my latest projects. It allows you to do just that, have a configuration parameters stored in constants with values coming from environment variables and ability to provide defaults or have required parameters (i.e. fail if missing).

I just released version 0.0.1 of constfig to rubygems. Sources are of course on github.

Installation

Add this line to your application's Gemfile:

gem 'constfig'

And then execute:

$ bundle

Or install it yourself as:

$ gem install constfig

Usage

There is only one function provided by the gem: define_config.

With a default (optional variable)

You can call it with a default, like this:

define_config :DEFAULT_DOMAIN, "astrails.com"

In which case it will first look if ENV['DEFAULT_DOMAIN'] is available, and if not will use the 'astrails.com'. A constant DEFAULT_DOMAIN will be defined.

Without a default (required variable)

Or you can call it without the default:

define_config :DEFAULT_DOMAIN

In which case it will raise exception Constfig::Undefined if ENV['DEFAULT_DOMAIN'] is not available.

Variable type

One last thing. Non-string variables are supported. If you provide a non-string default (boolean, integer, float or symbol), the value that is coming from ENV will be converted to the same type (using to_i, to_f, and to_symbol). For the true/false types "true", "TRUE", and "1" will be treated as true, anything else will be treated as false.

In the case of required variables, you can supply a Class in place of the default, and it will be used for the type conversion. Like this:

define_config :EXPIRATION_DAYS, Fixnum

For boolean variables you can supply either TrueClass, or FalseClass.

Existing constants

This gem will not re-define existing constants, which can be used to define defaults for non-production environments.

Rails on Heroku

There is one caveat with Rails on Heroku. By default Heroku doesn't provide environment variables to your application during the rake assets:precompile stage of slug compilation. If you don't take care of it your application will fail to compile its assets and might fail to work in production. To take care of it you can either use Heroku Labs user-env-compile option, or (and this is what I'd recommend) you can use development defaults during assets:precompile.

For example in Rails you con do this:

if Rails.env.development? || Rails.env.test? || ARGV.join =~ /assets:precompile/
  DEFAULT_DOMAIN = 'myapp.dev'
end

define_config :DEFAULT_DOMAIN

In development and test environments it will use 'myapp.dev' ad PRIMARY_DOMAIN, but in production and staging environment it will fail unless PRIMARY_DOMAIN is provided by environment.

NOTE: make sure those configuration variables are not actually used for asset compilation. If they are, I'd go with user-env-compile.

Managing environment

You can use the dotenv gem to manage your ENV.

The Rails 4 Way on Leanpub.com

We are proud to announce that our own Vitaly Kushner co-authored the new edition of 'The Rails 4 Way' together with Rails legend Obie Fernandes of Hashrocket fame.

'The Rails 4 Way' is the latest edition of the most comprehensive, authoritative guide to delivering production-quality code with Rails 4.

Pow is great. The single thing that bothered me was problems with using debugger while pow'ing away.

Did you ever put a debugger into your controller, connected with rdebug, hit refresh but it didn't stop?

The problem is most probably with POW_WORKERS setting. You see, by default, pow will start 2 ruby processes per application.

About half a year ago Vitaly posted a post about how simple it is today to use ruby patches bundled with rvm installation to dramatically reduce big rails app loading times and make your dev environment a much happier place.

Since then Ruby advanced with new patchlevels and there are new patches to use, so let's go over this once again.

Rails Conference 2012, first time in Israel, was a great deal of fun. Lot's of presenters both local and from all over the world, well, more like from all over the world of Rails. There were talks from Github, Heroku, Engine Yard, Gogobot, Get Taxi and lots and lots of others. Solid organization from Raphael Fogel People and Computers guys. Hordes of interesting people to talk to, nice and abundant food and coffee, lots of great content from the speakers and to sign off the day Github guys invited everyone to an open bar drinkup event.

We gave 2 talks, Vitaly's "Performance - When, What and How" and Boris' "Rails Missing Features". Check out slides and videos of those talks.

UPDATE [30 Apr 2013]: The new, updated and faster version of this blog post covering Ruby 1.9.3 patchelevel 392 and railsexpress patch set.

I knew that there are various Ruby patches available but I've never tried them as I didn't want to waste time for something that I thought I don't really need.

Then I found out that rvm comes bundled with patches ready to be applied, so I'm just a simple command line away from having a patched Ruby. Don't know how I missed that before.

Finally, there is a Rails Conference here in Israel. Long overdue. We're sponsoring it and giving one keynote and one lecture.

I had a lot of things to do last Thursday, Feb-17. I met a friend from abroad 3am at Ben Gurion Airport and spent several hours talking before we went to sleep, signed a contract for developing killer web app at 1:30am, and finally gave a presentation at The Junction at 4:30pm.

We presented on IGTCloud Ruby On Rails Day today.

Agenda was a bit different this time, not only technical presentations but also a few words about modern approach of building web applications.

Find the slides below.

I was working on tests for blender when I came upon a need to mock or stub a back-tick operator e.g.:

`shell command`

Apparently this is just a shortcut for calling Kernel method :`.

The OPEN 2010 conference was very well organized and had many interesting talks.

We are giving 4 out of 20 sessions at Open 2010, an Israeli open source conference.

I just recently reinstalled my MacBook Pro, this time with Snow Leopard.

So I'm again going through various installation problems I already forgot about from few years back when I installed Leopard.

Anyway, just had to hunt down a problem with mysql gem installation on Snow Leopard.

http://markupslicer.com

Supports ERB and HAML for now, vote on site for more formats.

Beautifully crafted, totally free and it's kinda fun.

About a week ago about 15 people were gathered in People and Computers offices thanks to Raphael Fogel.

I've just read "Do You Believe In Magic?" and the following quote resonated particularly well:

"It's not magic. There is no magic involved. It's just, if you never learnt Lisp, then you never learned to program, and Ruby's power is exposing a deficiency in your skills."

Yeah, I know, MVC is the "Only True Way™". But sometimes, just sometimes, you need your link_to or html helpers working in the model.

For example, sometimes the cleanest way to implement something is to have to_html in the model (widgets anyone?).

Doing this will most probably require generating some urls, and you need a controller for that. Usually I solved this by passing controller to the to_html function, but it always felt wrong.

Clicktale is a service that allows you to record and later playback behavior of your users while they are using your site. And Rails is Rails, you know.

And those two are getting along just fine, until the user logs in. After that clicktale service is cut out of the html pages this user gets and can't record the session. But it just started to get interesting...

Thanks a lot to Amit Hurvitz for providing a file of Virtual Disk Image (VDI) of VirtualBox, containing an up and running JRuby on Rails on Glassfish with Mysql. Image also contains some examples (actually solutions to the code camp exercises), all running on top of an OpenSolaris guest OS (can be run on many host systems).

Grab the image ~1.5GB archive.

Grab the exercises ~9.7MB archive.

We participated in JRuby on Rails with GlassFish Code Camp hosted by Sun Microsystems Inc. I was speaking about the framework in general trying to infect Java developers with Ruby On Rails. Slides are available.

Amit Hurvitz gave exciting presentation about GlassFish and short introduction into DTrace. Find out more details about the Code Camp.

I was the last person in our company working with ERB to render templates. While all the rest switched to HAML. At the beginning it was quite hard for me to read HAML comparing to ERB. HAML looked for me like some completely alien thing with weird percent marks all over the place and the significant whitespace never did it for me. On the other hand ERB felt like warm home after years we spent together.

Until I did the switch.

Recently we looked for video transcoding/hosting solution to use in one of our client's projects.

The best thing we've found is Panda. It runs on Amazon stack of services including ec2, s3, and simpledb.

Using amazon has many advantages. no contracts, pay as you go, easy and fast scaling in case your site explodes :)

Unfortunately the image that is refered in the Getting Started (ami-05d7336c) is not safe for production - it has openssh version with a serious security bug, but don't worry, we will explain how to fix it.

This blog-post is mostly targeted at non-Rails developers. Rails devs should know all this by heart :) Many times we need to explain to our customers what is 'proper deployment' and why their current one sucks :) Now we'll be able to just point them to this post...

On one of our projects we needed to do some caching for an action with an expensive db query. Fragment caching took care of the rendering but we needed a way to skip the db if we have a cache hit. And checking for an existence of the fragment file in the controller just didn’t seem right.