We usually use dragonfly to handle user generated assets in almost all the projects. But sometimes dragonfly with ImageMagick doesn’t play nicely in a limited environments like heroku.
We were getting tons of R14 - Memory quota exceeded errors after analyzing even small images using ImageMagick’s identify command.
Here is how we solved it.
First of all the context.
We use direct S3 upload on the client side in order to reduce the heroku servers load. Client goes to the Rails server with a sign request and gets back a policy, a signature and a key (aka path) of the resource to be uploaded to S3. There are more details about jQuery-File-Upload flow here.
Once the file is uploaded client goes to ImagesController and creates a record for the image.
class My::ImagesController < InheritedResources::Base
before_filter :authenticate_user!
actions :create, #...
respond_to :json
# ...
end
“create” action receives the only parameter original_image_uid which is passed to the model.
class Image < ActiveRecord::Base
belongs_to :user
dragonfly_accessor :original_image
def original_uid=(value)
self.original_image_uid = value
self.original_image_width = original_image.analyse(:width)
self.original_image_height = original_image.analyse(:height)
self.original_image_size = original_image.file.size
end
end
This is where all the (image-) magic happens. Before the model is saved we analyze the image width, height and the file size in order to use later according to the application business needs.
First call to the original_image will download the file from S3, files can be upto a few megabytes, so it takes about a second in the production environment. Than original_image.analyse calls the ImageMagick’s identify command and cache its results.
So everything is quite straightforward. But, we started to get R14 errors on heroku after the images#create requests. We were under impression that some huge memory leak eats up all the memory, but it turned out that it was not garbage collected memory bloat that happens right after the identify command returns.
It looks like ImageMagick’s identify tries to get as much memory as possible with no particular reason from my perspective. So we had to fight these bloats in a few different ways.
First is to run garbage collection. Check out gctools, the only thing we had to do is to add these lines to config.ru
require 'gctools/oobgc'
if defined?(Unicorn::HttpRequest)
use GC::OOB::UnicornMiddleware
end
It works with unicorn running on ruby 2.1. Learn more about it in the Aman Gupta’s blog.
And everything got back to normal, no R14 any more because the memory was cleaned up properly after each request.
But, why should we allow identify to take so much memory at the first place? And here comes the solution: passing limits to the identify command.
Another line of code added to initializers/dragonfly.rb
Dragonfly.app.configure do
plugin :imagemagick, identify_command: "identify -limit memory 0 -limit map 0"
#...
end
So, the ImageMagick doesn’t eat so much memory any more, and even if it does the bloat will be garbage collected after the request.
