Simple Object persistency library for Cassandra
Motivation & History (you can skip it :)
I was developing an multiplayer online game for a client (TBD: link when
released :) and we decided to use Cassandra
for performance and scaling benefits. Also the game’s internal data structures
mapped very well to key-value semantics.
I did some research but couldn’t find anything that was Ready at the time
to be used for development.
I did find the BigRecord project which somewhat
kind of supported Cassandra, but I got an impression that Cassandra support was
‘bolted on’ after the fact, and their way of running a java Cassandra driver
talking to some JRuby and interfacing with it through DRB was … can’t quite
find the word for it, lets call it ‘awkward’ :).
So I started to work on my own library, “steaing” liberally from BigObject,
which at least at the time was very close to ActiveRecord with just some
parts of the code commented out or replaced.
Somewhere midway through the implementation I found the
CassandraObject project by
NZKoz. Now that was a much better one. So much
better that for some time I just considered dumping all I did and using it
as-is. After some thinking I decided against it though. The primary reason was
that its data model was quite different from what I intended to use for the
game. In CassandraObject all attributes are stored as columns in a simple
ColumnFamily, and indexes and
associations stored in separate ColumnFamilies. This has a benefit of not
having restrictions on the number of associated object, but it does require
additional DB queries to access associated objects.
I wanted to use
Supercolumns
instead, and store attributes and associations in different supercolumns for
the given key. This has the benefit of being able to fetch all the data at
once, but does restrict the number of associated objects. Since in my intended
use-case the number of associations was rather small I decided to continue the
development.
But, this doesn’t mean I didn’t use CassandraObject to “steal” some code from it too :).
There were too many good ideas to pass by. I ended up copying a lot
of code from it and throwing all of the BigRecord heritage. May be someday I’ll find
a way to ‘combine’ SmallRecord back into CassandraObject. Meanwhile though I’m going
to work on this one.
(Ugh, that ended up being rather long explanation :)
Status
This is work in progress. It is currently used in a client project and works
well, but it still has rough edges and might have problems in your specific
environment. The code base is quite small and modular though, so you should
have no problems jumping in and fixing or extending it for your use case.
Data Model
This library is intended to be used with Supercolumns Families only (for now :).
Model’s attributes are stored inside “attributes” supercolumn, with attributes themselves
being columns inside it. Associations are stored as separate supercolumns, with each association
id being a column inside.
Example (json notation):
users: {
"1": {
"attributes": {
"name": "Vitaly Kushner",
"company": "Astrails"
},
"account_ids": {
"123": "1",
"456": "1"
}
}
accounts: {
"123": {
"url": "http://astrails.com",
"username": "vitaly",
"password": "234987234509827345"
},
"456": {
"url": "http://rubyonrails.org",
"username": "vitaly",
"password": "3084573945873945"
}
}
As you can see we have a one-to-many association here. but contrary to how it is handled
in ActiveRecord we don’t store the user_id in the account ‘record’, instead we store all the
account_ids in the user record. This is because otherwise we would have no way of querying user.accounts
except for the full accounts ‘table’ scan.
Implementation
Connecting to the DB
SmallRecord will look for the file config/small_record.yml :
production:
adapter: cassandra
host: 127.0.0.1
port: 9160
keyspace: astrails
development:
adapter: mock
test:
adapter: mock
Notice the :mock adapter. This is just a simple
Cassandra emulation using in-memory ruby hash. This is what I’m using
for development and testing. It doesn’t require cassandra running.
The emulation is not 100% off course but it does the job. And I didn’t
yet have any bugs related to the difference b/w the mock and the real thing.
Just remember that if you run develoment server with mock all the data will be gone
once you restart. But that is probably not such a bad thing for development. Or
it is. You decide.
Basics
You define your models the usual way:
class User < SmallRecord::Base
...
end
user = User.new :foo => "bar"
user.save
User.find(user.id)
User.first
Attributes
What is different from the ActiveRecord is that you have to tell SmallRecord
about all your attributes since it can’t infer it from the database schema like
ActiveModel does. There is no database schema duh! The attributes support
mostly came form CassandraObject but changed quite a bit since then. One day
I’ll document the differences :)
class User < SmallRecord::Base
attribute :name
attribute :age, :type => :integer
attribute :create_at, :type => :time
end
ActiveModel
This library is built upon Rails’s ActiveModel pulling in many of the familiar features of
the ActiveRecord. The following is supported:
Callbacks
before_save :do_something
The following callbacks are supported:
:before_init,
:after_init,
:before_find,
:after_find,
:before_save,
:after_save,
:before_create,
:after_create,
:before_update,
:after_update,
:before_destroy,
:after_destroy,
:before_validation,
:before_validation_on_create,
:before_validation_on_update
You can also define your own callbacks:
class User < SmallRecord::Base
define_callbacks :after_activation
after_activation :send_confirmation
def activate!
...
run_callbacks(:after_activation)
end
end
Dirty attributes
>> user.changed
=> []
>> user.name = "foo"
=> "foo"
>> user.changed
=> ["name"]
>> user.name_changed?
=> true
When saving an object it will only save changed attributes:
>> user.save
=> true
>> user.name = "qwe"
=> "qwe"
>> user.save
User Insert (0.000043) insert(aa421ea0-c407-46fe-986f-09b2d749b1be, {"attributes"=>{"name"=>"\"qwe\"", "schema_version"=>"0"}}, {})
=> true
Validations
class User < SmallRecord::Base
validates_presence_of :name
end
Associations
Association support in SmallRecord is rather basic. Only has_many is supported at the moment (feel free to add more :).
class User < SmallRecord::Base
has_many :accounts
end
user.accounts
user.accounts.create
user.create_account
user.account_ids
user.accounts.first
SmallRecord tries hard to do the minimal required amount of work. the association is
lazily loaded and only when really needed.
Migrations
Migrations are very different then what you are used to with ActiveRecord (this is too comes from CassandraObject).
You see, there might be a LOT of records in a Cassandra DB. To the point of it being quite impractical to run
a full migration. Instead each ‘record’ contains its schema-version and we migrate it on read if its outdated.
i.e. if we load a record into memory with schema_version that is less then the currently defined in the code
we will migrate this record. If you save the record after that it will be saved with the updated version.
migrations are defined using blocks:
class User
migrate 1 do |attrs|
attrs[:foo] = attrs.delete(:bar)
end
...
end
More
Read the code :)
TODO
There are a couple of things that I want to fix first:
-
The elephant in the room is the total lack of testing! Well, in the project I’m
extracting this from the test coverage is quite high, so all the code was implicitly
tested, but now that this is a separate project I need to add some specs.
-
There is this ugly read/write_data business. Apart from the bad naming
(I still can’t think of a good one) all the supercolumns except for the
‘attributes’ are not managed. They are currently written directly into db on
every change. Need to unify the ‘dirty’ handling in attributes with the rest
of supercolumns. For that I think I’ll need to drop the Dirty mixin from the
ActiveModel and just roll my own.
-
Documentation. This is also lacking at the moment and you will need to look at the code.
-
Need to research the possibility of merging with CassandraObject. Thought I’m not sure this
is practical.
Copyright
Copyright (c) 2010 Astrails Ltd. See LICENSE for details.
