Pages Menu
Twitter
Categories Menu

Posted by on Apr 1, 2015 in Ops, Ruby On Rails | 0 comments

Rails: Use JSON to serialize model attributes

Scalling a Rails application’s response is often done by using another programming language. As we might know, Twitter has started as a Rails application, and ended up as Scala, or later as a Javascript backend application.

In my oppinion, a first step to this kind of migration would consist in normalising all the serialised data you have in your database.

Personally, i use ActiveRecord::Base#serialize method to handle most of the custom data that can be resulted by a STI model, or to store any dynamic extra data. I consider to be a good example the situation when you need to keep some additional information about the user, like the company info if is a company .

Some of the Ruby on Rails projects are starting to use plain serialize method, which can be exemplified by Ryan Bates tutorial named PayPal Notifications, or by Diaspora’s codebase or Spree codebase.

One easy trick that you can do in any 3.x & 4.x application is to define your serialize method like :

class User < ActiveRecord::Base 
  serialize :other_data, JSON
end

This way your application will use JSON column coder, which in my opinion, is a better alternative which fixes some of the problems for you:

  • Allows you to use same database backend for multiple applications, written in multiple languages (if is the case)
  • Avoids Syck vs Psych serialization problems (Psych is default Yaml-er starting with Ruby 1.9.3)
  • Json is much faster than Yaml (check here, here and here)

Some of the problems caused by YAML are described in Arne Brasseur’s post.

Updating an existing application to use JSON serialized fields

In order to make your existing application to use JSON serialized field, you would need to perform some changes to your models, mainly to convert:

class User < ActiveRecord::Base 
  serialize :other_data
end

To

class User < ActiveRecord::Base 
  serialize :other_data, JSON
end

Notice the “JSON”, argument for the serialize method.

Other change that you would need to do is to add a migration to change your existing data, from YAML serialized string to JSON serialized string. To do so, you would need to add a migration or a code snippet somewhere in your application to perform the conversion operation:

class ChangeSerializationOnUser < ActiveRecord::Migration
  class YamlUser < ActiveRecord::Base
    self.table_name ="users"
    serialize :other_data
  end

  class JsonUser  < ActiveRecord::Base
    self.table_name ="users"
    serialize :other_data, JSON
  end

  def up
    YamlUser.where(other_data: '---
    ').update_all(other_data: nil)

    YamlUser.find_each do |yaml_user|
      next unless yaml_user.other_data.present?
      next unless yaml_user.other_data.respond_to?(:to_hash)
      hash = yaml_user.other_data.to_hash

      JsonUser.where(id: yaml_user.id).update_all(other_data: nil)
      json_user = JsonUser.find(yaml_user.id)
      json_user.other_data = hash || {}
      json_user.save!
    end
  end

  def down
    raise ActiveRecord::IrreversibleMigration
  end
end

The migration above is doing the following things:

  • Defines a YamlUser class that would handle the Yaml serialize part of your migration. Assuming you added JSON parameter to your class, YamlUser is performing the simple task of converting from string to whatever data you have serialized.
  • Defines a JsonUser class that would handle the JSON serialize part of your migration. This class is defined to perform one single thing, that to convert and save the serialized info field, without validations, without ActiveRecord callbacks.
  • Cleans up all the empty serialized objects. Depending of your data, you might add also an update for  ‘— \n[]’
  • Sometimes, the information you have saved might come as an HashWithIndifferentAccess, which for this operation would require a manual deserialization. That is why, i am using .to_hash
  • Before instantiating a JsonUser object, we would need to update the record in order to avoid any errors caused by the object hydration.
  • Of course, i consider this to be an “ActiveRecord::IrreversibleMigration”

I consider this to be a first step in order to migrate to multiple backend applications.

Read more:

Read More

Posted by on Mar 30, 2015 in Programming, Ruby On Rails | 2 comments

Exposing serialised fields – meta programming way

class SomeClass < ActiveRecord::Base
  belongs_to :user

  def self.serialize(attr_name, class_name = Object, exposed_fields = [])
    super(attr_name, class_name)
    serialized_attr_accessor attr_name, exposed_fields
  end

  def self.serialized_attr_accessor(attr_name, *args)
    args.first.each do |method_name|
      eval "
        def #{method_name}
          (self[:#{attr_name}] || {})[:#{method_name}]
        end
        def #{method_name}=(value)
          self[:#{attr_name}] ||= {}
          self[:#{attr_name}][:#{method_name}] = value
        end
        attr_accessible :#{method_name}
      "
    end
  end

  serialize :other_data,
    Hash,
    %w(some other values you want to store in serialized field)
end
Read More

Posted by on Jan 29, 2014 in Optimisations, Programming, Ruby On Rails, Server | 0 comments

Horizontal scaling using Db Charmer

I was looking for a way to scale horizontally a Ruby on Rails application, and i have tried several methods to scale it. A method would be using a MySQL cluster, but that would require some serious database administrator skills, which unfortunately i don’t have.

dbreplication173Mainly i have an application that is read intensive (80% reads vs 20% writes) so, i have considered to use a MySQL master – slave configuration. The problem is that there is nothing about it in Rails documentation, however, after a short look in ruby-toolbox.com I have discovered that I am not the only one who encountered this problem.

I have tried octopus as my first choice, but i have soon discovered that is not fit for my application. For some reasons, not all my “read” queryes were passed to my slave connection. I have tried to see why, but because I was kind of pressed by time, i have dismissed this gem, even if i love the simplicity of the models.

After dismissing octopus, I have tried db charmer gem, which is pretty active. This is yet another Active Record Sharding gem that offers you the possibility to split database reads and writes.

The method i have chosen for my first try was to split my actions that were 100% reads, and push them to a slave. That was pretty simple using a before filter in my rails controllers.

class ProfilesController < Application
  force_slave_reads :only =>  [ :show, :index ]
end

This action allowed me to scale the application by keeping the same amount of servers, but the main effect was a drop in the response time of the applications.

The second action i have taken was to get all the heavy queries like counts out of the mysql master server and move them to slave.

class User < ActiveRecord::Base
  def some_some_heavy_query
    self.on_slave.joins(:profile, :messages).count(:group => ['messages.thread_id'])
  end
end

In my enthusiasm of having a mysql slave I have thought that it would be nice to have “ready” 3 slave instances in my config. I have later realised that this “optimisation” caused problems because those 3 connections multiplied by the number of max_child in my apache configuration and also multiplied by the number of the servers exceded the number of the max_connection on my mysql slave server.

After a small fix in my database.yml files I was back online with a more performant application.

Read More

Posted by on Mar 28, 2013 in Programming, Ruby On Rails | 0 comments

How to reduce code duplication

By using Ruby or Pyton you might know about the fact that both are dynamically typed, as a result a developer can choose some of the variants available to remove some of the duplication. There are at least 2 methods that can be used to reduce code duplication.

Dynamic declaration:

class MyClass
  STATUS_ARRAY = [ :pending, :closed, :rejected, :waiting_for_payment ]
  STATUS_ARRAY.each do |method|
    class_eval %{
      scope :#{method}, where('column_name = ?', #{method.to_sym}) 
      def #{method}?
        self[:column_name] == '#{method}'
      end
      alias :is_#{method}?, #{method}?
    }
  end
end

In the example above we managed to write 8 lines that compress the code by declaring dynamically the equivalent of a 20 lines.

Inheritance

In Ruby you can easily use modules that you can include into your classes or you could use extend.

The below example you might encounter in all the Rails application upon class definitions, but this method is somehow limiting as you cannot extend with more than 1 class. In the below example we are extending the Foo class from Class1. Better said, Class1 is an ancestor of Foo

class Foo < Class1
end

However you could also use multiple inheritance by using the following example:

class Foo
  extend Class1
  extend Class2
end

The above example allows you to use several classes as a parent for Foo, however this might not be the best approach, because defies some OOP principles. As a workaround, is better to use modules.

Modules usage

class Foo < Class1
  include Module1 
  include Module2
end

I have shown here how to create a module

Read More

Posted by on Feb 3, 2013 in Programming, Ruby On Rails | 0 comments

How to implement Enum Fields in your Ruby on Rails models

If you reached this page, it might be because you have searched how the hell you could create an enum field in Ruby On Rails.

Short answer: Ruby On Rails prior to version 4 does not know how to deal Enum fields, so you cannot declare enum fields. However, you can hack your model, and implement your own enum 🙂

Code your own Enum Field

Long answer: Many developers need for a reason or another to have enum fields in their database. Maybe because they need to save a status of their object, or they need something more complex. By default Rails allows you to do this by using string fields, which later you could use to fetch your information by using scopes, or custom ActiveRecord or by declaring your own methods.

Personally i needed to have some enum fields in my Ruby on Rails applications, and i have seen that i am usually repeat the same stuff all over again, thing that lead me to write the snippet at the end of this post which later allowed me to use plain vanilla Enum fields into my Database.

First is i am creating my migration or my model where i add something like this:

class CreateProducts < ActiveRecord::Migration  
  def up
    create_table :products do |t|
       t.string :name
       t.integer :my_status_field, :limit => 1  #as a TinyInt 
       t.timestamps
    end
    add_index :products, :my_status_field
  end

  def down
    drop_table :products
  end
end

After i am creating the needed migration and the model, we are going to implement our status column inside a model.

class Product < ActiveRecord::Base
  STATUS_ARRAY = {
    :pending             => 1,
    :open                => 2, 
    :closed              => 3, 
    :rejected            => 4, 
    :waiting_for_payment => 5 
  }

  has_enum_field :my_status_field, STATUS_ARRAY
end

By using a snippet like the one above, you can easily use this kind of syntax:

# Active Record Scopes:
pending_products = Product.pending
open_products    = Product.open
# etc 

# inside an object, the following syntax

object = Product.new 
object.my_status_field = :open 

object.is_pending?  # => false 

object.my_status_field = STATUS_ARRAY[:closed]
object.pending?    # => false 
object.closed?     # => true
object.is_closed?  # => true

But, wait, there is more:
1. you have presence validators
2. you can disable the number 1 validators
3. you can disable the boolean columns
4. you can diable the scopes

We are gonna take the above product class, and we will add another hash to our customize our enum snippet, by disabling the validators and scopes.

class Product < ActiveRecord::Base
  STATUS_ARRAY = {
    :pending             => 1,
    :open                => 2, 
    :closed              => 3, 
    :rejected            => 4, 
    :vaiting_for_payment => 5 
  }

  ENUM_SETTINGS = {
    :validate => false, 
    :scopes => false, 
    :booleans => true
  }

  has_enum_field :my_status_field, STATUS_ARRAY, ENUM_SETTINGS 
end

Before running the below example, you might want to add in your “environment.rb” or “application.rb” a require statement to include the module globbaly into your project

Finally the module:

module EnumField
  class << self
    def included(klass)
      klass.class_eval do
        extend ClassMethods
        include InstanceMethods
      end
    end
  end 

  module InstanceMethods
  end

  module ClassMethods
    def has_enum_field(column_name, data_set, options = {:validate => true, :scopes => true, :booleans => true})
      data_set.keys.each do |ds|
        dat = data_set[ds.to_sym]
        
        class_eval %{
          validates_inclusion_of :#{column_name}, :in => #{data_set}.keys  
        } if options[:validate] 

        class_eval %{
          scope :#{ds}, where('#{column_name} = ?', dat)
        } if options[:scopes] 

        class_eval %{
          def #{ds}?
            self[:#{column_name}] == #{data_set}[:#{ds}]
          end

          alias_method :is_#{ds}?, :#{ds}?
        } if options[:booleans] 

        class_eval %{
          def #{column_name}=(value)
            self[:#{column_name}] = value.is_a?(Integer) ? value : #{data_set}[value.to_sym]
          end

          def #{column_name}
            #{data_set}.key(self[:#{column_name}])
          end
        }
      end
    end
  end
end

ActiveRecord::Base.send(:include, EnumField)

This module might have a problem thought, because is not 100% bullet proof. In order to achieve that, you might need to override the write_attribute method from Rails.

Update:

Once with release of Rails 4, this post can be deprecated, as it has been introduced by default in Rails. Please read more on the official wiki: http://api.rubyonrails.org/v4.1.0/classes/ActiveRecord/Enum.html

Read More