Effective querying using SQL over Rails

Most of the rails developers write queries on daily basis. Some on the other day by mistake we follow rails way instead of SQL way to do tasks like sorting etc. But you might ask what’s the problem regarding which approach I take as long as I am able to get correct results. Yes, you may get correct results but you will end up with performance problems in future. Following a set of scenarios I am observed that SQL querying out powered rails:

1) Apply sorting on a combination of different tables:

I have three tables namely deposits, purchases and withdraws. Now, I want to display latest three records from a combination of three tables.

Let’s check rails way

  Benchmark.ms {(Deposit.all + Purchase.all + Withdraw.all).sort{|x, y| x.updated_at <=> y.updated_at}}

  Deposit Load (0.6ms)  SELECT "deposits".* FROM "deposits"
  Purchase Load (0.4ms)  SELECT "purchases".* FROM "purchases"
  Withdraw Load (0.7ms)  SELECT "withdraws".* FROM "withdraws"
 => 121.91500002518296

Let’s check SQL way

   (SELECT 'deposit', authorization_code, amount, status, updated_at
      FROM DEPOSITS
      ORDER BY updated_at DESC
      LIMIT 3)

      UNION

      (SELECT 'purchase', ref, amount, status, updated_at
      FROM PURCHASES
      ORDER BY updated_at DESC
      LIMIT 3)

      UNION

      (SELECT 'withdraw', ref, amount, status, updated_at
      FROM WITHDRAWS
      ORDER BY updated_at DESC
      LIMIT 3)

      ORDER BY updated_at DESC
      LIMIT 3

  => SQL (2.0ms)

2) Apply sorting on different types of data:

Let’s say a table stores gifs, videos of different duration. Now I want sort videos, gifs in ascending order of content duration separately.

Let’s check rails way

   Benchmark.ms do
      videos = SpotBooking.where(content_type: 'video').order('content_duration DESC')
      gifs = SpotBooking.where(content_type: 'gif').order('content_duration DESC')
      videos + gifs
   end

    SpotBooking Load (0.4ms)  SELECT "spot_bookings".* FROM "spot_bookings" WHERE      "spot_bookings"."content_type" = $1 ORDER BY content_duration DESC  [["content_type", 3]]

  SpotBooking Load (0.4ms)  SELECT "spot_bookings".* FROM "spot_bookings" WHERE "spot_bookings"."content_type" = $1 ORDER BY content_duration DESC  [["content_type", 2]]

 => 12.477000011131167

Let’s check SQL way

   SELECT id, content_duration, row_number()
   OVER (PARTITION BY content_type ORDER BY content_duration DESC)
   FROM spot_bookings

  => SQL (0.8ms) 

3) Formatting the data while querying:

Let’s check rails way:

  Benchmark.ms do
     h = {}
     pricings = SpotPricing.order('starts_at ASC')

     pricings.each do |b|
       h[b.starts_at.strftime("%H %M %P")] = b.price.to_f
     end
  end

  SpotPricing Load (0.5ms)  SELECT "spot_pricings".* FROM "spot_pricings" ORDER BY starts_at ASC

 => 35.459000151604414

    # output
 => {"02:19 AM"=>"1.00", "03:19 AM"=>"2.00", "04:19 AM"=>"3.00",
     "05:19 AM"=>"4.00", "06:19 AM"=>"5.00", "07:19 AM"=>"6.00", "08:19 AM"=>"7.00",
     "09:19 AM"=>"8.00", "10:19 AM"=>"9.00", "11:19 AM"=>"10.00", "12:19 PM"=>"11.00", 
     "13:19 PM"=>"12.00", "14:19 PM"=>"13.00", "15:19 PM"=>"14.00", "16:19 PM"=>"15.00", 
     "17:19 PM"=>"16.00", "18:19 PM"=>"17.00", "19:19 PM"=>"18.00", "20:19 PM"=>"19.00", 
     "21:19 PM"=>"20.00", "23:19 PM"=>"22.00", "00:19 AM"=>"23.00"}

Let’s check SQL way:

  SpotPricing.order('starts_at ASC').pluck("to_char(starts_at, 'HH24:MI AM')", "to_char(price, 'FM999999999.00')")
  
  SELECT to_char(starts_at, 'HH24:MI AM'), to_char(price, 'FM999999999.00') FROM "spot_pricings" ORDER BY ASC

  => SQL (0.5ms)

 => {"02:19 AM"=>"1.00", "03:19 AM"=>"2.00", "04:19 AM"=>"3.00",
     "05:19 AM"=>"4.00", "06:19 AM"=>"5.00", "07:19 AM"=>"6.00", "08:19 AM"=>"7.00",
     "09:19 AM"=>"8.00", "10:19 AM"=>"9.00", "11:19 AM"=>"10.00", "12:19 PM"=>"11.00", 
     "13:19 PM"=>"12.00", "14:19 PM"=>"13.00", "15:19 PM"=>"14.00", "16:19 PM"=>"15.00", 
     "17:19 PM"=>"16.00", "18:19 PM"=>"17.00", "19:19 PM"=>"18.00", "20:19 PM"=>"19.00", 
     "21:19 PM"=>"20.00", "23:19 PM"=>"22.00", "00:19 AM"=>"23.00"}

In all above scenarios, SQL plays best compare to conventional rails way. Sometimes I would feel it’s not a big deal if you not follow design patterns as long as your ability to provide the best outcome.

Any suggestions and feedback would be welcome!!!

Happy querying!!!

Live streaming using ROR, cloud front and S3

Streaming videos are one of the difficult jobs for many developers. Compare to images and remaining media formats videos are large in size. Hence, it is a frustrating experience for user waiting for long time to download the entire video. Then how to solve this problem? Here comes streaming. Streaming means constantly sending data into small chunks to end-user continuously to avoid problems mentioned above. But how to implement this solution. Here comes Amazon cloud front. As per Amazon here is the definition

“Amazon CloudFront is a global content delivery network (CDN) service that accelerates delivery of your websites, APIs, video content or other web assets through CDN caching. It integrates with other Amazon Web Services products to give developers and businesses an easy way to accelerate content to end users with no minimum usage commitments.”

Using cloud front we can stream the video content that has been stored on S3. Below are the steps to follow:

Creating Cloud front distribution:

First sign in into AWS account, then go to cloud front under Content & Network delivery section. Since we are doing live streaming we need to register a web distribution. Two options are most important while creating a distribution. 1) Origin Domain name and 2) SSL certificate. Origin Domain name is nothing but S3 bucket where your media content has been stored. For SSL certificate I would prefer to go with Default CloudFront Certificate since we want to stream through cloud front instead of custom domain. For remaining, you can go with default options.

Origin settings:

Since we want to stream the content through cloud front we need to restrict S3 bucket access to cloud front user only. To make this happen we need to add restriction policy to our S3 bucket through policy generator. Once you added a policy, it would available in your origin identities. By adding this we can restrict S3 bucket will be accessible through cloud front only.

For more details about configuration go through this video tutorial

Generating streaming URLs:

1) Create cloud front key pair to create cloud front URLs. You can create this by using cloud front key pairs under My security credentials.

2) Add aws-sdk to your Gemfile

3)

      # app/controllers/streams_controller.rb

      # To create signed cloud front URLs. Using this we can restrict video access.
      signer = Aws::CloudFront::UrlSigner.new(key_pair_id: <cloud front access ket>,
                                            private_key_path: <cloud front public key>)

      @url = signer.signed_url(<cloud front URL>/<Name of the video which you want to stream>, 
                               expires: Time.zone.now + 10.minutes)
       
   

Note: You can get cloud front URL from cloud front distribution.

Streaming:

To stream video, I am using html5box player.

# app/views/streams/show.html.erb

<div style="display: block; margin: 0px auto; position: relative; max-width: 100%; height:700px;"   class="html5gallery" data-responsive="true" data-responsivefullscreen="true" data-html5player="true" data-skin="darkness" data-width="800" data-height="450" data-showtitle=="false" data-autoplayvideo="true"  data-autoslide="true" data-effect="slide" data-showimagetoolbox="none" data-showplaybutton="false" data-showprevbutton='false' data-shownextbutton='false' data-showcarousel='false'>

  <a href="<%= @url %>"><img src="http://img.youtube.com/vi/YE7VzlLtp-4/2.jpg"></a> 
</div>

Note: html5box not only renders video but also it renders images & gifs as well.

Once you access show page then it starts streaming through cloud front. You can verify from your network tab of your browser.

Happy streaming!!!

seldom used but powerful aggregation operators

Most programmers who use MongoDB may have heard about its aggregation framework but only a few have used it to its full potential. Recently, in one of our projects, we got a complex requirement where the user needed to generate a report of sales done for any day, month or year. Using just MongoDB queries would have been very slow compared to the aggregation framework. So I decided to take up the challenge and learn the aggregation operators and make my code awesome! Here are the operators that helped me:

  # example we would refer throughout our post:

  class Booking
    include Mongoid::Document
    include Mongoid::TimeStamp

    field :agreement_amount, type: Float, default: 0.0
    field :project_name, default: ''
    field :is_active, type: Boolean, default: false
    field :amount_collected, type: Float, default: 0.0
    field :amount_refunded, type: Float, default: 0.0
    field :amount_dishonored, type: Float, default: 0.0
    field :due_installment_amount, type: Float, default: 0.0

    belongs_to :offer
  end

1. $and

$and is one of the MongoDB aggregation conditional operators where you can specify multiple conditions. Any document which satisfies all these conditions would get forwarded to next stage in the pipeline which is nothing but $group stage. Here is an example

# Today's sales value

{ 
  '$project' => 
     'todaysales' =>                                                                                   
         { '$cond' =>                                                                                    
            [                                                                                             
              { '$and' =>                                                                                 
                [                                                                                         
                  { '$gte' => ['$created_at', Time.zone.now.beginning_of_day.mongoize] },                      
                  { '$lte' => ['$created_at', Time.zone.now.end_of_day.mongoize] }
                  { '$eq' => ['$is_cancelled', false] }                                                   
                ]                                                                                         
              }, '$agreement_amount', 0                                                                   
            ]                                                                                             
         },  
     }
}

The above example returns the $agreement_amount if it is today’s booking.

2. $add and $subtract

Similar to conditional operators, we have arithmetic operators where you can do addition or subtraction. Here is an example:

{ 
  '$project' => 
     'yearlycollection' =>                                                                                   
         { '$cond' =>                                                                                    
            [                                                                                             
              { '$and' =>                                                                                 
                [                                                                                         
                  { '$gte' => ['$created_at', Time.zone.now.beginning_of_year.mongoize] },                      
                  { '$lte' => ['$created_at', Time.zone.now.end_of_day.mongoize] }
                  { '$eq' => ['$is_cancelled', false] }                                                   
                ]                                                                                         
              }, "$subtract" => [ { "$amount_collected", "$add" => [ "$amount_refunded", "$amount_dishonoured" ] } ] }, 0                                                                   
            ]                                                                                             
         },  
     }
}

In the above example, if the condition has been satisfied then it adds the $amount_refunded and $amount_dishonored and subtracts the result from the $amount_collected, otherwise it returns 0.

$ifNull:

This is similar to the conditional operator with a short syntax. It returns the value if value is null. Here is an example:

  {
    $project: {
      due_installment_amount: { $ifNull: [ "$offer_id", "$due_installment_amount" ] }
    }
 }

In the above example if offer_id is Null, only then does it return the due_installment_amount of that document.

4. $let:

Can we store an aggregated value in a variable and use in next expression? Yes! $let lets us do just that 🙂

{ 
  '$project' => 
    {
     'monthlycollection' =>                                                                                  
       {
         '$let' => { 
             collection: { 
                  $cond' =>                                                                                    
                    [                                                                                             
                     { '$and' =>                                                                                 
                         [                                                                                         
                           { '$gte' => ['$created_at', Time.zone.now.beginning_of_year.mongoize] },                      
                           { '$lte' => ['$created_at', Time.zone.now.end_of_day.mongoize] }
                           { '$eq' => ['$is_cancelled', false] }                                                   
                         ]                                                                                         
                     }, "$subtract" => [ { "$amount_collected", 
                                           "$add" => [ "$amount_refunded", "$amount_dishonored" ] } ] }, 0                                                                   
                   ] 
             }
          }
          in: { $cond: { '$gt': ['$$collection', 0] }, '$$collection', 0 }
       }
    }
}
        

In the above example, if amount_collected - (amount_refunded + amount_dishonored) is less than 0 for a particular booking then it returns 0 instead of -ve value. Notice that $$ is the convention to access variables within the aggregation pipeline.

Most of these seem like standard conditional operators but their uses are phenomenal. From MongoDB 3.2 onwards, aggregation framework provides next generation mongodb operators like $lookup (performs left outer join), $switch and others that we shall discuss in my next post.

Any suggestions would be welcome.

Enforcing coding guidelines using CircleCI and Rubocop

Being a rubyist we don’t just write code, we write beautiful code. But sometimes, in a hurry to complete the requirement some developers, especially new comer, might not follow the coding guidelines which would become headache later. Rubocop is here to rescue us from this scenario.

Let’s see how to integrate rubocop:

1. Add rubocop and generate config file:

      # Gemfile
      gem "rubocop", require: false
  
      # .rubocop.yml

      AllCops:
        Exclude:
          # You could specify set of files or dirs while you want to be ignored by rubocop
          - 'vendor/**/*'
          - 'spec/**/*'
          - 'config/**/*'
          - 'db/**/*'
          - 'lib/**/*'
          - 'bin/*'
          - 'Gemfile'
          - 'Gemfile.lock'
          - 'Rakefile'
      Documentation: # Disabling documentation cop
        Enabled: false
    

Here is the file where you can find list of cops that rubocop uses.

2. Run rubocop or through CircleCI:

      $ bundle exec rubocop -R

      Inspecting 47 files
        ...............................................

        47 files inspected, no offenses detected
   

where -R option make rubocop to run rails cops as well.

     #circle.yml

     test:
       post: 
         - bundle exec rubocop -R
   

This would run rubocop upon every commit on CircleCI. Though your test cases have been passed but if rubocop fails then your build fails.

Now let’s see how rubocop enforces you write to better & clean code:

Scenario-1: Using else with unless:

     # Before 
     unless @new_unit.open?
       'Unit is not available!!' 
     else 
       'Remark is empty!!' 
     end 
     
     # Comply to rubocop
     if @new_unit.open?
       'Remark is empty!!'
     else 
       'Unit is not available!!' 
     end
 

As a matter of fact, using else with if would make more sense than using with unless.

Scenario-2: Use next to skip iteration:

     # Before 
     @payment_vouchers.each do |pv|
       if pv.items.any?
         # code here
       end
     end
     
     # Comply to rubocop
     @payment_vouchers.each do |pv|
       next unless pv.items.any?
       # code here
     end
 

As you see after using next, code is more readable than before.

Rubocop not only highlights ruby code related issues but also pin points rails related issues as well.
Scenario-3: Use find_by instead of where.first:

     # Before 
     addr = addresses.where(type: 'primary').first
     
     # Comply to  rubocop
     addr = addresses.find_by(type: 'primary')
 

Scenario-4: Use proc instead of Proc.new:

     # Before 
     validates :address, presence: true, if: Proc.new{ |addr| add.type == 'primary' }
     
     # Comply to rubocop
     validates :address, presence: true, if: -> (addr) { addr.type == 'primary' }
 

Rubocop not only helps in covering all scenarios also insist on coding guidelines. Hope this would be helpful. Any feedback or suggestions are welcome.

What makes rspec3 so interesting?

Being an rspec fan I have been waiting for quite some time for the rspec3 final release. rspec3 has finally been released and it’s ready for production use. There are many interesting changes that has been incorporated in rspec3. Thanks to myron, Andy and david and other contributors. Here are the few changes that makes testing more fun:

Changes in rspec-expectations:

Compound Expectations: Composing a expectation using two or more expectations is called compound expectation. Here is the example:

  # In rspec2
  describe String do
    it "expect to be a instance of String and it should be equal to 'RUBY IS AWESOME'"
      string = "RUBY IS AWESOME"

      expect(string).to be_a(String)
      expect(string).to eql("RUBY IS AWESOME")
    end
  end

  # In rspec3
  RSpec.describe String do
    example "expect to be a instance of String and it should be equal to 'RUBY IS AWESOME'"
      string = "RUBY IS AWESOME"

      expect(string).to be_a(String).and eql("RUBY IS AWESOME")
    end
  end

Note: Not only combined pre-defined matchers you can also combined custom matchers also. Below is the example:

  RSpec::Matchers.define :be_a_first do |expected|
    match do |actual|
      actual.first == expected
    end
  end

  RSpec::Matchers.define :be_a_last do |expected|
    match do |actual|
      actual.last == expected
    end
  end

  RSpec.describe Array do
    describe "#sort" do
      example "expect 'AWESOME' be a first and 'RUBY' at last elements respectively" do
        expect("RUBY IS AWESOME".split(" ")).to be_a_first("AWESOME").and be_a_last("RUBY")
      end
    end
  end

Composing Expectation: Composing a expectation with pre-defined expectations or custom expectations so that you can express exactly what you want. Below is the example:

  RSpec.describe Array do
    describe "#sort" do
      example "expect 'AWESOME' be a first and 'RUBY' at last elements respectively" do
        expect("RUBY IS AWESOME".split(" ").sort).to match(be_a_first("AWESOME")).and match(be_a_last("RUBY"))
      end
    end
  end

Changes in rspec-mocks: The following changes would be interesting if you use mocks in your test suite.

Stub chain of methods: Once in a while you might face an issue how to stub chain of methods? In rspec2 you have done as follows:

  describe "String" do
    it "able to call upcase.split.reverse on a string object" do
      string = "ruby is awesome"

      allow(string).to receive(:upcase).and_return(upcase = "RUBY IS AWESOME")
      allow(upcase).to receive(:split).and_return(string_split = ["RUBY", "IS", "AWESOME"])
      allow(string_split).to receive(:reverse).and_return(["AWESOME", "IS", "RUBY"])

      expect(string.upcase.split(" ").reverse).to eql(["AWESOME", "IS", "RUBY"])
    end
  end

While in rspec3 you can do

   describe "String" do
    it "able to call upcase.split.reverse on a string object" do
      string = "ruby is awesome"

      allow(string).to receive_message_chain(:upcase, :split, :reverse).and_return(["AWESOME", "IS", "RUBY"])
      expect(string.upcase.split(" ").reverse).to eql(["AWESOME", "IS", "RUBY"])
    end
  end

Scopes:
Till now you couldn’t stub methods in before(:all) scope. Although you can do in rspec3, rspec team suggested not to use all the times. Here is the example how to use:

   RSpec.describe String do
      before(:context) do
        RSpec::Mocks.with_temporary_scope do
           string = "ruby is awesome"
           allow(string).to receive(:upcase).and_return("RUBY IS AWESOME")
           @hash = string.upcase
        end
      end 

      example "able to split the string" do
        expect(@hash.split(" ")).to eq(["RUBY", "IS", "AWESOME"])
      end 

      example "able to split and reverse" do
        expect(@hash.split(" ").reverse).to eq(["AWESOME", "IS", "RUBY"])
      end
    end

Changes in rspec-rails

In rspec-rails they have changed the structure a little. From rspec3 onwards, rspec team segregated core rspec configuration from rspec-rails configuration. All core rspec configuration like expectations.syntax = :expect, profile_examples = 10 moved to spec_helper and all rspec-rails configuration like use_transactional_fixtures moved to rails_helper. verify partial doubles: Prior to verifying partial double let see what is partial double. Partial double is a real object which acts as a test double in testing. Using verify_partial_doubles option you can cross check that by unknowingly you mock a method which is not there in the class. Here is the example:

  RSpec.describe Array do
    describe "#to_hash" do
      example "convert array to hash" do
        a = [1,2,3]

        allow(a).to receive(:to_hash).and_return({"1" => 0, "2" => 0, "3" => 0})
        expect(a.to_hash).to eql({"1" => 0, "2" => 0, "3" => 0})
       end
     end
   end

   # spec/spec_helper

   config.mock_with :rspec do |mocks|
     mocks.verify_partial_doubles = true
   end

Here is output when I run the specifications.

Failures:

  1) Array#to_hash convert array to hash
     Failure/Error: allow(a).to receive(:to_hash).and_return({"1" => 0, "2" => 0, "3" => 0})
       [1, 2, 3] does not implement: to_hash
     # ./spec/models/person_spec.rb:63:in `block (3 levels) in '

There is no method called to_hash exists on Array class. But I mock that method on that partial double. As method is not defined and I set verify_partial_doubles to true the above spec has failed. Till now we have seen changes in rspec-rails, rspec-mocks and rspec-expectations. Now we will see changes in rspec-core.

If you have observed carefully, I have not used 'it' throughout the post. In rspec3, it is suggested not to use 'it' to define a test case. Instead use 'example' to define a test case as 'it' might deprecated on future versions. Not just 'it', but a few other aliases have been added that make the test cases more readable. In rspec3, before(:example) is now an alias for before(:each) and before(:context) is an alias of before(:all). Not only can you use pre-defined aliases but also define your own aliases. Here is the example:

   RSpec.configure do |config|
     config.alias_example_to :frun, run: true
   end

Now you can use frun to define a test case with metadata as run:true. Here is the example:

   frun "should adds 'run:true' in example metadata" do
     expect(1+1).to eq(2)
   end

Now you can run the test case using run tag as follows:

  $rspec spec/models/person_spec.rb -t run
   Run options: include {:run=>true}
   .

   Finished in 0.02857 seconds (files took 1.1 seconds to load)
   1 example, 0 failures

Apart from few additions there are few things has been removed from rspec3. From no onwards you can’t do the following:

   RSpec.describe "Array" do
     subject { [1,2,3] }

     its(:length) { is_expected.to eq(3)}

     example "contains three items" do
       expect([1,2,3]).to have_at_least(3).items
     end
   end

To make above example work you have to include rspec-its and rspec-collection_matchers gems in your Gemfile. I hope these notable changes in rspec3 help you. Please help me expand this post by commenting on things I may have missed.

Real Time notifications using slanger and sidekiq

With the increasing expectation of web-applications, everyone wants real time updates or real time notifications to improve the web portals user experience. Understandably, my project required Real time notifications too. I successfully implemented and deployed on production. During my development I found some interesting gems, javascript libraries that I came across. So I thought it would be helpful if I share my experience. To make it more useful I have created a sample repository for demonstrating Real Time notifications for your further reference.

The AIM: Notify the user with a reminder on specific user defined date and time using Web notifications.

I choose slanger gem. (Recently updated by Jiren to make it compatible with rails4). To make it work, add pusher to rails app and for slanger instead of adding it in rails Gemfile, I created a sub directory called ‘slanger’ in my rails project and added slanger. Now we are done with installation. Now let’s see some configuration:

  #config/initializers/pusher.rb

  Pusher.app_id = app-id
  Pusher.key = app-key
  Pusher.secret = app-secret
  Pusher.port   = 4567
  Pusher.host   = 'localhost'
 

where app-id, app-key and secret could be any secure random number for authentication purpose.

Note: Add the IP address of the machine if your running app is in production mode or staging mode.

Now it’s time to add a sidekiq job to push notifications on a particular user defined time. Here is code snippet how to add sidekiq job:

# app/models/reminder.rb

class Reminder
  include Mongoid::Document
  field :text, type: String
  field :notify_at, type: Time

  after_create :add_sidekiq_job
  before_destroy :remove_sidekiq_job

  def add_sidekiq_job
    id = ReminderWorker.perform_at(scheduled_at.second, {id: self.id.to_s})
    self.update_attributes(job_id: id)
  end 

  def remove_sidekiq_job
    queue =  Sidekiq::ScheduledSet.new
    job = queue.find_job(self.job_id)
    job.delete if job
  end 

  def scheduled_at
    notify_at.to_time.to_i - Time.now.to_i
  end
end

The above code snippet adds a sidekiq job after reminder object has been created.

Now let’s trigger some notifications:

  # app/worker/reminder_worker.rb

 class ReminderWorker
  include Sidekiq::Worker

  def perform(args)
    reminder = Reminder.find(args['id'])
    Pusher.trigger("reminder", "reminder_12345", {notification_text: "Reminder: #{reminder.text}"})
  end
end

where trigger takes three arguments. First argument is ‘channel’ and second argument is ‘event’ and third one is data that you want to push to client.

Note: reminder_12345 is a unique way to identify to which user we have to push notification.

Once data has been published on channel the browser should respond and display data on front end. To do this every client should subscribe reminder channel using following javascript snippet:

  #app/assets/javascripts/remainder.js  

  var reminder = pusher.subscribe('reminder');

Prior to subscribe, pusher needs to authenticate. To do this, use following snippet:

  # app/assets/javascripts/remainder.js 

  window.pusher = new Pusher(PUSHER_API_KEY, {authEndpoint: '/pusher/authentication', authTransport: 'ajax',activityTimeout: 120000, disableStats: true});

Where PUSHER_API_KEY would unique key to identify your application. To know more about the remaining options please go through this.

Once you authenticated, client should respond once pusher triggers(nothing but event) a notification. To achieve this use following snippet:

  #app/assets/javascripts/remainder.js  

  reminder.bind('reminder_12345', function(data){
    $.notify(data['notification_text']);
   });

Once you watched above code snippet carefully you will get doubt about what is $.notify and why we need it?

notifyjs is a beautiful javascript library to display notifications on your front end. To know how to customize your notifications please go through the documentation.

So finally coalition of sidekiq, slanger and notifyjs has done job for me. It would be great if you share your thoughts on this coalition. Do play around with the demonstration on Github to see this in action.

Writing unit tests the right way

We all know what unit tests are and which gems we need to use for unit testing. Unit testing means testing a method in isolation. Unit tests are light weight and execute fast when compared to integration tests. You can write unit and integration tests either using rspec or minitest along with any mocking frameworks. Both frameworks have built-in support for mocking frameworks but in rspec they are extracted as a separate gem called rspec-mocks. Here are some pointers

Tests should help in debugging code.

We write tests to check whether we have covered all scenarios for our code. But we forgot to cross-check whether the test cases helps in debugging when an problem raises. Consider the following scenario

  # app/models/project.rb

  class Project
    has_many :buildings
    has_many :units
    has_many :bookings
  end

  # app/models/building.rb

 class Building
   belongs_to :project
   has_many :units
 end

  # app/models/unit.rb

  class Unit
    belongs_to :building
    has_many :bookings
  end

  # app/models/booking.rb

  class Booking
    belongs_to :unit
    belongs_to :project
    has_many :unit_installments
  end

There are 4 kinds of users called as DSGadmin, Partner, Area Sales manager(ASM), Sales Manger(SM) working under same company(technically called as Distributor in my system) are there. Here is the requirement:

• DSGadmin can see Partner, ASM, SM bookings and his bookings.
• Partner can see ASM, SM bookings and his bookings
• ASM can see see SM and his bookings
• SM can see his bookings only

Now, to test this I wrote some integration tests, thinking that they will be robust and quick.

   describe "Distributor Report", dont_clean: true do
     context "When DSG admin and partner has done a booking each" do
       before(:all) do
         @today = Date.today.to_s

         @project = create(:project)
         @franchise = create(:franchise)
         @dsg_admin = create(:dsg_admin, resource: @franchise)
         @partner = create(:partner, resource: @franchise, parent: @parent)

         @project.distributors = [@franchise]
         @project.save

         create(:distributor_admin_booking, resource: @franchise, sold_by: @dsg_admin, project: @project.id)
         create(:distributor_staff_booking, resource: @franchise, sold_by: @partner, project: @project.id)
       end 

       it "then DSG admin able to see 2 bookings" do
         login(@dsg_admin)

         post sales_reports_path, search: { project_id: @project.id, start_date: @today, end_date: @today, resource_key: "#{@franchise.id}@@Distributor"}

         expect(assigns[:report].first[1]).to eq(2)
       end 

       it "then partner able to see 1 booking" do
         login(@partner)

         post sales_reports_path, search: { project_id: @project.id, start_date: @today, end_date: @today, resource_key: "#{@franchise.id}@@Distributor"}

         expect(assigns[:report].first[1]).to eq(1)
       end
     end
   end

Distributor Report
  When DSG admin and partner has done a booking each
    then DSG admin able to see 2 bookings
    then partner able to see 1 booking (FAILED - 1)

Failures:

  1) Distributor Report When DSG admin and partner has done a booking each then partner able to see 1 booking
     Failure/Error: expect(assigns[:report].first[1]).to eq(1)
       
       expected: 1
            got: 2
       
       (compared using ==)
     # ./spec/requests/distributor_reports_spec.rb:172:in `block (3 levels) in '

Finished in 2.17 seconds
2 examples, 1 failure

As we can see, when I ran tests then one of my integration tests failed. However, it did not tell me where things failed. Now I realized the need to write some good unit tests and got down to it.

  describe "#b_match" do
    context "should return criteria as" do
      before do
        @sales.stub(:c_project).and_return(nil)
        @sales.stub(:resource).and_return(nil)
        @sales.stub(:default_match).and_return({'deleted_at' => nil})
      end 

      it "returns {'deleted_at' => nil} if no project or resource has selected" do
        @sales.b_match
        expect(@sales).to have_received(:default_match)
      end 

      it "returns {{'deleted_at' => nil, sold_by_id: {'$in' => @user.assigned_users_ids}} if selected resource is distributor" do
        expect(@sales.b_match).to eq('$match' => {'deleted_at' => nil, sold_by_id: {'$in' => @partner.assigned_users_ids}})
      end
    end
  end

  Sales 
    #b_match 
      returns criteria as {{'deleted_at' => nil, sold_by_id: {'$in' => @user.assigned_users_ids}} if selected resource is distributor


Failures:

  1) Sales#b_match should return criteria as returns {{'deleted_at' => nil, sold_by_id: {'$in' => @user.assigned_users_ids}} if selected resource is distributor
     Failure/Error: expect(@sales.b_match).to eq('$match' => {'deleted_at' => nil, sold_by_id: {'$in' => @partner.assigned_users_ids}})
       
       expected: {"$match"=>{"deleted_at"=>nil, :sold_by_id=>{"$in"=>["5378c281472d208bc100000d"]}}}
            got: {"$match"=>{"deleted_at"=>nil}}
       
       (compared using ==)
       
       Diff:
       @@ -1,2 +1,2 @@
       -"$match" => {"deleted_at"=>nil, :sold_by_id=>{"$in"=>["5378c281472d208bc100000d"]}}
       +"$match" => {"deleted_at"=>nil}
       
     # ./spec/reports/sales_spec.rb:75:in `block (4 levels) in '

Finished in 0.45649 seconds
9 examples, 1 failure

Now, with this failure, I found the issue in b_match method. In the fix below, it was obvious that the I required the `merge!` instead of the `merge` method. However, the integration tests never pointed to this particular failure – the unit test did!

  def b_match
     match = default_match
     if @current_resource.class == Distributor
+      match.merge!(sold_by_id: {'$in' => @user.assigned_users_ids})
-      match.merge(sold_by_id: {'$in' => @user.assigned_users_ids})
     end
     {'$match' => match}
   end

Unit tests should be independent

Unit tests should be less tightly coupled to avoid dependency. Consider the scenario where we need to calculate the compound interest if a customer didn’t pay their instalment on time. Here is a code snippet

  # app/models/unit_installment.rb

  class UnitInstallment
    field :percentage
    field :amount, type: Float, default: 0
    field :order, type: Integer
    field :received_amount, type: Float, default: 0
    field :interest_amount, type: Integer, default: 0
    field :is_due, type: Boolean, default: false

    belongs_to :booking

    before_create :initialize_is_due

    def initialize_is_due
      self.is_due = unit.building.building_schemes.where(payment_schedule_id: payment_schedule_id, scheme_id: scheme_id).first.try(:due)
      make_due
    end
  end

Now I have written the code to calculate the interest for a installment amount. Here is my test case to check interest calculation:

require 'spec_helper'

describe InterestAmount do
  before do
    UnitInstallment.any_instance.stub(:initialize_is_due).and_return(true)
    UnitInstallment.any_instance.stub(:update_booking).and_return(true)
    UnitInstallment.any_instance.stub(:valid?).and_return(true)
  end 

  let!(:booking) { create(:org_booking) }
  let!(:installment) { create(:unit_installment, booking: booking) }
  let!(:project) { create(:project) }

  context "#calculate" do
    it "should update interest amount on installment" do
      interest = InterestAmount.new(installment: installment, project: project, booking: booking)
      interest.stub_chain(:interest, :round).and_return(100)
      interest.calculate

      expect(installment.reload.interest_amount).to eq(100)
      expect(interest).to have_received(:interest)
    end
  end

Notice that, I have stubbed before_create callback which deals with unit, payment_schedule, scheme objects. So if either unit, payment_schedule or scheme are invalid then I can’t create unit installment object which eventually can’t check interest calculation.

I hope it helps you understanding how to write unit tests that help in debugging.

Handy gems for minitest-rails part2

Mocks and stubs are not new concepts that have been introduced in minitest. If you want to get detailed understanding about what are mocks and stubs you should read mocks aren’t stubs written by Martin Fowler. To achieve mocking and stubbing in minitest-rails you don’t need to include any separate gem like rspec-mocks in rspec. Let see how to do work with mocks and stubs in minitest then check what are gems available to add extra functionality.

Method stubbing
Stubbing a method means, set a pre-defined return value of a method in test-case. Let’s check how to do method stubbing in minitest:

  # app/models/user.rb
  class User < ActiveRecord::Base
    def foo
      "Executes foo"
    end

    def bar
      "Executes bar"
    end

    def new_method
      foo
      bar
    end
  end

  it "stubbing known method" do
    user = User.new
    user.stub(:foo, "Return from test foo") do
      user.new_method.must_equal "Executes bar"
    end
  end

In this example, I stub foo and check what would be the output of new_method. As matter of fact it won’t call actual foo method instead it calls stub foo method. Now that we have  seen how to stub and instance method, let check how to stub on any instance.

minitest stub any instance:

Using this gem you can stub a method on any instance of a class. The below example demonstrates how to do that:

  # app/models/user.rb
  class User < ActiveRecord::Base
    def foo
      "Executes foo"
    end

    def bar
      "Executes bar"
    end

    def new_method
      foo
      bar
    end
  end

  it "stubbing on any instance" do
    User.stub_any_instance(:foo, "Return from test foo") do
      p User.new.foo
      User.new.foo.must_equal "Return from test foo"
    end
    p User.new.foo
  end

Output:
1 tests, 1 assertions, 0 failures, 0 errors, 0 skips
Run options: –seed 36985

# Running tests:

“Return from test foo”
“Executes foo”

Notice that the stub method on any instance of User is valid only in the stub_any_instance block.

minitest-stub-const:

As name suggests you can stub constants of a class. Here is the example:

  it "stubbing a constant" do
    User.stub_const(:CONSTANT, "Test constant") do
      User::CONSTANT.must_equal "Test constant"
    end
  end

Here CONSTANT is a constant defined on User class. Using minitest-stub-const we can not only stub constants but also class methods. Here is the example:

  # app/models/user.rb
  def self.add
    "User created on development"
  end

  def add_user
    User.add
  end

  it "stub a class method" do
    m = MiniTest::Mock.new
    m.expect(:add, "Created user from testing")

    User.stub_const(:User, m) do
      User.new.add_user.must_equal "Created user from testing"
    end

    m.verify
  end

In above example, I stub User with mock object m. Now whenever User is referred, it is replaced with the mock object m. Now, we call the method add on the Mock object but this method is already mocked using m.expect(:add, "Created user from testing").

Mocking:

Mocking an object is one of the ways to do unit testing. Mock objects are the objects that we use in a test environment instead of real objects. We can create mocks using double in rspec to create dummy objects, where as in minitest you can do as Minitest::Mock.new which creates mock object. On mock object we can set expected messages. Here is an example:

  it "mocking a method" do
    user = Minitest::Mock.new
    user.expect(:another_method, "from test bar")

    user.another_method.must_equal "from test bar"

    user.verify
  end

You can also mock objects using minitest firemock as follows:

  it "mocking a method" do
    user = Minitest::FireMock.new("User")
    user.expect(:another_method, "from test bar")

    user.another_method.must_equal "from test bar"

    user.verify
  end

The only difference between Minitest::Mock and Minitest::FireMock is, when you try to mock undefined method then Minitest::FireMock raises an exception whereas Minitest::Mock won’t.

Hope this helps while you writing mocks and stubs in minitest. Any suggestions, queries would be welcome.

Handy gems for minitest-rails

Welcome to ‘minitest’ world! Minitest is yet another ruby testing framework but it’s not a replacement to rspec. In rspec you can readily use things like ‘subject’, ‘metadata’ etc. However, minitest, as the name suggests, is a light-weight testing framework and does not include a lot of these features automatically. Here is a list of gems that you can use to enhance minitest-rails.

1. m

m is a test runner similar to rspec command in rspec framework. Using this we can run single test using line numbers too. Users who don’t like this gem can also run tests using default rake tasks provided by minitest.

2. minitest-metadata

metadata is one of the cool features in rspec. Using this we can tag scenarios across files. Unfortunately minitest doesn’t support this feature by default. We have to include a separate gem called minitest-metadata. We can implement rspec conditional hooks functionality through this gem as follows:

  require "test_helper"

  describe UsersController do

    before do
      @user = create(:user) if metadata[:before_each] == true
    end

    it "must create user" do
      assert_difference('User.count') do
        post :create, user: attributes_for(:user)
      end

      assert_redirected_to user_path(assigns(:user))
    end

    it "must show user", before_each: true do
      get :show, id: @user
      assert_response :success
    end
  end

When we run the file before code block will run only for the latter test case.

3. minitest-implicit-subject

subject is one of the best ways to dry up your specs. To dry up your specs in minitest you have explicity include a gem called minitest-implicit-subject. Here is an example:

  require "test_helper"

  describe User do
    it "foo returns bar" do
      subject.must_equal User
    end
  end

There is slight difference that I have observed in minitest subject from rspec subject. In minitest if you write subject it returns current class name under which it has included. In this scenario it returns output as User class not User object. But in rspec it returns User object.

4. minitest-spec-expect

rspec expectation framework lets you code example more readable. In minitest there is separate gem called minitest-spec-expect available to make your code example readable as same as in rspec. Here is an example:

  require "test_helper"

  describe User do
    it "must be valid" do
      expect(subject.new.valid?).to_equal true
    end
  end

5. minitest-rails-capybara

Using capybara is one of the ways to implement your integration test suite in rspec. We need to include separate gem called minitest-rails-capybara in minitest to write integration tests using the capybara DSL. Here is a simple example:

   require "test_helper"

   feature "User" do
     scenario "should be able save" do
       visit users_path
       click_link 'New User'
       fill_in "user_first_name", with: 'Siva'
       click_button 'Create User'

       expect(User.all.count).to_equal 1
     end
   end

In my next post I will talk about the gems that allow us to mock and stub requests in minitest.

Pro tips for writing better rspec tests

Writing ‘good code’ is always a challenge for every project. Unfortunately, we always associate it with development and not tests. What about automation test code? Ever noticed that you have to write more automation code than the actual lines of development code?

Improper automation code leads to slow running specs, huge LOC in your spec files and a whole lot of confusion when you open a spec file. So, how does one avoid these problems?

Here are a few guidelines that I learned from my experience:

Tip #1: Follow Rspec style guide

That was obvious, wasn’t it? As surprising as it may seem, we invariably do not follow in detail.

• How often do you see more than one expectation in an example? bad
• How often do you see lots of stubs and mocks in test code? bad – keep it as simple as you can.
• How often do you see the use of context, especially in controllers? good

A lot of them are mentioned in the Style guide and we shall repeat below a few which require more details.

Tip #2: Independent tests

We could use the word idempotent here but as a thumb rule there should not be any dependencies across specs and if we run the test multiple times in succession, it should give the same result So, it’s very important that every spec  run in clean database environment. Use database cleaner gem properly, so that you don’t end of having to seed the database before each test! So, if you don’t want roles and countries being deleted,

config.before(:suite) do
  DatabaseCleaner.strategy = :truncation, {:except => %w[roles countries]}
end

Tip #3: Use expect and not should syntax

It’s just nice and clean!  Using the Transpec gem helps you to convert should syntax to expect syntax

expect(User.all.count).to eq(1)

instead of

User.all.count.should == 1

Tip#4: Use the new syntax for object creation

Use the new FactoryGirl syntax that is supported in factory_girl4. This helps the code to be clean of any harsh gem dependent syntax of class names!

create(:user)

instead of

FactoryGirl.create(:user)

Tip #5: Use let variables instead of instance variables

This makes code look cleaner. For better understanding about let read the answer given by Myron. Here is an example

describe Project do
  let(:project) { build(:project) }

  it "project should be valid" do
    expect(project.valid?).to eq(true)
  end
end

Now, when the example executes it defines a method called project if its not defined already or simply returns the value. Fast and efficient.

Tip #6: Use correct expectations for a spec

Obvious as it seems – the test we write should test the write stuff – not just make the test pass! Here is an example of making a test case pass.

context "#edit" do
  it "user should be able to edit his profile" do
    get :edit, id: user.id
    expect(response).to be_success
  end
end

So, your expectation should ideally be

describe UsersController do
  context "#edit" do
    #render_views
    it "user should be able to edit his profile" do
      get :edit, id: user.id
      expect(response).to render_template("edit")
    end
  end
end

Here #edit is a method defined in UsersController. So I defined a context in my specs.

Do remember that by default all controller specs stub the template rendering unless and until you include render_views

Tip #7: Use shared examples.

When you want to test the same functionality across different subjects, for example different kinds of users, use shared examples.

context "#edit" do
  it "admin should be able to edit his profile"
  it "distributor should be able to edit his profile"
end

You can write as

context "#edit" do
  shared_examples "profile_editing" do
    it "user should be able to edit his profile"
  end

  context "admin" do
    include_examples "profile_editing"
    it "should be able see to list of distributors"
  end

  context "distributor" do
    include_examples "profile_editing"
    it "should not able to see list of distributors"
  end
end

A word of caution – you may loose readability if you use too many shared examples in single file.

Tip #8: Use conditional hooks

Sometimes if you want to execute some piece of code only for one example but not for all examples in a context then you can use conditional hooks.

context "#update" do
  before(:each, run: true) do
    # do something
  end

  it "example1", run: :true
  it "example2"
end

Now before(:each) will run only for first example but not for second example.

Tip #9: Profiling your examples

rspec gives you an option to examine how much time does a example group take to run.  Simply run specs with the –profile flag.

rspec ./spec/models/user_spec.rb:51 --profile
# output:
0.22431 seconds ./spec/models/user_spec.rb:51

Tip #10: Tagging

When you have a large test suite and you changed some code, how do you ensure that your change didn’t breaks existing functionality? One way is to run your specs locally then push then your Continuous Integration server that runs all specs. This is time consuming if you want to test only a specific set of tests.

Tag them! Run your module specified specs by using tags. For example, suppose we have a Commission feature and we have written model specs, controller specs and integration specs for it. We can easily tag each file with commission:true.  So when we change something related to commissions,  we can run only commission related specs!. We can tag our scenarios as below

   # spec/models/commission_spec.rb
   describe Commission, commission: true do
     # do something
   end

rspec --tag commission

Note: We can tag entire context or a single example in the spec file. For more information please follow this link

All these tips are a results of practical problems I have faced and overcome and a result of finding out the best we can do to improve our own testing. More suggestions and tips are welcome!