using redis to track and uniquify impressions with rails

as redis has become the number one contender for memcaches throne of key-value storage engines. it deserves a closer look why so many rails developers are flocking to it.

redis has a set of commands and functions that enable endless possibilities to those creative enough to use them in a new way.

today i want to show how we use redis to count the picture views in our spray can app.

first to explain what we want to do and why:
in our ios app users can upload and view the paintings of other users. we track those views and use them as a scoring mechanism to rate and order the pictures. when we started we quickly discovered that users were gaming the system be simply viewing a picture over and over again to get better ratings.
so the views had to be unique per user/picture in order to prevent gaming the system.
the user base at this time was rather small so i decided to go with a simple postgresql table and an unique index across two columns. but when we started to hit up as many as 20M views per day my postgresql database was becoming a serious bottle neck.

so i devised a clever solution:

we use the built in ‘set’ feature of redis where keys can be grouped into a such a set and are automatically
uniquified.

the basic configuration of the redis gem is like in my previous post.

so as a metal poller (yes this is an rails 2.3 example) i have:
app/metal/poller.rb

# Allow the metal piece to run in isolation
require(File.dirname(__FILE__) + "/../../config/environment") unless defined?(Rails)

class Poller
  def self.call(env)
    if env["PATH_INFO"] =~ /^\/pictures\/viewed/
      request = Rack::Request.new(env)
      params = request.params
      $redis_cache.sadd('spraycan_recent_views', [params['uuid'], params['pic_id']])      
      [200, {"Content-Type" => "text/json"}, ["OK"]]
    else
      [404, {"Content-Type" => "text/html"}, ["Not Found"]]
    end
  end
end

here i simply write the pair of device identifier and picture identifier into a set.
now i can build a rake task which gets the unique new views of the last few minutes and save the count to my pictures.

lib/tasks/process_views.rake

task "process_views" do
  require "./config/environment"

  @view_sets = $redis_cache.sdiff('spraycan_recent_views', 'spraycan_views_today')
  $redis_cache.sunionstore("spraycan_views_today","spraycan_recent_views","spraycan_views_today")
  $redis_cache.del('spraycan_recent_views')
  
  @view_sets = @view_sets.map{|e| JSON.parse e}
  @view_sets = @view_sets.inject(Hash.new(0)){|h,e| h[e.last]+=1;h} #kudos to manuel
  
  @view_sets.each do |pic_id, counter|
    begin
      @picture = Picture.find(pic_id.to_i)
      @picture.update_attribute(:views, @picture.views.to_i + counter)
      @picture.update_score
    rescue
      #picture not found, happens very rarely and shouldn't stop the rake task
    end
  end
  
end

so let’s have a closer look. since i want to have my views unique over a certain period of time (i.e. a whole day) but also want to update my view counter every 10 minutes i have to store the views of the day in another set called ‘spraycan_views_today’.
that means, that the difference between my recent views (last 10 minutes) and all the prior views of the day are the new unique views i want to count and register in my database.
(the adding ofnthe recent views to todays views using union and then clearing the recent views again should be easy to grasp)
so i take these views and transform them to a Hash that holds picture identifiers and the count of devices that looked at it.

now i only have to save that to my database and i am all done.

i hope you liked this quick example of what awesome stuff you can do with redis once you understand the possibilities.

have fun.