Rough counters on Google App Engine

On Google App Engine the maximum frequency of updates for a single database entity is limited. If you need to count something with high frequency such as page views on a web site or similar you need to do some magic to manage it.

The solution found in documents by Google is to use sharded counters. The principle is that you have several database entities, pick one at random and update that entity. The number of updates possible scales with the number of entities you use. However to get the value of the counter you need to get all entities and sum them. Also the code to do this is a bit complex. You pay for that complexity with cpu-resources.

If you need a counter that allows a high update frequency but don't want the fuzz of sharded counters and it doesn't matter if you occasionally miss a count or two there is a better solution. For some of my applications I instead use rough counters. The basic principle is that you let your instance cache the counts locally and occasionally write the counter to the database. You'll waste far less resources, you can read the value of the counter directly.

Note: If your instance shuts down you loose the current counter value, the error whill likely be less than a tenth of a percent, but it isn't exact.

The code:

import javax.jdo.JDOObjectNotFoundException;
import javax.jdo.PersistenceManager;
import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.NotPersistent;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;

@PersistenceCapable(detachable="true")
public class RoughCounter {
 @PrimaryKey
 @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
 private String mName;
 
 @Persistent
 private long mRoughCount;
 
 @NotPersistent
 private long mLastUpdate;
 
 public RoughCounter(String name) {
  mName = name;
  mRoughCount = 0;
  mLastUpdate = 0;
 }
    
 public void increase() {
      mRoughCount += 1;
      if (mLastUpdate > System.currentTimeMillis() - 10 * 1000) return;
      mLastUpdate = System.currentTimeMillis();
     
      long count = mRoughCount;
      mRoughCount -= count;
     
  PersistenceManager pm = PMF.get().getPersistenceManager();
  try {
   RoughCounter rc = pm.getObjectById(RoughCounter.class, mName);
   rc.mRoughCount += count;
   pm.makePersistent(rc);
  } catch (JDOObjectNotFoundException ex) {
   RoughCounter c = new RoughCounter(mName);
   c.mRoughCount += count;
   pm.makePersistent(c);
  } finally {
   pm.close();
  }
 }    
}

This counter writes the results to the database around once every ten seconds. The counting can occur at ANY speed, this is not depending on database performance, only on execution speed which is extremely high in comparison to database update performance.