söndag 15 maj 2011

Rough counters on Google App Engine

On Google App Engine the maximum frequency of updates for a single database entity is limited. If you need to count something with high frequency such as page views on a web site or similar you need to do some magic to manage it.

The solution found in documents by Google is to use sharded counters. The principle is that you have several database entities, pick one at random and update that entity. The number of updates possible scales with the number of entities you use. However to get the value of the counter you need to get all entities and sum them. Also the code to do this is a bit complex. You pay for that complexity with cpu-resources.

If you need a counter that allows a high update frequency but don't want the fuzz of sharded counters and it doesn't matter if you occasionally miss a count or two there is a better solution. For some of my applications I instead use rough counters. The basic principle is that you let your instance cache the counts locally and occasionally write the counter to the database. You'll waste far less resources, you can read the value of the counter directly.

Note: If your instance shuts down you loose the current counter value, the error whill likely be less than a tenth of a percent, but it isn't exact.

The code:

import javax.jdo.JDOObjectNotFoundException;
import javax.jdo.PersistenceManager;
import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.NotPersistent;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;

public class RoughCounter {
@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private String mName;

private long mRoughCount;

private long mLastUpdate;

public RoughCounter(String name) {
mName = name;
mRoughCount = 0;
mLastUpdate = 0;

public void increase() {
mRoughCount += 1;
if (mLastUpdate > System.currentTimeMillis() - 10 * 1000) return;
mLastUpdate = System.currentTimeMillis();

long count = mRoughCount;
mRoughCount -= count;

PersistenceManager pm = PMF.get().getPersistenceManager();
try {
RoughCounter rc = pm.getObjectById(RoughCounter.class, mName);
rc.mRoughCount += count;
} catch (JDOObjectNotFoundException ex) {
RoughCounter c = new RoughCounter(mName);
c.mRoughCount += count;
} finally {

This counter writes the results to the database around once every ten seconds. The counting can occur at ANY speed, this is not depending on database performance, only on execution speed which is extremely high in comparison to database update performance.

tisdag 19 april 2011

Key value store on Google App Engine

Just created a quick hack to use Google App Engine (GAE) as a simple key value store with a nice rest api. This allows usage as a data store for web applications and also a flexible way to serve web sites. It can also be used as an application server with similar thinking to CouchApp.

Have a look at:

torsdag 20 januari 2011

Sums in Microsoft Excel

I often use MS Excel (2007) to evaluate data... simple and convenient for many cases...

Unless you try to sum several integer values and they exceed 1'000'000'000'000'000. By then Excel starts losing precision. I'd expect 64bit integer math to work correctly, but it doesn't!