Redis Lua scripting is badass

Roughly a year ago Salvatore Sanfilippo the author of Redis wrote a blog post discussing the inclusion of Lua as a scripting language. I finally decided to try this out, and let’s just say it’s pretty badass.

Lua is a great fit for Redis, they have similar philosophies, being simple, small, and fast. Suppose for example you have 200,000 jobs, each represented in Redis as a hash, and you want to map/reduce the job duration, the new scripting capabilities make this really easy!

Here’s the node setup script to generate these jobs:

  var redis = require('redis')    , db = redis.createClient();  var n = 500000    , pending = n    , ms;  while (n--) {    ms = Math.random() * 200 | 0;    db.hset('job:' + n, 'duration', ms, function(){      --pending || process.exit();    })  }

Next here is what you might consider scripting in your host language without the new Redis scripting feature, manually reducing the value:

  var redis = require('redis')    , db = redis.createClient();  var n = 200000    , start = new Date    , pending = n    , ms = 0;  while (n--) {    db.hget('job:' + n, 'duration', function(err, n){      if (err) throw err;      ms += ~~n;      --pending || (function(){        console.log('%d minutes spent processing jobs', ms / (1000 * 60) | 0);        console.log('took %ds', (new Date - start) / 1000 | 0);        process.exit();      })();    })  }

On my Air this took roughly 7s, not too great, keep in mind that there is no throttling here I’m just plastering it with 200k commands. Now let’s try it with Lua! The following script is ad-hoc, but it’ll do the trick. To signal an error all you have to do is return a table with the err slot. redis.call() is effectively the public Redis API exposed to your Redis script, so you can use it just like you would your host language Redis bindings or redis-cli(1).

   local sum = 0   for i = 0, 200000, 1 do     local key = "job:" .. i     local ms = tonumber(redis.call("hget", key, "duration"))     if ms == nil then return { err = key .. " is not an integer" } end     sum = sum + ms   end   return sum

Here I’ve embedded it in the JS script, but you could of course generate these, load them from files etc (beware of redis-injection?).

   var redis = require('redis')     , db = redis.createClient();   var script = '   local sum = 0    for i = 0, 200000, 1 do      local key = "job:" .. i      local ms = tonumber(redis.call("hget", key, "duration"))      if ms == nil then return { err = key .. " is not an integer" } end      sum = sum + ms    end    return sum';   var start = new Date;   db.eval(script, 0, function(err, ms){     if (err) throw err;     console.log('%d minutes spent processing jobs', ms / (1000 * 60) | 0);     console.log('took %dms', new Date - start | 0);     process.exit();   });

After running the script with the new EVAL command what previously took several seconds dropped to ~850ms, much better. EVAL and EVALSHA are actually a lot more flexible than I’ve explained here, accepting keys and arbitrary arguments.

Check out antirez.com/post/an-update-on-redis-and-lua.html for more.

• Tags:
  • redis
  • lua
  • node
  • js

• March 14, 2012, 6:23pm

        17 notes

• Permalink