We are in the process of deploying memcached in one of our applications to server as a flexible, lightweight distributed cache. I’ve been familiar with the technology for years, but this was my first time to be able to directly play with it.
Memcached is written with the idea of do only one thing, and do it really really well. Its mission in life is to be a robust, high scalable in memory cache. When deployed in a distributed cluster, it completely ignores the problem inter-node communication. In fact, the nodes in a memcached cluster are completely unaware of each other’s existence. Instead, it pushes the problem down on the client, which is the only point that is actually aware of the existence of all the nodes. If a node is down, the client can simply adjust its hashing algorithm to start looking for data on another node.
I’m the kind of person who loves to pop open the hood and see how things work. Years ago, when my network admins were building out a pair of firewalls, I insisted on playing with the “heartbeat” cable. I would unplug to simulate a failure, watch the firewalls failover, and then reconnect to see it failback. I always smile and nod when people tell me about “automatic failover”, but I’ve been burned enough times that I always want to see it in action.
With memcached, naturally, I wanted to see how it handled failover as well. I wanted to take a cluster of 12 servers, stick a value in the cache, and then shut that server down and see it pop up somewhere else the next time a server accessed it (and trigger its reload into the cache).
Unfortunately, this is a little harder than it sounds. Memcached has a telnet interface that allows you to access a server and query information like server statistics, as well as view, add, and delete cache values. However, since each server is standalone, you have no way of knowing where a particular value might be. There are various memcached management tools out there, but they are more focused server status than querying the cache.
In the end, one of my team members write a simple powershell script that could take list of servers and a command and then execute the command on each server. In other words, it functions like a broadcast tool that allows you to send a message to all the servers in a cluster and see the results.
This is a little trickier than you would expect, since telnet does not lend itself well to being scripted. However, he found a blog post by an engineer in Australia about scripting an SMTP telnet session in powershell and adapted it to do something similar for a telnet session.
Here is the memcached script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
# param( [string] $command = $(throw "command is required. example .memdata.ps1 ""get SOMEKEYNAME""") ) # function readResponse($stream) { $encoding = new-object System.Text.AsciiEncoding $buffer = new-object System.Byte[] 1024 while ($stream.DataAvailable) { $read = $stream.Read($buffer, 0, 1024) Write-Output ($encoding.GetString($buffer, 0, $read)) } } # function callport($remoteHost, $port = 11211) { try { $socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) if ($socket -eq $null) { return; } $stream = $socket.GetStream() $writer = new-object System.IO.StreamWriter($stream) $writer.WriteLine($command) $writer.Flush() start-sleep -m 500 readResponse $stream } catch [Exception] { $ex = $_.Exception Write-Output ("Error: {0}" -f $ex.Message) } } # all servers in the cluster $servers = @( "SERVER1", "SERVER2", "SERVER3") $servers |% { $remoteHost = [string]$_ Write-Output $remoteHost # change value below to your memcached port if not the default of 11211 # and call callport as callport $remoteHost YourPortNumber callport $remoteHost } # script end |
To execute, you just run it with your desired telnet command in quotes:
.memcacheQuery.ps1 "get KEY1"
.memcacheQuery.ps1 "delete KEY1"
.memcacheQuery.ps1 "stats"
Updated: one of my former colleagues pointed out some issues with the script and offered some corrections. I’ve also migrated it over to github, since Posterous’s formatting was wreaking havoc with it.
Jeremy Rothman-Shore Blog 