Wakatta!

Like Eureka!, only cooler

Seven Databases in Seven Weeks Redis Day 1

After a long winter hiatus, the elves at Pragmatic Bookshelf delivered a late but welcome present: the third beta of Seven Databases in Seven Weeks. The book is not complete yet (the chapter on CouchDB is still missing), but it now covers Redis.

Redis is basically a key-value store, like Riak, but while Riak is agnostic about the values, Redis values can be data structures (lists, queues, dictonaries, …, or even messaging queues). This allows Redis to act as a synchronized shared memory for cooperating applications.

Complex Datatypes

Redis values can have structure, and specific commands manipulate these values in appropriate ways. Redis supports strings, which can also behave as numbers if they have the right format, lists which can also be seen as queues, and support blocking reads, sets, hashes (that is, dictionaries), and sorted sets.

Transactions

All Redis commands are atomic, and it is possible to group a sequence of commands into a transaction for an all or nothing execution with the command MULTI. But a Redis transaction is not similar to a transaction in relational databases: it just queues all the commands and executes them when it receives the EXEC command. This means it is not possible to read any data while in a transaction.

Expiry

Perhaps nothing labels Redis as a datastore for transient data more than expiry: keys can be marked for expiration (either relative from the current time, or absolute).

Messaging

Redis also supports messaging but this is a topic for Day 2.

This post has a more detailed but still balanced coverage of Redis.

Exercises

Redis command documentation

The documentation is well done and easy to navigate. Of all the databases I have seen so far, this is probably the base (PostgreSQL being a strong second).

Create a Redis client

I’m using Java and the Jedis client library.

The code is simple enough:

simple redis client (RedisFirst.java) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
package jp.wakatta;

import redis.clients.jedis.Jedis;
import redis.clients.jedis.Transaction;

public class RedisFirst {
  public static void main(final String...args) throws Exception {
      // connect
      Jedis jedis = new Jedis("localhost");
      // set the key first to 5
      jedis.set("first", "5");
      
      // start a transaction
      Transaction trans = jedis.multi();
      // increase by 4
      trans.incrBy("first", 4);
      trans.exec();
      
      // retrieve the value
      System.out.println("Value is now: " + jedis.get("first"));
  }
}

The pom.xml file:

(pom.xml) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>jp.wakatta</groupId>
  <artifactId>redis-first</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <properties>
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-compiler-plugin</artifactId>
              <version>2.3.2</version>
              <configuration>
                  <source>1.6</source>
                  <target>1.6</target>
              </configuration>
          </plugin>
      </plugins>
  </build>

  <dependencies>
      <dependency>
          <groupId>redis.clients</groupId>
          <artifactId>jedis</artifactId>
          <version>2.0.0</version>
      </dependency>
  </dependencies>
</project>

Create a pair of Redis clients

This one is simple as well, but having a reader and a writer allowed me to try one writer and two readers.

First the writer program:

Redis Push (RedisPush.java) download
1
2
3
4
5
6
7
8
9
10
11
package jp.wakatta;

import redis.clients.jedis.Jedis;

public class RedisPush {
  public static void main(final String...args) throws Exception {
      Jedis jedis = new Jedis("localhost");
      jedis.lpush("msg:queue", "A new message");
      System.out.println("Message inserted");
  }
}

The poml.xml is a bit more complex, as it creates a self-contained jar with MANIFEST.MF (so I can run it from the command line easily):

(pom.xml) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>jp.wakatta</groupId>
  <artifactId>redis-push</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <properties>
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-compiler-plugin</artifactId>
              <version>2.3.2</version>
              <configuration>
                  <source>1.6</source>
                  <target>1.6</target>
              </configuration>
          </plugin>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-assembly-plugin</artifactId>
              <version>2.2.2</version>
              <configuration>
                  <descriptorRefs>
                      <descriptorRef>jar-with-dependencies</descriptorRef>
                  </descriptorRefs>
                  <archive>
                      <manifest>
                          <mainClass>jp.wakatta.RedisPush</mainClass>
                      </manifest>
                  </archive>
              </configuration>
              <executions>
                  <execution>
                      <id>make-assembly</id>
                      <phase>package</phase>
                      <goals>
                          <goal>attached</goal>
                      </goals>
                  </execution>
              </executions>
          </plugin>
      </plugins>
  </build>

  <dependencies>
      <dependency>
          <groupId>redis.clients</groupId>
          <artifactId>jedis</artifactId>
          <version>2.0.0</version>
      </dependency>
  </dependencies>
</project>

The reader program:

Redis Pop (RedisPop.java) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
package jp.wakatta;

import java.util.List;

import redis.clients.jedis.Jedis;

public class RedisPop {
  public static void main(final String...args) throws Exception {
      boolean again = true;
      while (again) {
          System.out.println("Waiting for messages");
          Jedis jedis = new Jedis("localhost");
          List<String> msgs = jedis.blpop(300, "msg:queue");
          
          System.out.println("Messages received:");
          
          for (String msg: msgs) {
              System.out.println(msg);
              if (msg.equals("finish"))
                  again = false;
          }
      }
      
      System.out.println("No more messages");
  }
}

with its pom.xml:

(pom.xml) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>jp.wakatta</groupId>
  <artifactId>redis-pop</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <properties>
      <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>

  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-compiler-plugin</artifactId>
              <version>2.3.2</version>
              <configuration>
                  <source>1.6</source>
                  <target>1.6</target>
              </configuration>
          </plugin>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-assembly-plugin</artifactId>
              <version>2.2.2</version>
              <configuration>
                  <descriptorRefs>
                      <descriptorRef>jar-with-dependencies</descriptorRef>
                  </descriptorRefs>
                  <archive>
                      <manifest>
                          <mainClass>jp.wakatta.RedisPop</mainClass>
                      </manifest>
                  </archive>
              </configuration>
              <executions>
                  <execution>
                      <id>make-assembly</id>
                      <phase>package</phase>
                      <goals>
                          <goal>attached</goal>
                      </goals>
                  </execution>
              </executions>
          </plugin>
      </plugins>
  </build>

  <dependencies>
      <dependency>
          <groupId>redis.clients</groupId>
          <artifactId>jedis</artifactId>
          <version>2.0.0</version>
      </dependency>
  </dependencies>
</project>

The blpop command can block on several lists, so when it receives something it is always at least a pair: the list key, and the value.

Now, I can open three terminals to test the code: two with readers:

1
java -jar target/redis-pop-0.0.1-SNAPSHOT-jar-with-dependencies.jar

and one with the writer (which must be started last):

1
java -jar target/redis-push-0.0.1-SNAPSHOT-jar-with-dependencies.jar

The writer will simply state

1
Message inserted

One of the readers will get the message:

1
2
3
4
5
Waiting for messages
Messages received:
msg:queue
A new message
Waiting for messages

but the other one will just keep waiting:

1
Waiting for messages

So Redis blocking queues can only server one blocking reader at a time (as it should).

The reader programs can be stopped with Ctrl-c, or by pushing finish into msg:queue from a Redis client (twice, once for each client):

1
2
3
4
redis 127.0.0.1:6379> lpush "msg:queue" "finish"
(integer) 1
redis 127.0.0.1:6379> lpush "msg:queue" "finish"
(integer) 1

And that’s all for today.

Comments