Redis: the basic data structures

The first chapter is for you get an overview of the core Redis data structures - STRING, LIST, SET, HASH and SORTED SET. A practical application is used to explain the concepts

This is by no means a complete list of all the Redis data structures

News sharing app

This application is loosely inspired by Hacker News. It's features are as follows

• Basic

  • User registration (using a unique name)

  • Submit a news item

  • Get a news item

  • View all news items sorted in descending order by upvotes

• Comments

  • Post a comment on a news item

  • View all comments for a news item

• Upvote

  • Upvote a news item

  • Get number of upvotes for a news item

All the functionality (registration, news items submission etc.) is exposed via a REST API

Technical stack

The following components have been used

• Gin is the HTTP web framework for REST API
• go-redis as the Redis client
• Docker

  • pre-built Docker image for Redis
  • Docker Compose to orchestrate all the application components

Schema

Here is a quick overview of the application specific entities and the Redis data structures they map to

• users - a SET which stores all users (just the user names to keep it simple)

• news:[id] - news details are stored in a HASH (id is the news ID)

• news-id-counter - a STRING data type

• news:[id]:comments - a LIST to store comments associated with a news item (id is the news ID)

• comments-id-counter - STRING key which acts as a counter for generating comment ID

• news-upvotes - SORTED SET with news ID as key and number of upvotes as value

Implementation details

Its time to dive into the nitty gritty. This section focuses on the code and the details of how to test the application is covered in the next one

Source code is available on Github

The code package structure is as follows

│   main.go
│
├───api
│       comments-api.go
│       middleware.go
│       newsitem-api.go
│       upvotes-api.go
│       user-api.go
│
└───model
        models.go

The api directory contains logic for the REST API handlers for news items, comments etc. models contains a single file models.go which has the Go structs representations for the JSON data which will be exchanged between the client and the API. Finally, main.go serves as the entry point

Let's move on to the details

New user registration

The new user registration process is implemented using a REST endpoint which accepts the user name as an input (via HTTP POST). It adds to the users set using SADD (e.g. SADD users abhirockzz)

func UserRegistrationHandler(c *gin.Context) {

    userByte, _ := ioutil.ReadAll(c.Request.Body)
    ...

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    num, err := client.SAdd(usersSet, string(userByte)).Result()
    ...

    if num == 0 {
        fmt.Println("user already exists")
        c.Status(http.StatusConflict)
        return
    }

    ...
    c.Status(http.StatusNoContent)
}

• A Set stores unique elements only - same applies to SETs in Redis

• SADD returns 0 in case the element (the user name in this case) is not added to the set - this allows registration request for a duplicate user to be rejected

Redis connection management

In the above code snippet, notice the following lines

clientCtxObj, _ := c.Get("redis")
client := clientCtxObj.(*redis.Client)

client is nothing but a handle to the redis.Client object - but was the connection established in the first place ? A Gin middleware was used to achieve this. As a result

• connection to Redis is established before invoking any of the HTTP handler (in response to a API request)

  the request is aborted in case the connection is not successful (there is no point proceeding)

• the Redis connection is closed after the request

    func RedisConnHandler(c *gin.Context) {
  
        connected := true
  
        redisServer := os.Getenv("REDIS_SERVER")
        if redisServer == "" {
            redisServer = "192.168.99.100:6379"
        }
  
        client := redis.NewClient(&redis.Options{Addr: redisServer})
        err := client.Ping().Err()
        if err != nil {
            c.AbortWithError(http.StatusInternalServerError, err)
            connected = false
        }
  
        c.Set("redis", client)
  
        c.Next()
  
        if client != nil && connected {
            client.Close()
        }
    }

Redis server endpoint is extracted from the environment variable REDIS_SERVER. On successful connection, the object is set in a (Gin) context specific variable called redis - the HTTP request handler(s) (as seen above), Gets the client object from the context variable

Submit a news item

The REST API for news item submission accepts the news item details via HTTP POST and it makes of the HASH, SORTED SET and STRING data structures

the API simply returns the (unique) ID for the submitted news item (as the response)

func PostNewsItemHandler(c *gin.Context) {

    var news model.NewItemSubmission
    c.ShouldBindJSON(&news)
    submittedBy := c.GetHeader("user")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    exists, sisMemberErr := client.SIsMember(usersSet, submittedBy).Result()

    if sisMemberErr != nil {
        c.AbortWithError(500, sisMemberErr)
        return
    }

    if !exists {
        c.AbortWithError(500, errors.New("Invalid user "+submittedBy))
        return
    }

    newsID, err := client.Incr(redisNewsIdCounter).Result()
    if err != nil {
        c.AbortWithError(500, err)
        return
    }

    newsHashKey := redisNewsHashPrefix + ":" + strconv.Itoa(int(newsID))

    newsDetails := map[string]interface{}{"title": news.Title, "url": news.Url, "submittedBy": submittedBy}
    _, hmsetErr := client.HMSet(newsHashKey, newsDetails).Result()

    if hmsetErr != nil {
        c.AbortWithError(500, hmsetErr)
        return
    }

    _, zaddErr := client.ZAdd(redisNewsUpvotesSortedSet, redis.Z{Member: newsID, Score: 0}).Result()

    if zaddErr != nil {
        c.AbortWithError(500, zaddErr)
        return
    }

    c.JSON(http.StatusCreated, newsID)
}

• the users SET is first checked to see if a registered user has submitted the request

• The STRING data type (news-id-counter) is used as a unique (news) ID generator. The interesting part is that it is integer aware and can be incremented in an atomic manner using INCR command (e.g. INCR news-id-counter) which makes it ideal for implementing counters

• Once the unique news ID is generated, the news item details are stored in a HASH (using HMSET command). Think of HASH as a Map-like data structure which is a great fit for storing objects. In this case, we're storing the url, title and submittedBy attributes for a news item in HASH whose naming convention is news:[id] e.g. news:42. Here is a what the an example command look like - HMSET news:42 title "Why did I spend 1.5 months creating a Gameboy emulator?" url "http://blog.rekawek.eu/2017/02/09/coffee-gb/" submittedBy "tosh"

• Finally, we add the news item ID and the number of upvotes (zero to begin with) to a SORTED SET (called news-upvotes). This data structure is like a SET (unique elements only) but with an additional property that the elements/members can be associated with a score which opens up a bunch of interesting options (discussed in upcoming section)

Comment on a news item

Implementing a REST API for comments on a post is as as simple as using the LPUSH command to add it to a Redis LIST whose naming pattern is news:[id]:comments (e.g. news:42:comments). LPUSH inserts the comment to the beginning of the list such that the older comments are pushed further in. This way, when fetched (details below), the latest (newer) comments will show up first

func PostCommentsForNewsItemHandler(c *gin.Context) {
    commentsBytes, _ := ioutil.ReadAll(c.Request.Body)
    newsID := c.Param("newsid")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    commentsListName := "news:" + newsID + ":comments"
    _, lpushErr := client.LPush(commentsListName, string(commentsBytes)).Result()

    if lpushErr != nil {
        c.AbortWithError(500, lpushErr)
        return
    }

    c.Status(http.StatusNoContent)
}

Fetch comments for a news item

The REST API to get all the comments associated with a specific news item makes use of the LRANGE command which returns a subset of the list (given start and stop index). In this case we want all the contents, hence our start index is 0 and the stop index is the length of the list itself e.g. LRANGE news:42:comments 0 10. We find out the length of the list using LLEN command

func GetCommentsForNewsID(c *gin.Context) {
    newsID := c.Param("newsid")
    commentListName := "news:" + newsID + ":comments"

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    numOfComments, llenErr := client.LLen(commentListName).Result()
    if llenErr != nil {
        return
    }

    if numOfComments == 0 {
        c.Data(200, "text/plain", []byte("no comments found"))
        return
    }

    comments, lrangeErr := client.LRange(commentListName, 0, numOfComments).Result()
    if lrangeErr != nil {
        c.AbortWithError(500, lrangeErr)
        return
    }

    c.JSON(200, model.NewsItemComments{newsID, comments})
}

Upvote a news item

Upvoting a news item is achieved by incrementing its score in the SORTED SET (news-upvotes) using the ZINCRBY command

func UpvoteNewsItemHandler(c *gin.Context) {
    newsID := c.Param("newsid")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    _, zincrErr := client.ZIncr(redisNewsUpvotesSortedSet, redis.Z{Member: newsID, Score: 1}).Result()

    if zincrErr != nil {
        c.AbortWithError(500, zincrErr)
        return
    }

    c.Status(http.StatusNoContent)
}

Fetch number of upvotes for a news item

Upvotes for a news item are represented as a score in a Redis SORTED SET which means we can just use the ZSCORE on SORTED SET (news-upvotes) to get the number of upvotes i.e. ZSCORE news-upvotes 42 (where 42 is the news ID)

func GetUpvotesForNewsItemHandler(c *gin.Context) {
    newsID := c.Param("newsid")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    upvotes, zscoreErr := client.ZScore(redisNewsUpvotesSortedSet, newsID).Result()

    if zscoreErr != nil {
        c.AbortWithError(500, zscoreErr)
        return
    }

    c.Writer.WriteString(strconv.Itoa((int(upvotes))))

} 

Fetch a news item by its ID

The REST endpoint for news item details accepts a news ID (as a Path Parameter of a HTTP GET request) and returns information about the news item which includes its URL, title, who it was submitted by along with the number of upvotes and comments

func GetNewsItemByIDHandler(c *gin.Context) {
    newsID := c.Param("newsid")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    newsItem, err := getNewsItemDetails(newsID, client)
    if err != nil {
        c.AbortWithError(500, err)
        return
    }

    if newsItem.NewsID == "" {
        c.Status(http.StatusNoContent)
    } else {
        c.JSON(http.StatusOK, newsItem)
    }

}

The real workhorse is the getNewsItemDetails function. It fetches the details from the the Redis HASH specific to the news item (e.g. news:42 where 42 is the news ID) and creates a model.NewsItem instance which is the representation of the details to be returned

func getNewsItemDetails(newsID string, client *redis.Client) (model.NewsItem, error) {
    newsItemHashKey := redisNewsHashPrefix + ":" + newsID

    var item model.NewsItem
    newsItemDetailMap, hgetallErr := client.HGetAll(newsItemHashKey).Result()

    if hgetallErr != nil {
        return item, hgetallErr
    }

    if len(newsItemDetailMap) == 0 {
            fmt.Println("No news item with ID", newsID)
            return item, nil
        }

    upvotes, zscoreErr := client.ZScore(redisNewsUpvotesSortedSet, newsID).Result()

    if zscoreErr != nil {
        return item, zscoreErr
    }

    commentListName := "news:" + newsID + ":comments"

    comments, llenErr := client.LLen(commentListName).Result()
    if llenErr != nil {
        return item, llenErr
    }

    item = model.NewsItem{newsID, newsItemDetailMap["title"], newsItemDetailMap["submittedBy"], newsItemDetailMap["url"], strconv.Itoa(int(upvotes)), strconv.Itoa(int(comments))}
    return item, nil
}

Fetch all the news items

This is an extension of the above functionality which returns all the news items. Important thing to note is that the returned list of news items are sorted in descending order of the number of upvotes they have received i.e. the news items with the maximum upvotes shows up first

This is where the SORTED SET shines. Since we had stored the upvotes for a news item as a score in the news-upvotes Redis SORTED SET, getting back sorted list of these news items (in descending order) is achieved using the ZREVRANGE command which accepts the start and stop index (in this case it's the length of the SORTED SET which is found using ZCARD)

func GetAllNewsItemsHandler(c *gin.Context) {

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    numItems, zcardErr := client.ZCard(redisNewsUpvotesSortedSet).Result()

    if zcardErr != nil {
        c.AbortWithError(500, zcardErr)
        return
    }

    itemIDs, zrevrangeErr := client.ZRevRange(redisNewsUpvotesSortedSet, 0, numItems-1).Result()

    if zrevrangeErr != nil {
        c.AbortWithError(500, zrevrangeErr)
        return
    }

    var newsItems []model.NewsItem
    for _, newsID := range itemIDs {
        newsItem, err := getNewsItemDetails(newsID, client)
        if err == nil {
            newsItems = append(newsItems, newsItem)
        }
    }

    c.JSON(http.StatusOK, newsItems)
}

As earlier, the details of the each of the news item is extracted from the respective HASH and the entire result is sent back to client as a slice of model.NewsItems

Docker setup

The docker-compose.yml defines the redis and news-sharing-app services

version: '3'
services:
    redis:
        image: redis
        container_name: redis
        ports:
            - '6379:6379'
    news-sharing-app:
        build: .
        environment:
            - REDIS_SERVER=redis:6379
            - PORT=9090
        ports:
            - '8080:9090'
        depends_on:
            - redis

REDIS_SERVER environment variable is used by the application code and PORT variable is used by Gin

The redis service is based on the Redis image from Docker Hub and the news-sharing-app is based on a custom Dockerfile

FROM golang as build-stage
WORKDIR /go/
RUN go get -u github.com/go-redis/redis && go get -u github.com/gin-gonic/gin
COPY src/ /go/src
RUN cd /go/src && CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o news-sharing-app

FROM scratch
COPY --from=build-stage /go/src/news-sharing-app /
CMD ["/news-sharing-app"]

A multi-stage build process is used wherein a different image is used for building our Go app and a different image is used as the base for running it

• golang Dockerhub image is used for the build process which results in a single binary (for linux)
• since we have the binary with all dependencies packed in, all we need is the minimal image for running it and thus we use the lightweight scratch image for this purpose

Test drive

• Install curl, Postman or any other HTTP tool to interact with the REST endpoints of the service
• Get the project - git clone https://github.com/abhirockzz/practical-redis.git

• cd practical-redis/news-sharing-service/

• Invoke the startup script ./run.sh (this in turn invokes docker-compose commands)

• Stop the application by invoking ./stop.sh from another terminal

Replace DOCKER_IP with the IP address of your Docker instance which you can obtain using docker-machine ip. The port (8080 in this case) is the specified in docker-compose.yml

Create a few users

curl -X POST \
  http://DOCKER_IP:8080/users \
  -H 'content-type: text/plain' \
  -d abhirockzz

-d accepts the payload - in this case, the user name

You should see HTTP 204 status as the response

Submit a couple of news items

curl -X POST \
  http://DOCKER_IP:8080/news \
  -H 'content-type: application/json' \
  -H 'user: abhirockzz' \
  -d '{
    "url":"https://simplydistributed.wordpress.com/2018/05/24/redis-geo-lua-example-using-go/",
    "title":"Redis geo.lua example using Go",
    "submittedBy": "abhirockzz"
}'

If successful, the news ID is returned as a payload in the HTTP 200 response

and another one

curl -X POST \
  http://DOCKER_IP:8080/news \
  -H 'content-type: application/json' \
  -H 'user: abhirockzz' \
  -d '{
    "url":"https://simplydistributed.wordpress.com/2018/05/02/nats-on-kubernetes-example/",
    "title":"NATS on Kubernetes example",
    "submittedBy": "abhirockzz"
}'

Get details for a specific news item

• For news item 1 - curl -X GET http://DOCKER_IP:8080/news/1

• For news item 2 - curl -X GET http://DOCKER_IP:8080/news/2

A JSON payload representing the news item details is returned

{
    "newsID": "1",
    "url": "https://simplydistributed.wordpress.com/2018/05/24/redis-geo-lua-example-using-go/",
    "title": "Redis geo.lua example using Go",
    "submittedBy": "abhirockzz",
    "numUpvotes": "0",
    "numComments": "0"
}

Comment on news items

curl -X POST \
  http://DOCKER_IP:8080/news/1/comments \
  -H 'content-type: text/plain' \
  -d 'I din't know this was possible!'

the 1 in the URL http://DOCKER_IP:8080/news/1/comments is the news ID

curl -X POST \
  http://DOCKER_IP:8080/news/1/comments \
  -H 'content-type: text/plain' \
  -d 'Details would have been better'

Let's post a comment for news item 2 as well

curl -X POST \
  http://DOCKER_IP:8080/news/2/comments \
  -H 'content-type: text/plain' \
  -d 'Did not work for me :-('

You should see HTTP 204 status as the response

Get comments for a specific news items

For news item 1

curl -X GET \
  http://DOCKER_IP:8080/news/1/comments \
  -H 'content-type: text/plain'

For news item 2

curl -X GET \
  http://DOCKER_IP:8080/news/2/comments \
  -H 'content-type: text/plain'

This API returns a JSON response similar to below

{
    "newsID": "1",
    "comments": [
        "I din't know this was possible!",
        "Details would have been better"
    ]
}

Upvote news items

Execution of this command is equal to one upvote. So, repeat it for as many upvotes you like and please note that 1 is the news ID

Let's give 3 upvotes to news ID 1 (repeat this thrice)

curl -X POST \
  http://DOCKER_IP:8080/news/1/upvotes \
  -H 'content-type: text/plain'

.. and 2 upvotes for news ID 2 (repeat twice)

curl -X POST \
  http://DOCKER_IP:8080/news/2/upvotes \
  -H 'content-type: text/plain'

You should see HTTP 204 status as the response

Get all news items

curl -X GET http://DOCKER_IP:8080/news

The JSON representation of multiple news items is returned

{
    "newsItems": [
        {
            "newsID": "1",
            "url": "https://simplydistributed.wordpress.com/2018/05/24/redis-geo-lua-example-using-go/",
            "title": "Redis geo.lua example using Go",
            "submittedBy": "abhirockzz",
            "numUpvotes": "3",
            "numComments": "2"
        },
        {
            "newsID": "2",
            "url": "https://simplydistributed.wordpress.com/2018/05/02/nats-on-kubernetes-example/",
            "title": "NATS on Kubernetes example",
            "submittedBy": "abhirockzz",
            "numUpvotes": "2",
            "numComments": "1"
        }
    ]
}