# Redis: the basic data structures

The first chapter is for you get an overview of the core Redis data structures - `STRING`, `LIST`, `SET`, `HASH` and `SORTED SET`. A practical application is used to explain the concepts

> This is by no means a complete list of all the Redis data structures

## News sharing app

This application is loosely inspired by [Hacker News](https://news.ycombinator.com/). It's features are as follows

* **Basic**
  * User registration (using a unique name)
  * Submit a news item
  * Get a news item
  * View all news items sorted in descending order by upvotes
* **Comments**
  * Post a comment on a news item
  * View all comments for a news item
* **Upvote**
  * Upvote a news item
  * Get number of upvotes for a news item

> All the functionality (registration, news items submission etc.) is exposed via a REST API

### Technical stack

The following components have been used

* [Gin](https://github.com/gin-gonic/gin) is the HTTP web framework for REST API
* [go-redis](https://github.com/go-redis/redis) as the Redis client 
* [Docker](https://www.docker.com/)
  * [pre-built Docker image](https://hub.docker.com/_/redis/) for Redis
  * [Docker Compose](https://docs.docker.com/compose/) to orchestrate all the application components

### Schema

Here is a quick overview of the application specific entities and the Redis data structures they map to

* `users` - a `SET` which stores all users (just the user names to keep it simple)
* `news:[id]` - news details are stored in a `HASH` (`id` is the news ID)
* `news-id-counter` - a `STRING` data type
* `news:[id]:comments` - a `LIST` to store comments associated with a news item (`id` is the news ID) 
* `comments-id-counter` - `STRING` key which acts as a counter for generating comment ID
* `news-upvotes` - `SORTED SET` with news ID as key and number of upvotes as value

## Implementation details

Its time to dive into the nitty gritty. This section focuses on the code and the details of how to test the application is covered in the next one

> Source code is available on [Github](https://github.com/abhirockzz/practical-redis/tree/master/news-sharing-service/go)

The code package structure is as follows

```
│   main.go
│
├───api
│       comments-api.go
│       middleware.go
│       newsitem-api.go
│       upvotes-api.go
│       user-api.go
│
└───model
        models.go
```

The `api` directory contains logic for the REST API handlers for news items, comments etc. `models` contains a single file `models.go` which has the Go `struct`s representations for the JSON data which will be exchanged between the client and the API. Finally, `main.go` serves as the entry point

Let's move on to the details

### New user registration

The new user registration process is implemented using a REST endpoint which accepts the user name as an input (via HTTP `POST`). It adds to the `users` set using `SADD` (e.g. `SADD users abhirockzz`)

```
func UserRegistrationHandler(c *gin.Context) {

    userByte, _ := ioutil.ReadAll(c.Request.Body)
    ...

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    num, err := client.SAdd(usersSet, string(userByte)).Result()
    ...

    if num == 0 {
        fmt.Println("user already exists")
        c.Status(http.StatusConflict)
        return
    }

    ...
    c.Status(http.StatusNoContent)
}
```

* A Set stores unique elements only - same applies to `SET`s in Redis
* `SADD` returns 0 in case the element (the user name in this case) is not added to the set - this allows registration request for a duplicate user to be rejected

### Redis connection management

In the above code snippet, notice the following lines

```
clientCtxObj, _ := c.Get("redis")
client := clientCtxObj.(*redis.Client)
```

`client` is nothing but a handle to the `redis.Client` object - but was the connection established in the first place ? A [Gin middleware](https://github.com/gin-gonic/gin#using-middleware) was used to achieve this. As a result

* connection to Redis is established before invoking any of the HTTP handler (in response to a API request)

  > the request is aborted in case the connection is not successful (there is no point proceeding)
* the Redis connection is closed after the request

  ```
    func RedisConnHandler(c *gin.Context) {

        connected := true

        redisServer := os.Getenv("REDIS_SERVER")
        if redisServer == "" {
            redisServer = "192.168.99.100:6379"
        }

        client := redis.NewClient(&redis.Options{Addr: redisServer})
        err := client.Ping().Err()
        if err != nil {
            c.AbortWithError(http.StatusInternalServerError, err)
            connected = false
        }

        c.Set("redis", client)

        c.Next()

        if client != nil && connected {
            client.Close()
        }
    }
  ```

Redis server endpoint is extracted from the environment variable `REDIS_SERVER`. On successful connection, the object is set in a (Gin) context specific variable called `redis` - the HTTP request handler(s) (as seen above), `Get`s the client object from the context variable

### Submit a news item

The REST API for news item submission accepts the news item details via HTTP `POST` and it makes of the `HASH`, `SORTED SET` and `STRING` data structures

> the API simply returns the (unique) ID for the submitted news item (as the response)

```
func PostNewsItemHandler(c *gin.Context) {

    var news model.NewItemSubmission
    c.ShouldBindJSON(&news)
    submittedBy := c.GetHeader("user")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    exists, sisMemberErr := client.SIsMember(usersSet, submittedBy).Result()

    if sisMemberErr != nil {
        c.AbortWithError(500, sisMemberErr)
        return
    }

    if !exists {
        c.AbortWithError(500, errors.New("Invalid user "+submittedBy))
        return
    }

    newsID, err := client.Incr(redisNewsIdCounter).Result()
    if err != nil {
        c.AbortWithError(500, err)
        return
    }

    newsHashKey := redisNewsHashPrefix + ":" + strconv.Itoa(int(newsID))

    newsDetails := map[string]interface{}{"title": news.Title, "url": news.Url, "submittedBy": submittedBy}
    _, hmsetErr := client.HMSet(newsHashKey, newsDetails).Result()

    if hmsetErr != nil {
        c.AbortWithError(500, hmsetErr)
        return
    }

    _, zaddErr := client.ZAdd(redisNewsUpvotesSortedSet, redis.Z{Member: newsID, Score: 0}).Result()

    if zaddErr != nil {
        c.AbortWithError(500, zaddErr)
        return
    }

    c.JSON(http.StatusCreated, newsID)
}
```

* the `users` SET is first checked to see if a registered user has submitted the request
* The `STRING` data type (`news-id-counter`) is used as a unique (news) ID generator. The interesting part is that it is *integer aware* and can be **incremented in an atomic manner** using `INCR` command (e.g. `INCR news-id-counter`) which makes it ideal for implementing counters
* Once the unique news ID is generated, the news item details are stored in a `HASH` (using `HMSET` command). Think of `HASH` as a `Map`-like data structure which is a great fit for storing objects. In this case, we're storing the `url`, `title` and `submittedBy` attributes for a news item in `HASH` whose naming convention is `news:[id]` e.g. `news:42`. Here is a what the an example command look like - `HMSET news:42 title "Why did I spend 1.5 months creating a Gameboy emulator?" url "http://blog.rekawek.eu/2017/02/09/coffee-gb/" submittedBy "tosh"`
* Finally, we add the news item ID and the number of upvotes (zero to begin with) to a `SORTED SET` (called `news-upvotes`). This data structure is like a `SET` (unique elements only) but with an additional property that the **elements/members can be associated with a score** which opens up a bunch of interesting options (discussed in upcoming section)

### Comment on a news item

Implementing a REST API for comments on a post is as as simple as using the `LPUSH` command to add it to a Redis `LIST` whose naming pattern is `news:[id]:comments` (e.g. `news:42:comments`). `LPUSH` inserts the comment to the beginning of the list such that the older comments are pushed further in. This way, when fetched (details below), the latest (newer) comments will show up first

```
func PostCommentsForNewsItemHandler(c *gin.Context) {
    commentsBytes, _ := ioutil.ReadAll(c.Request.Body)
    newsID := c.Param("newsid")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    commentsListName := "news:" + newsID + ":comments"
    _, lpushErr := client.LPush(commentsListName, string(commentsBytes)).Result()

    if lpushErr != nil {
        c.AbortWithError(500, lpushErr)
        return
    }

    c.Status(http.StatusNoContent)
}
```

### Fetch comments for a news item

The REST API to get all the comments associated with a specific news item makes use of the `LRANGE` command which returns a subset of the list (given `start` and `stop` index). In this case we want all the contents, hence our `start` index is 0 and the `stop` index is the length of the list itself e.g. `LRANGE news:42:comments 0 10`. We find out the length of the list using `LLEN` command

```
func GetCommentsForNewsID(c *gin.Context) {
    newsID := c.Param("newsid")
    commentListName := "news:" + newsID + ":comments"

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    numOfComments, llenErr := client.LLen(commentListName).Result()
    if llenErr != nil {
        return
    }

    if numOfComments == 0 {
        c.Data(200, "text/plain", []byte("no comments found"))
        return
    }

    comments, lrangeErr := client.LRange(commentListName, 0, numOfComments).Result()
    if lrangeErr != nil {
        c.AbortWithError(500, lrangeErr)
        return
    }

    c.JSON(200, model.NewsItemComments{newsID, comments})
}
```

### Upvote a news item

Upvoting a news item is achieved by incrementing its score in the `SORTED SET` (`news-upvotes`) using the `ZINCRBY` command

```
func UpvoteNewsItemHandler(c *gin.Context) {
    newsID := c.Param("newsid")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    _, zincrErr := client.ZIncr(redisNewsUpvotesSortedSet, redis.Z{Member: newsID, Score: 1}).Result()

    if zincrErr != nil {
        c.AbortWithError(500, zincrErr)
        return
    }

    c.Status(http.StatusNoContent)
}
```

### Fetch number of upvotes for a news item

Upvotes for a news item are represented as a *score* in a Redis `SORTED SET` which means we can just use the `ZSCORE` on `SORTED SET` (`news-upvotes`) to get the number of upvotes i.e. `ZSCORE news-upvotes 42` (where `42` is the news ID)

```
func GetUpvotesForNewsItemHandler(c *gin.Context) {
    newsID := c.Param("newsid")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    upvotes, zscoreErr := client.ZScore(redisNewsUpvotesSortedSet, newsID).Result()

    if zscoreErr != nil {
        c.AbortWithError(500, zscoreErr)
        return
    }

    c.Writer.WriteString(strconv.Itoa((int(upvotes))))

} 
```

### Fetch a news item by its ID

The REST endpoint for news item details accepts a news ID (as a `Path Parameter` of a `HTTP GET` request) and returns information about the news item which includes its URL, title, who it was submitted by along with the number of upvotes and comments

```
func GetNewsItemByIDHandler(c *gin.Context) {
    newsID := c.Param("newsid")

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    newsItem, err := getNewsItemDetails(newsID, client)
    if err != nil {
        c.AbortWithError(500, err)
        return
    }

    if newsItem.NewsID == "" {
        c.Status(http.StatusNoContent)
    } else {
        c.JSON(http.StatusOK, newsItem)
    }

}
```

The real workhorse is the `getNewsItemDetails` function. It fetches the details from the the Redis `HASH` specific to the news item (e.g. `news:42` where `42` is the news ID) and creates a `model.NewsItem` instance which is the representation of the details to be returned

```
func getNewsItemDetails(newsID string, client *redis.Client) (model.NewsItem, error) {
    newsItemHashKey := redisNewsHashPrefix + ":" + newsID

    var item model.NewsItem
    newsItemDetailMap, hgetallErr := client.HGetAll(newsItemHashKey).Result()

    if hgetallErr != nil {
        return item, hgetallErr
    }

    if len(newsItemDetailMap) == 0 {
            fmt.Println("No news item with ID", newsID)
            return item, nil
        }

    upvotes, zscoreErr := client.ZScore(redisNewsUpvotesSortedSet, newsID).Result()

    if zscoreErr != nil {
        return item, zscoreErr
    }

    commentListName := "news:" + newsID + ":comments"

    comments, llenErr := client.LLen(commentListName).Result()
    if llenErr != nil {
        return item, llenErr
    }

    item = model.NewsItem{newsID, newsItemDetailMap["title"], newsItemDetailMap["submittedBy"], newsItemDetailMap["url"], strconv.Itoa(int(upvotes)), strconv.Itoa(int(comments))}
    return item, nil
}
```

### Fetch all the news items

This is an extension of the above functionality which returns *all* the news items. Important thing to note is that the returned list of news items are sorted in descending order of the number of upvotes they have received i.e. the news items with the maximum upvotes shows up first

This is where the `SORTED SET` shines. Since we had stored the upvotes for a news item as a *score* in the `news-upvotes` Redis `SORTED SET`, getting back sorted list of these news items (in descending order) is achieved using the `ZREVRANGE` command which accepts the `start` and `stop` index (in this case it's the length of the `SORTED SET` which is found using `ZCARD`)

```
func GetAllNewsItemsHandler(c *gin.Context) {

    clientCtxObj, _ := c.Get("redis")
    client := clientCtxObj.(*redis.Client)

    numItems, zcardErr := client.ZCard(redisNewsUpvotesSortedSet).Result()

    if zcardErr != nil {
        c.AbortWithError(500, zcardErr)
        return
    }

    itemIDs, zrevrangeErr := client.ZRevRange(redisNewsUpvotesSortedSet, 0, numItems-1).Result()

    if zrevrangeErr != nil {
        c.AbortWithError(500, zrevrangeErr)
        return
    }

    var newsItems []model.NewsItem
    for _, newsID := range itemIDs {
        newsItem, err := getNewsItemDetails(newsID, client)
        if err == nil {
            newsItems = append(newsItems, newsItem)
        }
    }

    c.JSON(http.StatusOK, newsItems)
}
```

As earlier, the details of the each of the news item is extracted from the respective `HASH` and the entire result is sent back to client as a slice of `model.NewsItem`s

## Docker setup

The `docker-compose.yml` defines the `redis` and `news-sharing-app` services

```
version: '3'
services:
    redis:
        image: redis
        container_name: redis
        ports:
            - '6379:6379'
    news-sharing-app:
        build: .
        environment:
            - REDIS_SERVER=redis:6379
            - PORT=9090
        ports:
            - '8080:9090'
        depends_on:
            - redis
```

> `REDIS_SERVER` environment variable is used by the application code and `PORT` variable is used by Gin

The `redis` service is based on the [Redis image from Docker Hub](https://hub.docker.com/_/redis/) and the `news-sharing-app` is based on a custom Dockerfile

```
FROM golang as build-stage
WORKDIR /go/
RUN go get -u github.com/go-redis/redis && go get -u github.com/gin-gonic/gin
COPY src/ /go/src
RUN cd /go/src && CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o news-sharing-app

FROM scratch
COPY --from=build-stage /go/src/news-sharing-app /
CMD ["/news-sharing-app"]
```

A [multi-stage build process](https://docs.docker.com/develop/develop-images/multistage-build/) is used wherein a different image is used for building our Go app and a different image is used as the base for running it

* `golang` [Dockerhub image](https://hub.docker.com/_/golang/) is used for the build process which results in a single binary (for linux)
* since we have the binary with all dependencies packed in, all we need is the minimal image for running it and thus we use the lightweight `scratch` [image](https://hub.docker.com/_/scratch/) for this purpose

## Test drive

* Install [curl](https://curl.haxx.se/), [Postman](https://www.getpostman.com/) or any other HTTP tool to interact with the REST endpoints of the service
* Get the project - `git clone https://github.com/abhirockzz/practical-redis.git`
* `cd practical-redis/news-sharing-service/`
* Invoke the startup script `./run.sh` (this in turn invokes `docker-compose` commands)
* Stop the application by invoking `./stop.sh` from another terminal

> Replace `DOCKER_IP` with the IP address of your Docker instance which you can obtain using `docker-machine ip`. The port (`8080` in this case) is the specified in `docker-compose.yml`

**Create a few users**

```
curl -X POST \
  http://DOCKER_IP:8080/users \
  -H 'content-type: text/plain' \
  -d abhirockzz
```

> `-d` accepts the payload - in this case, the user name

You should see HTTP `204` status as the response

**Submit a couple of news items**

```
curl -X POST \
  http://DOCKER_IP:8080/news \
  -H 'content-type: application/json' \
  -H 'user: abhirockzz' \
  -d '{
    "url":"https://simplydistributed.wordpress.com/2018/05/24/redis-geo-lua-example-using-go/",
    "title":"Redis geo.lua example using Go",
    "submittedBy": "abhirockzz"
}'
```

> If successful, the news ID is returned as a payload in the HTTP `200` response

and another one

```
curl -X POST \
  http://DOCKER_IP:8080/news \
  -H 'content-type: application/json' \
  -H 'user: abhirockzz' \
  -d '{
    "url":"https://simplydistributed.wordpress.com/2018/05/02/nats-on-kubernetes-example/",
    "title":"NATS on Kubernetes example",
    "submittedBy": "abhirockzz"
}'
```

**Get details for a specific news item**

* For news item `1` - `curl -X GET http://DOCKER_IP:8080/news/1`
* For news item `2` - `curl -X GET http://DOCKER_IP:8080/news/2`

A JSON payload representing the news item details is returned

```
{
    "newsID": "1",
    "url": "https://simplydistributed.wordpress.com/2018/05/24/redis-geo-lua-example-using-go/",
    "title": "Redis geo.lua example using Go",
    "submittedBy": "abhirockzz",
    "numUpvotes": "0",
    "numComments": "0"
}
```

**Comment on news items**

```
curl -X POST \
  http://DOCKER_IP:8080/news/1/comments \
  -H 'content-type: text/plain' \
  -d 'I din't know this was possible!'
```

> the `1` in the URL `http://DOCKER_IP:8080/news/1/comments` is the news ID

```
curl -X POST \
  http://DOCKER_IP:8080/news/1/comments \
  -H 'content-type: text/plain' \
  -d 'Details would have been better'
```

Let's post a comment for news item `2` as well

```
curl -X POST \
  http://DOCKER_IP:8080/news/2/comments \
  -H 'content-type: text/plain' \
  -d 'Did not work for me :-('
```

You should see HTTP `204` status as the response

**Get comments for a specific news items**

For news item `1`

```
curl -X GET \
  http://DOCKER_IP:8080/news/1/comments \
  -H 'content-type: text/plain'
```

For news item `2`

```
curl -X GET \
  http://DOCKER_IP:8080/news/2/comments \
  -H 'content-type: text/plain'
```

This API returns a JSON response similar to below

```
{
    "newsID": "1",
    "comments": [
        "I din't know this was possible!",
        "Details would have been better"
    ]
}
```

**Upvote news items**

> Execution of this command is equal to one upvote. So, repeat it for as many upvotes you like and please note that `1` is the news ID

Let's give 3 upvotes to news ID `1` (repeat this thrice)

```
curl -X POST \
  http://DOCKER_IP:8080/news/1/upvotes \
  -H 'content-type: text/plain'
```

.. and 2 upvotes for news ID `2` (repeat twice)

```
curl -X POST \
  http://DOCKER_IP:8080/news/2/upvotes \
  -H 'content-type: text/plain'
```

You should see HTTP `204` status as the response

**Get all news items**

```
curl -X GET http://DOCKER_IP:8080/news
```

The JSON representation of multiple news items is returned

```
{
    "newsItems": [
        {
            "newsID": "1",
            "url": "https://simplydistributed.wordpress.com/2018/05/24/redis-geo-lua-example-using-go/",
            "title": "Redis geo.lua example using Go",
            "submittedBy": "abhirockzz",
            "numUpvotes": "3",
            "numComments": "2"
        },
        {
            "newsID": "2",
            "url": "https://simplydistributed.wordpress.com/2018/05/02/nats-on-kubernetes-example/",
            "title": "NATS on Kubernetes example",
            "submittedBy": "abhirockzz",
            "numUpvotes": "2",
            "numComments": "1"
        }
    ]
}
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://abhishek-gupta.gitbook.io/practical-redis/news-sharing-app.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.