Document Store Using Redis

Miguel Sy

January 14, 2021

What is a Document Store?
  • Type of nonrelational database
  • Designed to store and query data as JSON-like documents
  • Makes it easy for developers to store and query data in a database using the same document-model format used in app

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image14_0df3e0eeb8.png"> </div>

<br/>

Difference of Document Store from Relational Database
  • Doesn’t require a schema before inserting data
  • Stores data for a single entity in a single document rather than multiple tables
  • More flexible in managing application changes over time
  • Relatively simpler

<br/>

What is Redis?
  • High-performance in-memory data structure store
  • Advanced key-value model that supports different data structures
  • Can be used as a database, cache, and message broker
  • Open source
  • Developed by Salvatore Sanfilippo and sponsored by Redis Labs

<br/>

Benefits of Using Redis as a Document Store
  • High-Performance
  • Natively written in C
  • Extremely efficient machine code and requires little overhead
  • Support for Arbitrary Data
  • Binary Safe - Can store any data from human readable text to encoded binaries
  • Supports a maximum string length of 512MB
  • In-memory Store
  • Allows the delivery of the fastest possible access times to the data for both read and write
  • Key-Based Access
  • Extremely optimized and efficient access times
  • Atomicity of Operations and Transactions
  • Has clients in all the popular programming languages
  • Redis Benchmark Example:

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image44_f9625addb2.png"> </div>

<br/>

Trade-offs of Using Redis
  • In-memory Store (Dataset always resides in RAM)
  • Expensive
  • Redis uses periodic memory dump to create a persistency snapshot on disk
  • May need manual configuration and triggering
  • May affect performance of live running Redis server
  • Non-Relational Database
  • Does not support structured query language (SQL) and useful complex relational queries

<br/>

Redis Commands and Data Structures Hands-on Exercise
  • Prerequisite
    • Install redis-cli via Homebrew
    • Start Redis Server

<div> <img class="img-fluid img-width-50 mb-2" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image8_5a6a4ee7fc.png"> </div>

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image51_88b36cb462.png"> </div>

  • Some Useful General Redis Client Commands
    • PING - Checks if connection to server is alive (redis-cli defaults connection to server 127.0.0.1 port 6379)
    • Connect to remote server via redis-cli: 'redis-cli -h &lt;host> -p &lt;port> -a &lt;password>'
    • FLUSHALL - Delete all keys from database

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image24_defc231df1.png"> </div>

  • Keys and Commands
    • Redis keys are generally in the form &lt;string>[:&lt;string>]...
    • Useful commands
      • KEYS &lt;pattern> - Finds all keys matching the specified pattern
      • DEL &lt;key> - Deletes the key if it exists
      • EXISTS &lt;key> - Checks if keys exists
      • RENAME &lt;key> - Changes the key name
      • EXPIRE &lt;key> &lt;seconds> - Sets expiry of key after specified time
      • TYPE &lt;key> - Returns the data type of the value stored in the key

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image7_17d51e6775.png"> </div>

  • Strings
    • Binary-safe data structure
    • Can store any kind of data (string, integer, float, JPEG image, etc.)
    • Example Commands
      • SET &lt;key> &lt;value> - Sets value at the specified key
      • GET &lt;key> - Gets the value of a key
      • MSET &lt;key1> &lt;value1> &lt;key2> &lt;value2> - Sets multiple values to multiple keys
      • MGET &lt;key1> &lt;key2> - Gets the value of given keys
      • GETSET &lt;key> &lt;value> - Sets the string value of a key and returns old value
      • STRLEN &lt;key> - Gets the length of the value stored in key
      • INCR &lt;key> - Increments the integer value of a key
      • DECR &lt;key> - Decrements the integer value of a key

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image45_06081d9e68.png"> </div>

  • Lists
    • Collection of string elements sorted according to order of insertion
    • Duplicates are allowed
    • Example Commands
    • LPUSH &lt;key> &lt;value1> &lt;value2> - Prepends one or more multiple values to a list
    • RPUSH &lt;key> &lt;value1> &lt;value2> - Appends one or more multiple values to a list
    • LRANGE &lt;key> &lt;start> &lt;stop> - Gets a range of elements from a list
    • LPOP/RPOP &lt;key> - Gets element from the list

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image1_c08aeb050b.png"> </div>

  • Sets
    • Unordered collection of unique string elements
    • Example Commands
      • SADD &lt;key> &lt;value1> &lt;value2> - Adds one or more members to a set
      • SMEMBERS &lt;key> - Gets all the members in a set
      • SPOP &lt;key> - Removes and returns a random member from a set
      • SISMEMBER &lt;key> &lt;value> - Determines if values is member of the set
      • SDIFFSTORE &lt;destination> &lt;key1> &lt;key2> - Subtracts multiple sets and stores to destination key

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image17_2d746c9138.png"> </div>

  • Sorted Sets
    • Collection of unique string elements sorted according to order of floating-point score
    • Example Commands
      • ZADD &lt;key> &lt;score1> &lt;value1> &lt;score2> &lt;value2> - Adds one or more members to a sorted set, or updates its score if already exists
      • ZRANGE &lt;key> &lt;start> &lt;stop> [WITHSCORES] - Returns a range of members in a sorted set, by index
      • ZRANGEBYSCORE &lt;key> &lt;min> &lt;max> [WITHSCORES] [LIMIT] - Returns a range of members in a sorted set, by score

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image23_46209c9554.png"> </div>

  • Hashes
    • Stores a set of field-value pairs
    • Best way to store objects with key &lt;object-type>:&lt;id>
    • Example Commands
      • HSET &lt;key> &lt;field> &lt;value> - Sets the value of a hash field
      • HGET &lt;key> &lt;field> - Gets the value of a hash field
      • HGETALL &lt;key> - Gets all the fields and values in a hash at the specified key
      • HMSET &lt;key> &lt;field1> &lt;value1> &lt;field2> &lt;value2> - Sets multiple hash fields to multiple values
      • HMGET &lt;key> &lt;field1> &lt;field2> - Gets values of multiple hash fields

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image11_3d494e46d4.png"> </div>

<br/>

Redis Node.JS Application Use Case: Key-Value Store
  • One effective way to use Redis is to use it as a database
    • Can be used alongside a relational database (e.g. MySQL)
    • Separation of Tasks
      • Relational Database: Data for Listing Page (Flat Table Implementation)
      • Redis Key Value Store: Data for Detail Page
    • App Example

<div> <img class="img-fluid img-width-50 mb-2" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image52_07ae73955e.png"> </div>

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image53_e4dad49634.png"> </div>

  • Performance Benefits
    • Relational database schema will be compact
    • Data needed for detail page is quickly accessible because of Redis’ key-value approach and high performance

<br/>

Node.JS Redis Client: ioredis
  • One of the most popular Redis clients for Node.js
  • Initialization and Connection to Redis Server

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image25_420fd70edc.png"> </div>

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image50_1d65cbf408.png"> </div>

  • Redis Commands
    • Returns a promise
    • General Form: redis.&lt;command>(&lt;arguments>)
    • Supports callbacks
    • Has Built-in Arguments and Replies Transformers

<div> <img class="img-fluid mb-2 img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image47_c511539f54.png"> </div>

<div> <img class="img-fluid mb-2 img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image18_13b9ade0e0.png"> </div>

<div> <img class="img-fluid mb-2 img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image34_cc111739df.png"> </div>

<div> <img class="img-fluid mb-2 img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image35_490432fe15.png"> </div>

  • Redis Pipelining
    • optimization approach in Redis to reduce network latency and improve round trip time between Redis client and server communication
    • main idea is to send multiple commands at once
    • ioredis pipeline exec returns a Promise

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image29_3feefb339a.png"> </div>

<br/>

Redis as Node.JS Key Value Store Hands-on Exercise
  • Goal: Add Redis Key Value Store on sample Node.JS Application
    • Post User
    • Get User
Exercise Steps

1. Open provided Express app and explore

2. Go to express folder and run npm install

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image33_1c33a4ba68.png"> </div>

3. Install ioredis

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image32_c77b5354e2.png"> </div>

4. Go to parent directory and run docker-compose-up

5. Initialize and connect ioredis to redis docker server in repository

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image42_c866d9845a.png"> </div>

6. Populate User Redis Key Value Store on Post API Call

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image10_e4feeeccbb.png"> </div>

7. Test Post User Using Postman and redis-cli

<div> <img class="img-fluid mb-2 img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image28_9b1f5a8570.png"> </div>

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image39_9b78029566.png"> </div>

8. Get User Data from Redis on Get User API Call

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image19_27987df49e.png"> </div>

9.Test Get User via Postman

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image16_d9d97dfee4.png"> </div>

10. Optimize Redis Key Value Store by Implementing Pipelining

a. Optimize Post User

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image15_4baf26c85a.png"> </div>

b. Test Optimized Post User Using Postman and redis-cli

<div> <img class="img-fluid mb-2 img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image5_62510ccb11.png"> </div>

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image55_f5db535fbf.png"> </div>

c. Optimize Get User

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image30_7b8f0c0cad.png"> </div>

d. Test Optimized Get User via Postman

<div> <img class="img-fluid img-width-50" src="https://strapi-saperium.s3.ap-southeast-1.amazonaws.com/redis_image31_a8aca6d219.png"> </div>