How Redis Works End to End

1. You start a Redis server - it runs as a single process on a machine.

2. Redis stores everything in-memory, so it’s blazing fast.

3. You connect to Redis using a client - could be from Java, Python, Node.js, etc.

4. You send commands like SET, GET, INCR, DEL - it responds instantly.

5. Redis stores data as key-value pairs - the value can be strings, hashes, lists, sets, or sorted sets.

6. It supports atomic operations - e.g., increment a counter with INCR and it’s safe from race conditions.

7. You can set expiry on keys - using EXPIRE or SET key value EX 60 to auto-delete after 60 seconds.

8. It’s often used as a cache - store data that’s expensive to compute or slow to fetch from DB.

9. Redis supports pub/sub - you can publish messages to a channel and other clients can subscribe and react in real time.

10. You can use Lists (like a queue), Sets (for unique items), and Sorted Sets (for leaderboards, ranked data).

11. Redis supports Lua scripting - you can run small programs on the server to perform complex logic atomically.

12. You can use Streams in Redis - great for real-time event data like logs or messaging.

13. Redis supports bitmaps, hyperloglogs, and geospatial indexes - niche but powerful tools for specific use cases.

14. Redis is single-threaded for commands but super fast due to memory and tight architecture.

15. It can persist data using RDB snapshots (periodic dumps) or AOF logs (write every operation).

16. You can choose to run it purely in-memory (ephemeral) or with persistence (survives reboots).

17. Redis supports replication - a master can have one or more read-only replicas.

18. If master goes down, you can use Redis Sentinel to handle automatic failover and monitoring.

19. For large-scale setups, you can use Redis Cluster - data is sharded across multiple nodes.

20. Redis also supports transactions using MULTI / EXEC - run multiple commands as one unit.

21. You can use it for rate limiting, session storage, real-time analytics, job queues, and leaderboards.

22. All of this with millisecond latency - that’s why Redis is a go-to for performance-critical systems.

Redis Technical Deep Dive

Core Architecture

Redis employs a simple yet powerful architecture:

1. Event Loop Model:

   • Uses an event-driven, non-blocking I/O model
   • Built around ae (Advanced Event) library
   • Similar to Node.js event loop but single-threaded
   • Processes commands sequentially (no context switching overhead)

2. Memory Management:

   • Custom memory allocator (jemalloc/tcmalloc)
   • Minimizes fragmentation
   • Memory is pre-allocated in chunks
   • No garbage collection pauses

3. Communication Protocol (RESP):

   • Redis Serialization Protocol
   • Simple text-based protocol
   • Binary-safe
   • Low overhead for parsing

4. Main Components:

   • Server core (event loop + command processing)
   • Data structures implementations
   • Persistence mechanisms
   • Replication system
   • Cluster management

Data Structures Implementation

Redis data types are implemented with sophisticated internal structures:

1. Strings:

   • Simple Dynamic Strings (SDS) implementation
   • Binary-safe (can contain any data)
   • Optimized small string encoding (for values < 44 bytes)
   • Int optimization (stores numbers as actual integers)

2. Lists:

   • Implemented as linked lists or ziplist (for small lists)
   • Optimized for fast push/pop at both ends (O(1))
   • Memory-efficient encodings for small lists

3. Hashes:

   • Small hashes use ziplist encoding
   • Larger hashes use hash tables (dictionaries)
   • Auto-conversion between encodings based on size

4. Sets:

   • Either hash tables (for regular sets) or intsets (for integer-only sets)
   • O(1) member lookups
   • Memory optimization for integer-only sets

5. Sorted Sets:

   • Combination of a hash table and a skip list
   • O(log N) insertions and score-based retrievals
   • Used for leaderboards, priority queues, range queries

6. Bitmaps:

   • Not a separate type (implemented on strings)
   • Bit-level operations on string values
   • Memory-efficient for counting unique items (1 bit per item)

7. HyperLogLog:

   • Probabilistic data structure for cardinality estimation
   • Extremely memory-efficient (uses ~12KB for counting millions of items)
   • 0.81% standard error

Command Processing Flow

How Redis handles a command:

1. Command Reception:

   • Client sends RESP-formatted command
   • Server reads input buffer during event loop
   • Parses command and arguments

2. Command Lookup:

   • Looks up command in command table (O(1) hash table)
   • Verifies argument count and access permissions

3. Command Execution:

   • Calls command implementation function
   • Manipulates data structures in memory
   • Prepares response

4. Response Generation:

   • Formats response in RESP protocol
   • Writes to client output buffer
   • Sends when socket is writable (next event loop iteration)

5. Command Propagation (if applicable):

   • Appends to AOF if enabled
   • Queues for replication to replicas
   • Logs for monitoring if enabled

Persistence Mechanisms

Redis offers two main persistence options:

1. RDB (Redis Database):

   • Point-in-time snapshots
   • Binary compact format
   • Created via:
     • Background process (BGSAVE)
     • Foreground save (SAVE)
     • Automatic based on configuration
   • Pros: Compact, faster restarts
   • Cons: Potential data loss between snapshots

2. AOF (Append-Only File):

   • Log of all write operations
   • Human-readable format (Redis commands)
   • Configurable fsync policies:
     • always: Safest, slowest
     • everysec: Good compromise
     • no: Fastest, OS controlled
   • Background rewriting to avoid file growth
   • Pros: Durability, easier corruption recovery
   • Cons: Larger files, slower restarts

3. Hybrid Persistence (RDB+AOF):

   • AOF can include RDB preamble
   • Combines restart speed with better durability

Memory Optimization Techniques

Redis employs several techniques to minimize memory usage:

1. Special Encodings:

   • Small lists, sets and hashes use compact encodings
   • Integer-only data uses special representations
   • Shared objects for common values

2. Memory Policies:

   • LRU (Least Recently Used) eviction
   • LFU (Least Frequently Used) eviction
   • Random eviction
   • TTL-based eviction
   • No-eviction (return errors when memory limit reached)

3. maxmemory Configuration:

   • Sets memory usage limit
   • Triggers eviction when limit is reached
   • Can be dynamically adjusted

4. Key Expiration:

   • Passive expiry (lazy deletion when keys are accessed)
   • Active expiry (periodic sampling of expired keys)
   • Avoids CPU spikes from many simultaneous expirations

Replication Architecture

Redis uses a master-replica model:

1. Replication Process:

   • Initial full sync (RDB transfer)
   • Followed by command streaming
   • Partial resync after disconnections (using replication buffer)

2. Replication Modes:

   • Disk-based: Master creates RDB on disk, then sends
   • Diskless: Master sends RDB directly to replicas

3. Consistency Model:

   • Asynchronous replication (by default)
   • Semi-synchronous with WAIT command
   • Eventual consistency guarantee

4. Replica Features:

   • Read-only by default (can be configured as writable)
   • Can have their own replicas (cascading replication)
   • Can persist data independently of master

Sentinel System

Redis Sentinel provides high availability:

1. Core Functions:

   • Monitoring: Checks if masters and replicas are working
   • Notification: Alerts administrators or other programs
   • Automatic failover: Promotes replica when master fails
   • Configuration provider: Clients connect to Sentinel to find current master

2. Quorum-based Decisions:

   • Multiple Sentinels must agree on failure
   • Prevents split-brain scenarios
   • Configurable quorum size

3. Failover Process:

   • Majority of Sentinels detect master failure
   • Leader election among Sentinels
   • Best replica selection
   • Reconfiguration of other replicas

Clustering Architecture

Redis Cluster for horizontal scaling:

1. Data Sharding:

   • Uses hash slot approach (16384 slots)
   • CRC16(key) mod 16384 determines slot
   • Each node handles a subset of slots
   • Hash tags allow multi-key operations ([key-name]) within same slot

2. Node Communication:

   • Gossip protocol for node discovery and heartbeats
   • Binary protocol for data exchange
   • Separate bus port for node-to-node communication

3. Fault Tolerance:

   • Master-replica pairs for each shard
   • Automatic failover when master is down
   • Automatic replica migration for better distribution

4. Consistency Guarantees:

   • No strong consistency guarantee
   • Asynchronous replication between master and replicas
   • Potential data loss during failovers
   • Redis Cluster prefers availability over consistency (AP in CAP theorem)

Performance Considerations

Key factors affecting Redis performance:

1. CPU Impact:

   • Single-threaded main process (commands execute sequentially)
   • IO threading since Redis 6 (for network handling)
   • CPU frequency more important than core count
   • Background tasks (e.g., persistence) affect performance

2. Memory Impact:

   • Data should fit in RAM for optimal performance
   • Swapping to disk severely impacts performance
   • Virtual memory fragmentation affects real memory usage

3. Network Factors:

   • Latency between client and server
   • Network bandwidth for large values
   • Connection pooling importance
   • Pipelining to reduce round-trips

4. Persistence Overhead:

   • RDB creation (fork and copy-on-write)
   • AOF rewriting (temporary increased memory usage)
   • fsync frequency affecting write latency