Pastebin System Design

Mental Model

A pastebin service allows users to store plain text (typically code snippets) and share them via a short URL. Designing a scale-out pastebin requires segregating raw document storage from search metadata, pre-allocating short-URL keys to avoid write-time locks, and accelerating read paths using CDN caches.

For interactive exploration of these behaviors, you can run the live visualization: Pastebin System Design Walkthrough.

Components and Why They Exist

Client Browser

Represents end-users who either compose new pastes (writes) or retrieve saved pastes (reads). Triggers API and CDN requests, generating network load on the gateway. In the demo, watch requests flow from the client browser.

CDN Edge Cache

Reduces latency and limits origin server load by caching and serving popular pastes at the network edge. Intercepts read requests and returns cached copy (Cache Hit) or forwards to Load Balancer on Cache Miss. In the demo, watch read requests hit this node and observe hits vs misses.

Load Balancer & API GW

Distributes traffic evenly and protects backend services from abuse. Applies sliding window rate limiting and forwards requests to the API Server Pool. In the demo, watch rate-limiting blocks return 429 Too Many Requests directly to the client browser.

API Application Server

Executes the core system business logic in a stateless manner. Coordinates with Key Generation Service, saves paste text bodies to S3, and writes search metadata to SQL Database. In the demo, watch it request keys, write to S3/DB, and invalidate the CDN.

Key Generation Service (KGS)

Prevents database locks and key collision checks at write-time. Generates unique Base62 strings offline, buffers them in memory, and allocates them instantly in $O(1)$ time. In the demo, watch it hand over pre-generated keys to the API Server.

Amazon S3 Object Store

Offers cheap, horizontally scalable storage for large raw text files, keeping database indexes clean. Persists raw paste payloads as text files. In the demo, watch raw content get uploaded here directly.

Metadata DB (SQL)

Serves as the indexed, relational query engine for mapping paste keys to metadata. Stores mapping details (short URL key, S3 path, expiration time, creation date, user ID). In the demo, observe row insertion operations (writes) and select queries (reads).

Expiration Cleaner

Garbage collects expired paste data in the background to free database index pages and S3 storage. Periodically searches SQL database for expired timestamps, deletes metadata rows, and deletes raw files from S3. In the demo, watch it issue SQL deletions and purge S3 blobs asynchronously.

Edge Cases and Misconceptions

Key Exhaustion and Recovery

A common misconception is that the Key Generation Service (KGS) never runs out of keys. If KGS crashes or its buffer runs dry due to a massive spike in write traffic, the API Application Server must fall back to dynamic key generation. Generating keys dynamically requires expensive database unique-key checks, leading to high write latencies and write lock escalations.

CDN Cache Invalidation vs. Natural Expiration

When a paste is updated or deleted, it is crucial to invalidate the CDN edge cache immediately. If the CDN cache is not invalidated, users will continue to retrieve the expired or old paste version from local edge caches until the Cache TTL expires naturally, leading to cache inconsistency.

Soft Delete vs. Hard Delete in Garbage Collection

Simply removing a metadata row from the SQL DB does not immediately free space on S3. Production systems mark the paste as inactive (soft delete) in SQL first. The Expiration Cleaner daemon subsequently processes this queue in batches, executing hard deletes on the Metadata DB and S3 asynchronously to avoid overloading database transaction logs.\n