Designing a URL Shortener Like Bitly
A system design breakdown covering ID generation, caching, database replication, and high availability.

A system design breakdown covering ID generation, caching, database replication, failure handling and the trade-offs behind the architecture.
A URL shortener is usually one of the first system design problems people practise. The basic implementation is straightforward: take a long URL, generate a short code, store the mapping and redirect users when they open the shortened link.
But the interesting engineering problems start when we think about scale.
How do we generate unique short codes across multiple application instances? How do we handle millions of redirects without overwhelming the database? What happens when a popular link suddenly receives millions of requests? And what happens to our ID counter when the primary database fails?
In this article, I’ll walk through how I would design a URL shortener like Bitly, starting with the basic requirements and gradually introducing the components needed to make the system scalable and highly available.
1. Understanding the Requirements
Before drawing boxes, let’s establish what we are actually building.
A user provides a long URL and receives a shorter URL. Anyone opening that shortened URL should be redirected to the original destination.
Functional Requirements
We’ll focus on these features:
- A user can submit a long URL and receive a shortened URL.
- A user can optionally specify a custom alias, such as
short.ly/my-launch. - A user can optionally specify an expiration time.
- Anyone accessing a valid short URL should be redirected to the original URL.
- The system should support disabling or taking down malicious URLs.
We’ll keep user account management and detailed click analytics outside the core requirements. However, we’ll discuss analytics and takedowns later because they affect the architecture.
Non-Functional Requirements
- Uniqueness: Every short code must map to exactly one long URL.
- Low latency: Target a redirect latency below 100 ms for most users, measured end-to-end where possible.
- High availability: Target 99.99% availability for redirects.
- Scalability: Support 1 billion stored URLs and 100 million daily active users.
- Durability: Successfully created URLs should not disappear after a database failure.
- Read-heavy optimisation: Most traffic will come from redirects rather than URL creation.
One important observation is the read-to-write ratio.
People click links far more often than they create them. Almost every architectural decision in this design follows from this observation.
2. Back-of-the-Envelope Estimation
Let’s put some numbers down before deciding on the architecture.
We’ll make the following assumptions:
- Daily active users (DAU): 100 million.
- Redirects per user per day: 5 request.
- New URLs created per day: 1 million.
- Total URLs stored: 1 billion.
- Average raw record size: 500 bytes.
Read Throughput
100 million users, with five redirects per day:
100M × 5 = 500M redirects/day
500M / 86,400 = 5,787 redirects/second
This is average traffic. Real traffic is never flat.
Assuming a 10x peak multiplier:
Peak reads ≈ 58,000 requests/second
Write Throughput
Assuming 1 million new URLs per day:
1M / 86,400 = 11.6 writes/second
Even with a 100x spike:
Peak writes ≈ 1,160 writes/second
Our estimated read-to-write ratio is approximately 500:1.
Storage Estimation
Assuming an average record size of 500 bytes:
1 billion × 500 bytes = 500 GB
This is only the raw data estimate. Actual storage requirements will be higher after accounting for indexes, WAL, database overhead, backups and operational headroom.
Still, this is a reasonable starting point for a well-provisioned PostgreSQL deployment.
Short Code Capacity
We’ll use Base62 encoding.
Base62 consists of:
- Lowercase letters: a-z (26)
- Uppercase letters: A-Z (26)
- Numbers: 0–9 (10)
Total: 62 characters.
The number of possible combinations depends on the length of the short code.
With seven characters:
62^7 = 3,521,614,606,208
For 1 billion URLs, we use approximately 0.028% of this namespace.
Seven characters provide sufficient room for our initial requirements.
What Do These Numbers Tell Us?
Write throughput is relatively low, and the dataset is manageable.
A single PostgreSQL primary should be a reasonable starting point. We can use caching and read replicas to handle the read-heavy workload.
There is no immediate justification for database sharding.
If someone proposes sharding on day one, ask them to explain which actual bottleneck requires it.
3. Data Model and API Design
Core Entity
We can represent the entire mapping in a single table.
PostgreSQL schema:
CREATE TABLE short_urls (
short_code TEXT PRIMARY KEY,
long_url TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
expiration_time TIMESTAMPTZ,
is_custom BOOLEAN NOT NULL DEFAULT FALSE,
owner_id BIGINT,
status VARCHAR(20) NOT NULL DEFAULT 'ACTIVE',
CHECK (status IN ('ACTIVE', 'DISABLED', 'TAKEN_DOWN'))
);
CREATE INDEX idx_short_urls_expiry
ON short_urls (expiration_time)
WHERE expiration_time IS NOT NULL;
The primary key gives us both uniqueness and an efficient lookup index.
The partial index on expiration_time will help our background cleanup job.
API Design
We need two primary APIs.
1. Create a Short URL
POST /api/v1/urls
Request:
{
"long_url": "https://www.example.com/products/some-very-long-path",
"custom_alias": "my-launch",
"expiration_time": "2027-03-31T23:59:59Z"
}
Both custom_alias and expiration_time are optional.
Response:
{
"short_url": "https://short.ly/aZ3kP9x",
"short_code": "aZ3kP9x"
}
2. Redirect to Original URL
GET /aZ3kP9x
Response:
HTTP/1.1 302 Found
Location: https://www.example.com/products/some-very-long-path
HTTP Status Codes
The 409 case is worth highlighting.
Suppose two users simultaneously try to create:
short.ly/my-launch
Checking whether the alias exists before inserting is not enough. Both requests could pass the check at the same time.
The database primary key must enforce uniqueness.
When the insert encounters a unique constraint violation, the Write Service translates it into HTTP 409 Conflict.
For generated codes, we retry with a fresh ID.
Why 302 Instead of 301?
A 301 represents a permanent redirect. Browsers can cache it, and subsequent clicks may never reach our service.
That creates problems if we want to expire, disable or change a URL.
A 302 represents a temporary redirect. It is not heuristically cacheable by default, but clients and CDNs can still cache it when explicit freshness headers are provided.
We control that behaviour through Cache-Control policies.
For our use case, 302 is a sensible default because we want to retain control over redirects.
4. High-Level Architecture
We have two very different workloads:
- URL creation: relatively low volume.
- URL redirection: significantly higher volume.
I would separate them into two services so that we can scale and operate them independently.
The main components are:
- API Gateway
- Write Service
- Read Service
- PostgreSQL Primary
- PostgreSQL Synchronous Standbys
- Redis Cluster
- CDN
- In-process LRU Cache (Caffeine/Ehcache)
One important distinction: public redirects should not require authentication. Someone clicking a link in a WhatsApp group should not need to log in.
Authentication belongs on the URL creation and management APIs. Public redirects need rate limiting and abuse protection, not user authentication.
Overall Architecture
Let’s go deeper into each path.
5. Deep Dive: How Do We Generate Unique Short Codes?
This is one of the most interesting parts of the design.
We need short codes that are:
- Unique.
- Short enough to fit in a URL.
- Cheap to generate.
- Safe to generate across multiple application instances.
Let’s look at some possible approaches.
Approach 1: Using a Prefix of the Long URL
This is obviously not going to work.
Consider:
https://example.com/products/iphone
https://example.com/products/macbook
Taking the first few characters would generate the same code.
We cannot guarantee uniqueness.
Approach 2: Hashing the Long URL
We could calculate SHA-256 of the URL, encode it using Base62 and take the first seven characters.
Long URL
https://www.example.com/blog/system-design/url-shortener?ref=newsletter
|
v
SHA-256 (256-bit digest, shown as hex)
26475c6e9119c2e960ce019d42a69b63567beb6d5ae0aecf61333caab455d7b5
|
v
Base62 Encoding (treat the digest as one big integer, then convert to base 62)
94mGJY3P9vPsEcCrQcWvhlLBvEGcyOM9D5qMWiqWdxF (43 characters)
|
v
First 7 characters
94mGJY3
|
v
Short Code
https://short.ly/94mGJY3
This looks attractive because hashing is deterministic and stateless.
However, there are two problems.
First, truncating the hash introduces collisions.
With a large number of URLs and a finite seven-character namespace, collisions become inevitable over time.
We need a unique constraint and a retry mechanism, possibly with a nonce or salt.
Second, the same long URL always produces the same hash.
Imagine two users shortening the same URL but wanting different expiration times or independent analytics.
A deterministic hash would give them the same short code.
We could introduce a user ID or random nonce into the hash, but that makes the design more complicated.
Approach 3: Using a Counter
A counter is simpler.
Every time we create a URL, we allocate a unique numeric ID and convert it into Base62.
1 -> 0000001
100 -> 000001c
1000 -> 00000G8
The exact characters depend on the Base62 implementation.
The advantage is that uniqueness is guaranteed by the ID allocator rather than by the probability of a hash not colliding.
The question now becomes: where should this counter live?
Why Not Redis INCR?
A common solution is to use Redis INCR.
Redis provides atomic increment operations, so multiple Write Service instances can safely request IDs.
It is a perfectly valid approach.
However, it introduces an additional stateful dependency whose primary responsibility is ID allocation.
We also need to think about counter durability, replication and failover. If the latest counter state is lost during failover, previously allocated IDs could potentially be reused.
Since PostgreSQL is already our durable source of truth, I’d prefer to use it.
PostgreSQL Sequence with Range Allocation
Instead of requesting one ID for every URL, we allocate a range of 1,000 IDs to each Write Service instance.
We create a PostgreSQL sequence:
CREATE SEQUENCE short_id_seq
START WITH 1
INCREMENT BY 1000
MINVALUE 1
NO CYCLE
CACHE 1;
Every time a Write Service instance needs a new range, it calls:
SELECT nextval('short_id_seq');
Suppose PostgreSQL returns 4001.
The service now owns:
4001 - 5000
It can generate IDs locally without making another database call for each URL.
When the range is exhausted, it requests another one.
Here’s the flow:
The nextval() operation is atomic, so concurrent application instances receive different sequence values.
If an instance crashes with 500 unused IDs, those IDs are simply discarded.
That’s fine. We need uniqueness, not gapless sequences.
PostgreSQL sequences are specifically designed for this kind of allocation. Their values are not rolled back when transactions abort, so gaps are expected. See the PostgreSQL sequence documentation.
Why **CACHE 1**?
PostgreSQL can cache sequence values per database session. Since we are already managing ranges in the application, keeping the sequence cache at one makes the allocation behaviour easier to reason about. It can still produce gaps; that is acceptable.
Converting IDs into Base62
Once we get the numeric ID, we convert it into Base62.
For example:
Numeric ID: 125000
Base62 Encoding:
125000 -> w7E
Fixed-length code:
0000w7E
I prefer keeping generated codes at exactly seven characters.
It gives us a predictable URL format and a fixed namespace.
Preventing Enumeration
There is one problem with a sequential counter.
Someone can guess the next short code.
short.ly/0000001
short.ly/0000002
short.ly/0000003
A script could enumerate codes and discover URLs.
To reduce casual enumeration, we can scramble the numeric ID before encoding it.
One option is a bijective affine permutation over the seven-character code space:
code_space = 62^7
scrambled = (n * K + C) mod code_space
short_code = base62(scrambled).padStart(7, '0')
Here, K must be coprime with the code-space size.
This preserves uniqueness while making sequential IDs less obvious.
A keyed Feistel-based permutation is another option when stronger scrambling is required.
However, neither approach is encryption or an access-control mechanism.
If a shortened URL points to sensitive information, the destination must enforce its own authentication and authorization. For genuinely unguessable private links, use longer cryptographically random tokens.
Handling Custom Aliases
Custom aliases introduce another possible collision.
Suppose the generated code is:
aZ3kP9x
A user might choose the same alias.
We don’t need a complicated solution. The primary key handles it.
- If the custom alias was created first, a generated-code collision results in a retry with another ID.
- If the generated code already exists, the custom alias request gets HTTP 409.
We can also reserve a separate alias format, such as requiring a hyphen in custom aliases, while keeping generated codes strictly seven-character Base62 strings.
Either way, the database constraint remains the final safety net.
6. Deep Dive: How Do We Make Redirects Fast?
Our peak traffic estimate is approximately 58,000 redirects per second.
Sending every request to PostgreSQL is unnecessary.
A primary-key lookup is efficient, but the database should not have to serve every redirect while also handling writes and replication.
We need multiple caching layers.
A CDN serves the redirect at the edge, so those clicks never reach our servers and are missed by our analytics. To limit this, we cache only viral URLs at the CDN and recover their counts from CDN logs.
The Read Path
Each layer protects the one below it.
Layer 1: CDN
The CDN is our first line of defence.
Viral URLs can be cached at geographically distributed points of presence.
Instead of reaching our application servers, a request can be answered directly from the nearest edge location.
We can cache redirect responses with explicit cache headers.
For example:
HTTP/1.1 302 Found
Location: https://example.com/products
Cache-Control: public, max-age=60
The actual TTL depends on whether the link can expire, be modified or be taken down.
A longer TTL improves performance but makes takedown and updates harder.
Layer 2: In-Process LRU Cache
If the CDN does not have the mapping, the request reaches our Read Service.
Before calling Redis, we check a local in-memory LRU cache.
LRU stands for Least Recently Used.
When the cache reaches its configured capacity, the least recently accessed entries are removed.
This is useful for frequently accessed URLs because we avoid a network call to Redis.
The limitation is that every application instance has its own cache.
If we have ten Read Service instances, we have ten independent LRU caches.
We need another shared layer.
Layer 3: Redis Cluster
Redis stores frequently accessed short-code mappings.
For example:
Key: short_url:aZ3kP9x
Value:
{
"long_url": "https://example.com/products",
"expiration_time": "2027-03-31T23:59:59Z",
"status": "ACTIVE"
}
We run Redis as a cluster with replicas rather than relying on a single Redis instance.
If a Redis node fails, we lose some cache capacity temporarily, but PostgreSQL remains our source of truth.
Layer 4: PostgreSQL
If all cache layers miss, we query PostgreSQL.
Normally, reads go to a read replica.
But there is an important edge case.
Imagine a user creates a URL and immediately tries to open it.
The primary has committed the record, but an asynchronous read replica has not received or applied the transaction yet.
The replica returns no result.
If we immediately return 404, we have a read-after-write consistency problem.
There are two mechanisms to address this:
Cache on write
After the database transaction commits, the Write Service populates Redis with the new mapping. It also removes any negative cache entry for that short code.
Primary fallback
If Redis and the read replica both miss, the Read Service checks the primary before returning 404.
This protects us against replication lag and makes the immediate post-creation experience reliable.
The primary fallback must be bounded and protected with rate limiting, connection pools and negative caching. Otherwise, random requests for invalid codes could overload the primary.
An alternative is to use synchronous_commit = remote_apply for transactions that must be visible on a synchronous standby before the commit returns. This increases write latency, so we should make the trade-off consciously.
Cache Expiration
The cache TTL should never exceed the remaining lifetime of a URL.
cache_ttl = min(default_cache_ttl, expiration_time - now)
That way, an expired URL does not remain in the cache indefinitely.
The Read Service should still check the expiration time before redirecting.
An expired link returns HTTP 410 Gone.
7. Handling Hot Keys
Traffic in a URL shortener is not uniformly distributed.
A single link shared by a celebrity or a popular news outlet can receive more traffic than millions of other links combined.
This is the hot-key problem.
Suppose one Redis shard receives a huge percentage of all requests because every request is asking for the same short code.
That shard becomes a bottleneck.
Our existing layers already help:
- CDN serves popular redirects.
- Local LRU caches absorb repeated requests.
- Redis handles the remaining shared traffic.
If a particular key is still too hot, we can replicate that cache entry across multiple Redis keys.
For example:
short_url:aZ3kP9x#0
short_url:aZ3kP9x#1
short_url:aZ3kP9x#2
short_url:aZ3kP9x#3
The Read Service selects one of these keys, spreading traffic across different Redis shards.
We should only do this for keys identified as hot through monitoring.
Replicating every key adds unnecessary memory usage and invalidation complexity.
8. Handling Cache Stampedes
This is one of the most important failure scenarios in the design.
Imagine a popular URL receives 10,000 requests per second.
Its Redis entry expires.
Suddenly, thousands of concurrent requests experience a cache miss and query PostgreSQL for the same record.
The database becomes overloaded. Requests start timing out, clients retry, and the situation gets worse.
This is called a cache stampede.
We can combine a few techniques to handle it.
8.1. Request Coalescing / Singleflight
Within one application instance, requests for the same key can be grouped together.
Only one request performs the database lookup.
Other requests wait for that result.
This reduces 100 database requests to one within that instance.
However, we may still have multiple Read Service instances.
8.2. Distributed Locking
To coordinate cache refreshes across instances, we can use Redis locks.
SET lock:short_url:aZ3kP9x <unique-token> NX PX 2000
NX: Acquire the lock only if it doesn’t exist.PX 2000: Lock expires after two seconds.unique-token: Identifies the owner of the lock.
The instance that acquires the lock queries PostgreSQL and refreshes Redis.
Other instances briefly wait and check the cache again, rather than immediately querying the database.
8.3. Safe Unlock Using Lua
We cannot simply execute:
DEL lock:short_url:aZ3kP9x
Imagine our lock expires after two seconds, another instance acquires it, and then the original instance tries to unlock.
It could accidentally delete the second instance’s lock.
We need an atomic check-and-delete operation.
Conceptually:
if redis.call("GET", KEYS[1]) == ARGV[1] then
return redis.call("DEL", KEYS[1])
else
return 0
end
This ensures that only the owner of the lock can release it.
For cache refresh, this lock is an optimisation to reduce duplicate work, not a correctness guarantee.
8.4. TTL Jitter
Suppose thousands of cache entries were populated at approximately the same time with identical TTLs.
They may expire together and create a sudden burst of database traffic.
We can introduce random variation:
actual_ttl = base_ttl + random_jitter
For example, 10–20% random jitter spreads expiration events over time.
8.5. Early Refresh
For heavily accessed keys, we can refresh the cache before it expires.
Instead of waiting for a cache miss, the Read Service detects that the entry is approaching expiry and refreshes it in the background.
This is different from normal cache-aside behaviour, where we populate the cache only after a miss.
8.6. Negative Caching
Not every request contains a valid short code.
Bots and scanners may repeatedly request non-existent codes.
Without negative caching, every request would hit PostgreSQL.
We can cache a not-found result for a short period, such as 30–60 seconds.
When a new URL is created with that code, the Write Service must invalidate the negative cache entry.
8.7. Bulkhead and Connection Limits
Finally, we protect PostgreSQL itself.
The Read Service should have:
- Bounded database connection pools.
- Timeouts on database requests.
- Limits on concurrent cache-miss lookups.
- Circuit breakers and controlled retry policies.
When the database is struggling, we should shed load rather than let every request wait indefinitely.
A slow database should degrade the redirect service gradually, not bring down the entire application.
9. Deep Dive: Database Replication and High Availability
Our PostgreSQL primary handles writes.
We have already reduced most read traffic using CDN, LRU and Redis.
Now the question is: what happens when the primary database fails?
A single database instance is a single point of failure.
We need replication and automatic failover.
Database Topology
I’d use one primary and two synchronous standbys in separate availability zones.
The configuration concept is:
synchronous_standby_names =
'ANY 1 (standby_a, standby_b)'
This means a transaction waits for acknowledgement from at least one of the configured synchronous standbys.
With synchronous_commit = on, PostgreSQL waits for the standby to flush the relevant WAL to durable storage.
It does not necessarily wait for that standby to replay the WAL and make the change visible to queries.
For that stronger visibility guarantee, PostgreSQL provides:
synchronous_commit = remote_apply
That waits until the synchronous standby has replayed the transaction.
We don’t necessarily need remote_apply for every transaction because our Read Service already has primary fallback.
The trade-off is straightforward:
on: Better write latency, with durable WAL on a synchronous standby.remote_apply: Stronger immediate read visibility on the synchronous standby, at the cost of additional write latency.
Official reference: PostgreSQL synchronous replication.
What Happens When the Primary Fails?
Synchronous replication protects committed data, but it doesn’t automatically promote a standby.
We need a cluster management mechanism.
I’d use:
- Patroni: Manages PostgreSQL cluster health, leader election and failover.
- etcd: Provides distributed coordination and leader information.
- Database proxy or stable endpoint: Directs applications to the current primary.
The important point is that quorum commit and leader election are two different problems.
Patroni’s synchronous mode tracks which standby nodes are eligible for safe promotion.
If a node cannot be proven to contain all successfully acknowledged commits, we should not automatically promote it when data loss is unacceptable.
This introduces an important trade-off: in some failure scenarios, we may temporarily stop accepting writes rather than promote a replica that could lose acknowledged data.
With quorum-based synchronous replication, Patroni maintains additional state to reason about which standby contains the latest committed WAL.
It is important to configure Patroni’s synchronous/quorum mode properly instead of assuming that a PostgreSQL ANY 1 setting alone provides a complete failover solution.
Reference: Patroni replication modes.
What Happens to Our Allocated ID Ranges During Failover?
This is an interesting edge case.
Suppose Write Service instance 1 already owns:
4001 - 5000
It has used only 200 IDs when the PostgreSQL primary fails.
There is no problem with the application continuing to use the unused IDs, provided the allocated range has been safely persisted on a standby.
The important rule is that an application must not start using a range until the nextval() operation has successfully completed under the configured synchronous commit policy.
If the service crashes, its unused IDs are discarded.
If the primary fails, Patroni must promote a standby that contains the acknowledged sequence advancement.
If a failover cannot prove that the most recent acknowledged allocation is present, automatic promotion must not be treated as lossless.
This is why ID allocation, synchronous replication and failover cannot be designed independently.
What About Redis?
Redis is a cache, not our source of truth.
If Redis fails, the system should continue working by falling back to PostgreSQL.
However, a complete Redis failure can produce a massive increase in database traffic.
We should therefore use Redis Cluster/Sentinel with replicas and monitor failovers, cache hit ratios and per-shard load.
Do We Need Multi-Region Deployment?
Not initially.
If we eventually need writes from multiple regions, we can allocate disjoint numeric ID ranges to each region.
For example:
Region A: 0 - 1 billion
Region B: 1 - 2 billion
Region C: 2 - 3 billion
Each region generates IDs from its own range, avoiding coordination for every new URL.
Regional CDNs and read replicas handle redirects.
We also need to think about replication lag between regions, regional failover and recovery of the ID allocator. Multi-region writes introduce substantially more complexity, so I would not add them without an actual requirement.
When Would I Introduce Database Sharding?
Not at this scale.
We have:
- Approximately 12 average writes per second.
- Around 1,160 writes per second during our assumed peak.
- Approximately 500 GB of raw data.
- Most reads handled by caching.
I’d first scale vertically, add read capacity and optimise the database.
If the primary eventually becomes a write bottleneck or the dataset exceeds the practical capacity of one database node, we can introduce sharding using short_code as the shard key.
The important thing is to introduce sharding when the measurements justify it, not because the architecture diagram looks more distributed.
10. Click Analytics Without Affecting Redirect Latency
Click analytics are not part of the core requirements, but they are a natural extension.
We may want to track:
- Total clicks.
- Country and city.
- Device and browser.
- Referrer.
- Clicks over time.
We should not update a PostgreSQL row synchronously on every redirect.
At 58,000 requests per second, that would put enormous write pressure on the database.
Instead, we publish click events asynchronously.
An example event:
{
"short_code": "aZ3kP9x",
"timestamp": "2026-11-14T18:30:00Z",
"country": "IN",
"device": "mobile",
"referrer": "whatsapp"
}
The redirect should not wait for analytics processing.
If the analytics pipeline is down, links should continue working.
One important detail: if the CDN caches redirect responses, those clicks never reach our Read Service.
So if we need complete click analytics, we should also consume CDN access logs or edge-generated events.
We can aggregate the data in a separate analytics-oriented database rather than putting this workload on PostgreSQL.
11. Handling URL Expiration and Takedowns
We have two different cases:
- A URL expires naturally.
- A URL needs to be disabled because of abuse, spam or a legal request.
Both need to invalidate cached entries.
Expiration Cleanup
Every redirect request checks whether the URL has expired.
If it has expired, we return:
HTTP/1.1 410 Gone
We don’t have to delete the row immediately.
A periodic background job can remove expired mappings in small batches.
For example:
WITH expired AS (
SELECT short_code
FROM short_urls
WHERE expiration_time < NOW()
ORDER BY expiration_time
LIMIT 10000
FOR UPDATE SKIP LOCKED
)
DELETE FROM short_urls s
USING expired e
WHERE s.short_code = e.short_code;
The FOR UPDATE SKIP LOCKED approach allows multiple cleanup workers to process different batches without waiting on rows already locked by another worker.
Small batches also reduce long-running transactions and replication pressure.
We can archive records before deletion if an audit trail is required.
Takedown Flow
Suppose a URL is reported for phishing.
Deleting the row from PostgreSQL is not sufficient.
The mapping may still exist in:
- CDN cache.
- Redis.
- Local LRU caches on multiple application instances.
We need a proper invalidation flow.
The sequence is:
- Update the URL status in PostgreSQL.
- Record an invalidation event in a transactional outbox.
- Publish the event to a durable event bus.
- Remove the mapping from Redis.
- Notify Read Service instances to invalidate their local LRU entries.
- Purge the corresponding CDN entry.
Why use an outbox?
If the database update succeeds but the application crashes before publishing the event, a simple publish-after-commit approach can lose the invalidation message.
The transactional outbox ensures the database update and event record are committed together. A separate publisher delivers the event.
There is one limitation worth calling out.
CDN purge and pub/sub propagation are not instantaneous guarantees.
If a CDN has cached a 302 response, it may continue serving that response until its cache expires or the purge takes effect.
For stricter takedown requirements, we need short CDN TTLs, reliable invalidation and potentially an authoritative revocation check at the edge.
We should not claim that pub/sub alone provides immediate global takedowns.
12. Abuse Prevention and Security
A URL shortener can easily become a platform for phishing and malicious redirects.
We should validate URLs at creation time.
Some basic checks include:
- Allow only
httpandhttpsschemes. - Reject malformed URLs and unsupported protocols.
- Check known malware and phishing databases.
- Apply per-user and per-IP creation limits.
- Monitor suspicious redirect patterns.
If we fetch submitted URLs for scanning or previews, that operation must run in an isolated environment.
Otherwise, a malicious destination could exploit our backend to access internal services or cloud metadata endpoints. The scanning service should block private, loopback and link-local addresses and protect against DNS rebinding.
Finally, short-code scrambling is not authorization.
For links containing sensitive information, the destination must still enforce access control.
13. Observability: How Do We Know the System Is Healthy?
A staff-level design should also cover how we operate the system after deployment.
Here are the metrics I’d start with:
ComponentImportant metricsCDNCache hit ratio, edge latency, purge latencyRead ServiceP95/P99 latency, redirect errors, request rateWrite ServiceCreation latency, 409 rate, failed requestsRedisHit ratio, per-shard QPS, memory, eviction ratePostgreSQLQuery latency, connection usage, replication lagID allocatorRange allocation failures, unused ranges, duplicate retriesKafkaConsumer lag, event processing throughputPatroniLeader changes, failover duration, standby health
I’d pay particular attention to three metrics:
1. Cache hit ratio: A sudden drop can increase PostgreSQL traffic dramatically.
2. Replication lag: This affects read consistency and the reliability of the primary fallback strategy.
3. Database connection pool saturation: This is often an early sign that cache misses or a slow database are creating a larger incident.
We should also track availability and redirect latency by geography, not just from a single monitoring location.
14. Final Architecture
Let’s put everything together.
Write Path Summary
- The client submits a URL through the authenticated API Gateway.
- The Write Service validates the request.
- It gets an ID from its locally held range, allocating a new range from PostgreSQL when required.
- The ID is scrambled and converted to a seven-character Base62 code.
- The mapping is inserted into the PostgreSQL primary.
- The transaction is synchronously replicated according to our durability policy.
- After commit, the Write Service populates Redis and clears any negative cache entry.
- The short URL is returned.
If the custom alias already exists, the insert fails with a unique constraint violation and we return 409.
Read Path Summary
- The browser requests the shortened URL.
- The CDN serves a cached redirect if available.
- Otherwise, the request reaches the Read Service through the public redirect route, with rate limiting but no user authentication.
- The service checks its local LRU.
- On a miss, it checks Redis.
- On another miss, it queries a PostgreSQL read replica.
- If the record is missing, it checks the primary before returning 404.
- It validates the status and expiration time, then returns a 302 redirect.
Request coalescing, distributed locks, TTL jitter, early refresh, negative caching and database bulkheads protect the cache-miss path.
15. What Is Expected at Each Interview Level?
I find it useful to think about the design in terms of what an interviewer expects from different engineering levels.
Mid-Level Engineer
You should be able to explain the complete flow:
- How a long URL becomes a short URL.
- How the mapping is stored.
- How redirects work.
- Why Base62 is useful.
- Why 302 is preferable to 301 for this use case.
- Why we need a database index and cache.
The interviewer may help you reach some of these decisions.
Senior Engineer
You should lead most of the discussion.
You should be able to:
- Compare hashing, random codes and counters.
- Explain how multiple application instances generate unique IDs.
- Justify PostgreSQL over Redis as the durable source of truth.
- Separate read and write paths.
- Design the caching layers.
- Discuss consistency, expiry and cache invalidation.
- Explain database replication and the trade-offs involved.
Staff Engineer and Above
The discussion should go beyond the happy path.
You should start with capacity numbers and use them to justify architectural decisions.
More importantly, you should proactively discuss failure scenarios:
- What happens when a popular Redis key expires?
- What happens when a Read Service instance crashes while holding an ID range?
- Can a database failover cause an ID range to be allocated twice?
- What happens when a synchronous standby is unavailable?
- Can a CDN continue redirecting users after a takedown?
- What happens when the analytics pipeline is down?
- How do we know the system is degrading before users report an issue?
You should also be able to explain why you are not introducing certain technologies.
For example, why not shard PostgreSQL when the workload is still manageable by one primary?
A good staff-level design is not the one with the most components. It is the one where every component has a reason to exist.
Closing Thoughts
The interesting part of designing a URL shortener isn’t generating a short string. That is relatively straightforward.
The real engineering challenges come from ensuring uniqueness across distributed application instances, making redirects fast, handling database failures and keeping cached data consistent.
I’d start with PostgreSQL as the primary data store, allocate IDs in batches, use Base62 encoding and separate the read and write paths.
For the read-heavy workload, I’d introduce CDN caching, in-process LRU and Redis before allowing requests to reach PostgreSQL.
I’d keep analytics asynchronous, build a reliable takedown mechanism and introduce sharding only when actual capacity requirements justify it.
One thing I would particularly emphasise in an interview is the relationship between ID allocation and database failover. It is easy to draw a sequence and assume that uniqueness is solved. But once application instances hold ranges in memory, we need to think about what happens when the primary changes.
The same applies to caching. Adding Redis is easy. Understanding what happens when a viral link expires from Redis is where the interesting design discussion begins.
Start with the numbers, identify the bottlenecks, and design for the failure cases. Don’t introduce complexity just to make the architecture look scalable.
If you found this article helpful, do leave a clap 👏 and follow for more insights on GenAI, Python, and backend engineering!
Discussion
Have a question, a correction, or a different approach? Sign in with GitHub to join the conversation.