Skip to content

Start typing to search…

Navigation
On this page

Designing a Dropbox-Like File Storage and Synchronisation System

A system-design deep dive covering content-defined chunking, block storage, and cross-device sync.

A system-design deep dive inspired by Dropbox, Google Drive and OneDrive

Dropbox looks like a very simple product. You save a file in a folder, and it appears on your phone, your laptop and your other devices. You share it with a friend, and your friend can open it too.

But the real fun starts when things go wrong:

  • Someone uploads a 50 GB file over a weak internet connection.
  • The app crashes when 90% of the file is already uploaded.
  • Two people edit the same document at the same time.
  • A user logs in from a new laptop that has 100,000 files in the account.
  • A shared file is downloaded by thousands of users.
  • One faulty client opens thousands of parallel upload connections.

So the challenge is not just storing files. The challenge is building a reliable file storage and synchronisation system.

Let us start.

1. Requirements

Functional requirements

  1. Users can upload files from different devices.
  2. Users can download files from different devices.
  3. Users can share files with other users, and view files shared with them.
  4. Changes made on one device should sync automatically to the other devices.
  5. Delta sync: when a large file is modified, we should reuse already-stored content and transfer only the chunks that are missing.
  6. Conflict handling: concurrent edits to the same file follow a Last Write Wins (LWW) policy. The latest successfully committed update becomes the current version.

Non-functional requirements

  1. High availability: files remain accessible even if some application instances fail.
  2. Scalability: millions of users and a very large number of files.
  3. Reliability: committed data and metadata must be durable, and file integrity must be verified.
  4. Performance: metadata operations and sync notifications should be fast. Large transfers should be resumable.
  5. Security: only authorised users can access files. Data is encrypted in transit and at rest.
  6. Recoverability: the system recovers from crashes and network failures.
  7. Large file support: files up to 50 GB.

Out of scope

To keep the design manageable, we will not cover:

  • Real-time collaborative editing (like Google Docs).
  • File previews and in-app document editing.
  • Per-user storage plans and billing.
  • File format conversion.
  • Users can restore old versions and deleted files within the retention period.

A note on consistency

We should not simply say that Dropbox chooses availability over consistency.

Different operations need different guarantees.

For example, another device receiving a file update a few seconds later is acceptable. But allowing a user whose access has been revoked to obtain new download permissions because of stale authorization data is not.

We’ll use strong consistency for critical metadata and permissions, while allowing asynchronous propagation for device synchronization.

2. Capacity Estimation

Let’s put some numbers down before deciding on the architecture.

System assumptions

  • Daily active users: 1 million
  • New uploads per user per day: 0.1
  • Average file size: 20 MB
  • Download volume: 4x uploaded data
  • Peak traffic multiplier: 5x
  • Average CDC chunk size: 8 MiB

All data-transfer calculations below use decimal TB and MB. Chunk sizes use binary MiB.

2.1 Upload Volume

First, calculate the number of uploads per day.

Uploads per day = 1,000,000 x 0.1 = 100,000 uploads/day

Now calculate daily logical upload volume.

Daily upload data = 100,000 x 20 MB = 2,000,000 MB = 2 TB/day

This represents the logical file data processed by clients before chunk reuse.

Because we use content-addressed chunks, the actual bytes transferred to S3 can be lower when chunks already exist in the owner’s storage scope.

Average ingress throughput:

2 TB / 86,400 seconds = approximately 23 MB/second

Assuming peak traffic is five times the average:

Peak ingress = 23 x 5 = approximately 115 MB/second

2.2 Download Volume

We assume downloads are four times the uploaded data volume.

Daily download data = 2 TB x 4 = 8 TB/day

Average aggregate download throughput:

8 TB / 86,400 seconds = approximately 93 MB/second

If we apply the same five-times peak assumption to downloads:

Peak download throughput = 93 x 5 = approximately 463 MB/second

2.3 Chunk Processing and Upload Requests

Our architecture uses content-defined chunking.

Assuming an illustrative average chunk size of 8 MiB:

Chunks per day = 2 TB / 8 MiB = approximately 238,000 chunks/day

Average chunk processing rate:

238,000 / 86,400 = approximately 2.8 chunks/second

This is an estimate before deduplication.

The client and backend still need to process the manifest entries, but only missing chunks are uploaded to S3.

2.4 Storage Growth

Assuming all uploaded data is retained:

Daily logical upload volume = 2 TB

30-day logical upload volume = 2 x 30 = 60 TB/month

Actual physical storage growth depends on:

  • Chunk deduplication.
  • Compression
  • Retained file versions.
  • User deletions and retention policies.
  • Replication and backup requirements.

2.5 What Does This Tell Us?

These estimates help identify which components need attention as the system grows.

  • S3: Stores independently uploaded content chunks.
  • CDN: Handles large download volumes and reduces origin traffic.
  • PostgreSQL: Stores metadata, versions, manifests and per-user change feeds.
  • Message Queue: Handles asynchronous event processing and notifications.
  • Redis: Can reduce frequently repeated metadata lookups.
  • Client sync agent: Handles chunk processing, persistent upload/download queues and bounded parallelism.

3. Core Entities

These are the main building blocks of our system.

  • User: a person using the system.
  • Folder: a directory that contains files and other folders. It refers to its parent folder, so we get a hierarchy.
  • File: a logical file. It stores name, owner, parent folder and the current version.
  • FileVersion: an immutable snapshot of the file’s content. It references the chunks that make up that version.
  • Chunk: an independently stored piece of file content, identified by its content hash.
  • FileVersionChunk: maps chunks to a file version and keeps their order using a chunk index.
  • UploadSession: tracks one ongoing upload.
  • UploadSessionChunk: temporary references to chunks needed by an upload, so that garbage collection does not delete them too early.
  • Share: maps a file to a user who has permission to access it.
  • Device: a laptop, phone or desktop installation. It remembers the last sync cursor it applied.
  • Change: a durable change-log entry (update, rename, delete, permission change). Devices use a cursor to fetch changes they have not yet applied.

Understanding File vs FileVersion

This is an important point. A File is a logical resource. A FileVersion is one immutable snapshot of its content.

When a user edits project-report.pdf, we never overwrite the old version. We create a new version and move the file’s current_version_id pointer. The old versions stay available for recovery. This one decision gives us version history, restore, and a clean way to handle concurrent edits.

Entity Relationships

The following diagram shows how the main entities relate to each other.

                         User
                          |
             +------------+------------+
             |            |            |
             v            v            v
          Folder        File         Device
             |            |            |
             |            |            +---- Change Feed
             |            |
             |            +---- current_version_id
             |            |              |
             |            |              v
             |            +--------> FileVersion
             |                           |
             |                           v
             |                    FileVersionChunk
             |                           |
             |                           v
             |                         Chunk
             |                           |
             |                           v
             |                        S3 Object
             |
             +---- parent_folder_id

----------------------------------------------------------------------------

            File
             |
             +---- UploadSession
             |         |
             |         v
             |    UploadSessionChunk
             |         |
             |         v
             |       Chunk
             |
             +---- Share ---- User (recipient)

4. APIs

We’ll expose REST APIs for the initial design.

File APIs

POST   /files
GET    /files/{fileId}
PATCH  /files/{fileId}
GET    /files/{fileId}/download
DELETE /files/{fileId}

Upload APIs

POST   /files/{fileId}/upload-sessions
GET    /upload-sessions/{uploadSessionId}
PUT    <presigned-url>
POST   /upload-sessions/{uploadSessionId}/commit
DELETE /upload-sessions/{uploadSessionId}

Version APIs

GET  /files/{fileId}/versions
GET  /files/{fileId}/versions/{versionId}/manifest

Sharing APIs

POST   /files/{fileId}/shares
DELETE /files/{fileId}/shares/{userId}
GET    /users/me/shared-files

Synchronization APIs

GET  /sync/initial
GET  /sync/changes?cursor={cursor}
POST /sync/acknowledge

These APIs coordinates uploads and downloads, verifies metadata, checks permissions, and generates short-lived pre-signed URLs.

Clients transfer the actual chunks directly to S3 or through the CDN.

4.1 Two important API contracts

1. Create an upload session. The client first splits the file into content-addressed chunks, then sends the ordered manifest.

POST /files/{fileId}/upload-sessions
{
  "chunkerVersion": 1,
  "contentSha256": "file-content-hash",
  "manifest": [
    {"hash": "h1", "size": 4194304},
    {"hash": "h2", "size": 4194304},
    {"hash": "h3", "size": 4194304},
    {"hash": "h4", "size": 4194304}
  ]
}

The backend checks which chunks already exist in the owner’s storage and gives pre-signed PUT URLs only for the missing ones.

{
  "uploadSessionId": "upload-123",
  "missingChunks": [ { "hash": "h3", "putUrl": "https://..." } ]
}

2. Retrieve a version manifest. The client needs the ordered manifest to rebuild the file.

GET /files/{fileId}/versions/{version}/manifest?pageToken=...
{
  "chunks": [
    { "index": 0, "hash": "h1", "size": 4194304, "downloadUrl": "https://..." },
    { "index": 1, "hash": "h2", "size": 4194304, "downloadUrl": "https://..." }
  ],
  "nextPageToken": "..."
}

The backend checks permission first and returns short-lived download URLs. For files with thousands of chunks, the manifest is paginated.

4.2 Idempotency

We also use idempotency keys for file creation, upload-session creation and version commit. A retried request must not create duplicate files or versions.

5. Data Model

We use PostgreSQL for metadata and Amazon S3 for actual file content. Our data is full of relationships (users, files, folders, versions, chunks, permissions), so PostgreSQL is a good starting point. DynamoDB could also work.

The key idea: PostgreSQL stores metadata and relationships. S3 stores content-addressed chunks.

Users, Folders and Files

users   (user_id BIGINT PK, email TEXT UNIQUE, created_at TIMESTAMPTZ)

folders (folder_id UUID PK, owner_id BIGINT, parent_folder_id UUID NULL,
         folder_name TEXT, created_at TIMESTAMPTZ, deleted_at TIMESTAMPTZ)

files   (file_id UUID PK, owner_id BIGINT, parent_folder_id UUID,
         file_name TEXT, current_version_id UUID,
         created_at TIMESTAMPTZ, updated_at TIMESTAMPTZ, deleted_at TIMESTAMPTZ)

CREATE INDEX idx_folders_owner_parent ON folders(owner_id, parent_folder_id);
CREATE INDEX idx_files_owner_folder   ON files(owner_id, parent_folder_id);

A NULL parent folder means a root-level folder. A files row is only a logical resource. current_version_id points to the latest successfully committed version.

File versions

file_versions (
  version_id UUID PK, file_id UUID, version_number BIGINT,
  file_size BIGINT, content_sha256 TEXT, manifest_hash TEXT,
  chunker_version INT, created_by BIGINT, created_at TIMESTAMPTZ,
  UNIQUE(file_id, version_number)
)

Notice there is no object_key here. A version does not point to one S3 object. It points to an ordered list of chunks. Every committed version is immutable.

Chunks and the version-to-chunk mapping

chunks (
  chunk_id UUID PK, owner_id BIGINT, chunk_hash TEXT, chunk_size BIGINT,
  object_key TEXT UNIQUE, storage_checksum TEXT, status TEXT,
  created_at TIMESTAMPTZ, UNIQUE(owner_id, chunk_hash)
)

file_version_chunks (
  version_id UUID, chunk_index INT, chunk_id UUID,
  PRIMARY KEY(version_id, chunk_index)
)

CREATE INDEX idx_file_version_chunks_chunk ON file_version_chunks(chunk_id);

The chunk_hash is the SHA-256 of the original chunk. It tells us if the chunk already exists. We scope deduplication per owner, not globally, so that nobody can find out whether another user’s files contain some particular content.

Upload sessions

upload_sessions (
  upload_session_id UUID PK, file_id UUID, owner_id BIGINT,
  expected_size BIGINT, content_sha256 TEXT, manifest_hash TEXT,
  chunker_version INT, status TEXT,
  committed_version_id UUID NULL, expires_at TIMESTAMPTZ, created_at TIMESTAMPTZ
)

upload_session_chunks (
  upload_session_id UUID, chunk_index INT, chunk_hash TEXT, chunk_size BIGINT,
  status TEXT, lease_expires_at TIMESTAMPTZ,
  PRIMARY KEY(upload_session_id, chunk_index)
)

If the app crashes or the network drops, the client can resume with the same session (if not expired). committed_version_id lets us return the same version if the client retries a commit. The upload_session_chunks table protects chunks from garbage collection while an upload is in progress.

Shares

file_shares (
  share_id UUID PK, file_id UUID, recipient_id BIGINT,
  permission TEXT, created_at TIMESTAMPTZ, revoked_at TIMESTAMPTZ
)

CREATE INDEX idx_shares_recipient ON file_shares(recipient_id, file_id)
  WHERE revoked_at IS NULL;
CREATE UNIQUE INDEX idx_unique_active_share ON file_shares(file_id, recipient_id)
  WHERE revoked_at IS NULL;

We keep revoked shares as history, and still allow a fresh share later.

Devices, change log and outbox

devices (device_id UUID PK, user_id BIGINT, last_ack_cursor BIGINT, last_seen_at TIMESTAMPTZ)

user_changes (
  user_id BIGINT, sequence_id BIGINT, resource_type TEXT, resource_id UUID,
  event_type TEXT, version_id UUID, created_at TIMESTAMPTZ,
  PRIMARY KEY(user_id, sequence_id)
)

user_change_counters (user_id BIGINT PK, last_sequence BIGINT)
outbox (
  event_id UUID PK, aggregate_type TEXT, aggregate_id UUID,
  event_type TEXT, payload JSONB, created_at TIMESTAMPTZ, published_at TIMESTAMPTZ
)
  • Each device keeps its own cursor. A laptop and a phone can be at different points in the feed.
  • Whenever we create a change, we increment the user’s counter and insert the change row in the same transaction. This gives every user a monotonically increasing sequence.
  • The outbox table helps us publish events reliably. When a version is committed, we update the pointer, insert change-log rows and insert an outbox event in one transaction. A background publisher later sends outbox rows to the message broker.

The change log is the source of truth for sync. The message broker only delivers notifications.

6. High-Level Design

Let’s start with the basic architecture.

                         +-------------------------+
                         |         Clients         |
                         |   Web / Desktop / App   |
                         +------------+------------+
                                      |
                                      v
                         +-------------------------+
                         |       DNS / CDN         |
                         |          WAF            |
                         +------------+------------+
                                      |
                                      v
                         +-------------------------+
                         |       External LB       |
                         +------------+------------+
                                      |
                    +-----------------+-----------------+
                    |                                   |
                    | HTTPS API                         | WebSocket / SSE
                    v                                   v
           +--------------------+             +----------------------+
           |    API Gateway     |             |   Realtime Gateway   |
           | Auth / Rate Limit  |             | Persistent Client    |
           +---------+----------+             | Connections          |
                     |                        +----------+-----------+
          +----------+----------+                        ^
          |          |          |                        |
          v          v          v                        |
    +----------+ +----------+ +-----------+              |
    |   File   | |   Sync   | |   Share   |              |
    | Service  | | Service  | |  Service  |              |
    +----+-----+ +----+-----+ +-----+-----+              |
         |            |             |                    |
         +------------+-------------+                    |
                      |                                  |
                      v                                  |
           +-------------------------+                   |
           |       PostgreSQL        |                   |
           |                         |                   |
           | File / Folder Metadata  |                   |
           | Shares / Permissions    |                   |
           | File Versions           |                   |
           | Ordered Chunk Manifests |                   |
           | Chunk Metadata          |                   |
           | Upload Sessions         |                   |
           | Durable Change Feed     |                   |
           | Transactional Outbox    |                   |
           +------------+------------+                   |
                        ^                                |
                        | Poll / Read Outbox             |
               +--------+---------+                      |
               |  Outbox Publisher|                      |
               +--------+---------+                      |
                        |                                |
                        v                                |
               +-------------------+                     |
               |   Message Queue   |                     |
               |    SQS / Kafka    |                     |
               +--------+----------+                     |
                        |                                |
                        v                                |
               +-------------------+                     |
               | Notification      |---------------------+
               | Service           |
               +-------------------+


DIRECT FILE DATA TRANSFER
-------------------------
    UPLOAD
    Client ----------------------------------------> S3
           Pre-signed PUT URL
           Individual content-addressed chunks
    DOWNLOAD
    Client ---------------------> CDN ----------------> S3
           Request chunk          Cache miss / origin
    Client <--------------------- CDN
           Chunk data

There are three important parts:

  1. Application services handle authentication, authorisation, metadata, upload coordination, version commits and sync. They do not carry the actual file bytes.
  2. PostgreSQL stores metadata, versions, permissions, upload sessions, chunk manifests and the change feed.
  3. Amazon S3 stores the actual content-addressed chunks.

Redis can be added to cache metadata and permissions, but PostgreSQL stays the source of truth.

7. High-Level Design: Requirement by Requirement

7.1 Upload a File

Let’s start with a simple approach and improve it.

Bad: Upload the file through our application server

The client sends the entire file to the File Service, and the File Service uploads it to S3.

Client
  |
  | 50 GB file
  v
File Service
  |
  | 50 GB file
  v
 S3

This means the application server handles the entire file transfer.

For large files, this makes our application servers a data-transfer bottleneck. They consume network bandwidth, connection capacity and compute resources. If the server buffers file contents, memory pressure also becomes a concern.

We should avoid routing large file contents through our application servers.

Better: Upload directly to S3 using pre-signed URLs

A pre-signed URL is a temporary URL that grants permission to perform a specific operation on an S3 object.

The backend generates the URL using its AWS credentials. Generating the signature is a local cryptographic operation; it does not require an S3 API request.

The basic flow looks like this:

Client          File Service       PostgreSQL          S3
  |                  |                  |               |
  |-- Initiate ----->|                  |               |
  |                  |-- Save metadata->|               |
  |                  |                  |               |
  |<-- Pre-signed URL|                  |               |
  |                  |                  |               |
  |-- Upload directly to S3 --------------------------->|
  |                  |                  |               |
  |<-- Upload response ---------------------------------|

The backend coordinates the upload, but the file bytes travel directly from the client to S3.

However, a single pre-signed URL for an entire file is not sufficient for our design. We also need delta sync, chunk reuse and immutable file versions.

Best: Upload Content-Addressed Chunks

Instead of storing each complete file as one S3 object, we split the file into smaller chunks and store those chunks independently.

We use Content-Defined Chunking (CDC) to determine chunk boundaries. A rolling fingerprint algorithm identifies content-dependent boundaries, while SHA-256 identifies the resulting chunk content.

The client creates an ordered manifest containing each chunk’s index, hash and size. It also calculates the complete file’s SHA-256 hash and includes the chunker version.

The upload process then follows these steps:

  1. The client sends the chunk manifest to the File Service through the upload-session API.
  2. The backend checks which chunks already exist in the owner’s storage scope.
  3. It reuses existing chunks and generates pre-signed PUT URLs only for missing chunks.
  4. The client uploads missing chunks directly to S3.
  5. Once the required chunks are available, the client calls the commit API to publish a new file version.

The backend does not receive or proxy the chunk bytes.

The key idea is that the application backend manages the upload lifecycle, while clients transfer the actual chunks directly to S3.

7.2 Download a File

The same principle applies to downloads.

Bad: Proxy the file through our application server

S3 sends the file to our backend, and our backend forwards it to the client.

Client             File Service                S3
  |                      |                      |
  |-- Download request ->|                      |
  |                      |-- GET file --------->|
  |                      |<-- File bytes -------|
  |<-- File bytes -------|                      |

This makes our application servers responsible for transferring the entire file.

Better: Download directly through a CDN/S3

The backend checks permissions and provides the client with authorized download URLs.

The client downloads the actual file data directly from the CDN/S3.

Client          File Service          CDN             S3
  |                  |                 |               |
  |-- Download ----->|                 |               |
  |                  |-- Check access  |               |
  |                  |-- Fetch metadata                |
  |                  |                 |               |
  |<-- Signed URLs --|                 |               |
  |                  |                 |               |
  |-- GET chunks --------------------->|               |
  |                  |                 |-- Cache miss->|
  |                  |                 |<-- Chunk -----|
  |<-------------------- Chunk data ---|               |

Best: Downloading Using Chunk Manifests

Our architecture does not store a complete file as one S3 object for every version. Instead, each immutable FileVersion contains an ordered mapping to the chunks required to reconstruct that version.

The download process follows these steps:

  1. The client requests a file download using GET /files/{fileId}/download.
  2. The File Service checks whether the authenticated user owns the file or has an active share permission.
  3. The backend resolves the requested version and provides access to its chunk manifest.
  4. The client checks which required chunks are already available in its local cache.
  5. It downloads missing chunks through the CDN/S3.
  6. The client verifies the chunks and reconstructs the original file using the manifest order.

For large files, the client can download multiple missing chunks in parallel, using bounded concurrency to avoid exhausting network and device resources.

7.3 Why Do We Need a CDN?

Imagine Alice shares a file with 10,000 users across different countries.

Without a CDN, downloads repeatedly reach the S3 origin. A CDN caches frequently requested chunks at edge locations closer to users, reducing latency and S3 traffic while handling popular downloads more efficiently.

Our content-addressed, immutable chunks are ideal for long-lived caching because their content never changes.

One caveat: revoking access prevents new authorizations but does not automatically invalidate already-issued credentials.

7.4 Share a File

Suppose Alice wants to share a file with Bob.

We need to answer two questions:

  1. Who has access to this file?
  2. Which files have been shared with a particular user?

Bad Approach: Keep a List of Shared Users Inside the File Record

Files
-----
file_id
owner_id
file_name
shared_users = [Bob, Charlie, David]

Now consider the query:

“Show me all files shared with Bob.”

We may need to scan a large number of file records.

Better Approach: Maintain a Separate Mapping

File Shares
-------------------------------------
recipient_id       file_id
-------------------------------------
Bob                File-101
Bob                File-102
Charlie            File-101
David              File-103

Now we can query using recipient_id.

SELECT file_id
FROM file_shares
WHERE recipient_id = :user_id
  AND revoked_at IS NULL;

We can also check whether a user has permission to access a particular file.

Download Request
       |
       v
 Check Ownership
       |
       +---- Owner? ---- Yes ----> Allow
       |
       No
       |
       v
 Check file_shares
       |
       +---- Valid VIEW/EDIT permission? ---- Yes ----> Allow
       |
       No
       |
       v
 Return 403 Forbidden

We are using the file_shares table as our Access Control List (ACL) mapping.

For our initial design, we will focus on direct file sharing. We will not support folder sharing for now.

7.5 Sync Files Across Devices

This is one of the most interesting parts of Dropbox.

Suppose a user has two devices:

  • Laptop A
  • Laptop B

The user edits a document on Laptop A. How does Laptop B learn about the changes and download the updated file?

We need to handle two directions:

  • Local device to cloud.
  • Cloud to other devices.

A. Local Device to Cloud

We need a background sync agent running in the desktop application.

The agent monitors the local file system for changes. Operating systems provide different mechanisms:

  • Windows: ReadDirectoryChangesW.
  • macOS: FSEvents.
  • Linux: inotify.

When a file changes, the operating system sends an event to our agent.

If a file changes multiple times within a short window, we wait for the changes to settle and enqueue one upload job.

            Local Folder
                  |
                  v
          +------------------+
          | File System      |
          | Watcher          |
          +--------+---------+
                   |
                   v
          +------------------+
          | Debounce /       |
          | Coalesce         |
          +--------+---------+
                   |
                   v
          +------------------+
          | Stable File      |
          | Check            |
          +--------+---------+
                   |
                   v
          +------------------+
          | SQLite Database  |
          | Persistent Jobs  |
          +--------+---------+
                   |
                   v
          +------------------+
          | Upload Workers   |
          +--------+---------+
                   |
            +------+------+
            |             |
            v             v
       File Service      S3
       Create Session  Direct PUT
       Get URLs        Missing Chunks
            |             ^
            |             |
            +------+------+
                   |
                   v
               Commit API

SQLite provides the sync agent with persistent local storage for upload jobs, chunk-level progress, file synchronization state, and the device’s last applied cloud change-feed cursor.

A local file fingerprint identifies the file snapshot and detects changes, while CDC’s rolling fingerprint determines chunk boundaries.

Why SQLite?

The upload queue must survive application restarts.

If we keep pending jobs only in memory, restarting the laptop may cause us to lose track of pending uploads.

SQLite provides a persistent local database for the sync agent.

Handling Missed Filesystem Events

Filesystem watchers are useful, but they cannot guarantee that every change will be detected.

Events can be duplicated, coalesced or lost due to buffer overflow or application downtime.

So we need periodic reconciliation.

File System Watcher
         |
         v
    SQLite Queue
         |
         v
    Upload Worker
         |
         v
       Cloud

-----------------------------------------------

Periodic Reconciliation
         |
         v
   Scan Local Folder
         |
         v
 Compare with Local DB
         |
         v
 Find Missed Changes
         |
         v
   Add Missing Jobs

The watcher provides fast change detection. Periodic reconciliation protects us against missed events.

The agent should also distinguish changes made by the user from changes applied by the sync agent itself. Otherwise, downloading a cloud update could trigger another upload and create a synchronization loop.

B. Cloud to Local Device

Now let’s understand how Laptop B learns about changes made on Laptop A.

We have two common approaches.

Polling

The client periodically asks the backend:

GET /sync/changes?cursor=12345

This is easy to implement, but polling too frequently creates unnecessary requests. Polling too slowly delays synchronization.

WebSockets or Server-Sent Events

The server maintains an open connection and sends notifications when something changes.

This provides faster updates, but connections can drop and notifications can be missed.

Recommended Approach: Hybrid

Use WebSockets or SSE as the fast path, and cursor-based polling as the recovery path.

                Laptop A
                    |
                    | Upload file
                    v
                File Service
                    |
                    v
               PostgreSQL
                    |
              +-----+-----+
              |           |
              v           v
          Change Feed  Outbox Event
                           |
                           v
                    Outbox Publisher
                           |
                           v
                     Message Queue
                           |
                           v
                 Notification Service
                           |
                           v
                    WebSocket / SSE
                           |
                           v
                       Laptop B
                           |
                           | Fetch changes
                           v
                       Sync API

When Laptop A commits a new file version, the backend writes the change-feed record and transactional outbox event in the same PostgreSQL transaction.

The Outbox Publisher publishes the event to the message queue.

The Notification Service consumes it and notifies connected devices through WebSocket or SSE.

The notification contains lightweight information such as the file ID, version ID and change sequence. It does not contain the actual file contents.

An important principle:

A push notification is only a signal. The durable change feed is the source of truth.

Even if Laptop B misses a notification, it can retrieve all pending changes from the durable feed.

Each device maintains its own cursor because different devices can have different synchronization progress.

When Laptop B receives a notification, it fetches changes after its last successfully applied cursor. It applies those changes locally and acknowledges its progress only after successfully applying them.

How Does Laptop B Download the Updated File?

When Laptop B receives a FILE_UPDATED event:

  1. It fetches the manifest for the specified immutable version.
  2. It checks which chunks are already available locally.
  3. It downloads missing chunks through the CDN/S3.
  4. It verifies the chunk hashes.
  5. It reconstructs the file in the correct order.
  6. It atomically replaces the local file.

The sync agent must also ensure that changes applied from the cloud do not trigger new upload jobs. If the local file has unsynchronized user edits, the agent must preserve that pending snapshot rather than blindly overwrite it with the incoming cloud version.

7.6 How Do Shared File Changes Reach Other Users?

There is an additional problem.

Alice owns a file and shares it with Bob and Charlie.

Alice modifies the file.

How do Bob and Charlie receive the change?

A file update must reach the change feeds of the owner and all users who currently have access to that file.

We have two approaches.

Fan-out on Write

When Alice updates the file, the system creates change-feed entries for the owner and all active recipients.

            Alice updates File-101
                       |
                       v
                  Change Event
                       |
              +--------+--------+
              |        |        |
              v        v        v
            Alice     Bob    Charlie
            Feed     Feed      Feed

Each user’s change feed contains a separate entry referencing the same file and version.

For example:

user_changes
-----------------------------------------
user_id   sequence   event_type
-----------------------------------------
Alice       501      FILE_UPDATED
Bob         421      FILE_UPDATED
Charlie     351      FILE_UPDATED

Each user has an independent, monotonically increasing sequence because different users can have different synchronization progress.

This makes recipient synchronization queries simple because each user can read changes from their own feed.

The outbox publisher publishes notifications through the message queue.

However, a file shared with thousands of users creates many database writes. This is known as write amplification.

For example, a file shared with 100,000 users could require up to 100,000 change-feed writes for a single update.

Fan-out on Read

Instead of writing a change to every recipient’s feed, store the resource-level change once.

When a user synchronizes, resolve changes from the resources they currently have access to.

            Alice updates File-101
                       |
                       v
                Resource Change
                     Feed
                       |
              +--------+--------+
              |                 |
              v                 v
             Bob             Charlie
            Query              Query
              |                 |
              v                 v
        Resolve Access    Resolve Access
              |                 |
              v                 v
         Return Changes   Return Changes

This reduces write amplification because the system stores the resource change only once.

However, sync queries become more complex because the backend must identify all relevant shared resources, check permissions and combine changes into a consistent response.

Cursor management also becomes more complicated because changes can come from multiple resource-level feeds rather than a single per-user feed.

Recommended Approach: Hybrid

For our initial design, use fan-out-on-write for direct file sharing.

  • When a file is updated, create change-feed entries for its owner and all active recipients.
  • When a file is shared, add a FILE_SHARED event to the recipient’s change feed so their devices discover the new file.
  • When access is revoked, add a FILE_ACCESS_REVOKED event to the affected recipient’s feed so their devices can remove the file from their synchronized view.

8. Supporting 50 GB Files

Supporting files up to 50 GB introduces additional challenges.

Two things are particularly important:

  • The user should see meaningful upload progress.
  • The upload should resume after network or application failure.

Our architecture already uses Content-Defined Chunking (CDC) and independently stored, content-addressed chunks.

This approach allows us to process and upload smaller chunks independently without sending the entire file through our application servers.

For illustration, assume an average chunk size of 8 MiB.

A 50 GiB file would produce approximately 6,400 chunks.

8.1 How Does Uploading a 50 GB File Work?

Let’s walk through the upload process for a large file.

Step 1: Split the file into chunks

The desktop sync agent reads the file incrementally and applies CDC.

It calculates the SHA-256 hash and size of every chunk. The result is an ordered manifest.

For example:

File: movie.mp4 (50 GiB)

+----------+----------+----------+----------+
| Chunk  1 | Chunk  2 | Chunk  3 | Chunk  4 |
|    h1    |    h2    |    h3    |    h4    |
+----------+----------+----------+----------+
                    ...
+----------+----------+----------+----------+
|Chunk 6397|Chunk 6398|Chunk 6399|Chunk 6400|
+----------+----------+----------+----------+

Step 2: Create an upload session

The client sends the manifest to the File Service:

POST /files/{fileId}/upload-sessions

The backend creates an UploadSession in PostgreSQL and checks which chunks already exist in the owner’s storage scope.

The backend reuses those chunks and generates pre-signed PUT URLs for the remaining missing chunks.

For a large manifest, upload instructions and pre-signed URLs can be returned in batches rather than one enormous response.

Step 3: Upload missing chunks

The client uploads missing chunks directly to S3.

Client                         S3
  |                             |
  |-- PUT chunk h1001 --------->|
  |                             |
  |-- PUT chunk h1002 --------->|
  |                             |
  |-- PUT chunk h1003 --------->|
  |                             |
  |          ...                |
  |                             |
  |-- PUT chunk h6400 --------->|
  |                             |
  |<-- Upload responses --------|

Each request uploads one independently stored content-addressed chunk.

Step 4: Commit the upload session

Once all required chunks are available, the client calls:

POST /upload-sessions/{uploadSessionId}/commit

The backend verifies that every chunk referenced by the manifest is available and passes the required integrity checks.

It then commits the new immutable file version.

8.2 How Do We Resume an Interrupted Upload?

Suppose the client uploads 60% of the chunks and then the laptop crashes.

We should not restart the entire 50 GB upload.

The desktop sync agent maintains upload progress in local SQLite.

For example, we can maintain the following tables:

sync_jobs

--------------------------------
job_id
file_id
local_path
file_fingerprint
upload_session_id
manifest_hash
upload_status
retry_count

sync_job_chunks

--------------------------------
job_id
chunk_index
chunk_hash
chunk_size
upload_status

Notice that we track chunk-level progress rather than S3 part numbers or ETags.

When the application restarts:

  1. It identifies unfinished upload jobs from SQLite.
  2. It checks that the original file is still available.
  3. It verifies that the file has not changed since the upload session was created.
  4. It calls the upload-session status API.
  5. The backend returns the status of the expected chunks.
  6. The client uploads only the missing chunks.
  7. Once all chunks are available, the client commits the session.

The status API is:

GET /upload-sessions/{uploadSessionId}

The client should not blindly trust its local SQLite state.

The backend reconciles the session against the actual S3 objects and their expected checksums.

If the original file has changed, the agent must not continue uploading an old snapshot as though it represented the new file.

Instead, it should calculate a new manifest and initiate a new upload session.

Previously uploaded chunks can still be reused if they appear in the new manifest.

The file_fingerprint helps the agent identify whether the original local file has changed. This is different from the Rabin rolling fingerprint used by CDC to determine chunk boundaries.

The important point is that we resume at the chunk level, not from the beginning of the entire file.

8.3 How Do We Track Upload Progress?

Since the file is divided into chunks, tracking progress becomes easier.

The client can maintain:

  • Total file size.
  • Total number of chunks.
  • Number of chunks already available in cloud storage.
  • Number of chunks successfully uploaded.
  • Number of chunks still pending.
  • Bytes uploaded and bytes remaining.

For example:

File: movie.mp4

Total Size:                50 GiB
Chunks Identified:          6,400
Chunks Already Available:   1,000
Chunks Uploaded:            3,000
Chunks Remaining:           2,400

Upload Progress:
[#################-------] 60%

8.4 How Do We Verify the Uploaded Chunks?

We cannot blindly trust the client to tell us that a chunk was uploaded successfully.

Our design uses two important values:

1. chunk_hash

This is the SHA-256 hash of the original chunk content.

It identifies the content and enables deduplication.

2. storage_checksum

This verifies the actual stored payload.

If we store the chunk without transforming it, the storage checksum can correspond to the original chunk content.

If compression or encryption is applied before storage, the checksum of the stored payload is different from the original content hash.

In that case, we maintain both values separately.

S3 supports checksum validation during object uploads. The backend can use the expected object size and stored checksum information to verify the uploaded chunks.

Why Can’t We Use ETag as Our Content Hash?

An ETag is not universally equivalent to SHA-256.

Its meaning depends on how the object was uploaded and stored.

We should use our own SHA-256 chunk_hash as the content identity and explicit checksums for storage integrity.

During download, the client verifies the original chunk hash after any required decryption or decompression.

It can also verify the complete file hash after reconstructing the file.

The backend commits the new file version only after all required chunks have been verified.

9. Upload Completion and Data Integrity

Uploading all chunks does not mean a file is ready for users.

Before publishing a file version, the backend must verify that every required chunk exists in S3 and its content is intact.

Our completion flow has three stages:

Client uploads chunks
        |
        v
  Verify integrity
        |
        v
  Commit file version
        |
        v
  File available

9.1 Verify Uploaded Chunks

Once the client finishes uploading, it calls:

POST /upload-sessions/{uploadSessionId}/commit

The File Service retrieves the expected chunk manifest and verifies every unique chunk.

For each chunk, it checks:

  • Does the S3 object exist?
  • Does its size match the expected size?
  • Does its checksum match the expected checksum?

We can use S3 HeadObject with checksum retrieval enabled.

There are two types of checksums to consider:

  • chunk_hash: SHA-256 hash of the original chunk content.
  • storage_checksum: Checksum of the actual stored payload.

9.2 Handle Integrity Failures

Consider a file containing 6,400 chunks, where one chunk is missing or has an invalid checksum.

We must not commit the file version.

Instead, the backend returns the affected chunks and moves the upload session back to UPLOADING.

       VERIFY CHUNKS
             |
       +-----+------+
       |            |
       v            v
   All valid    Missing/Invalid
       |            |
       v            v
    COMMIT      Return chunk IDs
                    |
                    v
               Re-upload
                    |
                    v
              Verify again

The client only needs to upload the affected chunks. It does not need to restart the entire upload.

9.3 Commit the File Version

Once all chunks pass verification, the backend starts a PostgreSQL transaction.

It performs the following operations:

  1. Creates an immutable FileVersion.
  2. Inserts the ordered FileVersionChunk mappings.
  3. Updates files.current_version_id.
  4. Creates change-feed and transactional outbox records.
  5. Marks the upload session as COMMITTED.
                 Verified Chunks
                       |
                       v
             PostgreSQL Transaction
                       |
          +------------+--------------+
          |            |              |
          v            v              v
       Create     Insert Chunk   Update Current
       File       Mappings       Version Pointer
       Version        |               |
          |           |               |
          +-----------+---------------+
                       |
                 +-----+------+
                 |            |
                 v            v
             Change Feed   Transactional
                              Outbox
                 |            |
                 +-----+------+
                       |
                       v
              Transaction Commit
                       |
                       v
              New Version Available

9.4 Handle Retries and Failed Commits

What happens if S3 uploads succeed but the PostgreSQL transaction fails?

The chunks remain in S3, but the file version is not published. The upload session remains retryable, so the backend can verify the existing chunks and try the commit again.

There is another important case: PostgreSQL commits successfully, but the client never receives the response.

To prevent duplicate versions, we store committed_version_id in the upload session.

      Commit Request
             |
             v
     PostgreSQL Commit
             |
             v
      Response Lost
             |
             v
       Client Retries
             |
             v
   Session Already Committed?
             |
             v
      Return Existing Version

The session status and committed_version_id are updated in the same transaction that creates the new version.

9.5 Clean Up Orphaned Chunks

An abandoned upload can leave chunks in S3 without a committed file version.

We cannot delete them immediately because another active upload session might still need them.

Before deleting a chunk, garbage collection checks whether it has:

  • A reference from a committed FileVersionChunk.
  • An active UploadSessionChunk reference or lease.
                S3 Chunk
                    |
                    v
             Check References
                    |
             +------+------+
             |             |
             v             v
       Committed       Active Upload
       Version Ref?    Session Ref?
             |             |
            Yes           Yes
             |             |
             v             v
          Keep Chunk    Keep Chunk

--------------------------------------------------  
        No references from either
                    |
                    v
          Retention Period Passed?
                    |
                    v
             Mark DELETING
                    |
                    v
               Delete S3
                    |
                    v
             Remove DB Record

Reference checks and deletion must be coordinated using PostgreSQL transactions and row-level locking. This prevents garbage collection from racing with an active upload or commit.

The key idea: S3 stores the chunks, but PostgreSQL decides when a complete, verified and immutable file version becomes available to users.

Deep Dives

1. Downloading Large Files

1.1 Manifest and Parallel Downloads

For a large file, the client fetches a paginated manifest using the same immutable versionId throughout the download:

GET /files/{fileId}/versions/{versionId}/manifest?pageToken=...

The manifest contains ordered chunk references, hashes, sizes and the complete-file checksum. The client reuses verified cached chunks and downloads missing chunks through a bounded worker pool.

File Manifest
                 |
        Identify Missing Chunks
                 |
       +---------+---------+
       |         |         |
    Worker 1  Worker 2  Worker 3
       |         |         |
    Chunk 1   Chunk 2   Chunk 3
       |         |         |
       +---------+---------+
                 |
         Local Chunk Cache

This avoids sequential downloads and excessive concurrent requests.

1.2 Resuming Interrupted Downloads

Track progress in local SQLite:

  • download_jobs: File, version, manifest hash, destination and overall status.
  • download_job_chunks: Chunk index, hash and download status.

After a restart, retrieve the same version’s manifest, reuse verified chunks, download missing chunks and reconstruct the file. Incomplete chunks can be downloaded again as CDC bounds their size.

1.3 Reconstruction and Verification

      Local Chunk Cache
               |
     Read in Manifest Order
               |
    Verify storage_checksum
               |
      Decrypt / Decompress
               |
       Verify chunk_hash
               |
    Stream into Temporary File
               |
     Verify content_sha256
               |
         Atomic Rename

Verify the stored representation, original chunks and complete file. Re-download invalid chunks. Stream into a temporary file to limit memory usage, and move it to the destination only after complete verification.

1.4 Logical Byte Ranges

An S3 Range request applies to one chunk object, not the complete logical file. Use the manifest and cumulative chunk sizes to identify overlapping chunks, decode them and extract the requested bytes.

Chunk 1     Chunk 2     Chunk 3     Chunk 4
0–8 MB      8–16 MB     16–24 MB    24–32 MB
               |           |
               +-----+-----+
                     |
           Requested: 10–20 MB
                     |
             Fetch Chunks 2, 3
                     |
               Extract Range

Download complete chunks initially. Add logical-range and partial-chunk retrieval later if streaming or random access requires them.

1.5 URL Expiry and Progress

CloudFront checks signed URL expiry when each request starts. An ongoing request can finish after expiry, but subsequent requests and retries fail. Generate URLs in batches and refresh them for pending chunks.

Calculate progress using verified bytes, including cached chunks, rather than chunk counts because CDC produces variable-sized chunks.

2. Reliable Synchronisation

The protocol must handle new devices, long offline periods, crashes and duplicate events without losing changes.

2.1 Initial Sync: New Device

Return metadata in pages:

GET /sync/initial?limit=1000
{
  "snapshotId": "snapshot-101",
  "files": [
    {
      "fileId": "file-101",
      "name": "document.pdf",
      "versionId": "version-105",
      "parentFolderId": "folder-10"
    }
  ],
  "nextPageToken": "token-2",
  "snapshotCursor": 50000
}

Fetch subsequent pages from the same snapshot:

GET /sync/initial?snapshotId=snapshot-101&pageToken=token-2

All pages must represent one point in time. Paginating live records can miss or duplicate files changed between requests.

          Consistent PostgreSQL Snapshot
                       |
             +---------+---------+
             |                   |
       Capture Feed       Capture Visible
          Cursor          Metadata + Permissions
             |                   |
             +---------+---------+
                       |
          Materialise Short-lived Snapshot
                       |
               Paginated Results
                       |
          Fetch Changes after Snapshot Cursor

Capture visible metadata, permissions and the feed cursor within one consistent database snapshot. Materialise the result so pagination does not keep a transaction open.

The snapshot and cursor must form a gap-free boundary: earlier changes appear in the snapshot; later changes are available after its cursor. Write metadata changes and corresponding feed records in the same PostgreSQL transaction.

After loading the snapshot, fetch changes made during pagination:

GET /sync/changes?cursor=50000

Retain the required feed history while the snapshot is active. Restart initial sync if the snapshot expires or its required history is unavailable.

Initial sync downloads metadata, not all content. Show files as Online Only and fetch chunks when opened, marked for offline access or prefetched.

2.2 Incremental Sync: Existing Device

The client stores its last applied cursor in SQLite and requests subsequent changes:

GET /sync/changes?cursor=59828
{
  "changes": [
    { "sequence": 59829, "eventType": "FILE_UPDATED", "fileId": "file-101", "versionId": "version-106" },
    { "sequence": 59830, "eventType": "FILE_CREATED", "fileId": "file-102" },
    { "sequence": 59831, "eventType": "FILE_DELETED", "fileId": "file-103" }
  ],
  "nextCursor": 59831,
  "hasMore": false
}

Client                                  Backend
  |                                        |
  |--- GET changes?cursor=59828 ---------->|
  |<-- Changes 59829–59831 ----------------|
  |                                        |
  | Apply changes and save cursor          |
  | atomically in SQLite                   |
  |                                        |
  |--- ACK cursor=59831 ------------------>|
  |<-- ACK successful ---------------------|

Advance the cursor only after applying the corresponding changes. Commit metadata updates and cursor advancement in one SQLite transaction. Make processing idempotent, and use repeatable filesystem operations, temporary files and atomic renames.

Each device progresses independently. Update devices.last_ack_cursor only after acknowledgement. Acknowledge only the highest contiguous applied position; parallel processing must not skip unfinished events. If processing stops after 59830, the saved cursor must not advance to 59831.

2.3 Expired Cursors and Full Re-sync

Reject cursors older than the retained history rather than silently skipping missing events:

HTTP/1.1 410 Gone
{
  "error": "CURSOR_EXPIRED",
  "message": "Change history is no longer available.",
  "action": "FULL_SYNC_REQUIRED"
}
        Cursor within Retention?
                  |
            +-----+-----+
            |           |
           Yes          No
            |           |
      Return Deltas   410 CURSOR_EXPIRED
                        |
                   Initial Sync
                        |
              Reconcile Local State

Reconcile the new snapshot with local state: add missing files, update changed versions and remove deleted or inaccessible files. Preserve unsynchronised edits as recovery copies for an explicit recovery or conflict workflow. Never silently overwrite them or automatically recreate cloud-deleted files.

Track the earliest supported cursor per user feed. Retention cleanup must preserve events required by unexpired snapshots.

2.4 Deletions and Tombstones

Record logical deletion in files.deleted_at and write a FILE_DELETED event to affected users’ feeds in the same transaction.

          Delete File
                |
       +--------+---------+
       |                  |
Set files.deleted_at   Write FILE_DELETED
       |                  |
       +--------+---------+
          Same Transaction
                |
         Durable Change Feed
                |
        Other Devices Remove File

The tombstone records deletion; the event informs devices. Together, they prevent stale offline copies from being uploaded as new files. For shared files, notify users who had access. Use a separate event, such as FILE_ACCESS_REVOKED, when only access is removed.

Coordinate tombstone cleanup with feed retention and device acknowledgements. After history is purged, older cursors require full reconciliation. Logical deletion does not immediately remove S3 chunks; reference-based garbage collection handles them separately (Section 4.5).

3. Concurrent File Edits

3.1 Last Write Wins

Use Last Write Wins (LWW): the latest successful server-side commit becomes current. Do not rely on client timestamps.

Clients upload chunks independently without locking the file. Serialise only the final commit using a PostgreSQL row-level lock:

SELECT *
FROM files
WHERE file_id = :file_id
FOR UPDATE;

Within the transaction:

  1. Lock the file and assign the next version number.
  2. Create an immutable FileVersion and ordered FileVersionChunk mappings.
  3. Update files.current_version_id.
  4. Write change-feed records and the transactional outbox event.
  5. Mark the upload session as committed.

Other commits for the same file wait until the transaction finishes.

3.2 Concurrent and Stale Updates

Suppose Alice and Bob both edit Version 10:

                Version 10
                /        \
          Alice edits   Bob edits
               |            |
        Upload Session A  Upload Session B
               |            |
               +------+-----+
                      |
              Serialised Commits
                      |
            Version 11: Alice
                      |
            Version 12: Bob (Current)

If Alice commits first, she creates Version 11. Bob’s later commit creates Version 12 containing his snapshot, not a merge of both edits. Alice’s version remains recoverable under the version-retention policy.

Do not reject stale updates through row_version, expectedVersion or HTTP 409 checks. Both uploads can succeed if validation passes. Version numbers reflect commit order, not when editing started.

3.3 Restoring Earlier Versions

Restore by creating a new immutable version that reuses historical chunks. Do not modify history or move the current pointer backwards.

Version 10
    |
Version 11: Alice’s changes
    |
Version 12: Bob’s changes
    |
Version 13: Restore Version 11 (Current)

3.4 Unsynchronised Local Changes

Preserve pending local content before applying an incoming remote version. If the client later commits its local snapshot, LWW applies again. This preserves unsynchronised work but does not merge concurrent edits.

4. Content-Defined Chunking

4.1 Why CDC?

Fixed-size boundaries shift after an insertion, changing hashes for otherwise unchanged content. Content-Defined Chunking (CDC) uses content-based boundaries that can resynchronise after an edit, reducing re-uploads.

Original:                 [ A  ][ B  ][ C  ][ D  ]
Fixed-size after insert:  [ A' ][ B' ][ C' ][ D' ]
CDC after local edit:     [ A  ][ B' ][ C  ][ D  ]

The CDC example is illustrative; boundary preservation is not guaranteed for every edit.

4.2 Boundaries and Chunk Identity

Use a rolling hash, such as a Rabin fingerprint, to find boundaries. Configure minimum, target and maximum sizes to avoid very small or large chunks. An illustrative target is 8 MiB; tune the limits to the workload.

After chunking, calculate SHA-256 for each chunk. The rolling hash finds boundaries; SHA-256 identifies content.

          Original File
                 |
      CDC: Rabin Fingerprint
                 |
       +---------+---------+
       |         |         |
    Chunk A   Chunk B   Chunk C
       |         |         |
    SHA-256   SHA-256   SHA-256
       |         |         |
       h1        h2        h3
       |         |         |
       +---------+---------+
                 |
          Ordered Manifest

The manifest records chunk index, hash, size and chunker version identifying the algorithm and parameters. At upload-session creation, the File Service checks existing chunks and requests only missing ones.

Scope deduplication to the owner or tenant, including reuse across files and versions. Do not expose whether another user stores matching content.

4.3 Independent Chunk Storage

Store each chunk as an S3 object, for example {owner_id}/{chunk_hash}. PostgreSQL maintains version metadata and ordered FileVersionChunk mappings.

            File Version
                  |
          Ordered Manifest
                  |
        +---------+---------+
        |         |         |
     Chunk 1   Chunk 2   Chunk 3
        |         |         |
     S3 Object S3 Object S3 Object
        |         |         |
        +---------+---------+
                  |
         Client Reconstruction

Versions reuse unchanged chunks:

Version 1: [h1, h2, h3, h4]
Version 2: [h1, h2, h5, h4]
Version 3: [h1, h2, h5, h6]

4.4 CDC vs S3 Multipart Upload

Chunk versioning and block-level deduplication against object storage
Content-Defined Chunking and Block Deduplication

Content-Defined Chunking and Block Deduplication

Use regular pre-signed S3 PUT requests for missing chunks, not CreateMultipartUpload, UploadPart and CompleteMultipartUpload as the primary mechanism.

4.5 Reference-Based Garbage Collection

Delete a chunk only when no retained file version or active upload session references it. Deleting one version must not remove chunks still required elsewhere.

5. Merkle Trees: Future Optimisation

A Merkle tree allows chunk-membership verification against a trusted root without downloading all other chunks.

              Merkle Root
              /         \
          Hash AB      Hash CD
           /   \        /   \
         H(A) H(B)    H(C) H(D)
          |    |       |    |
        CH-A CH-B     CH-C CH-D

Each parent hashes its children. To verify Chunk B, combine its hash with H(A) and Hash CD in the correct order, then compare the calculated root with the file version’s authenticated root.

The root must come from authenticated metadata. Retain tree nodes or sufficient hash information to generate proofs; the root alone is insufficient.

Merkle trees are not required initially. Chunk SHA-256 hashes, the complete-file checksum and manifest hash already meet current integrity needs. Add Merkle trees only for independent membership proofs, partial-manifest verification or untrusted-storage proofs that justify their computation, storage and proof-generation cost.

6. Performance Optimisations

6.1 CDN Caching

Cache immutable, content-addressed chunks for long periods. New versions reuse unchanged cache entries without invalidation. Use shorter caching for mutable metadata and current-version information.

6.2 Selective Compression

Compress chunks independently so downloads, decoding and retries remain chunk-local. Text, JSON and CSV usually benefit; already-compressed images, audio, video and archives often do not.

Retain encoding metadata. chunk_hash identifies original content; storage_checksum validates the stored representation. If application-level encryption is used, compress before encrypting. Skip compression when savings are negligible.

6.3 Adaptive Parallelism

Adjust bounded upload/download concurrency using throughput, latency, failures, retries and device resources. Excessive concurrency can overwhelm the connection or client without improving throughput.

Keep CDC parameters and chunker version consistent within an upload. Adapt worker count, not chunk boundaries.

6.4 LAN and Peer-to-Peer Sync

Nearby devices could exchange verified chunks directly, with cloud fallback:

Laptop A ── Direct LAN Transfer ──> Laptop B
                                      |
                                Cloud Fallback
                                      |
                                     CDN
                                      |
                                      S3

This can reduce egress and improve speed but requires peer discovery, authentication, encryption, NAT/firewall handling and secure fallback. Verify received chunks against the authenticated manifest. Keep peer-to-peer sync out of the initial scope.

7. Keeping Files Secure

7.1 Encryption in Transit

Use HTTPS/TLS for client-to-API, direct S3 uploads, CDN downloads and internal service communication. Never transfer file content or credentials over unencrypted connections.

7.2 Encryption at Rest

  • S3 encrypts new objects by default using SSE-S3. Use SSE-KMS for additional key control, permissions and auditing; KMS manages keys while S3 encrypts and decrypts objects.
  • Encrypt PostgreSQL storage, backups and snapshots containing metadata, permissions, versions, manifests and object references.

Use server-side encryption initially.

References: S3 server-side encryption, SSE-KMS.

7.3 Access Control and Signed URLs

Keep S3 private. Never provide clients with AWS credentials or unrestricted bucket access. Authorise each operation before issuing temporary object-specific URLs:

  • Upload: Issue a pre-signed S3 PUT URL restricted to the object and method. Bind the expected checksum where supported.
  • Download: Check ownership or active sharing permissions, then issue a signed CloudFront URL or appropriately scoped cookie. Validate authorisation before cache hits and origin fetches. Restrict S3 origin access using CloudFront OAC.

Signed URLs are bearer credentials. Use HTTPS, short expiry and narrow scope. Do not expose full URLs in logs, analytics or errors.

Revoking database permissions does not invalidate issued URLs; they can remain usable until expiry or signing credentials/policies are invalidated. Near-immediate revocation requires current-permission checks at the edge or on each request, adding complexity and potential latency. Handle expiry during large downloads as described in Section 1.5.

References: S3 pre-signed URLs, CloudFront signed URLs.

7.4 Malware Scanning

Malware scanning is outside the initial scope. If introduced, reconstruct the original content as a stream in manifest order, after required decryption or decompression. Scanning chunks separately can miss malicious patterns spanning boundaries.

        Chunks Uploaded
                |
      Upload Manifest Verified
                |
        Malware Scanner Worker
                |
       Read Chunks in Order
                |
    Decode and Reconstruct Stream
                |
           Scan Content
                |
          +-----+-----+
          |           |
        Clean      Malicious
          |           |
    Allow Version  Quarantine
       Commit        Upload

Large-file scanning can run asynchronously, but downloads and sharing must remain blocked until it passes. Add a scan-status gate to the upload lifecycle when this feature is introduced.

Production Considerations

A design that works on paper can face different challenges at production scale. Here are some areas to consider:

  • Storage Costs: S3 storage and egress can become expensive. Consider Cloudflare R2, which offers zero egress fees, along with CDN caching, deduplication and lifecycle policies.
  • Database Scaling: Start with PostgreSQL, proper indexing and read replicas. Introduce partitioning or owner-based sharding when actual load requires it.
  • Rate Limiting: Enforce per-account quotas, limit active upload sessions and use bounded transfer concurrency. Control direct S3 access through scoped, short-lived pre-signed URLs.
  • High Availability: Use automated backups, database replication and a defined disaster-recovery plan with clear RPO and RTO targets.
  • Observability: Monitor upload failures, download retries, synchronization lag, queue health and garbage-collection backlog. Use distributed tracing to investigate failures across services.

The goal is to keep the initial architecture simple while having a clear path to address these challenges as the system grows.

Final Architecture

Let’s put everything together.

Actual file contents are transferred directly between clients and S3/CDN.

Complete Architecture

                         +-------------------------+
                         |         Clients         |
                         |   Web / Desktop / App   |
                         +------------+------------+
                                      |
                                      v
                         +-------------------------+
                         |       DNS / WAF         |
                         +------------+------------+
                                      |
                                      v
                         +-------------------------+
                         |       External LB       |
                         +------------+------------+
                                      |
                    +-----------------+-----------------+
                    |                                   |
                    v                                   v
          +---------------------+           +---------------------+
          |     API Gateway     |           |   Realtime Gateway  |
          |  Auth / Rate Limit  |           |    WebSocket / SSE  |
          +----------+----------+           +----------^----------+
                     |                                 |
          +----------+----------+                      |
          |          |          |                      |
          v          v          v                      |
      +--------+ +--------+ +--------+                 |
      |  File  | |  Sync  | | Share  |                 |
      | Service| | Service| | Service|                 |
      +---+----+ +---+----+ +---+----+                 |
          |          |          |                      |
          +----------+----------+                      |
                     |                                 |
                     v                                 |
          +---------------------------+                |
          |        PostgreSQL         |                |
          |                           |                |
          | File / Folder Metadata    |                |
          | File Versions             |                |
          | Ordered Chunk Manifests   |                |
          | Chunk Metadata            |                |
          | Upload Sessions           |                |
          | Shares / Permissions      |                |
          | Durable Change Feed       |                |
          | Device Sync Cursors       |                |
          | Transactional Outbox      |                |
          +-------------+-------------+                |
                        |                              |
                        | Outbox Publisher polls       |
                        v                              |
          +---------------------------+                |
          |     Outbox Publisher      |                |
          +-------------+-------------+                |
                        |                              |
                        v                              |
          +---------------------------+                |
          |       Message Queue       |                |
          |        SQS / Kafka        |                |
          +-------------+-------------+                |
                        |                              |
                        v                              |
          +---------------------------+                |
          |   Notification Service    |----------------+
          +---------------------------+

---------------------------------------------------------------------
                    DIRECT FILE DATA TRANSFER
                    -------------------------
    UPLOAD
    Client -------------------------------------> S3
           Pre-signed PUT URLs
           Individual content-addressed chunks
    DOWNLOAD
    Client <---------------- CDN <--------------> S3
                   Immutable chunk delivery
                    CLIENT-SIDE COMPONENTS
                    ----------------------
                Local File System
                       |
                       v
                File System Watcher
                       |
                       v
                Debounce / Coalesce
                       |
                       v
                 Stable File Check
                       |
                       v
                  SQLite Database
              (Persistent Jobs / State)
                       |
                       v
                  Sync Agent
                       |
             +---------+---------+
             |                   |
             v                   v
        Upload Workers       Sync Workers
             |                   |
       +-----+-----+       +-----+------+
       |           |       |            |
       v           v       v            v
   File Service   S3   Sync Service    CDN
   Sessions /    Direct  Cursors /   Chunk
   Commit        PUT     Changes     Downloads
       Periodic Local Filesystem Reconciliation

The architecture has three important paths:

  1. Control plane: The backend handles authentication, authorization, metadata, upload coordination, version commits and synchronization.
  2. Data plane: Clients upload chunks directly to S3 and download them through the CDN.
  3. Synchronization path: PostgreSQL maintains durable changes, while the transactional outbox, message queue and notification service deliver real-time notifications.

19. Conclusion

Building a Dropbox-like system is more than integrating S3 with a file-upload API. The real challenge is maintaining data integrity, handling concurrent changes and keeping multiple devices synchronized despite failures.

The architecture starts with simple, reliable building blocks: PostgreSQL, S3, Content-Defined Chunking and a durable change feed. More advanced optimizations can be introduced as the system grows.

The priority is to get correctness and reliability right before optimizing for scale.

If you found this article helpful, do leave a clap 👏 and follow for more insights on GenAI, AI Agents, LLM, RAG, System Desing and backend engineering!

Discussion

Have a question, a correction, or a different approach? Sign in with GitHub to join the conversation.

100%