File Sync System Design: How Dropbox-Style Sync Works

You save a 300 MB presentation on your laptop, and a minute later it opens on your phone. Then you fix a typo on one slide, and the update takes a second, because only the changed part travels. A file sync service makes this look easy, but underneath it is a careful split between two very different kinds of data, plus a protocol that tolerates flaky networks, offline edits and devices that wake up days later.

This post designs a Dropbox-style sync service: requirements and numbers, the split between metadata and content, chunking and content hashes, the upload and commit protocol, the metadata journal and cursors, notifications, conflicts and garbage collection, with an architecture diagram.

Requirements

• Files saved on one device appear on the user's other devices within seconds to minutes.
• Only changed data uploads and downloads.
• Edits made offline sync when the device reconnects, and concurrent edits never silently overwrite each other.
• Shared folders work across accounts.
• No data loss, ever; version history for recovery.

Capacity

Assume 10 million active devices, each syncing 20 changes a day, with an average of 256 KB of new data per change after deduplication:

The metadata rate is modest; the data volume is large but simple to store. That asymmetry drives the design.

Split metadata from content

The most important decision is to treat file contents and file metadata as two different systems:

• Content is stored as immutable blocks, addressed by the hash of their bytes. Blocks are large, write-once and easy to replicate in object storage.
• Metadata records which file at which path, in which version, consists of which blocks. It is small, changes constantly, and needs transactions and strict ordering.

Dropbox described this split in its streaming sync post: every file is partitioned into 4 MB blocks, each block is hashed with SHA-256, and a file's contents are identified by the list of those hashes, its block list. The public API still exposes the same idea as a content hash: SHA-256 of each 4 MB block, then SHA-256 of the concatenated block hashes.

Chunking: fixed or content-defined

Fixed-size blocks are simple and fast to compute. Their weakness shows when bytes are inserted near the start of a file: every block boundary after the insertion shifts, and every block hash changes.

Content-defined chunking places boundaries where a rolling hash over the content matches a pattern, as rsync's rolling checksum and newer algorithms such as FastCDC do. Boundaries move with the content, so an insertion changes only the chunks around it.

For files that are mostly appended to or replaced wholesale, fixed blocks are fine. For files edited in the middle, such as large documents, disk images or databases, content-defined chunking saves far more bandwidth.

The upload and commit protocol

1. The desktop client notices a change through the operating system's file watcher, waits until the file stops changing, then chunks and hashes it.
2. It sends a commit to the metadata service: this path, this new block list, based on parent version N.
3. The metadata service checks which hashes it already stores and answers with the ones it needs.
4. The client uploads only those blocks to the block service, in parallel, each verified by its hash. An interrupted upload resumes from the missing blocks.
5. The client retries the commit. Now every block exists, so the metadata service appends a new version of the file to the journal.

The order is the safety property: a version that references blocks is never committed until all of them are stored. A crash at any point leaves either the old version or the new one, never a file pointing at missing data.

Content addressing also gives deduplication for free. A file copied into three folders, or reverted to an earlier version, uploads nothing new. Cross-account deduplication saves more space, but it can reveal that someone else stored the same file, so many designs deduplicate only within an account or a shared folder.

The metadata journal and cursors

Group files into namespaces, such as a user's own folder tree or one shared folder. For each namespace, the metadata service keeps an append-only journal: every change gets the next sequence number, with the path, version and block list.

Each client stores a cursor per namespace: the last sequence number it has applied. Syncing down is then a single question: "give me everything in this namespace after my cursor." The journal makes this cheap, ordered and restartable. A device that was offline for a week asks the same question and gets a week of changes. Shard the metadata by namespace, since changes in one namespace never need a transaction with another.

Notifications: telling devices to sync

Polling the journal from every idle device would be wasteful. Instead, idle desktop clients hold a long-poll connection to a notification service. When a namespace's journal advances, the service answers the waiting requests with "changes available"; each client then lists changes after its cursor and downloads the blocks it does not have. Dropbox's public API offers the same pattern as list_folder/longpoll and list_folder/continue.

Mobile apps cannot hold connections open in the background, so they rely on platform push notifications and sync when the app opens.

The client side

The desktop client is half of the system. It keeps a local database of the last synced state of every file: path, version, block list, size and modification time. Comparing three states decides each action:

A local block cache avoids downloading blocks the device already holds in other files.

Conflicts

Two devices edit the same file while offline. Both later commit with parent version N. The metadata service accepts the first commit, creating version N+1, and rejects the second because its parent is no longer current. Instead of merging binary files it cannot understand, the losing client saves its version next to the original as a conflicted copy and syncs both. The user decides.

This is optimistic concurrency: every commit names the version it was based on. The same check handles directory operations, which are harder in practice: a rename on one device and an edit on another, or a delete racing an edit. Write these cases down explicitly; they are where sync engines lose data.

Version history and garbage collection

Old versions keep their block lists, so restoring a file is just committing an old block list as the newest version. Blocks become garbage only when no version within the retention period references them. Run garbage collection as mark-and-sweep with a grace period, never as an eager delete on the last reference, because a client may be about to commit a block list that reuses a block the server thinks is orphaned.

The file sync diagram

How to read it:

• The laptop on the left commits a block list to the metadata service and gets back the hashes the server is missing.
• Block uploads go to the block service and land in object storage, keyed by hash.
• The metadata service appends the new version to the namespace journal in Postgres and publishes the change.
• The notification service answers the phone's long poll; the phone lists changes after its cursor and downloads only missing blocks.
• The Markdown block lists the six protocol steps in order.

Drawing sync systems clearly

• Draw metadata and content as separate paths. Different stores, different scaling, different failure modes.
• Number the protocol steps on the edges. Commit, missing hashes, upload, commit again: the order is the correctness argument.
• Show the second device. Sync is about the reader as much as the writer; a diagram with one client hides notifications and cursors.
• Label the partition key: namespace for metadata, hash for blocks.

For a general approach, see the system design diagram example. The push side resembles the notification system design, chunked uploads to object storage also appear in the video streaming architecture diagram, and partitioning metadata follows the database sharding diagram.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add two Device nodes from the Network pack: a laptop and a phone.
3. Add a Core Frame for the sync backend with an API Gateway from the Cloud pack at its edge.
4. Inside the frame, add three Compute nodes from the Cloud pack for the metadata, block and notification services, a Postgres node from the Data pack for the journal, an Object Storage node from the Cloud pack for blocks and a Kafka node from the Data pack for journal events.
5. Number the edges in protocol order: commit, missing hashes, upload, commit again, notify, list and download.
6. Add a Markdown block from the Core pack with the protocol and capacity.

The example uses 11 diagram nodes, within the Free plan.

FAQ

Why do sync services split files into blocks? Blocks let the client upload only what changed, resume interrupted transfers and deduplicate identical data. A block's hash is its identity, so the server can tell immediately which blocks it already has.

Should I use fixed-size or content-defined chunking? Fixed-size blocks are simpler and cheaper to compute. Content-defined chunking keeps chunk boundaries stable when bytes are inserted in the middle of a file, which saves bandwidth for files edited in place.

How does a client know what changed on the server? It keeps a cursor per namespace and asks the metadata service for every journal entry after it. A long-poll or push notification tells it when to ask.

How are conflicting edits handled? Each commit names the version it was based on. If another device committed first, the commit fails, and the client keeps its edit as a conflicted copy instead of overwriting the other one.

Does deduplication work across users? It can, but it may reveal that another account stored the same file, and it does not work with per-user encryption. Many services limit deduplication to one account or shared folder.

Start free — one seat, no credit card.