System Design

Building Reliable Upload Systems

Chunked uploads, progress tracking, retry logic, and why the naive approach fails at scale.

2026-06-05·10 min·intermediate

#upload#architecture#node#storage

326 words · 100 lines

Xonoxc

Problem

Building an upload system seems trivial: accept a file, save it to disk/S3. The complexity appears when you need to handle:

Network interruptions
Large files (2GB+)
Concurrent uploads
Progress tracking
Virus scanning
Multipart encoding
Rate limiting

Naive Approach

The simplest version:

code

app.post("/upload", async (req, res) => {
  const file = req.files?.file
  await fs.promises.writeFile(`/uploads/${file.name}`, file.data)
  res.json({ 
    success: true 
  })
})

This fails for any file over 50MB, blocks the event loop, and offers zero resilience.

Architecture

Client → Chunk Splitter → Upload API → Chunk Assembler → Scan Queue → Storage
           ↓                                                             ↑
      Progress Tracker                                            CDN Invalidator
           ↓
      Client Polling / WebSocket

Problems Encountered

Chunk Boundary Alignment

Progress Tracking Accuracy

Key Insight

Progress should be calculated at the chunk level, not the byte level:

function calculateProgress(upload: Upload): number {
  const completedBytes = upload.completedChunks
    .reduce(
        (sum, c) => sum + c.size,
        0
    )

  return completedBytes / upload.totalSize * 100
}

Simple average based on chunks completed, weighted by chunk size.

Tradeoffs

Tradeoffs.md

Strategy	Reliability	Latency	Storage	Complexity
Single POST	Low	Low	Temp	Minimal
Chunked + Retry	High	Medium	Temp	High
Streaming	Medium	Low	None	Medium
Resumable (TUS)	Very High	Medium	Temp	Very High

Final Mental Model

Reliable uploads are state machines disguised as file transfers.