System Design

Building Reliable Upload Systems

Chunked uploads, progress tracking, retry logic, and why the naive approach fails at scale.

2026-06-0510 minintermediate

#upload#architecture#node#storage

EST CONFIG|No response

326 words · 100 lines

Xonoxc⚏

Problem

Building an upload system seems trivial: accept a file, save it to disk/S3. The complexity appears when you need to handle:

Network interruptions
Large files (2GB+)
Concurrent uploads
Progress tracking
Virus scanning
Multipart encoding
Rate limiting

Naive Approach

The simplest version:

app.post("/upload", async (req, res) => {
  const file = req.files?.file
  await fs.promises.writeFile(`/uploads/${file.name}`, file.data)
  res.json({ 
    success: true 
  })
})

This fails for any file over 50MB, blocks the event loop, and offers zero resilience.

Architecture

Client → Chunk Splitter → Upload API → Chunk Assembler → Scan Queue → Storage
           ↓                                                             ↑
      Progress Tracker                                            CDN Invalidator
           ↓
      Client Polling / WebSocket

Problems Encountered

Chunk Boundary Alignment

[!] Pitfall

I naively split files into fixed 5MB chunks. This broke because:

S3 multipart upload has a 10000 part limit
Some chunks were too small (S3 minimum is 5MB except the last)
Retrying a chunk meant re-uploading arbitrary byte ranges

The fix was to use adaptive chunk sizing. Start with 5MB, adjust based on file size to stay under 10000 parts.

Progress Tracking Accuracy

> Key Insight

Progress should be calculated at the chunk level, not the byte level:

function calculateProgress(upload: Upload): number {
  const completedBytes = upload.completedChunks
    .reduce(
        (sum, c) => sum + c.size,
        0
    )

  return completedBytes / upload.totalSize * 100
}

Simple average based on chunks completed, weighted by chunk size.

Tradeoffs

[ Tradedoffs.md ]

Strategy	Reliability	Latency	Storage	Complexity
Single POST	Low	Low	Temp	Minimal
Chunked + Retry	High	Medium	Temp	High
Streaming	Medium	Low	None	Medium
Resumable (TUS)	Very High	Medium	Temp	Very High

Final Mental Model

Reliable uploads are state machines disguised as file transfers.