Block, File and Object Storage

1The core idea#

There are three ways to hand bytes to an application. They differ in exactly one thing that drives everything else: what the storage system knows about your data.

Block storage: a raw disk.

• Gives you fixed-size blocks, usually 512 bytes or 4 KB. The layer knows nothing about files or structure.
• Exposes blocks over a protocol like iSCSI or NVMe. Your OS puts a filesystem on top.
• This is what a hard disk or an AWS EBS volume looks like to a server: one contiguous device you format and mount.
• Lowest latency, highest control, and exactly one machine attaches it at a time (mostly).

File storage: a shared folder.

• Adds a hierarchy: directories, files, paths, permissions, and a lock manager.
• You talk to it over NFS or SMB. Many clients mount the same share and see the same tree.
• The server tracks metadata: who owns /reports/q3.xlsx, when it changed, who has it open.
• That shared-tree convenience is also the scaling ceiling, because the metadata server becomes a bottleneck.

Object storage: a key-value store for big blobs.

• Throws away the hierarchy. You get a flat namespace of buckets and objects.
• Each object has a key, the blob of data, and arbitrary metadata, reached over HTTP with GET and PUT.
• No partial in-place edits: to change one byte you re-upload the whole object.
• In return: near-infinite scale, eleven nines of durability, and a price near zero. S3 is the canonical example.

2How it works#

The deepest difference is the access unit and where the intelligence lives.

• Block. The client sends "read logical block 48213." The device returns 4 KB. The filesystem logic (inodes, directories, journaling) runs on your server. The disk is dumb on purpose.
• File. The client sends "open /a/b/c.txt, read 100 bytes at offset 2000." The server resolves the path, checks the lock, and returns the bytes. The intelligence moved into the file server.
• Object. The client sends GET /bucket/key. The server returns the whole object, or a byte range if you ask. Metadata, replication, and placement are the server's problem, not yours.

A worked example: can object storage back my video thumbnails?#

Say I run a video app serving 10,000 thumbnail reads per second, each thumbnail 40 KB. I need to decide between EBS (block, behind my own service) and S3 (object).

Block path (EBS gp3).

• A gp3 volume gives a baseline 3,000 IOPS and 125 MB/s, provisionable up to 16,000 IOPS. At 10,000 reads/sec I must provision ~10,000 IOPS.
• Throughput check: 10,000 reads x 40 KB = 400 MB/s, well past the 125 MB/s baseline, so I provision throughput toward 1,000 MB/s.
• Cost at current gp3 rates: ~$0.005 per provisioned IOPS-month above 3,000, so ~7,000 extra IOPS is about $35. Plus ~$0.04 per provisioned MB/s-month above 125, so ~875 extra MB/s is about $35. Plus the volume itself.
• Catch: this serves one availability zone from one attached instance. To scale reads I now need a read-replica fleet or a cache in front. I own the filesystem, the failover, and the backups.

Object path (S3).

• S3 charges per request plus per GB stored and transferred. 10,000 GET/sec = 36M requests/hour.
• At $0.0004 per 1,000 GETs that is ~$14.40/hour of pure request cost. That is high, so in practice I put CloudFront in front.
• The CDN serves 95%+ of thumbnails from the edge cache, so S3 sees maybe 500 origin GETs/sec. Request cost drops ~20x to roughly $0.72/hour.
• Storage is $0.023/GB-month, durability is eleven nines, ops work is zero. I attach it from every instance in every region at once.

The verdict.

• The math pushes me to object storage plus a CDN for this read-heavy, immutable, globally served workload.
• I would only choose block if I needed in-place mutation or sub-millisecond latency per read, which thumbnails do not.
• The deciding factor was not cost alone. It was that thumbnails are immutable and read far more than written, which is exactly where object storage wins.

Where each sits#

3Trade-offs#

Here is the comparison I actually sketch on the whiteboard.

The real decisions behind the table:

Latency vs scale: you pick one per layer.

• Block wins latency because there is no path resolution, no HTTP, no metadata lookup: just "give me block 48213."
• Object wins scale because there is no shared mutable tree to coordinate across nodes.
• You cannot get both from one primitive, so you layer: database on block, cold data on object.

Mutability is the sharpest line.

• If you need to change bytes in place, object storage is simply the wrong tool.
• A Postgres data file gets written constantly at specific offsets, say a 4 KB page update deep inside a 2 GB file.
• Putting that on S3 would mean re-uploading the whole object per write.
• This is why databases live on block.

Cost drives the lifecycle.

• Object is 5 to 10x cheaper per GB than block: roughly $0.023/GB-month for S3 Standard versus $0.08 to $0.10/GB-month for gp3.
• So the pattern is: hot, mutable, latency-sensitive data on block; large, immutable, infrequently accessed data on object.
• Then add lifecycle rules, for example move to S3 Infrequent-Access after 30 days and Glacier after 90.

4Where it is used#

• Block. Every production database you have touched: RDS and self-managed Postgres or MySQL sit on EBS. Boot volumes for EC2 instances. Kafka brokers often use block for the commit log because of the write latency. Reach for block when a single process owns the data and mutates it in place at specific offsets.

• File. Shared home directories, media teams passing large files, legacy apps that assume a POSIX filesystem and cannot be rewritten to speak HTTP. AWS EFS, Azure Files, on-prem NetApp. Reach for file when multiple machines must see the same mutable tree and you cannot change the app to use object APIs. I treat this as the "I have no choice" option more than the "I would design it this way" option.

• Object. S3, Google Cloud Storage, Azure Blob, Cloudflare R2, MinIO on-prem. User uploads, images and video, data-lake files (Parquet), ML training sets, logs, backups, static website assets, Terraform state. Reach for object for anything large, write-once-read-many, and served to the world. It is the default storage layer for modern cloud apps, and most of your bytes end up here.

The mature answer layers all three.

• Netflix-style video: master files and encoded renditions live in object storage.
• The CDN pulls from object into edge caches.
• The microservices' databases (user, catalog, billing) run on block.
• If any internal tool needs a shared POSIX mount, it uses file.
• One company, all three, each for its strength. That layering is the mature answer, not picking a single winner.

The senior answer is never "which one." It is "which one for this layer, and why."

5Common pitfalls#

• Putting a database on object storage. The classic mistake. Object has no in-place writes and tens-of-ms latency; your DB needs both. The exception is purpose-built engines like data warehouses that query Parquet on S3 directly, but that is OLAP scan workload, not OLTP random writes.

• Treating file storage as infinitely scalable. The metadata server and shared lock manager are a real ceiling. Thousands of clients hammering one NFS share will serialize on metadata. Object storage has no such single tree to coordinate, which is exactly why it scales.

• Forgetting object storage consistency quirks. S3 gives strong read-after-write for new objects now, but listing and some cross-region behaviors can lag. Do not build a locking or coordination scheme on top of a bucket.

• Ignoring request cost at scale. The worked example showed 10,000 GET/sec costing ~$14/hour in pure request charges. People size for storage cost and get surprised by request cost. A CDN or cache in front is not optional at high read rates.

• Re-uploading whole objects to change metadata. You can often update object metadata, or use multipart and range reads, instead of rewriting the full blob. Know the API before you assume it is all-or-nothing.

• Mounting block storage to two instances expecting shared writes. A standard EBS volume attaches to one instance. Multi-attach exists but needs a cluster-aware filesystem, or you corrupt data. If you want shared writes, you wanted file or object.

6How it shows up in interviews#

Interviewers rarely ask "define object storage." They weave it into a larger design: "Design a photo-sharing app," or "where do you store the uploaded videos and why." What they are probing is whether you map a workload to the right primitive and can defend it with latency, mutability, cost, and scale.

Interviewer: You are designing a photo-sharing app. Where does the user's uploaded image go, and where does the post metadata go?

Candidate: The image goes to object storage under a key like userId/photoId.jpg. It is immutable, large, and read far more than written. The metadata row, caption, author, the object key, goes in Postgres, which sits on block storage because the DB does constant in-place writes that object storage cannot serve.

Interviewer: Why not just store the image as a blob in the database?

Candidate: It bloats the DB, makes backups huge, and wastes expensive block IOPS on data that never mutates. The DB should hold the pointer, not the payload.

Other probes they reach for:

• "How do you keep storage cost down as data ages?" Lifecycle policies to infrequent-access, then archive tiers.
• "Your read rate 10x'd. What breaks?" Object request cost and origin load; the answer is a CDN.

Weak answer vs strong answer

Weak: "I'll store the images in object storage like S3 because it's scalable and cheap, and the data in a database."

That is a memorized mapping with no reasoning. It names the right boxes but never says why, so the interviewer cannot tell if you understand the mechanism or just the buzzwords. Every junior says it, and the next probe ("why not the database for the image?") usually exposes the gap.

Strong: "User uploads are immutable, large, and write-once-read-many, so object storage. I'll store each image under a key like userId/photoId.jpg and keep only that key plus metadata in Postgres, which sits on block storage because the DB does constant in-place writes that object storage cannot serve. At 10k reads/sec, S3 request cost alone is ~$14/hour, so I front it with a CDN that absorbs 95% of reads and cuts origin requests ~20x to under $1/hour. As photos age I add a lifecycle rule to drop them to infrequent-access after 30 days, roughly halving storage cost on cold data. I'd avoid file storage here because nothing needs a shared POSIX tree and the metadata server would be a bottleneck."

Why the strong answer wins:

• It names the workload properties (immutable, write-once, read rate) and picks the primitive from those properties.
• It defends the database-on-block choice instead of assuming it.
• It shows cost awareness with an actual number, including the CDN and lifecycle levers.

The one line that always lands: when they ask "why block for the database," say "because a database writes to specific byte offsets in place constantly, and object storage replaces whole objects, so the write amplification would be catastrophic." That sentence tells them you understand the mechanism, not just the mapping.

See the related Design Interview Question: "Design an image hosting service" to practice placing block, file, and object storage in a full system.