I/O and the I/O Models

The process post noted processes spend much time Blocked on I/O; this post covers I/O itself — how programs read and write files, networks, and devices, and the crucial I/O models (blocking, non-blocking, asynchronous) that determine how well a program scales. I/O is where a lot of real-world performance lives (programs are often I/O-bound, not CPU-bound), and the model you choose is one of the most consequential system-design decisions. This connects to the networking, concurrency, and LLM-serving series.

Files, descriptors, and the I/O abstraction

The OS abstracts all I/O — files, network sockets, pipes, devices — behind a uniform interface. The central abstraction is the file descriptor (fd): a small integer handle representing an open I/O resource, whatever it is:

• Everything is a file(-like thing) — on Unix, an open file, a network socket, a pipe, even some devices are all accessed through file descriptors with the same operations (read, write, close). This uniformity means the same I/O concepts apply whether you’re reading a file or a network connection.
• I/O goes through syscalls — read(fd, ...) and write(fd, ...) are system calls (the first post): your program asks the kernel to move data to/from the resource. So every I/O operation crosses the user/kernel boundary, and the kernel does the actual device/network interaction. This is why I/O has cost (the boundary crossing) and why it involves the kernel.

The OS also buffers I/O — data is often staged in kernel buffers (and library buffers) rather than each byte going straight to the device — which improves efficiency (batching) but means “written” doesn’t always mean “on disk” (why fsync exists, from the database-internals WAL post). With the abstraction (fds, read/write syscalls, buffering) in mind, the crucial question is: what does your program do while an I/O operation is in progress? That’s the I/O model.

Blocking I/O: simple but limited

The default, simplest model is blocking I/O: when you call read() and the data isn’t ready (e.g. waiting on a network response or disk), your thread blocks — it stops and waits (goes to the Blocked state, from the process post) until the I/O completes, then continues:

Blocking read:
   read(fd) → (no data yet) → THREAD BLOCKS, waiting... → data arrives → returns
   → the thread does nothing else while waiting

Blocking I/O is easy to program (call read, get data, continue — straightforward sequential code) but has a scaling problem: one thread can only wait on one thing at a time. To handle many concurrent I/O operations (e.g. many network connections) with blocking I/O, you need many threads — typically one thread per connection, each blocking on its connection. And that runs into the thread-scaling limits from the scheduling post:

• Threads are limited — each thread has memory (its stack) and context-switch overhead; you can’t have hundreds of thousands of threads efficiently. Thread-per-connection tops out at maybe thousands of connections before context-switching overhead and memory dominate.
• Mostly waiting — with I/O-bound work, those threads spend most of their time blocked (not computing), so you’re paying for many threads that are mostly idle-waiting — wasteful.

This is the classic scaling wall: blocking I/O with thread-per-connection doesn’t scale to very high concurrency (the “C10K problem” — handling 10,000+ concurrent connections). It’s simple and fine for modest concurrency, but it’s why high-concurrency servers need a different model.

Non-blocking and asynchronous I/O

To handle massive concurrency, you need a model where one thread manages many I/O operations without blocking on each. Two related approaches:

• Non-blocking I/O + event notification (the event loop) — set file descriptors to non-blocking (so read returns immediately, with data or “not ready”), and use an OS mechanism (epoll on Linux, kqueue on BSD/macOS, IOCP on Windows) that lets one thread watch many fds at once and be told which are ready. The thread runs an event loop: ask the OS “which of these thousands of connections have data ready?”, handle those, repeat. One thread (or a few) handles tens of thousands of connections, because it never blocks — it only works on ready ones and lets the OS watch the rest.

Event loop (one thread, many connections):
   epoll_wait() → OS returns the fds that are READY → handle each ready fd → loop
   → one thread multiplexes thousands of connections, blocking on none

• Asynchronous I/O — you start an I/O operation and register a callback (or await a future/promise) to run when it completes; your code doesn’t block, and the completion is delivered later. This is the model behind async/await in modern languages: await read(...) doesn’t block the thread — it suspends that task and lets the thread do other work, resuming the task when the I/O completes.

The key idea in both: decouple “many concurrent I/O operations” from “many threads.” Instead of one blocked thread per operation, one thread (or a small pool) multiplexes many operations via non-blocking I/O + event notification. This is how high-concurrency servers (nginx, Node.js, Go’s runtime, async Python/Rust) handle enormous numbers of connections on few threads — they use event loops / async I/O, not thread-per-connection. It scales because it removes the per-connection thread cost (memory + context switches, from the scheduling post) — a mostly-idle-waiting connection costs almost nothing (just an fd the OS watches), not a whole thread.

Choosing an I/O model

The model is a real design decision with clear trade-offs:

• Blocking (thread-per-connection or thread pool) — simplest to write (straightforward sequential code), fine for modest concurrency and CPU-bound-ish work. Use it when connection counts are low-to-moderate and simplicity matters. It’s not “wrong” — it’s the right default until concurrency demands more.
• Async / event-loop (non-blocking) — necessary for high concurrency (many thousands+ of connections), especially I/O-bound work (lots of connections mostly waiting). It scales far better (few threads, many connections) but is more complex to write (callbacks or async/await, avoiding blocking the event loop) — a single blocking call in an event loop stalls everything, a classic bug. Use it when you need high concurrency.
• Hybrid / language runtimes — some runtimes give you the simplicity of blocking-style code with the scaling of async underneath: Go’s goroutines (cheap, and its runtime multiplexes them over threads with non-blocking I/O), and async/await (write sequential-looking code that’s actually non-blocking). These aim for “easy to write and scales,” and are the modern sweet spot for many servers.

The through-line, and why this matters: most servers are I/O-bound (limited by waiting on I/O, not CPU), so the I/O model is often the dominant scaling factor — the difference between handling hundreds vs hundreds of thousands of connections on the same hardware. Understanding blocking vs non-blocking/async explains why async/await and event loops exist, why “don’t block the event loop” is a rule, why Go and Node scale to huge concurrency, and how to choose for your own systems. This connects directly to the LLM-serving series (async, batching, high-concurrency serving) and the networking series (handling many connections).

I/O, understood

The takeaway: the OS abstracts all I/O behind file descriptors and read/write syscalls (crossing the user/kernel boundary, buffered for efficiency), and the crucial choice is the I/O model — what your program does while I/O is in progress. Blocking I/O is simple but forces thread-per-operation, which doesn’t scale to high concurrency (the C10K wall). Non-blocking I/O with event loops (epoll/kqueue) and asynchronous I/O (async/await) decouple concurrency from threads, letting one thread multiplex many operations — which is how high-concurrency servers scale. Choose blocking for simplicity/modest concurrency, async for high concurrency (or use runtimes like Go/async that give both). Since most servers are I/O-bound, the I/O model is often the scaling decision. The final post ties the series together: system calls and why OS knowledge makes you a better engineer.

Key takeaways

• The OS abstracts all I/O (files, sockets, pipes, devices) behind file descriptors with uniform read/write operations, done via syscalls (crossing the user/kernel boundary) and buffered for efficiency — so the same I/O concepts apply everywhere, and I/O involves the kernel (hence its cost, and why “written” isn’t always “persisted” — fsync).
• Blocking I/O (the default) makes a thread wait (block) until an operation completes — simple sequential code, but one thread waits on one thing, so handling many concurrent operations needs many threads (thread-per-connection), which hits thread-scaling limits (memory + context-switch overhead) — the C10K wall.
• Non-blocking I/O with event notification (epoll/kqueue) lets one thread run an event loop watching many fds and handling only ready ones, and asynchronous I/O (async/await) starts operations and resumes on completion without blocking — both decouple “many concurrent operations” from “many threads,” so few threads multiplex tens of thousands of connections.
• Choose by concurrency need: blocking for simplicity and modest concurrency (a fine default), async/event-loop for high concurrency (thousands+, especially I/O-bound) at the cost of complexity (“don’t block the event loop”), or runtimes like Go’s goroutines / async-await that give blocking-style simplicity with async scaling.
• Because most servers are I/O-bound (limited by waiting, not CPU), the I/O model is often the dominant scaling factor — the difference between hundreds and hundreds of thousands of connections on the same hardware — and it explains why async/await, event loops, and Go/Node’s concurrency exist.

Further reading

• The memory hierarchy and caching (previous post)
• Computer Networking for Backend Engineers — connections, and handling many of them
• LLM Inference and Serving — async and high-concurrency serving