The difference between a server that handles a hundred connections and one that handles a hundred thousand on the same hardware usually comes down to one choice: how it does I/O. Blocking, non-blocking, and asynchronous I/O aren't interchangeable styles — they're fundamentally different models with different scaling limits, and understanding them explains async/await, event loops, and why the network stack works the way it does.
The difference between a server handling a hundred connections and one handling a hundred thousand usually comes down to one choice: how it does I/O. Blocking, non-blocking, and asynchronous I/O are fundamentally different models with different scaling limits.
The single most counterintuitive fact in performance engineering: accessing memory is not one speed. A value in the CPU cache is hundreds of times faster to reach than one in main memory, which is thousands of times faster than disk. Your code's speed often depends less on how many operations it does than on where the data lives — and understanding the memory hierarchy is what lets you see that.
The most counterintuitive fact in performance: accessing memory is not one speed. A value in CPU cache is hundreds of times faster to reach than one in RAM. Your code's speed often depends less on how many operations it does than on where the data lives.
Kubernetes has to answer a question every time a pod is created: which machine should run it? The scheduler answers it, and the quality of that answer depends entirely on information you provide — resource requests and limits. Get those right and the cluster packs efficiently and stays stable; get them wrong and you get waste, evictions, and mysterious outages.
Kubernetes answers a question every time a pod is created: which machine runs it? The scheduler answers it, and the quality depends on resource requests and limits you provide. Get those wrong and you get waste, evictions, and mysterious outages.
Your machine runs hundreds of processes on a handful of CPU cores, and yet everything feels like it's running at once. That illusion is the CPU scheduler's doing — rapidly switching the cores between processes, dozens of times a second, deciding who runs and for how long. Understanding scheduling explains why your program isn't always running, why context switches cost, and why "add more threads" doesn't always mean faster.
Your machine runs hundreds of processes on a handful of cores, yet everything feels simultaneous. That illusion is the scheduler's doing — rapidly switching cores between processes. It explains why your program isn't always running and why more threads isn't always faster.
There is no single "make it fast" for LLM serving — latency and throughput pull against each other, and both trade against cost. The job isn't to maximize one number; it's to hit your latency targets at the lowest cost per token, which means knowing exactly which knob moves which metric and in which direction.
There is no single 'make it fast' for LLM serving — latency and throughput pull against each other, and both trade against cost. The job is to hit your latency targets at the lowest cost per token, knowing which knob moves which metric.
SQL is a language where you say what you want, not how to get it — and the component that invents the "how" is the query planner, the closest thing a database has to a brain. When a query is mysteriously slow, the answer is almost always in the plan, which is why reading `EXPLAIN` is the single most valuable database skill you can learn.
SQL says what you want, not how to get it — and the component that invents the 'how' is the query planner, the closest thing a database has to a brain. Reading EXPLAIN is the single most valuable database skill you can learn.
At some point a model doesn't fit on one GPU, or the traffic doesn't, and you have to spread inference across many. The choices — which kind of parallelism, how to place replicas, when to autoscale — are governed by one unforgiving resource (GPU memory) and one expensive one (inter-GPU communication). Get the memory math right and most scaling decisions follow.
At some point a model doesn't fit on one GPU, or the traffic doesn't. The scaling choices are governed by one unforgiving resource (GPU memory) and one expensive one (inter-GPU communication) — get the memory math right and most decisions follow.
The reason a long analytics query doesn't block every writer in your database — and vice versa — is a single elegant idea: never overwrite data, keep multiple versions, and give each transaction a consistent snapshot in time. MVCC is how nearly every modern database delivers isolation without readers and writers fighting over locks.
The reason a long analytics query doesn't block every writer — and vice versa — is a single elegant idea: never overwrite data, keep multiple versions, and give each transaction a consistent snapshot in time. That's MVCC.
HTTP is the protocol your applications actually speak, and it has quietly reinvented itself twice to fight one persistent enemy: head-of-line blocking, where one slow thing stalls everything behind it. The journey from HTTP/1.1 to HTTP/2 to HTTP/3 is the story of chasing that problem down the stack — and understanding it explains why modern connections are so much faster.
HTTP is the protocol your applications speak, and it has quietly reinvented itself twice to fight one enemy: head-of-line blocking, where one slow thing stalls everything behind it. The journey from HTTP/1.1 to 2 to 3 is chasing that problem down the stack.
Vectors are big, and storing millions of them in full precision is where vector search gets expensive. Quantization compresses each vector into a fraction of its size — trading a little recall for large memory savings — and it's the technique that lets both IVF and HNSW scale from millions of vectors to billions without a memory budget that breaks the bank.
Vectors are big, and storing millions in full precision is where vector search gets expensive. Quantization compresses each vector into a fraction of its size — trading a little recall for large memory savings — and it's what lets IVF and HNSW scale to billions.
You don't assemble the KV cache, continuous batching, quantization, and speculative decoding by hand — you use a serving engine that has already solved the hard parts. And the idea that ties them together, PagedAttention, is a borrowed operating-systems trick: manage the KV cache like virtual memory, in pages, and the waste that throttled everything disappears.
You don't assemble the KV cache, batching, and quantization by hand — you use a serving engine. And PagedAttention, the idea that ties them together, is a borrowed OS trick: manage the KV cache like virtual memory, in pages.
The claim that agents should write code isn't just aesthetic — Hugging Face backs it with measured results: code agents take fewer steps, make fewer LLM calls, and score better on hard benchmarks. This post examines the evidence and the mechanism, so you understand not just that code actions win but why.
The claim that agents should write code isn't just aesthetic — Hugging Face backs it with measured results: code agents take fewer steps, make fewer LLM calls, and score better on hard benchmarks. This examines the evidence and the mechanism.
Decode is slow because it's sequential — one token at a time, each waiting for the last. Speculative decoding cheats that limit with a beautiful trick: let a small, fast model guess several tokens ahead, then let the big model verify them all in a single pass. When the guesses are good, you get several tokens for the price of one — with mathematically identical output.
Decode is slow because it's sequential. Speculative decoding cheats that with a beautiful trick: a small fast model guesses several tokens ahead, the big model verifies them in one pass — several tokens for the price of one, with identical output.
An index is a data structure that lets a database find rows without reading the whole table — the difference between flipping to a book's index and reading every page. It's the highest-leverage performance tool a database gives you, and also the most misused: every index you add speeds up reads and slows down writes, so the skill is knowing exactly which ones earn their cost.
An index lets a database find rows without reading the whole table — the highest-leverage performance tool a database gives you, and the most misused: every index speeds up reads and slows down writes.
LoRA shrank the trainable parameters, but you still had to hold the full base model in memory to train against it — and for a large model that alone needs serious hardware. QLoRA closes the gap: quantize the frozen base model to 4-bit, train LoRA adapters on top, and fine-tune a large model on a single consumer GPU with almost no quality loss. It's what truly democratized fine-tuning.
LoRA shrank the trainable parameters, but you still had to hold the full base model in memory. QLoRA closes the gap: quantize the frozen base to 4-bit, train LoRA adapters on top, and fine-tune a large model on a single consumer GPU with almost no quality loss.
Quantization shrinks a model by storing its numbers in fewer bits — and because LLM decode is bottlenecked on moving those numbers from memory, making them smaller makes inference both cheaper to host and faster to run. It's the rare optimization that improves memory, cost, and speed at once, if you respect its limits on quality.
Quantization stores a model's numbers in fewer bits — and because decode is bottlenecked on moving those numbers from memory, making them smaller makes inference cheaper to host and faster to run, if you respect the limits on quality.
Of every AI cost lever, one dominates the rest, and all the major vendors agree on it: prompt caching. On long prompts and agentic workloads the reported savings are the largest anywhere — because caching attacks the specific way agent costs explode. If you optimize one thing, optimize this.
Of every AI cost lever, one dominates and all the major vendors agree on it: prompt caching. On agentic workloads the reported savings are the largest anywhere — because caching attacks the specific way agent costs explode. If you optimize one thing, optimize this.
IP gets packets to a host but promises nothing about whether they arrive, in order, or intact. The transport layer is where that gap is filled — or deliberately left open. TCP builds a reliable, ordered connection on top of unreliable IP; UDP declines to, trading guarantees for speed. Choosing between them, and understanding TCP's costs, is core backend knowledge.
IP gets packets to a host but promises nothing about whether they arrive, in order, or intact. The transport layer fills that gap — or deliberately leaves it open: TCP builds a reliable ordered connection, UDP declines to and stays fast.
Full fine-tuning updates every weight in a model — billions of numbers — which needs enormous memory and produces a full-size copy per task. LoRA sidesteps all of it with one insight: the change a model needs for a task is "low-rank," so you can train a tiny pair of matrices instead of the whole model. It's the technique that put fine-tuning within reach of anyone with a single GPU.
Full fine-tuning updates every weight — billions of numbers — needing enormous memory. LoRA sidesteps it with one insight: the change a model needs for a task is low-rank, so you train a tiny pair of matrices instead of the whole model.
The most underrated vector index is no index at all. Brute-force search — compare the query to every vector — is the one method with perfect recall, zero build time, and no tuning, and for a surprising number of real systems it's not just adequate but optimal. Knowing when you don't need an ANN index is as valuable as knowing how they work.
The most underrated vector index is no index at all. Brute-force search has perfect recall, zero build time, and no tuning, and for a surprising number of real systems it's optimal — knowing when you don't need ANN is as valuable as knowing how it works.
On the cloud, quantization is an optimization you reach for to cut costs. On a phone, it's the difference between the model running and not running at all. Nearly every on-device LLM you'll ever ship is quantized, because full-precision weights simply don't fit — so understanding the bit-width trade-off is non-negotiable for edge AI.
On the cloud, quantization cuts costs. On a phone, it's the difference between the model running and not running at all — nearly every on-device LLM you'll ship is quantized, so the bit-width trade-off is non-negotiable for edge AI.
A single request leaves an expensive GPU almost entirely idle. Batching is how you fill it — and the leap from naive static batching to continuous batching is the single biggest throughput improvement in modern LLM serving, often several times more requests from the exact same hardware.
A single request leaves an expensive GPU almost entirely idle. Batching fills it — and the leap from static to continuous batching is the single biggest throughput improvement in modern LLM serving.
The buffer pool is where a database spends most of its memory and wins or loses most of its performance. It's a cache of disk pages in RAM, and the difference between a query that hits it and one that misses is the difference between a microsecond and a millisecond — a thousandfold gap that decides whether your database feels fast.
The buffer pool is where a database spends most of its memory and wins or loses most of its performance — a cache of disk pages in RAM, where a hit versus a miss is a thousandfold latency gap.
Metrics are the cheapest, most efficient telemetry you have — a handful of numbers that summarize millions of events and tell you, at a glance, whether your system is healthy. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies to you.
Metrics are the cheapest, most efficient telemetry — a handful of numbers summarizing millions of events. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies.
A phone is not a small server — it's a fundamentally different environment with four hard limits a data center never imposes: a tight memory budget, modest and heterogeneous compute, a battery that your model drains, and a thermal ceiling that throttles you when you push it. Every on-device AI decision is a negotiation with these four.
A phone is not a small server — it's a different environment with four hard limits a data center never imposes: a tight memory budget, modest heterogeneous compute, a battery your model drains, and a thermal ceiling that throttles you.
The KV cache is the optimization that makes LLM generation fast enough to be practical — and the memory hog that makes it expensive. Almost every hard problem in LLM serving, from how many users you can batch to why long contexts cost so much, traces back to this one data structure.
The KV cache is the optimization that makes LLM generation fast enough to be practical — and the memory hog that makes it expensive. Almost every hard problem in serving traces back to this one data structure.
Running an LLM is not one computation — it's two very different ones stitched together: a compute-heavy pass over your prompt, then a long, memory-bound slog generating one token at a time. Almost every serving optimization that follows makes sense only once you see that inference has these two phases with opposite bottlenecks.
Running an LLM is two very different computations stitched together: a compute-heavy pass over your prompt, then a long, memory-bound slog generating one token at a time. Almost every serving optimization makes sense only once you see these two phases.
The two specialist lenses a reviewer switches on for a diff — thinking like an attacker to catch the injection and the missing authorization check, and thinking like production to catch the N+1 query — while knowing exactly where the human eye stops and a scanner, profiler, or load test has to take over.
The specialist lenses: security review (untrusted input to a sink, missing authz/BOLA, secrets, SSRF, new dependencies) and performance review (N+1 queries, unbounded queries, missing indexes) — flag the smells, defer depth to scanners and profilers.
When to graduate from the hosted API Catalog to inference you run yourself — deploying a NIM container, reaching for Triton and its Python client, the TensorRT-LLM levers that raise throughput and cut latency, and the one base_url change that carries your Python client across unchanged.
Move from the hosted API Catalog to self-hosted inference: run a NIM container (your Python client works unchanged by swapping base_url), use Triton directly via tritonclient, and understand the TensorRT-LLM levers — quantization, in-flight batching, paged KV cache, tensor parallelism.
Streaming is gRPC's most powerful feature and its most misused. Sending a stream of messages sounds simple until one side produces faster than the other can consume — then, without flow control, you get unbounded memory growth and crashes. gRPC inherits HTTP/2's flow control to prevent exactly this. Understanding backpressure is the difference between streaming that scales and streaming that falls over under load.
Streaming is gRPC's most powerful feature and its most misused. Sending a stream sounds simple until one side produces faster than the other consumes — then, without flow control, you get unbounded memory growth and crashes. gRPC inherits HTTP/2's flow control to prevent exactly this. Backpressure is the difference between streaming that scales and streaming that falls over.
Making an LLM system faster and cheaper without touching the weights — the levers an application engineer actually controls, from streaming and caching to token trimming, model routing, and Go's real superpower: concurrency with a rate limiter.
Make LLM systems faster and cheaper without retraining: TTFT vs throughput, exact and semantic response caching, prompt caching, token reduction, model routing and cascades, and Go concurrency with a worker pool and rate limiter.
The highest-leverage tool for latency and scale — and the source of its hardest problem, invalidation. Where caches live, the patterns for filling them, how they evict, and why keeping them correct is the part that stays hard.
Caching as the highest-leverage latency tool — and its hardest problem: where caches live, the patterns (cache-aside/read-through/write-through/write-behind), eviction, and invalidation including cache stampede, penetration, and hot keys.
Caching giveth performance and taketh away your sanity. The same technique that makes systems fast introduces a whole category of subtle, intermittent, hard-to-debug problems — stale data that appears randomly, a cache that collapses under load at the worst moment, bugs that only happen when the cache is cold or full. This closing post catalogs the pitfalls that bite real systems, and distills the practical wisdom of the series: cache deliberately, expect the failure modes, and remember that the two genuinely hard things are still hard.
Caching giveth performance and taketh away your sanity. The same technique that makes systems fast introduces subtle, intermittent, hard-to-debug problems — stale data appearing randomly, a cache collapsing under load, bugs that only happen when the cache is cold. This catalogs the pitfalls and the practical wisdom.
Every time a web page loads instantly on a repeat visit, or a video streams smoothly from halfway around the world, caching is the reason. The web is layered with caches — in your browser, at CDN edge servers near you, in front of origin servers — all applying the same caching principle to make the internet fast. And remarkably, much of it is coordinated by a few HTTP headers that let servers tell caches exactly what to store and for how long. Understanding web and CDN caching is understanding how the internet stays fast at global scale.
Every time a web page loads instantly on a repeat visit, or a video streams smoothly from across the world, caching is the reason. The web is layered with caches — browser, CDN edge, origin — all applying the same principle, much of it coordinated by a few HTTP headers.
A cache inside a single application process is easy — but it doesn't scale, and every instance of your app has its own separate copy. The moment you run many application servers, you want a shared cache they all use, which means a cache that lives across the network on its own machines: a distributed cache like Redis or Memcached. This unlocks scale and sharing, but introduces the distributed-systems problems that a local cache never had. Understanding distributed caching is understanding how caching works at real scale.
A cache inside a single process is easy — but it doesn't scale, and every instance of your app has its own separate copy. Run many servers and you want a shared cache across the network on its own machines: a distributed cache like Redis or Memcached. This unlocks scale and sharing, but introduces distributed-systems problems.
Knowing to cache is one thing; wiring the cache into your application correctly is another. Should the application manage the cache itself, or should the cache sit transparently in front of the source? Should writes go to the cache, the database, or both — and in what order? These questions have standard answers — the caching patterns — and choosing the right one shapes your consistency, performance, and complexity. Getting the pattern right is how caching goes from "store some stuff" to a coherent, correct design.
Knowing to cache is one thing; wiring the cache into your application correctly is another. Should the application manage the cache itself, or should it sit transparently in front of the source? Should writes go to the cache, the database, or both? These questions have standard answers — the caching patterns.
This is the hard one. "There are only two hard things in computer science: cache invalidation and naming things" names it directly — cache invalidation is genuinely, notoriously difficult. The moment you cache a copy of data, you've created a second source of truth that can drift from the first, and keeping them in sync (or deciding how much drift you'll tolerate) is a problem with no clean, universal solution. Understanding why it's hard, and the strategies for managing it, is the difference between caching that helps and caching that causes baffling bugs.
This is the hard one. 'There are only two hard things in computer science: cache invalidation and naming things' names it directly. The moment you cache a copy of data, you've created a second source of truth that can drift from the first, and keeping them in sync has no clean universal solution.
A cache is a small space pretending to be a big one, and the pretense only works if it's clever about what to keep. When a bounded cache fills up, every new item forces out an old one — and which one you evict determines your hit rate, which determines whether the cache is worth having at all. Eviction policies are the algorithms that make this choice, and understanding them (especially the workhorse, LRU) is essential to building caches that actually stay effective.
A cache is a small space pretending to be a big one, and the pretense only works if it's clever about what to keep. When a bounded cache fills up, every new item forces out an old one — and which one you evict determines your hit rate. Eviction policies are the algorithms that make this choice.
A cache lives or dies by one number: its hit rate. Every cache access is a small bet — that the data will be there (a hit, served fast) rather than missing (a miss, served slow, plus the overhead of caching it). Whether caching helps at all comes down to how often that bet pays off, and understanding hits, misses, and hit rate — and what you should and shouldn't cache — is the foundation of using caches effectively. Get these fundamentals right, and the rest of caching makes sense.
A cache lives or dies by one number: its hit rate. Every cache access is a small bet — that the data will be there (a hit, served fast) rather than missing (a miss, served slow). Whether caching helps at all comes down to how often that bet pays off.
There's an old joke that there are only two hard things in computer science: cache invalidation and naming things. It's a joke because caching is everywhere and sounds simple — just keep a copy of stuff you'll need again — and it's true because getting caching right is genuinely, surprisingly hard. Caching is one of the most universal and powerful ideas in computing, appearing at every layer from CPU to CDN, and understanding why it exists and when it helps is foundational to building fast systems.
There's an old joke that there are only two hard things in computer science: cache invalidation and naming things. It's a joke because caching sounds simple, and true because getting it right is genuinely hard. Caching is one of the most universal ideas in computing, appearing at every layer from CPU to CDN.
Spanner partitions by primary-key range. A monotonically-increasing PK like a timestamp or UUID-v1 funnels all writes to one server. The fix changes everything from your sequence strategy to your tenant model.
The Picnic social platform served 1M+ users across a graph of Go microservices behind a GraphQL gateway. The latency win came from a counter-intuitive move: fewer services, tighter contracts.
Rust's central promise — the one that runs through this entire curriculum — is that you don't have to choose between safety and speed. The abstractions that make Rust pleasant to write (iterators, closures, traits, generics) compile down to code as fast as hand-written low-level equivalents. This closing post steps back to that promise, to what "idiomatic Rust" really means, and to where you go after the fundamentals. It's the capstone of Rust from the Ground Up — a look at the whole, and the road ahead.
Rust's central promise — the one that runs through this whole curriculum — is that you don't have to choose between safety and speed. This closing post steps back to that promise, to what 'idiomatic Rust' really means, and to where you go after the fundamentals. The capstone of Rust from the Ground Up.