You can build software for years treating the operating system as a black box — and then one day a production mystery (a service that's slow for no reason, a memory crash, a concurrency heisenbug, a server that won't scale) has an answer that lives entirely below your framework. OS knowledge is what lets you see down there. This closing post shows how everything in the series connects, through the system-call boundary and the diagnostic power it gives you.
You can build software for years treating the OS as a black box — then a production mystery has an answer that lives entirely below your framework. OS knowledge is what lets you see down there. How everything connects, through the system-call boundary and the diagnostic power it gives.
The difference between a server that handles a hundred connections and one that handles a hundred thousand on the same hardware usually comes down to one choice: how it does I/O. Blocking, non-blocking, and asynchronous I/O aren't interchangeable styles — they're fundamentally different models with different scaling limits, and understanding them explains async/await, event loops, and why the network stack works the way it does.
The difference between a server handling a hundred connections and one handling a hundred thousand usually comes down to one choice: how it does I/O. Blocking, non-blocking, and asynchronous I/O are fundamentally different models with different scaling limits.
The hardest part of platform engineering isn't the technology — it's building a platform people actually adopt, and knowing whether it's working. This closing post covers treating the platform as a product, measuring it with DORA metrics and adoption, structuring the team, and the failure modes that sink platforms. Getting these right is what turns platform engineering from a project into a lasting capability.
The hardest part of platform engineering isn't the technology — it's building a platform people actually adopt, and knowing whether it's working. Treat the platform as a product, measure it with DORA metrics and adoption, and avoid the failure modes that sink platforms.
The single most counterintuitive fact in performance engineering: accessing memory is not one speed. A value in the CPU cache is hundreds of times faster to reach than one in main memory, which is thousands of times faster than disk. Your code's speed often depends less on how many operations it does than on where the data lives — and understanding the memory hierarchy is what lets you see that.
The most counterintuitive fact in performance: accessing memory is not one speed. A value in CPU cache is hundreds of times faster to reach than one in RAM. Your code's speed often depends less on how many operations it does than on where the data lives.
Pods are disposable — destroyed and recreated constantly — which raises two problems: how do you give a pod its configuration without baking it into the image, and how does any data survive a pod's death? ConfigMaps and Secrets answer the first; volumes and StatefulSets answer the second. This is how stateless-by-default Kubernetes handles config and the state it can't avoid.
Pods are disposable, which raises two problems: how to give a pod its config without baking it into the image, and how any data survives a pod's death. ConfigMaps and Secrets answer the first; volumes and StatefulSets answer the second.
A golden path is the well-lit, paved road through your platform — the supported, opinionated way to build and ship a service, so a developer can go from idea to production without making a hundred infrastructure decisions. Developer experience is the measure of how good that road feels. Together they're what makes a platform actually reduce cognitive load rather than just relocate it.
A golden path is the well-lit, paved road through your platform — the supported, opinionated way to build and ship a service, so a developer goes from idea to production without a hundred infrastructure decisions. Developer experience is how good that road feels.
Rust has no exceptions. Errors and absent values are ordinary data — enum values you must handle — so the compiler forces you to deal with the possibility of failure instead of letting it propagate invisibly. It sounds tedious and turns out to be one of Rust's quiet strengths: you cannot forget to handle an error.
Rust has no exceptions. Errors and absent values are ordinary data — enum values you must handle — so the compiler forces you to deal with failure instead of letting it propagate invisibly. It turns out to be one of Rust's quiet strengths.
The internal developer platform is the product platform engineering builds: a self-service layer that packages all the infrastructure complexity — CI/CD, Kubernetes, cloud, IaC — into paved roads a developer can use without understanding any of it. Getting the concept right, especially the "platform as a product" mindset, is the difference between a platform developers love and one they route around.
The internal developer platform is the product platform engineering builds: a self-service layer that packages all the infrastructure complexity into paved roads a developer can use without understanding any of it. The 'platform as a product' mindset makes or breaks it.
Rust's enums are not the feeble named-constants of other languages — they're full algebraic data types that can hold data, and combined with pattern matching they become one of Rust's most loved features. Together with structs, they're how you model your domain, and the compiler makes sure you handle every case.
Rust's enums are not the feeble named-constants of other languages — they're full algebraic data types that hold data, and combined with pattern matching they become one of Rust's most loved features. Together with structs, they're how you model your domain.
A thread lets one process do several things at once — and the moment you have two threads touching the same memory, you've entered the hardest territory in all of programming: concurrency. Race conditions, deadlocks, and the need for synchronization are not exotic edge cases; they're the fundamental consequences of shared mutable state, and understanding them is what separates working concurrent code from code that fails mysteriously.
A thread lets one process do several things at once — and the moment two threads touch the same memory, you're in the hardest territory in programming: concurrency. Race conditions, deadlocks, and synchronization are the fundamental consequences of shared mutable state.
All the theory pays off in a handful of habits that separate resilient backend code from code that falls over the first time the network misbehaves — which it will. Set timeouts on everything, reuse connections, retry idempotently, and know how to read the layers when something breaks. This closing post turns the stack into a working checklist.
All the theory pays off in a handful of habits that separate resilient backend code from code that falls over the first time the network misbehaves — which it will. Set timeouts on everything, reuse connections, retry idempotently, and debug by layer.
Knowing the pillars is not the same as having an observable system. In practice, observability is built incrementally, costs real money you have to manage, and only pays off if the whole team treats telemetry as part of building software — not something added after the outage. This closing post turns the concepts into a way of working.
Knowing the pillars is not the same as having an observable system. In practice, observability is built incrementally, costs real money you must manage, and only pays off if the whole team treats telemetry as part of building software — not something added after the outage.
SQL is a language where you say what you want, not how to get it — and the component that invents the "how" is the query planner, the closest thing a database has to a brain. When a query is mysteriously slow, the answer is almost always in the plan, which is why reading `EXPLAIN` is the single most valuable database skill you can learn.
SQL says what you want, not how to get it — and the component that invents the 'how' is the query planner, the closest thing a database has to a brain. Reading EXPLAIN is the single most valuable database skill you can learn.
Kafka's defaults will run; whether they'll survive a broker failure, a traffic spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor — that are far cheaper to make now than to retrofit later.
Kafka's defaults will run; whether they survive a broker failure, a spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor.
A process is the OS's answer to "what is a running program?" — and it's more than the code: it's the code plus its own private memory, its own resources, and its own isolated view of the machine, as if it owned the computer. That isolation is what lets many programs run at once without corrupting each other, and understanding it explains a huge amount of how systems behave.
A process is the OS's answer to 'what is a running program?' — more than the code: it's the code plus its own private memory, resources, and isolated view of the machine. That isolation is what lets many programs run at once without corrupting each other.
Almost nothing on the modern internet talks directly to the server that answers it. In between sit proxies and load balancers — the traffic directors that spread load across many servers, terminate TLS, cache responses, and shield your backends. Understanding this layer is understanding how a single domain name serves millions of users from hundreds of machines.
Almost nothing on the modern internet talks directly to the server that answers it. In between sit proxies and load balancers — the traffic directors that spread load, terminate TLS, cache, and shield your backends — turning one domain into a system that scales.
An alert that fires when nothing is actually wrong trains people to ignore alerts — and an ignored alert is worse than no alert, because it creates false confidence while the real incident scrolls past unnoticed. Good alerting is ruthlessly selective: page a human only for things that are both urgent and real, alert on what users feel, and treat every noisy alert as a bug to fix.
An alert that fires when nothing is wrong trains people to ignore alerts — and an ignored alert is worse than none. Good alerting is ruthlessly selective: page a human only for things both urgent and real, alert on what users feel, and treat every noisy alert as a bug.
The reason a long analytics query doesn't block every writer in your database — and vice versa — is a single elegant idea: never overwrite data, keep multiple versions, and give each transaction a consistent snapshot in time. MVCC is how nearly every modern database delivers isolation without readers and writers fighting over locks.
The reason a long analytics query doesn't block every writer — and vice versa — is a single elegant idea: never overwrite data, keep multiple versions, and give each transaction a consistent snapshot in time. That's MVCC.
Consensus is the problem of getting a group of unreliable machines to agree on a single value despite crashes, delays, and lost messages. It sounds narrow, but it's the hidden foundation under leader election, distributed locks, configuration, and every "exactly one node is in charge" guarantee. Raft is the algorithm that finally made it understandable.
Consensus is getting unreliable machines to agree on a single value despite crashes and lost messages — the hidden foundation under leader election, distributed locks, and every 'exactly one node is in charge' guarantee. Raft made it understandable.
Kafka gives you a durable log; these patterns are what you build on it — event sourcing, CQRS, the outbox, sagas, and the choice between choreography and orchestration — the vocabulary of real event-driven systems.
Kafka gives you a durable log; these patterns are what you build on it — event sourcing, CQRS, the outbox, sagas, and the choice between choreography and orchestration.
You write applications that run on top of an operating system every day, and mostly you can ignore it — until a performance mystery, a concurrency bug, or a resource limit forces you to understand what's underneath. The OS is doing two jobs for you constantly: managing the hardware's finite resources, and giving you clean abstractions over messy reality. Understanding those two jobs is understanding the machine your code actually runs on.
You write applications on top of an OS every day and mostly ignore it — until a performance mystery, concurrency bug, or resource limit forces you to understand it. The OS does two jobs: managing finite hardware, and abstracting messy reality. Understanding them is understanding the machine your code runs on.
CI/CD is the assembly line of modern software — the automated path from a developer's commit to running production code. It's the practice that turned releases from rare, terrifying events into routine, boring ones, and "boring releases" is one of the highest compliments in software. It's also the first capability any platform provides.
CI/CD is the assembly line of modern software — the automated path from a developer's commit to running production code. It turned releases from rare, terrifying events into routine, boring ones, and 'boring releases' is one of the highest compliments in software.
Compliance requires you to collect and keep a great deal of personal data; privacy law requires you to protect it, limit it, and sometimes delete it. Data privacy is the compliance domain that governs the data all the other controls depend on — and it turns "how you handle personal information" from a courtesy into a set of enforceable engineering obligations.
Compliance requires you to collect and keep a great deal of personal data; privacy law requires you to protect it, limit it, and sometimes delete it. Data privacy governs the data all the other controls depend on — turning data handling into enforceable engineering obligations.
HTTP is the protocol your applications actually speak, and it has quietly reinvented itself twice to fight one persistent enemy: head-of-line blocking, where one slow thing stalls everything behind it. The journey from HTTP/1.1 to HTTP/2 to HTTP/3 is the story of chasing that problem down the stack — and understanding it explains why modern connections are so much faster.
HTTP is the protocol your applications speak, and it has quietly reinvented itself twice to fight one enemy: head-of-line blocking, where one slow thing stalls everything behind it. The journey from HTTP/1.1 to 2 to 3 is chasing that problem down the stack.
"Is the system reliable?" is a useless question until you define reliability as a number. SLOs turn reliability from a vague aspiration into a measurable target, and the error budget — the small amount of unreliability you're allowed — turns it into a tool that settles the eternal fight between shipping features and keeping things stable. This is where telemetry becomes a way to run engineering, not just watch it.
'Is the system reliable?' is useless until you define reliability as a number. SLOs turn reliability into a measurable target, and the error budget — the unreliability you're allowed — settles the eternal fight between shipping features and keeping things stable.
A transaction is a promise that a group of operations happens all-or-nothing and doesn't get corrupted by everyone else doing the same thing at once. Most developers know the word ACID; far fewer know that the "I" — isolation — is a dial with several settings, and that the default setting in most databases allows anomalies they've never heard of.
Most developers know ACID; far fewer know that the 'I' — isolation — is a dial with several settings, and that the default in most databases allows anomalies they've never heard of.
In an event-driven system your events are a public API that outlives every service that reads them, so how you shape them and how you evolve them without breaking consumers is not a detail — it is the contract the whole architecture rests on.
In an event-driven system your events are a public API that outlives every service that reads them, so how you shape and evolve them without breaking consumers is the contract the whole architecture rests on.
The primitives in this series are unbreakable in practice — and yet crypto keeps failing in the real world. That's the paradox of applied cryptography: almost every vulnerability is a misuse of a sound primitive, not a broken one. A predictable random number, a comparison that returns early, a reused nonce, a missing authentication check — each is a one-line mistake that silently voids the guarantee. This closing post catalogs the pitfalls that matter and distills the whole series into a decision guide.
The primitives are unbreakable in practice, yet crypto keeps failing — because almost every vulnerability is a misuse of a sound primitive, not a broken one. A predictable random number, an early-exit comparison, a reused nonce: one-line mistakes that silently void the guarantee. Plus a decision guide.
DevOps promised to tear down the wall between development and operations — and it worked, but it accidentally built a new problem: it made every developer responsible for a sprawling stack of infrastructure, pipelines, and tooling nobody has time to master. Platform engineering is the industry's answer: give developers a paved road instead of a pile of tools. Understanding that evolution is the key to the whole discipline.
DevOps tore down the wall between dev and ops — and accidentally built a new problem: it made every developer responsible for a sprawling stack nobody has time to master. Platform engineering is the answer: give developers a paved road instead of a pile of tools.
In most languages, variables vary — that's the default, and you opt into constancy. Rust flips it: variables are immutable unless you say otherwise. That one inverted default, plus a strong static type system with inference, quietly shapes how Rust code is written and prevents a whole class of bugs before you meet ownership.
In most languages variables vary by default; Rust flips it — variables are immutable unless you say otherwise. That one inverted default, plus a strong static type system with inference, quietly shapes how Rust is written and prevents a class of bugs.
Compliance ultimately comes down to one demand: prove it. Prove you verified the customer, prove you screened the transaction, prove you investigated the alert. The audit trail is how you prove it — an immutable, complete record of what happened and why — and it's the quiet backbone that makes every other compliance control defensible.
Compliance ultimately comes down to one demand: prove it. Prove you verified the customer, screened the transaction, investigated the alert. The audit trail is how you prove it — an immutable, complete record — the backbone that makes every other control defensible.
The "s" in HTTPS is TLS, and it does three things at once that most engineers conflate: it encrypts the connection, verifies you're talking to the real server, and detects tampering. Understanding how — the handshake, the certificates, the chain of trust — demystifies the padlock icon and the certificate errors that block deploys, and it's foundational to every secure connection you make.
The 's' in HTTPS is TLS, and it does three things engineers often conflate: encrypts the connection, verifies you're talking to the real server, and detects tampering. Understanding the handshake, certificates, and chain of trust demystifies the padlock and the cert errors.
Before OpenTelemetry, instrumenting a system meant picking a vendor and wiring their proprietary agent into all your code — and switching vendors meant re-instrumenting everything. OpenTelemetry ended that: one open, vendor-neutral standard for producing metrics, logs, and traces, so you instrument once and send the data anywhere. It's become the default way to make systems observable.
Before OpenTelemetry, instrumenting meant picking a vendor and wiring their agent into all your code — switching vendors meant re-instrumenting everything. OpenTelemetry ended that: one open, vendor-neutral standard, so you instrument once and send the data anywhere.
An index is a data structure that lets a database find rows without reading the whole table — the difference between flipping to a book's index and reading every page. It's the highest-leverage performance tool a database gives you, and also the most misused: every index you add speeds up reads and slows down writes, so the skill is knowing exactly which ones earn their cost.
An index lets a database find rows without reading the whole table — the highest-leverage performance tool a database gives you, and the most misused: every index speeds up reads and slows down writes.
"Exactly-once" is the most misunderstood phrase in streaming — it is real in Kafka, but only within a specific boundary, and outside that boundary the honest and usually-correct answer is at-least-once plus idempotent consumers.
'Exactly-once' is the most misunderstood phrase in streaming — it's real in Kafka, but only within a specific boundary, and outside it the honest answer is at-least-once plus idempotent consumers.
DNS is the internet's phone book — it turns the names humans use into the addresses machines route to — and it's also the internet's most common outage cause and a frequent source of baffling latency. "It's always DNS" is a running joke among engineers precisely because DNS is invisible until it breaks, and then it breaks everything.
DNS is the internet's phone book — it turns the names humans use into the addresses machines route to — and it's also the internet's most common outage cause. 'It's always DNS' is a running joke because DNS is invisible until it breaks, and then it breaks everything.
When a request touches ten services and comes back slow, metrics tell you it's slow and logs tell you what each service did — but neither shows you the one thing you need: where, along that journey, the time actually went. Distributed tracing is the pillar built for exactly this, following a single request across every service it touches and showing you the whole path at once.
When a request touches ten services and comes back slow, metrics say it's slow and logs say what each service did — but neither shows where the time went. Distributed tracing follows a single request across every service and shows the whole path at once.
Durability — the promise that a committed transaction survives a crash — comes down to one deceptively simple rule: write down what you're about to do before you do it. The write-ahead log is that rule made concrete, and it's the reason a database can be both fast and crash-safe, two goals that otherwise pull in opposite directions.
Durability comes down to one deceptively simple rule: write down what you're about to do before you do it. The write-ahead log is that rule made concrete — the reason a database can be both fast and crash-safe.
The most dangerous line of code in a distributed system is the one that trusts a timestamp. Physical clocks on different machines disagree, drift, and jump backward — so "which event happened first?" cannot be answered by comparing wall-clock times. Logical clocks answer it instead, by tracking causality rather than time.
The most dangerous line in a distributed system is the one that trusts a timestamp. Logical clocks — Lamport timestamps and vector clocks — order events by causality instead of unreliable wall-clock time.
A single consumer reading a topic is easy; the elegant part is how Kafka lets a group of consumers share the work automatically, rebalance when members come and go, and each remember exactly where it left off.
The elegant part of Kafka is how a group of consumers shares the work automatically, rebalances when members come and go, and each remembers exactly where it left off.
TLS is the protocol securing nearly every connection you make, and it's not a single cryptographic trick — it's the whole toolkit orchestrated into one handshake. Key exchange, certificates, signatures, and authenticated encryption each solve one sub-problem, and TLS composes them so that two parties who've never met can establish a private, tamper-proof, authenticated channel over a hostile network. Understanding the handshake is understanding how every earlier piece fits.
TLS secures nearly every connection you make, and it's not a single trick — it's the whole toolkit orchestrated into one handshake. Key exchange, certificates, signatures, and AEAD each solve one sub-problem, and TLS composes them into a private, authenticated channel over a hostile network.
Rust makes a promise that sounds impossible: memory safety without a garbage collector, and fearless concurrency without data races — all checked at compile time, with no runtime cost. The price is a compiler that argues with you until your program is correct. Understanding that bargain is the key to understanding why Rust exists and why people love it.
Rust makes a promise that sounds impossible: memory safety without a garbage collector, and fearless concurrency without data races — all checked at compile time, with no runtime cost. The price is a compiler that argues with you until your program is correct.
IP gets packets to a host but promises nothing about whether they arrive, in order, or intact. The transport layer is where that gap is filled — or deliberately left open. TCP builds a reliable, ordered connection on top of unreliable IP; UDP declines to, trading guarantees for speed. Choosing between them, and understanding TCP's costs, is core backend knowledge.
IP gets packets to a host but promises nothing about whether they arrive, in order, or intact. The transport layer fills that gap — or deliberately leaves it open: TCP builds a reliable ordered connection, UDP declines to and stays fast.
Logs are the oldest and most detailed telemetry — the granular record of what actually happened. But the log line you write for a human to read with grep is nearly useless at scale; the one you write as structured data for a machine to query is the one that saves you at 3 a.m. The shift from text logs to structured logs is the single biggest upgrade most teams can make.
Logs are the most detailed telemetry — the record of what actually happened. But the log line you write for a human to grep is nearly useless at scale; the one you write as structured data for a machine to query is the one that saves you at 3 a.m.
The buffer pool is where a database spends most of its memory and wins or loses most of its performance. It's a cache of disk pages in RAM, and the difference between a query that hits it and one that misses is the difference between a microsecond and a millisecond — a thousandfold gap that decides whether your database feels fast.
The buffer pool is where a database spends most of its memory and wins or loses most of its performance — a cache of disk pages in RAM, where a hit versus a miss is a thousandfold latency gap.
A producer looks trivial — send a record to a topic — but the three decisions it makes (which partition, how durably, how safely on retry) determine your ordering, your durability, and whether retries create duplicates.
A producer looks trivial, but the three decisions it makes — which partition, how durably, how safely on retry — determine your ordering, your durability, and whether retries create duplicates.
The internet layer performs a small miracle billions of times a second: it gets a packet from any machine to any other machine on Earth, across networks owned by thousands of independent organizations, with no central controller and no guarantee it'll arrive. Understanding IP — addresses, packets, routing, and why it's deliberately unreliable — is understanding the foundation everything else is built on.
The internet layer performs a small miracle billions of times a second: it gets a packet from any machine to any other on Earth, across networks owned by thousands of organizations, with no central controller and no guarantee it'll arrive.
Metrics are the cheapest, most efficient telemetry you have — a handful of numbers that summarize millions of events and tell you, at a glance, whether your system is healthy. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies to you.
Metrics are the cheapest, most efficient telemetry — a handful of numbers summarizing millions of events. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies.
Almost every database on earth stores its data in one of two structures: a B-tree that updates in place, or an LSM-tree that only ever appends. This one choice ripples through everything — read speed, write speed, space usage, and latency predictability — so knowing which your database uses tells you more about its behavior than almost anything else.
Almost every database stores data in one of two structures: a B-tree that updates in place, or an LSM-tree that only appends. This one choice ripples through read speed, write speed, space, and latency predictability.
Almost everything Kafka does follows from one deceptively simple idea — an append-only, ordered, durable log — and once that clicks, topics, partitions, and offsets stop being jargon and become obvious.
Almost everything Kafka does follows from one deceptively simple idea — an append-only, ordered, durable log — and once that clicks, topics, partitions, and offsets stop being jargon and become obvious.
Symmetric encryption has a chicken-and-egg problem: to share a secret key securely, you seem to need a secure channel — which is what you were trying to build. Public-key cryptography is the astonishing idea that broke that loop: two mathematically-linked keys where knowing the public one doesn't reveal the private one. It's the foundation of key exchange, digital signatures, and essentially all secure communication over the open internet.
Symmetric encryption has a chicken-and-egg problem: sharing a secret key securely seems to need a secure channel. Public-key cryptography broke that loop — two mathematically-linked keys where the public one doesn't reveal the private. It's the foundation of key exchange and signatures.
In a regulated business, compliance is not paperwork bolted onto the product — it's a set of hard requirements woven through the code, and treating it as an engineering discipline rather than a legal afterthought is what separates companies that scale safely from ones that get shut down. RegTech is the practice of building compliance in, as software.
In a regulated business, compliance is not paperwork bolted onto the product — it's hard requirements woven through the code, and treating it as an engineering discipline rather than a legal afterthought separates companies that scale safely from ones that get shut down.
Every backend engineer relies on the network constantly and understands it vaguely — until a mysterious timeout, a TLS error, or a latency spike forces a reckoning. The layered model of networking is the map that makes those problems legible: each layer does one job, hides the one below it, and fails in its own characteristic way. Learn the layers and the network stops being magic.
Every backend engineer relies on the network constantly and understands it vaguely — until a mysterious timeout or TLS error forces a reckoning. The layered model is the map that makes those problems legible: learn the layers and the network stops being magic.
Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside. In a world of distributed services where failures are novel and emergent, that difference — between watching known dashboards and investigating unknown problems — is the difference between guessing and knowing.
Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside — the difference between watching known dashboards and investigating unknown problems.
A database is not magic — it's a program that turns your rows into bytes on a disk and finds them again quickly, correctly, and without losing them when the power fails. Understanding the machine underneath the SQL is what separates someone who writes queries from someone who knows why they're slow.
A database is a program that turns your rows into bytes on a disk and finds them again quickly, correctly, and without losing them when the power fails. Understanding the machine underneath the SQL is what separates writing queries from knowing why they're slow.
A distributed system is one where a machine you've never heard of failing can stop your program from working. That single property — partial failure — is the root of almost everything that makes this field hard, and pretending it away is the most common and most expensive mistake in backend engineering.
A distributed system is one where a machine you've never heard of failing can stop your program from working. That single property — partial failure — is the root of almost everything that makes the field hard.
Synchronous request/response quietly welds your services together until a change in one breaks three others; event-driven architecture breaks that weld by making the event — a fact that happened — the thing services share.
Synchronous request/response quietly welds your services together until a change in one breaks three others; event-driven architecture breaks that weld by making the event — a fact that happened — the thing services share.
"Just hash the password" is advice that's simultaneously right and dangerous — right that you never store plaintext, dangerous because a fast cryptographic hash like SHA-256 is exactly the wrong tool for passwords. Hashing, message authentication, and password storage are three different jobs that all involve hashing, and confusing them is a classic source of real breaches.
'Just hash the password' is simultaneously right and dangerous — right that you never store plaintext, dangerous because a fast hash like SHA-256 is the wrong tool for passwords. Hashing, MACs, and password storage are three different jobs that all involve hashing.
A gRPC service that works on localhost is a long way from one that runs reliably at scale. Production raises questions localhost never does: how do calls get load-balanced when connections are long-lived? How do you secure them, expose them to browsers, observe them, and evolve the contract without breaking anyone? This closing post covers what it takes to run gRPC for real.
A gRPC service that works on localhost is far from one that runs reliably at scale. Production raises questions localhost never does: how do calls get load-balanced when connections are long-lived? How do you secure them, expose them to browsers, observe them, and evolve the contract without breaking anyone? This closing post covers running gRPC for real.
Symmetric encryption is the workhorse of confidentiality — the same fast primitive protecting your disk, your database fields, and every byte inside a TLS connection. But "encrypt this" is a trap: raw encryption alone doesn't stop tampering, block ciphers need a mode of operation, and modes need nonces that must never repeat. The single right answer for almost every case is authenticated encryption, and this post explains why.
Symmetric encryption is the workhorse of confidentiality — but 'encrypt this' is a trap. Raw encryption doesn't stop tampering, block ciphers need a mode, and nonces must never repeat. The right answer for almost every case is authenticated encryption (AEAD), and this post explains why.
Taking an Amazon Bedrock Go service from a working prototype to something you can run on-call — least-privilege IAM, credentials without static keys, tuning the SDK's built-in retryer, tracking token cost, and wiring up logging and metrics with aws-sdk-go-v2.
Taking a Bedrock Go service to production: least-privilege IAM and role-based credentials, tuning the SDK's built-in retryer for throttling, token-based cost tracking, and observability via model-invocation logging, structured metrics, and request IDs.
Streaming is gRPC's most powerful feature and its most misused. Sending a stream of messages sounds simple until one side produces faster than the other can consume — then, without flow control, you get unbounded memory growth and crashes. gRPC inherits HTTP/2's flow control to prevent exactly this. Understanding backpressure is the difference between streaming that scales and streaming that falls over under load.
Streaming is gRPC's most powerful feature and its most misused. Sending a stream sounds simple until one side produces faster than the other consumes — then, without flow control, you get unbounded memory growth and crashes. gRPC inherits HTTP/2's flow control to prevent exactly this. Backpressure is the difference between streaming that scales and streaming that falls over.
Most engineers reach for cryptography wanting "make this secure," but crypto doesn't provide "secure" — it provides four specific, separable guarantees, and using the wrong one (encrypting when you needed to authenticate, hashing when you needed to encrypt) is how most real-world crypto failures happen. This series is about using cryptography correctly as an engineer who builds on top of it, not about inventing it.
Most engineers reach for cryptography wanting 'make this secure' — but crypto doesn't provide 'secure.' It provides four specific, separable guarantees: confidentiality, integrity, authenticity, and non-repudiation. Using the wrong one is how most real-world crypto failures happen.
How to put Amazon Bedrock Guardrails in front of a model from Go — attaching one to a Converse call, screening raw text with ApplyGuardrail, and reading whether the guardrail actually intervened.
Guardrails for Amazon Bedrock in Go: content filters, denied topics, PII/sensitive-information filters, and contextual grounding — attaching a guardrail to a Converse call and screening arbitrary text with ApplyGuardrail, checking for intervention.
The last post in the series: what changes when the LLM system you built across posts 1-14 has to run for real — reliability, security, cost, observability, evaluation gates, and versioning, from a Go engineer's seat, with code where it earns its place.
The capstone: running an LLM system in production from a Go engineer's seat — reliability (timeouts, retries, fallbacks), security (injection, least-privilege tools, secrets), cost and observability, CI eval gates, and versioning models and prompts.
Errors are part of every API's contract, and gRPC has a specific, structured model for them: a fixed set of status codes, a message, and optional rich details — not the HTTP status codes you know from REST. Using this model well is what makes failures actionable for callers instead of opaque. Getting it wrong turns every error into a debugging session.
Errors are part of every API's contract, and gRPC has a specific, structured model: a fixed set of status codes, a message, and optional rich details — not the HTTP status codes you know from REST. Using it well makes failures actionable for callers instead of opaque; getting it wrong turns every error into a debugging session.
How to invoke a managed Agent for Amazon Bedrock from Go — where the server owns the reason-act loop, and your job is to call InvokeAgent, range the event stream, accumulate the answer chunks, and read the trace for observability.
Agents for Amazon Bedrock from Go: the managed reason-act loop that runs server-side (vs the DIY Converse loop), invoking an agent alias with InvokeAgent, streaming the response and trace events, and keeping multi-turn state with a SessionId.
Making an LLM system faster and cheaper without touching the weights — the levers an application engineer actually controls, from streaming and caching to token trimming, model routing, and Go's real superpower: concurrency with a rate limiter.
Make LLM systems faster and cheaper without retraining: TTFT vs throughput, exact and semantic response caching, prompt caching, token reduction, model routing and cascades, and Go concurrency with a worker pool and rate limiter.
The capstone of the Evaluating Agents in Go series: how to score a conversation instead of a single reply, how to attribute errors across a coordinator and its sub-agents, and how to build rubric, safety, and hallucination judges in Go when the framework hands you no eval package.
The capstone of the Evaluating Agents in Go series: how to score a conversation instead of a single reply, how to attribute errors across a coordinator and its sub-agents, and how to build rubric, safety,...
A production RPC is more than a request and a response. Every call should carry a deadline so it can't hang forever, metadata for cross-cutting concerns like auth and tracing, and it should pass through interceptors that apply logging, authentication, and metrics uniformly. These three mechanisms are how a gRPC system becomes observable, secure, and resilient — and they're the pieces beginners most often skip.
A production RPC is more than a request and response. Every call should carry a deadline so it can't hang forever, metadata for cross-cutting concerns like auth and tracing, and pass through interceptors that apply logging, authentication, and metrics uniformly. These three mechanisms make a gRPC system observable, secure, and resilient.
How to query a Knowledge Base for Amazon Bedrock from Go — the managed retrieve-then-read layer — using both the low-level Retrieve call and the one-shot RetrieveAndGenerate, with citations wired through.
RAG on Bedrock in Go with Knowledge Bases: the retrieve-then-read pattern via Retrieve, the one-shot managed path via RetrieveAndGenerate with citations, and when to reach for each — plus reading grounding so you keep RAG's trust benefit.
How to know whether an LLM system actually works — building an eval dataset, the four metric families (deterministic checks, text overlap, embedding similarity, LLM-as-judge) in Go, task-specific eval for RAG and classification, and wiring a scored regression gate into CI so you measure instead of vibe.
How to know whether an LLM system works when outputs are non-deterministic: build an eval dataset, score with deterministic checks, embedding similarity, and LLM-as-judge (with its biases), evaluate RAG and classification, and gate regressions in CI.
How to wire agent evaluations into continuous integration in Go — running a slow, model-calling eval harness under `go test`, setting per-metric thresholds that fail the build on a regression, and living honestly with the fact that these gates are softer than unit tests.
How to wire agent evaluations into continuous integration in Go — running a slow, model-calling eval harness under `go test`, setting per-metric thresholds that fail the build on a regression, and living...
The magic that makes an RPC feel like a local function call is code generation. You run a compiler over your `.proto` file and out come typed classes and methods for your language — the client stub you call and the server interface you implement. Understanding what's generated, and the boundary between generated and hand-written code, is what turns gRPC from a black box into a tool you control.
The magic that makes an RPC feel like a local function call is code generation. Run a compiler over your .proto and out come typed classes and methods — the client stub you call and the server interface you implement. Understanding what's generated, and the boundary between generated and hand-written code, turns gRPC from a black box into a tool you control.
How to give an Amazon Bedrock model real Go functions — declaring tools, catching the tool-use stop reason, executing your code, and returning results — using the full round-trip loop in aws-sdk-go-v2.
Giving a Bedrock model tools in Go via the Converse API: declaring a ToolConfiguration, the ToolUse round-trip loop, echoing ToolUseId, returning tool results as a user message, and handling parallel tool calls.
Give the hand-rolled Go agent from post 11 a memory it can carry between turns and a plan it can follow across many steps — a compacting conversation buffer, retrieval over the post-8 vector store, and a plan-then-execute-then-reflect loop, all built from scratch.
Give the agent memory and planning in Go: a compacting short-term conversation buffer, long-term memory as timestamped embeddings in the vector store, and planning — plan-then-execute, reflection and re-planning when observations contradict the plan, and task decomposition.
Where good eval cases actually come from — seeding by hand, harvesting from production telemetry, and curating a golden dataset in Go that doesn't rot the moment your prompt changes.
Where good eval cases actually come from — seeding by hand, harvesting from production telemetry, and curating a golden dataset in Go that doesn't rot the moment your prompt changes.
gRPC isn't just request-and-response. Because it rides on HTTP/2, it offers four distinct call patterns: unary, server streaming, client streaming, and bidirectional streaming. Each fits a different shape of problem, and choosing the right one is a real design decision — it changes how your API feels, how it performs, and how it handles data that arrives over time rather than all at once.
gRPC isn't just request-and-response. Riding on HTTP/2, it offers four call patterns: unary, server streaming, client streaming, and bidirectional. Each fits a different shape of problem, and choosing the right one is a real design decision — it changes how your API feels, performs, and handles data that arrives over time rather than all at once.
How to stream Amazon Bedrock responses token-by-token with the aws-sdk-go-v2 Converse API, decode the event stream with a double type-switch, and account for tokens and cost from the metadata event — accurately, in Go.
Streaming responses and accounting for tokens and cost on Bedrock in Go: ranging the ConverseStream event stream, the nested delta unions, checking stream.Err(), and computing cost from the metadata usage event with a formula you fill in.
Building a real agent loop in Go by hand — an LLM in a loop that picks tools, runs them, reads the results, and repeats until the task is done — so you can see there is no magic behind LangGraph, MAF, or ADK.
Build a minimal but real agent loop in Go by hand: an Agent with a tool registry and a reason-act Run loop, an iteration budget, validation against hallucinated tools, feeding tool errors back as observations, and parallel tool calls — the loop frameworks formalize, demystified.
How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating Agents in Go.
How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating...
Protocol Buffers are the foundation gRPC is built on — both the language you write your API contract in and the binary format your data travels as. A `.proto` file is a strict, versioned schema; the encoding is a compact, tag-based binary that's a fraction of JSON's size. Understanding both halves — the schema language and how it serializes — is what lets you design APIs that stay compatible as they evolve.
Protocol Buffers are gRPC's foundation — both the language you write your API contract in and the compact binary format your data travels as. A .proto file is a strict, versioned schema; the encoding is tag-based binary a fraction of JSON's size. Understanding both halves is what lets you design APIs that stay compatible as they evolve.
Your first real inference call in Go against Amazon Bedrock — using the unified, model-agnostic Converse API and the AWS SDK for Go v2, from client construction to reading tokens back off the response.
Your first real inference call on Bedrock in Go via the unified Converse API: building the client, the ConverseInput message/content-block union, extracting the assistant text, and reading stop reason and token usage — with the content-block union explained.
Why the naive RAG pipeline from post 9 underperforms in production, and the concrete, evaluation-driven fixes — structure-aware chunking, hybrid search, reranking, query transformation, and deliberate context construction — each explained with the reasoning and a real Go sketch.
Why naive RAG underperforms and the techniques that fix it: measure recall@k first, then structure-aware chunking, hybrid dense+BM25 search fused with RRF, over-retrieve-then-rerank, query transformation (HyDE, multi-query), and deliberate context construction against lost-in-the-middle.
How to score what an agent did, not just what it said — building trajectory metrics in Go from an exact-match baseline up to arg-aware, order-tolerant scoring, with a readable diff of expected vs. actual.
How to score what an agent did, not just what it said — building trajectory metrics in Go from an exact-match baseline up to arg-aware, order-tolerant scoring, with a human-readable diff of expected vs....
REST won the public web, but inside a system — between the dozens of services that make up a modern backend — a different model dominates: RPC. Instead of modeling everything as resources and verbs over HTTP, you call a remote function as if it were local. gRPC is the modern, high-performance realization of that idea, built on Protocol Buffers and HTTP/2. Understanding why it exists explains when to reach for it over REST.
REST won the public web, but inside a system — between the dozens of services in a modern backend — RPC dominates: you call a remote function as if it were local. gRPC is the modern, high-performance realization of that idea, built on Protocol Buffers and HTTP/2. Understanding why it exists explains when to reach for it over REST.
The opener for a Go series on building LLM and agent applications with Amazon Bedrock — what the service actually is, why it sits between your Go code and a dozen foundation models, and which aws-sdk-go-v2 packages you will lean on for the rest of the way.
The opener to a series on building LLM and agent applications on Amazon Bedrock in Go: what Bedrock actually is, what it adds over calling a provider API directly (one API across models, IAM auth, data residency), and the aws-sdk-go-v2 packages you'll use.
Wire the embedding client, vector store, and chat client from the last five posts into one working RAG pipeline in Go — ingest and chunk documents, retrieve the top matches for a question, inject them as grounded context, and generate a cited answer, all from scratch.
Assemble embeddings and vector search into a working RAG pipeline in Go: chunk documents, embed and store them, retrieve the top-k for a query, augment the prompt with grounded context (and cite sources), then generate — a baseline end-to-end Answer() built from scratch.
The core of the series: a minimal, original evaluation harness in Go. Run an agent under test through adk-go's runner, capture the tool-call trajectory and the final response behind an adapter you own, and score them with `go test`.
The core of the series: a minimal, original evaluation harness in Go. Run an agent under test through adk-go's runner, capture the tool-call trajectory and the final response behind an adapter you own, and...
Build a working in-memory vector store and exact k-nearest-neighbor search in Go by hand — no vector database — then understand precisely what HNSW, FAISS, and pgvector optimize when brute force finally runs out of road.
Build an in-memory vector store and exact k-NN search in Go by hand: a VectorStore with Add and Search, top-k selection with container/heap, normalize-on-insert, an honest look at when brute force is right, and when ANN (HNSW, FAISS, pgvector) earns its keep.
Before you can evaluate an agent in Go, you need a mental model of what "evaluating an agent" even means. This post unpacks the conceptual core of Google's Agent Development Kit eval framework — cases, trajectories, metrics, thresholds — the parts that are language-agnostic, so the rest of this series can implement them as plain Go types and functions.
Before you can evaluate an agent in Go, you need a mental model of what "evaluating an agent" even means. This post unpacks the conceptual core of Google's Agent Development Kit eval framework — cases,...
Turn text into a `[]float32` that places meaning in space — what an embedding is, cosine similarity implemented by hand in Go, calling an OpenAI-compatible /embeddings endpoint with net/http, and a worked pairwise-similarity example that scores related sentences higher.
Turn text into a []float32 that places meaning in space — what an embedding is, cosine similarity implemented by hand in Go, calling an OpenAI-compatible /embeddings endpoint with net/http, and a worked pairwise-similarity example that scores related sentences higher.
The opener to a series on evaluating agents in Go: why an agent isn't a function you can unit-test, why "it worked in the demo" doesn't survive contact with production, and the two things actually worth measuring — the steps it took and the answer it gave.
The opener to a series on evaluating agents in Go: why an agent isn't a function you can unit-test, why "it worked in the demo" doesn't survive contact with production, and the two things actually worth...
Almost everything interesting in JavaScript is asynchronous — network calls, file reads, timers — and TypeScript types all of it through one generic: `Promise<T>`. But typing async code well means understanding the runtime it describes: the single-threaded event loop that makes non-blocking concurrency work. Types and runtime together are what let you write async code that's both correct and comprehensible.
Almost everything interesting in JavaScript is asynchronous, and TypeScript types all of it through one generic: Promise<T>. But typing async code well means understanding the runtime it describes — the single-threaded event loop that makes non-blocking concurrency work. Types and runtime together let you write async code that's correct and comprehensible.
Treating the prompt as a real engineering artifact — grounded in how a next-token predictor actually works — with roles, specificity, few-shot examples, decomposition, chain-of-thought, grounding, temperature, injection defense, and versioned Go templates you can test.
Prompt engineering as a real engineering discipline: roles, specificity, few-shot, decomposition, chain-of-thought and its cost, grounding, and prompt injection — plus building prompts as versioned, testable Go text/template templates you treat like code.
From-scratch Go for the two mechanisms that turn an LLM from a text generator into a component you can wire into real software — schema-constrained JSON and function calling — both spoken over the same OpenAI-compatible chat JSON.
Getting reliable machine-readable output from an LLM in Go: structured output (json-schema mode, decode into a typed struct, validate with a bounded retry) and tool/function calling (the full round-trip loop, decoding tool arguments, returning results tied to the call id).
Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming — no framework required.
Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming.
The unit a language model actually reads is neither a word nor a character — it is a token, and once you see the world the way the model does, half of its strange behavior stops being strange.
The unit a language model actually reads is neither a word nor a character but a token. How byte-pair encoding builds a vocabulary, why tokenization explains half of an LLM's strange behavior, and how to count tokens exactly in Go.
The working mental model an AI engineer needs — next-token prediction, attention, training, and sampling — without the transformer math, and with every fact tied back to a decision you make in code.
The working mental model an AI engineer needs — next-token prediction, attention at an intuition level, pretraining vs post-training, and sampling — with every fact tied back to a concrete decision you make in code.
The opener to a from-scratch series on building applications on top of foundation models in Go — what AI engineering actually is, how it differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.
The opener to a from-scratch AI-engineering-in-Go series: what AI engineering actually is, how building on foundation models differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.
Lesson 7 of Harness Engineering in Go — a sensitive action pauses for a human decision, and the whole suspension is nothing more than a Lesson 2 checkpoint marked awaiting_approval.
Series finale, Lesson 7: a sensitive action pauses for human approval, where suspension is just a Lesson 2 checkpoint marked awaiting_approval, the deadline is checked first so a late yes is void, and the action must be idempotent.
Lesson 6 of Harness Engineering in Go — a supervisor splits a task, fans out to concurrent workers behind a semaphore, and fans the results back in decomposition order, with each worker's failure (or panic) isolated to one result.
Lesson 6: bounded fan-out behind a semaphore, ordered fan-in via a pre-sized results slice, and per-worker fault isolation so one sub-agent panicking becomes one failed result instead of crashing the whole run.
Lesson 5 of Harness Engineering in Go — a triage step that first-matches a keyword and hands the request to a specialist, and the exact place a substring table stops being able to think.
Lesson 5: a triage router first-matches a keyword to hand intent to a specialist, and the exact point a substring table stops being able to think.
Lesson 4 of Harness Engineering in Go — three collaborating stores (a thread, a knowledge index, and a summarizer) behind interfaces, and an honest accounting of where each local stand-in leaks.
Lesson 4: memory is three stores, not one — an append-only thread, a keyword knowledge index, and a lossy first-and-last summarizer — and an honest account of where each local stand-in leaks against Azure.
Lesson 3 of Harness Engineering in Go — how a context deadline and `exec.CommandContext` reap a runaway snippet, why the two-shaped `Result` distinguishes a timeout from a failure, and the leak that makes a local subprocess a teaching tool, not a security boundary.
Lesson 3: run agent-written code behind a hard timeout with exec.CommandContext, distinguish OK from TimedOut, and face the leak — a subprocess is not a security boundary.
How ADK's config loader turns a declarative YAML file into a fully-built agent — and why treating an agent as data changes who gets to edit it.
Defining an agent declaratively in YAML and loading it via from_config — the loader reads, resolves, and validates the config into a built agent, so config-as-data works without writing code.
Lesson 2 of Harness Engineering in Go — a workflow that saves its progress after each step and picks up exactly where it died, proven by a test that kills a real subprocess mid-run.
Lesson 2: a workflow that checkpoints after every step and resumes from the last one after a crash, why at-least-once execution forces idempotent steps, and the atomic-rename store that survives a killed process.
How caching a large, stable prompt prefix cuts latency and cost — and the ADK config that decides when it pays off.
Context caching cuts latency and cost by caching large, stable context — system prompt, reference docs, tool definitions — so repeated calls don't re-send and re-process the same tokens.
Lesson 1 of Harness Engineering in Go — why the input guardrail is a hard block, not a warning, and how a plain `net/http` handler wraps the model call so it tests without a running server.
Lesson 1: why the input guardrail is a hard block rather than flag-and-pass, why it counts runes instead of bytes, and how a plain net/http handler wraps the (stubbed) model call so it tests with httptest.
Post 23 of 26 in "Google ADK, Concept by Concept" — how a planner turns one-shot answers into inspectable plan-then-act reasoning.
Structuring an agent's reasoning: planners that make the model plan-then-act (ReAct-style), the built-in thinking feature, and how a planner improves multi-step tool use over naive prompting.
The final lesson: a recipe agent whose JSON replies are turned into trackable state snapshots by a middleware, so the client can render the recipe as it evolves.
The server side of AG-UI state management: middleware emits a DataContent state snapshot from the model's JSON so the client can adopt shared state across turns.
Seven patterns that turn a bare model call into production agent infrastructure — each written first as offline Go behind an interface, so the leap to Azure is a swap, not a rewrite.
Seven patterns that turn a bare model call into production agent infrastructure, each written first as offline Go behind an interface (the seam) so the leap to Azure is a swap, not a rewrite.
How ADK closes the write-code, run-it, read-the-output loop — and why "unsafe" is a warning, not a typo.
Letting an agent write and run code: built-in and container-based code executors, safe sandboxed execution, how results flow back into the conversation, and the security tradeoffs.
The server hosts an agent with one approval-gated tool — the model may propose calling it, but the framework refuses to run it until a human on the other end says yes.
The server side of AG-UI human-in-the-loop: gate a tool with tool.ApprovalRequiredFunc so the server suspends the run and resumes on the client's decision.
How Go's `context` package carries a cancellation signal, a deadline, and a small bag of request-scoped values across every API and goroutine boundary in a request — and the handful of rules that keep it from leaking or lying to you.
How Go's `context` package carries a cancellation signal, a deadline, and a small bag of request-scoped values across every API and goroutine boundary in a request — and the handful of rules that keep it...
How ADK skills bundle instructions, tools, and resources into folders an agent can browse and load on demand.
Skills package reusable capabilities — instructions, tools, resources, including file-based skills — so they can be discovered and attached to agents, promoting reuse across projects.
The mirror image of backend tools: the server hosts the agent but the tools live on the client, so the handler must forward tool calls instead of running them.
The server side of AG-UI frontend tools: set DisableFuncAutoCall so the server forwards tool calls to the client to execute and awaits the result.
Wrapping Go agents in A2A, MCP, and AG-UI transports, then a full client/server app that ties the whole series together.
Wrap Go agents in A2A, MCP, and AG-UI transports, then run the end-to-end client/server capstone where a host agent uses remote specialists as tools.
What a data race actually is, why it is undefined behavior rather than "just a wrong number," the happens-before rules that make concurrent code correct, and when `sync/atomic` is the right tool — and when it quietly is not.
What a data race actually is, why it is undefined behavior rather than "just a wrong number," the happens-before rules that make concurrent code correct, and when `sync/atomic` is the right tool — and when...
cross-cutting concerns registered once on the Runner instead of copied onto every agent
Plugins are cross-cutting hooks that apply globally across every agent, tool, and runner — logging, policy, metrics, caching — as opposed to per-agent callbacks. When a plugin beats a callback.
Same AG-UI transport as the getting-started server, but now the hosted agent carries a tool the model can call server-side while it answers.
The server side of AG-UI backend tools: attach server-owned function tools to the Foundry agent so the tool round-trip stays entirely server-side.
Durable workflows in Go: checkpoint and rehydrate, pause on a RequestPort for a human, nest sub-workflows, and coordinate through scoped shared state.
Durable Microsoft Agent Framework workflows in Go: checkpoint and rehydrate a fresh graph, pause on a RequestPort for a human, nest sub-workflows, and coordinate via scoped shared state.
When channels are the wrong tool — a working guide to shared memory and locks in Go: Mutex, RWMutex, WaitGroup, Once, Cond, Map, Pool, and the race detector that keeps you honest.
When channels are the wrong tool — a working guide to shared memory and locks in Go: Mutex, RWMutex, WaitGroup, Once, Cond, Map, Pool, and the race detector that keeps you honest.
How ADK's model abstraction lets you swap Gemini for Claude, GPT, or Ollama without touching a line of agent code
ADK is model-agnostic: use Gemini natively or plug in Claude, GPT, or Ollama via LiteLLM and a model registry, swapping the model without changing agent code.
The client half streams events over SSE; this is the process on the other end of the wire — a Foundry agent wrapped in an HTTP handler that speaks AG-UI.
The AG-UI server half: wrap a Foundry agent in aguiprovider.NewJSONHTTPHandler and serve it over HTTP/SSE for any AG-UI client to drive.
The prebuilt orchestration builders in agent-framework-go — Sequential, Concurrent, Group Chat — plus wrapping a whole workflow as one agent.
The Sequential, Concurrent, and Group Chat orchestration builders in agent-framework-go, plus wrapping a whole workflow as one nestable agent.
A working guide to Go's channels — unbuffered rendezvous vs buffered capacity, send/receive/close semantics, directional types in APIs, and how `select` multiplexes, times out, and disables cases with a nil channel.
A working guide to Go's channels — unbuffered rendezvous vs buffered capacity, send/receive/close semantics, directional types in APIs, and how `select` multiplexes, times out, and disables cases with a nil...
Post 18 of 26 in "Google ADK, Concept by Concept" — retrieval tools, grounding metadata, rendering citations, and the retrieve→augment→generate loop.
Grounding answers in real data: retrieval tools, grounding metadata returned with responses, rendering citations from that metadata, and the retrieve-augment-generate RAG pattern in ADK.
The final lesson ties the whole Go tutorial into one small product: an assistant that answers questions about your docs — grounded, cited, and refusing to guess.
The capstone: a grounded DocQA agent that answers only from embedded docs via a search_docs tool, cites sources, and refuses to guess — with an optional reviewer.
Agents as graph executors in Go: switch routing, fan-out/fan-in barriers, and mixing typed function nodes with agent nodes.
Agents as Go workflow executors: switch routing, fan-out and fan-in barriers, and mixing typed function nodes with agent nodes in one graph.
What a goroutine actually is, why it is cheaper than a thread, and the one rule that separates working concurrent code from a program that quietly leaks itself to death: every goroutine you start must have a way to stop.
What a goroutine actually is, why it is cheaper than a thread, and the one rule that separates working concurrent code from a program that quietly leaks itself to death: every goroutine you start must have...
Two open protocols that let an agent reach outside its own process — one to borrow tools, one to call other agents as peers.
Two interoperability protocols: MCP lets an agent consume tools from external servers, and A2A — HTTP for agents — lets one agent discover and call another remote agent as a peer over HTTP.
This lesson hosts one specialized Foundry agent over the A2A protocol, publishing an agent card the client can discover.
Host one Foundry agent over A2A: newMux pins the card interface URL, wraps the agent in an a2aprovider executor, and serves the card plus JSON-RPC routes.
The graph model underneath every multi-agent app: executors as nodes, edges as data flow, and typed events streaming out of `WatchStream` as it runs.
The Microsoft Agent Framework workflow model in Go: executors bound to IDs, AddEdge wiring, WithOutputFrom, and typed WatchStream events - plus an upstream route-builder fix.
Go handles ordinary failure with values, not exceptions — so what are panic and recover actually for? A working guide to how panic unwinds the stack through your defers, why recover only fires inside a deferred function, and the narrow set of places where catching a panic is the right call rather than a code smell.
Go handles ordinary failure with values, not exceptions — so what are panic and recover actually for? A working guide to how panic unwinds the stack through your defers, why recover only fires inside a...
Stack the guardrails — callbacks, model filters, restricted tools, and clean-room sandboxing — so that if one layer misses, the next one catches
Layered defense-in-depth for agents: input/output guardrails via callbacks, Gemini safety settings, restricting tools, and sandboxing untrusted actions.
This lesson builds a Foundry host agent that discovers remote A2A agents by their cards and calls each one as a tool.
A Foundry host agent resolves remote A2A agent cards and turns each remote agent into a callable tool with agenttool.New over the a2aprovider.
Wrap every run in an OpenTelemetry span, gate risky tool actions behind a permission handler, and swap model providers behind one agent.Agent.
Wrap every Microsoft Agent Framework run in an OpenTelemetry span, gate risky tool actions behind a permission handler, and swap Anthropic, OpenAI, Gemini, Copilot, and Azure behind one agent.
Go treats errors as ordinary values, not exceptions — which means everything you know about passing, comparing, and inspecting values applies. This is a working guide to sentinel errors, wrapping with %w, and the two verbs that make error chains navigable: errors.Is and errors.As.
Go treats errors as ordinary values, not exceptions — which means everything you know about passing, comparing, and inspecting values applies. This is a working guide to sentinel errors, wrapping with %w,...
How OpenTelemetry traces, structured logs, and token metrics turn an agent's event stream into something you can debug in production.
Seeing inside a running agent: OpenTelemetry tracing with spans for agent, model, and tool steps, structured logging, and exporting traces to debug latency and tool-call trajectories.
How a whole workflow binds as a single executor inside a larger workflow, so pipelines compose two levels deep.
inproc.BindSubworkflowAsExecutor binds a whole workflow as one node; an order pipeline nests Payment and FraudCheck two levels deep, and inner events still bubble up.
A guardrail that blocks a run before the model, chained ahead of a logger — and the upstream nil-update panic I fixed along the way.
A guardrail middleware that blocks an Microsoft Agent Framework Go run before the model, chained ahead of a logger — plus the tool-approval nil-update panic I fixed upstream in PR #472.
How type parameters and constraints actually work in Go 1.18+ — writing functions and data structures that are type-safe across many types, when the compiler can infer type arguments for you, and the harder question of when a plain interface is still the better tool.
How type parameters and constraints actually work in Go 1.18+ — writing functions and data structures that are type-safe across many types, when the compiler can infer type arguments for you, and the harder...
How `adk deploy` builds, pushes, and ships an agent in a single step — and the ack-after-invocation rule that keeps event-driven agents reliable.
Deploying an agent: adk deploy with its cloud_run and agent_engine subcommands, containerizing the app, and reliability rules like ack-after-invocation so failures are redelivered, not dropped.
How one executor stores a value in scoped shared state and later executors read it back — instead of copying a whole payload down every edge.
One executor stores a document with QueueStateUpdate; fan-out counters read it back with ReadState, and an AddFanInBarrierEdge aggregates once both deliver. Offline.
Decoding typed structs from a run, draining a streamed response, and sending an image with `RunMessage`.
Decode typed structs from a run, drain a streamed response by ranging it, and send an image with RunMessage and DataContent.
How Go recovers a concrete type from an interface value, why the comma-ok form exists, and the two-word memory layout that explains the single most surprising bug in the language — the non-nil interface holding a nil pointer.
How Go recovers a concrete type from an interface value, why the comma-ok form exists, and the two-word memory layout that explains the single most surprising bug in the language — the non-nil interface...
How ADK turns "did the agent behave correctly?" into a number you can gate a merge on.
Measuring agent quality: eval sets, scoring both the trajectory (right tools, right order) and the final response, criteria configs, and the adk eval CLI — a Python-first workflow today.
How WithTelemetry instruments a whole workflow with OpenTelemetry spans, then agentworkflow.NewAgent wraps the graph so it behaves like one agent.
WithTelemetry instruments a French-to-English workflow with OpenTelemetry spans, then agentworkflow.NewAgent wraps the graph so RunText drives the whole pipeline.
A Session threads history into each run; a ContextProvider carries memory across sessions — and because a Session is JSON, both survive a process restart.
A Session threads history into each run and a ContextProvider carries memory across sessions. Because a Session is JSON, both survive a process restart.
How Go turns "what a value can do" into a first-class type — with implicit satisfaction, small contracts, the consumer-defined-interface rule, and the typed-nil trap that catches everyone once.
How Go turns "what a value can do" into a first-class type — with implicit satisfaction, small contracts, the consumer-defined-interface rule, and the typed-nil trap that catches everyone once.
token streaming, the accumulate-and-reconcile consumer pattern, and full-duplex live streaming for voice
Consuming output as it is produced: partial events and token streaming, the accumulate-and-reconcile consumer pattern, and bidi/live streaming for voice and interactive UIs.
How a workflow graph forms a cycle: two executors feed each other until one yields an output — the workflow analogue of a while-loop.
A workflow graph is not a DAG: two edges form a GuessNumber to Judge cycle that runs a binary search and exits only when Judge yields an output. The while-loop analogue.
Wrap a typed Go function as a tool, let the model call it, and compose whole agents as tools.
Wrap a typed Go function as a tool with functool.MustNew, let the model call it, and compose whole agents as tools with agenttool.New.
How Go attaches behavior to types without classes — the receiver, the value-versus-pointer decision, method sets and what they mean for interfaces, and the addressability rules that trip people up when a value lives in a map.
How Go attaches behavior to types without classes — the receiver, the value-versus-pointer decision, method sets and what they mean for interfaces, and the addressability rules that trip people up when a...
How an agent actually runs — a Runner drives an invocation and hands you back a stream of events, not a single answer.
How ADK runs an agent: the Runner drives an invocation that yields a stream of Event objects — content, tool calls, state deltas, control signals. The event loop explains streaming, callbacks, and state.
How a RequestPort pauses a workflow, emits a request to a human, and feeds their answer back into the graph.
A RequestPort emits a RequestInfoEvent and suspends the workflow; the driver answers with SendResponse. A guess-the-number cycle shows the pause/resume mechanism.
The minimal loop in Go: a Foundry provider, an Agent with instructions, run collected and streamed — and what RunText hands back.
The minimal Microsoft Agent Framework loop in Go: a foundryprovider agent, DefaultAzureCredential, and one RunText you either Collect or range over as a Go 1.23 iterator.
How Go builds aggregate types from value semantics up — why a struct is a copy, when it stops being comparable, what embedding actually promotes (and what it deliberately doesn't), and how a backtick string in a field definition ends up steering `encoding/json`.
How Go builds aggregate types from value semantics up — why a struct is a copy, when it stops being comparable, what embedding actually promotes (and what it deliberately doesn't), and how a backtick string...
before/after the agent, model, and tool steps — and the single short-circuit rule that turns them into guardrails
Callbacks are lifecycle hooks around the agent, model, and tool steps — before/after each — used for guardrails (short-circuit by returning a response), logging, and mutating requests and responses.
How an edge assigner delivers one message to a chosen subset of targets — a multi-way switch that may fall through to more than one case.
WithEdgeAssigner yields target indexes via an iter.Seq[int], so a single message reaches several branches at once — a long email goes to both assistant and summary, offline.
Why I learned the whole framework in Go by writing one runnable lesson per concept, against Azure AI Foundry, instead of reading the docs top to bottom.
I learned the whole Microsoft Agent Framework in Go by building one runnable lesson per concept against Azure AI Foundry. Here is the 12-track map.
How Go's built-in hash table really behaves — reference semantics, the nil-write panic, comma-ok, randomized iteration, why `&m[k]` is illegal, and the presizing and concurrency rules that separate correct map code from the code that bites you at 2 a.m.
How Go's built-in hash table really behaves — reference semantics, the nil-write panic, comma-ok, randomized iteration, why `&m[k]` is illegal, and the presizing and concurrency rules that separate correct...
ReadonlyContext, CallbackContext, ToolContext, InvocationContext — and why the read-only vs mutable distinction is a feature, not a limitation.
The context objects ADK passes into tools and callbacks — InvocationContext, ToolContext, CallbackContext, ReadonlyContext — what each exposes and why the read-only vs mutable split matters.
How a switch builder fans one edge into three mutually-exclusive branches — the workflow analogue of switch/case/default.
AddSwitch/AddCase/WithDefault fans one edge into three mutually-exclusive branches with a guaranteed fallback — the workflow analogue of switch/case/default, fully offline.
Why an array is a value and a slice is a view — the three-word header, how `append` really grows, the aliasing trap that silently corrupts data, and the small habits (three-index slices, `copy`, pre-sizing) that keep it from biting you.
Why an array is a value and a slice is a view — the three-word header, how `append` really grows, the aliasing trap that silently corrupts data, and the small habits (three-index slices, `copy`, pre-sizing)...
Session state is for small text and JSON. When your agent produces a PNG, a PDF, or a WAV, it belongs in the artifact store — binary-native, versioned, and out of the session record.
Artifacts are binary/file data agents produce or consume: ArtifactService saves and versions named artifacts, loaded and saved via context, keeping large blobs out of session state.
How a graph workflow forks: a predicate on each edge decides whether a message may pass, turning the graph into an if/else expressed as data flow.
The first 03-workflows lesson: AddDirectEdge attaches a func(any) bool to each edge so a spam-detection graph forks into send vs. handle-spam branches, offline.
What a Go string actually is under the hood — an immutable read-only slice of bytes, not a sequence of characters — and how bytes, runes, and code points relate, so you stop shipping the classic multibyte bugs.
What a Go string actually is under the hood — an immutable read-only slice of bytes, not a sequence of characters — and how bytes, runes, and code points relate, so you stop shipping the classic multibyte bugs.
State remembers things inside one chat; Memory is the searchable archive that lets an agent recall what you told it weeks ago.
Memory is long-term recall across sessions, distinct from per-session state: MemoryService stores and retrieves prior context, surfaced via a recall tool so an agent remembers a user over time.
How to express the classic MapReduce word-count as a five-stage workflow graph — fan-out to mappers, barrier to a shuffler, fan-out to reducers, barrier to completion.
MapReduce word-count as a five-stage workflow: fan-out to mappers, barrier to a shuffler, fan-out to reducers, barrier to completion, coordinated via shared state.
What a Go pointer actually is, why there's no pointer arithmetic, `new(T)` versus `&T{}`, the addressability rules that decide what `&` will even compile against, and when reaching for a pointer helps versus when it just adds indirection and GC pressure.
What a Go pointer actually is, why there's no pointer arithmetic, `new(T)` versus `&T{}`, the addressability rules that decide what `&` will even compile against, and when reaching for a pointer helps...
A `Session` is the conversation; `state` is the key-value bag agents and tools read and write — and the prefix on a key decides how long it lives.
A Session holds a conversation; state is a scoped key-value store (session/user/app/temp) read and written by tools, injected into instructions via {state} templating, and persisted by SessionService.
How to broadcast one input to several executors in parallel and join their answers with a barrier — the core workflow graph primitives, with no LLM in the way.
A start executor broadcasts a question to two experts via a fan-out edge; a fan-in barrier edge joins both answers before the aggregator yields output.
How Go treats functions as ordinary values — and what that buys you: the (result, error) idiom, variadic APIs, closures over shared state, and the decorator/middleware/option patterns that fall out of passing functions around.
How Go treats functions as ordinary values — and what that buys you: the (result, error) idiom, variadic APIs, closures over shared state, and the decorator/middleware/option patterns that fall out of...
How a plain function becomes a callable tool, how ToolContext reaches session state, and how long-running tools pause a run for a human.
Tools give agents capabilities: a plain function becomes a tool with its signature as the schema, plus ToolContext, built-in tools, and long-running/human-in-the-loop tools across Python and Go.
How to pause a workflow to ask a human, checkpoint every super-step, then rewind the whole graph to a saved checkpoint and replay.
A RequestPort pauses to ask a human for a guess while every super-step is checkpointed, then RestoreCheckpoint rewinds the whole graph including tries state.
Go's control flow is deliberately small — one loop keyword, a switch that doesn't fall through, an `if` that can scope its own variable — and then there's `defer`, the one construct that repays close reading. A tour of the whole surface, with the sharp edges labelled.
Go's control flow is deliberately small — one loop keyword, a switch that doesn't fall through, an `if` that can scope its own variable — and then there's `defer`, the one construct that repays close...
How one agent routes work to specialists — and why the description field is the most important string you write.
Agent hierarchies and LLM-driven delegation: sub_agents, how the description field drives auto-transfer, and coordinator/dispatcher patterns — contrasted with deterministic workflow agents.
How to snapshot a workflow after every super-step, then rewind the same live run to an earlier checkpoint and replay from it.
Same guess-the-number graph, but RestoreCheckpoint rewinds the live run to the 6th snapshot and replays forward — no fresh workflow instance needed.
How Go's declaration forms, scope rules, and its unusual constant system fit together — including the untyped-constant model that makes numeric literals feel effortless, and the `iota` patterns that turn enums and bit-flags into a few tidy lines.
How Go's declaration forms, scope rules, and its unusual constant system fit together — including the untyped-constant model that makes numeric literals feel effortless, and the `iota` patterns that turn...
When you want fixed control flow, don't ask the model — wire it yourself.
Sequential, Parallel, and Loop agents compose sub-agents in fixed patterns — deterministic orchestration where you, not the model, decide control flow, with state flowing between steps.
How to snapshot a running workflow at every super-step, then throw the instance away and rebuild a fresh workflow that resumes from a saved checkpoint.
A cyclic guess-the-number workflow snapshots state each super-step, then a brand-new workflow instance is rehydrated from a checkpoint via ResumeStreaming.
How Go's type system actually behaves — predeclared types, the zero-value guarantee that removes a whole class of null bugs, the "no implicit conversions" rule and why it exists, and the difference between a named type and a mere alias.
How Go's type system actually behaves — predeclared types, the zero-value guarantee that removes a whole class of null bugs, the "no implicit conversions" rule and why it exists, and the difference between...
description, instruction, generation params, and structured output — the dials on almost every agent you'll build
The four knobs on almost every ADK agent: description (for delegation), instruction with {state} templating, generation params, and structured output — Python Pydantic model vs Go genai.Schema.
How to collapse a whole multi-agent workflow into a single agent so callers use it exactly like any leaf agent.
A concurrent French+English workflow is hosted as one agent via agentworkflow.NewAgent, with IncludeOutputsInResponse surfacing the merged output.
Why Go is shaped the way it is, and how its toolchain — go run, build, test, fmt, vet, mod, doc — turns a small language into a fast, predictable team workflow.
Why Go is shaped the way it is, and how its toolchain — go run, build, test, fmt, vet, mod, doc — turns a small language into a fast, predictable team workflow.
An `LlmAgent`, a `Runner`, a `Session`, and a CLI that runs it all — the four pieces the other 25 concepts sit on top of.
The smallest ADK agent and the machinery around it: an LlmAgent, the Runner, a Session, and the adk CLI (adk web / adk run) that runs your agent with a dev UI or REPL — no server code.
How a multi-agent group chat pauses to ask a human for approval before one gated tool is allowed to run.
A QA and DevOps agent collaborate in a group chat where DeployToProduction is gated by tool.ApprovalRequiredFunc, pausing the workflow for human approval.
How two Foundry agents cooperate inside a cyclic graph workflow where one of them owns the loop control.
Two Foundry agents cooperate in a cyclic workflow: a SloganWriter drafts, a FeedbackProvider critiques and owns loop control via YieldOutput vs SendMessage.
This lesson teaches how to build a cyclic workflow where a Writer and Critic loop until approval, using `AddSwitch` to route on structured output.
A cyclic Writer-Critic-Summary workflow: the Critic emits a structured CriticDecision, AddSwitch routes on Approved, and Context state caps the revision loop at maxIterations.
This lesson teaches how deterministic function executors and agent-backed executors compose in one graph with the same `AddEdge` wiring.
One workflow that mixes deterministic executors with two Foundry agent nodes for jailbreak detection and response, joined by the same AddEdge wiring and TurnToken triggering.
This lesson teaches how to embed a whole built workflow as a single executor inside a larger one — workflows compose.
Embed an entire built workflow as one executor with inproc.BindSubworkflowAsExecutor, composing a Prefix to SubWorkflow to PostProcess parent graph that runs fully offline.
This lesson teaches how to chain three role-specialised agents — researcher → fact_checker → reporter — into one sequential workflow and stream each stage.
A sequential workflow of three role-specialised Foundry agents — researcher, fact_checker, reporter — built with NewSequentialWorkflowBuilder and streamed stage by stage.
This lesson teaches that orchestration is a property of the workflow, not the agents — the same three agents drop into three different built-in graph shapes.
The same three agents dropped into three built-in agentworkflow builders — sequential, concurrent, and round-robin group chat — showing orchestration is a property of the graph, not the agents.
This lesson teaches how to host real `agent.Agent`s as workflow executors and chain them into a sequential pipeline.
Host three translation agents as workflow executors and chain them French to Spanish to English, using DisableForwardIncomingMessages and a TurnToken to drive a strict pipeline.
This lesson teaches how to build a two-executor pipeline and watch it run as a live stream of typed events.
Your first Agent Framework Go workflow: two string executors wired by an edge, run with RunStreaming and watched as a stream of typed events, fully offline.
Compose three kinds of Agent Skill — code-defined, struct-based, and file-based — into a single agent through one skills context provider.
One skills ContextProvider blending three origins: in-memory volume and temperature skills plus a file-based unit-converter, unified behind one tool surface for the model.
An Agent Skill — on-demand instructions, resources, and scripts — defined entirely in Go, with no SKILL.md files on disk.
An Agent Skill built from Go closures: instructions, a static and a runtime-generated resource, and a convert script that runs in-process with no SKILL.md files.
Teach an agent a capability from a folder of files — a SKILL.md manifest, resources, and scripts — loaded on demand via progressive disclosure.
Teach an agent from a SKILL.md manifest, resources, and scripts on disk: fsskills scans the tree and a skills ContextProvider exposes load, read, and run tools.
The provider is the swappable back end: the same Joker agent, one-shot and streaming, now through the OpenAI API.
The same Joker agent, one-shot and streaming, through openaiprovider: openai.NewClient reads OPENAI_API_KEY and the model name lives in AgentConfig, not a Foundry deployment.
A provider whose "credential" is a local process, gated by a human-in-the-loop permission handler for every action the model wants to take.
A provider whose credential is a local copilot CLI process, with an OnPermissionRequest handler that approves or rejects each shell action the model wants to run.
How swapping in the Gemini provider changes the constructor and credential — and nothing else about your agent.
The Joker agent, unchanged, now backed by Google Gemini: geminiprovider.NewAgent takes a genai.Client and an API key instead of an Azure token credential.
This lesson teaches how to connect to a remote MCP server and decorate the tools it exposes with your own local behavior before an agent uses them.
List a remote MCP server's tools, then wrap each FuncTool with a logging decorator via embedding plus one Call override — MCP tools compose like any tool.FuncTool.
This lesson teaches how to attach the hosted web-search tool and read the citation annotations the service returns with its answer.
Attach hostedtool.WebSearch so Foundry grounds its answer on live results, then pull CitationAnnotations off content headers with a pure, unit-testable extractor.
This lesson teaches the hosted tool: a marker that lets the Foundry service run code the model writes, instead of a Go function you implement.
Attach hostedtool.CodeInterpreter, a zero-value marker with no Run method, so Foundry executes model-written Python in a sandbox to solve sin(x) + x^2 = 42.
All the components — pipelines, warehouses, models, batch and streaming, the modern stack, quality and governance — come together in a single job: keep reliable, usable data flowing to the people and systems that need it. Doing that in the real world is less about any one technology than about a mindset: treating data pipelines as production software that must be reliable, tested, observed, and maintained. This closing post is about data engineering as it's actually practiced, and where it's heading as AI makes good data more valuable than ever.
All the components come together in one job: keep reliable, usable data flowing to the people and systems that need it. Doing that in the real world is less about any one technology than about a mindset — treating data pipelines as production software that must be reliable, tested, observed, and maintained. Plus where the field is heading as AI makes good data more valuable than ever.
This lesson teaches how to bundle related function tools onto a Go type and attach the whole group to an agent as one slice.
Bundle GetWeather and GetCurrentTime onto one Go type, expose them as a tool.Tool slice, and let the model call both in a single turn — plugins as an organizing idea.
The most dangerous failure in data engineering is the one that doesn't announce itself. A crashed pipeline gets noticed; silently wrong data flows straight into dashboards and models, where people trust it and make decisions on it — wrong decisions, confidently made. Data quality and governance are the disciplines that guard against this: ensuring data is correct, trustworthy, and well-managed. They're the least glamorous and most important part of production data engineering, because without them, all the pipelines and warehouses just deliver garbage efficiently.
The most dangerous failure in data engineering is the one that doesn't announce itself. A crashed pipeline gets noticed; silently wrong data flows straight into dashboards and models, where people trust it and make wrong decisions confidently. Data quality and governance are the disciplines that guard against this.
This lesson teaches a middleware chain where a guardrail can block a run before the model is ever called, chained ahead of a logger.
Chain a guardrail middleware that yields its own refusal and skips the model ahead of a logger — short-circuiting a harmful request offline before Foundry is called.
A decade ago, building a data platform meant heavy, monolithic, on-premises systems and long projects. Today it's assembled from specialized cloud tools that snap together around a central cloud data warehouse — ingest here, transform there, visualize over there — each best-in-class at one job. This "modern data stack" is less a specific set of products than an architecture and a philosophy, and understanding its shape (and what drove it) is understanding how data platforms are actually built now.
A decade ago, a data platform meant heavy, monolithic, on-premises systems. Today it's assembled from specialized cloud tools that snap together around a central cloud data warehouse — each best-in-class at one job. This 'modern data stack' is an architecture and a philosophy, and understanding its shape is understanding how data platforms are built now.
This lesson teaches the simplest form of multi-agent delegation: wrapping one agent as a tool another agent can call.
Wrap a specialist WeatherAgent with agenttool.New into a tool.Tool and hand it to a French TravelAgent — one model orchestrating another with no workflow engine.
How fresh does the data need to be? That one question splits data engineering into two paradigms. Batch processing handles data in large chunks on a schedule — simpler, cheaper, and fine when yesterday's data is good enough. Stream processing handles data continuously as it arrives — more complex and costly, but necessary when you need to know now. Choosing between them (and knowing when each fits) is one of the most consequential architectural decisions in a data platform, and it's driven by real requirements, not by which sounds more impressive.
How fresh does the data need to be? That one question splits data engineering into two paradigms. Batch processing handles data in chunks on a schedule — simpler and cheaper. Stream processing handles data continuously as it arrives — more complex, but necessary when you need to know now.
This lesson teaches how to send a question and an image together in one message to a vision-capable Foundry agent.
Put a TextContent and a base64 DataContent JPEG in one message and hand it to a vision agent — multimodal input with no upload step, embedded at compile time.
The same data can be organized in ways that make analytical queries fast, intuitive, and cheap — or slow, confusing, and expensive. Data modeling is the craft of that organization, and it's where a counterintuitive truth lives: the careful normalization that's correct for application databases is often exactly wrong for analytics. Analytical data wants to be shaped differently, around how questions are asked rather than how data is written, and understanding dimensional modeling is understanding how to make a warehouse actually usable.
The same data, organized differently, makes analytical queries fast and intuitive — or slow and confusing. Data modeling is the craft of that organization, and it holds a counterintuitive truth: the normalization that's correct for application databases is often exactly wrong for analytics.
This lesson teaches how a Foundry agent borrows tools from a remote MCP server and calls them as if they were local functions.
Connect a Foundry agent to Microsoft Learn's public MCP endpoint over streamable HTTP, list its tools, and hand the whole slice to the agent as proxied tools.
"Just put it in a database" stops working the moment you're dealing with analytics at scale — because the database that runs your application is optimized for exactly the wrong thing. Analytical data needs different storage: systems built to scan and aggregate huge volumes, not to serve fast individual transactions. The landscape of analytical storage — data warehouses, data lakes, and the newer lakehouses — is where data engineering decisions about where data lives get made, and understanding the differences (starting with OLTP vs OLAP) is essential.
'Just put it in a database' stops working the moment you're doing analytics at scale — the database that runs your app is optimized for exactly the wrong thing. Analytical data needs different storage: warehouses, lakes, and lakehouses. Understanding the differences (starting with OLTP vs OLAP) is essential.
How one middleware wraps every agent run in an OpenTelemetry span tagged with the gen_ai semantic attributes — the same one-line hook you use for logging.
otelprovider.NewMiddleware wraps every run in an OpenTelemetry span tagged with gen_ai attributes — the same one-line middleware hook you use for logging.
The core artifact of data engineering is the pipeline: an automated flow that pulls data from somewhere, reshapes it, and lands it somewhere useful. And the single most consequential shift in modern data engineering is captured in three reordered letters — ETL became ELT — a change driven by cheap, powerful cloud data warehouses that flipped when and where transformation happens. Understanding pipelines, and the ETL-to-ELT shift, is understanding how data actually moves and gets made usable.
The core artifact of data engineering is the pipeline: an automated flow that pulls data from somewhere, reshapes it, and lands it somewhere useful. And the single most consequential modern shift is captured in three reordered letters — ETL became ELT — driven by cheap, powerful cloud data warehouses that flipped when and where transformation happens.
How to serialize an agent.Session to disk and resume it later, so a follow-up prompt still remembers earlier turns across a process restart.
An agent.Session serializes with encoding/json, so you can marshal it to disk or a database and resume it later with the model still remembering earlier turns.
Every dashboard, every analytics query, every machine-learning model, and every "data-driven decision" rests on an invisible foundation: someone built the pipelines that collect, move, clean, and organize the data so it's actually usable. That someone is a data engineer, and their work is the unglamorous, essential plumbing beneath everything data. When it works, no one notices; when it breaks, every downstream report and model breaks with it. Understanding data engineering is understanding how raw data becomes something a business can actually use.
Every dashboard, analytics query, and ML model rests on an invisible foundation: someone built the pipelines that collect, move, clean, and organize the data. That someone is a data engineer, and their work is the essential plumbing beneath everything data. When it works, no one notices; when it breaks, every downstream report and model breaks with it.
How to make the model return a typed Go struct instead of prose — hand RunText a pointer, get back populated fields.
Pass RunText a pointer to a Go struct via WithStructuredOutput and the framework asks the model for matching JSON, then unmarshals it straight into your fields.
Caching giveth performance and taketh away your sanity. The same technique that makes systems fast introduces a whole category of subtle, intermittent, hard-to-debug problems — stale data that appears randomly, a cache that collapses under load at the worst moment, bugs that only happen when the cache is cold or full. This closing post catalogs the pitfalls that bite real systems, and distills the practical wisdom of the series: cache deliberately, expect the failure modes, and remember that the two genuinely hard things are still hard.
Caching giveth performance and taketh away your sanity. The same technique that makes systems fast introduces subtle, intermittent, hard-to-debug problems — stale data appearing randomly, a cache collapsing under load, bugs that only happen when the cache is cold. This catalogs the pitfalls and the practical wisdom.
How to gate a sensitive tool behind human consent: the run pauses, hands you an approval request, and only calls the tool once you approve.
Wrap a tool with ApprovalRequiredFunc and the run pauses with a ToolApprovalRequestContent; approve it, feed the decision back on the same session, and the tool fires.
Every time a web page loads instantly on a repeat visit, or a video streams smoothly from halfway around the world, caching is the reason. The web is layered with caches — in your browser, at CDN edge servers near you, in front of origin servers — all applying the same caching principle to make the internet fast. And remarkably, much of it is coordinated by a few HTTP headers that let servers tell caches exactly what to store and for how long. Understanding web and CDN caching is understanding how the internet stays fast at global scale.
Every time a web page loads instantly on a repeat visit, or a video streams smoothly from across the world, caching is the reason. The web is layered with caches — browser, CDN edge, origin — all applying the same principle, much of it coordinated by a few HTTP headers.
How to wrap an ordinary Go function as a tool the model can call, with the schema derived automatically from its types.
Wrap a plain Go func with functool.MustNew and the framework derives its JSON schema and drives the tool call inside a single RunText — two model round-trips, one call.
A cache inside a single application process is easy — but it doesn't scale, and every instance of your app has its own separate copy. The moment you run many application servers, you want a shared cache they all use, which means a cache that lives across the network on its own machines: a distributed cache like Redis or Memcached. This unlocks scale and sharing, but introduces the distributed-systems problems that a local cache never had. Understanding distributed caching is understanding how caching works at real scale.
A cache inside a single process is easy — but it doesn't scale, and every instance of your app has its own separate copy. Run many servers and you want a shared cache across the network on its own machines: a distributed cache like Redis or Memcached. This unlocks scale and sharing, but introduces distributed-systems problems.
How to keep a conversation's history on the Foundry service instead of in local memory, so the client never resends the transcript.
Create a Foundry project conversation, bind its ID into a session with WithServiceID, and let the service keep the transcript across turns without resending it.
Knowing to cache is one thing; wiring the cache into your application correctly is another. Should the application manage the cache itself, or should the cache sit transparently in front of the source? Should writes go to the cache, the database, or both — and in what order? These questions have standard answers — the caching patterns — and choosing the right one shapes your consistency, performance, and complexity. Getting the pattern right is how caching goes from "store some stuff" to a coherent, correct design.
Knowing to cache is one thing; wiring the cache into your application correctly is another. Should the application manage the cache itself, or should it sit transparently in front of the source? Should writes go to the cache, the database, or both? These questions have standard answers — the caching patterns.
How an agent.Session carries conversation history so a second prompt can refer back to the first.
A client-side agent.Session carries conversation history so a second prompt can refer back to the first. Create it with CreateSession, thread it with WithSession.
This is the hard one. "There are only two hard things in computer science: cache invalidation and naming things" names it directly — cache invalidation is genuinely, notoriously difficult. The moment you cache a copy of data, you've created a second source of truth that can drift from the first, and keeping them in sync (or deciding how much drift you'll tolerate) is a problem with no clean, universal solution. Understanding why it's hard, and the strategies for managing it, is the difference between caching that helps and caching that causes baffling bugs.
This is the hard one. 'There are only two hard things in computer science: cache invalidation and naming things' names it directly. The moment you cache a copy of data, you've created a second source of truth that can drift from the first, and keeping them in sync has no clean universal solution.
How the Foundry provider snaps a project endpoint, a credential, and a model deployment name into a runnable agent.
The Foundry provider is three inputs: a project endpoint, a credential, and a model deployment name. See how foundryprovider.NewAgent snaps them into an agent.
A cache is a small space pretending to be a big one, and the pretense only works if it's clever about what to keep. When a bounded cache fills up, every new item forces out an old one — and which one you evict determines your hit rate, which determines whether the cache is worth having at all. Eviction policies are the algorithms that make this choice, and understanding them (especially the workhorse, LRU) is essential to building caches that actually stay effective.
A cache is a small space pretending to be a big one, and the pretense only works if it's clever about what to keep. When a bounded cache fills up, every new item forces out an old one — and which one you evict determines your hit rate. Eviction policies are the algorithms that make this choice.
Back the agent with Azure OpenAI's Responses API, and toggle whether conversation state lives on the server or locally.
Back an agent with Azure OpenAI Responses via NewResponsesAgent, and use DisableStoreOutput to keep chat history local instead of stored server-side.
Build a GDPR Article 22 compliant explanation endpoint in Go that turns audit logs and eval stores into regulator-friendly answers for AI decisions.
Request → N-eyes approve → window-of-time → automatic expiry, with every transition written to a hash-chained audit log. The package that closes Gap #1 from the PCSE map.
A cache lives or dies by one number: its hit rate. Every cache access is a small bet — that the data will be there (a hit, served fast) rather than missing (a miss, served slow, plus the overhead of caching it). Whether caching helps at all comes down to how often that bet pays off, and understanding hits, misses, and hit rate — and what you should and shouldn't cache — is the foundation of using caches effectively. Get these fundamentals right, and the rest of caching makes sense.
A cache lives or dies by one number: its hit rate. Every cache access is a small bet — that the data will be there (a hit, served fast) rather than missing (a miss, served slow). Whether caching helps at all comes down to how often that bet pays off.
The same agent primitive, wired to Azure OpenAI's Chat Completions API where the model name is your deployment name.
Back an agent with Azure OpenAI Chat Completions via NewChatCompletionsAgent, where the Model field is your Azure deployment name, not a catalog model ID.
There's an old joke that there are only two hard things in computer science: cache invalidation and naming things. It's a joke because caching is everywhere and sounds simple — just keep a copy of stuff you'll need again — and it's true because getting caching right is genuinely, surprisingly hard. Caching is one of the most universal and powerful ideas in computing, appearing at every layer from CPU to CDN, and understanding why it exists and when it helps is foundational to building fast systems.
There's an old joke that there are only two hard things in computer science: cache invalidation and naming things. It's a joke because caching sounds simple, and true because getting it right is genuinely hard. Caching is one of the most universal ideas in computing, appearing at every layer from CPU to CDN.
Reach a Foundry-hosted model through the OpenAI-compatible Responses API by configuring a plain openai.Client with three request options.
Reach a Foundry model through the OpenAI-compatible API by configuring an openai.Client with base URL, an Azure token credential, and the ai.azure.com scope.
Point the same agent at an Azure AI Foundry project endpoint using the project Responses API mode.
Run an agent against an Azure AI Foundry project: foundryprovider.NewAgent plus ModelDeployment selects project Responses API mode from the project endpoint.
Five interfaces hold the whole platform together. The 30-line orchestrator closure that makes the rest of the architecture testable, auditable, and safe to evolve.
The same agent primitive as the Foundry lessons, backed by the Anthropic (Claude) provider instead of Azure AI Foundry.
A provider swap onto Anthropic Claude: build an anthropic.Client, pass it to anthropicprovider.NewAgent, and run the identical agent surface.
PostgreSQL row-level security as HIPAA defence in depth. Why fail-open application filtering isn't enough, and how 'append-only at DB GRANTs' carries more of the §164.312(b) burden than people realise.
This lesson swaps the LLM behind an agent for another agent, reached over the A2A protocol instead of a model API.
Back a Go agent with another agent over the A2A protocol: resolve the remote card, open a gRPC client, and wrap it with a2aprovider.NewAgent.
The 21st Century Cures Act §3060 CDS carve-out criterion 4 expressed as a code-level queue, lossless on reject, with audit-recorded reviewer rationale. Build it once, satisfy GDPR Article 22 for free.
How to borrow tools from a remote Model Context Protocol server and hand them to a Foundry agent as ordinary tools.
Connect to Microsoft Learn's public MCP server with mcptool.Connect and ListTools, then hand the borrowed tools to a Foundry agent as ordinary agent.Config.Tools.
How a single sprint of specialty-rule work — guided by a benchmark that wasn't afraid to print embarrassing numbers — turned a 'demo respiratory differential' into a five-condition rule-based diagnostic engine.
How a shared state snapshot rides along with every AG-UI turn, so client and server-hosted agent stay in sync on evolving structured data.
Share a JSON state snapshot across AG-UI turns — server middleware emits a DataContent snapshot from the model's JSON and the client adopts it via toStateContent and extractState.
What HIPAA looks like when you express it as Go interfaces — governance policies, append-only audit at DB GRANTs, PHI redaction at the logger seam, and HITL as the §3060 CDS carve-out criterion 4.
How an approval-required tool makes the server pause a run and wait for a human on the client to say yes or no.
Gate a tool behind tool.ApprovalRequiredFunc so the AG-UI server pauses the run, then answer the approval from the client with a message round-trip loop.
How the server-hosted agent can call a tool that actually runs on the client, and the one flag that makes it work.
Let a server-hosted agent call a client-side tool over AG-UI — the server sets DisableFuncAutoCall and forwards the call to the client, which runs the Go function locally.
How an AG-UI-hosted agent runs a server-side function tool while the thin client just streams the conversation.
Give an AG-UI-hosted agent a server-side search_restaurants tool via functool while the thin SSE client just streams the reply — the tool round-trip stays invisible.
We built a small Go + Python service that parses a project's INFORMATION_SCHEMA, asks Gemini to classify each top-spending query against a catalog of anti-patterns, and recommends a rewrite. It is not a magic box; it is a pipeline that cuts the human review time per query from 20 minutes to 90 seconds.
How to take the same Foundry agent from earlier lessons and serve it over the AG-UI protocol so a separate client can drive it over HTTP+SSE.
Serve an unchanged Foundry agent over the AG-UI protocol with one aguiprovider.NewJSONHTTPHandler call, then drive it from a credential-free SSE client.
How a memory ContextProvider backed by an Azure AI Foundry store lets an agent recall you in a brand-new session.
Attach a Foundry-backed memory ContextProvider that retrieves before and stores after each run, so a fresh session still recalls facts keyed by a scope.
How a real shell tool lets the model run commands, and an environment provider tells it which shell it is driving.
Pair a run_shell tool with an EnvironmentProvider that probes the shell once and injects OS-correct idioms, contrasting stateless and persistent shell modes.
Notes from contributing to Google's open-source Spanner Migration Tool (HarbourBridge). Where to start reading the codebase, where the load-bearing logic lives, and the parts that look simple but aren't.
Threads are great for CPU-bound parallelism, but for handling thousands of network connections — each mostly waiting — spawning a thread per connection doesn't scale (recall the C10K problem from the OS series). Async/await is Rust's answer: write code that looks sequential but doesn't block a thread while waiting, letting a handful of threads handle enormous concurrency. Rust's async is powerful and zero-cost, with one distinctive twist — you bring your own runtime to actually run the async code.
For handling thousands of connections each mostly waiting, spawning a thread per connection doesn't scale. Async/await is Rust's answer: write code that looks sequential but doesn't block a thread while waiting. Rust's async is powerful and zero-cost, with one distinctive twist — you bring your own runtime.
How to keep a long-running conversation inside the context window by chaining compaction strategies from gentle to aggressive.
Chain four compaction strategies from tool-result collapse to summarization to sliding window to truncation, each gated by a trigger and a preservation floor.
Spanner partitions by primary-key range. A monotonically-increasing PK like a timestamp or UUID-v1 funnels all writes to one server. The fix changes everything from your sequence strategy to your tenant model.
There are two great philosophies of concurrency: share memory (with locks, as the previous posts covered) or share nothing and communicate by passing messages. The message-passing school has a famous slogan — "do not communicate by sharing memory; instead, share memory by communicating" — and Rust supports it fully with channels. Instead of multiple threads carefully locking shared state, ownership of data is transferred from one thread to another through a channel, and Rust's ownership system makes that transfer clean and safe.
There are two great philosophies of concurrency: share memory (with locks) or share nothing and communicate by passing messages. Rust supports channels fully — and its ownership system makes message passing especially natural, because sending data through a channel is transferring ownership.
How a ContextProvider injects extra messages and tools into every run — a live todo list and calendar — with state that survives serialize and resume.
A ContextProvider injects a live todo list and calendar into each run and contributes session-mutating tools, with state that serializes to JSON and resumes.
How does the Rust compiler actually know that an `Arc<Mutex<T>>` is safe to share across threads but an `Rc<T>` isn't? The answer is two of the most elegant ideas in Rust: a pair of marker traits, `Send` and `Sync`, that encode thread-safety directly into the type system. They're rarely written by hand and often invisible, yet they're the machinery that makes fearless concurrency work — the compiler reasons about thread-safety by checking these traits, automatically, at compile time.
How does the compiler know an Arc<Mutex<T>> is safe to share across threads but an Rc<T> isn't? The answer is two elegant marker traits — Send and Sync — that encode thread-safety directly into the type system. Rarely written by hand and often invisible, they're the machinery that makes fearless concurrency work.
How wrapping one agent as a tool.Tool lets an orchestrator agent delegate to a specialist — composition all the way down, with no routing code.
Wrap a specialist agent with agenttool.New so an orchestrator calls it like any tool, cascading two levels deep from orchestrator to weather agent to leaf function.
Moving data into a single thread is safe but limiting — sometimes multiple threads genuinely need to share and mutate the same data. This is exactly where data races live in other languages, and where Rust's guarantees shine brightest. The answer is a pair of types, `Arc` and `Mutex`, that let you share mutable state across threads — and the compiler will refuse to compile code that shares it unsafely. You literally cannot forget the lock, because the data lives inside it.
Sometimes multiple threads genuinely need to share and mutate the same data — exactly where data races live in other languages, and where Rust's guarantees shine brightest. The answer is Arc and Mutex, which let you share mutable state across threads while the compiler refuses to compile unsafe sharing. You literally cannot forget the lock.
How one message can bundle a text prompt and an image, and why that forces RunMessage instead of the RunText shortcut.
Bundle a TextContent prompt and a base64 DataContent image into one message.Message and run it with RunMessage instead of the text-only RunText shortcut.
Rust's boldest promise is "fearless concurrency" — the claim that you can write multithreaded code and have the compiler guarantee, at compile time, that you have no data races. Coming from languages where concurrency bugs are a dark art of subtle, intermittent horror, this sounds too good to be true. It isn't: the same ownership and borrowing rules that give Rust memory safety extend naturally to threads. This module explores concurrency, starting with the basics — spawning threads and moving data into them.
Rust's boldest promise is fearless concurrency — write multithreaded code and have the compiler guarantee, at compile time, that you have no data races. It isn't too good to be true: the same ownership and borrowing rules that give memory safety extend naturally to threads. Module 3 begins with the basics — spawning threads and moving data into them.
How to publish an agent as a discoverable tool on a Model Context Protocol server so any MCP client can invoke it.
Wrap an agent with agenttool.New then register it with mcptool.AddTool to serve it over MCP on stdio, so any MCP client can discover and invoke it.
How Go's constructor-over-an-interface idiom lets you inject a real Foundry agent into a service — and a fake into its test — through the same seam.
Inject a Foundry agent into a service through a two-method ChatAgent interface, so main injects the real agent and the test injects a fake through the same constructor.
How to turn every agent run into an OpenTelemetry span with the SDK's otelprovider middleware.
Wire otelprovider.NewMiddleware into agent.Config.Middlewares to open a gen_ai span around every run, exported to whatever TracerProvider you register globally.
How to teach an agent to load and persist conversation memory in your own store via a custom history provider.
Implement a custom agent.HistoryProvider with Provide and Store hooks so the agent loads and persists history in your store; DisableStoreOutput keeps it the single source of truth.
The Picnic social platform served 1M+ users across a graph of Go microservices behind a GraphQL gateway. The latency win came from a counter-intuitive move: fewer services, tighter contracts.
How to serialize an agent.Session to storage and resume the same conversation later, even from a new process.
Serialize an agent.Session with encoding/json, store the bytes, and reload them into a fresh Session in another process to resume the exact same conversation.
Test coverage and observability are the boring infrastructure that makes the interesting changes safe. Notes on how the Picnic team built both, and the on-call experience they enabled.
How to make the agent return a typed Go struct instead of free-form prose, via a JSON schema derived from your type.
Two ways to get typed output from an agent: per-run agent.WithStructuredOutput, or agent.WithResponseFormat with jsonformat.MustFor baked into Config.RunOptions.
The transaction engine had to absorb 30K+ TPS across partner integrations, never lose a transaction, and survive partial failures. The architecture: Go, Kafka, Pub/Sub, Redis, K8s, with idempotency at every layer.
How to pause a run for human approval before the framework is allowed to execute a tool.
Wrap a tool with tool.ApprovalRequiredFunc so the run pauses and returns a ToolApprovalRequestContent; approve or decline, then resume with RunMessage on the same session.
A single layer of idempotency will eventually fail. Three independent layers gives you a margin. Here is the pattern that worked across ingest, worker, and emit boundaries.
How to wrap a plain typed Go function as a tool the model can call mid-run.
Wrap a plain typed Go function with functool.MustNew and attach it via agent.Config.Tools so the model can call it; the SDK infers the JSON schema from the signature.
Status-code-based dispatch made every worker grow a longer and longer switch. Normalising every partner-specific error into an enumerated set let the orchestration logic stop changing as new partners landed.
How an agent.Session carries conversation history across RunText calls so the model remembers earlier turns.
Thread conversation history across RunText calls with an agent.Session and agent.WithSession, so a follow-up prompt builds on the previous turn; a new session starts fresh.
5K+ loans per month. Three credit bureaus. Multiple payment gateways. The thing that has to be right is the ledger. Notes on what invariants the database enforces vs what the application enforces.
How to wrap every agent run with a middleware — the standard hook for logging, tracing, and guardrails.
Wrap every agent run with an agent.Middleware in the Foundry Go SDK: one pass-through function that observes messages and updates without altering the result.
How to resume a long-running remote agent stream after it drops — using a continuation token to reconnect instead of re-sending the expensive query.
Capture the continuation token from a streamed remote run, then reconnect with WithContinuationToken and no messages to resume the same task to completion.
100K+ votes, 10K+ concurrent users during a live AFL Brownlow Medal broadcast. The architecture: Go on Cloud Run, GraphQL + gRPC behind a CDN, vote integrity through Cloud KMS + Security Command Center. Notes on what makes a live-broadcast load shape unusual.
How to pin which transport a client uses when a remote A2A agent advertises several bindings at once.
Pin the transport an A2A client uses via PreferredTransports so binding negotiation is deterministic, while the wrapped remote agent stays a plain agent.Agent.
How to drive a slow remote agent that answers with a continuation token, then poll that token to completion instead of blocking on one long call.
Ask a remote A2A agent with AllowBackgroundResponses, get a continuation token, and poll with WithContinuationToken and no messages until the token clears.
How to discover a remote A2A agent's advertised skills and hand each one to a local host agent as an ordinary function tool.
Resolve a remote A2A agent card, turn each advertised skill into a function tool with derived names and descriptions, and hand them to a local Foundry host agent.
What it actually takes to build a unified cloud API library — and why "write once, run anywhere" still doesn't quite work, even for the patterns where it almost does.
The other primitive: a directed graph of executors wired by edges, running fully offline with no model, no credential, no Foundry.
Wire two executors into an uppercase to reverse pipeline with the fluent workflow builder and run it fully offline via inproc.Default, iterating ExecutorCompletedEvents.
Every Professional Cloud Security Engineer exam bullet, mapped to a file path in an RBI FREE-AI aligned Go platform. Where the implementation matches, where the analog substitutes, and where the honest gaps are.
A custom ContextProvider gives the agent memory: it reads stored facts before a run and learns new ones after — all through the Session.
A custom ContextProvider wires Provide and Store hooks around every run so the agent reads remembered facts before the call and learns new ones after, all in the Session.
One Session threaded through every call turns two stateless one-shots into a single conversation that remembers.
Thread one agent.Session through every RunText call with agent.WithSession to turn stateless one-shots into a single conversation that remembers.
Hand the agent a plain Go function it can decide to call mid-conversation — the model requests it, the framework runs it, the result flows back.
Register a plain Go function as a tool with functool.MustNew; the model decides when to call it, the framework runs it, and the result flows back into the answer.
The smallest useful agent: give a model instructions and a name, hand it a message, get a response — collected or streamed.
Build the smallest useful agent from instructions and a model, then run the same RunText call two ways — collected all at once and streamed token-by-token.
The zero-network preflight that confirms your Foundry config and Azure credential chain exist before you run a single agent.
A zero-network preflight that confirms your Foundry endpoint, model deployment, and Azure credential chain exist before you run a single agent.
Stdlib over libraries, single binary over framework, fail-closed defaults over forgiveness. The boring-on-purpose case for choosing Go to ship a multi-agent system into a regulated environment.
Microsoft's Multi-Agent Reference Architecture in Go. Protocol, registry, bus, governance, orchestration, observability, evaluation — and how the seven hold each other up.
Passkeys are FIDO2; FIDO2 is the spec; Ed25519 is the signature algorithm. The full registration + assertion flow in 200 lines of stdlib Go.
Two signals do most of the work for detecting compromised sessions: impossible travel between consecutive logins, and credential-stuffing density across an IP range. The Go implementation.
Google's A2A spec standardises how agents talk to other agents (not just tools). The Go client is small; the conceptual shift is what matters.
A saga is fine when every step succeeds. The interesting code is what runs when step 3 of 5 fails and you have to undo 1 and 2 in the right order. The patterns I use.
Postgres over the latest vector DB. Go stdlib over the framework du jour. Single binary over Kubernetes operator. The choices that bore reviewers and delight on-call engineers.
Go's embed.FS bundles files into the binary at compile time. The pattern collapses what would be a multi-artefact deploy into one binary. Three places it pays back daily.
GOMEMLIMIT tells the Go runtime to keep memory below a soft cap by running GC harder when it's close. For containers with hard memory limits, this prevents OOM kills. The setting every Go service in K8s should have.
Patterns I confidently recommended five years ago that I'd argue against today. The list of "things you used to do in Go that don't pay back anymore."
Range-over-function landed in Go 1.23. `iter.Seq` lets you write iterators that compose. The patterns that pay back; the ones that don't.
Fan out to N agents; first error cancels the rest; collect successful results. errgroup is the right tool for this; the patterns are concise but worth getting exactly right.
An honest retrospective on the open-source Genie project after a year. The patterns that held up; the ones we rebuilt; the code we deleted because it solved problems we didn't actually have.