You can have the right customer, sharp positioning, the right motion, and smart pricing — and still sell nothing, because no one knows you exist. Demand generation and channels are how you solve the awareness problem: getting the right people to discover you, and moving them from "never heard of it" toward "customer." For technical builders this is the least intuitive part of GTM, because it can't be reasoned out at a desk — it's found by testing where your customers actually are.
You can have the right customer, sharp positioning, the right motion, and smart pricing — and still sell nothing, because no one knows you exist. Demand generation and channels are how you solve the awareness problem, and it can't be reasoned out at a desk — it's found by testing where your customers actually are.
Most agent code is tested by running it and eyeballing the output — because testing agents properly is genuinely hard. Pydantic AI's whole design has been quietly building toward making it easy: type safety, dependency injection, and test models combine so you can unit-test agent logic deterministically, offline, without ever calling a real LLM. This is arguably the framework's biggest practical advantage.
Most agent code is tested by running it and eyeballing the output, because testing agents properly is hard. Pydantic AI's design builds toward making it easy: type safety, dependency injection, and test models let you unit-test agent logic deterministically, offline, with no real LLM.
A benchmark score is trustworthy only if the model hasn't seen the answers and no one has optimized directly for the test. Both assumptions fail constantly. Training data contamination inflates scores by rewarding memorization; optimizing for a benchmark turns it from a measure into a target and destroys its meaning. This post is about the ways evaluation gets corrupted — and how to defend against them.
A benchmark score is trustworthy only if the model hasn't seen the answers and no one optimized directly for the test — and both assumptions fail constantly. Training-data contamination inflates scores by rewarding memorization; optimizing for a benchmark turns it from a measure into a target. Here's how evaluation gets corrupted, and how to defend it.
The scores that dominate model announcements — MMLU, GSM8K, HumanEval, and the rest — are benchmarks: standardized public tests that let the whole field compare models on common ground. They've driven enormous progress, but a benchmark is only as good as its design, and a number without understanding of what it measures is easy to misread. Knowing how benchmarks are built, and what makes a good one, is how you read a leaderboard critically instead of credulously.
The scores in model announcements — MMLU, GSM8K, HumanEval — are benchmarks: standardized public tests for comparing models on common ground. They've driven huge progress, but a benchmark is only as good as its design, and a number without understanding is easy to misread. Here's how they're built and what makes a good one.
Rust's tooling is one of its quiet superpowers — a single tool, Cargo, handles building, dependencies, testing, and more, and it's good enough that Rust developers rarely think about build systems at all. Before the language's hard ideas, meet the tooling that makes working in Rust pleasant.
Rust's tooling is a quiet superpower — a single tool, Cargo, handles building, dependencies, testing, and more, and it's good enough that Rust developers rarely think about build systems at all. Meet the tooling that makes working in Rust pleasant.
Metrics and judges are ingredients; a harness is the kitchen. An evaluation harness is the system that takes a dataset of test cases, runs your LLM system over them, scores the outputs, and reports the results — reproducibly, every time. Building one well is what turns evaluation from a one-off spreadsheet into an engineering asset you run on every change, like a test suite.
Metrics and judges are ingredients; a harness is the kitchen. An eval harness takes a dataset of test cases, runs your system over them, scores the outputs, and reports results — reproducibly, every time. Building one well turns evaluation from a one-off spreadsheet into an engineering asset you run on every change.
Everything in the series so far comes together here: a small but complete Model Context Protocol server, in Python, exposing a tool, a resource, and a prompt, runnable and testable in minutes.
Build a complete MCP server in Python with the official SDK — a notes service with a tool, a resource, and a prompt — and test it end to end with the MCP Inspector.
The most striking frontier result is a meta-agent that writes agents — defining them as code, testing them, archiving the good ones, and inventing architectures that outperform the best humans hand-built.
The most striking frontier result is a meta-agent that writes agents — defining them as code, testing them, archiving the good ones, and inventing architectures that outperform the best humans hand-built.
A single pre-launch red-team decays the moment your model, prompt, or tools change — turning adversarial testing into a sustained program is what keeps an AI system safe past day one.
The capstone: making red-teaming a sustained program — the remediation loop where findings become regression tests, blending automated/manual/external modes, rules of engagement, ship-blocking severity thresholds, governance evidence, and culture.
The finale of the API Security series — how to bake security into the way APIs are designed, built, tested, shipped, and operated, so that every control from the previous seven posts becomes a repeatable part of the pipeline instead of a one-time heroic effort.
The capstone: baking security into the API lifecycle — shift-left threat modeling, CI gates (SAST, SCA, secret scanning, spec-driven authz/BOLA and DAST tests that fail the build), an OWASP-API-Top-10-to-control map, and incident response.
How APIs are designed, shipped, and kept consistent at scale — the design review, the enforceable style guide, spec linting as policy-as-code, contract tests, an API catalog, and the org model that makes ten teams' APIs feel like one.
The capstone: designing and shipping consistent APIs at scale — the lifecycle, design review and style guides, linting the spec as policy-as-code in CI, contract testing so changes don't break consumers, an API catalog with ownership, and production observability.
The series finale: how to proactively find AI security failures before attackers do — turning injection, leakage, and excessive-agency risks into a repeatable adversarial test suite that runs in CI, measuring attack success honestly, and standing up incident response for the day a control fails.
Find AI security failures before attackers do: adversarial testing of the whole system, a test taxonomy mapped to the series, automated tooling (PyRIT, garak), a CI security-regression suite that fails the build, honest attack-success-rate measurement, and AI incident response.
Why an API is only as good as a developer's ability to succeed with it — the OpenAPI contract as the source of truth, reference docs versus guides, errors as documentation, and the DX niceties that turn a first request into a shipped integration.
An API is only as good as a developer's ability to succeed with it: OpenAPI as the machine-readable contract (docs, SDKs, mocks, contract tests), reference plus guides, errors as documentation, and optimizing time-to-first-successful-call.
Half of review quality is decided before the reviewer ever opens the diff — by the author. Small focused PRs, a description that states what and why, a self-review pass, and clean commits are how you optimize the one resource a review really spends: the reviewer's attention.
Half of review quality is set before the reviewer arrives: small focused PRs, splitting big changes, great descriptions (what/why/how-to-test), self-review first, and keeping refactor noise out of behavior changes.
Reasoning models are a genuine advance — and it's precisely because they're so impressive that their limits are worth stating plainly. A model that writes out careful, step-by-step reasoning invites you to trust the reasoning, and that trust is not always warranted. The chain of thought may not be why the model actually answered. More thinking eventually stops helping. And some problems no amount of test-time compute will solve. Knowing the edges is part of using the capability wisely.
Reasoning models are a genuine advance — and precisely because they're so impressive, their limits are worth stating plainly. The chain of thought may not be why the model actually answered. More thinking eventually stops helping. And some problems no amount of test-time compute will solve.
A defender's tour of the attacks that target the model and its data — prompt and context extraction, training-data memorization, membership inference and model inversion, model stealing, poisoning and backdoors, and evasion — with what a red-teamer tests and what actually stops each one.
Attacks on the model and its data: system-prompt and training-data extraction, membership inference and model inversion, model stealing, and poisoning/backdoors — what to test and defend, and the honest risk difference between using hosted models and training your own.
The habits that made you good at prompting standard models can actively hurt you with reasoning models. "Let's think step by step" is redundant — even counterproductive — when the model already thinks natively. Few-shot examples can constrain reasoning that's better left free. The shift from standard to reasoning models isn't just picking a different model; it's unlearning some prompting reflexes and learning where deep thinking actually earns its cost.
The habits that made you good at prompting standard models can actively hurt you with reasoning models. 'Let's think step by step' is redundant when the model already thinks natively. The shift isn't just picking a different model — it's unlearning some prompting reflexes and learning where deep thinking earns its cost.
The single highest-leverage setup step for Claude Code is a good CLAUDE.md — the file where you write down, once, the context and conventions you'd otherwise repeat every session.
The highest-leverage setup step: a good CLAUDE.md that gives durable project context (build/test commands, conventions, gotchas, what not to do), how it layers, and the settings/permissions that tune autonomy safely and shareably.
An FDE's superpower is turning a vague problem into something the customer can see and touch within days — because a rough working demo teaches more than a month of meetings.
Turn a vague problem into something the customer can touch in days: build the thinnest slice that tests the riskiest assumption, run a tight demo loop, and manage the prototype's lifespan so 'it demoed' doesn't get shipped as 'it's done'.
A defender's field guide to the injection and jailbreak techniques a red-teamer probes for — the taxonomy, why each one works, and how to turn it into a re-runnable test suite that maps every passed test to a concrete fix.
A deeper, test-focused look at injection and jailbreak families — direct vs indirect (the RAG/agent threat), role-play, obfuscation, many-shot, multi-turn, cross-lingual — plus a red-team methodology: build a probe suite, mutate, test input and retrieval paths, measure, re-test.
How the four families of automated security tests — static analysis, dynamic analysis, secret scanning, and instrumented runtime testing — fit together across a pipeline, and why tuning signal-to-noise matters more than adding scanners.
Automated security testing in the pipeline: SAST vs DAST vs IAST and their trade-offs, secret scanning (including git history), where each runs, and making findings actionable so false-positive fatigue doesn't get the scanner muted.
How to wire agent evaluations into continuous integration in Go — running a slow, model-calling eval harness under `go test`, setting per-metric thresholds that fail the build on a regression, and living honestly with the fact that these gates are softer than unit tests.
How to wire agent evaluations into continuous integration in Go — running a slow, model-calling eval harness under `go test`, setting per-metric thresholds that fail the build on a regression, and living...
Test-time compute reframes a question engineers rarely had to ask before: how much is a correct answer worth? Because thinking now costs money and time in direct proportion to how much of it you do, reasoning is no longer free — it's a purchase. A reasoning model can generate many times more tokens working through a problem than a standard model uses to answer it, and you pay for every one. Deciding when that's worth it is the core practical skill of the reasoning era.
Test-time compute reframes a question engineers rarely had to ask: how much is a correct answer worth? Thinking now costs money and time in direct proportion to how much of it you do. A reasoning model can generate many times more tokens than a standard model — and you pay for every one.
Getting great results from Claude Code is less about clever prompts and more about a disciplined loop: give context, specify clearly, let it work, review, and steer.
The disciplined loop that gets great results: explore → plan → execute → review, specifying like you'd brief a colleague, steering actively, managing context, and right-sizing delegation to the checks (tests) the agent can loop against.
A reviewer's mental checklist ordered by leverage — design first, then correctness, then tests, then readability, with style and formatting handed off to the tools that were built for it.
A reviewer's checklist ordered by leverage: design first, then correctness and edge cases, then tests, then readability — and what to leave entirely to tools (formatting, lint) so human attention goes where it matters.
A reasoning model thinks harder on its own — but you can spend test-time compute on top of any model, reasoning or not, to squeeze out more accuracy. Generate many answers and vote. Generate many and pick the best with a verifier. Search a tree of reasoning steps, pruning the bad branches. These techniques share one shape — do more work at inference, then choose well — and understanding them turns test-time compute from a model property into a toolkit you control.
A reasoning model thinks harder on its own — but you can spend test-time compute on top of any model to squeeze out more accuracy. Generate many answers and vote. Generate many and pick the best with a verifier. Search a tree of reasoning steps. These techniques share one shape: do more work, then choose well.
The opening post of a hands-on series for builders: what it means to systematically stress-test an AI system — model, prompts, retrieval, tools, and guardrails — to surface its failures before adversaries or ordinary users do, how that differs from pentesting and robustness testing, and the frameworks and ethics that keep the work rigorous.
The opener to a defensive AI red-teaming series: stress-testing AI systems (models + prompts + RAG + tools + guardrails) to find failures before adversaries do — how it differs from pentesting, the frameworks (OWASP GenAI, MITRE ATLAS, NIST AI 100-2), and ethical rules of engagement.
How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating Agents in Go.
How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating...
You can't teach deep reasoning by showing a model more examples of good reasoning — because the best reasoning for a hard problem often isn't in any dataset, and imitation caps a model at the quality of what it imitates. The breakthrough behind modern reasoning models was to stop imitating and start rewarding: let the model try to solve problems, check whether it got them right, and reinforce whatever thinking led to correct answers. That shift — from imitation to reinforcement on verifiable outcomes — is why reasoning models can think in ways no one wrote down.
You can't teach deep reasoning by showing more examples — imitation caps a model at what it imitates. The breakthrough was to stop imitating and start rewarding: let the model try to solve problems, check whether it got them right, and reinforce whatever thinking led to correct answers.
The dominant story of AI progress for years was training-time scale: bigger models, more data, more training compute. Test-time compute is a second, independent axis — spend more computation when you run the model, not when you train it, and get better answers on hard problems. It reframes a trained model not as a fixed-capability artifact but as one whose performance you can dial up per query by letting it think more.
The dominant story of AI progress was training-time scale: bigger models, more data. Test-time compute is a second, independent axis — spend more computation when you *run* the model, not when you *train* it, and get better answers on hard problems. It reframes a model as one whose performance you can dial up per query.
The core of the series: a minimal, original evaluation harness in Go. Run an agent under test through adk-go's runner, capture the tool-call trajectory and the final response behind an adapter you own, and score them with `go test`.
The core of the series: a minimal, original evaluation harness in Go. Run an agent under test through adk-go's runner, capture the tool-call trajectory and the final response behind an adapter you own, and...
The observation that launched reasoning models was almost embarrassingly simple: if you ask a language model to "think step by step" before answering, it gets dramatically better at hard problems — with no change to the model at all. That a few words of prompting could unlock reasoning already latent in the model was a clue about something deep: the ability to reason was there, waiting to be elicited, and giving the model room to work was the key.
The observation that launched reasoning models was almost embarrassingly simple: ask a model to 'think step by step' and it gets dramatically better at hard problems, with no change to the model. That a few words could unlock latent reasoning was a clue about something deep.
For years, the way to make a language model better was to make it bigger — more parameters, more training data. Reasoning models represent a different bet: instead of only scaling what the model knows, scale how much it thinks before answering. A reasoning model spends extra computation at inference time working through a problem step by step before committing to an answer — and on hard problems, that changes what's possible.
For years, making a language model better meant making it bigger. Reasoning models represent a different bet: scale how much the model *thinks* before answering, not just what it knows. A reasoning model spends extra computation working through a problem step by step — and on hard problems, that changes what's possible.
The opener to a series on evaluating agents in Go: why an agent isn't a function you can unit-test, why "it worked in the demo" doesn't survive contact with production, and the two things actually worth measuring — the steps it took and the answer it gave.
The opener to a series on evaluating agents in Go: why an agent isn't a function you can unit-test, why "it worked in the demo" doesn't survive contact with production, and the two things actually worth...
The last mile of a financial system isn't code — it's who can do what, who approves it, and how you prove the whole thing is correct.
The last mile isn't code — it's who can do what, who approves it, and how you prove it's correct. Segregation of duties, four-eyes, an auditable change trail, and property tests that assert ledger invariants over thousands of scenarios.
Anyone can stay motivated when things are going well. The test — and the skill — is what happens when the project stalls, the code won't work, the feedback stings, or the effort drags on with no payoff in sight. Motivation and resilience are the emotional-intelligence skills of managing your own drive and bouncing back from setbacks, and they're what turn talent into sustained achievement. Without them, ability leaks away in the face of the frustration and failure that all real work involves.
Anyone can stay motivated when things are going well. The test — and the skill — is what happens when the project stalls, the code won't work, or the feedback stings. Motivation and resilience are the EQ skills of managing your own drive and bouncing back, turning talent into sustained achievement.
The full pipeline should run in CI with zero API keys and zero network. A deterministic classifier is the test double that makes an agentic system testable.
Put the seam at the router: same interface, a deterministic classifier for tests. The whole orchestration, routing, gateway, human-in-the-loop, and checkpointing, runs in CI with zero API keys and zero network.
Same interface, two implementations: cloud-backed stores when config exists, in-memory when it doesn't. The whole service boots with zero external dependencies.
Same store interface, two implementations: cloud-backed when config exists, in-memory when it doesn't. Don't let a subscription be a hard dependency for make test, it's the ports-and-adapters move teams skip under pressure.
Treat agent output quality like a test. A baseline file plus a gate that fails the build on regression turns "the agents got worse" into a red X.
Agent systems rot because nobody catches quality regressions until users do. A committed baseline plus a gate that fails the build turns the agents got worse into a red X and a reviewable diff in the pull request.
A company can grow revenue explosively, raise huge rounds, and dominate headlines — and still be doomed, if it loses money on every customer. Unit economics is the question underneath all the aggregate financials: does a single customer, on its own, make money? Get that right and scale is the amplifier of a good thing; get it wrong and scale just multiplies the losses. It's the most important economic idea for judging whether a business actually works — and the one flashy growth numbers most often hide.
A company can grow revenue explosively, raise huge rounds, and dominate headlines — and still be doomed, if it loses money on every customer. Unit economics is the question underneath all the aggregate financials: does a single customer, on its own, make money? It's the most important test of whether a business actually works.
Turn ICT asset and third-party risk, incident classification, and reporting into a system you can actually run — with a resilience-testing loop that keeps it honest.
Engineer DORA operational resilience: ICT asset/third-party risk, incident classification, reporting, and resilience testing.
Why Go is shaped the way it is, and how its toolchain — go run, build, test, fmt, vet, mod, doc — turns a small language into a fast, predictable team workflow.
Why Go is shaped the way it is, and how its toolchain — go run, build, test, fmt, vet, mod, doc — turns a small language into a fast, predictable team workflow.
Single-turn evals check one decision. Multi-turn evals check the whole trajectory. A Python harness with three evaluators, an offline test suite, and the judge prompt that actually works.
Most languages treat testing as an afterthought — a separate framework you bolt on, a separate directory, a separate mental mode. Rust treats it as a first-class, built-in feature: testing is part of the language and its tooling, you write tests right next to the code they test, and `cargo test` just works. This tight integration, combined with Rust's culture of correctness, makes testing in Rust unusually pleasant and encourages a habit that pairs perfectly with the compiler's guarantees.
Most languages treat testing as an afterthought. Rust treats it as first-class and built-in: testing is part of the language and tooling, you write tests right next to the code, and cargo test just works. This tight integration makes testing in Rust unusually pleasant.
How Go's constructor-over-an-interface idiom lets you inject a real Foundry agent into a service — and a fake into its test — through the same seam.
Inject a Foundry agent into a service through a two-method ChatAgent interface, so main injects the real agent and the test injects a fake through the same constructor.
Test coverage and observability are the boring infrastructure that makes the interesting changes safe. Notes on how the Picnic team built both, and the on-call experience they enabled.
Twelve months of running multi-agent AI in a regulated context. SLIs that matter, the incident runbook, drift detection, continuous adversarial testing, secret rotation, compliance posture as code.
Fallback agents plus a CI step that replaces the primary agent with one that always errors. If the fallback doesn't produce a usable answer, the PR can't merge.
A recruiter spends 90 seconds on your GitHub before deciding to talk to you. What they're looking for; what makes them skip; what signals matter more than the README.