Archive
1290 posts · Page 12 of 108. ← Blog
Computer-use agents are impressive in demos and frustrating in production, and the gap is reliability. Driving a human interface — imprecise perception, imprecise actions, brittle UIs, long multi-step tasks — accumulates errors in ways that make these agents genuinely hard to trust for important work. This post is an honest look at the reliability challenges, why they're inherent, and how to work within them.
Impressive in demos, frustrating in production — the gap is reliability. Error accumulation over long tasks (0.95^20 ≈ 36%), brittleness to UI change, imprecise perception/action, and weak visual verification make these agents structurally hard to trust, and how to work within the limits (short tasks, checkpoints, human-in-the-loop).
Now we open up the mechanism at the heart of modern LLMs. A Mixture-of-Experts layer replaces the transformer's single feed-forward block with many parallel "experts" and a router that sends each token to just a few of them. The result is a model with enormous total capacity that spends only a little compute on any given token — the decoupling the first post promised, made concrete.
A Mixture-of-Experts layer replaces the transformer's single feed-forward block with many parallel experts and a router that sends each token to just a few of them. The result is a model with enormous total capacity that spends only a little compute on any given token — the decoupling of capacity from per-token compute, made concrete. Experts, routing, and how outputs combine.
Continuous Integration is the least glamorous and most important half of CI/CD. It's also the most misunderstood — teams install a build server, call it "CI," and miss the actual practice, which is a discipline about how often you merge, not which tool runs your tests. This post is about the real thing: integrating small, integrating often, and keeping the mainline always green.
Continuous Integration is the least glamorous and most important half of CI/CD, and the most misunderstood — teams install a build server, call it CI, and miss the actual practice, which is a discipline about how often you merge, not which tool runs your tests. This post is the real thing: integrate small, integrate often, keep the mainline always green.
Everything before this post raised the probability barrier against injection. This post lowers the impact — and impact is what actually protects you. The load-bearing defense against prompt injection isn't a prompt or a filter; it's an architecture where a fully-hijacked model still can't do anything catastrophic, because it was never granted the power to.
Everything before this raised the probability barrier against injection; this post lowers the impact — and impact is what actually protects you. The load-bearing defense isn't a prompt or a filter but an architecture where a fully-hijacked model still can't do anything catastrophic: least privilege, the confused-deputy trap, human-in-the-loop, and the dual-LLM pattern.
AI governance fails most often not because an organization lacks principles, but because nobody owns them. Fairness, safety, and accountability are everyone's job in the abstract and no one's job in practice, so they fall through the cracks between data science, legal, risk, and engineering. This series is about governance as an operating model you can actually run — and it starts with the unglamorous foundation: clear ownership, defined roles, and the mechanisms that turn good intentions into enforced practice.
AI governance fails from an ownership gap, not an absence of principles. The operating model: decision rights and accountability via the three lines of defense, named roles laddering to an executive owner, and the mechanisms that give them teeth — an AI system registry, risk tiering, and lifecycle gates.
A computer-use agent can perceive the screen and decide "click Submit," but it still has to answer a deceptively hard question: where exactly is Submit, and how do I act on precisely that spot? This UI grounding problem — connecting a decision to a precise screen location and element — is where many computer-use agents stumble, because a click a few pixels off hits the wrong thing. This post is about how agents locate and target UI elements reliably.
Deciding 'click Submit' is easy; knowing exactly where Submit is and clicking precisely there is the grounding problem — where a click a few pixels off hits the wrong thing. Coordinate vs element-based vs set-of-marks grounding, why precise localization is hard for vision, and why mis-clicks are a top failure mode.
Mixture of Experts is a modification to the transformer, so you can't understand modern LLM architecture without a clear picture of the transformer itself. This post is a focused recap of the parts that matter for the rest of the series: attention as the mechanism that mixes information across tokens, the feed-forward block that MoE replaces, and how the pieces stack into the model everyone is now modifying.
Mixture of Experts is a modification to the transformer, so you can't understand modern LLM architecture without a clear picture of the transformer itself. This post is a focused recap of the parts that matter: attention as the mechanism that mixes information across tokens, the feed-forward block that MoE replaces, and how the pieces stack into the model everyone is now modifying.
CI/CD is one of those acronyms everyone uses and few can define precisely. Strip away the tooling and it's an answer to a simple, painful question: how do you take a change a developer just wrote and get it safely into users' hands, quickly and repeatedly, without fear? This opening post defines the terms from first principles and explains the problem they solve.
CI/CD is an answer to a simple, painful question: how do you take a change a developer just wrote and get it safely into users' hands, quickly and repeatedly, without fear? This opening post defines the terms from first principles — CI, Continuous Delivery vs Deployment, the pipeline — and explains the integration-hell and deployment-fear problems they were invented to kill.
Between filtering the input and re-architecting the system sits a tempting middle ground: make the prompt itself more resistant. Delimiters, spotlighting, instruction placement, and defensive system prompts all raise the cost of an attack. None of them close the hole — because they are all still text in the one channel the attacker also writes to — but used well they meaningfully shift the odds.
Between filtering input and re-architecting the system sits a tempting middle ground: make the prompt itself more resistant. Delimiters, spotlighting, instruction placement, and defensive system prompts all raise the cost of an attack — but none close the hole, because they're all still text in the one channel the attacker also writes to. The value is knowing exactly how much they buy.
Having understood the screen, a computer-use agent must act on it — move the cursor, click the right spot, type the right text — and then look again to see what happened. This perceive-act loop, running over a graphical interface with imprecise actions and visual feedback, is the engine of a computer-use agent, and it's both simple in shape and hard in execution. This post covers how agents act on a GUI and the loop that ties perception and action together.
An agent acts by simulating human input — mouse, keyboard, navigation — translating intent into precise physical actions on a visual surface where a click can mechanically succeed yet do the wrong thing. The perceive-act loop (visual on both ends, slow, per-step-fallible), and why closing it (verify, adapt) beats brittle scripted RPA.
The frontier language models of the last few years share a structural secret that isn't obvious from the outside: most of them aren't dense. They're sparse — built from Mixture-of-Experts layers that let a model have hundreds of billions of parameters while only using a fraction of them on any given token. Understanding why models moved from dense to sparse is the key to understanding how modern LLMs are actually built.
Frontier language models share a structural secret that isn't obvious from the outside: most aren't dense. They're sparse — built from Mixture-of-Experts layers that let a model have hundreds of billions of parameters while using only a fraction on any given token. Understanding why models moved from dense to sparse is the key to understanding how modern LLMs are built.
The first instinct when facing prompt injection is to inspect the input and block the bad stuff. It's a reasonable layer — but a treacherous one, because it creates a feeling of safety far larger than the protection it provides. This post covers the input-side defenses that are genuinely worth having, and draws a hard line around what they can and cannot do.
The first instinct against injection is to inspect the input and block the bad stuff. It's a reasonable layer but a treacherous one — it creates a feeling of safety far larger than the protection it provides. This post covers the input defenses genuinely worth having, and draws a hard line around what they can't do (starting with: indirect injection bypasses them entirely).