Building private, offline-capable AI that runs entirely on the user's phone — the case for on-device AI, edge constraints, quantization, the on-device runtime, running Gemma with flutter_gemma, on-device RAG, privacy-by-architecture, and shipping.
For a decade the default answer to "where does the AI run?" was "someone else's GPU, over the network." On-device AI flips that: the model runs on the phone in the user's hand, and once you internalize what that changes — privacy, offline capability, latency, and cost all at once — a whole class of apps becomes possible that a cloud API can't build.
On-device AI flips the default — the model runs on the phone in the user's hand. Once you internalize what that changes — privacy, offline capability, latency, and cost all at once — a whole class of apps becomes possible that a cloud API can't build.
A phone is not a small server — it's a fundamentally different environment with four hard limits a data center never imposes: a tight memory budget, modest and heterogeneous compute, a battery that your model drains, and a thermal ceiling that throttles you when you push it. Every on-device AI decision is a negotiation with these four.
A phone is not a small server — it's a different environment with four hard limits a data center never imposes: a tight memory budget, modest heterogeneous compute, a battery your model drains, and a thermal ceiling that throttles you.
On the cloud, quantization is an optimization you reach for to cut costs. On a phone, it's the difference between the model running and not running at all. Nearly every on-device LLM you'll ever ship is quantized, because full-precision weights simply don't fit — so understanding the bit-width trade-off is non-negotiable for edge AI.
On the cloud, quantization cuts costs. On a phone, it's the difference between the model running and not running at all — nearly every on-device LLM you'll ship is quantized, so the bit-width trade-off is non-negotiable for edge AI.
Between your quantized model file and a running feature sits the runtime — the engine that loads the model, maps it onto the phone's CPU, GPU, or NPU, and turns tokens into text. You rarely write inference math yourself; you pick a runtime and let it handle the brutal complexity of executing a neural network across thousands of different devices.
Between your quantized model file and a running feature sits the runtime — the engine that loads the model, maps it onto the phone's CPU, GPU, or NPU, and turns tokens into text across thousands of different devices.
This is where the concepts become code. With a quantized Gemma model and the flutter_gemma plugin, running a real LLM entirely on the user's phone is a handful of Dart calls — load the model, open a chat session, stream tokens into your UI. The plugin hides the runtime; you write an app.
This is where the concepts become code. With a quantized Gemma model and the flutter_gemma plugin, running a real LLM entirely on the user's phone is a handful of Dart calls — load the model, open a chat session, stream tokens into your UI.
An on-device model only knows what's baked into its weights — nothing about the user's notes, messages, or documents. On-device RAG fixes that by doing retrieval entirely on the phone: embed the user's data locally, store the vectors locally, and retrieve locally, so the model can reason over personal data that never touches a server. It's the technique that makes a private assistant actually useful.
An on-device model only knows what's in its weights. On-device RAG fixes that by doing retrieval entirely on the phone — embed locally, store locally, retrieve locally — so the model can reason over personal data that never touches a server.
On-device AI's biggest promise is privacy — but that promise is only real if the architecture actually keeps data on the device. Privacy isn't a feature you add; it's a property of a design where sensitive data has no path off the phone. This post is about building that property in deliberately, and about the honest hybrid designs for when pure local isn't enough.
On-device AI's biggest promise is privacy — but only if the architecture actually keeps data on the device. Privacy isn't a feature you add; it's a property of a design where sensitive data has no path off the phone.
The demo runs on your phone; now you have to get it running on thousands of phones you'll never touch. Shipping on-device AI has its own hard problems — a model file too big to bundle, first-run downloads, a fleet of wildly different devices, and updates you can't push instantly — and handling them is what separates a hackathon project from a product.
The demo runs on your phone; now you have to get it running on thousands of phones you'll never touch — a model file too big to bundle, first-run downloads, a fleet of different devices, and updates you can't push instantly.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.