#Data Engineering

Articles about Data Engineering — exploring patterns, best practices, and real-world implementations in production systems.

25 posts tagged with data engineering. ← Blog home

#A/B Testing (3)#A2A (14)#ADK (8)#AG-UI (6)#AI (9)#AI Adoption (9)#AI Agents (355)#AI Architecture (22)#AI Cost (10)#AI Cost Optimization (8)#AI Engineering (316)#AI Evaluation (9)#AI Financial Engineering (14)#AI Gateway (8)#AI Governance (43)#AI Red Teaming (9)#AI Research (17)#AI Safety (12)#AI Security (35)#AI Strategy (8)#AI in Production (12)#AML (3)#API Design (17)#API Security (9)#APIs (69)#AWS (17)#Accountability (8)#Accounting (9)#Agent Memory (8)#Agent Skills (11)#Agentic AI (25)#Agentic Commerce (12)#Agentic Engineering (12)#Agentic RAG (8)#Agents (4)#Amazon Bedrock (8)#Analytics (3)#Architecture (45)#Attention (3)#Audit (4)#Authentication (11)#Authorization (3)#Automation (13)#Azure (11)#Azure AI Foundry (9)#Backend Engineering (322)#Benchmarks (3)#Best Practices (5)#BigQuery (6)#Billing (8)#Build Systems (4)#Business Finance (8)#Business Strategy (55)#C (8)#CI/CD (23)#Caching (11)#Capital Markets (14)#Card Payments (12)#Cards (13)#Career (30)#Causal Inference (8)#Cellular Automata (8)#Change Management (3)#Checkpointing (4)#Claude Code (8)#Cloud (5)#Cloud Architecture (3)#Cloud Native (10)#Code Review (9)#Coding Agents (20)#Collaboration (6)#Communication (9)#Complexity (8)#Compliance (54)#Computer Networking (9)#Computer Science (40)#Computer Vision (15)#Computer-Use Agents (8)#Concurrency (42)#Consulting (26)#Containers (10)#Context Engineering (19)#Conversational AI (8)#Cost Optimisation (5)#Cost Optimization (3)#Credit (16)#Credit Risk (16)#CrewAI (8)#Crypto (12)#Cryptocurrency (12)#Cryptography (9)#Custody (9)#DSPy (8)#Data (14)#Data Engineering (25)#Data Governance (3)#Data Science (8)#Data Structures (10)#Databases (48)#Deep Learning (20)#Deployment (5)#Design (3)#Design Patterns (10)#DevOps (24)#DevSecOps (22)#Developer Experience (7)#Developer Productivity (15)#Developer Tools (5)#Differential Privacy (3)#Diffusion Models (10)#Discovery (5)#Distributed Systems (104)#Documentation (3)#Edge AI (8)#Embeddings (21)#Emotional Intelligence (8)#Energy (8)#Engineering (14)#Engineering Culture (3)#Engineering Judgment (7)#Engineering Practices (21)#Enterprise AI (8)#Error Handling (4)#Evaluation (79)#Event-Driven (9)#Event-Driven Architecture (8)#Explainability (3)#Explainable AI (8)#FREE-AI (8)#FX (5)#Feature Engineering (3)#Feature Flags (3)#Feedback (4)#FinOps (23)#FinTech (6)#Financial AI (14)#Financial Engineering (8)#Financial Systems (133)#Fine-Tuning (12)#Fintech (134)#Flutter (8)#Forecasting (10)#Foreign Exchange (5)#Forward Deployed Engineer (43)#Forward Deployed Engineering (8)#Fraud (10)#Fraud Detection (8)#Function Tools (5)#Functional Programming (3)#Fundamentals (23)#Fundraising (8)#GCP (5)#GPU (8)#GUI Automation (8)#Gemma (4)#Generative AI (13)#Git (8)#Go (232)#Go-to-Market (31)#Google ADK (36)#Governance (61)#Granite (6)#Graph Neural Networks (8)#GraphQL (3)#Growth (3)#Guardrails (35)#HIPAA (3)#HTTP (3)#Harness Engineering (8)#Hiring (9)#Hugging Face (8)#Human-in-the-Loop (10)#IBM watsonx (8)#Idempotency (3)#Identity (11)#Image Generation (10)#Inference (8)#Information Retrieval (8)#Integration (3)#Integrations (8)#Intellectual Property (8)#Interfaces (3)#Interpretability (8)#JavaScript (8)#KYC (11)#KYC and AML (12)#Kafka (11)#Knowledge Graphs (11)#Kubernetes (17)#LLM (5)#LLM Inference (8)#LLM Infrastructure (8)#LLM-as-Judge (3)#LLMOps (9)#LLMs (197)#LangChain (8)#LangGraph (11)#Leadership (33)#Ledger (12)#Legal (8)#Lending (14)#Linux (9)#LlamaIndex (8)#Load Balancing (3)#MCP (25)#MLOps (39)#Machine Learning (119)#Marketing (16)#Markets (4)#Mathematical Finance (8)#Memory (16)#Memory Management (5)#Metrics (7)#Microservices (3)#Microsoft Agent Framework (150)#Middleware (6)#Migration (9)#Migrations (8)#Mixture of Experts (5)#Model Risk (3)#Monetization (9)#Monitoring (5)#Monorepo (3)#Multi-Agent (10)#Multi-Agent AI (14)#Multi-Agent Systems (76)#Multimodal (3)#Multimodal AI (11)#NIM (5)#NVIDIA (8)#Networking (3)#OAuth (3)#OWASP (7)#Observability (53)#On-Device AI (8)#Ontology (8)#Open Source (7)#OpenAPI (3)#OpenTelemetry (5)#Operating Systems (9)#Operations (14)#Opinion (6)#Orchestration (10)#Organizational Design (8)#Payment Rails (16)#Payments (55)#People (8)#Performance (56)#Personalization (9)#Platform Engineering (17)#Policy as Code (3)#PreSales (8)#Pricing (4)#Privacy (14)#Privacy Engineering (3)#Process (4)#Product (30)#Product Management (9)#Production (22)#Programming (10)#Programming Languages (48)#Prompt Engineering (86)#Prompt Injection (16)#Proof of Concept (3)#Protocol Buffers (3)#Protocols (9)#Providers (4)#Pydantic AI (8)#Python (142)#Quality (3)#Quantitative Finance (22)#RAG (72)#RBI (3)#REST (5)#Rails (16)#Real-Time (8)#Reasoning Models (8)#Recommender Systems (8)#Reconciliation (5)#RegTech (8)#Regulation (10)#Reinforcement Learning (9)#Reliability (68)#Resilience (4)#Responsible AI (5)#Retrieval (3)#Risk (22)#Risk Management (11)#Rust (32)#SLSA (3)#SRE (23)#SaaS (9)#Sales (9)#Sales Engineering (8)#Scalability (6)#Search (10)#Security (103)#Security Engineering (8)#Self-Evolving Agents (24)#Semantic Web (8)#Sessions (3)#Settlement (9)#Soft Skills (8)#Software (3)#Software Architecture (36)#Software Delivery (9)#Software Engineering (152)#Spanner (4)#Specification Engineering (3)#Speech (8)#Startups (30)#Statistics (8)#Strands (8)#Strategy (6)#Stream Processing (9)#Streaming (38)#Structured Output (4)#Supply Chain Security (10)#Sustainability (9)#System Design (32)#Systems Programming (56)#Technical Sales (6)#Testing (79)#Threat Modeling (3)#Time Series (9)#Tool Use (23)#Tooling (11)#Tools (4)#Trading (8)#Transformers (3)#Treasury (6)#Trust (4)#Type System (3)#Type Systems (12)#TypeScript (8)#Validation (3)#Valuation (3)#Vector Databases (23)#Vector Search (12)#Venture Capital (8)#Version Control (9)#Versioning (7)#Voice AI (9)#Web Development (6)#Webhooks (9)#Workflow (4)#Workflows (14)#eBPF (8)#gRPC (13)#smolagents (8)
Pratik Dhanave · ·4 min read

The Label Problem — Delay, Noise, and the Data Your Own Decisions Hide

Supervised learning assumes someone hands you the truth. In fraud, the truth arrives 30 to 90 days late, labeled by processes with their own agendas, for only the transactions you approved — and silence means nothing, because unreported fraud and un-disputed legitimate purchases are indistinguishable until they aren't. Label engineering is the least glamorous layer of a fraud system and the one that decides whether everything upstream learns reality or a convenient fiction.

Labels are an engineered pipeline: maturity horizons (recent silence is not legitimacy), labels as versioned state trajectories, early proxies weighted then reconciled; noise handled by provenance; and censoring confronted with bought data — bounded exploration of would-be declines booked as data acquisition cost.

Pratik Dhanave · ·7 min read

The Data Foundation — Where Financial AI Lives or Dies

Every AI financial system is built on data, and in finance the data is uniquely treacherous: expensive, fragmented, revised after the fact, survivor-biased, and riddled with subtle ways to accidentally see the future. Getting the data foundation right matters more than any model — a brilliant model on leaky or biased data is a confident path to losing money. This post is about the financial data landscape and the traps that make it the hardest, most decisive part of the work.

Every financial AI system lives or dies on its data, and financial data is uniquely treacherous: expensive, fragmented, revised, survivor-biased, and riddled with ways to accidentally see the future. The landscape (market, document, proprietary data) and the traps — lookahead, survivorship, point-in-time correctness — that make it the most decisive part. (Educational, not financial advice.)

Pratik Dhanave · ·6 min read

The Semantic Layer — The Enterprise Ontology as Shared Meaning

In a large organization the hardest data problem isn't storage or speed — it's that nobody agrees what "customer," "revenue," or "active user" actually means, and every team computes them differently. A semantic layer is the ontology applied to the enterprise: a shared, governed definition of the business's concepts and relationships that sits above the scattered data and gives everyone — dashboards, analysts, and now AI agents — one consistent meaning to build on. This is where ontologies deliver their most concrete business value.

The enterprise ontology made operational: a governed layer that defines the business's entities, relationships, and metrics once, so dashboards, analysts, and AI agents all inherit the same definitions. Why 'revenue' means three things, and why AI made the semantic layer urgent — an agent is only as correct as its grasp of your terms.

Pratik Dhanave · ·7 min read

Building a Knowledge Graph — Construction and Entity Resolution

The hardest part of a knowledge graph isn't querying it or designing the ontology — it's building it from messy, real-world data. Most of your knowledge lives in databases that don't connect, documents that don't parse, and records that refer to the same thing by different names. Construction is the engineering of turning that mess into a clean, connected graph, and its central, unglamorous challenge is entity resolution: deciding when two records are actually the same thing. This post is about the work that makes or breaks a knowledge graph.

The hard part isn't the ontology or the queries — it's construction. The pipeline from heterogeneous sources, and its make-or-break challenge: entity resolution (deciding when two records are the same thing). Under-merging fragments; over-merging asserts false connections; connectivity amplifies every error.

Pratik Dhanave · ·6 min read

Knowledge Graphs — The Ontology, Populated

An ontology is the blueprint; a knowledge graph is the building. A knowledge graph is a network of real-world entities and the relationships between them, structured against an ontology so that both humans and machines can navigate it, query it, and reason over it. They power web search, recommendations, fraud detection, and increasingly the grounding of LLMs — and understanding what makes a graph a "knowledge" graph is understanding why this shape of data keeps winning for connected problems.

A knowledge graph is the ontology populated — entities as nodes, typed relationships as edges, integrated from many sources. RDF vs property graphs, and why graphs win for connected problems: relationships are first-class (traversal, not join cascades), the schema flexes, and meaning lives in structure.

Pratik Dhanave · ·5 min read

Regulatory Reporting

Every compliance control eventually produces something you must tell a regulator: a suspicious activity report, a transaction report, a periodic filing. Regulatory reporting is where your internal compliance data becomes an external, deadline-bound, precisely-formatted obligation — and it's an unglamorous data-engineering problem where accuracy and timeliness are legal requirements, not quality goals.

Every compliance control eventually produces something you must tell a regulator: a suspicious activity report, a transaction report, a periodic filing. Reporting is where internal compliance data becomes an external, deadline-bound obligation where accuracy and timeliness are legal requirements.

Pratik Dhanave · ·6 min read

Data: The Real Determinant

You can pick the perfect technique, tune every hyperparameter, and rent the best GPU — and still get a worse model than you started with, because your dataset was mediocre. Fine-tuning quality is decided overwhelmingly by data quality, and a few hundred excellent examples beat tens of thousands of sloppy ones. This is the post that actually determines whether your fine-tune works.

You can pick the perfect technique and rent the best GPU, and still get a worse model, because your dataset was mediocre. Fine-tuning quality is decided overwhelmingly by data quality, and a few hundred excellent examples beat tens of thousands of sloppy ones.

Pratik Dhanave · ·5 min read

Data Foundations: The Substrate Everything Depends On

Model quality is bounded by data quality, and the defects you tolerate here — poor lineage, silent drift, unmanaged PII, careless retrieval — resurface downstream as hallucinations, bias, privacy incidents, and un-auditable decisions.

Model quality is bounded by data quality; poor lineage, silent drift, unmanaged PII, and careless retrieval resurface as hallucinations and privacy incidents. Phase 2: the governed data foundation.

Pratik Dhanave · ·7 min read

Stream Processing in Practice — Building Real Pipelines

Knowing the concepts and the engines is one thing; building a stream-processing pipeline that's correct, reliable, and operable in production is another. This closing post assembles the series into practical judgment: how to design a streaming pipeline, the operational realities that trip teams up, and when to reach for streaming at all. It turns the theory of data-in-motion into something you can actually run.

Designing a streaming pipeline (source, processing, sink) and the operational realities — it runs forever, state is the crux, backpressure and lag, reprocessing, hard testing — plus the honest judgment: use streaming only when latency genuinely justifies its complexity over simpler batch.

Pratik Dhanave · ·7 min read

Stream Processing Systems — The Engines That Run It

The concepts of stream processing — events, time, windows, state, fault tolerance — are realized by a handful of systems, and choosing among them is one of the main practical decisions in building a streaming pipeline. This post maps the landscape: the engines, the models they embody, and how to think about which fits a problem. It's the bridge from understanding stream processing to actually running it.

The engines that implement stream processing: Flink (the heavyweight true stream processor), Spark Structured Streaming (micro-batch), Kafka Streams (library on Kafka), cloud-managed services, and the unifying Beam/Dataflow model. How to choose by state/correctness, ecosystem, ops capacity, and latency — plus the rise of streaming SQL.

Pratik Dhanave · ·6 min read

Fault Tolerance and Exactly-Once — Not Losing or Double-Counting

A stream-processing job runs forever, which means it will experience failures — nodes crash, networks partition, processes restart. The question isn't whether failure happens but what happens to your results when it does: do you lose events, double-count them, or get exactly the right answer anyway? Fault tolerance, and the famous "exactly-once" guarantee, are what separate a toy stream processor from one you can trust with money. This post is about surviving failure correctly.

A forever-running job will fail, so the delivery guarantee matters: at-most-once (may lose), at-least-once (may duplicate), or exactly-once (correct as if neither). How checkpointing and replayable sources achieve exactly-once semantics, and choosing the guarantee by need — with idempotency as the pragmatic route.

Pratik Dhanave · ·7 min read

State and Stateful Stream Processing

Simple stream operations — filter this event, transform that one — need no memory. But the interesting computations — counts, aggregations, joins, windows, pattern detection — require remembering things across events, and that memory is state. Managing state correctly over an unbounded stream, at scale, with fault tolerance, is the hardest engineering problem in stream processing. This post is about state: why it's needed, what makes it hard, and how stream processors manage it.

Interesting stream computations — counts, aggregations, joins, windows, patterns — need to remember across events, and that memory is state. Why state is hard on an unbounded stream (bounding, speed, partitioning, surviving failure), and how engines provide managed keyed state.

Pratik Dhanave · ·7 min read

Windowing — Turning an Infinite Stream into Finite Answers

You can't average an infinite stream, or count "all" the events in something that never ends. Windowing is how stream processing carves the unbounded flow into finite, computable chunks — the last minute, each hour, each user session — so that aggregations become meaningful. It's the single most-used concept in practical stream processing, and understanding the window types is understanding how most real streaming logic is expressed.

You can't aggregate an infinite stream, so windowing carves it into finite chunks. Tumbling, sliding, and session windows; triggers that control when a window emits; and how windowing sits on event time and watermarks to produce correct, finite results — the dominant pattern in real stream processing.

Pratik Dhanave · ·7 min read

Watermarks — Reasoning About Time in an Out-of-Order Stream

Post 2 ended on a hard problem: you need event-time results, but events arrive late and out of order, so you never know when you've seen everything for a given time. Watermarks are stream processing's elegant answer — a running estimate of how far event time has progressed, letting you decide when it's safe to produce a result. They're the mechanism that makes correct event-time computation possible over a messy, unbounded stream.

Watermarks are stream processing's answer to late, out-of-order events: a running estimate of how far event time has progressed, letting you decide when it's safe to emit a result. How they make the completeness-vs-latency tradeoff explicit and tunable, and how to handle late events.

Pratik Dhanave · ·7 min read

Events, Streams, and Time — The Core Model

Stream processing has a small, elegant core model: everything is an event, events form streams, and time is a first-class concern. Getting these three ideas precise — especially the subtle and crucial distinction between when an event happened and when you process it — is what makes the rest of stream processing (windowing, state, correctness) comprehensible. This post builds the core vocabulary that everything else depends on.

The core model: events, streams, and time as first-class. The crucial, most-misunderstood distinction between event time (when it happened) and processing time (when you handle it), why correct answers depend on event time, and why that makes time the central problem of streaming.

Pratik Dhanave · ·6 min read

What Stream Processing Is — Computing on Data in Motion

Most software processes data at rest: you store it, then query it. Stream processing flips that — it computes on data in motion, as each event arrives, continuously and forever. This shift from "store then process" to "process as it flows" underpins fraud detection, real-time analytics, monitoring, and the event-driven systems that react to the world as it happens. This series builds stream processing from first principles, starting with why computing on data in motion is a fundamentally different model.

Most software processes data at rest; stream processing computes on data in motion, as each event arrives, continuously and forever. Batch vs stream, why unboundedness makes streaming hard, and when latency justifies its complexity over simpler batch.

Pratik Dhanave · ·8 min read

Data Engineering in Practice

All the components — pipelines, warehouses, models, batch and streaming, the modern stack, quality and governance — come together in a single job: keep reliable, usable data flowing to the people and systems that need it. Doing that in the real world is less about any one technology than about a mindset: treating data pipelines as production software that must be reliable, tested, observed, and maintained. This closing post is about data engineering as it's actually practiced, and where it's heading as AI makes good data more valuable than ever.

All the components come together in one job: keep reliable, usable data flowing to the people and systems that need it. Doing that in the real world is less about any one technology than about a mindset — treating data pipelines as production software that must be reliable, tested, observed, and maintained. Plus where the field is heading as AI makes good data more valuable than ever.

Pratik Dhanave · ·8 min read

Data Quality and Governance

The most dangerous failure in data engineering is the one that doesn't announce itself. A crashed pipeline gets noticed; silently wrong data flows straight into dashboards and models, where people trust it and make decisions on it — wrong decisions, confidently made. Data quality and governance are the disciplines that guard against this: ensuring data is correct, trustworthy, and well-managed. They're the least glamorous and most important part of production data engineering, because without them, all the pipelines and warehouses just deliver garbage efficiently.

The most dangerous failure in data engineering is the one that doesn't announce itself. A crashed pipeline gets noticed; silently wrong data flows straight into dashboards and models, where people trust it and make wrong decisions confidently. Data quality and governance are the disciplines that guard against this.

Pratik Dhanave · ·8 min read

The Modern Data Stack

A decade ago, building a data platform meant heavy, monolithic, on-premises systems and long projects. Today it's assembled from specialized cloud tools that snap together around a central cloud data warehouse — ingest here, transform there, visualize over there — each best-in-class at one job. This "modern data stack" is less a specific set of products than an architecture and a philosophy, and understanding its shape (and what drove it) is understanding how data platforms are actually built now.

A decade ago, a data platform meant heavy, monolithic, on-premises systems. Today it's assembled from specialized cloud tools that snap together around a central cloud data warehouse — each best-in-class at one job. This 'modern data stack' is an architecture and a philosophy, and understanding its shape is understanding how data platforms are built now.

Pratik Dhanave · ·7 min read

Batch vs Streaming

How fresh does the data need to be? That one question splits data engineering into two paradigms. Batch processing handles data in large chunks on a schedule — simpler, cheaper, and fine when yesterday's data is good enough. Stream processing handles data continuously as it arrives — more complex and costly, but necessary when you need to know now. Choosing between them (and knowing when each fits) is one of the most consequential architectural decisions in a data platform, and it's driven by real requirements, not by which sounds more impressive.

How fresh does the data need to be? That one question splits data engineering into two paradigms. Batch processing handles data in chunks on a schedule — simpler and cheaper. Stream processing handles data continuously as it arrives — more complex, but necessary when you need to know now.

Pratik Dhanave · ·8 min read

Data Modeling for Analytics

The same data can be organized in ways that make analytical queries fast, intuitive, and cheap — or slow, confusing, and expensive. Data modeling is the craft of that organization, and it's where a counterintuitive truth lives: the careful normalization that's correct for application databases is often exactly wrong for analytics. Analytical data wants to be shaped differently, around how questions are asked rather than how data is written, and understanding dimensional modeling is understanding how to make a warehouse actually usable.

The same data, organized differently, makes analytical queries fast and intuitive — or slow and confusing. Data modeling is the craft of that organization, and it holds a counterintuitive truth: the normalization that's correct for application databases is often exactly wrong for analytics.

Pratik Dhanave · ·7 min read

Where Data Lives: Warehouses, Lakes, and Lakehouses

"Just put it in a database" stops working the moment you're dealing with analytics at scale — because the database that runs your application is optimized for exactly the wrong thing. Analytical data needs different storage: systems built to scan and aggregate huge volumes, not to serve fast individual transactions. The landscape of analytical storage — data warehouses, data lakes, and the newer lakehouses — is where data engineering decisions about where data lives get made, and understanding the differences (starting with OLTP vs OLAP) is essential.

'Just put it in a database' stops working the moment you're doing analytics at scale — the database that runs your app is optimized for exactly the wrong thing. Analytical data needs different storage: warehouses, lakes, and lakehouses. Understanding the differences (starting with OLTP vs OLAP) is essential.

Pratik Dhanave · ·7 min read

Data Pipelines and ETL/ELT

The core artifact of data engineering is the pipeline: an automated flow that pulls data from somewhere, reshapes it, and lands it somewhere useful. And the single most consequential shift in modern data engineering is captured in three reordered letters — ETL became ELT — a change driven by cheap, powerful cloud data warehouses that flipped when and where transformation happens. Understanding pipelines, and the ETL-to-ELT shift, is understanding how data actually moves and gets made usable.

The core artifact of data engineering is the pipeline: an automated flow that pulls data from somewhere, reshapes it, and lands it somewhere useful. And the single most consequential modern shift is captured in three reordered letters — ETL became ELT — driven by cheap, powerful cloud data warehouses that flipped when and where transformation happens.

Pratik Dhanave · ·6 min read

What Data Engineering Is

Every dashboard, every analytics query, every machine-learning model, and every "data-driven decision" rests on an invisible foundation: someone built the pipelines that collect, move, clean, and organize the data so it's actually usable. That someone is a data engineer, and their work is the unglamorous, essential plumbing beneath everything data. When it works, no one notices; when it breaks, every downstream report and model breaks with it. Understanding data engineering is understanding how raw data becomes something a business can actually use.

Every dashboard, analytics query, and ML model rests on an invisible foundation: someone built the pipelines that collect, move, clean, and organize the data. That someone is a data engineer, and their work is the essential plumbing beneath everything data. When it works, no one notices; when it breaks, every downstream report and model breaks with it.

All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse the blog or explore related topics using the tag cloud above.