The Data Foundation — Where Financial AI Lives or Dies

Post 1 framed the discipline; this post is its foundation. As in all ML (the ML foundations series), the data decides the outcome — but financial data has failure modes most ML practitioners never encounter, and missing them produces backtests that look spectacular and lose money in production. Understanding financial data is understanding where financial AI actually lives or dies.

The financial data landscape

Financial AI draws on several distinct kinds of data, each with its own character:

• Market data — prices, volumes, quotes, order books for traded instruments, often high-frequency and high-volume. The classic quant input: structured, abundant for liquid instruments, but sparse and unreliable for illiquid ones (much of private credit and structured finance has no market price at all).
• Reference and fundamental data — the descriptive and financial facts: instrument terms, company financials, ratings, corporate structure, holdings. Structured but fragmented across providers and frequently needing reconciliation.
• Document data — the vast unstructured corpus finance runs on: filings, offering memos, indentures, contracts, research, news. Historically unusable at scale by machines; now the frontier that LLMs unlock (post 7). This is often where the real, differentiated signal lives — and where AI’s new reading ability matters most.
• Alternative data — non-traditional signals (satellite imagery, transactions, web, sentiment) used to gain an edge, valuable but noisy, expensive, and often of uncertain provenance and legality.
• Internal/proprietary data — a firm’s own history: its past deals, decisions, monitoring records, and outcomes. This is frequently the most valuable data an institution has (it’s unique to them and encodes their judgment) and the foundation of institutional memory (post 12) — yet it’s usually scattered, unstructured, and underused.

The practical reality: financial data is expensive, fragmented across many vendors and internal systems, and inconsistent — much of the work is acquiring, cleaning, reconciling, and integrating it before any modeling. And crucially, the most valuable data is often the unstructured document data and the proprietary internal data, precisely the kinds AI is newly able to exploit — which is a big part of why AI financial engineering is possible now.

The traps that make financial data uniquely dangerous

Financial data has specific, severe failure modes that generic ML practitioners rarely face — and each one produces results that look great and are wrong, which is the dangerous kind:

• Lookahead bias / data leakage — using information that wouldn’t have been available at the decision time. This is the cardinal sin (as in time-series forecasting), and finance is full of subtle ways to commit it: using a financial figure before its actual release date (not its reporting date), using revised historical data that didn’t exist in that form at the time, or any feature that peeks ahead. Leakage inflates backtests spectacularly and vanishes in production — the number-one way financial AI fools its builders.
• Survivorship bias — analyzing only the entities that survived. A dataset of “current companies” or “funds that still exist” silently omits the ones that went bankrupt or closed — exactly the failures a credit or risk model most needs to learn from. Training on survivors makes everything look safer than it is, systematically underestimating risk.
• Point-in-time correctness. The fix for lookahead and revision bias: the data must reflect what was known when, not today’s corrected values. Financial figures get revised (restatements, rating changes, backfilled data), so a proper system stores data as it was known at each date (“point-in-time” or “bitemporal” data) so models only ever see what was actually available. Building point-in-time-correct data is hard, essential, and the thing most amateur financial-AI efforts get wrong.
• Non-stationarity (post 5) — the data-generating process changes over time (regime changes, structural breaks), so even leak-free historical data may not represent the future. The past is a flawed guide, and the data foundation must account for it (shorter relevant history, regime awareness).
• Sparsity and illiquidity — for private credit, structured products, and illiquid assets, there’s little or no market data and few comparable instruments, so you can’t rely on the data abundance that liquid-market quant assumes. Much of the signal must come from documents and fundamentals instead.
• Provenance and quality — vendor data has errors, gaps, and inconsistent definitions; alternative data has uncertain sourcing and legality. In a domain where decisions are audited, where each datum came from and how reliable it is must be tracked (provenance), tying into governance (post 11).

The unifying lesson: in finance, the ways data can betray you are more numerous and more subtle than in typical ML, and they share a signature — they produce optimistic, believable, wrong results. A financial AI system is only as trustworthy as its defenses against these traps.

Building the data foundation right

Given all this, the data foundation is the highest-leverage engineering in financial AI, and doing it well means:

• Point-in-time everything. Architect the data so every feature and every backtest sees only what was knowable at that moment — this single discipline prevents the most catastrophic errors (lookahead, revision bias) and is the foundation of trustworthy backtesting (post 5).
• Integrate structured and unstructured. The pipeline must bring together market/reference data and the information extracted from documents (post 7) and internal records, because the edge increasingly comes from combining them — structured facts plus what the documents actually say.
• Track provenance and quality relentlessly. Every datum should carry where it came from, when it was known, and how reliable it is — essential for auditability (post 11) and for debugging (most production incidents are data problems, as in all ML).
• Treat proprietary data as the crown jewel. A firm’s own deal, decision, and outcome history is its most defensible asset; structuring and preserving it (post 12) turns scattered institutional knowledge into a reusable foundation that competitors can’t buy.
• Respect sparsity and non-stationarity. Don’t assume liquid-market data abundance; lean on documents and fundamentals where market data is thin, and design for regime change rather than assuming a stable past.

The data foundation is unglamorous and decisive: it’s where most of the real effort goes, where financial AI most often fails silently, and where the defensible edge is actually built. Every later post — pricing, risk, ML, underwriting, monitoring — rests on it, and no model is better than the point-in-time-correct, provenance-tracked, leak-free data beneath it. In financial AI more than anywhere, get the data right and the rest is tractable; get it wrong and the fanciest model just loses money with confidence.

The throughline: financial AI lives or dies on its data foundation, drawing on market data (structured, abundant for liquid instruments, absent for private/illiquid), reference/fundamental data (fragmented, needs reconciliation), document data (the vast unstructured corpus LLMs now unlock — often where real signal lives), alternative data (edge but noisy/uncertain), and proprietary internal data (a firm’s own deals/decisions/outcomes — its most valuable, defensible asset, and the base of institutional memory). Financial data is expensive, fragmented, and inconsistent, and riddled with severe traps that produce optimistic-believable-wrong results: lookahead bias/leakage (the cardinal sin — using not-yet-available info), survivorship bias (omitting the failures a risk model most needs), revisions (demanding point-in-time correctness — store data as known at each date), non-stationarity, sparsity/illiquidity, and provenance/quality issues. So building it right means point-in-time everything, integrating structured + unstructured, tracking provenance/quality, treating proprietary data as the crown jewel, and respecting sparsity and regime change — the highest-leverage, most decisive, most often-silently-failed part of the work.

Key takeaways

• The financial data landscape: market data (structured, abundant for liquid instruments, absent for private credit/illiquid assets), reference/fundamental (fragmented, reconciliation-heavy), document data (the vast unstructured corpus — filings/memos/indentures/news — LLMs now unlock, often where the real signal is), alternative data (edge but noisy/uncertain provenance), and proprietary internal data (a firm’s own deals/decisions/outcomes — its most valuable, defensible asset; the base of institutional memory, post 12).
• Financial data is expensive, fragmented across vendors/systems, and inconsistent — most of the work is acquiring, cleaning, reconciling, and integrating it before any modeling.
• Severe, finance-specific traps all produce optimistic-believable-wrong results: lookahead bias/leakage (the cardinal sin — using info unavailable at decision time; release vs reporting dates, revised data), survivorship bias (training only on survivors omits the failures a credit/risk model most needs → underestimates risk), non-stationarity (the past is a flawed guide), sparsity/illiquidity (no data abundance for private/illiquid), and provenance/quality issues.
• The fix for the worst errors is point-in-time correctness: store data as it was known at each date (bitemporal), so models/backtests only ever see what was actually available — hard, essential, and what most amateur efforts get wrong.
• Build the foundation right: point-in-time everything, integrate structured + unstructured + internal data, track provenance and quality relentlessly (for audit, post 11, and debugging), treat proprietary data as the crown jewel (post 12), and respect sparsity and non-stationarity — it’s the highest-leverage, most decisive, most often-silently-failed part of financial AI, and every later post rests on it.

Further reading

• Market data — the structured quant input
• Alternative data (finance) — non-traditional signals and their pitfalls