Data Provenance For RAG Systems
A citation can tell a reader where to look. Provenance must explain which exact source version, transformation, chunk, index, retrieval decision, and model context produced the claim they are reading.
A URL is not enough provenance
The original RAG paper by Patrick Lewis and coauthors described generation grounded in explicit non-parametric memory and identified provenance as part of the motivation. In production, displaying a source URL is useful but incomplete. A page can change, a PDF can be replaced, a creator can be omitted, and a citation can point to a document that does not support the nearby claim.
Useful provenance answers a reproducibility question: what exact evidence did the system have at answer time? Preserve immutable source identity, content hash, version or retrieval timestamp, author or creator, publisher, license, access policy, parser output, chunk boundaries, embedding model, index snapshot, query transformations, ranking, selected context, prompt version, model, answer claims, and citation mapping.
OriginCan the user identify the creator, publisher, canonical source, rights, and exact content version?
TransformationCan operators reconstruct parsing, OCR, normalization, chunking, enrichment, and embedding?
AuthorizationWere source ACLs and tenant boundaries enforced at ingestion, retrieval, cache, and answer time?
AttributionDoes each material claim point to passages that actually support it, at useful granularity?
FreshnessCan changed, revoked, deleted, or superseded content be found and removed from future answers?
Model provenance as entities, activities, and agents
W3C PROV provides a practical mental model: entities such as documents, chunks, embeddings, indexes, prompts, and answers are generated or derived through activities such as ingestion, chunking, indexing, retrieval, and generation, associated with agents such as creators, publishers, services, users, and models.
OpenLineage can capture datasets, jobs, runs, and extensible facets for the ingestion pipeline. It does not replace answer-level lineage, but it can connect a RAG chunk and index snapshot back to the source pipeline run that produced them. Stable IDs are essential; filenames and display URLs are not identifiers.
RAG lineage graph from creator to answer claim
01 SourceCreator and publicationCreator, publisher, canonical URI, issue date, version, license, jurisdiction, and content hash.
02 AcquireIngestion eventConnector, fetch time, response metadata, signature or credential, ACL, consent, and retention policy.
03 TransformParsed documentParser or OCR version, extracted structure, normalization, language, tables, images, and quality flags.
04 SegmentChunk lineageStable chunk ID, parent document, byte or page range, heading path, overlap, and chunker version.
05 RepresentEmbedding and indexEmbedding model and dimensions, vector hash, metadata filters, collection, and immutable index snapshot.
06 RetrieveSelection decisionUser and tenant context, query variants, filters, candidate scores, reranker, selected chunks, and order.
07 GenerateAnswer contextPrompt template, model, parameters, context positions, truncation, tool results, and response identifier.
08 AttributeClaims and citationsClaim spans, supporting chunk spans, entailment checks, confidence, creator credit, and visible source link.
Retrieval logs must preserve the decision, not raw sensitive text
At answer time, log which index snapshot was queried, which authorization context and metadata filters applied, whether the query was rewritten, candidate IDs and scores, reranking version, deduplication, selected chunks, context order, and dropped passages. OpenTelemetry's Generative AI conventions include retrieval events and document identifiers that can help correlate these operations.
Avoid turning observability into data exfiltration. Query text, document content, and prompts may contain personal or tenant-sensitive data. Prefer stable IDs, hashes, classifications, bounded snippets, and protected evidence stores. Keep high-cardinality provenance in an audit or lineage system rather than normal metrics.
Citations should be claim-level and creator-aware
A citation should connect a claim to one or more exact supporting spans. The visible reference should credit the creator when available, name the work and publisher, show publication or update date, link to the canonical source, and disclose license or access restrictions where relevant. A generic domain name hides the people and organizations that created the knowledge.
Do not let the model invent references from memory. Give the generator opaque source IDs, then resolve visible citations server-side from the provenance registry. Validate that cited chunks were actually in context and support the claim. Research such as RAGAS, ARES, ReClaim, and CiteFix demonstrates that retrieval relevance, answer faithfulness, and citation correctness are separate problems.
| Failure | Why the answer can look trustworthy | Provenance evidence that exposes it | Control |
|---|---|---|---|
| Correct URL, wrong version | The linked page exists, but changed after indexing. | Source hash, version, fetch timestamp, archive or immutable object ID. | Display answer-time version and detect source drift before reuse. |
| Correct document, unsupported claim | The citation is topically related but does not entail the sentence. | Claim span to chunk span mapping and support evaluation result. | Claim-level citation validation and abstention for unsupported claims. |
| Stale or revoked source | The vector remains retrievable after deletion, correction, or permission change. | Source status, deletion event, index snapshot, cache keys, and affected answers. | Tombstone lineage descendants, rebuild indexes, invalidate caches, and audit replay. |
| Cross-tenant retrieval | The answer is fluent and may not reveal the leaked source directly. | User, tenant, ACL snapshot, filters, candidate IDs, and selected chunks. | Authorization before retrieval, security-trimmed indexes, and leakage tests. |
| Creator or license lost | The answer links to a mirror or strips publication metadata. | Canonical source, creator, publisher, license, acquisition path, and transformation record. | Require attribution metadata before indexing and preserve it through every derivative. |
| Chunk context distortion | A passage is quoted without heading, table, qualifier, or adjacent exception. | Parent range, heading path, neighboring chunks, parser structure, and context order. | Structure-aware chunking, expandable citations, and context-completeness checks. |
| Index or model regression | The same query suddenly produces a different supported answer. | Embedding, index, reranker, prompt, model, parameters, and evaluation versions. | Versioned replay sets, lineage-aware comparison, canaries, and rollback. |
Deletion and correction must propagate through the graph
When a source is corrected, revoked, expires, changes license, or receives a deletion request, the system must find every derived chunk, embedding, index snapshot, cache entry, evaluation sample, and stored answer. Provenance makes impact analysis possible; without it, teams can delete the original object while stale derivatives continue answering.
Keep tombstones and revocation evidence without retaining deleted sensitive content. Rebuild or incrementally repair affected indexes, invalidate semantic caches, and mark prior answers as stale when their supporting evidence is no longer valid. The provenance graph is also the deletion graph.
Evaluation should diagnose the stage that failed
A single answer-quality score is operationally weak. Measure source freshness and rights completeness, retrieval recall and relevance, ACL leakage, context completeness, answer faithfulness, citation precision and recall, creator attribution coverage, and abstention quality. Preserve the lineage for every evaluation result so regressions can be tied to a source, parser, chunker, embedding, retriever, reranker, prompt, or model change.
NIST's Generative AI Profile recommends evaluating content provenance in real scenarios and against defined metrics. For RAG, that means testing not only whether an answer sounds correct, but whether its support is traceable, authorized, current, and presented honestly.
What I would build
I would build a provenance registry with immutable IDs for sources, source versions, chunks, embeddings, indexes, retrieval runs, context bundles, answers, claims, and citations. Ingestion would emit OpenLineage-compatible events; online requests would emit retrieval and generation traces linked to the registry.
The reader interface would show concise citations by default and expand into creator, publication, version, license, exact supporting passage, retrieval time, and freshness status. Operators would get reverse lineage: starting from a bad source or parser version, find every affected answer.
The design principle
Trust in RAG does not come from adding footnote-shaped text. Preserve the graph from creator to claim, resolve citations from immutable evidence, enforce rights and authorization at every derivative, and make correction and deletion propagate through the same lineage.