Search⌘ K
AI Features
Home
Courses
Google FDE Interview Prep

Building Enterprise Data Pipelines

Explore the processes of building and managing enterprise data pipelines focusing on document ingestion, extraction quality, version control, and deletion proofing. Understand how to link authoritative sources to derived data, enforce access controls, handle failures, and maintain pipeline integrity using tools like BigQuery and Python. This lesson equips you to design resilient data systems that handle updates, corrections, and cleanup while preserving traceable, auditable evidence.

We'll cover the following...

A source owner deletes a manual. What would prove that the agent can no longer use it? An extraction job reporting success cannot answer that question: you need the authoritative document identity, its last revision, its derived outputs, the index’s acknowledged state, and the rule controlling which evidence reaches the agent. Those same mechanisms determine whether an update supplies current evidence or leaves contradictory instructions searchable.

Following D41 downstream

Consider a legacy maintenance repository supplying manuals and equipment records. BigQuery holds normalized equipment rows, extraction metadata, quarantine records, and a publication manifest; a separate retrieval index holds document chunks.

A publication manifest records the intended outputs for a source version and the evidence that each required destination accepted them. It connects source truth to serving eligibility. The source remains authoritative; both normalized rows and indexed chunks are derived representations.

For manual D41, deletion evidence must connect four things:

  • Source: An attributable deletion event carrying D41’s deletion revision.

  • Manifest: Durable state making every D41 revision ineligible.

  • Serving: Enforcement of that state whenever retrieved material enters an answer, including previously retrieved material reused by the application.

  • Cleanup: Acknowledged removal from derived rows, chunks, and indexes, with unresolved work retained for retry.

Logical exclusion can take effect before physical cleanup finishes. Physical purge, backup handling, and retention obligations need a separately agreed policy.

The maintenance repository supplies D41's source identity and revision, raw input where permitted, and parser or OCR fixtures; equipment records follow a separate validation path, versioned outputs publish independently to BigQuery normalized records and the retrieval index, and the BigQuery publication manifest records sink acknowledgments and immediately excludes deleted evidence at the serving eligibility check while physical cleanup remains retryable.
The maintenance repository supplies D41's source identity and revision, raw input where permitted, and parser or OCR fixtures; equipment records follow a separate validation path, versioned outputs publish independently to BigQuery normalized records and the retrieval index, and the BigQuery publication manifest records sink acknowledgments and immediately excludes deleted evidence at the serving eligibility check while physical cleanup remains retryable.

The serving check is essential during sink failures: a chunk’s physical presence alone says nothing about whether it remains admissible evidence.

From file to evidence

Ingestion is a repeatable transformation pipeline:

  1. Discover or receive source changes.

  2. Retain attributable source inputs where permitted.

  3. Select a parser or OCR path.

  4. Normalize extracted documents or incoming records.

  5. Validate content and structure.

  6. Attach provenance and access metadata.

  7. Stage versioned outputs and publish them.

  8. Reconcile derived state against the source.

Parsing extracts existing document structure and text. Optical character recognition, or OCR, recognizes text in images. A born-digital manual can provide headings, page boundaries, tables, units, and warnings through parsing; a scanned page needs OCR before equivalent normalization.

Preserve the relationships between extracted values. A flattened table can contain every word yet associate one equipment model with another model’s service interval. Extraction success and OCR confidence therefore cannot establish business-field correctness.

Keep the maintenance-record feed separate from document extraction. Its normalized row is:

{"equipment_id": "E17", "inspection_interval_days": 60}

Require a nonempty source-mapped equipment identifier and an integer interval in the approved domain; reject the string “60” unless an explicit normalization rule permits conversion. Give this feed its own source keys, schema version, and insert/update/delete semantics.

Attributing page 3

D41 revision 1 contains two page-3 sections:

  • Inspection interval: 90 days

  • Disconnect power before inspection

They become D41:r1:0 and D41:r1:1. Each chunk retains:

  • Source attribution: D41’s source location, revision, page and section references, and source-content hash.

  • Transformation versions: Parser version and schema version.

  • ...