feat(cookbooks): add Knowledge Catalog cookbook suite - #284
Open
hyunuk wants to merge 24 commits into
Open
Conversation
- Add verified metadata_as_code_dark_data.ipynb cookbook with 3-layer test verification - Add initial draft cookbooks (curated_data_products, lineage_graph_observability) and sample manuals
- Add verified curated_data_products.ipynb with 3-layer test verification - Add data contract SLA aspect and AI golden queries grounding
- Add verified lineage_graph_observability.ipynb with 3-layer test verification - Add programmatic Lineage API querying, topology visualization, and root-cause drift isolation
… and branding - Remove unused tabulate dependency and BadRequest import - Eliminate directional words and 1st-person plural pronouns - Replace em dashes with standard commas per style guide - Update Universal Catalog references to Knowledge Catalog
…a_products with style guide - Add architectural rationale for zero-copy data mesh packaging, machine-readable SLAs, and protobuf integer indexing - Explain golden queries grounding for AI agents and dual-consumer patterns - Eliminate directional words and 1st-person plural pronouns per Google Developer Style Guide - Replace em dashes with standard punctuation - Remove unused pyyaml dependency from setup
…ry data, HITL governance, and schema drift RCA - Ingest authentic e-commerce dataset from bigquery-public-data.thelook_ecommerce into 3-tier pipeline - Implement human-in-the-loop (HITL) metadata review and approval promotion - Inject upstream schema drift into users_raw and detect impact across lineage graph - Implement recursive root-cause analysis (trace_root_cause) to isolate originating fault - Add lineage explainability auditing with BigQuery dry-run syntax verification - Align with Google Developer Style Guide and pass 3-layer test suite
…uganizer IDs - Set utm_campaign=b543855766 for metadata_as_code_dark_data.ipynb - Set utm_campaign=b543859711 for curated_data_products.ipynb - Set utm_campaign=b543859650 for lineage_graph_observability.ipynb
…ing layers on live GCP - Set default LOCATION to US for seamless bigquery-public-data multi-region CTAS - Robustify provision_dataset to ensure dataset location matches target region - Use pure pandas DataFrame row-dict conversion for preview queries - Pass Layer 1 Unit Tests (16/16 tests), Layer 2 Cleanup, and Layer 3 E2E Execution on live environment
…nt automatically - Add explicit Callout Note in Cell 3 explaining bigquery-public-data US multi-region requirement - Automatically normalize LOCATION to US in Cell 4 for public data pipeline execution - Pass all 3 testing layers top-to-bottom on live GCP
…bformat v4 schema compliance
… with Gemini 3.6 Flash - Upgrades reasoning model to gemini-3.6-flash and removes deprecated sampling arguments - Aligns markdown headings with Google Developer Style Guide sentence-case conventions - Enhances fail-fast parameter validation with descriptive remediation instructions - Implements BigQuery, OpenLineage, Dataplex aspect types, and cryptographic audit certificates
…prise_governance_observability
…& Gemini - Refactor architecture to official Google Cloud Knowledge Catalog Data Products and Aspect Types - Implement deterministic in-table timestamp freshness SLA scorecard audit - Add live Knowledge Catalog metadata discovery and Gemini grounded query generation - Integrate BigQuery dry-run cost estimation and syntax safety guardrails - Streamline to 27 procedural cells with unnumbered sentence-case headings and clean English comments - Align top Open in Colab badge with GoogleCloudPlatform/knowledge-catalog main branch
…ebook and preserve WIP version - cookbooks/enterprise_governance_observability.ipynb: Update to final 3-Gate certified Level 200 notebook (BigQuery staging, negative drift falsification, HITL state machine, ACL search, Gemini 3.7 Flash, and Lineage process discovery). - cookbooks/enterprise_governance_observability_wip.ipynb: Preserve intermediate WIP notebook version. - cookbooks/enterprise_governance_observability_preview.md: Add companion markdown preview.
…ds and rebrand - Remediate AST taint and eliminate silent fallback assignments in except blocks - Enforce Google Cloud 2026 rebrand invariants and remove banned acronyms - Standardize headings to sentence-case and add DevRel UTM attribution tags - Harden teardown resilience and add bounded retry for asynchronous indexing
…tructure and standard parameter contracts
…ucture, and cleanup safety - Harmonize parameter contracts across all 3 cookbooks: standardize on PROJECT_ID = "your-project-id", REGION = "us-central1", GEMINI_LOCATION = "global", and MODEL_NAME = "gemini-3.7-flash" - Enforce atomic cell granularity: separate dependency installation, parameter configuration, and SDK client initialization - Enforce safe standalone teardown with ENABLE_CLEANUP = False boolean guard to prevent accidental resource deletion during Run All - Implement negative validation gates across cookbooks to verify contract error interception - Eliminate legacy acronyms and directional language in compliance with Google Cloud documentation standards
…bservability Colab badge - Point Colab badge URL to official GoogleCloudPlatform/knowledge-catalog repository - Add standard utm_source=devrel, utm_medium=colab_badge, and campaign parameters
… in dark data cookbook
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This pull request adds cookbooks for Google Cloud Knowledge Catalog. These recipes demonstrate end-to-end patterns for enterprise data governance, curated data products, and multimodal dark data extraction.
Included Cookbooks
metadata_as_code_dark_data.ipynb• Strict Pydantic schema validation
• Programmatic Knowledge Catalog custom aspect registration and tagging
• Round-trip aspect verification
curated_data_products.ipynb• Machine-readable contract SLAs
• Golden queries aspect for AI grounding
• Grounded BigQuery SQL generation.
enterprise_governance_observability.ipynb• OpenLineage lifecycle events & schema facets
• Automated data quality diagnostic gate & poisoning prevention
• Custom aspect governance.