Hive Lineage RFC #10
Hive Lineage RFC #10
Conversation
WalkthroughA new document titled "Hive Lineage" has been added. It details the process of capturing and tracking data lineage within the Hive ecosystem. The document describes how lineage information is intercepted from Hive engine events, processed by a LineageLogger Hook, transmitted via Kafka, and ultimately used to generate Metadata Change Proposals for integration with the Graph Metadata Service. Changes
Sequence Diagram(s)sequenceDiagram
participant Hive as Hive Engine
participant Logger as LineageLogger Hook
participant Kafka as Kafka Broker
participant Consumer as Kafka Consumer
participant GMS as Graph Metadata Service
Hive->>Logger: Emit event / intercept query
Logger->>Kafka: Send lineage data
Kafka->>Consumer: Deliver message
Consumer->>GMS: Generate and send MCP
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (9)
active/000-hive-lineage/000-hive-lineage.md (9)
1-5: Metadata Headers – Ensure Completeness
The header section includes key metadata (Start Date, RFC PR, etc.). Please verify that the "Discussion Issue" and "Implementation PR(s)" fields are populated or that leaving them blank is intentional.
6-10: Summary Clarity and Readability
The summary provides a comprehensive overview of Hive Lineage. Consider breaking the long sentence on line 10 into shorter sentences to improve readability and better emphasize the key steps (listening to events, intercepting queries, and capturing lineage).
12-22: Consistent Terminology
There is an inconsistency in spelling: line 12 uses “organisations” while line 22 uses “organizations.” Please standardize the spelling throughout the document to avoid confusion.🧰 Tools
🪛 LanguageTool
[style] ~16-~16: The phrase “a variety of” may be wordy. To make your writing clearer, consider replacing it.
Context: ...chieve several key benefits and support a variety of use cases. ### Use Cases Supported: 1...(A_VARIETY_OF)
[uncategorized] ~22-~22: Do not mix variants of the same word (‘organization’ and ‘organisation’) within a single text.
Context: ...e:** With detailed lineage information, organizations can comply with regulatory requirements...(EN_WORD_COHERENCY)
🪛 markdownlint-cli2 (0.17.2)
18-18: Trailing punctuation in heading
Punctuation: ':'(MD026, no-trailing-punctuation)
18-18: Markdown Heading Punctuation
Several headings include trailing colons, which does not comply with our Markdown style guidelines. Please remove the trailing punctuation from the following headings:
- “### Use Cases Supported:” (line 18)
- “### Expected Outcome:” (line 30)
- “### Implementation Details:” (line 45)
- “### Usage Example:” (line 60)
- “### Architecture:” (line 432)
- “### Terminology:” (line 440)
Also applies to: 30-30, 45-45, 60-60, 432-432, 440-440
🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
18-18: Trailing punctuation in heading
Punctuation: ':'(MD026, no-trailing-punctuation)
47-49: Comma for Clarity in Implementation Details
On line 48, the sentence “This ensures that the hook is invoked to capture lineage information after the query execution is completed ensuring the correct lineage is captured and doesn't get affected if the query execution fails.” would benefit from a comma to clearly separate the clauses. For example:-... after the query execution is completed ensuring the correct lineage is captured... +... after the query execution is completed, ensuring the correct lineage is captured...🧰 Tools
🪛 LanguageTool
[uncategorized] ~48-~48: Possible missing comma found.
Context: ...nformation after the query execution is completed ensuring the correct lineage is capture...(AI_HYDRA_LEO_MISSING_COMMA)
432-435: Architecture Section – Heading Punctuation and Image Verification
Remove the trailing colon from the “### Architecture:” heading. Also, please verify that the referenced image file (hive-lineage-architecture.png) is available and correctly linked.🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
432-432: Trailing punctuation in heading
Punctuation: ':'(MD026, no-trailing-punctuation)
436-442: Terminology Heading – Remove Trailing Colon
Please remove the trailing colon from the “### Terminology:” heading to adhere to Markdown style guidelines.🧰 Tools
🪛 markdownlint-cli2 (0.17.2)
440-440: Trailing punctuation in heading
Punctuation: ':'(MD026, no-trailing-punctuation)
450-454: Drawbacks Section – Rephrase for Clarity
The drawbacks section currently reads:"No effect on existing functionality/customers. Apart from the need for a separate topic and consumer, it does not need any more changes to existing systems. The current design will be leveraging existing lineage API and MCP structure to process lineage information."
Consider rephrasing for clarity. For example:
-No effect on existing functionality/customers. Apart from the need for a separate topic and consumer, it does not need any more changes to existing systems. +There is no impact on existing functionality or customers. The only new requirement is the introduction of a separate Kafka topic and consumer—no further modifications to existing systems are needed.🧰 Tools
🪛 LanguageTool
[style] ~452-~452: Consider an alternative to strengthen your wording.
Context: ...opic and consumer, it does not need any more changes to existing systems. The current design...(CHANGES_ADJUSTMENTS)
454-457: Rollout / Adoption Strategy – Minor Refinements
The rollout strategy is clearly described. You might consider a slight rephrasing for enhanced clarity and consistency; however, the current version is acceptable if it meets your communication goals.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
active/000-hive-lineage/hive-lineage-architecture.pngis excluded by!**/*.png
📒 Files selected for processing (1)
active/000-hive-lineage/000-hive-lineage.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
active/000-hive-lineage/000-hive-lineage.md
[style] ~16-~16: The phrase “a variety of” may be wordy. To make your writing clearer, consider replacing it.
Context: ...chieve several key benefits and support a variety of use cases. ### Use Cases Supported: 1...
(A_VARIETY_OF)
[uncategorized] ~22-~22: Do not mix variants of the same word (‘organization’ and ‘organisation’) within a single text.
Context: ...e:** With detailed lineage information, organizations can comply with regulatory requirements...
(EN_WORD_COHERENCY)
[uncategorized] ~48-~48: Possible missing comma found.
Context: ...nformation after the query execution is completed ensuring the correct lineage is capture...
(AI_HYDRA_LEO_MISSING_COMMA)
[style] ~452-~452: Consider an alternative to strengthen your wording.
Context: ...opic and consumer, it does not need any more changes to existing systems. The current design...
(CHANGES_ADJUSTMENTS)
🪛 markdownlint-cli2 (0.17.2)
active/000-hive-lineage/000-hive-lineage.md
18-18: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
30-30: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
45-45: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
60-60: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
432-432: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
440-440: Trailing punctuation in heading
Punctuation: ':'
(MD026, no-trailing-punctuation)
🔇 Additional comments (3)
active/000-hive-lineage/000-hive-lineage.md (3)
66-426: JSON Example – Well Structured
The JSON example is comprehensive and well-structured. It clearly illustrates the expected data format for lineage information.
428-431: Processing Lineage Information – Clear Explanation
The description of how the Kafka consumer processes JSON messages to generate Metadata Change Proposals (MCPs) is clear and informative.
458-460: Future Work Section – Clear and Forward Looking
The Future Work section succinctly outlines potential enhancements leveraging the captured Hive Lineage. No changes required.
|
|
||
| ## Detailed Design | ||
|
|
||
| The design for Hive Lineage involves leveraging Hive's LineageLogger Hook to capture lineage information, processing this information, and sending it to a Kafka topic. A Kafka consumer then processes these messages and sends metadata change proposals (MCPs) to the GMS (Graph Metadata Service). This design ensures efficient tracking and documentation of data flow within the Hive ecosystem. |
There was a problem hiding this comment.
It will support rest emitter as well, right?
There was a problem hiding this comment.
This architecture makes sense to ensure a robust system and the additional latency should be minimal to the Hive execution with the listener.
On the other hand, I wouldn't drop the option to generate mcp and send it to DataHub using rest emitter right away to ensure Kafka is an optional dependency.
Does LineageLogger block the query execution?
There was a problem hiding this comment.
As of now we are thinking to separate the rest emitter from the hook jar, the rest emitter will be a part of the consumer which actually consumes the lineage messages from the kafka.
One of the reasons for bringing in kafka is to address scalability and consistency. Considering hive loads that run in an order of 100k+ queries per day, we feel having a message queue (kafka) will help to address all sorts of typical producer-consumer issues.
Regarding the LineageLogger Hook, it is designed as a post-execution hook. This means it does not block the query execution. The hook is invoked after the query execution is completed and before the result is published to the user. To mitigate any latency in the user receiving the results due to lineage computation, we plan to spawn a worker thread dedicated to computing the lineage separately from the main thread executing the Hive query. This ensures that the user experience remains unaffected and the lineage is processed efficiently in the background.
Summary by CodeRabbit