feat: llma / error tracking integration #376
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Resolved conflicts: - CHANGELOG.md: Added 7.5.0 entry for error tracking integration - posthog/ai/utils.py: Kept refactored tag-based approach with sanitize_messages - posthog/version.py: Bumped to 7.5.0 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add 6 tests covering the new LLMA + error tracking integration: - capture_exception called on span/generation errors - $exception_event_id added to AI events - No capture when autocapture disabled - AI properties passed to exception event - Handles None return from capture_exception 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Export get_tags() from posthog module - Explicitly pass context tags to capture() in AI utils - Fix $ai_model fallback to extract from response.model - Fix ruff formatting in langchain test_callbacks.py 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
The new_context() defaults to capture_exceptions=True which would auto-capture any exception regardless of enable_exception_autocapture setting. This was inconsistent with LangChain callbacks which explicitly check the setting. Pass capture_exceptions=False to let exception handling be controlled explicitly by the enable_exception_autocapture setting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Use fresh=True to start with a clean context for each LLM call. This avoids inheriting $ai_* tags from parent contexts which could cause mismatched AI metadata due to the tag merge order bug in contexts.py (parent tags incorrectly override child tags). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
The collect_tags() method had a bug where parent tags would overwrite child tags, despite the comment saying the opposite. This fix ensures child context tags properly override parent tags. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
andrewm4894
requested review from
a team,
ablaszkiewicz,
hpouillot and
oliverb123
With the collect_tags() bug fixed, child tags properly override parent tags. LLM events can now inherit useful parent context tags (request_id, user info, etc.) while still having their $ai_* tags take precedence. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Verifies that in non-fresh contexts, child tags properly override parent tags with the same key while still inheriting other parent tags. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
ablaszkiewicz
approved these changes
Jan 7, 2026
Member
Summary of current stateThis PR adds error tracking integration for LLM observability, primarily for LangChain. What it doesLangChain integration (complete)
OpenAI/Anthropic/Gemini wrappers (partial)
Additional fixes included
Test coverage
|
Document that exception capture needs to be added for the direct SDK wrappers, similar to how it's implemented in LangChain callbacks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to PostHog/posthog-js#2593
exception_idto LLMA traces / spans so we can better integrate the two in the app