Add Experiments project access control docs - #39707
Draft
imran-at-datadog wants to merge 4 commits into
Draft
Conversation
Documents restricting an Agent Observability Experiments project with Data Access Control: what a restriction hides, how to author the policy on either the current or the redesigned access control page, the API equivalent, and the limitations (SDK-run spans keyed on the app's own ml_app, and event data ingested before the feature was available). Also corrects the Agent Observability entry on the Data Access Control page, which said only event data is restricted and that datasets are unsupported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Preview links (active after the
|
The previous wording explained the backfill mechanism (project ID is used as ml_app only when none is set), which reads as a lever the reader can pull. It is not: the SDK always sends an ml_app value, so the backfill never applies to SDK traffic. State the outcome and the remedy instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The overview promised that a restricted user cannot read the spans or evaluation metrics of a restricted project's runs, and the limitations section then took half of that back. For an SDK-run experiment the two differ: eval metrics carry no ml_app and are backfilled to the project id, while task spans carry the application's ml_app and are not matched. Say so once, up front. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page said holders of user_access_manage are not subject to restrictions. They are: the permission gates authoring a policy, while the access decision is taken over the user's teams and roles with no permissions input. Verified on staging as a non-granted admin — the project is hidden and its dataset records 404. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do? What is the motivation?
Documents Data Access Control for Agent Observability Experiments projects, which lets an admin restrict a single project so only chosen teams or roles can see it.
Adds
llm_observability/improve/experiments/access_controlcovering what a restriction hides (list views, direct links, spans, evaluation metrics, dataset records, and writes), how to author the policy, the API equivalent, and the limitations.Also corrects the Agent Observability entry on the Data Access Control page. It currently reads:
Both statements describe an earlier state of the feature. List views, metadata, and Experiments datasets are now restricted; annotation queues and managed prompts remain unsupported.
Files:
llm_observability/improve/experiments/access_control.mdaccount_management/rbac/data_access.mdllm_observability/data_privacy_security_and_rbac.mdconfig/_default/menus/main.en.yamlMerge readiness
Not ready — please review but do not merge yet. The feature is behind a feature flag that is currently off in production. The correction to
data_access.mddescribes behavior that is not live yet, so merging before the rollout would make that page inaccurate in the other direction. I will comment here when the flag is on and check the box.For Datadog employees:
Two review notes:
The page deliberately describes one workflow for two UIs. Datadog is rolling out a redesigned access control page (
unified_access_control_page), currently enabled for a small number of organizations, with a wider rollout planned. Rather than fork the instructions, the page gives one set of steps plus a callout naming what differs, and links to/organization-settings/data-access-controls— which redirects to the new page for organizations that have it, so one link is correct for both audiences and needs no change after the rollout completes. Flagging in case Docs would rather see tabs or asite-region-style split.The SDK limitation is newly stated in customer-facing terms. Running an experiment through the SDK stamps the application's own
ml_appon its task spans, so a policy keyed on the project does not match those spans (the project and its metadata are still hidden). This is a known and accepted gap, but it has not been documented before, so it is worth a second opinion on the wording.AI assistance
Drafted with Claude Code from the feature's design docs and from the QA notes on the implementation PRs, then reviewed and corrected by hand. AI-assisted research into which app URLs and UI states each customer cohort sees.
Additional notes
Preview links should show the new page under Agent Observability > Use Agent Observability to Improve Applications > Experiments > Restrict Access to Projects.