From 7659574754bd04a9c9c4bd882a835f24a0e2cc31 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Thu, 3 Sep 2026 11:21:17 -0400 Subject: [PATCH 1/4] Add Experiments project access control docs Documents restricting an Agent Observability Experiments project with Data Access Control: what a restriction hides, how to author the policy on either the current or the redesigned access control page, the API equivalent, and the limitations (SDK-run spans keyed on the app's own ml_app, and event data ingested before the feature was available). Also corrects the Agent Observability entry on the Data Access Control page, which said only event data is restricted and that datasets are unsupported. Co-Authored-By: Claude Opus 5 (1M context) --- hugo/config/_default/menus/main.en.yaml | 5 + .../en/account_management/rbac/data_access.md | 3 +- .../data_privacy_security_and_rbac.md | 4 +- .../improve/experiments/access_control.md | 110 ++++++++++++++++++ 4 files changed, 120 insertions(+), 2 deletions(-) create mode 100644 hugo/content/en/llm_observability/improve/experiments/access_control.md diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index 4f7486a03f5..01b99f3675f 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -5764,6 +5764,11 @@ menu: parent: llm_obs_experiments identifier: llm_obs_experiments_api weight: 50305 + - name: Restrict Access to Projects + url: llm_observability/improve/experiments/access_control + parent: llm_obs_experiments + identifier: llm_obs_experiments_access_control + weight: 50306 - name: Configure url: llm_observability/configure/ parent: llm_obs diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index f2102448e4e..1287d89e7eb 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -142,7 +142,7 @@ If you are concerned about unauthorized data access through monitors, Datadog re ### Agent Observability -* **Supported telemetry**: Agent Observability traces are supported. Experiment event data (spans and evaluation metrics) for experiments in a project is also restricted by `ml_app`-keyed Restricted Datasets. Only the event data is restricted—experiment list views and metadata are not. Datasets, annotation queues, and managed prompts are not supported. +* **Supported telemetry**: Agent Observability traces are supported. Experiments are also supported: a Restricted Dataset keyed on `ml_app` whose value is an Experiments project ID restricts that project's spans, evaluation metrics, datasets, and dataset records, and hides the project and its experiments from list views, search, and direct links. For instructions, see [Restrict Access to Experiments Projects][14]. Annotation queues and managed prompts are not supported. * **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries. @@ -270,3 +270,4 @@ When querying data through Datadog APIs with restrictions enabled, users without [11]: /logs/guide/logs-rbac/?tab=ui#restrict-access-to-logs [12]: /dashboards/sharing/shared_dashboards/ [13]: /llm_observability/instrument/otel_instrumentation/ +[14]: /llm_observability/improve/experiments/access_control/ diff --git a/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md b/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md index b7062d174c4..82a38d9089f 100644 --- a/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md +++ b/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md @@ -18,6 +18,8 @@ Agent Observability allows you to restrict access to potentially sensitive data Access controls in Agent Observability are built on Datadog's [Data Access Control][1] feature, which enables you to regulate access to data deemed sensitive. You can use the `ml_app` tag to identify and restrict access to specific LLM applications within your organization. +The same mechanism restricts individual Experiments projects, including their datasets and dataset records. See [Restrict Access to Experiments Projects][4]. + ## Redacting data with span processors You can redact or modify sensitive data at the application level before it is sent to Datadog. Use span processors in the Agent Observability SDK to conditionally modify input and output data on spans, or prevent spans from being emitted entirely. @@ -42,4 +44,4 @@ By proactively scanning for sensitive data, Agent Observability ensures that con [1]: /account_management/rbac/data_access [2]: /llm_observability/instrument/sdk/#span-processing [3]: /security/sensitive_data_scanner/ - +[4]: /llm_observability/improve/experiments/access_control/ diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md new file mode 100644 index 00000000000..b362ffd46fe --- /dev/null +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -0,0 +1,110 @@ +--- +title: Restrict Access to Experiments Projects +description: Use Data Access Control to restrict an Agent Observability Experiments project to specific teams or roles. +further_reading: +- link: "/account_management/rbac/data_access" + tag: "Documentation" + text: "Data Access Control" +- link: "/llm_observability/data_privacy_security_and_rbac" + tag: "Documentation" + text: "Data Privacy, Security, and RBAC" +- link: "/account_management/rbac/permissions/#access-management" + tag: "Documentation" + text: "Access management permissions" +--- + +## Overview + +Experiments projects can contain sensitive material: the prompts and expected outputs stored in your datasets, the traces produced by an experiment run, and the evaluation results attached to them. Datadog's [Data Access Control][1] lets you restrict an individual project so that only the teams or roles you name can see it. + +When a project is restricted, users outside the granted principals: + +- Do not see the project, or its experiments and datasets, in any list view or search result. +- Receive a *not found* response when they open a direct link to the project or to anything inside it. +- Cannot read the spans and evaluation metrics produced by the project's experiment runs. +- Cannot read the project's dataset records, including inputs and expected outputs. +- Cannot create, modify, or delete anything inside the project, even with an ID they obtained earlier. + +Restrictions are enforced consistently in the Datadog UI and in the API, including requests authenticated with an application key: an application key carries the identity of the user who owns it, and that user's access is what applies. + +## Prerequisites + +- Data Access Control is configured for your organization. See [Data Access Control][1]. +- You have the Datadog Admin role, or another role carrying the [`user_access_manage` permission][2]. +- The project you want to restrict already exists in Experiments. + +## Restrict a project + +Access to Agent Observability data is keyed on the **`ml_app`** tag, and that one key covers two different things: + +- An **ML app** value restricts the traces your instrumented application sends under that `ml_app` name. +- An **Experiments project ID** restricts that project: its experiments, datasets, dataset records, and the spans and evaluation metrics produced by its experiment runs. + +A project is matched by its ID, never by its name. + +1. Navigate to [Organization Settings > Data Access Controls][3]. +2. Create a new policy that restricts a subset of data. +3. Name the policy something that identifies the project it protects, for example `Experiments - Fraud Detection`. +4. Add a filter on the **Agent Observability** product with the key `ml_app`, then **select the project from the list of values**. The list contains two groups: your ML apps (the `ml_app` values your instrumented applications send), and your Experiments projects, offered by name. Selecting a project stores its ID, which is what the restriction matches on. +5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 principals can be attached to one policy. +6. Save the policy. + +The policy takes effect as soon as it is saved. The project, its experiments, its datasets, and its dataset records are hidden immediately, whatever their age. Spans and evaluation metrics are matched by a tag applied when they are ingested, so experiment runs from before your organization had this feature available are not covered; see [Limitations](#limitations). + +
Select the project from the list rather than typing its name. The value field also accepts free text, and a policy holding a project's name matches no Experiments data: the project stays visible to everyone while the policy looks like it is working. The same applies to a partially typed or misspelled name.
+ +Two things to expect while filling in the filter: + +- **Projects already covered by another policy do not appear in the list.** A project can belong to only one policy at a time. +- **The key may be locked to `ml_app`.** Data Access Control allows one tag key per telemetry type, so if your organization already has an Agent Observability policy, new ones reuse the same key. + +
Datadog is rolling out a redesigned access control page. Depending on your organization, step 2 is either New Restricted Dataset on the Data Access Controls page, or New Policy > Sensitive Data Partition on the Access Control page. Both configure the same restriction, and the link in step 1 takes you to whichever page your organization has. Selecting an Experiments project by name is only available on the redesigned page; on the older page, enter the project ID as the value.
+ +### Find a project's ID + +If your organization's access control page shows raw values rather than project names, take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. + +## Restrict a project through the API + +You can also create the policy with the [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: + +```json +{ + "data": { + "type": "dataset", + "attributes": { + "name": "Experiments - Fraud Detection", + "product_filters": [ + { + "product": "ml_obs", + "filters": ["ml_app:3547f4ac-3af4-4733-9a70-8fe596e1e76d"] + } + ], + "principals": ["team:f771276e-0847-4c24-a277-6744f8520bb4"] + } + } +} +``` + +## Grant and revoke access + +Access is granted by editing the policy's principals. Removing a team or role takes effect on their next request. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. + +Users with the `user_access_manage` permission are not subject to restrictions, so verify a restriction with an account that does not hold that permission. + +## Limitations + +- **Spans from an experiment run by the SDK are not restricted by project.** Datadog only fills in the project ID as the `ml_app` value when your code has not set one, and never overrides a value you set. Running an experiment through the SDK stamps your application's `ml_app` on every task span it produces, so those spans are matched by a policy on your application's `ml_app` value, not by one on the project. The project, its experiments, its datasets, and its dataset records are still hidden from the Experiments UI either way. To restrict the span contents as well, add a filter for your application's own `ml_app` value alongside the project. +- **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. +- **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. +- **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled Strict Mode for the telemetry type. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted principals. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /account_management/rbac/data_access/ +[2]: /account_management/rbac/permissions/#access-management +[3]: https://app.datadoghq.com/organization-settings/data-access-controls +[4]: /llm_observability/improve/experiments/api/ +[5]: /api/latest/datasets/ From d6c3875b541e2f2243ee59b0aa0e4a78cfdbd272 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Fri, 4 Sep 2026 13:36:11 -0400 Subject: [PATCH 2/4] Clarify the SDK span limitation The previous wording explained the backfill mechanism (project ID is used as ml_app only when none is set), which reads as a lever the reader can pull. It is not: the SDK always sends an ml_app value, so the backfill never applies to SDK traffic. State the outcome and the remedy instead. Co-Authored-By: Claude Opus 5 (1M context) --- .../en/llm_observability/improve/experiments/access_control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index b362ffd46fe..6e2c9b73479 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -94,7 +94,7 @@ Users with the `user_access_manage` permission are not subject to restrictions, ## Limitations -- **Spans from an experiment run by the SDK are not restricted by project.** Datadog only fills in the project ID as the `ml_app` value when your code has not set one, and never overrides a value you set. Running an experiment through the SDK stamps your application's `ml_app` on every task span it produces, so those spans are matched by a policy on your application's `ml_app` value, not by one on the project. The project, its experiments, its datasets, and its dataset records are still hidden from the Experiments UI either way. To restrict the span contents as well, add a filter for your application's own `ml_app` value alongside the project. +- **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. - **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. - **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled Strict Mode for the telemetry type. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted principals. From a3167934db95127612d8aa7e58786bf8cf6b5a39 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Fri, 4 Sep 2026 13:42:29 -0400 Subject: [PATCH 3/4] Narrow the span coverage claim in the overview The overview promised that a restricted user cannot read the spans or evaluation metrics of a restricted project's runs, and the limitations section then took half of that back. For an SDK-run experiment the two differ: eval metrics carry no ml_app and are backfilled to the project id, while task spans carry the application's ml_app and are not matched. Say so once, up front. Co-Authored-By: Claude Opus 5 (1M context) --- .../llm_observability/improve/experiments/access_control.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index 6e2c9b73479..e6f76e18596 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -21,8 +21,9 @@ When a project is restricted, users outside the granted principals: - Do not see the project, or its experiments and datasets, in any list view or search result. - Receive a *not found* response when they open a direct link to the project or to anything inside it. -- Cannot read the spans and evaluation metrics produced by the project's experiment runs. - Cannot read the project's dataset records, including inputs and expected outputs. +- Cannot read the evaluation metrics produced by the project's experiment runs. +- Cannot read the spans produced by those runs, unless the spans carry an `ml_app` value of their own. Experiments run through the SDK do produce such spans; see [Limitations](#limitations). - Cannot create, modify, or delete anything inside the project, even with an ID they obtained earlier. Restrictions are enforced consistently in the Datadog UI and in the API, including requests authenticated with an application key: an application key carries the identity of the user who owns it, and that user's access is what applies. @@ -94,7 +95,7 @@ Users with the `user_access_manage` permission are not subject to restrictions, ## Limitations -- **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. +- **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans, so for an experiment run this way the restriction covers its evaluation metrics but not the inputs and outputs recorded on its spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. - **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. - **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. - **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled Strict Mode for the telemetry type. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted principals. From 80c1227abf13a65eb845e7b4c870b3c9eb92af90 Mon Sep 17 00:00:00 2001 From: Imran Hendley Date: Fri, 4 Sep 2026 14:24:35 -0400 Subject: [PATCH 4/4] Correct the user_access_manage claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The page said holders of user_access_manage are not subject to restrictions. They are: the permission gates authoring a policy, while the access decision is taken over the user's teams and roles with no permissions input. Verified on staging as a non-granted admin — the project is hidden and its dataset records 404. Co-Authored-By: Claude Opus 5 (1M context) --- .../en/llm_observability/improve/experiments/access_control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md index e6f76e18596..568dcf60bd0 100644 --- a/hugo/content/en/llm_observability/improve/experiments/access_control.md +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -91,7 +91,7 @@ You can also create the policy with the [Datasets API][5]. The `ml_obs` product Access is granted by editing the policy's principals. Removing a team or role takes effect on their next request. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. -Users with the `user_access_manage` permission are not subject to restrictions, so verify a restriction with an account that does not hold that permission. +Access follows grant membership, not permission level. The `user_access_manage` permission lets you author and edit a policy; it does not exempt you from one. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would. ## Limitations