diff --git a/hugo/config/_default/menus/main.en.yaml b/hugo/config/_default/menus/main.en.yaml index 4f7486a03f5..01b99f3675f 100644 --- a/hugo/config/_default/menus/main.en.yaml +++ b/hugo/config/_default/menus/main.en.yaml @@ -5764,6 +5764,11 @@ menu: parent: llm_obs_experiments identifier: llm_obs_experiments_api weight: 50305 + - name: Restrict Access to Projects + url: llm_observability/improve/experiments/access_control + parent: llm_obs_experiments + identifier: llm_obs_experiments_access_control + weight: 50306 - name: Configure url: llm_observability/configure/ parent: llm_obs diff --git a/hugo/content/en/account_management/rbac/data_access.md b/hugo/content/en/account_management/rbac/data_access.md index f2102448e4e..1287d89e7eb 100644 --- a/hugo/content/en/account_management/rbac/data_access.md +++ b/hugo/content/en/account_management/rbac/data_access.md @@ -142,7 +142,7 @@ If you are concerned about unauthorized data access through monitors, Datadog re ### Agent Observability -* **Supported telemetry**: Agent Observability traces are supported. Experiment event data (spans and evaluation metrics) for experiments in a project is also restricted by `ml_app`-keyed Restricted Datasets. Only the event data is restricted—experiment list views and metadata are not. Datasets, annotation queues, and managed prompts are not supported. +* **Supported telemetry**: Agent Observability traces are supported. Experiments are also supported: a Restricted Dataset keyed on `ml_app` whose value is an Experiments project ID restricts that project's spans, evaluation metrics, datasets, and dataset records, and hides the project and its experiments from list views, search, and direct links. For instructions, see [Restrict Access to Experiments Projects][14]. Annotation queues and managed prompts are not supported. * **OpenTelemetry**: When using [OpenTelemetry instrumentation][13], some data sent to Agent Observability may also be written to APM traces, as well as metrics and monitors. If you are protecting sensitive data with a Restricted Dataset on Agent Observability, consider also configuring Restricted Datasets on APM, metrics, or monitors with matching data boundaries. @@ -270,3 +270,4 @@ When querying data through Datadog APIs with restrictions enabled, users without [11]: /logs/guide/logs-rbac/?tab=ui#restrict-access-to-logs [12]: /dashboards/sharing/shared_dashboards/ [13]: /llm_observability/instrument/otel_instrumentation/ +[14]: /llm_observability/improve/experiments/access_control/ diff --git a/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md b/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md index b7062d174c4..82a38d9089f 100644 --- a/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md +++ b/hugo/content/en/llm_observability/data_privacy_security_and_rbac.md @@ -18,6 +18,8 @@ Agent Observability allows you to restrict access to potentially sensitive data Access controls in Agent Observability are built on Datadog's [Data Access Control][1] feature, which enables you to regulate access to data deemed sensitive. You can use the `ml_app` tag to identify and restrict access to specific LLM applications within your organization. +The same mechanism restricts individual Experiments projects, including their datasets and dataset records. See [Restrict Access to Experiments Projects][4]. + ## Redacting data with span processors You can redact or modify sensitive data at the application level before it is sent to Datadog. Use span processors in the Agent Observability SDK to conditionally modify input and output data on spans, or prevent spans from being emitted entirely. @@ -42,4 +44,4 @@ By proactively scanning for sensitive data, Agent Observability ensures that con [1]: /account_management/rbac/data_access [2]: /llm_observability/instrument/sdk/#span-processing [3]: /security/sensitive_data_scanner/ - +[4]: /llm_observability/improve/experiments/access_control/ diff --git a/hugo/content/en/llm_observability/improve/experiments/access_control.md b/hugo/content/en/llm_observability/improve/experiments/access_control.md new file mode 100644 index 00000000000..568dcf60bd0 --- /dev/null +++ b/hugo/content/en/llm_observability/improve/experiments/access_control.md @@ -0,0 +1,111 @@ +--- +title: Restrict Access to Experiments Projects +description: Use Data Access Control to restrict an Agent Observability Experiments project to specific teams or roles. +further_reading: +- link: "/account_management/rbac/data_access" + tag: "Documentation" + text: "Data Access Control" +- link: "/llm_observability/data_privacy_security_and_rbac" + tag: "Documentation" + text: "Data Privacy, Security, and RBAC" +- link: "/account_management/rbac/permissions/#access-management" + tag: "Documentation" + text: "Access management permissions" +--- + +## Overview + +Experiments projects can contain sensitive material: the prompts and expected outputs stored in your datasets, the traces produced by an experiment run, and the evaluation results attached to them. Datadog's [Data Access Control][1] lets you restrict an individual project so that only the teams or roles you name can see it. + +When a project is restricted, users outside the granted principals: + +- Do not see the project, or its experiments and datasets, in any list view or search result. +- Receive a *not found* response when they open a direct link to the project or to anything inside it. +- Cannot read the project's dataset records, including inputs and expected outputs. +- Cannot read the evaluation metrics produced by the project's experiment runs. +- Cannot read the spans produced by those runs, unless the spans carry an `ml_app` value of their own. Experiments run through the SDK do produce such spans; see [Limitations](#limitations). +- Cannot create, modify, or delete anything inside the project, even with an ID they obtained earlier. + +Restrictions are enforced consistently in the Datadog UI and in the API, including requests authenticated with an application key: an application key carries the identity of the user who owns it, and that user's access is what applies. + +## Prerequisites + +- Data Access Control is configured for your organization. See [Data Access Control][1]. +- You have the Datadog Admin role, or another role carrying the [`user_access_manage` permission][2]. +- The project you want to restrict already exists in Experiments. + +## Restrict a project + +Access to Agent Observability data is keyed on the **`ml_app`** tag, and that one key covers two different things: + +- An **ML app** value restricts the traces your instrumented application sends under that `ml_app` name. +- An **Experiments project ID** restricts that project: its experiments, datasets, dataset records, and the spans and evaluation metrics produced by its experiment runs. + +A project is matched by its ID, never by its name. + +1. Navigate to [Organization Settings > Data Access Controls][3]. +2. Create a new policy that restricts a subset of data. +3. Name the policy something that identifies the project it protects, for example `Experiments - Fraud Detection`. +4. Add a filter on the **Agent Observability** product with the key `ml_app`, then **select the project from the list of values**. The list contains two groups: your ML apps (the `ml_app` values your instrumented applications send), and your Experiments projects, offered by name. Selecting a project stores its ID, which is what the restriction matches on. +5. Grant access to the teams or roles that should keep access to the project. A maximum of 50 principals can be attached to one policy. +6. Save the policy. + +The policy takes effect as soon as it is saved. The project, its experiments, its datasets, and its dataset records are hidden immediately, whatever their age. Spans and evaluation metrics are matched by a tag applied when they are ingested, so experiment runs from before your organization had this feature available are not covered; see [Limitations](#limitations). + +
Select the project from the list rather than typing its name. The value field also accepts free text, and a policy holding a project's name matches no Experiments data: the project stays visible to everyone while the policy looks like it is working. The same applies to a partially typed or misspelled name.
+ +Two things to expect while filling in the filter: + +- **Projects already covered by another policy do not appear in the list.** A project can belong to only one policy at a time. +- **The key may be locked to `ml_app`.** Data Access Control allows one tag key per telemetry type, so if your organization already has an Agent Observability policy, new ones reuse the same key. + +
Datadog is rolling out a redesigned access control page. Depending on your organization, step 2 is either New Restricted Dataset on the Data Access Controls page, or New Policy > Sensitive Data Partition on the Access Control page. Both configure the same restriction, and the link in step 1 takes you to whichever page your organization has. Selecting an Experiments project by name is only available on the redesigned page; on the older page, enter the project ID as the value.
+ +### Find a project's ID + +If your organization's access control page shows raw values rather than project names, take the project ID from the URL of the project in Experiments, or from the `id` field returned by the [Experiments API][4] when listing projects. + +## Restrict a project through the API + +You can also create the policy with the [Datasets API][5]. The `ml_obs` product filter takes the project ID as its `ml_app` value: + +```json +{ + "data": { + "type": "dataset", + "attributes": { + "name": "Experiments - Fraud Detection", + "product_filters": [ + { + "product": "ml_obs", + "filters": ["ml_app:3547f4ac-3af4-4733-9a70-8fe596e1e76d"] + } + ], + "principals": ["team:f771276e-0847-4c24-a277-6744f8520bb4"] + } + } +} +``` + +## Grant and revoke access + +Access is granted by editing the policy's principals. Removing a team or role takes effect on their next request. Deleting the policy removes the restriction entirely, and the project becomes visible again to everyone in the organization with Agent Observability read access. + +Access follows grant membership, not permission level. The `user_access_manage` permission lets you author and edit a policy; it does not exempt you from one. An admin who is not in a granted team or role sees the project as not found, exactly as any other user would. + +## Limitations + +- **Spans from an experiment run by the SDK carry your application's `ml_app`, not the project.** The SDK sends an `ml_app` value with every span it produces, taken from the value you configured or from your service name, and Datadog does not replace it. A policy on the project does not match those spans, so for an experiment run this way the restriction covers its evaluation metrics but not the inputs and outputs recorded on its spans. The project, its experiments, its datasets, and its dataset records are hidden either way. To restrict the span contents as well, add a second filter for your application's own `ml_app` value to the same policy. +- **Spans and evaluation metrics ingested before this feature became available are not restricted.** The project ID is attached as a tag at ingestion time and past events are not re-tagged, so a policy on a project does not hide the event data of experiment runs that predate it. List views, metadata, and dataset records are unaffected by this and are hidden regardless of age. +- **Annotation queues and managed prompts are not supported** by Data Access Control. See [Data Access Control][1] for the full list of supported telemetry. +- **A project with no restriction policy is visible to everyone** with Agent Observability read access. Data Access Control is permissive by default unless your organization has enabled Strict Mode for the telemetry type. A policy whose value matches no project silently restricts nothing, so confirm every new restriction with a user outside the granted principals. + +## Further reading + +{{< partial name="whats-next/whats-next.html" >}} + +[1]: /account_management/rbac/data_access/ +[2]: /account_management/rbac/permissions/#access-management +[3]: https://app.datadoghq.com/organization-settings/data-access-controls +[4]: /llm_observability/improve/experiments/api/ +[5]: /api/latest/datasets/