Skip to content

azure.md

Latest commit

 

History

History
511 lines (348 loc) · 30.7 KB

azure.md

File metadata and controls

511 lines (348 loc) · 30.7 KB
Raw
title Getting Started with Azure
aliases
/integrations/guide/azure-manual-setup/
/integrations/guide/azure-programmatic-management/
description Connect Microsoft Azure with Datadog using Azure app registration integration options. Configure metric collection, log forwarding, and Agent installation.
further_reading
link tag text
Documentation
Microsoft Azure integration
link tag text
Guide
Why should I install the Datadog Agent on my cloud instances?
link tag text
Blog
Accelerate your Azure integration setup with guided onboarding
link tag text
Blog
Avoid Azure secret rotation with secretless authentication

Overview

Datadog offers multiple configuration options for the Azure integration. This guide provides an overview of the various options available for getting started with Azure, with links to Azure resources and tutorials that address specific use cases.

Prerequisites

If you haven't already, create a Datadog account.

{{% collapse-content title="Permissions required for integration setup" level="h3" expanded=false id="required-permissions" %}}

In Azure

Your Microsoft Entra ID user needs the following permissions:

Permission to create an app registration

One of the following must be true for the user:

Admin roles within your subscriptions

Within the subscriptions you wish to monitor, you must have either:

  • The {{< ui >}}Owner{{< /ui >}} role
  • Both the {{< ui >}}Contributor{{< /ui >}} and {{< ui >}}User Access Admin{{< /ui >}} roles

Permission to add and grant consent for Graph API permissions

The Privileged Role Administrator role contains the required permissions.

In Datadog

The Datadog Admin Role, or any other role with the azure_configurations_manage permission.

{{% /collapse-content %}}

{{< site-region region="us3" >}}

Cloud Cost Management and Log Archives require the app registration setup method. For Datadog accounts using the Azure Native integration, follow the setup steps on this page to create an app registration. If a subscription is connected through both methods, a redundancy warning appears in the Azure integration tile. This warning can be safely ignored for Cloud Cost Management and Log Archives.

{{< /site-region >}}

Setup

Follow the instructions on this page to set up the {{< ui >}}Azure integration{{< /ui >}} through an app registration, available for all Datadog sites. During setup, choose whether the app registration authenticates with Datadog using Secretless Auth (recommended) or a Client Secret.

{{< img src="/getting_started/integrations/azure/GSwAzure_siteSelector.mp4" alt="Site selector for US3 site" video=true >}}

Authentication methods

{{< site-region region="gov,gov2" >}}

Secretless Auth is not supported on the US1-FED and US2-FED sites. Use the Client Secret method to authenticate.
{{< /site-region >}}

The Azure integration supports two authentication methods:

Method How it works What you manage
Secretless Auth (recommended) Datadog and Azure establish a trusted identity relationship using an Azure federated credential and OpenID Connect (OIDC) workload identity federation. The app registration trusts Datadog's OIDC identity instead of a stored secret, and Datadog never stores a secret. Nothing to rotate or renew, which reduces the risk of ingestion gaps from expired credentials.
Client Secret Datadog stores and authenticates with a client secret from your app registration. Keep the secret up to date before it expires.

You select the authentication method when you create or configure an app registration.

{{< img src="/getting_started/integrations/azure/GSwAzure_authenticationMethod.png" alt="The Authentication Method step in the Azure integration setup, showing the recommended Secretless Auth and Client Secret options" style="width:100%;" >}}

{{% collapse-content title="Quickstart (recommended)" level="h3" expanded=false id="quickstart-setup" %}}

Choose the Quickstart setup method if...

  • You are setting up Datadog for the first time.
  • You prefer a UI-based workflow and want to minimize the time it takes to create a service principal with the required monitoring permissions.
  • You want to automate setup steps in scripts or CI/CD pipelines.

Instructions

  1. In the Azure integration tile, click {{< ui >}}+ Add New App registration{{< /ui >}}, then select {{< ui >}}Quickstart{{< /ui >}}.
  2. Copy the setup script, and run it in the Azure Cloud shell.
  3. Return to the Datadog UI. You should see {{< ui >}}CONNECTED{{< /ui >}} at the top right corner of the setup script.
  4. Select the subscriptions and management groups to collect data from.
  5. Under {{< ui >}}Authentication Method{{< /ui >}}, select {{< ui >}}Secretless Auth{{< /ui >}} (recommended) to authenticate with a federated credential, or {{< ui >}}Client Secret{{< /ui >}} to store and authenticate with a client secret.
  6. Optionally, click the metric collection toggle to disable all metric collection from Azure. You can also expand the {{< ui >}}Advanced Configuration{{< /ui >}} dropdown to filter metrics by:
    • Resource provider
    • Tags
    • Hosts
    • App Service Plans
    • Container Apps

You can also click to enable custom metric collection from Azure Application Insights, and disable the collection of usage metrics.

  1. Optionally, click the resource collection toggle to disable the collection of configuration information from your Azure resources.

  2. Enable log collection to set up and configure the services and diagnostic settings needed to forward logs to Datadog:

    1. If a log forwarder already exists in the tenant, it is modified to extend its scope. Any changed settings apply to existing as well as newly-selected subscriptions or management groups.
    2. If you're creating a new log forwarder:
      1. Enter a resource group name to store the log forwarder control plane
      2. Select a control plane subscription for the log-forwarding orchestration (LFO).
      3. Select a region for the control plane.
        Note: The resource group name, control plane subscription, and region fields only appear when creating a new log forwarder.
    3. Optionally, open {{< ui >}}Log filtering options{{< /ui >}} to filter logs by tags, or apply filtering for specific information (such as PII) using regex.

    See the Architecture section of the automated log forwarding guide for more information about this architecture.

  3. Click {{< ui >}}Confirm{{< /ui >}} to finish the setup.

{{% /collapse-content %}}

{{% collapse-content title="Terraform" expanded=false level="h3" id="terraform-setup" %}}

Choose the Terraform setup method if...

  • You manage infrastructure as code and want to keep the Datadog Azure integration under version control.
  • You need to configure multiple tenants or subscriptions consistently with reusable provider blocks.
  • You want a repeatable, auditable deployment process that fits into your Terraform-managed environment.

Instructions

Follow these steps to deploy the Datadog Azure integration through Terraform.

{{< tabs >}} {{% tab "Create an app registration" %}}

  1. In the Azure integration tile, click {{< ui >}}+ Add New App registration{{< /ui >}}, then select {{< ui >}}Terraform{{< /ui >}}.

  2. Select the subscriptions and management groups to collect data from.

  3. Under {{< ui >}}Authentication Method{{< /ui >}}, select {{< ui >}}Secretless Auth{{< /ui >}} (recommended) to authenticate with a federated credential, or {{< ui >}}Client Secret{{< /ui >}} to store and authenticate with a client secret.

  4. Optionally, click the metric collection toggle to disable all metric collection from Azure. You can also expand the {{< ui >}}Advanced Configuration{{< /ui >}} dropdown to filter metrics by:

    • Resource provider
    • Tags
    • Hosts
    • App Service Plans
    • Container Apps

    You can also click to enable custom metric collection from Azure Application Insights, and disable the collection of usage metrics.

  5. Optionally, click the resource collection toggle to disable the collection of configuration information from your Azure resources.

  6. Copy and run the command under {{< ui >}}Initialize and apply the Terraform{{< /ui >}}.

{{% /tab %}}

{{% tab "Use an existing app registration" %}}

  • You already have an app registration configured with the {{< ui >}}Monitoring Reader{{< /ui >}} role for Datadog to monitor the provided scope (subscriptions or management groups), and don't want to create new resources.

  1. Configure the Datadog Terraform provider to interact with the Datadog API through a Terraform configuration.

  2. Set up your Terraform configuration file using the example below as a base template. Update the following parameters before you apply the changes:

    • tenant_name: Your Azure Active Directory ID.
    • client_id: Your Azure application (client) ID.
    • secretless_auth_enabled: Set to true to authenticate with a federated credential instead of a client secret (recommended). To authenticate with a client secret instead, remove this parameter and set client_secret to the app registration's secret value.

    Secretless Auth requires a federated credential on the app registration that trusts Datadog's identity. See Authentication methods.

    See the Datadog Azure integration resource page in the Terraform registry for further example usage and the full list of optional parameters, as well as additional Datadog resources.

{{< code-block lang="hcl" filename="" disable_copy="false" collapsible="false" >}}

resource "datadog_integration_azure" "sandbox" { tenant_name = "<AZURE_TENANT_NAME>" client_id = "<AZURE_CLIENT_ID>" secretless_auth_enabled = true }

{{< /code-block >}}

  1. Run terraform apply. Wait up to 10 minutes for data to start being collected, and then view the out-of-the-box Azure overview dashboard to see metrics sent by your Azure resources.

{{% /tab %}} {{< /tabs >}}

Managing multiple subscriptions or tenants

You can use multiple provider blocks with aliases to manage Terraform resources across multiple subscriptions or tenants. Read Provider Configuration for more information.

Monitor the integration status

After the integration is configured, Datadog begins running a continuous series of calls to Azure APIs to collect critical monitoring data from your Azure environment. Sometimes these calls return errors (for example, if the provided credentials have expired). These errors can inhibit or block Datadog's ability to collect monitoring data.

When critical errors are encountered, the Azure integration generates events in the Datadog Events Explorer, and republishes them every five minutes. You can configure an Event Monitor to trigger when these events are detected and notify the appropriate team.

Datadog provides a monitor template to help you get started. To use the monitor template:

  1. In Datadog, go to {{< ui >}}Monitors{{< /ui >}} and click the {{< ui >}}Browse Templates{{< /ui >}} button.
  2. Search for and select the monitor template titled [Azure] Integration Errors.
  3. Make any desired modifications to the search query or alert conditions. By default, the monitor triggers whenever a new error is detected, and resolves when the error has not been detected for the past 15 minutes.
  4. Update the notification and re-notification messages as desired. Note that the events themselves contain pertinent information about the event and are included in the notification automatically. This includes detailed information about the scope, error response, and common steps to remediate.
  5. Configure notifications through your preferred channels (email, Slack, PagerDuty, or others) to make sure your team is alerted about issues affecting Azure data collection.

{{% /collapse-content %}}

{{% collapse-content title="Use an existing app registration" level="h3" expanded=false id="existing-app-registration-setup" %}}

Choose the existing app registration setup method if..

  • You already have an app registration configured with the {{< ui >}}Monitoring Reader{{< /ui >}} role for Datadog to monitor the provided scope (subscriptions or management groups), and don't want to create new resources.

If you need to set up an app registration for Datadog, see the Quickstart or Terraform setup methods.

Instructions

  1. In the Datadog Azure integration tile, select {{< ui >}}Add Existing{{< /ui >}}.
  2. In the {{< ui >}}Tenant ID{{< /ui >}} field, paste your Directory (tenant) ID.
  3. In the {{< ui >}}Client ID{{< /ui >}} field, paste the application (client) ID.
  4. In the {{< ui >}}Client Secret Value{{< /ui >}} field, paste the value of the app registration's client secret.
  5. Optionally, click the {{< ui >}}Monitor Automuting{{< /ui >}} toggle to disable monitor automuting.
  6. Optionally, click the metric collection toggle to disable all metric collection from Azure. You can also expand the {{< ui >}}Advanced Configuration{{< /ui >}} dropdown to filter metrics by:
    • Resource provider
    • Tags
    • Hosts
    • App Service Plans
    • Container Apps

You can also click to enable custom metric collection from Azure Application Insights, and disable the collection of usage metrics.

  1. Optionally, click the resource collection toggle to disable the collection of configuration information from your Azure resources.
  2. Click {{< ui >}}Create Configuration{{< /ui >}}.

{{% /collapse-content %}}

{{% collapse-content title="Migrate an existing app registration to Secretless Auth" level="h4" expanded=false id="secretless-migration-setup" %}}

If you already connected an app registration using a client secret, you can migrate it to Secretless Auth in place. You don't need to recreate the app registration.

  1. In the Azure integration tile, on the {{< ui >}}Configuration{{< /ui >}} tab, select the app registration you want to migrate.

  2. On the {{< ui >}}General{{< /ui >}} tab, under {{< ui >}}Authentication{{< /ui >}}, click {{< ui >}}Set Up Secretless Auth{{< /ui >}}.

    {{< img src="/getting_started/integrations/azure/GSwAzure_secretlessMigration.png" alt="The General tab for an app registration, with a callout recommending Secretless Auth and a Set Up Secretless Auth button" style="width:100%;" >}}

In the {{< ui >}}Secretless Authentication Setup{{< /ui >}} dialog, select a setup method, then complete the steps for that method:

{{< img src="/getting_started/integrations/azure/GSwAzure_secretlessSetupModal.png" alt="The Secretless Authentication Setup dialog with the Azure CLI, Terraform, and Azure Portal setup methods and steps to create a federated credential, verify it, and complete setup" style="width:100%;" >}}

{{< tabs >}} {{% tab "Azure CLI" %}}

  1. Run the provided az ad app federated-credential create command in a terminal with the Azure CLI configured for the correct tenant, or click {{< ui >}}Open Azure Cloud Shell{{< /ui >}}.
  2. Click {{< ui >}}Verify Credential{{< /ui >}} to test the authentication.
  3. Click {{< ui >}}Confirm{{< /ui >}}. Your client secret is removed when setup succeeds.

{{% /tab %}} {{% tab "Terraform" %}}

This method requires the azuread provider version 3.7.0 or later.

  1. Add the federated credential resource shown in the dialog to the same configuration that manages your Datadog Azure integration, then apply it. Copy the issuer and subject values from the dialog.

    {{< code-block lang="hcl" filename="" disable_copy="false" collapsible="false" >}} resource "azuread_application_federated_identity_credential" "datadog_federated_credential" {

Reference the ID of this app registration

application_id = azuread_application.datadog_app.id display_name = "datadog" description = "Federated credential that permits Datadog to authenticate without storing a client secret" audiences = ["api://AzureADTokenExchange"] issuer = "" subject = "" } {{< /code-block >}}

  1. Click {{< ui >}}Verify Credential{{< /ui >}} to test the authentication.
  2. Modify your datadog_integration_azure resource to use Secretless Auth:
    • Add the federated credential resource to the integration's dependency list.
    • Replace the client_secret setting with secretless_auth_enabled = true.
    • Apply the changes.

{{% /tab %}} {{% tab "Azure Portal" %}}

  1. In the Azure portal, open the app registration and go to {{< ui >}}Certificates & secrets{{< /ui >}} > {{< ui >}}Federated credentials{{< /ui >}} > {{< ui >}}Add credential{{< /ui >}}.
  2. Select {{< ui >}}Other issuer{{< /ui >}}, then enter the {{< ui >}}Issuer{{< /ui >}}, {{< ui >}}Subject{{< /ui >}}, and {{< ui >}}Audience{{< /ui >}} values exactly as shown in the dialog.
  3. Click {{< ui >}}Verify Credential{{< /ui >}} to test the authentication.
  4. Click {{< ui >}}Confirm{{< /ui >}}. Your client secret is removed when setup succeeds.

{{% /tab %}} {{< /tabs >}}

{{% /collapse-content %}}

Metric collection

Datadog's Azure integration is built to collect all metrics from Azure Monitor. The Integrations page shows a curated list of predefined sub-integrations that provide additional out-of-the-box dashboards and monitors for specific Azure services. Many of these integrations are installed by default when Datadog recognizes data coming in from your Azure account. However, Datadog can ingest metrics from any Azure Monitor-supported resource, even if it doesn't have a dedicated sub-integration tile.

You can find your Azure metrics in the metrics summary page in the Datadog platform by navigating to Metrics > Summary and searching for Azure.

{{< img src="/getting_started/integrations/azure/GSwAzure_metricExplorer.png" alt="Metric summary image" style="width:100%;" >}}

Resource tag filtering for metrics

Use tag filters to control which Azure resources have their metrics collected by Datadog. Configure tag filters in the {{< ui >}}Configuration{{< /ui >}} tab of the Azure integration tile. A tag filter is a comma-separated list of tags in the form key:value. Only resources that match at least one tag in the filter have their metrics collected.

You can use wildcards in your tag filters:

  • ? matches a single character.
  • * matches multiple characters.

To exclude resources with a given tag, prefix the tag with !. Exclusion takes precedence over inclusion. A resource matches the filter if it matches any tag in the list.

For example: datadog:monitored,env:production,!plan_tier:basic,instance-type:c1.*

This filter collects metrics from resources tagged with datadog:monitored or env:production, excludes resources tagged with plan_tier:basic, and includes resources with an instance-type tag matching c1.*.

If no tag filter is set, Datadog collects metrics from all Azure resources.

Enable log collection

You can use the automated log forwarding feature to setup and configure the services and diagnostic settings needed to forward logs to Datadog. If an automated log forwarding control plane already exists in the tenant, this flow modifies it and extends its scope to include the selected subscriptions or management groups. For more detail, see Azure Automated Log Forwarding Setup.

Datadog recommends using the Agent or DaemonSet to send logs from Azure. If direct streaming isn't possible, use the {{< ui >}}Configure Log Forwarding{{< /ui >}} flow in the Azure integration to set up and manage automated log forwarding directly in Datadog. You can also deploy log forwarding with an Azure Resource Manager (ARM) template. Both methods automatically manage and scale log forwarding services.

{{% collapse-content title="Automated (recommended)" level="h3" expanded=false id="automated-log-forwarding-setup" %}}

Choose the automated log forwarding setup method if...

  • You haven't already set up logs through the Quickstart setup method.
  • You prefer a UI-based workflow and want to minimize the time it takes to create a service principal with the required monitoring permissions.
  • You want to automate setup steps in scripts or CI/CD pipelines.

Instructions

Configure Log Forwarding (recommended)

Use the {{< ui >}}Configure Log Forwarding{{< /ui >}} flow to set up new or manage existing log forwarders directly in Datadog:

  1. In Datadog, navigate to {{< ui >}}Integrations{{< /ui >}} > {{< ui >}}Azure{{< /ui >}}.
  2. Click {{< ui >}}Configure Log Forwarding{{< /ui >}}.
  3. Copy the provided command and paste it in your Azure Cloud Shell.
  4. Select the subscriptions to forward logs from.
  5. Optionally, add or remove log filters.
  6. Click {{< ui >}}Confirm{{< /ui >}}.

For more details, see Azure Automated Log Forwarding Setup.

ARM template

Alternatively, deploy log forwarding with an Azure Resource Manager (ARM) template:

  1. Open the Automated Log Forwarding ARM template in Azure.
  2. Configure your Azure project and instance details on the Basics tab.
  3. Enter your Datadog credentials on the Datadog Configuration tab.
  4. Acknowledge deployment warnings on the Deployment tab.
  5. Start the deployment process on the Review + create tab.

{{< site-region region="us3" >}}

Log Archives require the app registration setup method. For Datadog accounts using the Azure Native integration, follow the steps on this page to create an app registration.

{{< /site-region >}}

See Azure Automated Log Forwarding Architecture for more details.

{{% /collapse-content %}}

{{% collapse-content title="Container App" level="h3" expanded=false id="container-app-log-forwarding-setup" %}}

Choose the Container App log forwarding method if...

  • You prefer to manually configure diagnostic settings on the resources you want to forward logs from.

Instructions

  1. Click the button below, and fill in the form on the Azure Portal. Datadog automatically deploys the Azure resources required to forward logs into your Datadog account.

    Deploy to Azure

  2. After the template deployment finishes, set up diagnostic settings for each log source to send Azure platform logs (including resource logs) to the Storage Account created during deployment.

Note: Resources can only stream to a Storage Account in the same Azure region.

{{% /collapse-content %}}

{{% azure-log-archiving %}}

Resource tag filtering for logs

Use tag filters to control which Azure resources have their logs forwarded to Datadog. To configure tag filters for logs, click {{< ui >}}Configure Log Forwarding{{< /ui >}} in the Azure integration tile and follow the flow. A tag filter is a comma-separated list of tags in the form key:value. Only resources that match at least one tag in the filter have their logs forwarded.

You can use wildcards in your tag filters:

  • ? matches a single character.
  • * matches multiple characters.

To exclude resources with a given tag, prefix the tag with !. Exclusion takes precedence over inclusion. A resource matches the filter if it matches any tag in the list.

For example: datadog:monitored,env:production,!plan_tier:basic,instance-type:c1.*

This filter forwards logs from resources tagged with datadog:monitored or env:production, excludes resources tagged with plan_tier:basic, and includes resources with an instance-type tag matching c1.*.

If no tag filter is set, Datadog forwards logs from all Azure resources.

Get more from the Datadog Platform

Install the Agent for greater visibility into your application

After you set up your Azure integration, Datadog crawlers automatically collect Azure metrics, but you can gain even deeper visibility into your Azure instances with the Datadog Agent. Installing the Datadog Agent into your environment allows you to collect additional data including, but not limited to:

  • Application health
  • Process utilization
  • System-level metrics

You can also use the built-in StatsD client to send custom metrics from your applications, to correlate what's happening with your applications, users, and system. See the guide on Why should I install the Datadog Agent on my cloud instances? for more information on the benefits of installing the Datadog Agent on your instances.

Use the Azure extension to install the Datadog Agent on Windows VMs, Linux x64 VMs, and Linux ARM-based VMs. You can also use the AKS Cluster Extension to deploy the Agent to your AKS Clusters.

{{< tabs >}} {{% tab "VM Extension" %}}

  1. In the Azure portal, select the appropriate VM.
  2. From the left sidebar, under {{< ui >}}Settings{{< /ui >}}, select {{< ui >}}Extensions + applications{{< /ui >}}.
  3. Click {{< ui >}}+ Add{{< /ui >}}.
  4. Search for and select the {{< ui >}}Datadog Agent{{< /ui >}} extension.
  5. Click {{< ui >}}Next{{< /ui >}}.
  6. Enter your Datadog API key and Datadog site, and click {{< ui >}}OK{{< /ui >}}.

To install the Agent based on operating system or CI and CD tool, see the Datadog Agent installation instructions.

Note: Domain controllers are not supported when installing the Datadog Agent with the Azure extension.

{{% /tab %}}

{{% tab "AKS Cluster Extension" %}}

The Datadog AKS Cluster Extension allows you to deploy the Datadog Agent natively within Azure AKS, avoiding the complexity of third-party management tools. To install the Datadog Agent with the AKS Cluster Extension:

  1. Go to your AKS cluster in the Azure portal.
  2. From the left sidebar of the AKS cluster, select {{< ui >}}Extensions + applications{{< /ui >}} under {{< ui >}}Settings{{< /ui >}}.
  3. Search for and select the {{< ui >}}Datadog AKS Cluster Extension{{< /ui >}}.
  4. Click {{< ui >}}Create{{< /ui >}}, and follow the instructions in the tile using your Datadog credentials and Datadog site.

{{% /tab %}} {{< /tabs >}}

Troubleshooting

See Troubleshooting in the Azure Advanced Configuration guide.

Still need help? Contact Datadog support.

Further Reading

{{< partial name="whats-next/whats-next.html" >}}