Skip to content

Commit 304a409

Browse files
elihschiffelihschiff
authored
Document Azure and Google Cloud persistent volume allocation (#39497)
* Document Azure and GCP persistent volume allocation * Remove outdated StatefulSet limitation
1 parent 774e2db commit 304a409

1 file changed

Lines changed: 61 additions & 5 deletions

File tree

‎hugo/content/en/cloud_cost_management/allocation/container_cost_allocation.mdoc.md‎

Lines changed: 61 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ The following table presents the list of collected features and the minimal Agen
6666
|---|---|---|
6767
| Container Cost Allocation | 7.27.0 | 1.11.0 |
6868
| GPU Container Cost Allocation | 7.54.0 | 7.54.0 |
69+
| Azure Persistent Volume Allocation | 7.46.0 | 1.11.0 |
6970

7071
1. Configure the Azure Cost Management integration on the [Cloud Cost Setup page][2].
7172
1. Install the [Datadog Agent][3] in a Kubernetes environment and ensure that you enable the [Orchestrator Explorer][4] in your Agent configuration.
@@ -87,6 +88,7 @@ The following table presents the list of collected features and the minimal Agen
8788
| Container Cost Allocation | 7.27.0 | 1.11.0 |
8889
| GPU Container Cost Allocation | 7.54.0 | 7.54.0 |
8990
| Local Storage Cost Allocation | 7.27.0 | 1.11.0 |
91+
| Google Cloud Persistent Volume Allocation | 7.46.0 | 1.11.0 |
9092

9193
1. Configure the Google Cloud Cost Management integration on the [Cloud Cost Setup page][2].
9294
1. Install the [Datadog Agent][3] in a Kubernetes environment and ensure that you enable the [Orchestrator Explorer][4] in your Agent configuration.
@@ -170,6 +172,12 @@ Next, Datadog looks at all of the pods running on that node for the day. The cos
170172

171173
All other costs are given the same value and tags as the source metric `azure.cost.amortized`.
172174

175+
### Persistent volume storage
176+
177+
For Kubernetes persistent volume storage allocation, Persistent Volumes (PV), Persistent Volume Claims (PVC), nodes, and pods are joined with their associated Azure Managed Disk costs. All associated PV, PVC, node, and pod tags are added to the disk cost line items.
178+
179+
Next, Datadog examines all pods that claimed the volume that day. The cost of the volume is allocated to each pod based on the resources it used and the length of time it ran. These resources include provisioned storage capacity, IOPS, and throughput. The allocated cost is enriched with all of the pod's tags.
180+
173181
{% /if %}
174182

175183
<!-- Google -->
@@ -195,6 +203,12 @@ For Kubernetes local storage allocation, a Kubernetes node is joined with Google
195203

196204
Next, Datadog examines all pods running on that node for the day. For each pod, Datadog reserves the larger of its ephemeral-storage request and its daily peak usage. The average usage determines the usage cost, the remainder of the reservation is workload idle, and capacity not reserved by any pod is cluster idle. The allocated cost is enriched with the pod's tags.
197205

206+
### Persistent volume storage
207+
208+
For Kubernetes persistent volume storage allocation, Persistent Volumes (PV), Persistent Volume Claims (PVC), nodes, and pods are joined with their associated Google Cloud persistent disk costs. All associated PV, PVC, node, and pod tags are added to the disk cost line items.
209+
210+
Next, Datadog examines all pods that claimed the volume that day. The cost of the volume is allocated to each pod based on the resources it used and the length of time it ran. These resources include provisioned storage capacity, IOPS, and throughput. The allocated cost is enriched with all of the pod's tags.
211+
198212
### Agentless Kubernetes costs
199213

200214
To view the costs of GKE clusters without enabling Datadog Infrastructure Monitoring, use [GKE cost allocation][18]. Enable GKE cost allocation on unmonitored GKE clusters to access this feature set. This approach comes with the following limitations.
@@ -239,8 +253,6 @@ The cost of an EBS volume has three components: IOPS, throughput, and storage. E
239253
| Workload idle | Cost of provisioned IOPS, throughput, or storage that are reserved and allocated but not used by workloads. Storage cost is based on the maximum amount of volume storage used that day, while IOPS and throughput costs are based on the average amount of volume storage used that day. This is the difference between the total resources requested and the average usage. **Note:** This tag is only available if you have enabled {% ui %}Resource Collection{% /ui %} in your [AWS Integration][21]. To prevent being charged for {% ui %}Cloud Security Posture Management{% /ui %}, ensure that during the {% ui %}Resource Collection{% /ui %} setup, the {% ui %}Cloud Security Posture Management{% /ui %} box is unchecked. |
240254
| Cluster idle | Cost of provisioned IOPS, throughput, or storage that are not reserved by any pods that day. This is the difference between the total cost of the resources and what is allocated to workloads. |
241255

242-
**Note**: Persistent volume allocation is only supported in Kubernetes clusters, and is only available for pods that are part of a Kubernetes StatefulSet.
243-
244256
### Data transfer
245257

246258
Costs are allocated into the following spend types:
@@ -267,6 +279,16 @@ Costs are allocated into the following spend types:
267279
| Workload idle | Cost of resources (such as memory, CPU, and GPU) that are reserved and allocated but not used by workloads. This is the difference between the total resources requested and the average usage. |
268280
| Cluster idle | Cost of resources (such as memory, CPU, and GPU) that are not reserved by workloads in a cluster. This is the difference between the total cost of the resources and what is allocated to workloads. |
269281

282+
### Persistent volume
283+
284+
The cost of an Azure Managed Disk can include storage, provisioned IOPS, and provisioned throughput. Each component is allocated according to a pod's usage when the volume is mounted.
285+
286+
| Spend type | Description |
287+
| -----------| ----------- |
288+
| Usage | Cost of provisioned storage, IOPS, or throughput used by workloads. Storage usage is based on the maximum storage used that day. IOPS and throughput usage are based on the average operations and bytes transferred that day. |
289+
| Workload idle | Cost of provisioned storage, IOPS, or throughput reserved for a workload but not used. This is the difference between the provisioned capacity allocated to the workload and its usage. |
290+
| Cluster idle | Cost of provisioned storage, IOPS, or throughput not allocated to any pod that day. |
291+
270292
{% /if %}
271293

272294
<!-- Google -->
@@ -298,6 +320,16 @@ Local storage costs are allocated into the following spend types:
298320

299321
Local storage is tied to the life cycle of a node. It does not include persistent volumes, which retain data independently of pods and nodes.
300322

323+
### Persistent volume
324+
325+
The cost of a Google Cloud persistent disk can include storage, provisioned IOPS, and provisioned throughput. Each component is allocated according to a pod's usage when the volume is mounted.
326+
327+
| Spend type | Description |
328+
| -----------| ----------- |
329+
| Usage | Cost of provisioned storage, IOPS, or throughput used by workloads. Storage usage is based on the maximum storage used that day. IOPS and throughput usage are based on the average operations and bytes transferred that day. |
330+
| Workload idle | Cost of provisioned storage, IOPS, or throughput reserved for a workload but not used. This is the difference between the provisioned capacity allocated to the workload and its usage. |
331+
| Cluster idle | Cost of provisioned storage, IOPS, or throughput not allocated to any pod that day. |
332+
301333
{% /if %}
302334

303335
### Aggregated minor pod usage
@@ -367,7 +399,7 @@ Depending on the cloud provider, certain resources may or may not be available f
367399
|---:|---:|---|---|
368400
| CPU | {% x/ %} | {% x/ %} | {% x/ %} |
369401
| Memory | {% x/ %} | {% x/ %} | {% x/ %} |
370-
| {% tooltip contents="Storage resources within a cluster, provisioned by administrators or dynamically, that persist data independently of pod life cycles." %} Persistent volumes {% /tooltip %} | {% x/ %} | | |
402+
| {% tooltip contents="Storage resources within a cluster, provisioned by administrators or dynamically, that persist data independently of pod life cycles." %} Persistent volumes {% /tooltip %} | {% x/ %} | {% x/ %} | {% x/ %} |
371403
| {% tooltip contents="Cost of associated fees charged by the cloud provider for managing the cluster, such as fees for managed Kubernetes services or other container orchestration options." %} Managed service fees {% /tooltip %} | {% x/ %} | {% x/ %} | {% x/ %} |
372404
| ECS costs | {% x/ %} | N/A | N/A |
373405
| Data transfer costs | {% x/ %} | Limited* | Limited* |
@@ -394,15 +426,15 @@ When the prerequisites are met, the following cost metrics automatically appear.
394426

395427
| Cost Metric | Description |
396428
| --- | ----------- |
397-
| `azure.cost.amortized.shared.resources.allocated` | Azure VM costs allocated by the CPU & memory used by a pod or container task, using a 60:40 split for CPU & memory respectively and a 95:3:2 split for GPU, CPU, & memory respectively if a GPU is used by a pod. Also includes allocated Azure costs. <br> *Based on `azure.cost.amortized`* |
429+
| `azure.cost.amortized.shared.resources.allocated` | Azure VM costs allocated by the CPU & memory used by a pod or container task, using a 60:40 split for CPU & memory respectively and a 95:3:2 split for GPU, CPU, & memory respectively if a GPU is used by a pod. Also includes allocated Azure Managed Disk costs. <br> *Based on `azure.cost.amortized`* |
398430

399431
{% /if %}
400432
<!-- Google -->
401433
{% if equals($platform, "google") %}
402434

403435
| Cost Metric | Description |
404436
| --- | ----------- |
405-
| `gcp.cost.amortized.shared.resources.allocated` | Google Compute Engine costs allocated by the CPU, memory, and local storage used by a pod. When the bill does not provide a specific CPU and memory split, costs use a 60:40 split. GPU hosts use a 95:3:2 split for GPU, CPU, and memory. <br> *Based on `gcp.cost.amortized`* |
437+
| `gcp.cost.amortized.shared.resources.allocated` | Google Compute Engine costs allocated by the CPU, memory, and local storage used by a pod. When the bill does not provide a specific CPU and memory split, costs use a 60:40 split. GPU hosts use a 95:3:2 split for GPU, CPU, and memory. Also includes allocated Google Cloud persistent disk costs. <br> *Based on `gcp.cost.amortized`* |
406438

407439
{% /if %}
408440

@@ -496,6 +528,18 @@ In addition to Kubernetes pod and Kubernetes node tags, the following non-exhaus
496528
| `allocated_resource:data_transfer` | The tracking and allocation of costs associated with data transfer activities used by Azure services or workloads. |
497529
| `allocated_resource:local_storage` | The tracking and allocation of costs at a host level associated with local storage resources used by Azure services or workloads. |
498530

531+
#### Persistent volume
532+
533+
In addition to Kubernetes pod and node tags, the following out-of-the-box tags are applied to persistent volume cost metrics:
534+
535+
| Out-of-the-box tag | Description |
536+
| --- | --- |
537+
| `persistent_volume_reclaim_policy` | The Kubernetes reclaim policy on the Persistent Volume. |
538+
| `storage_class_name` | The Kubernetes Storage Class used to instantiate the Persistent Volume. |
539+
| `volume_mode` | The Volume Mode of the Persistent Volume. |
540+
| `azure_disk_storage_account_type` | The Azure storage account type for the managed disk. |
541+
| `allocated_resource:persistent_volume` | Identifies costs allocated from an Azure Managed Disk to a Persistent Volume. |
542+
499543
{% /if %}
500544
<!-- Google -->
501545
{% if equals($platform, "google") %}
@@ -514,6 +558,18 @@ In addition to Kubernetes pod and Kubernetes node tags, the following non-exhaus
514558
| `allocated_resource:gpu` | The tracking and allocation of costs at a host level associated with GPU resources used by Google Cloud services or workloads. |
515559
| `allocated_resource:local_storage` | Identifies local storage costs allocated to Google Cloud Kubernetes workloads. |
516560

561+
#### Persistent volume
562+
563+
In addition to Kubernetes pod and node tags, the following out-of-the-box tags are applied to persistent volume cost metrics:
564+
565+
| Out-of-the-box tag | Description |
566+
| --- | --- |
567+
| `persistent_volume_reclaim_policy` | The Kubernetes reclaim policy on the Persistent Volume. |
568+
| `storage_class_name` | The Kubernetes Storage Class used to instantiate the Persistent Volume. |
569+
| `volume_mode` | The Volume Mode of the Persistent Volume. |
570+
| `gcp_disk_type` | The Google Cloud disk type for the persistent disk. |
571+
| `allocated_resource:persistent_volume` | Identifies costs allocated from a Google Cloud persistent disk to a Persistent Volume. |
572+
517573
{% /if %}
518574

519575
[1]: https://app.datadoghq.com/cost/containers

0 commit comments

Comments
 (0)