Skip to content
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ public static class AzureKubernetesPersistentVolumeExtensions
/// .WithCapacity("20Gi");
///
/// builder.AddProject<Projects.Api>("api")
/// .WithPersistentVolume(data, "/data");
/// .WithPersistentVolume(data, "/data", env: "DATA_PATH");
/// </code>
/// </example>
[AspireExport]
Expand Down
8 changes: 5 additions & 3 deletions src/Aspire.Hosting.Azure.Kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Add a persistent volume to the AKS environment and mount it into a workload:
var data = aks.AddPersistentVolume("data")
.WithCapacity("20Gi");

myService.WithPersistentVolume(data, "/data");
myService.WithPersistentVolume(data, "/data", env: "DATA_PATH");
```

**TypeScript**
Expand All @@ -60,10 +60,12 @@ myService.WithPersistentVolume(data, "/data");
const data = await aks.addPersistentVolume("data");
await data.withCapacity("20Gi");

await myService.withKubernetesPersistentVolumeMount(data, "/data");
await myService.withKubernetesPersistentVolumeMount(data, "/data", { env: "DATA_PATH" });
```

When no storage class is specified, the generated claim uses the cluster's default storage class. A standard AKS cluster dynamically provisions an Azure managed disk. To request Premium SSD storage explicitly, call `WithStorageClass("managed-csi-premium")` in C# or `withStorageClass("managed-csi-premium")` in TypeScript.
When a project or executable runs locally, `DATA_PATH` points to a persistent directory in the AppHost's Aspire store. That store is normally under the AppHost intermediate-output directory, so cleaning build outputs can remove the local data. Local containers use a worktree-scoped container volume instead, and one persistent-volume resource cannot be shared between local containers and local projects or executables.

In AKS, `DATA_PATH` contains `/data`, the mounted volume path. When no storage class is specified, the generated claim uses the cluster's default storage class. A standard AKS cluster dynamically provisions an Azure managed disk. To request Premium SSD storage explicitly, call `WithStorageClass("managed-csi-premium")` in C# or `withStorageClass("managed-csi-premium")` in TypeScript.

## Additional documentation

Expand Down
20 changes: 20 additions & 0 deletions src/Aspire.Hosting.Docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,26 @@ builder.AddDockerComposeEnvironment("compose");
await builder.addDockerComposeEnvironment("compose");
```

### Volumes

Use an environment variable so projects and executables can use a local Aspire store directory while Docker Compose mounts the published named volume:

**C#**

```csharp
builder.AddProject<Projects.Api>("api")
.WithVolume("data", "/data", env: "DATA_PATH");
```

**TypeScript**

```typescript
const api = await builder.addNodeApp("api", "../api", "server.js");
await api.withVolume("/data", { name: "data", env: "DATA_PATH" });
Comment thread
mitchdenny marked this conversation as resolved.
Outdated
```

In run mode, projects and executables receive a workload-scoped directory through `DATA_PATH`. Containers receive `/data` and use a local container volume. Named storage is preserved independently of resource lifetime: session resources stop with the AppHost and reuse their storage on the next run, while persistent resources can keep the compute instance alive. In the generated Compose service, all compute resource types receive `/data` and a named volume mounted at that path.

```shell
aspire publish -o docker-compose-artifacts
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,25 @@ namespace Aspire.Hosting.Kubernetes.Annotations;
/// promote the workload to a <c>StatefulSet</c>.
/// </summary>
/// <param name="volume">The persistent volume resource the workload binds to.</param>
internal sealed class KubernetesPersistentVolumeBindingAnnotation(KubernetesPersistentVolumeResource volume) : IResourceAnnotation
/// <param name="environmentVariableName">The environment variable that exposes the effective mount path, if configured.</param>
/// <param name="runModeContainerVolumeName">The worktree-scoped container volume name to apply in run mode, if required.</param>
internal sealed class KubernetesPersistentVolumeBindingAnnotation(
KubernetesPersistentVolumeResource volume,
string? environmentVariableName = null,
string? runModeContainerVolumeName = null) : IResourceAnnotation
{
/// <summary>
/// Gets the persistent volume resource bound to the workload.
/// </summary>
public KubernetesPersistentVolumeResource Volume { get; } = volume ?? throw new ArgumentNullException(nameof(volume));

/// <summary>
/// Gets the environment variable that exposes the effective mount path, if configured.
/// </summary>
public string? EnvironmentVariableName { get; } = environmentVariableName;

/// <summary>
/// Gets the worktree-scoped container volume name to apply in run mode, if required.
/// </summary>
public string? RunModeContainerVolumeName { get; } = runModeContainerVolumeName;
}
147 changes: 144 additions & 3 deletions src/Aspire.Hosting.Kubernetes/KubernetesEnvironmentExtensions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

using Aspire.Hosting.ApplicationModel;
using Aspire.Hosting.Kubernetes;
using Aspire.Hosting.Kubernetes.Annotations;
using Aspire.Hosting.Kubernetes.Extensions;
using Aspire.Hosting.Pipelines;
using Microsoft.Extensions.DependencyInjection;
Expand All @@ -27,9 +28,8 @@ internal static IDistributedApplicationBuilder AddKubernetesInfrastructureCore(t
//
// The per-environment work (creating Kubernetes service resources and DeploymentTargetAnnotations)
// is registered as a separate per-environment pipeline step on KubernetesEnvironmentResource.
// This global step only validates that no resource has a PublishAsKubernetesService annotation
// when there are no KubernetesEnvironmentResource instances or Kubernetes-backed
// compute environments in the model.
// This global step validates model-wide Kubernetes configuration before those steps filter
// resources to their selected compute environments.
if (builder.Services.All(d => d.ServiceType != typeof(KubernetesPipelineStepMarker)))
{
builder.Services.AddSingleton<KubernetesPipelineStepMarker>();
Expand All @@ -38,6 +38,8 @@ internal static IDistributedApplicationBuilder AddKubernetesInfrastructureCore(t
name: KubernetesPipelineStepMarker.StepName,
action: ctx =>
{
ValidateAndFinalizePersistentVolumeBindings(ctx);

if (!ctx.ExecutionContext.IsPublishMode)
{
return Task.CompletedTask;
Expand All @@ -60,12 +62,151 @@ internal static IDistributedApplicationBuilder AddKubernetesInfrastructureCore(t

return Task.CompletedTask;
},
dependsOn: WellKnownPipelineSteps.ValidateComputeEnvironments,
Comment thread
mitchdenny marked this conversation as resolved.
requiredBy: WellKnownPipelineSteps.BeforeStart);
}

return builder;
}

private static void ValidateAndFinalizePersistentVolumeBindings(PipelineStepContext context)
{
var bindings = GetPersistentVolumeBindings(context);

if (context.ExecutionContext.IsRunMode)
{
ValidateRunModePersistentVolumeBindings(bindings);
ApplyRunModeContainerVolumeNames(bindings);
return;
}

ValidatePublishModePersistentVolumeBindings(bindings);
}

private static PersistentVolumeBinding[] GetPersistentVolumeBindings(PipelineStepContext context)
{
// GetComputeResources intentionally represents publishable workloads and excludes plain
// executables. Run mode must inspect every compute resource because those executables can
// consume the local IAspireStore-backed volume path.
var computeResources = context.ExecutionContext.IsRunMode
? context.Model.Resources.Where(resource => resource is IComputeResource)
: context.Model.GetComputeResources();

return computeResources
.SelectMany(resource =>
resource.Annotations
.OfType<KubernetesPersistentVolumeBindingAnnotation>()
.Select(binding => new PersistentVolumeBinding(resource, binding)))
.ToArray();
}

private static void ValidateRunModePersistentVolumeBindings(PersistentVolumeBinding[] bindings)
{
foreach (var (resource, annotation) in bindings)
{
if (annotation.EnvironmentVariableName is not null &&
resource is not ProjectResource and not ExecutableResource and not ContainerResource)
{
throw new DistributedApplicationException(
$"Resource '{resource.Name}' cannot resolve the '{annotation.EnvironmentVariableName}' persistent-volume path in run mode. " +
$"Only project, executable, and container resources are supported.");
}
}

ValidateRunModeBackingStoreCompatibility(bindings);
}

private static void ValidateRunModeBackingStoreCompatibility(PersistentVolumeBinding[] bindings)
{
// Host processes use an IAspireStore directory while containers use a named runtime
// volume. Treating those as one logical volume would silently split the data in run mode.
foreach (var environmentGroup in bindings.GroupBy(
item => item.Annotation.Volume.Parent.Name,
StringComparer.OrdinalIgnoreCase))
{
foreach (var volumeGroup in environmentGroup.GroupBy(
item => item.Annotation.Volume.Name,
StringComparer.OrdinalIgnoreCase))
{
var hasContainer = volumeGroup.Any(item => item.Resource is ContainerResource);
var hasHostProcess = volumeGroup.Any(item => item.Resource is ProjectResource or ExecutableResource);

if (hasContainer && hasHostProcess)
Comment thread
mitchdenny marked this conversation as resolved.
Outdated
{
var volume = volumeGroup.First().Annotation.Volume;
var resourceNames = string.Join(", ", volumeGroup.Select(item => $"'{item.Resource.Name}'"));
throw new DistributedApplicationException(
$"Kubernetes persistent volume '{volume.Name}' is used by both local container and host-process resources ({resourceNames}). " +
$"Run mode cannot provide one shared backing store across those execution types. Use only containers or only projects/executables for this volume.");
}
}
}
}

private static void ApplyRunModeContainerVolumeNames(PersistentVolumeBinding[] bindings)
{
foreach (var (resource, annotation) in bindings)
{
ApplyRunModeContainerVolumeName(resource, annotation);
}
}

private static void ValidatePublishModePersistentVolumeBindings(PersistentVolumeBinding[] bindings)
{
foreach (var (resource, annotation) in bindings)
{
var targetEnvironment = resource.GetComputeEnvironment();
var volumeEnvironment = annotation.Volume.Parent;

// AKS owns an inner Kubernetes environment, so a binding is valid when the workload
// targets either the Kubernetes environment directly or its owning compute environment.
if (targetEnvironment != volumeEnvironment &&
targetEnvironment != volumeEnvironment.OwningComputeEnvironment)
Comment thread
mitchdenny marked this conversation as resolved.
{
var targetName = targetEnvironment?.Name ?? "<none>";
var supportedTargetName = (volumeEnvironment.OwningComputeEnvironment ?? volumeEnvironment).Name;
throw new DistributedApplicationException(
$"Resource '{resource.Name}' is assigned to compute environment '{targetName}' but binds " +
$"Kubernetes persistent volume '{annotation.Volume.Name}' which belongs to environment " +
$"'{volumeEnvironment.Name}'. A workload can only bind persistent volumes declared on its " +
$"Kubernetes compute environment. Declare the volume on the workload's Kubernetes environment, " +
$"or assign the workload to '{supportedTargetName}' with WithComputeEnvironment.");
}
}
}

private static void ApplyRunModeContainerVolumeName(
IResource resource,
KubernetesPersistentVolumeBindingAnnotation binding)
{
if (resource is not ContainerResource || binding.RunModeContainerVolumeName is not { } localVolumeName)
{
return;
}

// Resolve after the model is complete so WithPersistentVolume and WithVolume remain
// order-independent while publish mode can continue matching the original claim name.
//
// Replace each mount in place rather than Remove-then-Add. ContainerMountAnnotation is a
// record, so Remove matches by value and a Remove/Add pair would both relocate the mount to
// the end of the annotation collection and risk removing a value-identical sibling.
// Assigning through the indexer preserves position and swaps atomically.
var annotations = resource.Annotations;

for (var i = 0; i < annotations.Count; i++)
{
if (annotations[i] is ContainerMountAnnotation { Type: ContainerMountType.Volume } mount &&
string.Equals(mount.Source, binding.Volume.Name, StringComparison.Ordinal))
{
annotations[i] = new ContainerMountAnnotation(localVolumeName, mount.Target, mount.Type, mount.IsReadOnly);
Comment thread
mitchdenny marked this conversation as resolved.
}
}
}

private readonly record struct PersistentVolumeBinding(
IResource Resource,
KubernetesPersistentVolumeBindingAnnotation Annotation);

private sealed class KubernetesPipelineStepMarker
{
public const string StepName = "validate-kubernetes";
Expand Down
25 changes: 0 additions & 25 deletions src/Aspire.Hosting.Kubernetes/KubernetesEnvironmentResource.cs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
using System.Security.Cryptography.X509Certificates;
using Aspire.Hosting.ApplicationModel;
using Aspire.Hosting.Dcp.Process;
using Aspire.Hosting.Kubernetes.Annotations;
using Aspire.Hosting.Kubernetes.Extensions;
using Aspire.Hosting.Kubernetes.Resources;
using Aspire.Hosting.Pipelines;
Expand Down Expand Up @@ -473,30 +472,6 @@ private async Task PrepareDeploymentTargetsAsync(PipelineStepContext context)
ConfigureOtlp(r, otlpGrpcEndpoint);
}

// Fail the publish if this workload binds a persistent volume owned by a
// different Kubernetes environment. The two charts would render into
// separate namespaces/clusters, so the workload's claimName reference would
// resolve to a PVC that does not exist alongside it — Kubernetes would fail
// to schedule the pod with "persistentvolumeclaim not found". Detect and
// surface it here where we have both the resolved workload compute
// environment and the annotated PV.
if (r.TryGetAnnotationsOfType<KubernetesPersistentVolumeBindingAnnotation>(out var pvBindings))
{
foreach (var binding in pvBindings)
{
if (binding.Volume.Parent != this)
{
throw new InvalidOperationException(
$"Resource '{r.Name}' is assigned to Kubernetes environment '{Name}' but binds " +
$"persistent volume '{binding.Volume.Name}' which belongs to environment " +
$"'{binding.Volume.Parent.Name}'. A workload can only bind persistent volumes " +
$"declared on its own environment. Move the AddPersistentVolume call to " +
$"'{Name}', or assign the workload to '{binding.Volume.Parent.Name}' with " +
$"WithComputeEnvironment.");
}
}
}

// Create a Kubernetes compute resource for the resource
var serviceResource = await environmentContext.CreateKubernetesResourceAsync(r, executionContext, cancellationToken).ConfigureAwait(false);
serviceResource.AddPrintSummaryStep();
Expand Down
Loading
Loading