Skip to content

feat(controllers): Enable multiple controllers with different label selectors for the same entity type #911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 89 additions & 0 deletions docs/docs/operator/building-blocks/controllers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,95 @@ public class V1DemoEntityController(
}
```

## Label Selectors

Label selectors allow you to filter which resources your controller watches based on Kubernetes labels. This is useful when you want a controller to only process resources with specific labels.

### Basic Label Selector

By default, controllers use the `DefaultEntityLabelSelector<TEntity>` which doesn't apply any filtering (watches all resources of the specified type).

### Custom Label Selector

To create a custom label selector, implement the `IEntityLabelSelector<TEntity, TSelf>` interface:

```csharp
public class V1DemoEntityLabelSelector : IEntityLabelSelector<V1DemoEntity, V1DemoEntityLabelSelector>
{
public ValueTask<string?> GetLabelSelectorAsync(CancellationToken cancellationToken)
{
// Return a Kubernetes label selector string
return ValueTask.FromResult<string?>("app=demo,environment=production");
}
}
```

### Implementing a Controller with Label Selector

You can implement a controller that directly uses a custom label selector by implementing the `IEntityController<TEntity, TLabelSelector>` interface:

```csharp
public class V1DemoEntityController : IEntityController<V1DemoEntity, V1DemoEntityLabelSelector>
{
private readonly ILogger<V1DemoEntityController> _logger;
private readonly IKubernetesClient _client;

public V1DemoEntityController(
ILogger<V1DemoEntityController> logger,
IKubernetesClient client)
{
_logger = logger;
_client = client;
}

public async Task ReconcileAsync(V1DemoEntity entity, CancellationToken token)
{
_logger.LogInformation("Reconciling entity {Entity} with custom label selector.", entity);
// Implement your reconciliation logic here
}

public async Task DeletedAsync(V1DemoEntity entity, CancellationToken token)
{
_logger.LogInformation("Deleting entity {Entity} with custom label selector.", entity);
// Implement your cleanup logic here
}
}
```

### Using Label Selectors with Controllers

There are multiple ways to register a controller with a label selector:

#### New Style (Recommended)

For controllers that implement `IEntityController<TEntity, TLabelSelector>` as shown above:

```csharp
// In your Startup.cs or Program.cs
services.AddKubernetesOperator()
.AddController<V1DemoEntityController, V1DemoEntity, V1DemoEntityLabelSelector>();
```

#### Backward Compatibility

By default, controllers that implement the simpler `IEntityController<TEntity>` interface use `DefaultEntityLabelSelector<TEntity>` behind the scenes, which doesn't apply any filtering.

However, you can still use a custom label selector with these controllers:

```csharp
// Controller implementation using the simpler interface
public class V1DemoEntityController : IEntityController<V1DemoEntity>
{
// Implementation details...
}

// In your Startup.cs or Program.cs
services.AddKubernetesOperator()
.AddController<V1DemoEntityController, V1DemoEntity, V1DemoEntityLabelSelector>(null);
```

The null parameter is used for method overload disambiguation. This approach allows you to use a custom label selector with controllers that implement the simpler interface, providing backward compatibility.

## Resource Watcher

When you create a controller, KubeOps automatically creates a resource watcher (informer) for your entity type. This watcher:
Expand Down
18 changes: 17 additions & 1 deletion src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,24 @@
/// <typeparam name="TLabelSelector">Label Selector type.</typeparam>
/// <returns>The builder for chaining.</returns>
IOperatorBuilder AddController<TImplementation, TEntity, TLabelSelector>()
where TImplementation : class, IEntityController<TEntity, TLabelSelector>
where TEntity : IKubernetesObject<V1ObjectMeta>
where TLabelSelector : class, IEntityLabelSelector<TEntity, TLabelSelector>;

/// <summary>
/// Add a controller implementation for a specific entity to the operator with backward compatibility.
/// This overload allows controllers that implement IEntityController&lt;TEntity&gt; to be registered
/// with a specific label selector for backward compatibility.
/// </summary>
/// <typeparam name="TImplementation">Implementation type of the controller.</typeparam>
/// <typeparam name="TEntity">Entity type.</typeparam>
/// <typeparam name="TLabelSelector">Label Selector type.</typeparam>
/// <param name="_">Unused parameter for method overload disambiguation.</param>
/// <returns>The builder for chaining.</returns>
IOperatorBuilder AddController<TImplementation, TEntity, TLabelSelector>(TImplementation? _ = null)

Check warning on line 57 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Testing

This method signature overlaps the one defined on line 42, the default parameter value can't be used.

Check warning on line 57 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Testing

This method signature overlaps the one defined on line 42, the default parameter value can't be used.

Check failure on line 57 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Analyze

This method signature overlaps the one defined on line 42, the default parameter value can't be used. (https://rules.sonarsource.com/csharp/RSPEC-3427)

Check failure on line 57 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Analyze

This method signature overlaps the one defined on line 42, the default parameter value can't be used. (https://rules.sonarsource.com/csharp/RSPEC-3427)

Check failure on line 57 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Analyze

This method signature overlaps the one defined on line 42, the default parameter value can't be used. (https://rules.sonarsource.com/csharp/RSPEC-3427)

Check failure on line 57 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Analyze

This method signature overlaps the one defined on line 42, the default parameter value can't be used. (https://rules.sonarsource.com/csharp/RSPEC-3427)
where TImplementation : class, IEntityController<TEntity>
where TEntity : IKubernetesObject<V1ObjectMeta>
where TLabelSelector : class, IEntityLabelSelector<TEntity>;
where TLabelSelector : class, IEntityLabelSelector<TEntity, TLabelSelector>;

/// <summary>
/// Add a finalizer implementation for a specific entity.
Expand Down Expand Up @@ -75,4 +90,5 @@
/// </param>
/// <returns>The builder for chaining.</returns>
IOperatorBuilder AddCrdInstaller(Action<CrdInstallerSettings>? configure = null);
}

Check warning on line 93 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Testing

File is required to end with a single newline character

Check warning on line 93 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Testing

File is required to end with a single newline character

Check failure on line 93 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Analyze

Check failure on line 93 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Analyze

Check failure on line 93 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Analyze

Check failure on line 93 in src/KubeOps.Abstractions/Builder/IOperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Analyze


43 changes: 40 additions & 3 deletions src/KubeOps.Abstractions/Controller/IEntityController{TEntity}.cs
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
using k8s;
using k8s.Models;
using KubeOps.Abstractions.Entities;

namespace KubeOps.Abstractions.Controller;

/// <summary>
/// Generic entity controller. The controller manages the reconcile loop
/// for a given entity type.
/// for a given entity type with a selector.
/// </summary>
/// <typeparam name="TEntity">The type of the Kubernetes entity.</typeparam>
/// <typeparam name="TSelector">The type of the label selector for the entity.</typeparam>
/// <example>
/// Simple example controller that just logs the entity.
/// <code>
/// public class V1TestEntityController : IEntityController&lt;V1TestEntity&gt;
/// public class V1TestEntityController : IEntityController&lt;V1TestEntity, DefaultEntityLabelSelector&lt;V1TestEntity&gt;&gt;
/// {
/// private readonly ILogger&lt;V1TestEntityController&gt; _logger;
///
Expand All @@ -33,8 +35,9 @@ namespace KubeOps.Abstractions.Controller;
/// }
/// </code>
/// </example>
public interface IEntityController<in TEntity>
public interface IEntityController<TEntity, in TSelector>
where TEntity : IKubernetesObject<V1ObjectMeta>
where TSelector : class, IEntityLabelSelector<TEntity, TSelector>
{
/// <summary>
/// Called for `added` and `modified` events from the watcher.
Expand All @@ -54,3 +57,37 @@ public interface IEntityController<in TEntity>
/// </returns>
Task DeletedAsync(TEntity entity, CancellationToken cancellationToken);
}

/// <summary>
/// Generic entity controller. The controller manages the reconcile loop
/// for a given entity type. This is a compatibility interface that inherits
/// from the new two-parameter interface with a default selector.
/// </summary>
/// <typeparam name="TEntity">The type of the Kubernetes entity.</typeparam>
/// <example>
/// Simple example controller that just logs the entity.
/// <code>
/// public class V1TestEntityController : IEntityController&lt;V1TestEntity&gt;
/// {
/// private readonly ILogger&lt;V1TestEntityController&gt; _logger;
///
/// public V1TestEntityController(
/// ILogger&lt;V1TestEntityController&gt; logger)
/// {
/// _logger = logger;
/// }
///
/// public async Task ReconcileAsync(V1TestEntity entity, CancellationToken token)
/// {
/// _logger.LogInformation("Reconciling entity {Entity}.", entity);
/// }
///
/// public async Task DeletedAsync(V1TestEntity entity, CancellationToken token)
/// {
/// _logger.LogInformation("Deleting entity {Entity}.", entity);
/// }
/// }
/// </code>
/// </example>
public interface IEntityController<TEntity> : IEntityController<TEntity, DefaultEntityLabelSelector<TEntity>>
where TEntity : IKubernetesObject<V1ObjectMeta>;
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

namespace KubeOps.Abstractions.Entities;

public class DefaultEntityLabelSelector<TEntity> : IEntityLabelSelector<TEntity>
public class DefaultEntityLabelSelector<TEntity> : IEntityLabelSelector<TEntity, DefaultEntityLabelSelector<TEntity>>
where TEntity : IKubernetesObject<V1ObjectMeta>
{
public ValueTask<string?> GetLabelSelectorAsync(CancellationToken cancellationToken) => ValueTask.FromResult<string?>(null);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,13 @@ namespace KubeOps.Abstractions.Entities;
// An alternative would be to use a KeyedSingleton when registering this however that's only valid from .NET 8 and above.
// Other methods are far less elegant
#pragma warning disable S2326
public interface IEntityLabelSelector<TEntity>
public interface IEntityLabelSelector<TEntity, TSelf>
where TEntity : IKubernetesObject<V1ObjectMeta>
{
ValueTask<string?> GetLabelSelectorAsync(CancellationToken cancellationToken);
}

public interface IEntityLabelSelector<TEntity> : IEntityLabelSelector<TEntity, DefaultEntityLabelSelector<TEntity>>
where TEntity : IKubernetesObject<V1ObjectMeta>;

#pragma warning restore S2326
71 changes: 46 additions & 25 deletions src/KubeOps.Operator/Builder/OperatorBuilder.cs
Original file line number Diff line number Diff line change
Expand Up @@ -38,50 +38,66 @@

public IOperatorBuilder AddController<TImplementation, TEntity>()
where TImplementation : class, IEntityController<TEntity>
where TEntity : IKubernetesObject<V1ObjectMeta> =>
AddController<TImplementation, TEntity, DefaultEntityLabelSelector<TEntity>>();

public IOperatorBuilder AddController<TImplementation, TEntity, TLabelSelector>()
where TImplementation : class, IEntityController<TEntity, TLabelSelector>
where TEntity : IKubernetesObject<V1ObjectMeta>
where TLabelSelector : class, IEntityLabelSelector<TEntity, TLabelSelector>
{
Services.AddHostedService<EntityRequeueBackgroundService<TEntity>>();
Services.TryAddScoped<IEntityController<TEntity>, TImplementation>();
Services.AddHostedService<EntityRequeueBackgroundService<TEntity, TLabelSelector>>();

// Register the implementation for the two-parameter interface
Services.TryAddScoped<IEntityController<TEntity, TLabelSelector>, TImplementation>();
Services.TryAddSingleton(new TimedEntityQueue<TEntity>());
Services.TryAddTransient<IEntityRequeueFactory, KubeOpsEntityRequeueFactory>();
Services.TryAddTransient<EntityRequeue<TEntity>>(services =>
services.GetRequiredService<IEntityRequeueFactory>().Create<TEntity>());
Services.TryAddTransient<EntityRequeue<TEntity>>(
services => services.GetRequiredService<IEntityRequeueFactory>().Create<TEntity>()
);

Check warning on line 57 in src/KubeOps.Operator/Builder/OperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Testing

Closing parenthesis should be on line of last parameter
Services.TryAddSingleton<IEntityLabelSelector<TEntity, TLabelSelector>, TLabelSelector>();

if (_settings.EnableLeaderElection)
{
Services.AddHostedService<LeaderAwareResourceWatcher<TEntity>>();
Services.AddHostedService<LeaderAwareResourceWatcher<TEntity, TLabelSelector>>();
}
else
{
Services.AddHostedService<ResourceWatcher<TEntity>>();
Services.AddHostedService<ResourceWatcher<TEntity, TLabelSelector>>();
}

return this;
}

public IOperatorBuilder AddController<TImplementation, TEntity, TLabelSelector>()
// Overload for controllers that implement IEntityController<TEntity> but are being registered with a specific label selector
// This allows backward compatibility for users who call AddController<Controller, Entity, LabelSelector>()
// even when their controller only implements IEntityController<Entity>
public IOperatorBuilder AddController<TImplementation, TEntity, TLabelSelector>(TImplementation? _ = null)
where TImplementation : class, IEntityController<TEntity>
where TEntity : IKubernetesObject<V1ObjectMeta>
where TLabelSelector : class, IEntityLabelSelector<TEntity>
where TLabelSelector : class, IEntityLabelSelector<TEntity, TLabelSelector>
{
Services.AddHostedService<EntityRequeueBackgroundService<TEntity>>();
Services.TryAddScoped<IEntityController<TEntity>, TImplementation>();
Services.TryAddSingleton(new TimedEntityQueue<TEntity>());
Services.TryAddTransient<IEntityRequeueFactory, KubeOpsEntityRequeueFactory>();
Services.TryAddTransient<EntityRequeue<TEntity>>(services =>
services.GetRequiredService<IEntityRequeueFactory>().Create<TEntity>());
Services.TryAddSingleton<IEntityLabelSelector<TEntity>, TLabelSelector>();

if (_settings.EnableLeaderElection)
// Check if TImplementation actually implements IEntityController<TEntity, TLabelSelector>
if (typeof(IEntityController<TEntity, TLabelSelector>).IsAssignableFrom(typeof(TImplementation)))
{
Services.AddHostedService<LeaderAwareResourceWatcher<TEntity>>();
// If it does, call the main method
return AddController<TImplementation, TEntity, TLabelSelector>();
}
else

// If the controller only implements IEntityController<TEntity>, we can only register it with DefaultEntityLabelSelector
// We cannot support arbitrary label selectors with controllers that don't implement the two-parameter interface
if (typeof(TLabelSelector) != typeof(DefaultEntityLabelSelector<TEntity>))
{
Services.AddHostedService<ResourceWatcher<TEntity>>();
throw new InvalidOperationException(
$"Controller {typeof(TImplementation).Name} only implements IEntityController<{typeof(TEntity).Name}> "
+ $"and cannot be used with label selector {typeof(TLabelSelector).Name}. "
+ $"Either implement IEntityController<{typeof(TEntity).Name}, {typeof(TLabelSelector).Name}> "
+ $"or use AddController<{typeof(TImplementation).Name}, {typeof(TEntity).Name}>() instead."
);

Check warning on line 96 in src/KubeOps.Operator/Builder/OperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Testing

Closing parenthesis should be on line of last parameter
}

return this;
// If TLabelSelector is DefaultEntityLabelSelector<TEntity>, delegate to the two-parameter method
return AddController<TImplementation, TEntity>();
}

public IOperatorBuilder AddFinalizer<TImplementation, TEntity>(string identifier)
Expand All @@ -90,9 +106,12 @@
{
Services.TryAddKeyedTransient<IEntityFinalizer<TEntity>, TImplementation>(identifier);
Services.TryAddTransient<IEventFinalizerAttacherFactory, KubeOpsEventFinalizerAttacherFactory>();
Services.TryAddTransient<EntityFinalizerAttacher<TImplementation, TEntity>>(services =>
services.GetRequiredService<IEventFinalizerAttacherFactory>()
.Create<TImplementation, TEntity>(identifier));
Services.TryAddTransient<EntityFinalizerAttacher<TImplementation, TEntity>>(
services =>
services
.GetRequiredService<IEventFinalizerAttacherFactory>()
.Create<TImplementation, TEntity>(identifier)
);

Check warning on line 114 in src/KubeOps.Operator/Builder/OperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Testing

Closing parenthesis should be on line of last parameter

return this;
}
Expand Down Expand Up @@ -131,7 +150,9 @@
Services.TryAddTransient<EventPublisher>(services =>
services.GetRequiredService<IEventPublisherFactory>().Create());

Services.AddSingleton(typeof(IEntityLabelSelector<>), typeof(DefaultEntityLabelSelector<>));
// Register default entity label selector for all entities
// Note: We cannot register the open generic types directly due to arity mismatch
// The registration happens in AddController when specific types are needed

Check warning on line 155 in src/KubeOps.Operator/Builder/OperatorBuilder.cs

View workflow job for this annotation

GitHub Actions / Testing

Single-line comments should not be followed by blank line

if (_settings.EnableLeaderElection)
{
Expand Down
8 changes: 5 additions & 3 deletions src/KubeOps.Operator/Queue/EntityRequeueBackgroundService.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
using k8s.Models;

using KubeOps.Abstractions.Controller;
using KubeOps.Abstractions.Entities;
using KubeOps.KubernetesClient;

using Microsoft.Extensions.DependencyInjection;
Expand All @@ -10,12 +11,13 @@

namespace KubeOps.Operator.Queue;

internal sealed class EntityRequeueBackgroundService<TEntity>(
internal sealed class EntityRequeueBackgroundService<TEntity, TSelector>(
IKubernetesClient client,
TimedEntityQueue<TEntity> queue,
IServiceProvider provider,
ILogger<EntityRequeueBackgroundService<TEntity>> logger) : IHostedService, IDisposable, IAsyncDisposable
ILogger<EntityRequeueBackgroundService<TEntity, TSelector>> logger) : IHostedService, IDisposable, IAsyncDisposable
where TEntity : IKubernetesObject<V1ObjectMeta>
where TSelector : class, IEntityLabelSelector<TEntity, TSelector>
{
private readonly CancellationTokenSource _cts = new();
private bool _disposed;
Expand Down Expand Up @@ -119,7 +121,7 @@ private async Task ReconcileSingleAsync(TEntity queued, CancellationToken cancel
}

await using var scope = provider.CreateAsyncScope();
var controller = scope.ServiceProvider.GetRequiredService<IEntityController<TEntity>>();
var controller = scope.ServiceProvider.GetRequiredService<IEntityController<TEntity, TSelector>>();
await controller.ReconcileAsync(entity, cancellationToken);
}
}
Loading
Loading