Skip to content

Allow callers to cancel their request to cancel the script #1085

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Allow callers to cancel their request to cancel the script #1085

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d99ded1
Allow callers to cancel their request to cancel the script
LukeButters Apr 3, 2025
1e60c41
.
LukeButters Apr 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 13 additions & 7 deletions source/Octopus.Tentacle.Client/ITentacleClient.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,37 +34,43 @@ Task<ScriptExecutionResult> ExecuteScript(
CancellationToken scriptExecutionCancellationToken);

/// <summary>
/// Start the script.
///
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we delete this summary? Yeah, it's simple, but it's better than none.

/// </summary>
/// <param name="command"></param>
/// <param name="startScriptIsBeingReAttempted"></param>
/// <param name="logger"></param>
/// <param name="requestCancellationToken">Cancels the inflight request</param>
/// <returns>The result, which includes the CommandContext for the next command</returns>
Task<ScriptOperationExecutionResult> StartScript(ExecuteScriptCommand command,
StartScriptIsBeingReAttempted startScriptIsBeingReAttempted,
ITentacleClientTaskLog logger,
CancellationToken scriptExecutionCancellationToken);
CancellationToken requestCancellationToken);


/// <summary>
/// Get the status.
/// </summary>
/// <param name="commandContext">The CommandContext from the previous command</param>
/// <param name="logger"></param>
/// <param name="scriptExecutionCancellationToken"></param>
/// <param name="requestCancellationToken">Cancels the inflight request</param>
/// <returns>The result, which includes the CommandContext for the next command</returns>
Task<ScriptOperationExecutionResult> GetStatus(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken scriptExecutionCancellationToken);
Task<ScriptOperationExecutionResult> GetStatus(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken requestCancellationToken);

/// <summary>
/// Cancel the script.
/// </summary>
/// <param name="commandContext">The CommandContext from the previous command</param>
/// <param name="logger"></param>
/// <param name="requestCancellationToken">Cancels the inflight request</param>
/// <returns>The result, which includes the CommandContext for the next command</returns>
Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext, ITentacleClientTaskLog logger);
Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken requestCancellationToken);

/// <summary>
/// Complete the script.
/// </summary>
/// <param name="commandContext">The CommandContext from the previous command</param>
/// <param name="logger"></param>
/// <param name="scriptExecutionCancellationToken"></param>
Task<ScriptStatus?> CompleteScript(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken scriptExecutionCancellationToken);
/// <param name="requestCancellationToken">Cancels the inflight request</param>
Task<ScriptStatus?> CompleteScript(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken requestCancellationToken);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering, what's the consequence of killing a request to cancel?

I would have thought this would result in the tentacle potentially running the script after we have told the user that we are cancelled, right?

Is this going to be an issue? I.e., is the trade off of shutting down Octopus Server quicker worth it?

}
}
4 changes: 2 additions & 2 deletions source/Octopus.Tentacle.Client/ScriptExecutor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,12 +60,12 @@ public async Task<ScriptOperationExecutionResult> GetStatus(CommandContext comma
return await scriptExecutor.GetStatus(commandContext, cancellationToken);
}

public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext)
public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext, CancellationToken cancellationToken)
{
var scriptExecutorFactory = CreateScriptExecutorFactory();
var scriptExecutor = scriptExecutorFactory.CreateScriptExecutor(commandContext.ScripServiceVersionUsed);

return await scriptExecutor.CancelScript(commandContext);
return await scriptExecutor.CancelScript(commandContext, cancellationToken);
}

public async Task<ScriptStatus?> CompleteScript(CommandContext commandContext, CancellationToken cancellationToken)
Expand Down
2 changes: 1 addition & 1 deletion source/Octopus.Tentacle.Client/Scripts/IScriptExecutor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Task<ScriptOperationExecutionResult> StartScript(ExecuteScriptCommand command,
/// </summary>
/// <param name="commandContext">The CommandContext from the previous command</param>
/// <returns>The result, which includes the CommandContext for the next command</returns>
Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext);
Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext, CancellationToken cancellationToken);

/// <summary>
/// Complete the script.
Expand Down
5 changes: 2 additions & 3 deletions source/Octopus.Tentacle.Client/Scripts/KubernetesScriptServiceV1Executor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ async Task<KubernetesScriptStatusResponseV1> GetStatusAction(CancellationToken c
return Map(kubernetesScriptStatusResponseV1);
}

public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext)
public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext, CancellationToken cancellationToken)
{
async Task<KubernetesScriptStatusResponseV1> CancelScriptAction(CancellationToken ct)
{
Expand All @@ -173,8 +173,7 @@ async Task<KubernetesScriptStatusResponseV1> CancelScriptAction(CancellationToke
CancelScriptAction,
logger,
clientOperationMetricsBuilder,
// We don't want to cancel this operation as it is responsible for stopping the script executing on the Tentacle
CancellationToken.None).ConfigureAwait(false);
cancellationToken).ConfigureAwait(false);
return Map(kubernetesScriptStatusResponseV1);
}

Expand Down
3 changes: 2 additions & 1 deletion source/Octopus.Tentacle.Client/Scripts/ObservingScriptOrchestrator.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,8 @@ async Task<ScriptOperationExecutionResult> ObserveUntilComplete(
{
if (scriptExecutionCancellationToken.IsCancellationRequested)
{
lastResult = await scriptExecutor.CancelScript(lastResult.ContextForNextCommand).ConfigureAwait(false);
// We don't want to cancel this operation as it is responsible for stopping the script executing on the Tentacle
lastResult = await scriptExecutor.CancelScript(lastResult.ContextForNextCommand, CancellationToken.None).ConfigureAwait(false);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CancellationToken.None is moved up here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason for this PR is so that we can cancel the cancellation request, so that the cancellation request does not block Octopus Server from shutting down.

But this made me wonder, why is this not an issue for the non-resilient pipeline? Surely it would get told to cancel a running deployment when it shuts down too, right? I would have thought that would also block.

}
else
{
Expand Down
4 changes: 2 additions & 2 deletions source/Octopus.Tentacle.Client/Scripts/ScriptServiceV1Executor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ public async Task<ScriptOperationExecutionResult> GetStatus(CommandContext comma
return Map(scriptStatusResponseV1);
}

public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext)
public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext, CancellationToken cancellationToken)
{
var response = await rpcCallExecutor.ExecuteWithNoRetries(
RpcCall.Create<IScriptService>(nameof(IScriptService.CancelScript)),
Expand All @@ -119,7 +119,7 @@ public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext co
},
logger,
clientOperationMetricsBuilder,
CancellationToken.None).ConfigureAwait(false);
cancellationToken).ConfigureAwait(false);

return Map(response);
}
Expand Down
19 changes: 9 additions & 10 deletions source/Octopus.Tentacle.Client/Scripts/ScriptServiceV2Executor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ static ScriptOperationExecutionResult Map(ScriptStatusResponseV2 scriptStatusRes

public async Task<ScriptOperationExecutionResult> StartScript(ExecuteScriptCommand executeScriptCommand,
StartScriptIsBeingReAttempted startScriptIsBeingReAttempted,
CancellationToken scriptExecutionCancellationToken)
CancellationToken requestCancellationToken)
{
var command = Map(executeScriptCommand);
ScriptStatusResponseV2 scriptStatusResponse;
Expand Down Expand Up @@ -98,11 +98,11 @@ void OnErrorAction(Exception ex)
OnErrorAction,
logger,
clientOperationMetricsBuilder,
scriptExecutionCancellationToken).ConfigureAwait(false);
requestCancellationToken).ConfigureAwait(false);

return Map(scriptStatusResponse);
}
catch (Exception ex) when (scriptExecutionCancellationToken.IsCancellationRequested)
catch (Exception ex) when (requestCancellationToken.IsCancellationRequested)
{
// If the call to StartScript is in-flight (being transferred to the Service) or we are retrying the StartScript call
// then we do not know if the script has been started or not on Tentacle so need to call CancelScript and CompleteScript
Expand All @@ -126,7 +126,7 @@ void OnErrorAction(Exception ex)



public async Task<ScriptOperationExecutionResult> GetStatus(CommandContext commandContext, CancellationToken scriptExecutionCancellationToken)
public async Task<ScriptOperationExecutionResult> GetStatus(CommandContext commandContext, CancellationToken requestCancellationToken)
{
async Task<ScriptStatusResponseV2> GetStatusAction(CancellationToken ct)
{
Expand All @@ -142,11 +142,11 @@ async Task<ScriptStatusResponseV2> GetStatusAction(CancellationToken ct)
GetStatusAction,
logger,
clientOperationMetricsBuilder,
scriptExecutionCancellationToken).ConfigureAwait(false);
requestCancellationToken).ConfigureAwait(false);
return Map(scriptStatusResponseV2);
}

public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext)
public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext, CancellationToken requestCancellationToken)
{
async Task<ScriptStatusResponseV2> CancelScriptAction(CancellationToken ct)
{
Expand All @@ -166,12 +166,11 @@ async Task<ScriptStatusResponseV2> CancelScriptAction(CancellationToken ct)
CancelScriptAction,
logger,
clientOperationMetricsBuilder,
// We don't want to cancel this operation as it is responsible for stopping the script executing on the Tentacle
CancellationToken.None).ConfigureAwait(false);
requestCancellationToken).ConfigureAwait(false);
return Map(scriptStatusResponseV2);
}

public async Task<ScriptStatus?> CompleteScript(CommandContext lastStatusResponse, CancellationToken scriptExecutionCancellationToken)
public async Task<ScriptStatus?> CompleteScript(CommandContext lastStatusResponse, CancellationToken requestCancellationToken)
{
try
{
Expand All @@ -180,7 +179,7 @@ async Task<ScriptStatusResponseV2> CancelScriptAction(CancellationToken ct)

using var completeScriptCancellationTokenSource = new CancellationTokenSource();

await using var _ = scriptExecutionCancellationToken.Register(() => completeScriptCancellationTokenSource.CancelAfter(onCancellationAbandonCompleteScriptAfter));
await using var _ = requestCancellationToken.Register(() => completeScriptCancellationTokenSource.CancelAfter(onCancellationAbandonCompleteScriptAfter));

await rpcCallExecutor.ExecuteWithNoRetries(
RpcCall.Create<IScriptServiceV2>(nameof(IScriptServiceV2.CompleteScript)),
Expand Down
16 changes: 8 additions & 8 deletions source/Octopus.Tentacle.Client/TentacleClient.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ public async Task<ScriptOperationExecutionResult> StartScript(
ExecuteScriptCommand command,
StartScriptIsBeingReAttempted startScriptIsBeingReAttempted,
ITentacleClientTaskLog logger,
CancellationToken scriptExecutionCancellationToken)
CancellationToken requestCancellationToken)
{
var scriptExecutor = new ScriptExecutor(
allClients,
Expand All @@ -203,10 +203,10 @@ public async Task<ScriptOperationExecutionResult> StartScript(
clientOptions,
OnCancellationAbandonCompleteScriptAfter);

return await scriptExecutor.StartScript(command, startScriptIsBeingReAttempted, scriptExecutionCancellationToken);
return await scriptExecutor.StartScript(command, startScriptIsBeingReAttempted, requestCancellationToken);
}

public async Task<ScriptOperationExecutionResult> GetStatus(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken scriptExecutionCancellationToken)
public async Task<ScriptOperationExecutionResult> GetStatus(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken requestCancellationToken)
{
var scriptExecutor = new ScriptExecutor(
allClients,
Expand All @@ -217,10 +217,10 @@ public async Task<ScriptOperationExecutionResult> GetStatus(CommandContext comma
clientOptions,
OnCancellationAbandonCompleteScriptAfter);

return await scriptExecutor.GetStatus(commandContext, scriptExecutionCancellationToken);
return await scriptExecutor.GetStatus(commandContext, requestCancellationToken);
}

public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext, ITentacleClientTaskLog logger)
public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken requestCancellationToken)
{
var scriptExecutor = new ScriptExecutor(
allClients,
Expand All @@ -231,10 +231,10 @@ public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext co
clientOptions,
OnCancellationAbandonCompleteScriptAfter);

return await scriptExecutor.CancelScript(commandContext);
return await scriptExecutor.CancelScript(commandContext, requestCancellationToken);
}

public async Task<ScriptStatus?> CompleteScript(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken scriptExecutionCancellationToken)
public async Task<ScriptStatus?> CompleteScript(CommandContext commandContext, ITentacleClientTaskLog logger, CancellationToken requestCancellationToken)
{
var scriptExecutor = new ScriptExecutor(
allClients,
Expand All @@ -245,7 +245,7 @@ public async Task<ScriptOperationExecutionResult> CancelScript(CommandContext co
clientOptions,
OnCancellationAbandonCompleteScriptAfter);

return await scriptExecutor.CompleteScript(commandContext, scriptExecutionCancellationToken);
return await scriptExecutor.CompleteScript(commandContext, requestCancellationToken);
}

public void Dispose()
Expand Down