[BUG]: A hard error on the final worker attempt discards the previous attempt's completed output

[justintime4tea]

Summary

execute_task retries a worker once when it produces output but never calls submit_result. The first attempt's raw output is stashed in last_raw_response and the loop continues:

// crates/aura/src/orchestration/orchestrator.rs:3320-3324
None => {
    tracing::info!("Worker attempt {} for task {} did not call submit_result. Retrying with correction.", attempt, task_id);
    last_raw_response = raw_response;
    continue;
}

If the second attempt then dies with a hard error, the loop breaks with last_error set:

// crates/aura/src/orchestration/orchestrator.rs:3328-3331
Err(e) => {
    last_error = Some(e);
    break; // Hard errors are not retried
}

and the post-loop branch keys solely off last_error, so last_raw_response is never read:

// crates/aura/src/orchestration/orchestrator.rs:3336-3352
if let Some(ref e) = last_error {
    self.persist_worker_execution(task_id, ..., Err(e.as_ref()), None, &base_worker_prompt).await;
    Err(format!("Worker failed task {} after {} attempts: {}", task_id, actual_attempts, e).into())
} else {
    // ... only this branch preserves last_raw_response
}

The task is reported as a total failure even though a complete attempt's worth of work exists in memory at that moment. The None if is_final_attempt branch at orchestrator.rs:3295 — which exists precisely to preserve unstructured output via the artifact flow — is unreachable once a hard error kills the final attempt.

Two things are lost, not one:

• The return value: downstream tasks and the coordinator see only the error string, so dependents are marked blocked and the plan collapses.
• The persistence record: the None => continue path does not call persist_worker_execution (compare the success path at 3274 and the final-attempt path at 3302), and the post-loop error path at 3338 persists the error. So task-N.attempt-1.response.txt is never written and the output is unrecoverable from the run directory too.

Expected: a hard error on the final attempt should fall back to the output the earlier attempt already produced, exactly as the final-attempt-no-submit_result path does.

Reproduction

Pre-requisites

• AURA 0.2.4 (05169982), orchestration enabled with memory_dir set

Steps

1. Run an orchestrated query where a worker does substantial tool work and then stops without calling submit_result (attempt 1 stashes its output and retries).
2. Cause attempt 2's first provider call to fail hard — e.g. exhaust the account's credit, revoke the API key, or point the worker at an invalid model.
3. [BUG] The task is reported as Worker failed task N after 2 attempts: <error>. Attempt 1's output appears nowhere in the response, and <memory_dir>/<session>/<run>/iteration-1/ contains no task-N.attempt-1.response.txt.

Relevant log output

Real run, 2026-08-24T23:23:17Z – 23:24:27Z. Attempt 1 ran 8 conversation turns and ~20 tool calls over 73 seconds:

2026-08-24T23:23:18.492317Z INFO aura::orchestration::persistence: Written tool output artifact (6387 chars) ...
2026-08-24T23:23:29.629147Z INFO aura::orchestration::persistence: Written tool output artifact (13422 chars) ...
2026-08-24T23:23:46.795803Z INFO aura::orchestration::persistence: Written tool output artifact (13491 chars) ...
2026-08-24T23:24:16.178407Z INFO aura::orchestration::persistence: Written tool output artifact (17232 chars) ...
   (~20 such lines; rig reported conversation depth reaching 8/15 — the worker stopped well
    short of its 15-turn budget, it simply never called submit_result)

2026-08-24T23:24:2x  INFO aura::orchestration::orchestrator: Worker attempt 1 for task 0 did not
  call submit_result. Retrying with correction.

2026-08-24T23:24:27.303827Z INFO rig::agent::prompt_request::streaming: Current conversation depth: 1/15
2026-08-24T23:24:27.531880Z ERROR aura::logging: error="CompletionError: ProviderError: SSE Error:
  Invalid status code 400 Bad Request ..."          <-- attempt 2 dies 228ms in

2026-08-24T23:24:27.532368Z  WARN aura::orchestration::orchestrator: Worker 'github_analyst' failed
  task 0 after 73121ms (agent_error): Worker failed task 0 after 2 attempts: ...

2026-08-24T23:24:27.532532Z  WARN aura::orchestration::orchestrator: No ready tasks but plan not
  finished — blocked tasks remaining after failure (dependency chain broken)
2026-08-24T23:24:27.535405Z  WARN aura::orchestration::orchestrator: Execution had failures:
  1 failed (1 agent_error), 1 blocked

73.1 seconds of billed generation and tool execution, reported as 0 of 3 tasks succeeded. Attempt 2 contributed 0.2s of that; every bit of the remaining 72.9s was discarded.

Additional Context

The individual tool output artifacts visible in the log above do survive on disk, but they are written by an unrelated path (ExecutionPersistence::write_tool_output_artifact, persistence.rs:624, invoked from the tool-interception flow at orchestrator.rs:2888). They are raw per-tool dumps with no synthesis. The worker's actual composed response — the thing that would have been usable — is what gets dropped.

This bug is what turns an unrecoverable provider failure into an expensive one. It compounds with the misclassification in the companion report (billing/quota errors categorized as AgentError): that report covers why the orchestrator kept calling a dead provider; this one covers why the work already completed was thrown away.

Suggested fix

Make the post-loop fallback prefer real output over the error. Rather than branching on last_error alone:

if let Some(ref e) = last_error {
    if !last_raw_response.is_empty() {
        // An earlier attempt produced output; ship it through the same
        // artifact flow the final-attempt path uses, and note the error.
        tracing::warn!(
            "Worker task {} hard-failed on attempt {} but attempt {} produced output; preserving it: {}",
            task_id, actual_attempts, actual_attempts - 1, e
        );
        self.persist_worker_execution(
            task_id, task_description, actual_attempts - 1, duration_ms,
            Ok(&last_raw_response), None, &base_worker_prompt,
        ).await;
        return Ok(TaskExecutionResult { result: last_raw_response, structured_output: None });
    }
    // ... existing error path
}

Separately, the None => continue path at 3320 should call persist_worker_execution with Ok(&raw_response) so each attempt's response lands in the run directory regardless of what happens next — the success and final-attempt paths already do this, and the gap is what makes the lost output unrecoverable post-mortem.

Worth deciding explicitly whether this fallback should apply to every hard error or only to non-ContextOverflow ones; shipping a partial result after a context blowup may be preferable to shipping nothing, but it is a behavior change that deserves a conscious call.

Searched Issues

• No similar issues found

Code of Conduct

• I agree to follow this project's Code of Conduct