OneDrive prune fails when server-honored Retry-After consumes the 12-retry budget

[guipace]

Summary

A scheduled prune against a OneDrive backend reliably fails when OneDrive
returns a burst of 503s with long Retry-After headers. Duplicacy honors
the Retry-After values correctly, but each honored sleep counts against the
hardcoded 12-attempt budget in OneDriveClient.call, so the operation aborts
with Maximum number of retries reached even though OneDrive is explicitly
telling the client when to come back.

The chunks for the snapshot revision had already been deleted by the time
this happens; only the final snapshot file delete fails. The repository is
left with an orphaned snapshot until the next successful prune.

Environment

• Duplicacy CLI bundled with saspus/duplicacy-web (Docker)
• Storage backend: OneDrive (personal)
• Operation: scheduled prune (no -exclusive)

Log excerpt

... thousands of successful CHUNK_DELETE lines ...
05:05:36 INFO ONEDRIVE_RETRY Response code: 503; retry after 611 ms
05:05:37 INFO ONEDRIVE_RETRY Response code: 503; retry after 1532 ms
05:05:39 INFO ONEDRIVE_RETRY Response code: 503; retry after 2070 ms
05:05:41 INFO ONEDRIVE_RETRY Response code: 503; retry after 4299 ms
05:05:45 INFO ONEDRIVE_RETRY Response code: 503; retry after 13317 ms
05:05:59 INFO ONEDRIVE_RETRY Response code: 503; retry after 23299 ms
05:06:22 INFO ONEDRIVE_RETRY Response code: 503; retry after 59494 ms
05:07:22 INFO ONEDRIVE_RETRY Response code: 503; retry after 111185 ms
05:09:13 INFO ONEDRIVE_RETRY Response code: 503; retry after 244996 ms
05:13:18 INFO ONEDRIVE_RETRY Response code: 503; retry after 254199 ms
05:17:32 INFO ONEDRIVE_RETRY Response code: 503; retry after 142863 ms
05:19:56 INFO ONEDRIVE_RETRY Response code: 503; retry after 178443 ms
05:22:54 ERROR SNAPSHOT_DELETE Failed to delete the snapshot AppData
         at revision 87: Maximum number of retries reached

Twelve retries, all with Retry-After from the server, then abort.

Root cause

In src/duplicacy_oneclient.go (the call method), the retry loop is:

for i := 0; i < 12; i++ {
    ...
    } else if response.StatusCode > 401 && response.StatusCode != 404 {
        delay := int((rand.Float32()*0.5 + 0.5) * 1000.0 * float32(backoff))
        if backoffList, found := response.Header["Retry-After"]; found && len(backoffList) > 0 {
            retryAfter, _ := strconv.Atoi(backoffList[0])
            if retryAfter*1000 > delay {
                delay = retryAfter * 1000
            }
        }
        LOG_INFO("ONEDRIVE_RETRY", "Response code: %d; retry after %d ms", ...)
        time.Sleep(time.Duration(delay) * time.Millisecond)
        backoff *= 2
        ...
        continue
    }
}
return nil, 0, fmt.Errorf("Maximum number of retries reached")

The 12-attempt cap was reasonable for transient errors, but it conflates two
very different cases:

1. The server returned an error and we're guessing when to retry → counting
   against a budget makes sense.
2. The server explicitly said "come back in N seconds" via Retry-After →
   this is a cooperative throttle, not a failure. Counting it against the
   budget effectively punishes the client for doing what Microsoft's
   throttling guidance
   tells it to do.

During a heavy prune, OneDrive can issue a sustained throttle window long
enough to exhaust 12 retries even when each retry behaves correctly.

Proposed fixes (any of these would help, in order of smallest scope)

1. Don't count server-honored Retry-After waits against the retry
   budget. When the response carries a Retry-After header, sleep but do
   not increment i (or use a separate, larger counter for this branch).
   ~10-line change, fully contained in oneclient.go.

2. Make max retries configurable — e.g. a -onedrive-max-retries flag
   or DUPLICACY_ONEDRIVE_MAX_RETRIES env var, defaulting to 12. Lets users
   on flaky/throttled OneDrive accounts opt into longer patience without
   patching the binary.

3. Make prune resilient to a single-object delete failure. Today, one
   stuck SNAPSHOT_DELETE aborts the whole prune. Logging the failure,
   continuing with the rest of the deletion plan, and surfacing a summary of
   skipped objects at the end would let prune make forward progress; the
   orphaned snapshot would simply be picked up by the next run.

(1) is the cheapest and addresses the reported failure directly. (2) is a
nice escape hatch. (3) is the most useful long-term and is also relevant to
other backends, not just OneDrive.

[gilbertchen]

This issue has been mentioned on Duplicacy Forum. There might be relevant details there:

https://forum.duplicacy.com/t/onedrive-retry-response-code-503/10994/2