Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver API client interprets JSON syntax errors as 503 #8903

Closed
jcsp opened this issue Sep 3, 2024 · 0 comments · Fixed by #9052
Closed

pageserver API client interprets JSON syntax errors as 503 #8903

jcsp opened this issue Sep 3, 2024 · 0 comments · Fixed by #9052
Assignees
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug t/on_call_followup triaged bugs that were already triaged

Comments

@jcsp
Copy link
Collaborator

jcsp commented Sep 3, 2024

        mgmt_api::Error::ReceiveErrorBody(str) => {
            // Presume errors receiving body are connectivity/availability issues
            ApiError::ResourceUnavailable(
                format!("{node} error receiving error body: {str}").into(),
            )
        }

We should interpret truncated responses as 503 (dropped connection), but if we get a full body and it can't be parsed, then we should use a more clearly fatal error code & message.

This caused some confusion when debugging a situation where there was a version incompatibility between storage controller and pageserver.

@jcsp jcsp added t/bug Issue Type: Bug c/storage/pageserver Component: storage: pageserver t/on_call_followup labels Sep 3, 2024
@skyzh skyzh self-assigned this Sep 3, 2024
@jcsp jcsp added the triaged bugs that were already triaged label Sep 5, 2024
skyzh added a commit that referenced this issue Sep 20, 2024
close #8903

In #8903 we observed JSON
decoding error to have the following error message in the log:

```
Error processing HTTP request: Resource temporarily unavailable: 3956 (pageserver-6.ap-southeast-1.aws.neon.tech) error receiving body: error decoding response body
```

This is hard to understand. In this patch, we make the error message
more reasonable.

## Summary of changes

* receive body error is now an internal server error, passthrough the
`reqwest::Error` (only decoding error) as `anyhow::Error`.
* instead of formatting the error using `to_string`, we use the
alternative `anyhow::Error` formatting, so that it prints out the cause
of the error (i.e., what exactly cannot serde decode).

I would expect seeing something like `error receiving body: error
decoding response body: XXX field not found` after this patch, though I
didn't set up a testing environment to observe the exact behavior.

---------

Signed-off-by: Alex Chi Z <[email protected]>
davidgomes pushed a commit that referenced this issue Sep 21, 2024
close #8903

In #8903 we observed JSON
decoding error to have the following error message in the log:

```
Error processing HTTP request: Resource temporarily unavailable: 3956 (pageserver-6.ap-southeast-1.aws.neon.tech) error receiving body: error decoding response body
```

This is hard to understand. In this patch, we make the error message
more reasonable.

## Summary of changes

* receive body error is now an internal server error, passthrough the
`reqwest::Error` (only decoding error) as `anyhow::Error`.
* instead of formatting the error using `to_string`, we use the
alternative `anyhow::Error` formatting, so that it prints out the cause
of the error (i.e., what exactly cannot serde decode).

I would expect seeing something like `error receiving body: error
decoding response body: XXX field not found` after this patch, though I
didn't set up a testing environment to observe the exact behavior.

---------

Signed-off-by: Alex Chi Z <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug t/on_call_followup triaged bugs that were already triaged
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants