-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[fetchart] Add retry with exponential backoff #4639
base: master
Are you sure you want to change the base?
Conversation
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixes the issue as advertised, nice work!
However, a recent breaking change in urllib3
makes it so we have to jump through some hoops to get it to work.
Tested by running beet fetchart
over a large library with itunes
as the only source
- Before this PR, the logs showed multiple requests failing with HTTP 403 like so
fetchart: trying source itunes for album Chromatics - In the City fetchart: getting URL: https://itunes.apple.com/search?term=Chromatics+In+the+City&entity=album&media=music&limit=200 fetchart: iTunes search failed: 403 Client Error: Forbidden for url: https://itunes.apple.com/search?term=Chromatics+In+the+City&entity=album&media=music&limit=200 fetchart: Chromatics - In the City: no art found
- With this PR, those failing requests do not appear anymore -- instead the process pauses once in a while before carrying on.
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
The iTunes source returns 403 as a basic form of rate limiting. To get around this, we add a retry with exponential backoff to all requests. We also include some other error codes to retry.
dbefbec
to
fc115f4
Compare
@ybnd Rebased on master and removed the contentious |
@richardmitic looks good! |
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Description
The iTunes source returns 403 as a basic form of rate limiting. To get around this, we add a retry with exponential backoff to all requests made by
_logged_get()
. We also include some other generic error codes to retry.Fixes #3827, which was closed with no fix. Without this change, two different runs of
beet fetchart -f
will download different sets of album art because the iTunes has art available but will fail some requests with 403.To Do
docs/
to describe it.)docs/changelog.rst
near the top of the document.)