Implement retries backoffs

[nkuba]

This issue is a followup from #409. We added retry delays for retries loops as a temporary solution. We should replace them with an elegant solution using exponential backoff (e.g. https://github.com/cenkalti/backoff).