Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use requests library for raw_iter, to support custom http_session #229

Merged
merged 2 commits into from
Apr 11, 2019

Conversation

OriHoch
Copy link
Contributor

@OriHoch OriHoch commented Oct 24, 2018

This fixes a problem with loading data from urls which require a custom http session (e.g. for http auth)

While tabulator accepts http_session option to support this, the raw_iter method which is used to infer encoding doesn't

To fix it - changed the raw_iter to use requests library and optionally use the http_session option from table options

I think that in the long term we should have the encoding detection done in tabulator - to keep a consistent loading method

@OriHoch OriHoch requested review from akariv and roll October 24, 2018 14:35
@akariv
Copy link
Member

akariv commented Oct 24, 2018

lgtm

@roll
Copy link
Member

roll commented Oct 29, 2018

@OriHoch
Thanks!

There is one thing I'm concerned about. You use requests.raw (http://docs.python-requests.org/en/master/user/quickstart/#raw-response-content). But as far as I can remember I failed to use it with tabulator because it doesn't decompress the stream. Not sure it has been changed for requests@3. I think we at least need to add a few tests to figure it out.

@roll roll merged commit 29a9e34 into frictionlessdata:master Apr 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants