Do you know what industries are available?

Do you know what industries are available?

I guess the ones used by Yahoo Screener.
See here: https://finance.yahoo.com/research-hub/screener/equity
In requests, they should be specified exactly as they appear on Yahoo Finance.

I keep experimenting with industry screening.
It seems to work except for ones with "—" in their names.
Is there any idea how to fix it?

The problem here is that yahoo uses specifically the 'em dash' character. When calling the query request the body gets passed as a json to the requests library. Down the chain the 'json.dumbs' function gets called, which changes the 'em dash' to '\u2014'. Obviously the resulting string is not a valid industry for yahoo, so it returns 0 items.

One fix (temporary) would be to go into the requests library to file models.py to line 511. Add 'ensure_ascii=False' as parameter to the complexjson.dumps function.
Obviously since it is a underlying library it might have impact on other requests in the code. So somebody with more knowledge in python project structuring has to come in now.

The problem here is that yahoo uses specifically the 'em dash' character. When calling the query request the body gets passed as a json to the requests library. Down the chain the 'json.dumbs' function gets called, which changes the 'em dash' to '\u2014'. Obviously the resulting string is not a valid industry for yahoo, so it returns 0 items.

One fix (temporary) would be to go into the requests library to file models.py to line 511. Add 'ensure_ascii=False' as parameter to the complexjson.dumps function. Obviously since it is a underlying library it might have impact on other requests in the code. So somebody with more knowledge in python project structuring has to come in now.

I see, thanks.

@ggeeoorrgg What do you think of my pr? Does it work how you would expect?

@ggeeoorrgg What do you think of my pr? Does it work how you would expect?

I really appreciate your efforts, thanks.
I just haven't had enough time to explore that, but I'll try.
By the way, could you please show an example of how to use your improvements?

P.S. I've just tried to explore your suggestion, but unfortunately, I didn’t quite get it. I still need a use case.

One fix (temporary) would be to go into the requests library...

Well, to avoid digging too deep, it should be sufficient to do something like this:
In line 390 of Data.py, replace:
request_args['json'] = body
with:

body = json.dumps(body, cls=CustomJSONEncoder, ensure_ascii=True)
request_args['data'] = body
request_args["headers"].update({"Content-Type": "application/json"})

And define this somewhere:

class CustomJSONEncoder(json.JSONEncoder):
    def encode(self, obj):
        json_str = super().encode(obj)
        return json_str.replace(r'\u2014', '—')

It works, but I haven’t done extensive testing.
I also discovered that industry names with an em dash should be specified without spaces, e.g., "REIT—Mortgage".