[IDEV-2271] RE:RTUF stream responses: address PR feedbacks

[briluza]

Changes:

• Remove wait/sleep for RTTF feed results
• Disable rate limiting for RTTF products
• Remove x-api-key in vcr cassetes
• Minor code adjustments (comments, imports)

[kelvinatorr]

You don't need to yield and the self.latest_feeds_status_code line above. You can instead just call self.setStatus() here instead of in line :252

[briluza]

done. Thanks!

[kelvinatorr]

If you want to just raise an Exception for the user you don't need to layer the try: except here and in the other functions. You can just remove all the try: except:'s and let the exception happen with the error text.

For example if the user has a bad value with after=abc here is what the error will look like:

(venv) (IDEV-2271)~/github.com/python_api$ python tester.py 
Traceback (most recent call last):
  File "/home/kdealca/github.com/python_api/domaintools/results.py", line 252, in _get_results
    self.setStatus(self.latest_feeds_status_code)  # set the status already
  File "/home/kdealca/github.com/python_api/domaintools/base_results.py", line 201, in setStatus
    raise BadRequestException(code, reason)
domaintools.exceptions.BadRequestException: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kdealca/github.com/python_api/domaintools/results.py", line 270, in response
    yield from feed_response_generator
  File "/home/kdealca/github.com/python_api/domaintools/results.py", line 257, in _get_results
    self.setStatus(500, reason_text=e)
  File "/home/kdealca/github.com/python_api/domaintools/base_results.py", line 207, in setStatus
    raise InternalServerErrorException(code, reason)
domaintools.exceptions.InternalServerErrorException: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kdealca/github.com/python_api/tester.py", line 9, in <module>
    for r in results.response():
  File "/home/kdealca/github.com/python_api/domaintools/results.py", line 279, in response
    self.setStatus(500, reason_text=e)
  File "/home/kdealca/github.com/python_api/domaintools/base_results.py", line 207, in setStatus
    raise InternalServerErrorException(code, reason)
domaintools.exceptions.InternalServerErrorException: None
(venv) (IDEV-2271)~/github.com/python_api$

It is 3 exceptions deep which makes it difficult to understand. And the helpful text from the server:

(IDEV-2271)~/github.com/python_api$ curl -H "X-API-Key: $DT_API_KEY" "https://api.domaintools.com/v1/feed/nod?after=abc"
{"error":"Bad request: invalid value in after query parameter; see documentation for examples"}
(IDEV-2271)~/github.com/python_api$

Is nowhere to be seen.

Instead you can do something like this:

    def _make_request(self) -> Generator:
        """
        Creates and manages the httpx stream request, yielding data line by line.
        This is the core generator that communicates with the DT frontend API server.
        """
        session_info = self._get_session_params_and_headers()
        headers = session_info.get("headers")
        parameters = session_info.get("parameters")

        with httpx.stream(
            "GET",
            self.url,
            headers=headers,
            params=parameters,
            verify=self.api.verify_ssl,
            proxy=self.api.proxy_url,
            timeout=None,
        ) as r:
            error_text = ""
            if r.status_code != 200 or r.status_code != 206:
                r.read()
                error_text = r.text
            self.setStatus(r.status_code, reason_text=error_text)

            for line in r.iter_lines():
                yield line

    def data(self) -> Generator:
        self._data = self._get_results()
        return self._data

    def response(self) -> Generator:
        while self.status != 200:
            yield from self.data()
            if not self.kwargs.get("sessionID"):
                # we'll only do iterative request for queries that has sessionID. Otherwise, we will have an infinite
                # request if sessionID was not provided but the required data asked is more than the maximum (1 hour of data)
                break
        self._status = None

If there is an error there will only be one, and any message from the server will be shown:

(venv) (IDEV-2271)~/github.com/python_api$ python tester.py 
Traceback (most recent call last):
  File "/home/kdealca/github.com/python_api/tester.py", line 9, in <module>
    for r in results.response():
  File "/home/kdealca/github.com/python_api/domaintools/results.py", line 248, in response
    yield from self.data()
  File "/home/kdealca/github.com/python_api/domaintools/results.py", line 237, in _make_request
    self.setStatus(r.status_code, reason_text=error_text)
  File "/home/kdealca/github.com/python_api/domaintools/base_results.py", line 201, in setStatus
    raise BadRequestException(code, reason)
domaintools.exceptions.BadRequestException: {"error":"Bad request: invalid value in after query parameter; see documentation for examples"}

(venv) (IDEV-2271)~/github.com/python_api$

[briluza]

thanks for the feedback @kelvinatorr! will do the corresponding changes

[briluza]

done.

[kelvinatorr]

You can now get rid of _get_results() completely and use _make_request() directly in data()

[briluza]

done. thanks!

[kelvinatorr]

time is now not used.

[briluza]

removed.

[kelvinatorr]

Thanks for addressing my concerns