Port forwarding not working with python requests library #569

rami-foo · 2025-01-31T17:34:05Z

Which project are you reporting a bug for?

kr8s

What happened?

kubectl create namespace argo
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

after the apply you will get an initial root token, put this token as the password in the payload and it should be reproduced

This works (with manual port-forwarding)

import requests
from utils.kubernetes_utils import K8

url = "http://127.0.0.1:1009/api/v1/session"
payload = {"username": "admin", "password": "---"}
headers = {"Content-Type": "application/json"}

response = requests.post(url, json=payload, headers=headers, verify=False)

print(response.status_code, response.text)

This does not work

import requests
from utils.kubernetes_utils import K8

payload = {"username": "admin", "password": "---"}
headers = {"Content-Type": "application/json"}

with K8().get_service("argocd-server", namespace="argo").portforward(remote_port=80, local_port="auto") as local_port:
    url = f"http://127.0.0.1:{local_port}/api/v1/session"
    response = requests.post(url, json=payload, headers=headers, verify=False)
    print(response.status_code, response.text)

Anything else?

Error Log

Traceback (most recent call last):
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\urllib3\connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\urllib3\connection.py", line 507, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py", line 1395, in getresponse
    response.begin()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py", line 294, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\requests\adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\urllib3\connectionpool.py", line 843, in urlopen       
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\urllib3\util\retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\urllib3\util\util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\urllib3\connectionpool.py", line 789, in urlopen       
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request 
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\urllib3\connection.py", line 507, in getresponse       
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py", line 1395, in getresponse
    response.begin()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py", line 294, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\User\Desktop\FOO_\Project Setup Automation\test.py", line 38, in <module>
    response = requests.post(url, json=payload, headers=headers, verify=False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\requests\api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\User\Desktop\FOO_\Project Setup Automation\.venv\Lib\site-packages\requests\adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The text was updated successfully, but these errors were encountered:

rami-foo · 2025-02-03T10:00:19Z

From the additional info I have gathered :

The error still happens with kr8s v 0.19.1.
Connecting to HASHICORP vault with portforwarding works as intended.
The error happens here when the switch protocols happens:

2025-02-03 10:09:27,316 - httpx - INFO - HTTP Request: GET https://AWS_IP/api/v1/namespaces/argo/pods/argocd-server-888b5d99f-zq4mg "HTTP/1.1 200 OK"
2025-02-03 10:09:27,317 - httpcore.http11 - DEBUG - receive_response_body.started request=<Request [b'GET']>
2025-02-03 10:09:27,318 - httpcore.http11 - DEBUG - receive_response_body.complete
2025-02-03 10:09:27,318 - httpcore.http11 - DEBUG - response_closed.started
2025-02-03 10:09:27,318 - httpcore.http11 - DEBUG - response_closed.complete
2025-02-03 10:09:27,333 - httpcore.http11 - DEBUG - send_request_headers.started request=<Request [b'GET']>
2025-02-03 10:09:27,333 - httpcore.http11 - DEBUG - send_request_headers.complete
2025-02-03 10:09:27,333 - httpcore.http11 - DEBUG - send_request_body.started request=<Request [b'GET']>
2025-02-03 10:09:27,333 - httpcore.http11 - DEBUG - send_request_body.complete
2025-02-03 10:09:27,333 - httpcore.http11 - DEBUG - receive_response_headers.started request=<Request [b'GET']>
2025-02-03 10:09:27,450 - httpcore.http11 - DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 101, b'Switching Protocols', [(b'Upgrade', b'websocket'), (b'Connection', b'Upgrade'), (b'Sec-WebSocket-Accept', b'GHY385CBC70ha/yn35vdCKJtIwo='), (b'Sec-WebSocket-Protocol', b'')])
2025-02-03 10:09:27,450 - httpx - INFO - HTTP Request: GET https://AWS_IP/api/v1/namespaces/argo/pods/argocd-server-888b5d99f-zq4mg/portforward?name=argocd-server-888b5d99f-zq4mg&namespace=argo&ports=80&_preload_content=false "HTTP/1.1 101 Switching Protocols"
2025-02-03 10:09:27,453 - httpcore.http11 - DEBUG - response_closed.started
2025-02-03 10:09:27,453 - httpcore.http11 - DEBUG - response_closed.complete

jacobtomlinson · 2025-02-03T10:47:53Z

I tried a simple example by starting an nginx web server Pod and getting the homepage with a port forward and requests.

This works as expected:

# Create web server Pod
kubectl run webserver --image nginx --port 80
kubectl expose po webserver

import kr8s
import requests

[svc] = kr8s.get("svc", "webserver")
with svc.portforward(remote_port=80, local_port="auto") as local_port:
    resp = requests.get(f"http://localhost:{local_port}/")
    print(resp.text)
# <!DOCTYPE html>\n<html>\n<head>\n<title>Welcome to nginx!</title>...

# Clean up
kubectl delete svc/webserver po/webserver

Given that you are also able to access Vault via a port forward it sounds like there is something specific to the Argo request you are making. The error messages you shared show the remote server not responding and requests timing out. This could be due to the port forward not passing data back and forth, but it's unclear why this is happening.

The logs you shared up to the Switching Protocols line are as expected, the port forward uses websockets and the last thing it does before the data exchange starts is switch protocols. However, after that line the connection closes which suggests the argo server closes the connection.

To debug this further we need a small example like the one I shared above that reproduces the issue. That way I can attach a debugger and see what is going on.

rami-foo · 2025-02-03T11:45:35Z

edited my issue to provide with hopefully a reproducable kubectl cli command

rami-foo added the bug Something isn't working label Jan 31, 2025

jacobtomlinson added the needs reproducible example label Feb 3, 2025

jacobtomlinson mentioned this issue Feb 3, 2025

Sync port forward context managed hangs on exit for around a minute #570

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port forwarding not working with python requests library #569

Port forwarding not working with python requests library #569

rami-foo commented Jan 31, 2025 •

edited

Loading

rami-foo commented Feb 3, 2025 •

edited by jacobtomlinson

Loading

jacobtomlinson commented Feb 3, 2025 •

edited

Loading

rami-foo commented Feb 3, 2025

Port forwarding not working with python requests library #569

Port forwarding not working with python requests library #569

Comments

rami-foo commented Jan 31, 2025 • edited Loading

Which project are you reporting a bug for?

What happened?

Anything else?

rami-foo commented Feb 3, 2025 • edited by jacobtomlinson Loading

jacobtomlinson commented Feb 3, 2025 • edited Loading

rami-foo commented Feb 3, 2025

rami-foo commented Jan 31, 2025 •

edited

Loading

rami-foo commented Feb 3, 2025 •

edited by jacobtomlinson

Loading

jacobtomlinson commented Feb 3, 2025 •

edited

Loading