Skip to content

Commit bdc850f

Browse files
KontinuationHyukjinKwon
authored andcommitted
[SPARK-54745][PYTHON] Fix PySpark import error caused by missing UnixStreamServer on Windows
### What changes were proposed in this pull request? This PR fixes an error caused by `socketserver.UnixStreamServer` not being available on Windows. We define a fallback `AccumulatorUnixServer` to raise an exception on construction and inform the user to disable `spark.python.unix.domain.socket.enabled`. ### Why are the changes needed? `import pyspark` fails with the following message on Windows since PySpark 4.1.0: ``` sedona\spark\__init__.py:19: in <module> import pyspark .venv\Lib\site-packages\pyspark\__init__.py:71: in <module> from pyspark.accumulators import Accumulator, AccumulatorParam .venv\Lib\site-packages\pyspark\accumulators.py:324: in <module> class AccumulatorUnixServer(socketserver.UnixStreamServer): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E AttributeError: module 'socketserver' has no attribute 'UnixStreamServer' ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually test this on Windows 11 ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53546 from Kontinuation/fix-uds-windows-compat. Authored-by: Kristin Cowalcijk <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit a8f817d) Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent 89cdd63 commit bdc850f

File tree

1 file changed

+30
-15
lines changed

1 file changed

+30
-15
lines changed

python/pyspark/accumulators.py

Lines changed: 30 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -321,21 +321,36 @@ def shutdown(self) -> None:
321321
self.server_close()
322322

323323

324-
class AccumulatorUnixServer(socketserver.UnixStreamServer):
325-
server_shutdown = False
326-
327-
def __init__(
328-
self, socket_path: str, RequestHandlerClass: Type[socketserver.BaseRequestHandler]
329-
):
330-
super().__init__(socket_path, RequestHandlerClass)
331-
self.auth_token = None
332-
333-
def shutdown(self) -> None:
334-
self.server_shutdown = True
335-
super().shutdown()
336-
self.server_close()
337-
if os.path.exists(self.server_address): # type: ignore[arg-type]
338-
os.remove(self.server_address) # type: ignore[arg-type]
324+
# socketserver.UnixStreamServer is not available on Windows yet
325+
# (https://github.com/python/cpython/issues/77589).
326+
if hasattr(socketserver, "UnixStreamServer"):
327+
328+
class AccumulatorUnixServer(socketserver.UnixStreamServer):
329+
server_shutdown = False
330+
331+
def __init__(
332+
self, socket_path: str, RequestHandlerClass: Type[socketserver.BaseRequestHandler]
333+
):
334+
super().__init__(socket_path, RequestHandlerClass)
335+
self.auth_token = None
336+
337+
def shutdown(self) -> None:
338+
self.server_shutdown = True
339+
super().shutdown()
340+
self.server_close()
341+
if os.path.exists(self.server_address): # type: ignore[arg-type]
342+
os.remove(self.server_address) # type: ignore[arg-type]
343+
344+
else:
345+
346+
class AccumulatorUnixServer(socketserver.TCPServer): # type: ignore[no-redef]
347+
def __init__(
348+
self, socket_path: str, RequestHandlerClass: Type[socketserver.BaseRequestHandler]
349+
):
350+
raise NotImplementedError(
351+
"Unix Domain Sockets are not supported on this platform. "
352+
"Please disable it by setting spark.python.unix.domain.socket.enabled to false."
353+
)
339354

340355

341356
def _start_update_server(

0 commit comments

Comments
 (0)