Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adbc_driver_postgresql.dbapi causes silent program exit on connection attempt to db #1531

Open
programmingwithalex opened this issue Feb 8, 2024 · 34 comments

Comments

@programmingwithalex
Copy link

Config:

  • Windows
  • python=3.11.7
  • pandas=2.2.0
  • adbc-driver-manager=0.9.0
  • adbc-driver-postgresql=0.9.0
  • pyarrow=15.0.0

Program exits silently (no exceptions raised) on adbc_driver_postgresql.dbapi.connect(uri).

import adbc_driver_postgresql.dbapi

uri = (
    'postgresql://' +
    f'{config["USERNAME"]}:' +
    f'{config["PASSWORD"]}@' +
    f'{config["HOST"]}:' +
    '5432/' +
    f'{config["DB_NAME"]}'
)

with adbc_driver_postgresql.dbapi.connect(uri) as conn:  # program exits silently here
   pass

Debugging

  • Validated database can be connected to with:
import sqlalchemy

uri = (
    'postgresql://' +
    f'{config["USERNAME"]}:' +
    f'{config["PASSWORD"]}@' +
    f'{config["HOST"]}:' +
    '5432/' +
    f'{config["DB_NAME"]}'
)

engine = sqlalchemy.create_engine(uri)

with engine.connect() as conn:
    pass
@lidavidm lidavidm added this to the ADBC Libraries 0.10.0 milestone Feb 8, 2024
@lidavidm
Copy link
Member

lidavidm commented Feb 8, 2024

Thanks for the report. Windows support is a bit difficult for me but I will spin up a VM to investigate when I get a chance.

@lidavidm
Copy link
Member

lidavidm commented Feb 8, 2024

Hmm, I am unable to reproduce this on a Windows 11 developer VM

PS C:\Users\User> python
Python 3.11.6 (tags/v3.11.6:8b6ee5b, Oct  2 2023, 14:57:12) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import adbc_driver_postgresql.dbapi
>>> adbc_driver_postgresql.dbapi.connect("postgresql://postgres:[email protected]:5432/postgres")
<adbc_driver_manager.dbapi.Connection object at 0x0000025F132CB850>
>>> exit()
PS C:\Users\User> pip freeze
adbc-driver-manager==0.9.0
adbc-driver-postgresql==0.9.0
colorama==0.4.6
duckdb==0.9.2
iniconfig==2.0.0
numpy==1.26.4
packaging==23.2
pandas==2.2.0
pluggy==1.4.0
pyarrow==15.0.0
pytest==8.0.0
python-dateutil==2.8.2
pytz==2024.1
six==1.16.0
typing_extensions==4.9.0
tzdata==2023.4

I tried the context manager as well in case that was the issue, but no dice.

PS C:\Users\User> python
Python 3.11.6 (tags/v3.11.6:8b6ee5b, Oct  2 2023, 14:57:12) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import adbc_driver_postgresql.dbapi
>>> import adbc_driver_postgresql.dbapi
KeyboardInterrupt
>>> conn = adbc_driver_postgresql.dbapi.connect("postgresql://postgres:[email protected]:5432/postgres")
>>> with conn:
...     pass
...
>>> conn
<adbc_driver_manager.dbapi.Connection object at 0x0000027E1B888650>
>>> with adbc_driver_postgresql.dbapi.connect("postgresql://postgres:[email protected]:5432/postgres") as conn:
...     with conn.cursor() as cur:
...         cur.execute("SELECT 1")
...         print(cur.fetchone())
...
(1,)
>>> exit()

@programmingwithalex
Copy link
Author

programmingwithalex commented Feb 8, 2024

I should have been more specific to simulate the error. The PostgreSQL database is hosted on AWS RDS and I'm running Windows 10 now.

It's so strange that you don't replicate the error though either. I can't think of any AWS RDS db configurations that would allow connections with sqlalchemy but not adbc_driver_postgresql.dbapi. Could you please try to replicate the issue once more on Windows 10?

@lidavidm
Copy link
Member

lidavidm commented Feb 8, 2024

Do you know of a way to get a Windows 10 developer VM?

@programmingwithalex
Copy link
Author

Unfortunately I don't and based on my research Windows no longer supports them. I'm hoping someone else has the same issue and discovers this thread to contribute.

The only other information I can give is when I attempt the connection I get the hourglass next to the mouse indicating some sort of processing is going on for a second before the python program dies.

@lidavidm
Copy link
Member

lidavidm commented Feb 9, 2024

Ok. It turns out, I dug up an old laptop (that I was planning on donating/recycling) but it happens to still be on Windows 10. When I can find more time again I'll give it another shot.

@programmingwithalex
Copy link
Author

You're the man David!

@lidavidm
Copy link
Member

No dice, unfortunately:

PS C:\Windows\system32> python
Python 3.11.8 (tags/v3.11.8:db85d51, Feb  6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import adbc_driver_postgresql.dbapi
>>> uri = "postgresql://postgres:[email protected]:5432/postgres"
>>> adbc_driver_postgresql.dbapi.connect(uri)
<adbc_driver_manager.dbapi.Connection object at 0x0000029912EDED50>
>>> conn = _
>>> conn.close()
>>> with adbc_driver_postgresql.dbapi.connect(uri) as conn:
...  with conn.cursor() as cur:
...   cur.execute("SELECT 1")
...   print(cur.fetchall())
...
[(1,)]
>>> exit()
PS C:\Windows\system32> pip freeze
adbc-driver-manager==0.9.0
adbc-driver-postgresql==0.9.0
numpy==1.26.4
pandas==2.2.0
pyarrow==15.0.0
python-dateutil==2.8.2
pytz==2024.1
six==1.16.0
tzdata==2024.1
PS C:\Windows\system32> [Environment]::OSVersion

Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.18362.0 Microsoft Windows NT 10.0.18362.0

@lidavidm
Copy link
Member

Are you possibly able to load WinDbg (it's in the Windows Store) and get a backtrace?

windbgx -g python myscript.py

should pop up the GUI and run the program

@programmingwithalex
Copy link
Author

Thanks for the suggestion. I ran the tool you suggested and indeed got a backtrace. It seems the error is related to the package itself and an integrity error on a single file.

(6ea4.33f0): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_postgresql\libadbc_driver_postgresql.so
libadbc_driver_postgresql!PostgresqlDriverInit+0x13f172:
00007ffa`05e4b9c2 f00fc119        lock xadd dword ptr [rcx],ebx ds:feeefeee`feeeffa6=????????

@lidavidm
Copy link
Member

Bleh. Any chance you have more of the traceback than that?

That's quite odd since that function implicated just fills out a struct, basically. I'm not great at reading disassembly, but I think I found the instruction implicated; the disassembly is extremely oddly long (hundreds of thousands of instructions), too.

@lidavidm
Copy link
Member

Ok, the disassembly is a few hundred lines of what I would expect, a ret, and then 600,000 instructions of garbage, and it appears the crash you have is inside that garbage. Based on that, it seems like the problem is a few levels up; something is jumping to the wrong address for you. So if there's a full backtrace from windbg, that would be interesting.

@programmingwithalex
Copy link
Author


************* Preparing the environment for Debugger Extensions Gallery repositories **************
   ExtensionRepository : Implicit
   UseExperimentalFeatureForNugetShare : true
   AllowNugetExeUpdate : true
   AllowNugetMSCredentialProviderInstall : true
   AllowParallelInitializationOfLocalRepositories : true

   -- Configuring repositories
      ----> Repository : LocalInstalled, Enabled: true
      ----> Repository : UserExtensions, Enabled: true

>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.297 seconds
   ----> Repository : UserExtensions, Enabled: true, Packages count: 0
   ----> Repository : LocalInstalled, Enabled: true, Packages count: 36

Microsoft (R) Windows Debugger Version 10.0.25921.1001 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\python.exe .\pyarrow_faster_reads.py

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
ModLoad: 00007ff7`fb940000 00007ff7`fb95a000   python.exe
ModLoad: 00007ffa`937f0000 00007ffa`939e8000   ntdll.dll
ModLoad: 00007ffa`92fb0000 00007ffa`9306f000   C:\WINDOWS\System32\KERNEL32.DLL
ModLoad: 00007ffa`914d0000 00007ffa`917a2000   C:\WINDOWS\System32\KERNELBASE.dll
ModLoad: 00007ffa`913d0000 00007ffa`914d0000   C:\WINDOWS\System32\ucrtbase.dll
ModLoad: 00007ffa`3e480000 00007ffa`3ea53000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\python311.dll
ModLoad: 00007ffa`88e20000 00007ffa`88e3d000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\VCRUNTIME140.dll
ModLoad: 00007ffa`92770000 00007ffa`927db000   C:\WINDOWS\System32\WS2_32.dll
ModLoad: 00007ffa`80c10000 00007ffa`80c1a000   C:\WINDOWS\SYSTEM32\VERSION.dll
ModLoad: 00007ffa`92500000 00007ffa`92625000   C:\WINDOWS\System32\RPCRT4.dll
ModLoad: 00007ffa`92400000 00007ffa`9249e000   C:\WINDOWS\System32\msvcrt.dll
ModLoad: 00007ffa`910a0000 00007ffa`910c7000   C:\WINDOWS\System32\bcrypt.dll
ModLoad: 00007ffa`93520000 00007ffa`935ce000   C:\WINDOWS\System32\ADVAPI32.dll
ModLoad: 00007ffa`93440000 00007ffa`934dc000   C:\WINDOWS\System32\sechost.dll
ModLoad: 00007ffa`91340000 00007ffa`913c2000   C:\WINDOWS\System32\bcryptprimitives.dll
ModLoad: 0000022d`b67b0000 0000022d`b67bf000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\python3.DLL
ModLoad: 0000022d`b67b0000 0000022d`b67bf000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\python3.DLL
ModLoad: 00007ffa`24350000 00007ffa`243a2000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_manager\_lib.cp311-win_amd64.pyd
ModLoad: 00007ff9`fe870000 00007ff9`fe90b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_manager.libs\msvcp140-a6ccfe46d550248f88945c0533c21bb8.dll
ModLoad: 00007ff9`fd980000 00007ff9`fdc28000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\lib.cp311-win_amd64.pyd
ModLoad: 00007ff9`fe7d0000 00007ff9`fe86b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\MSVCP140.dll
ModLoad: 00007ff9`fcff0000 00007ff9`fd13f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow_python.dll
ModLoad: 00007ff9`e0bc0000 00007ff9`e2952000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow.dll
ModLoad: 0000022d`b7070000 0000022d`b8e02000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow.dll
ModLoad: 00007ffa`910d0000 00007ffa`91226000   C:\WINDOWS\System32\CRYPT32.dll
ModLoad: 00007ffa`93120000 00007ffa`932c1000   C:\WINDOWS\System32\USER32.dll
ModLoad: 00007ffa`91800000 00007ffa`91822000   C:\WINDOWS\System32\win32u.dll
ModLoad: 00007ffa`6e650000 00007ffa`6eb29000   C:\WINDOWS\SYSTEM32\WININET.dll
ModLoad: 00007ffa`935d0000 00007ffa`935fb000   C:\WINDOWS\System32\GDI32.dll
ModLoad: 00007ffa`90ee0000 00007ffa`90fef000   C:\WINDOWS\System32\gdi32full.dll
ModLoad: 00007ffa`91230000 00007ffa`912cd000   C:\WINDOWS\System32\msvcp_win.dll
ModLoad: 00007ffa`8fe20000 00007ffa`8ff2c000   C:\WINDOWS\SYSTEM32\WINHTTP.dll
ModLoad: 00007ffa`91ba0000 00007ffa`922e4000   C:\WINDOWS\System32\SHELL32.dll
ModLoad: 00007ffa`76100000 00007ffa`7610c000   C:\WINDOWS\SYSTEM32\Secur32.dll
ModLoad: 00007ffa`92640000 00007ffa`9276a000   C:\WINDOWS\System32\ole32.dll
ModLoad: 00007ffa`91830000 00007ffa`91b85000   C:\WINDOWS\System32\combase.dll
ModLoad: 00007ffa`909b0000 00007ffa`909d7000   C:\WINDOWS\SYSTEM32\ncrypt.dll
ModLoad: 00007ffa`90de0000 00007ffa`90e0e000   C:\WINDOWS\SYSTEM32\USERENV.dll
ModLoad: 00007ffa`906f0000 00007ffa`906fc000   C:\WINDOWS\SYSTEM32\CRYPTBASE.DLL
ModLoad: 00007ffa`90da0000 00007ffa`90dd2000   C:\WINDOWS\SYSTEM32\SSPICLI.DLL
ModLoad: 00007ffa`90970000 00007ffa`909ab000   C:\WINDOWS\SYSTEM32\NTASN1.dll
ModLoad: 00007ffa`934e0000 00007ffa`93512000   C:\WINDOWS\System32\IMM32.DLL
ModLoad: 00007ffa`27f40000 00007ffa`27f7f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_decimal.pyd
ModLoad: 00007ff9`fbb20000 00007ff9`fbdf8000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\core\_multiarray_umath.cp311-win_amd64.pyd
ModLoad: 00007ff9`d8940000 00007ff9`da9f6000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy.libs\libopenblas64__v0.3.23-293-gc2f4bdbb-gcc_10_3_0-2bde3a66a51006b2b53eb373ff767a3f.dll
ModLoad: 00007ffa`4d940000 00007ffa`4d954000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\core\_multiarray_tests.cp311-win_amd64.pyd
ModLoad: 00007ffa`76030000 00007ffa`7604f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_ctypes.pyd
ModLoad: 00007ffa`936e0000 00007ffa`937ad000   C:\WINDOWS\System32\OLEAUT32.dll
ModLoad: 00007ffa`7e0c0000 00007ffa`7e0cb000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\libffi-8.dll
ModLoad: 00007ffa`4d6a0000 00007ffa`4d6bd000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\linalg\_umath_linalg.cp311-win_amd64.pyd
ModLoad: 00007ffa`48f10000 00007ffa`48f2e000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\fft\_pocketfft_internal.cp311-win_amd64.pyd
ModLoad: 00007ff9`fe4b0000 00007ff9`fe547000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\mtrand.cp311-win_amd64.pyd
ModLoad: 00007ffa`3fc80000 00007ffa`3fcac000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\bit_generator.cp311-win_amd64.pyd
ModLoad: 00007ffa`3c150000 00007ffa`3c17f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_common.cp311-win_amd64.pyd
ModLoad: 00007ffa`48a00000 00007ffa`48a11000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_hashlib.pyd
ModLoad: 00007ffa`3df80000 00007ffa`3e471000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\libcrypto-3.dll
ModLoad: 00007ffa`270e0000 00007ffa`27123000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_bounded_integers.cp311-win_amd64.pyd
ModLoad: 00007ffa`45720000 00007ffa`45736000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_mt19937.cp311-win_amd64.pyd
ModLoad: 00007ffa`40ce0000 00007ffa`40cf5000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_philox.cp311-win_amd64.pyd
ModLoad: 00007ffa`3dde0000 00007ffa`3ddf9000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_pcg64.cp311-win_amd64.pyd
ModLoad: 00007ffa`60310000 00007ffa`60320000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_sfc64.cp311-win_amd64.pyd
ModLoad: 00007ff9`fd8d0000 00007ff9`fd97f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\numpy\random\_generator.cp311-win_amd64.pyd
ModLoad: 00007ffa`7a590000 00007ffa`7a5a5000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_socket.pyd
ModLoad: 00007ffa`90270000 00007ffa`902ab000   C:\WINDOWS\SYSTEM32\IPHLPAPI.DLL
ModLoad: 00007ffa`88820000 00007ffa`88829000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\select.pyd
ModLoad: 00007ffa`8be00000 00007ffa`8be09000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_queue.pyd
ModLoad: 00007ffa`3b520000 00007ffa`3b544000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_hdfsio.cp311-win_amd64.pyd
ModLoad: 00007ff9`fcf50000 00007ff9`fcfec000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_dataset.cp311-win_amd64.pyd
ModLoad: 00007ff9`fb9d0000 00007ff9`fbb19000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow_dataset.dll
ModLoad: 00007ff9`fb890000 00007ff9`fb9ca000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow_acero.dll
ModLoad: 00007ff9`e0600000 00007ff9`e0bb4000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\parquet.dll
ModLoad: 00007ffa`2c850000 00007ffa`2c87b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_acero.cp311-win_amd64.pyd
ModLoad: 00007ff9`fcc40000 00007ff9`fccf1000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_compute.cp311-win_amd64.pyd
ModLoad: 00007ffa`24200000 00007ffa`24255000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_fs.cp311-win_amd64.pyd
ModLoad: 00007ffa`1fe40000 00007ffa`1fe7c000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_csv.cp311-win_amd64.pyd
ModLoad: 00007ffa`3c290000 00007ffa`3c2a4000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_json.cp311-win_amd64.pyd
ModLoad: 00007ffa`763c0000 00007ffa`763d5000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_bz2.pyd
ModLoad: 00007ffa`66e90000 00007ffa`66eb9000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_lzma.pyd
ModLoad: 00007ffa`5fda0000 00007ffa`5fdab000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\pandas_parser.cp311-win_amd64.pyd
ModLoad: 00007ffa`5ba50000 00007ffa`5ba5b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\pandas_datetime.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7db0000 00007ff9`f7ec7000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\interval.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7be0000 00007ff9`f7da8000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\hashtable.cp311-win_amd64.pyd
ModLoad: 00007ffa`23420000 00007ffa`23450000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\missing.cp311-win_amd64.pyd
ModLoad: 00007ffa`1a5d0000 00007ffa`1a5fd000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\dtypes.cp311-win_amd64.pyd
ModLoad: 00007ffa`2caf0000 00007ffa`2cb05000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\ccalendar.cp311-win_amd64.pyd
ModLoad: 00007ffa`1fe20000 00007ffa`1fe40000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\np_datetime.cp311-win_amd64.pyd
ModLoad: 00007ffa`19bf0000 00007ffa`19c2b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\conversion.cp311-win_amd64.pyd
ModLoad: 00007ffa`1ab20000 00007ffa`1ab31000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\base.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7b00000 00007ff9`f7bdd000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\offsets.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7a70000 00007ff9`f7af1000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\timestamps.cp311-win_amd64.pyd
ModLoad: 00007ffa`18c40000 00007ffa`18c75000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\nattype.cp311-win_amd64.pyd
ModLoad: 00007ff9`f79f0000 00007ff9`f7a68000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\timedeltas.cp311-win_amd64.pyd
ModLoad: 00007ffa`18c00000 00007ffa`18c3a000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\timezones.cp311-win_amd64.pyd
ModLoad: 00007ffa`59e20000 00007ffa`59e2d000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_zoneinfo.pyd
ModLoad: 0000022d`c7f40000 0000022d`c7f43000   C:\WINDOWS\SYSTEM32\tzres.dll
ModLoad: 0000022d`c7f40000 0000022d`c7f43000   C:\WINDOWS\SYSTEM32\tzres.dll
ModLoad: 00007ffa`11300000 00007ffa`11346000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\fields.cp311-win_amd64.pyd
ModLoad: 00007ffa`186a0000 00007ffa`186e0000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\tzconversion.cp311-win_amd64.pyd
ModLoad: 00007ffa`1aa40000 00007ffa`1aa54000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\properties.cp311-win_amd64.pyd
ModLoad: 00007ff9`fd870000 00007ff9`fd8c3000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\parsing.cp311-win_amd64.pyd
ModLoad: 00007ff9`fcf00000 00007ff9`fcf4e000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\strptime.cp311-win_amd64.pyd
ModLoad: 00007ff9`fb820000 00007ff9`fb887000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\period.cp311-win_amd64.pyd
ModLoad: 00007ff9`fc880000 00007ff9`fc8b2000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslibs\vectorized.cp311-win_amd64.pyd
ModLoad: 00007ffa`590d0000 00007ffa`590e0000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\ops_dispatch.cp311-win_amd64.pyd
ModLoad: 00007ff9`f6de0000 00007ff9`f6fa2000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\algos.cp311-win_amd64.pyd
ModLoad: 00007ff9`f73e0000 00007ff9`f7485000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\lib.cp311-win_amd64.pyd
ModLoad: 00007ff9`f5050000 00007ff9`f5168000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\unicodedata.pyd
ModLoad: 00007ff9`f73a0000 00007ff9`f73d5000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\ops.cp311-win_amd64.pyd
ModLoad: 00007ff9`fe7a0000 00007ff9`fe7cb000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\hashing.cp311-win_amd64.pyd
ModLoad: 00007ffa`1a5b0000 00007ffa`1a5cc000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\arrays.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7350000 00007ff9`f7392000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\tslib.cp311-win_amd64.pyd
ModLoad: 00007ff9`f4f70000 00007ff9`f5041000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\sparse.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7200000 00007ff9`f7250000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\internals.cp311-win_amd64.pyd
ModLoad: 00007ffa`19bd0000 00007ffa`19be1000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\indexing.cp311-win_amd64.pyd
ModLoad: 00007ff9`f4eb0000 00007ff9`f4f64000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\index.cp311-win_amd64.pyd
ModLoad: 00007ff9`f7070000 00007ff9`f70a3000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\writers.cp311-win_amd64.pyd
ModLoad: 00007ff9`f4d90000 00007ff9`f4ea6000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\join.cp311-win_amd64.pyd
ModLoad: 00007ff9`f6d80000 00007ff9`f6dd7000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\window\aggregations.cp311-win_amd64.pyd
ModLoad: 00007ffa`59030000 00007ffa`5903c000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\VCRUNTIME140_1.dll
ModLoad: 00007ff9`f4cf0000 00007ff9`f4d8b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas.libs\msvcp140-fa0758dedafbbe194d3ee96e3dc2b9a3.dll
ModLoad: 00007ff9`fcc10000 00007ff9`fcc3b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\window\indexers.cp311-win_amd64.pyd
ModLoad: 00007ff9`f6710000 00007ff9`f674f000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\reshape.cp311-win_amd64.pyd
ModLoad: 00007ff9`e03e0000 00007ff9`e05f4000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\groupby.cp311-win_amd64.pyd
ModLoad: 00007ffa`8c140000 00007ffa`8c147000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_uuid.pyd
ModLoad: 00007ffa`066b0000 00007ffa`066c1000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\json.cp311-win_amd64.pyd
ModLoad: 00007ff9`f6480000 00007ff9`f64e6000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\parsers.cp311-win_amd64.pyd
ModLoad: 00007ffa`00680000 00007ffa`00699000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pandas\_libs\testing.cp311-win_amd64.pyd
ModLoad: 00007ff9`f51d0000 00007ff9`f520c000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_dataset_parquet.cp311-win_amd64.pyd
ModLoad: 00007ff9`f4c90000 00007ff9`f4ce9000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_parquet.cp311-win_amd64.pyd
ModLoad: 00007ff9`ffb20000 00007ff9`ffb34000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_dataset_parquet_encryption.cp311-win_amd64.pyd
ModLoad: 00007ff9`fcbd0000 00007ff9`fcbfa000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\_parquet_encryption.cp311-win_amd64.pyd
ModLoad: 00007ffa`58780000 00007ffa`5878d000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\pyarrow\arrow_python_parquet_encryption.dll
ModLoad: 00007ff9`fde80000 00007ff9`fde9b000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_manager\_reader.cp311-win_amd64.pyd
ModLoad: 00007ffa`67420000 00007ffa`6744d000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_ssl.pyd
ModLoad: 00007ffa`4db00000 00007ffa`4dbc3000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\libssl-3.dll
ModLoad: 00007ffa`90560000 00007ffa`905ca000   C:\WINDOWS\system32\mswsock.dll
ModLoad: 00007ff9`fde30000 00007ff9`fde50000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\_elementtree.pyd
ModLoad: 00007ff9`f4c50000 00007ff9`f4c82000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\DLLs\pyexpat.pyd
ModLoad: 00007ff9`dfff0000 00007ff9`e03d8000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\lxml\etree.cp311-win_amd64.pyd
ModLoad: 00007ff9`fc060000 00007ff9`fc089000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\lxml\_elementpath.cp311-win_amd64.pyd
ModLoad: 00007ff9`dfd70000 00007ff9`dffee000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\PIL\_imaging.cp311-win_amd64.pyd
ModLoad: 00007ffa`902b0000 00007ffa`9037a000   C:\WINDOWS\SYSTEM32\DNSAPI.dll
ModLoad: 00007ffa`91b90000 00007ffa`91b98000   C:\WINDOWS\System32\NSI.dll
ModLoad: 00007ffa`87d00000 00007ffa`87d0a000   C:\Windows\System32\rasadhlp.dll
ModLoad: 00007ffa`7f420000 00007ffa`7f4a0000   C:\WINDOWS\System32\fwpuclnt.dll
ModLoad: 00007ff9`df8a0000 00007ff9`dfd62000   C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_postgresql\libadbc_driver_postgresql.so
ModLoad: 00007ffa`93680000 00007ffa`936d6000   C:\WINDOWS\System32\WLDAP32.dll
ModLoad: 00007ffa`6f090000 00007ffa`6f099000   C:\WINDOWS\SYSTEM32\WSOCK32.dll
ModLoad: 00007ffa`8ee30000 00007ffa`8f5c2000   C:\WINDOWS\SYSTEM32\windows.storage.dll
ModLoad: 00007ffa`90780000 00007ffa`907b0000   C:\WINDOWS\System32\Wldp.dll
ModLoad: 00007ffa`92350000 00007ffa`923fd000   C:\WINDOWS\System32\SHCORE.dll
ModLoad: 00007ffa`93620000 00007ffa`93675000   C:\WINDOWS\System32\shlwapi.dll
ModLoad: 00007ffa`90830000 00007ffa`90848000   C:\WINDOWS\SYSTEM32\CRYPTSP.dll
ModLoad: 00007ffa`8fd90000 00007ffa`8fdc4000   C:\WINDOWS\system32\rsaenh.dll
(7108.5f2c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for C:\Users\<user>\.pyenv\pyenv-win\versions\3.11.7\Lib\site-packages\adbc_driver_postgresql\libadbc_driver_postgresql.so
libadbc_driver_postgresql!PostgresqlDriverInit+0x13f172:
00007ff9`dfb2b9c2 f00fc119        lock xadd dword ptr [rcx],ebx ds:feeefeee`feeeffa6=????????

I hope this helps. I only didn't include it initially because it didn't appear to have any clues as to what might be causing the issue.

@CurtHagenlocher
Copy link
Contributor

CurtHagenlocher commented Feb 13, 2024

Okay, this is a super-naive question but is a dynamic library generated by a toolchain producing an ".so" file even compatible with what the driver manager is expecting to load?

EDIT to minimize spamming: I thought checksum failure might be indicative of something, but if this weren't a valid DLL then we presumably wouldn't be referencing a symbol inside of it.

@lidavidm
Copy link
Member

It's a DLL, just that I didn't bother to special case the extension in the build pipeline

@lidavidm
Copy link
Member

Hmm, that gives everything except for the rest of the traceback which is what I'm curious about: how/why are we jumping into what appears to not even be valid code?

@lidavidm
Copy link
Member

It does appear I should try with pyenv instead of Chocolatey python, though

@lidavidm
Copy link
Member

Or apparently pyenv-win? Is that what you're using?

@lidavidm
Copy link
Member

No dice, tried Python 3.11.7 via pyenv-win and it did not crash either.

@reinieroost
Copy link

reinieroost commented Mar 6, 2024

I have the same issue on windows 11.

  • python 3.10
  • adbc-driver-manager 0.10.0
  • abdc-driver-postgresql 0.10.0
  • pandas 2.2.1
  • pyarrow 15.0.0

When running dbapi.connect(uri) it crashes with: Process finished with exit code -1073740940 (0xC0000374)

On ubuntu I haven't had this issue when connecting to the same database.

@lidavidm
Copy link
Member

lidavidm commented Mar 6, 2024

Thanks, good to know. I'll try to find time to replicate this again but I've not been having any luck, or else I'll try to figure out the backtrace handler (#1583) and see if we can provide a nightly wheel to test with

@lidavidm
Copy link
Member

lidavidm commented Mar 7, 2024

Oh, some more questions:

  • Are either of you running in a notebook?
  • Are either of you able to get the full backtrace, a crash dump, etc.?

@reinieroost
Copy link

I'm not running it on a notebook but in pycharm.

I ran faulthander before the crash:

Windows fatal exception: code 0xc0000374

Thread 0x00008adc (most recent call first):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 217 in acquire
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 225 in __enter__
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 803 in process_internal_commands
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\pydevd.py", line 151 in _on_run
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 219 in run
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 1016 in _bootstrap_inner
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 973 in _bootstrap

Thread 0x00007710 (most recent call first):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 293 in _on_run
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 219 in run
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 1016 in _bootstrap_inner
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 973 in _bootstrap

Thread 0x0000715c (most recent call first):
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 324 in wait
  File "C:\Users\~\anaconda3\envs\~\lib\queue.py", line 180 in get
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 368 in _on_run
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2023.3.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 219 in run
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 1016 in _bootstrap_inner
  File "C:\Users\~\anaconda3\envs\~\lib\threading.py", line 973 in _bootstrap

Current thread 0x00008194 (most recent call first):
  File "C:\Users\~\anaconda3\envs\~\lib\site-packages\adbc_driver_postgresql\__init__.py", line 40 in connect
  File "C:\Users\~\anaconda3\envs\~\lib\site-packages\adbc_driver_postgresql\dbapi.py", line 121 in connect
Process finished with exit code -1073740940 (0xC0000374)

@lidavidm
Copy link
Member

lidavidm commented Mar 7, 2024

Thanks. That gives the Python stack, but unfortunately not the C-level stack that could/might explain how we're getting so sidetracked...

@reinieroost
Copy link

reinieroost commented Mar 8, 2024

Does this give you more insights?

Faulting application name: python.exe, version: 3.10.13150.1013, time stamp: 0x64ff158d
Faulting module name: ntdll.dll, version: 10.0.22621.3085, time stamp: 0xbced4b82
Exception code: 0xc0000374
Fault offset: 0x000000000010c7a9
Faulting process ID: 0x0x4910
Faulting application start time: 0x0x1DA70C7FF12C621
Faulting application path: C:\Users\~\anaconda3\envs\~\python.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report ID: 4f42f518-c955-411f-a2be-e10a50518adc
Faulting package full name: 
Faulting package-relative application ID: 
Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: python.exe
P2: 3.10.13150.1013
P3: 64ff158d
P4: StackHash_c67d
P5: 10.0.22621.3085
P6: bced4b82
P7: c0000374
P8: PCH_40_FROM_ntdll+0x000000000009FEC4
P9: 
P10: 

@LaurensBosscher
Copy link

I can confirm this is also an issue on:

  • Windows 11
  • Python 3.11
  • adbc-driver-manager 0.10.0
  • abdc-driver-postgresql 0.10.0
  • pandas 2.2.1
  • pyarrow 15.0.0

Also an AWS RDS database. Connecting using username+password.

@lidavidm
Copy link
Member

Hmm.

Is everyone here trying RDS? Does it crash with a local or other non-AWS Postgres instance?

@reinieroost
Copy link

I'm not using AWS

@mkleinbort-ic
Copy link

I have the same issue, on an Azure hosted db.

I'm running locally and it crashes the process/notebook when ran either in jupyter, IPython, or just Python.

I was suspecting it'd be the antivirus blocking it - but no idea.

On Windows 11, Python 3.10.9

@lidavidm
Copy link
Member

I have built some wheels here with debug info (scroll all the way down): https://github.com/apache/arrow-adbc/actions/runs/8331028521?pr=1599

Would it be possible for anyone to give that a shot, preferably under windbgx, and see if they can get a full stack trace?

@lidavidm
Copy link
Member

I'm still unable to reproduce this. If anyone is able to produce a full stack trace (not just the last frame), and/or a reproducer, that would be much appreciated.

As far as I can see, the original crash happens before any connection attempt is made (it happens when trying to load the client library), so it doesn't seem to be related to RDS/Azure/etc.

@lidavidm
Copy link
Member

Or even knowing how things were installed would help (e.g. I just saw Anaconda above, but we don't ship Conda packages on Windows!)

@lidavidm
Copy link
Member

Hmm, no dice either with Anaconda Python + ADBC from wheels

@lidavidm
Copy link
Member

Or actually, please use the updated wheels here (scroll down to "python3.12-windows" etc): https://github.com/apache/arrow-adbc/actions/runs/8347053159?pr=1599

Also, please extract "adbc_driver_postgresql.pdb" from the ZIP file and place it next to your script/in the PWD so that the backtrace handler can find it. If all goes well, a crash should print something like this, which will let us debug the issue:

#5    Object "", at 00007FF860185F99, in PyBytes_Repeat
#4    Object "", at 00007FF899441D61, in AdbcDriver::operator=
#3    Object "", at 00007FF89946D5C6, in AdbcConnectionSetOption
#2    Source "C:\Users\User\arrow-adbc\c\driver\sqlite\sqlite.c", line 254, in SqliteConnectionSetOption [00007FF84A115712]
#1    Source "C:\Users\User\arrow-adbc\c\driver\sqlite\sqlite.c", line 145, in ExecuteQuery [00007FF84A115215]
#0    Object "", at 00007FF860EAFF3E, in memset

@lidavidm lidavidm removed this from the ADBC Libraries 0.11.0 milestone Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants