Install python toolchain fails

[Comradin]

Hello,

just tried to install the python toolchain and it fails with a hardcoded dependency for python3.5?

% srclib toolchain install python Python (sourcegraph.com/sourcegraph/srclib-python) ============================ failed to install/upgrade Python (sourcegraph.com/sourcegraph/srclib-python) toolchain: nopython3.5` found in PATH; to install, visit https://www.python.org/downloads/

FAILED: srclib toolchain install python (failed to install/upgrade Python (sourcegraph.com/sourcegraph/srclib-python) toolchain: no python3.5 found in PATH; to install, visit https://www.python.org/downloads/
)
`

I am working with CentOS7, did a fresh install of the srclib tool itself via go get.

Does srclib support only python3?

On RedHat based distributions one can install it via the software collections mechanism, But after enabling the python35 it fails with a mixture of python2.7 and python3.5 errors:

`% source /opt/rh/rh-python35/enable
% srclib toolchain install python
Python (sourcegraph.com/sourcegraph/srclib-python) ============================
Downloading Python toolchain
Cloning into 'srclib-python'...
remote: Counting objects: 2318, done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 2318 (delta 70), reused 0 (delta 0), pack-reused 2177
Receiving objects: 100% (2318/2318), 2.06 MiB | 1.69 MiB/s, done.
Resolving deltas: 100% (1300/1300), done.
mkdir -p /home_marcus/.srclib/sourcegraph.com/sourcegraph
ln -s /home_marcus/src/sourcegraph.com/sourcegraph/srclib-python /home_marcus/.srclib/sourcegraph.com/sourcegraph/srclib-python
Building Python toolchain program
Running [make] in /home_marcus/src/sourcegraph.com/sourcegraph/srclib-python
virtualenv -p python3.5 .env
Running virtualenv with interpreter /opt/rh/rh-python35/root/usr/bin/python3.5
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/enum/init.py", line 371, in getattr
return cls.member_map[name]
KeyError: '_convert'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/virtualenv.py", line 23, in
import subprocess
File "/opt/rh/rh-python35/root/usr/lib64/python3.5/subprocess.py", line 364, in
import signal
File "/opt/rh/rh-python35/root/usr/lib64/python3.5/signal.py", line 8, in
_IntEnum._convert(
File "/usr/lib/python2.7/site-packages/enum/init.py", line 373, in getattr
raise AttributeError(name)
AttributeError: _convert
make: *** [.env] Error 1
failed to install/upgrade Python (sourcegraph.com/sourcegraph/srclib-python) toolchain: command ["make"] failed: exit status 2

FAILED: srclib toolchain install python (failed to install/upgrade Python (sourcegraph.com/sourcegraph/srclib-python) toolchain: command ["make"] failed: exit status 2
)
`

Thats because I had installed virtualenv support just for the system python2.7 and not for the scl based python3.5. Maybe the installer could provide better error messages like:
Did you install python3.5 and according virtualenv support beforehand?

After installing the rh-python35-python-virtualenv package I was able to install the python support. So, the only question, that remains, is, is python2.7 unsupported?

With kind regards,
Marcus

[alexsaveliev]

Hi,

Does srclib support only python3?

Actually, it's python3.5 (or later). I don't know much details, but I've heard the following, quote, type annotations were added in 3.5, end quote. Probably https://github.com/sourcegraph/srclib-python developers will chime in

[beyang]

srclib-python must be run with a python3.5 environment currently. You can use virtualenv if you don't want to mess with your system Python. It should be able to process code in both Python 3 and 2, however (with some exceptions that we hope to fix).

[macsj200]

@beyang Could you elaborate a bit on how one might set up the virtualenv? Specifically, does that mean we should construct a virtualenv with a python3.5 binary alias?

[beyang]

Python toolchain requires 3.5 for dependency parsing, if I recall correctly.

Here's a Dockerfile showing how we run it:

#
# Docker image for srclib-python

#
# Dependencies
#
FROM python:3.5.1-slim
RUN apt-get update
RUN apt-get install -qq make git tar python2.7
RUN pip3.5 install virtualenv

#
# Install srclib-python executable
#
ENV SRCLIBPATH /srclib/toolchains
ARG TOOLCHAIN_URL
ADD $TOOLCHAIN_URL /toolchain/t
RUN (cd /toolchain && tar xfz t && rm t && mv * /toolchain/t) || true
RUN mkdir -p $SRCLIBPATH/sourcegraph.com/sourcegraph && mv /toolchain/t $SRCLIBPATH/sourcegraph.com/sourcegraph/srclib-python
WORKDIR $SRCLIBPATH/sourcegraph.com/sourcegraph/srclib-python
RUN make install

#
# Install srclib binary (assumes this has been built on the Docker host)
#
ADD ./srclib /bin/srclib

# Run environment
ENTRYPOINT srclib config && srclib make

[macsj200]

I resolved the python3.5 binary issues, but now the installation fails with

.env/bin/mypy --silent-imports grapher
grapher/pydepwrap.py: note: In function "requirements_from_requirements_txt":
grapher/pydepwrap.py:55: error: No overload variant of "itemgetter" matches argument types [builtins.str]
grapher/pydepwrap.py: note: In function "requirements":
grapher/pydepwrap.py:62: error: No overload variant of "itemgetter" matches argument types [builtins.str]
grapher/pydepwrap.py:65: error: No overload variant of "itemgetter" matches argument types [builtins.str]
make: *** [check] Error 1
failed to install/upgrade Python (sourcegraph.com/sourcegraph/srclib-python) toolchain: command ["make"] failed: exit status 2

FAILED: srclib toolchain install go ruby javascript python (failed to install/upgrade Python (sourcegraph.com/sourcegraph/srclib-python) toolchain: command ["make"] failed: exit status 2
)

Have you seen this issue before?