pending python 3 upgrade tasks

[alastair]

✔️ This item is fixed in #115

There is still some python code which only runs in python 2.
Most importantly are uses of basestring and unicode:

gaia/src/bindings/pythonic.i

Lines 24 to 31 in ed433ed

	if not isinstance(key, basestring):
	raise TypeError('A key in the map is not a string; can not convert it')
	# until python 3, everything should always be utf-8 encoded bytestrings
	if isinstance(key, unicode):
	key = key.encode('utf-8')
	if isinstance(value, unicode):
	value = value.encode('utf-8')

These will have to be converted, but we have to understand the use of the types in this method. Theoretically both basestring and unicode can be changed to str, but in python 3, we should double-check where items should be actual strings, and where they should be encoded bytes.

[alastair]

gaia/src/bindings/pygaia/classification/classificationtask.py

Line 200 in c72b8f1

config = cPickle.load(sys.stdin)

This doesn't work in python3, as pickle.load requires bytes, but sys.stdin is a string. Ideally this would be fixed by #96, allowing us to remove cluster mode, but in the meantime we should work out a solution to this specific problem.

[alastair]

In python 3, taskhash is generating different hashes for the same combination of parameters:

gaia/src/bindings/pygaia/classification/classificationtaskmanager.py

Line 42 in f227e41

def taskhash(config):

when I run this script multiple times, it doesn't skip already done jobs as it does in python 2:

gaia/src/bindings/pygaia/classification/classificationtask.py

Line 174 in c72b8f1

if exists(resultFilename):

Running the same script multiple types in the same docker container results in different hashes each time. There is a random seed set in the project file.

I've replaced it with sha1(json.dumps(config)), and now it's stable

[alastair]

Running in python 3, clustermode=False, after about 290 jobs I get an exception

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/concurrent/futures/process.py", line 295, in _queue_management_worker
    shutdown_worker()
  File "/usr/lib/python3.6/concurrent/futures/process.py", line 253, in shutdown_worker
    call_queue.put_nowait(None)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 129, in put_nowait
    return self.put(obj, False)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 83, in put
    raise Full
queue.Full

This has happened more than once. Is something not cleaning up properly? Maybe a difference between concurrent.futures in python 3.6 and the backport we use in python 2?