non-Numpy data in Variogram and MetricSpace

[jmmcd]

In our research project we have some non-numerical data points which we would like to put into Variogram as the coordinates. There is a distance function which computes distance values between pairs of points. Also, there is a measure of fitness (quality) for each point, so this would go in as the values.

This is conceptually ok, but Variogram and MetricSpace put their data (coordinates and coords respectively) into Numpy arrays, so it's a crash. Below are my investigations, ending in a hack which seems to work.

Would you be interested in a pull request which allows MetricSpace(dist_matrix, metric='precomputed'), or some similar design?

# test non-numpy data

import numpy as np
from skgstat import MetricSpace, Variogram
import skgstat

print(skgstat.__version__)
# '1.0.23' - I used pip install -e from GitHub

# x data is not Numpy-compatible
# just an example: our real data is made up of complex nested tuples
x = [(1, 2, 3), (2, 3), (3, 1, 2), (4, 1, 2, 3), (1, 2, 3, 5, 4)]
y = [len(xi) for xi in x]
n = len(x)

def dist(a, b): return abs(len(a) - len(b)) 
dist_matrix = np.zeros((n, n))
for i in range(n):
    for j in range(n):
        dist_matrix[i, j] = dist(x[i], x[j])
print(dist_matrix) # abs(len(a) - len(b)) 
# [[0. 1. 0. 1. 2.] 
#  [1. 0. 1. 2. 3.]
#  [0. 1. 0. 1. 2.]
#  [1. 2. 1. 0. 1.]
#  [2. 3. 2. 1. 0.]]

print("The x data can't go directly into a Numpy array, so this is a crash in Variogram:")
try:
    v = Variogram(x, y, dist_func=dist)
except Exception as e:
    print(e)

print("Same for MetricSpace:")
try:
    ms = MetricSpace(x, dist_metric=dist)
except Exception as e:
    print(e)

print("This method I thought might work, by providing precomputed distances")
print("It doesn't crash, but it is treating the distance matrix (nxn) as")
print("coordinates, not distances, ie n points in an n-dimensional space")
ms = MetricSpace(dist_matrix)
print(ms.dists)
# [[0.         2.23606798 0.         2.23606798 4.        ]
#  [2.23606798 0.         2.23606798 3.46410162 4.58257569]
#  [0.         2.23606798 0.         2.23606798 4.        ]
#  [2.23606798 3.46410162 2.23606798 0.         2.23606798]
#  [4.         4.58257569 4.         2.23606798 0.        ]]

print("This crashes, as scipy doesn't expect 'precomputed'")
print("Something like this could be a nice API?")
try:
    ms = MetricSpace(dist_matrix, dist_metric="precomputed")
except Exception as e:
    print(e)


print("Claude suggested this hack, which seems to work ok")
ms = MetricSpace(dist_matrix)
dummy = np.random.random((n, 2)) # dummy coords, we will over-write dists
ms = MetricSpace(dummy) 
ms._dists = dist_matrix
print(ms.dists)
# [[0. 1. 0. 1. 2.]
#  [1. 0. 1. 2. 3.]
#  [0. 1. 0. 1. 2.]
#  [1. 2. 1. 0. 1.]
#  [2. 3. 2. 1. 0.]]
v = Variogram(coordinates=ms, values=y)

[mmaelicke]

Hi,
Thanks for opening this issue.
What do you think about adding a classmethod to MetricSpace like:

ms = MetricSpace.from_dists(dist_matrix)
vario = Variogram(coordinates=ms, values=y)

In my opinion, that would be clean approach, as we do not pass something as coordinates to MetricSpace, which is something else. We could also set a new flag, ie, MetricSpace.precomputed to signal that the distance matrix and not coordinates were passed.

If you use Claude anyway, please make sure to instruct it to check how the Kriging algorithm and the transform method might make use of the metric space, so that we do not introduce any logical errors. I cannot guarantee that I did not use a MetricSpace._* method or property somewhere, which might break then.

If you are unfamiliar with class methods, you can of course come back to me.

Can you maybe also give me a short idea about your research, so that I have a better understanding of you use case?

Thanks again,
Best

Mirko

[jmmcd]

Right. If you agree, it will be like this (without the dist_func here, Variogram will complain that the distance functions don't match):

ms = MetricSpace.from_dists(dist_matrix)
vario = Variogram(coordinates=ms, values=y, dist_func="precomputed")

I have code for this, but it needs more testing first.

About our research - we are studying evolutionary computation (genetic algorithms, genetic programming, and similar). The search space can be a vector space, but in our case, often it consists of tree structures. They could be represented as nested tuples, for example. In other research they could be arbitrary data structures. Variable-length tuples is sufficient to demonstrate the Numpy crash, above. Also, every individual has a fitness value, ie quality, which we try to maximise in our search - this gives the y-values for the Variogram. When studying fitness landscapes we calculate distances between pairs of these objects, giving the distance matrix.