PyGAD 2.7.0 Regression Support

Changes in PyGAD 2.7.0 (11 September 2020):

1. The `learning_rate` parameter in the `pygad.nn.train()` function defaults to **0.01**.
2. Added support of building neural networks for regression using the new parameter named `problem_type`. It is added as a parameter to both `pygad.nn.train()` and `pygad.nn.predict()` functions. The value of this parameter can be either **classification** or **regression** to define the problem type. It defaults to **classification**.
3. The activation function for a layer can be set to the string `"None"` to refer that there is no activation function at this layer. As a result, the supported values for the activation function are `"sigmoid"`, `"relu"`, `"softmax"`, and `"None"`.

To build a regression network using the `pygad.nn` module, just do the following:

1. Set the `problem_type` parameter in the `pygad.nn.train()` and `pygad.nn.predict()` functions to the string `"regression"`.
2. Set the activation function for the output layer to the string `"None"`. This sets no limits on the range of the outputs as it will be from `-infinity` to `+infinity`. If you are sure that all outputs will be nonnegative values, then use the ReLU function.

Check the documentation of the `pygad.nn` module for an example that builds a neural network for regression. The regression example is also available at [this GitHub project](https://github.com/ahmedfgad/NumPyANN): https://github.com/ahmedfgad/NumPyANN

To build and train a regression network using the `pygad.gann` module, do the following:

1. Set the `problem_type` parameter in the `pygad.nn.train()` and `pygad.nn.predict()` functions to the string `"regression"`.
2. Set the `output_activation` parameter in the constructor of the `pygad.gann.GANN` class to `"None"`.

Check the documentation of the `pygad.gann` module for an example that builds and trains a neural network for regression. The regression example is also available at [this GitHub project](https://github.com/ahmedfgad/NeuralGenetic): https://github.com/ahmedfgad/NeuralGenetic

To build a classification network, either ignore the `problem_type` parameter or set it to `"classification"` (default value). In this case, the activation function of the last layer can be set to any type (e.g. softmax).

Author: ahmedfgad

Fish.csv

@@ -0,0 +1,160 @@
+Species,Weight,Length1,Length2,Length3,Height,Width
+Bream,242,23.2,25.4,30,11.52,4.02
+Bream,290,24,26.3,31.2,12.48,4.3056
+Bream,340,23.9,26.5,31.1,12.3778,4.6961
+Bream,363,26.3,29,33.5,12.73,4.4555
+Bream,430,26.5,29,34,12.444,5.134
+Bream,450,26.8,29.7,34.7,13.6024,4.9274
+Bream,500,26.8,29.7,34.5,14.1795,5.2785
+Bream,390,27.6,30,35,12.67,4.69
+Bream,450,27.6,30,35.1,14.0049,4.8438
+Bream,500,28.5,30.7,36.2,14.2266,4.9594
+Bream,475,28.4,31,36.2,14.2628,5.1042
+Bream,500,28.7,31,36.2,14.3714,4.8146
+Bream,500,29.1,31.5,36.4,13.7592,4.368
+Bream,340,29.5,32,37.3,13.9129,5.0728
+Bream,600,29.4,32,37.2,14.9544,5.1708
+Bream,600,29.4,32,37.2,15.438,5.58
+Bream,700,30.4,33,38.3,14.8604,5.2854
+Bream,700,30.4,33,38.5,14.938,5.1975
+Bream,610,30.9,33.5,38.6,15.633,5.1338
+Bream,650,31,33.5,38.7,14.4738,5.7276
+Bream,575,31.3,34,39.5,15.1285,5.5695
+Bream,685,31.4,34,39.2,15.9936,5.3704
+Bream,620,31.5,34.5,39.7,15.5227,5.2801
+Bream,680,31.8,35,40.6,15.4686,6.1306
+Bream,700,31.9,35,40.5,16.2405,5.589
+Bream,725,31.8,35,40.9,16.36,6.0532
+Bream,720,32,35,40.6,16.3618,6.09
+Bream,714,32.7,36,41.5,16.517,5.8515
+Bream,850,32.8,36,41.6,16.8896,6.1984
+Bream,1000,33.5,37,42.6,18.957,6.603
+Bream,920,35,38.5,44.1,18.0369,6.3063
+Bream,955,35,38.5,44,18.084,6.292
+Bream,925,36.2,39.5,45.3,18.7542,6.7497
+Bream,975,37.4,41,45.9,18.6354,6.7473
+Bream,950,38,41,46.5,17.6235,6.3705
+Roach,40,12.9,14.1,16.2,4.1472,2.268
+Roach,69,16.5,18.2,20.3,5.2983,2.8217
+Roach,78,17.5,18.8,21.2,5.5756,2.9044
+Roach,87,18.2,19.8,22.2,5.6166,3.1746
+Roach,120,18.6,20,22.2,6.216,3.5742
+Roach,0,19,20.5,22.8,6.4752,3.3516
+Roach,110,19.1,20.8,23.1,6.1677,3.3957
+Roach,120,19.4,21,23.7,6.1146,3.2943
+Roach,150,20.4,22,24.7,5.8045,3.7544
+Roach,145,20.5,22,24.3,6.6339,3.5478
+Roach,160,20.5,22.5,25.3,7.0334,3.8203
+Roach,140,21,22.5,25,6.55,3.325
+Roach,160,21.1,22.5,25,6.4,3.8
+Roach,169,22,24,27.2,7.5344,3.8352
+Roach,161,22,23.4,26.7,6.9153,3.6312
+Roach,200,22.1,23.5,26.8,7.3968,4.1272
+Roach,180,23.6,25.2,27.9,7.0866,3.906
+Roach,290,24,26,29.2,8.8768,4.4968
+Roach,272,25,27,30.6,8.568,4.7736
+Roach,390,29.5,31.7,35,9.485,5.355
+Whitefish,270,23.6,26,28.7,8.3804,4.2476
+Whitefish,270,24.1,26.5,29.3,8.1454,4.2485
+Whitefish,306,25.6,28,30.8,8.778,4.6816
+Whitefish,540,28.5,31,34,10.744,6.562
+Whitefish,800,33.7,36.4,39.6,11.7612,6.5736
+Whitefish,1000,37.3,40,43.5,12.354,6.525
+Parkki,55,13.5,14.7,16.5,6.8475,2.3265
+Parkki,60,14.3,15.5,17.4,6.5772,2.3142
+Parkki,90,16.3,17.7,19.8,7.4052,2.673
+Parkki,120,17.5,19,21.3,8.3922,2.9181
+Parkki,150,18.4,20,22.4,8.8928,3.2928
+Parkki,140,19,20.7,23.2,8.5376,3.2944
+Parkki,170,19,20.7,23.2,9.396,3.4104
+Parkki,145,19.8,21.5,24.1,9.7364,3.1571
+Parkki,200,21.2,23,25.8,10.3458,3.6636
+Parkki,273,23,25,28,11.088,4.144
+Parkki,300,24,26,29,11.368,4.234
+Perch,5.9,7.5,8.4,8.8,2.112,1.408
+Perch,32,12.5,13.7,14.7,3.528,1.9992
+Perch,40,13.8,15,16,3.824,2.432
+Perch,51.5,15,16.2,17.2,4.5924,2.6316
+Perch,70,15.7,17.4,18.5,4.588,2.9415
+Perch,100,16.2,18,19.2,5.2224,3.3216
+Perch,78,16.8,18.7,19.4,5.1992,3.1234
+Perch,80,17.2,19,20.2,5.6358,3.0502
+Perch,85,17.8,19.6,20.8,5.1376,3.0368
+Perch,85,18.2,20,21,5.082,2.772
+Perch,110,19,21,22.5,5.6925,3.555
+Perch,115,19,21,22.5,5.9175,3.3075
+Perch,125,19,21,22.5,5.6925,3.6675
+Perch,130,19.3,21.3,22.8,6.384,3.534
+Perch,120,20,22,23.5,6.11,3.4075
+Perch,120,20,22,23.5,5.64,3.525
+Perch,130,20,22,23.5,6.11,3.525
+Perch,135,20,22,23.5,5.875,3.525
+Perch,110,20,22,23.5,5.5225,3.995
+Perch,130,20.5,22.5,24,5.856,3.624
+Perch,150,20.5,22.5,24,6.792,3.624
+Perch,145,20.7,22.7,24.2,5.9532,3.63
+Perch,150,21,23,24.5,5.2185,3.626
+Perch,170,21.5,23.5,25,6.275,3.725
+Perch,225,22,24,25.5,7.293,3.723
+Perch,145,22,24,25.5,6.375,3.825
+Perch,188,22.6,24.6,26.2,6.7334,4.1658
+Perch,180,23,25,26.5,6.4395,3.6835
+Perch,197,23.5,25.6,27,6.561,4.239
+Perch,218,25,26.5,28,7.168,4.144
+Perch,300,25.2,27.3,28.7,8.323,5.1373
+Perch,260,25.4,27.5,28.9,7.1672,4.335
+Perch,265,25.4,27.5,28.9,7.0516,4.335
+Perch,250,25.4,27.5,28.9,7.2828,4.5662
+Perch,250,25.9,28,29.4,7.8204,4.2042
+Perch,300,26.9,28.7,30.1,7.5852,4.6354
+Perch,320,27.8,30,31.6,7.6156,4.7716
+Perch,514,30.5,32.8,34,10.03,6.018
+Perch,556,32,34.5,36.5,10.2565,6.3875
+Perch,840,32.5,35,37.3,11.4884,7.7957
+Perch,685,34,36.5,39,10.881,6.864
+Perch,700,34,36,38.3,10.6091,6.7408
+Perch,700,34.5,37,39.4,10.835,6.2646
+Perch,690,34.6,37,39.3,10.5717,6.3666
+Perch,900,36.5,39,41.4,11.1366,7.4934
+Perch,650,36.5,39,41.4,11.1366,6.003
+Perch,820,36.6,39,41.3,12.4313,7.3514
+Perch,850,36.9,40,42.3,11.9286,7.1064
+Perch,900,37,40,42.5,11.73,7.225
+Perch,1015,37,40,42.4,12.3808,7.4624
+Perch,820,37.1,40,42.5,11.135,6.63
+Perch,1100,39,42,44.6,12.8002,6.8684
+Perch,1000,39.8,43,45.2,11.9328,7.2772
+Perch,1100,40.1,43,45.5,12.5125,7.4165
+Perch,1000,40.2,43.5,46,12.604,8.142
+Perch,1000,41.1,44,46.6,12.4888,7.5958
+Pike,200,30,32.3,34.8,5.568,3.3756
+Pike,300,31.7,34,37.8,5.7078,4.158
+Pike,300,32.7,35,38.8,5.9364,4.3844
+Pike,300,34.8,37.3,39.8,6.2884,4.0198
+Pike,430,35.5,38,40.5,7.29,4.5765
+Pike,345,36,38.5,41,6.396,3.977
+Pike,456,40,42.5,45.5,7.28,4.3225
+Pike,510,40,42.5,45.5,6.825,4.459
+Pike,540,40.1,43,45.8,7.786,5.1296
+Pike,500,42,45,48,6.96,4.896
+Pike,567,43.2,46,48.7,7.792,4.87
+Pike,770,44.8,48,51.2,7.68,5.376
+Pike,950,48.3,51.7,55.1,8.9262,6.1712
+Pike,1250,52,56,59.7,10.6863,6.9849
+Pike,1600,56,60,64,9.6,6.144
+Pike,1550,56,60,64,9.6,6.144
+Pike,1650,59,63.4,68,10.812,7.48
+Smelt,6.7,9.3,9.8,10.8,1.7388,1.0476
+Smelt,7.5,10,10.5,11.6,1.972,1.16
+Smelt,7,10.1,10.6,11.6,1.7284,1.1484
+Smelt,9.7,10.4,11,12,2.196,1.38
+Smelt,9.8,10.7,11.2,12.4,2.0832,1.2772
+Smelt,8.7,10.8,11.3,12.6,1.9782,1.2852
+Smelt,10,11.3,11.8,13.1,2.2139,1.2838
+Smelt,9.9,11.3,11.8,13.1,2.2139,1.1659
+Smelt,9.8,11.4,12,13.2,2.2044,1.1484
+Smelt,12.2,11.5,12.2,13.4,2.0904,1.3936
+Smelt,13.4,11.7,12.4,13.5,2.43,1.269
+Smelt,12.2,12.1,13,13.8,2.277,1.2558
+Smelt,19.7,13.2,14.3,15.2,2.8728,2.0672
+Smelt,19.9,13.8,15,16.2,2.9322,1.8792

README.md

@@ -1,12 +1,12 @@
 # NumPyANN: Building Neural Networks using NumPy
 
-[NumPyANN](https://github.com/ahmedfgad/NumPyCNN) is a Python project for training neural networks using NumPy. 
+[NumPyANN](https://github.com/ahmedfgad/NumPyCNN) is a Python project for building artificial neural networks using NumPy. 
 
-[NumPyANN](https://github.com/ahmedfgad/NumPyCNN) is part of [PyGAD](https://pypi.org/project/pygad) which is an open-source Python 3 library for implementing the genetic algorithm and optimizing machine learning algorithms. 
+[NumPyANN](https://github.com/ahmedfgad/NumPyCNN) is part of [PyGAD](https://pypi.org/project/pygad) which is an open-source Python 3 library for implementing the genetic algorithm and optimizing machine learning algorithms. Both regression and classification neural networks are supported starting from PyGAD 2.7.0.
 
 Check documentation of the [NeuralGenetic](https://github.com/ahmedfgad/NeuralGenetic) project in the PyGAD's documentation: https://pygad.readthedocs.io/en/latest/README_pygad_nn_ReadTheDocs.html
 
-The library is under active development and more features in the genetic algorithm will be added like working with binary problems. This is in addition to supporting more machine learning algorithms.
+The library is under active development and more features are added regularly. If you want a feature to be supported, please check the **Contact Us** section to send a request.
 
 Before using [NumPyANN](https://github.com/ahmedfgad/NumPyCNN), install PyGAD.
 

example_XOR_classification.py

@@ -0,0 +1,51 @@
+import numpy
+import pygad.nn
+
+"""
+This project creates a neural network where the architecture has input and dense layers only. More layers will be added in the future. 
+The project only implements the forward pass of a neural network and no training algorithm is used.
+For training a neural network using the genetic algorithm, check this project (https://github.com/ahmedfgad/NeuralGenetic) in which the genetic algorithm is used for training the network.
+Feel free to leave an issue in this project (https://github.com/ahmedfgad/NumPyANN) in case something is not working properly or to ask for questions. I am also available for e-mails at [email protected]
+"""
+
+# Preparing the NumPy array of the inputs.
+data_inputs = numpy.array([[1, 1],
+                           [1, 0],
+                           [0, 1],
+                           [0, 0]])
+
+# Preparing the NumPy array of the outputs.
+data_outputs = numpy.array([0, 
+                            1, 
+                            1, 
+                            0])
+
+# The number of inputs (i.e. feature vector length) per sample
+num_inputs = data_inputs.shape[1]
+# Number of outputs per sample
+num_outputs = 2
+
+HL1_neurons = 2
+
+# Building the network architecture.
+input_layer = pygad.nn.InputLayer(num_inputs)
+hidden_layer1 = pygad.nn.DenseLayer(num_neurons=HL1_neurons, previous_layer=input_layer, activation_function="relu")
+output_layer = pygad.nn.DenseLayer(num_neurons=num_outputs, previous_layer=hidden_layer1, activation_function="softmax")
+
+# Training the network.
+pygad.nn.train(num_epochs=100,
+               last_layer=output_layer,
+               data_inputs=data_inputs,
+               data_outputs=data_outputs,
+               learning_rate=0.01)
+
+# Using the trained network for predictions.
+predictions = pygad.nn.predict(last_layer=output_layer, data_inputs=data_inputs)
+
+# Calculating some statistics
+num_wrong = numpy.where(predictions != data_outputs)[0]
+num_correct = data_outputs.size - num_wrong.size
+accuracy = 100 * (num_correct/data_outputs.size)
+print("Number of correct classifications : {num_correct}.".format(num_correct=num_correct))
+print("Number of wrong classifications : {num_wrong}.".format(num_wrong=num_wrong.size))
+print("Classification accuracy : {accuracy}.".format(accuracy=accuracy))

example_classification.py

@@ -0,0 +1,51 @@
+import numpy
+import pygad.nn
+
+"""
+This project creates a neural network where the architecture has input and dense layers only. More layers will be added in the future. 
+The project only implements the forward pass of a neural network and no training algorithm is used.
+For training a neural network using the genetic algorithm, check this project (https://github.com/ahmedfgad/NeuralGenetic) in which the genetic algorithm is used for training the network.
+Feel free to leave an issue in this project (https://github.com/ahmedfgad/NumPyANN) in case something is not working properly or to ask for questions. I am also available for e-mails at [email protected]
+"""
+
+# Reading the data features. Check the 'extract_features.py' script for extracting the features & preparing the outputs of the dataset.
+data_inputs = numpy.load("dataset_features.npy") # Download from https://github.com/ahmedfgad/NumPyANN/blob/master/dataset_features.npy
+
+# Optional step for filtering the features using the standard deviation.
+features_STDs = numpy.std(a=data_inputs, axis=0)
+data_inputs = data_inputs[:, features_STDs > 50]
+
+# Reading the data outputs. Check the 'extract_features.py' script for extracting the features & preparing the outputs of the dataset.
+data_outputs = numpy.load("outputs.npy") # Download from https://github.com/ahmedfgad/NumPyANN/blob/master/outputs.npy
+
+# The number of inputs (i.e. feature vector length) per sample
+num_inputs = data_inputs.shape[1]
+# Number of outputs per sample
+num_outputs = 4
+
+HL1_neurons = 150
+HL2_neurons = 60
+
+# Building the network architecture.
+input_layer = pygad.nn.InputLayer(num_inputs)
+hidden_layer1 = pygad.nn.DenseLayer(num_neurons=HL1_neurons, previous_layer=input_layer, activation_function="relu")
+hidden_layer2 = pygad.nn.DenseLayer(num_neurons=HL2_neurons, previous_layer=hidden_layer1, activation_function="relu")
+output_layer = pygad.nn.DenseLayer(num_neurons=num_outputs, previous_layer=hidden_layer2, activation_function="softmax")
+
+# Training the network.
+pygad.nn.train(num_epochs=10,
+               last_layer=output_layer,
+               data_inputs=data_inputs,
+               data_outputs=data_outputs,
+               learning_rate=0.01)
+
+# Using the trained network for predictions.
+predictions = pygad.nn.predict(last_layer=output_layer, data_inputs=data_inputs)
+
+# Calculating some statistics
+num_wrong = numpy.where(predictions != data_outputs)[0]
+num_correct = data_outputs.size - num_wrong.size
+accuracy = 100 * (num_correct/data_outputs.size)
+print("Number of correct classifications : {num_correct}.".format(num_correct=num_correct))
+print("Number of wrong classifications : {num_wrong}.".format(num_wrong=num_wrong.size))
+print("Classification accuracy : {accuracy}.".format(accuracy=accuracy))

example_regression.py

@@ -0,0 +1,46 @@
+import numpy
+import pygad.nn
+
+"""
+This example creates a neural network for regression where the architecture has input and dense layers only. More layers will be added in the future. 
+The project only implements the forward pass of a neural network and no training algorithm is used.
+For training a neural network using the genetic algorithm, check this project (https://github.com/ahmedfgad/NeuralGenetic) in which the genetic algorithm is used for training the network.
+Feel free to leave an issue in this project (https://github.com/ahmedfgad/NumPyANN) in case something is not working properly or to ask for questions. I am also available for e-mails at [email protected]
+"""
+
+# Preparing the NumPy array of the inputs.
+data_inputs = numpy.array([[2, 5, -3, 0.1],
+                           [8, 15, 20, 13]])
+
+# Preparing the NumPy array of the outputs.
+data_outputs = numpy.array([0.1, 
+                            1.5])
+
+# The number of inputs (i.e. feature vector length) per sample
+num_inputs = data_inputs.shape[1]
+# Number of outputs per sample
+num_outputs = 1
+
+HL1_neurons = 2
+
+# Building the network architecture.
+input_layer = pygad.nn.InputLayer(num_inputs)
+hidden_layer1 = pygad.nn.DenseLayer(num_neurons=HL1_neurons, previous_layer=input_layer, activation_function="relu")
+output_layer = pygad.nn.DenseLayer(num_neurons=num_outputs, previous_layer=hidden_layer1, activation_function="None")
+
+# Training the network.
+pygad.nn.train(num_epochs=100,
+               last_layer=output_layer,
+               data_inputs=data_inputs,
+               data_outputs=data_outputs,
+               learning_rate=0.01,
+               problem_type="regression")
+
+# Using the trained network for predictions.
+predictions = pygad.nn.predict(last_layer=output_layer, 
+                         data_inputs=data_inputs, 
+                         problem_type="regression")
+
+# Calculating some statistics
+abs_error = numpy.mean(numpy.abs(predictions - data_outputs))
+print("Absolute error : {abs_error}.".format(abs_error=abs_error))

example_regression_fish.py

@@ -0,0 +1,47 @@
+import numpy
+import pygad.nn
+import pandas
+
+"""
+This example creates a neural network for regression where the architecture has input and dense layers only. More layers will be added in the future. 
+The project only implements the forward pass of a neural network and no training algorithm is used.
+For training a neural network using the genetic algorithm, check this project (https://github.com/ahmedfgad/NeuralGenetic) in which the genetic algorithm is used for training the network.
+Feel free to leave an issue in this project (https://github.com/ahmedfgad/NumPyANN) in case something is not working properly or to ask for questions. I am also available for e-mails at [email protected]
+"""
+
+data = numpy.array(pandas.read_csv("Fish.csv"))
+
+# Preparing the NumPy array of the inputs.
+data_inputs = numpy.asarray(data[:, 2:], dtype=numpy.float32)
+
+# Preparing the NumPy array of the outputs.
+data_outputs = numpy.asarray(data[:, 1], dtype=numpy.float32) # Fish Weight
+
+# The number of inputs (i.e. feature vector length) per sample
+num_inputs = data_inputs.shape[1]
+# Number of outputs per sample
+num_outputs = 1
+
+HL1_neurons = 2
+
+# Building the network architecture.
+input_layer = pygad.nn.InputLayer(num_inputs)
+hidden_layer1 = pygad.nn.DenseLayer(num_neurons=HL1_neurons, previous_layer=input_layer, activation_function="relu")
+output_layer = pygad.nn.DenseLayer(num_neurons=num_outputs, previous_layer=hidden_layer1, activation_function="None")
+
+# Training the network.
+pygad.nn.train(num_epochs=100,
+               last_layer=output_layer,
+               data_inputs=data_inputs,
+               data_outputs=data_outputs,
+               learning_rate=0.01,
+               problem_type="regression")
+
+# Using the trained network for predictions.
+predictions = pygad.nn.predict(last_layer=output_layer, 
+                         data_inputs=data_inputs, 
+                         problem_type="regression")
+
+# Calculating some statistics
+abs_error = numpy.mean(numpy.abs(predictions - data_outputs))
+print("Absolute error : {abs_error}.".format(abs_error=abs_error))