This GPGPU library uses multiple GPUs to train small-scale neural-networks with variable number of fully connected layers, fast and in a simple way.
- Vcpkg (for OpenCL)
- OpenCL (for GPGPU in CPUs, GPUs and FPGAs)
- C++17
It works on parallelized simulated annealing algorithm which supports load-balancing between multiple GPUs and CPUs. Every pipeline of GPU runs same neural network but with different input-output data pair. This makes single-pass training for whole data set with thousands of simulations in parallel. But parameters are stored on in-chip fast memory so the number of parameters can not exceed this small amount (sometimes as low as 64kB).
Class GPGPU::FastSimpleNeuralNetworkTrainer takes multiple template parameters starting with number of parallel simulations, ending with topology of the neural network to train & infer. To create a neural network with two hidden layers(of 10 neurons each), 1 input, 2 outputs following code is valid:
// 500 simulations in parallel, each with 1:10:10:2 topology & different parameters
GPGPU::FastSimpleNeuralNetworkTrainer<500, 1, 10, 10, 2> nn;then training data needs to be prepared:
// 1 input element, 2 output elements
GPGPU::TrainingData<1, 2> td;
// x and y are vectors for 1 input element, 2 output elements for network
td.AddInputOutputPair(x, y);
td.AddInputOutputPair(x2, y2);once data is prepared, training can start (it takes a callback function that is called whenever a better solution is found, until simulation ends):
std::vector<float> testInput = { 0.5f };
auto model = nn.Train(td, testInput, [testInput](std::vector<float> testOutput)
{
std::cout << "training: now square root of " << testInput[0] << " is " << testOutput[0] << std::endl;
});finally, the trained network can start computing test data:
auto result = model.Run({ 0.49 });
std::cout << "Square root of 0.49 = " << result[0] << std::endl;Following square-root approximation training completes in ~1.5 seconds using two GPUs (~10000 CUDA cores) with 1 : 10 : 20 : 10 : 1 neural network topology
#include <iostream>
#include"FastSimpleNeuralNetworkTrainer.h"
int main()
{
constexpr int numParallelSimulations = 1500;
constexpr int numInputs = 1;
constexpr int numOutputs = 1;
const int numThreadsPerSimulation = 256;
const float parameterScaling =1.1f;
//const GPGPU::ActivationFunction activation("sin(x)", [](float x) { return sin(x); });
//const GPGPU::ActivationFunction activation("tanh(x)", [](float x) { return tanh(x); });
//const GPGPU::ActivationFunction activation("x/(1.0f + exp(-x))", [](float x) { return x / (1.0f + exp(-x)); });
//const GPGPU::ActivationFunction activation("exp(-x*x)", [](float x) { return exp(-x*x); });
//const GPGPU::ActivationFunction activation("4.0f*tanh(x)", [](float x) { return 4.0f*tanh(x); });
//const GPGPU::ActivationFunction activation("exp(x)", [](float x) { return exp(x); });
const GPGPU::ActivationFunction activation("x*exp(x)", [](float x) { return x*exp(x); });
GPGPU::FastSimpleNeuralNetworkTrainer<numParallelSimulations, numInputs,10, 20, 10, numOutputs> nn(
numThreadsPerSimulation, parameterScaling, activation
);
GPGPU::TrainingData<numInputs, numOutputs> td;
// training data for: y = sqrt(x)
constexpr int numData =256;
for (int i = 0; i < numData; i++)
{
std::vector<float> x(numInputs);
std::vector<float> y(numOutputs);
// x
x[0] = i / (float)numData;
// weighting close-to-zero x values more because of floating-point accuracy
// y
y[0] = std::sqrt(x[0]);
td.AddInputOutputPair(x, y);
}
// neural network learns how to compute y = sqrt(x)
// more data = better fit, too much data = overfit, less data = generalization, too few data = not learning good
std::vector<float> testInput = { 0.5f };
float startTemperature = 1.0f;
float stopTemperature = 0.0001f;
float coolingRate = 1.1f;
int numRepeats = 5;
auto model = nn.Train(td, testInput, [testInput](std::vector<float> testOutput)
{
std::cout << "training: now square root of " << testInput[0] << " is " << testOutput[0] << std::endl;
}, startTemperature, stopTemperature, coolingRate, numRepeats);
double errPercent = 0.0;
double errTotal = 0.0;
double errMax = 0.0;
int ctr = 0;
for (double i = 0.00001; i < 1.0; i += 0.00001)
{
auto result = model.Run({ (float)i });
double err = std::abs(result[0] - sqrt(i));
errPercent += 100.0 * err / std::abs(sqrt(i));
errTotal += err;
if (errMax < err)
errMax = err;
ctr++;
}
std::cout << ctr << " samples between [0,1] have " << errPercent / ctr << "% average error, " << errTotal << " total error, "<<errMax<<" maximum error." << std::endl;
return 0;
}output:
lower energy found: 0.00489567
training: now square root of 0.5 is 0.704874
lower energy found: 0.00489242
training: now square root of 0.5 is 0.705003
lower energy found: 0.0048903
training: now square root of 0.5 is 0.705251
lower energy found: 0.00488862
training: now square root of 0.5 is 0.705173
total computation-time=1.48203 seconds (this includes debugging console-output that is slow)
---------------
OpenCL device info:
NVIDIA GeForce RTX 4070 computed 40.8667% of total work
NVIDIA GeForce RTX 4060 Ti computed 25.7333% of total work
NVIDIA GeForce RTX 4070 computed 21.5333% of total work
NVIDIA GeForce RTX 4060 Ti computed 11.8667% of total work
---------------
100000 samples between [0,1] have 0.708837% average error, 295.043 total error, 0.0253749 maximum error.
#include <iostream>
#include"FastSimpleNeuralNetworkTrainer.h"
int main()
{
constexpr int numParallelSimulations = 5000;
constexpr int numInputs = 2;
constexpr int numOutputs = 1;
GPGPU::ActivationFunction act("x*exp(x)", [](float x) {return x * exp(x); });
GPGPU::FastSimpleNeuralNetworkTrainer<numParallelSimulations, numInputs, 10, 20, 10, numOutputs> nn(256,1.1f,act);
GPGPU::TrainingData<numInputs, numOutputs> td;
// training data for: y = x1 or x2 where 1 means x>=0.5 and 0 means x<0.5
constexpr int numDataX = 32;
constexpr int numDataY = 32;
for (int i = 0; i < numDataX * numDataY; i++)
{
std::vector<float> x(numInputs);
std::vector<float> y(numOutputs);
x[0] = (i % numDataX) / (float)numDataX;
x[1] = (i / numDataX) / (float)numDataY;
y[0] = x[0] >= 0.5f or x[1] >= 0.5f;
td.AddInputOutputPair(x, y);
}
// neural network learns how to compute y = x1 or x2
// more data = better fit, too much data = overfit, less data = generalization, too few data = not learning good
std::vector<float> testInput = { 0.75f, 0.75f };
auto model = nn.Train(td, testInput, [testInput](std::vector<float> testOutput)
{
std::cout << "training: now " << (testInput[0] >= 0.5f) << " or " << (testInput[1] >= 0.5f) << " is " << (testOutput[0] >= 0.5f) << std::endl;
});
auto result = model.Run({ 0.3f, 0.3f });
std::cout << " 0 or 0 = " << (result[0] >= 0.5f) << std::endl;
result = model.Run({ 0.3f, 0.6f });
std::cout << " 0 or 1 = " << (result[0] >= 0.5f) << std::endl;
result = model.Run({ 0.6f, 0.3f });
std::cout << " 1 or 0 = " << (result[0] >= 0.5f) << std::endl;
result = model.Run({ 0.6f, 0.6f });
std::cout << " 1 or 1 = " << (result[0] >= 0.5f) << std::endl;
{
constexpr int numDataX = 500;
constexpr int numDataY = 500;
int err = 0;
for (int i = 0; i < numDataX * numDataY; i++)
{
std::vector<float> x(numInputs);
std::vector<float> y(numOutputs);
x[0] = (i % numDataX) / (float)numDataX;
x[1] = (i / numDataX) / (float)numDataY;
y[0] = x[0] >= 0.5f or x[1] >= 0.5f;
auto z = model.Run(x);
if (std::abs(y[0] - z[0]) > 0.01)
{
err++;
}
}
if (err > 0)
{
std::cout << "error: found training error. "<< numDataX * numDataX<<" samples have "<<err<<" errors" << std::endl;
}
}
return 0;
}
output:
lower energy found: 6.75636e-06
training: now 1 or 1 is 1
lower energy found: 6.72371e-06
training: now 1 or 1 is 1
lower energy found: 6.57961e-06
training: now 1 or 1 is 1
total computation-time=7.58356 seconds (this includes debugging console-output that is slow)
---------------
OpenCL device info:
NVIDIA GeForce RTX 4070 computed 25.9% of total work
NVIDIA GeForce RTX 4060 Ti computed 20.96% of total work
---------------
0 or 0 = 0
0 or 1 = 1
1 or 0 = 1
1 or 1 = 1
error: found training error. 250000 samples have 3907 errors
#include <iostream>
#include<algorithm>
#include <random>
#include"FastSimpleNeuralNetworkTrainer.h"
int main()
{
constexpr int numParallelSimulations =500;
constexpr int numInputs = 3;
constexpr int numOutputs = 3;
const GPGPU::ActivationFunction act("tanh(x)", [](float x) { return tanh(x); });
GPGPU::FastSimpleNeuralNetworkTrainer<numParallelSimulations, numInputs, 10, 20, 10, numOutputs> nn(256,1.1f,act);
GPGPU::TrainingData<numInputs, numOutputs> td;
std::random_device rd;
std::mt19937 rng{ rd() };
std::uniform_real_distribution<float> val(0, 1);
// training data for: unsorted array ==> sorted array
constexpr int numData = 4000;
for (int i = 0; i < numData; i++)
{
std::vector<float> x(numInputs);
x[0] = val(rng);
x[1] = val(rng);
x[2] = val(rng);
std::vector<float> y = x;
std::sort(y.begin(), y.end());
td.AddInputOutputPair(x, y);
}
// neural network learns how to sort an array
std::vector<float> testInput = { 0.75f, 0.45f, 0.9f };
auto model = nn.Train(td, testInput, [testInput](std::vector<float> testOutput)
{
std::cout << "training: { 0.75f, 0.45f, 0.9f } ==> {" << testOutput[0] << "," << testOutput[1] << "," << testOutput[2] << "}" << std::endl;
},1,0.00001f,1.05f,500);
for (int i = 0; i < 20; i++)
{
float v1, v2, v3;
auto result = model.Run({ v1 = val(rng), v2 = val(rng), v3 = val(rng) });
std::cout << v1 << " " << v2 << " " << v3 << " = > " << result[0] << " " << result[1] << " " << result[2] << std::endl;
}
return 0;
}
output:
training: { 0.75f, 0.45f, 0.9f } ==> {0.44334,0.764172,0.896679}
lower energy found: 2.38113
training: { 0.75f, 0.45f, 0.9f } ==> {0.443298,0.76415,0.896682}
reheating. num reheats left=1
lower energy found: 2.38113
training: { 0.75f, 0.45f, 0.9f } ==> {0.443315,0.764151,0.896683}
lower energy found: 2.38112
training: { 0.75f, 0.45f, 0.9f } ==> {0.443317,0.764172,0.896672}
total computation-time=1123.08 seconds (this includes debugging console-output that is slow)
---------------
OpenCL device info:
NVIDIA GeForce RTX 4070 computed 26.2% of total work
NVIDIA GeForce RTX 4060 Ti computed 22.8% of total work
NVIDIA GeForce RTX 4070 computed 29.4% of total work
NVIDIA GeForce RTX 4060 Ti computed 21.6% of total work
---------------
0.0581711 0.0220006 0.347504 = > 0.0141105 0.0670803 0.346016
0.0576147 0.0856164 0.0753108 = > 0.0212964 0.0836086 0.120738
0.699539 0.761743 0.32585 = > 0.329718 0.701058 0.75616
0.644279 0.929641 0.146303 = > 0.13175 0.661939 0.917223
0.43991 0.503481 0.939782 = > 0.436406 0.490325 0.935676
0.855676 0.365408 0.30679 = > 0.305092 0.371161 0.869841
0.928948 0.198322 0.144337 = > 0.135829 0.208741 0.928802
0.295339 0.467584 0.216512 = > 0.216227 0.298812 0.45997
0.988677 0.958554 0.206755 = > 0.184551 0.907675 0.973348
0.956393 0.0890611 0.482762 = > 0.0811025 0.469526 0.953046
0.766326 0.209337 0.368451 = > 0.220405 0.360087 0.774145
0.741995 0.111297 0.623226 = > 0.113182 0.629861 0.727476
0.461 0.722884 0.0677963 = > 0.0647887 0.47715 0.713786
0.773039 0.464691 0.795905 = > 0.453112 0.751761 0.81975
0.441087 0.440932 0.0803012 = > 0.0680988 0.417058 0.475103
0.39298 0.905184 0.489139 = > 0.402733 0.482062 0.916494
0.221201 0.808543 0.310247 = > 0.223572 0.300826 0.815217
0.278789 0.311042 0.451181 = > 0.267732 0.338308 0.42525
0.305794 0.937993 0.220696 = > 0.221226 0.312084 0.94049
0.43699 0.432962 0.0989809 = > 0.0895862 0.408906 0.467968