Standard error, T-stat, p value, F-stat for linear regression.

[fantested]

Can anyone please implement these?

[bwang22]

I second this

[JohnnyZhang0628]

Hello, Now how to calculation standard error of the coefficient in Polynomial Fitting?

[diluculo]

Here is the code for the polynomial regression:

Ref: https://www.originlab.com/doc/origin-Help/PR-Algorithm

EDIT:
t-value, probability, and confidence intervals are added.

// data and polynomial order are given by
double[] x = { 1, 2, 3, 4, 5 };
double[] y = { 6.2, 16.9, 33, 57.5, 82.5 };
var order = 2;

var Ns = x.Length;
var k = order + 1; // number of parameters
var dof = Ns - k; // degree of freedom

// Create the [Ns X k] design matrix, 
var X = Matrix<double>.Build.Dense(Ns, k, (i, j) => Math.Pow(x[i], j));

// Create the Y vector = [y0 y1 ...]
var Y = Vector<double>.Build.DenseOfArray(y);

// Calculate `covariance matrix`
var covariance = (X.Transpose() * X).Inverse();

// Calculate best-fitted parameters
var B = covariance * X.Transpose() * Y;
// Or use Fit.Polynomial
// var B = Vector<double>.Build.DenseOfArray(Fit.Polynomial(x, y, order));

// Calculate the residuals            
var residuals = X * B - Y;

// Calculate the `residual sum of squares`
var RSS = residuals.DotProduct(residuals);

// Calculate the `reduced Chi-square`
var reducedChiSquare = RSS / dof;

// Calculate the `residual standard deviation` (also called `standard error of estimate`, or `root mean square of the error`)
var rootMSE = Math.Sqrt(reducedChiSquare);
           
// Calculate the `standard errors of the coefficients`
var standardErrors = covariance.Diagonal().PointwiseSqrt().Multiply(rootMSE);

// Calculate the t-values and probabilities
var tValues = B.PointwiseDivide(standardErrors);
var pValues = tValues.Map(t => 2 * (1 - StudentT.CDF(0, 1, dof, Math.Abs(t))));

// Calculate the confidence intervals at 95% confidence level
double alpha = 0.05;
var LCL = B - standardErrors * StudentT.InvCDF(0, 1, dof, 1 - alpha / 2);
var UCL = B + standardErrors * StudentT.InvCDF(0, 1, dof, 1 - alpha / 2);
var CI = (UCL - LCL) / 2;

// Calculate the `total sum of square`
var yMean = Y.Average();
var TSS = Y.Select(v => Math.Pow(v - yMean, 2)).Sum();

// The goodness of fit can be evaluated by `Coefficient of Determination`(COD)
var Rsquare = 1.0 - RSS / TSS;

// The other statistics: R-Value, adjusted R square, and norm of residuals
var Rvalue = Math.Sqrt(Rsquare);
var adjRsquare = 1 - (1 - Rsquare) * (Ns - 1) / dof;
var normOfResiduals = Math.Sqrt(RSS);

// Y = B0 + B1*X + B2*X^2
// Parameter Value     Error     t-value    Pr(>|t|)    LCL         UCL         CI half_width
// --------------------------------------------------------------------------------------------
// B0        -0.24     3.07019   -0.07817   0.94481     -13.44995   12.96995    13.20995
// B1        3.46286   2.33969   1.48005    0.27700     -6.60401    13.52972    10.06686
// B2        2.64286   0.38258   6.90799    0.02032     0.99675     4.28897     1.64611
// --------------------------------------------------------------------------------------------
//
// Fit statistics
// -----------------------------------------
// Degree of freedom        2
// Reduced Chi-Sqr          2.04914
// Residual Sum of Sqaures  4.09829
// R Value                  0.99947
// R-Square(COD)            0.99893
// Adj. R-Square            0.99786
// Root-MSE(SD)             1.43148
// -----------------------------------------

[JohnnyZhang0628]

Thanks a lot.I try your code,it's all right.

[zzzaacchh]

Hello, I'm trying to get the same values for multiple linear regression. Can you please help?

[zzzaacchh]

Never mind, I figured it out and was able to match the output with the output from SPSS. This library is amazing, by the way, thank you. Here's the code for linear regression for anyone curious.

            // Enter predictor variables as an array of arrays, and dependent variables as an array
            double[][] x = { new[] { 1.75, 3.25, 2.10 }, new[] { 2.4, 4.3, 5.6 }, new[] { 3.1, 4.2, 3.3 }, new[] { 5.1, 3.5, 8.2 }, new[] { 24.7, 12.3, 19.7 } 
            };
            double[] y = { 17.2, 22.3, 10.10, 30.60, 111.21 };

            var N = x.Length; // number of observations
            var p = x[0].Length + 1; // number of parameters (+1 when adding intercept)

            var dfR = p - 1; // degree of freedom (Regression/Model)
            var dfE = N - p; // degree of freedom (Error)
            var dfT = N - 1; // degree of freedom (Total)

            // Create design matrix 
            var X = Matrix<double>.Build.DenseOfRowArrays(x);

            // Create the Y vector = [y0 y1 ...]
            var Y = Vector<double>.Build.DenseOfArray(y);

            // Augment design matrix for Y intercept with a column of 1s, and calculate `covariance matrix`
            var Xa = X.InsertColumn(0, Vector<double>.Build.DenseOfArray(addColumnToMatrix(N)));
            var covariance = (Xa.Transpose() * Xa).Inverse();

            // Calculate best-fitted parameters
            var B = Vector<double>.Build.DenseOfArray(Fit.MultiDim(x, y, true));

            // Calculate the predicted values and residuals: multiply augmented design matrix by coefficients, then subtract from Y       
            var predicted = Xa * B;
            var residuals = Y - predicted;

            // Calculate the `residual sum of squares`
            var RSS = residuals.DotProduct(residuals);

            // Calculate the `reduced Chi-square` (mean square of the error)
            var MSE = RSS / dfE;

            // Calculate the `residual standard deviation` (also called `standard error of estimate`, or `root mean square of the error`)
            var rootMSE = Math.Sqrt(MSE);

            // Calculate the `standard errors of the coefficients`
            var standardErrors = covariance.Diagonal().PointwiseSqrt().Multiply(rootMSE);

            // Calculate the t-values and probabilities
            var tValues = B.PointwiseDivide(standardErrors);
            var pValues = tValues.Map(t => 2 * (1 - StudentT.CDF(0, 1, dfE, Math.Abs(t))));

            // Calculate the confidence intervals at 95% confidence level
            double alpha = 0.05;
            var LCL = B - standardErrors * StudentT.InvCDF(0, 1, dfE, 1 - alpha / 2);
            var UCL = B + standardErrors * StudentT.InvCDF(0, 1, dfE, 1 - alpha / 2);
            var CI = (UCL - LCL) / 2;

            // Calculate the `total sum of square`
            var yMean = Y.Average();
            var TSS = Y.Select(v => Math.Pow(v - yMean, 2)).Sum();

            // The goodness of fit can be evaluated by `Coefficient of Determination`(COD)
            var Rsquare = 1.0 - RSS / TSS;

            // The other statistics: R-Value, adjusted R square, and norm of residuals (standard error of the estimate)
            var Rvalue = Math.Sqrt(Rsquare);
            var adjRsquare = 1 - (1 - Rsquare) * (N - 1) / dfE;
            var normOfResiduals = Math.Sqrt(RSS);

            // Calculate Mean square of regression/model
            var MSR = (TSS - RSS) / dfR;

            // Calculate F statistic
            var F = MSR / MSE;

            // Calculate p value from F statistic
            var sig = 1 - FisherSnedecor.CDF(dfR, dfE, F);

            private double[] addColumnToMatrix(int length)
            {
                var array = new double[length];
                for (int i = 0; i < length; i++)
                    array[i] = 1;            

                return array;
            }