Data Mining by Mehmed Kantardzic (good book recommendations TXT) 📕

(X1 = X, X2 = X2, and X3 = X3), it is possible to linearize the model and transform it into a multiple-regression problem, which can be solved by the method of least squares. It should be noted that the term linear in the general linear model applies to the dependent variable being a linear function of the unknown parameters. Thus, a general linear model might also include some higher order terms of independent variables, for example, terms such as X12, eβX, X1·X2, 1/X, or X23. The basis is, however, to select the proper transformation of input variables or their combinations. Some useful transformations for linearization of the regression model are given in Table 5.3.

TABLE 5.3. Some Useful Transformations to Linearize RegressionFunctionProper TransformationForm of Simple  Linear RegressionExponential:Y = α e βxY* = ln YRegress Y* against xPower:Y = α xβY* = logY; x* = log xRegress Y* against x*Reciprocal:Y = α + β(1/x)x* = 1/xRegress Y against x*Hyperbolic:Y = x/(α + βx)Y* = 1/Y; x* = 1/xRegress Y* against x*

The major effort, on the part of a user, in applying multiple-regression techniques lies in identifying the relevant independent variables from the initial set and in selecting the regression model using only relevant variables. Two general approaches are common for this task:

1. Sequential Search Approach. It is consists primarily of building a regression model with an initial set of variables and then selectively adding or deleting variables until some overall criterion is satisfied or optimized.

2. Combinatorial Approach. It is, in essence, a brute-force approach, where the search is performed across all possible combinations of independent variables to determine the best regression model.

Irrespective of whether the sequential or combinatorial approach is used, the maximum benefit to model building occurs from a proper understanding of the application domain.

Additional postprocessing steps may estimate the quality of the linear regression model. Correlation analysis attempts to measure the strength of a relationship between two variables (in our case this relationship is expressed through the linear regression equation). One parameter, which shows this strength of linear association between two variables by means of a single number, is called a correlation coefficient r. Its computation requires some intermediate results in a regression analysis.

where

The value of r is between −1 and 1. Negative values for r correspond to regression lines with negative slopes and a positive r shows a positive slope. We must be very careful in interpreting the r value. For example, values of r equal to 0.3 and 0.6 only mean that we have two positive correlations, the second somewhat stronger than the first. It is wrong to conclude that r = 0.6 indicates a linear relationship twice as strong as that indicated by the value r = 0.3.

For our simple example of linear regression given at the beginning of this section, the model obtained was B = 0.8 + 0.92A. We may estimate the quality of the model using the correlation coefficient r as a measure. Based on the available data in Figure 4.3, we obtained intermediate results

and the final correlation coefficient:

A correlation coefficient r = 0.85 indicates a good linear relationship between two variables. Additional interpretation is possible. Because r2 = 0.72, we can say that approximately 72% of the variations in the values of B is accounted for by a linear relationship with A.

5.5 ANOVA

Often the problem of analyzing the quality of the estimated regression line and the influence of the independent variables on the final regression is handled through an ANOVA approach. This is a procedure where the total variation in the dependent variable is subdivided into meaningful components that are then observed and treated in a systematic fashion. ANOVA is a powerful tool that is used in many data-mining applications.

ANOVA is primarily a method of identifying which of the β’s in a linear regression model are nonzero. Suppose that the β parameters have already been estimated by the least-square error algorithm. Then the residuals are differences between the observed output values and the fitted values:

The size of the residuals, for all m samples in a data set, is related to the size of variance σ2 and it can be estimated by:

assuming that the model is not over-parametrized. The numerator is called the residual sum while the denominator is called the residual degree of freedom (d.f.).

The key fact about S2 is that it allows us to compare different linear models. If the fitted model is adequate, then S2 is a good estimate of σ2. If the fitted model includes redundant terms (some β’s are really 0), S2 is still good and close to σ2. Only if the fitted model does not include one or more of the inputs that it ought to, will S2 tend to be significantly larger than the true value of σ2. These criteria are basic decision steps in the ANOVA algorithm, in which we analyze the influence of input variables on a final model. First, we start with all inputs and compute S2 for this model. Then, we omit inputs from the model one by one. If we omit a useful input the estimate S2 will significantly increase, but if we omit a redundant input the estimate should not change much. Note that omitting one of the inputs from the model is equivalent to forcing the corresponding β to the 0. In principle, in each iteration we compare two S2 values and analyze the differences between them. For this purpose, we introduce an F-ratio or F-statistic test in the form

If the new model (after removing one or more inputs) is adequate, then F will be close to 1; a value of F significantly larger than one will signal that the model is not adequate. Using this iterative ANOVA approach, we can identify which inputs are related to the output and which are not. The ANOVA procedure is only valid if the models being compared are nested; in other words,