// model evaluation & validation · theory
Evaluating a model is a very important step which is performed throughout the development of the model. Some methods such as the Artificial Neural Networks model do the evaluation when it performs backpropagation. However, we still perform the evaluation of a model manually through various methods. It is important to note that models can be evaluated successfully when working in a supervised learning environment as the actual values are available in order for the evaluation methods to function.
As seen in the modeling section, the models under supervised learning fall broadly under two categories - Regression problems and Classification problems. The methods of evaluating these models thus also fall under such two categories only: Evaluation of Regression Models and Evaluation of Classification problems. There is a fundamental difference between the methods used for evaluating a regression and classification model.
With regression, we deal with continuous values where the various techniques try to identify the error between the actual and predicted value. However, when we try to evaluate a classification model our focus is on the number of data points that are classified correctly. To evaluate a classification model correctly, we also have to consider the data points that were classified incorrectly. We also deal with two types of classification models: some produce class output such as KNN and SVM where the output is simply the class label, while others are probability producing models such as Logistic Regression and Random Forest, where the output is the probability of a data point belonging to a particular class, and through the use of a cut-off value we are able to convert these probabilities into class labels. A change in the cut-off value can alter the results of the evaluation and thus requires methods to figure out the correct threshold value.
Evaluation methods for Regression Models depend on the ways of calculating the difference between the actual and predicted values. There are methods such as the Sum Squared Error, Mean Squared Error and Root Mean Squared Error which are among the various ways of calculating this difference. On the other hand, there are more sophisticated methods such as Adjusted Coefficient of Determination which also take into account the problem of overfitting.
There are a lot of intriguing methods for evaluating a classification model. Among the most important is the confusion matrix. Then there are various methods that visually allow us to understand the performance of the model. Among such methods are the ROC and AUC curve that not only allow us to evaluate the model but also provide us with the much needed optimum cut-off value required for maximising the generalisation power of our model.
TM