// model evaluation & validation · application
Python is used to put both Model Evaluation and Model Validation techniques into practice, using libraries such as sklearn for computing evaluation metrics and for running cross-validation.
For Model Evaluation, regression models are evaluated by importing metrics from sklearn and using numpy to calculate values such as the RMSE, while classification models (using a Logistic Regression model fit on a defaulters dataset) are evaluated using metrics from sklearn along with the seaborn package, which helps visualise the confusion matrix.
For Model Validation, the library KFold found in sklearn.model_selection is used to run k-fold cross-validation, along with cross_val_score, which allows validation in a single function call. GridSearchCV is used for tuning hyperparameters while validating, and ShuffleSplit and StratifiedKFold (the latter only for classification problems) are explored as variations of the standard k-fold approach. A RandomForestRegressor from sklearn.ensemble is used to demonstrate nested cross-validation.
TM