Evaluate a model and decide what to do next.

Model Evaluation

Train Test Split

Split the dataset to: 70% for training and 30% for testing.

Train/test Procedure for Linear Regression

Fit params by minimizing const function J (contains regularization term) $$ J(\mathbf{w},b) = \frac{1}{2m_{train}} \sum\limits_{i = 1}^{m_{train}} (f_{\mathbf{w},b}(\mathbf{x}^{(i)}) - y^{(i)})^2 + \frac{\lambda}{2m_{train}} \sum_{j=1}^{n} w_j^2 $$

Compute Test Error: $$ J_{test}(\mathbf{w},b) = \frac{1}{2m_{test}} \sum\limits_{i = 1}^{m_{test}} (f_{\mathbf{w},b}(\mathbf{x_{test}}^{(i)}) - {y_{test}}^{(i)})^2 $$

Compute Train Error: $$ J_{train}(\mathbf{w},b) = \frac{1}{2m_{train}} \sum\limits_{i = 1}^{m_{train}} (f_{\mathbf{w},b}(\mathbf{x_{train}}^{(i)}) - {y_{train}}^{(i)})^2 $$

$J_{train}(\mathbf{w},b)$ will be low, and $J_{test}(\mathbf{w},b)$ will be high.

Train/test Procedure for Logistic Regression

Fit params by minimizing const function J (contains regularization term) $$ J(\vec{w},b) = -\frac{1}{m_{train}} \sum\limits_{i = 1}^{m_{train}} [y^{(i)}\log(f_{\vec{w}, b}(\vec{x}^{(i)})) + (1 - y^{(i)})\log(1 - f_{\vec{w}, b}(\vec{x}^{(i)}))] + \frac{\lambda}{2m_{train}}\sum_{j=1}^{n} w_j^2 $$

Compute Test Error: $$ J_{text}(\vec{w},b) = -\frac{1}{m_{test}} \sum\limits_{i = 1}^{m_{test}} [y_{test}^{(i)} \log(f_{\vec{w}, b}(\vec{x_{test}}^{(i)})) + (1 - y_{test}^{(i)})\log(1 - f_{\vec{w}, b}(\vec{x}_{test}^{(i)}))] $$

Compute Train Error: $$ J_{train}(\vec{w},b) = -\frac{1}{m_{train}} \sum\limits_{i = 1}^{m_{train}} [y_{train}^{(i)}\log(f_{\vec{w}, b}(\vec{x_{train}}^{(i)})) + (1 - y_{train}^{(i)})\log(1 - f_{\vec{w}, b}(\vec{x}_{train}^{(i)}))] $$

Common Method for train and test error in classification model: $J_{test}(\vec{w}, b)$ is the fraction of the test set that has been misclassified. $J_{train}(\vec{w}, b)$ is the fraction of the train set that has been misclassified.

Model Selection

Train Cross-Validation Test Split

Split dataset into 60% for training, 20% for cross-validation and 20% for testing.

Evaluate and Choose

Train all models on the training set.

Evaluate all models on cross-validation set and pick the best one.

Then give a fair estimate value of accuracy of the chosen model using test set.