What is a confusion matrix and how do you interpret it

A confusion matrix is a table used to evaluate the performance of a classification model. It provides a detailed breakdown of the model’s predictions compared to the actual class labels, allowing you to assess various aspects of its performance, such as accuracy, precision, recall, and more.

Components of a Confusion Matrix

For a binary classification problem, the confusion matrix has four components:

1. True Positive (TP): The number of instances that are correctly classified as positive.
2. True Negative (TN): The number of instances that are correctly classified as negative.
3. False Positive (FP): The number of instances that are incorrectly classified as positive (Type I error).
4. False Negative (FN): The number of instances that are incorrectly classified as negative (Type II error).

Here's what a confusion matrix typically looks like for binary classification:

Interpreting the Confusion Matrix

1. Accuracy:

   • Definition: The ratio of correctly predicted instances to the total number of instances.
   • Formula: Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN?
   • Use: Gives an overall measure of how well the model performs.

2. Precision (Positive Predictive Value):

   • Definition: The ratio of correctly predicted positive observations to the total predicted positives.
   • Formula: Precision=TPTP+FP\text{Precision} = \frac{TP}{TP + FP}Precision=TP+FPTP?
   • Use: Indicates how many of the predicted positives are actually positive. Important when the cost of false positives is high.

3. Recall (Sensitivity, True Positive Rate):

   • Definition: The ratio of correctly predicted positive observations to all actual positives.
   • Formula: Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}Recall=TP+FNTP?
   • Use: Indicates how many of the actual positives are correctly identified. Important when the cost of false negatives is high.

4. F1 Score:

   • Definition: The harmonic mean of precision and recall.
   • Formula: F1 Score=2⋅Precision⋅RecallPrecision+Recall\text{F1 Score} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}F1 Score=2⋅Precision+RecallPrecision⋅Recall?
   • Use: Provides a single metric that balances precision and recall. Useful when you need to balance false positives and false negatives.

5. Specificity (True Negative Rate):

   • Definition: The ratio of correctly predicted negative observations to all actual negatives.
   • Formula: Specificity=TNTN+FP\text{Specificity} = \frac{TN}{TN + FP}Specificity=TN+FPTN?
   • Use: Measures how well the model identifies negative instances. Important in cases where the cost of false positives is high.

6. False Positive Rate (FPR):

   • Definition: The ratio of false positives to the total actual negatives.
   • Formula: FPR=FPTN+FP\text{FPR} = \frac{FP}{TN + FP}FPR=TN+FPFP?
   • Use: Indicates how often the model incorrectly classifies negatives as positives.

7. False Negative Rate (FNR):

   • Definition: The ratio of false negatives to the total actual positives.
   • Formula: FNR=FNTP+FN\text{FNR} = \frac{FN}{TP + FN}FNR=TP+FNFN?
   • Use: Indicates how often the model fails to identify positives.

Example Scenario

Imagine you have a binary classification model that predicts whether an email is spam or not. After evaluating the model, you get the following confusion matrix:

From this confusion matrix:

• TP = 80 (spam correctly predicted as spam)

• TN = 85 (not spam correctly predicted as not spam)

• FP = 15 (not spam incorrectly predicted as spam)

• FN = 20 (spam incorrectly predicted as not spam)

• Accuracy = 80+8580+85+15+20=165200=0.825\frac{80 + 85}{80 + 85 + 15 + 20} = \frac{165}{200} = 0.82580+85+15+2080+85?=200165?=0.825 or 82.5%

• Precision = 8080+15=8095≈0.842\frac{80}{80 + 15} = \frac{80}{95} \approx 0.84280+1580?=9580?≈0.842 or 84.2%

• Recall = 8080+20=80100=0.8\frac{80}{80 + 20} = \frac{80}{100} = 0.880+2080?=10080?=0.8 or 80%

• F1 Score = 2⋅0.842⋅0.80.842+0.8≈0.8212 \cdot \frac{0.842 \cdot 0.8}{0.842 + 0.8} \approx 0.8212⋅0.842+0.80.842⋅0.8?≈0.821 or 82.1%

• Specificity = 8585+15=85100=0.85\frac{85}{85 + 15} = \frac{85}{100} = 0.8585+1585?=10085?=0.85 or 85%

• FPR = 1585+15=15100=0.15\frac{15}{85 + 15} = \frac{15}{100} = 0.1585+1515?=10015?=0.15 or 15%

• FNR = 2080+20=20100=0.2\frac{20}{80 + 20} = \frac{20}{100} = 0.280+2020?=10020?=0.2 or 20%

Summary

The confusion matrix provides a comprehensive view of a classification model's performance, highlighting how well the model predicts each class and where it makes errors. By examining the confusion matrix, you can derive several important metrics to assess and improve the model’s performance.