Evaluating Models
What is Evaluation
It is the process of checkming how well an AI model performs after it has been trained.
It helps us find out whether the model is giving correct and reliable predictions.
Why is Evaluation Important?
- To check if the model is working correctly.
- To identify errors made by the model.
- To compare different AI models.
- To improve the model's performance.
In Simple Words
Training = Teaching the model
Testing = Asking questions from the model
Evaluation = Checking how well the model answered
What is Train-Test Split?
- Train-Test Split is the process of dividing data into training and testing sets to build and evaluate an AI model.
- Training Data – Used to train the AI model.
- Testing Data – Used to test the AI model's performance.
Why is Train-Test Split Important?
- Prevents the model from simply memorizing data.
- Helps check how well the model works on new data.
- Provides a fair evaluation of the model.
Example
Suppose we have 100 student records.
DatasetNumber of RecordsTraining Data (80%)80Testing Data (20%)20
- The AI model learns from the 80 records.
- The remaining 20 records are used to test its performance.
Simple Analogy
Preparing for an Exam
📚 Training Data = Questions you practice before the exam.
📝 Testing Data = Questions that appear in the actual exam.
If you perform well in the exam, it means your learning was successful.
Evaluating Techniques in AI Models
1. Overfitting Model
An Overfitting Model learns the training data too well, including its mistakes and noise.
As a result, it performs very well on training data but poorly on new (test) data.
Characteristics
- High training accuracy ✔
- Low testing accuracy ✘
- Memorizes data instead of learning patterns
Example
A student memorizes all practice questions but cannot solve new questions in the exam.
Training Accuracy → High
Testing Accuracy → Low2. Underfitting Model
An Underfitting Model does not learn enough from the training data.
Therefore, it performs poorly on both training and testing data.
Characteristics
- Low training accuracy ✘
- Low testing accuracy ✘
- Fails to learn important patterns
Example
A student studies very little and performs poorly in both practice tests and the final exam.
Training Accuracy → Low
Testing Accuracy → Low3. Perfect Fit Model
A Perfect Fit Model learns the important patterns from the training data without memorizing it.
It performs well on both training and testing data.
Characteristics
- High training accuracy ✔
- High testing accuracy ✔
- Generalizes well to new data
A student understands concepts properly and performs well in both practice tests and the final exam.
Training Accuracy → High
Testing Accuracy → HighQuick SuExam Tip ⭐
Underfitting = Learns Too Little
Overfitting = Learns Too Much
Perfect Fit = Learns Just Right
Easy Example (Student Marks):
- Features: Study hours, attendance
- Label: Exam result (Pass/Fail)
- Labelled Data: Study hours + attendance + result
- Unlabelled Data: Study hours + attendance only
Evaluating Models
Accuracy and Error
What is Accuracy?
Accuracy tells us how many predictions made by the AI model are correct.
Definition
Accuracy is the percentage of correct predictions made by an AI model.
Formula
Accuracy = (Correct Predictions / Total Predictions ) X 100
Example
Suppose an AI model makes 100 predictions.
- Correct Predictions = 90
- Incorrect Predictions = 10
Accuracy = (90 ÷ 100) × 100 = 90%
Interpretation
- Higher Accuracy = Better Model ✔
- Lower Accuracy = Poor Model ✘
2. Error
What is Error?
Error tells us how many predictions made by the AI model are incorrect.
Easy Definition
Error is the percentage of incorrect predictions made by an AI model.
Formula
Error = (Incorrect Predictions / Total Predictions ) X 100
Evaluating Models
Relationship Between Accuracy and Error
Accuracy + Error = 100%
Example
- Accuracy = 90%
- Error = 10%
90% + 10% = 100%
Evaluating Models for Classification
What is Classification?
Classification is a type of AI task in which data is placed into predefined categories or classes.
Examples
- Email → Spam or Not Spam
- Animal → Cat or Dog
- Student Result → Pass or Fail
- Medical Report → Positive or Negative
Classification Metrics
Classification Metrics are used to evaluate the performance of a classification model.
The most common metrics are:
- Confusion Matrix
- Accuracy
- Precision
- Recall
What is a Confusion Matrix?
A Confusion Matrix is a table that shows the number of correct and incorrect predictions made by a classification model.
True Positive (TP)
Actual Positive and Predicted Positive.
Example: Sick person correctly identified as sick.
True Negative (TN)
Actual Negative and Predicted Negative.
Example: Healthy person correctly identified as healthy.
False Positive (FP)
Actual Negative but Predicted Positive.
Example: Healthy person wrongly identified as sick.
False Negative (FN)
Actual Positive but Predicted Negative.
Example: Sick person wrongly identified as healthy.
Evaluating Models
2. Classification Accuracy
What is Accuracy?
Accuracy measures the percentage of correct predictions made by the model.
Formula
Accuracy = (TP + TN) / (TP + TN + FP + FN)
Example
If a model correctly predicts 90 out of 100 cases:
Accuracy = 90%
Key Point
- Higher Accuracy = Better Performance
3. Precision
What is Precision?
Precision tells us how many of the predicted positive cases are actually positive.
Definition
Precision measures the correctness of positive predictions.
Formula
Precision = TP / (TP + FP)
Example
Suppose the model predicts 50 students will pass.
- Actually Passed = 45 (TP)
- Wrongly Predicted Pass = 5 (FP)
Precision = 45 / 50 = 0.90 (90%)
Key Point
- High Precision means fewer False Positives.
Learn with Orange Workflow
4. Recall
What is Recall?
Recall tells us how many actual positive cases were correctly identified by the model.
Easy Definition
Recall measures the model's ability to find all positive cases.
Formula
Recall = TP / (TP + FN)
Example
There are 50 actual positive cases.
- Correctly Identified = 45 (TP)
- Missed = 5 (FN)
Recall = 45 / 50 = 0.90 (90%)
Key Point
- High Recall means fewer False Negatives.
F1 Score
F1 Score is a metric that combines Precision and Recall into a single value.
It is used when we want a balance between Precision and Recall.
Definition
F1 Score is the harmonic mean of Precision and Recall.
Why is F1 Score Important?
Sometimes:
- Precision is high but Recall is low.
- Recall is high but Precision is low.
F1 Score helps us measure the overall performance by considering both together.
Formula
F1 Score = (2 X Precision X Recall) / (Precision + Recall)
Example
Suppose:
- Precision = 80% (0.8)
- Recall = 60% (0.6)
F1 Score = 2 × (0.8 × 0.6) ÷ (0.8 + 0.6)
F1 Score = 0.69 or 69%
Real-Life Example
Imagine an AI model that detects spam emails.
- Precision tells how many emails marked as spam are actually spam.
- Recall tells how many spam emails were successfully detected.
- F1 Score gives a balanced measure of both.