glossary terms
Accuracy
- Category
- Evaluation, Safety & Governance
- Difficulty
- Beginner
Definition
In machine learning, accuracy is a performance metric defined as the ratio of correctly predicted observations to the total number of observations in a dataset.
How It Works and Context
Accuracy is the most straightforward metric for evaluating classification models, calculated by dividing the number of correct predictions (true positives and true negatives) by the total number of predictions. Despite its simplicity, accuracy is frequently insufficient for real-world AI systems. A primary limitation is its failure to account for class imbalance; for example, in a medical diagnostic model where 99% of patients are healthy, a model that predicts 'healthy' for everyone will achieve 99% accuracy while failing to identify any sick patients. Consequently, practitioners often supplement accuracy with precision, recall, and F1-score to gain a more nuanced understanding of model behavior. Furthermore, accuracy treats all errors as equally significant, which is rarely the case in safety-critical applications like autonomous driving or financial fraud detection, where a false negative may be far more costly than a false positive.
Why It Matters
Accuracy provides a quick, high-level snapshot of model performance, making it a useful baseline for initial testing. However, relying solely on accuracy can lead to dangerous overconfidence in AI systems. Understanding its limitations is critical for developers and stakeholders to ensure models are not just 'correct' on average, but reliable in the specific, often rare, edge cases that define real-world safety and utility.
Real-world Example
Consider a spam filter trained on a dataset where 95% of emails are legitimate and 5% are spam. If the model is poorly trained and simply labels every incoming email as 'not spam,' it will achieve 95% accuracy. While this number sounds high, the model is useless because it fails to catch any spam, demonstrating why accuracy alone is a poor indicator of performance in imbalanced datasets.
Common Mistakes
- Assuming high accuracy implies the model is safe or reliable for all use cases.
- Using accuracy as the primary metric for datasets with significant class imbalance.
- Ignoring the cost of different types of errors (false positives vs. false negatives) by focusing only on the aggregate accuracy score.
- Failing to validate accuracy across different subsets or slices of data.
Frequently Asked Questions
When should I avoid using accuracy as a metric?
You should avoid accuracy whenever your dataset is imbalanced, meaning one class appears much more frequently than others, or when the cost of different types of errors is not equal.
How can I improve my model if accuracy is high but performance is poor?
You should switch to more granular metrics like precision, recall, or the F1-score, and perform error analysis to identify which specific classes or scenarios the model is failing to predict correctly.
Is accuracy the same as precision?
No. Accuracy measures the proportion of total correct predictions, whereas precision measures the proportion of positive predictions that were actually correct.