I was building a machine learning model to predict patient outcomes when I hit a wall. The model was accurate, but I couldn’t answer a simple, critical question: “How sure are you?” Traditional confidence scores felt like a guess. This sent me searching for a method that could attach a reliable, guaranteed measure of uncertainty to any prediction, for any model. I found it in conformal prediction. If you’ve ever needed more than just a prediction, if you’ve ever needed to know how much to trust it, this is for you.
Why should we care about uncertainty? Think about a model that suggests a medical diagnosis or a stock trade. A wrong prediction with high, misplaced confidence is worse than no prediction at all. Conformal prediction provides a mathematical guarantee. If you ask for a 90% confidence interval, it ensures the true answer falls within that interval 90% of the time, on average. This isn’t based on complex assumptions about your data; it’s a distribution-free, practical guarantee.
Let’s break down the core idea. It works through a simple but powerful process. You split your data: one part to train your model, and a separate part to calibrate it. The calibration step is key. You use this held-out data to measure how “wrong” your model typically is. You then use this real-world error measurement to create intervals for new predictions. It’s like learning your model’s consistent margin of error and applying it honestly.
How does this look in practice? Let’s start with a regression problem, predicting a continuous value. Here’s a basic implementation from scratch to demystify the process. We’ll create a simple class.
import numpy as np
from sklearn.base import BaseEstimator
class SimpleConformalRegressor(BaseEstimator):
def __init__(self, model, alpha=0.1):
self.model = model
self.alpha = alpha # 0.1 means 90% coverage
self.calibration_scores = None
def fit(self, X_train, y_train, X_calib, y_calib):
self.model.fit(X_train, y_train)
# Get predictions on the calibration set
y_calib_pred = self.model.predict(X_calib)
# Calculate the absolute errors (our nonconformity scores)
self.calibration_scores = np.abs(y_calib - y_calib_pred)
# Find the threshold that ensures (1-alpha) coverage
n_calib = len(y_calib)
q_level = np.ceil((n_calib + 1) * (1 - self.alpha)) / n_calib
self.score_threshold = np.quantile(self.calibration_scores, q_level, method='higher')
return self
def predict(self, X):
point_pred = self.model.predict(X)
lower = point_pred - self.score_threshold
upper = point_pred + self.score_threshold
return point_pred, lower, upperSee? The logic is straightforward. We train a model, see how far off it is on calibration data, and then apply that maximum error margin to future predictions. This gives us an interval. The guarantee comes from the quantile calculation, which accounts for the finite sample size.
But what about classification? Instead of predicting a number, we’re choosing a label. Here, conformal prediction gives us a set of possible labels, not a single guess. For a new email, instead of just saying “spam,” it might say {“spam”, “important”} with 95% confidence. The set size reflects uncertainty—a single label means high confidence, multiple labels mean the model is less sure. Isn’t it more useful to know when the model is struggling between options?
Let’s implement a basic version. We’ll use the model’s predicted probabilities to build these sets.
class SimpleConformalClassifier(BaseEstimator):
def __init__(self, model, alpha=0.1):
self.model = model
self.alpha = alpha
self.thresholds = None
def fit(self, X_train, y_train, X_calib, y_calib):
self.model.fit(X_train, y_train)
# Get probability predictions for the calibration set
calib_probs = self.model.predict_proba(X_calib)
# Get the probability assigned to the true class for each calibration point
true_class_probs = calib_probs[np.arange(len(y_calib)), y_calib]
# Find the probability threshold for the desired coverage
n_calib = len(y_calib)
q_level = np.ceil((n_calib + 1) * (1 - self.alpha)) / n_calib
self.prob_threshold = np.quantile(true_class_probs, q_level, method='higher')
return self
def predict_with_sets(self, X):
probs = self.model.predict_proba(X)
# Include all labels with a probability higher than the threshold
prediction_sets = []
for sample_probs in probs:
set_for_sample = np.where(sample_probs >= self.prob_threshold)[0]
prediction_sets.append(set_for_sample)
return prediction_setsYou don’t have to build everything yourself. Excellent libraries like MAPIE in Python handle the heavy lifting and offer advanced methods. For example, achieving valid intervals with a complex model like a gradient booster is just a few lines.
from mapie.regression import MapieRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import fetch_california_housing
data = fetch_california_housing()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train, X_calib, y_train, y_calib = train_test_split(X_train, y_train, test_size=0.2, random_state=42)
model = GradientBoostingRegressor()
conformal_model = MapieRegressor(model, method="split", cv="prefit")
conformal_model.fit(X_train, y_train, X_calib=X_calib, y_calib=y_calib)
y_pred, y_pis = conformal_model.predict(X_test, alpha=0.1)
coverage = np.mean((y_test >= y_pis[:, 0, 0]) & (y_test <= y_pis[:, 1, 0]))
print(f"Target Coverage: 90%, Achieved Coverage: {coverage:.1%}")This is where the power becomes clear. You can wrap any fitted model and get rigorous intervals. It works with neural networks, random forests, or even a simple linear regression. The model makes the point prediction; conformal prediction outfits it with a reliable confidence interval.
Where does this matter most? Consider an AI system reviewing loan applications. A conformal set that often contains multiple outcomes flags that case for human review. In a medical triage system, a wide prediction interval for a patient’s deterioration risk prompts immediate clinical assessment. It turns black-box scores into actionable, trustworthy guidance.
A common question is about data exchangeability, the main assumption. It means the order of your data shouldn’t matter. While i.i.d. data is exchangeable, time-series data, where order is crucial, is not. For such cases, adaptive methods exist that adjust thresholds over time, but they require more care. Always think about whether your calibration data truly represents the future data you’ll see.
Some practical advice: your calibration set is vital. It must be representative and of reasonable size—a few hundred samples is often a good start. The intervals you get are only as good as this calibration step. Also, remember the guarantee is marginal; it holds on average over many predictions, not for each individual one.
I started using this because I needed honest uncertainty. It changed how I deploy models. Now, I don’t just send out predictions; I send out predictions with a clear, guaranteed confidence statement. This builds trust and enables smarter decision-making. The code is simple, the guarantee is strong, and the impact is immediate.
Have you ever presented a model’s output only to be asked about its confidence? That moment is why this matters. Give conformal prediction a try. Wrap it around your next model. See how it changes your perspective on what your model knows, and what it doesn’t.
If this approach to reliable AI resonates with you, please share your thoughts in the comments. What’s a situation where you wish you had this kind of guaranteed uncertainty? Like and share this if you found it useful for your own projects.
As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!
101 Books
101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.
Check out our book Golang Clean Code available on Amazon.
Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!
📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!
Our Creations
Be sure to check out our creations:
Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools
We are on Medium
Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva
