I was trying to build an image search tool for a client. They had thousands of product images, but only a handful were labeled. Training a standard classifier was impossible—we simply didn’t have enough “cat photos” to teach the system what a cat was. This roadblock is common. It led me to a different kind of model, one that learns the essence of similarity itself. Instead of asking “what is this?”, it learns to answer “are these two things the same?” This approach is incredibly powerful when data is scarce.
Think about how you recognize a friend’s face. You don’t mentally compare it against millions of other faces you’ve seen; you have a mental model of their features. When you see them, you check for a match against that model. This is the core idea behind Siamese networks. They learn a flexible model of similarity that can be applied to new, unseen categories with very few examples.
So, how does it actually work? A Siamese network uses two identical neural networks, often called “twin towers.” They share the exact same weights. You feed two images in—one is an “anchor” (your reference), and the other is either a “positive” (a matching image) or a “negative” (a different image). The networks don’t output a class label. Instead, they each produce a compact numerical summary called an embedding.
Imagine converting a face into a unique 128-number code. The magic happens in the training. We train the network so that the codes for two images of the same person are very close together in a mathematical space, while codes for different people are far apart. Have you ever considered how a system learns to place things closer or farther apart without explicit rules? It uses a clever function called a loss function to guide it.
For Siamese networks, one of the most effective guides is the Triplet Loss. It doesn’t just look at a pair; it looks at three images at once: an Anchor (A), a Positive (P) of the same class, and a Negative (N) of a different class. The learning objective is simple: the distance between A and P should be smaller than the distance between A and N by at least a certain amount, called a margin.
Here’s a basic code outline of what that looks like in PyTorch:
import torch
import torch.nn.functional as F
def triplet_loss(anchor, positive, negative, margin=1.0):
"""
anchor, positive, negative: Embeddings from the Siamese network.
margin: How much farther the negative should be than the positive.
"""
pos_distance = F.pairwise_distance(anchor, positive)
neg_distance = F.pairwise_distance(anchor, negative)
# The core triplet loss formula
loss = torch.relu(pos_distance - neg_distance + margin)
return loss.mean()The torch.relu ensures we only penalize the network if the negative is not sufficiently far away (i.e., if pos_distance + margin > neg_distance). Otherwise, the loss is zero—the network has already solved that triplet. This pushes embeddings to organize themselves in space meaningfully.
Building the network itself is straightforward. You start with a standard feature extractor, like a ResNet, and replace its final classification layer with a new layer that outputs your embedding vector.
import torch.nn as nn
import torchvision.models as models
class EmbeddingNetwork(nn.Module):
def __init__(self, embed_size=128):
super().__init__()
# Use a pre-trained model as a strong starting point
backbone = models.resnet18(pretrained=True)
# Remove the final classification layer
self.features = nn.Sequential(*list(backbone.children())[:-1])
# Add a new layer to create our compact embedding
self.embedding_layer = nn.Linear(512, embed_size)
def forward(self, x):
x = self.features(x)
x = x.view(x.size(0), -1) # Flatten the features
x = self.embedding_layer(x)
# Normalizing is crucial for stable distance calculations
return F.normalize(x, p=2, dim=1)The Siamese network wrapper then uses two copies of this network (sharing weights) to process pairs or triplets of images. But here’s a practical question: how do you find good triplets for training? You can’t just use random combinations. If the negative is already obviously different, the network learns nothing. We need challenging negatives—ones that are somewhat similar to the anchor but belong to a different class. This is called “hard negative mining.”
During training, you often implement a strategy to find these hard triplets within each batch. It makes training slower but much more effective. The model is forced to learn finer distinctions. Can you see how this moves us beyond simple pattern matching to understanding more nuanced features?
Once trained, using the network is simple and fast. You pre-compute the embedding for your reference image (the “model” of your friend’s face). To verify a new image, you compute its embedding and measure the distance—typically Euclidean or cosine distance—to the reference embedding. If the distance is below a threshold, it’s a match.
def verify_image(model, ref_image, test_image, threshold=0.5):
model.eval() # Set the model to evaluation mode
with torch.no_grad():
ref_embed = model(ref_image)
test_embed = model(test_image)
distance = F.pairwise_distance(ref_embed, test_embed).item()
return distance < threshold, distanceThis opens up applications far beyond face recognition. I’ve used this for matching industrial parts from diagrams, finding similar documents, and even identifying plant diseases from leaf images where labeled data was minimal. The model learns a general skill: measuring visual similarity. You can then apply this skill to new problems it was never explicitly trained on.
The real advantage comes in production. You don’t retrain the model every time you add a new product or person. You just compute and store a new embedding vector for them. Matching becomes a fast database search for the nearest neighbor in this embedding space, which libraries like FAISS can do in milliseconds across millions of entries.
In my work, shifting to this similarity-based mindset was a breakthrough. It turned data scarcity from a show-stopper into a manageable challenge. The system becomes adaptable, learning a core competency that generalizes. It feels less like programming a rigid tool and more like teaching a fundamental skill.
If you’re facing the “not enough data” wall, I encourage you to explore this path. The code provided is a solid starting point. Try it on a small, personal project. See how it performs. I’d love to hear what unique problems you solve with it. Did this explanation clarify the concept for you? Share your thoughts or questions below—let’s keep the conversation going. If you found this guide useful, please like and share it with others who might be hitting the same data bottleneck
As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!
101 Books
101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.
Check out our book Golang Clean Code available on Amazon.
Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!
📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!
Our Creations
Be sure to check out our creations:
Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools
We are on Medium
Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva
