Ever since I saw a social media post with cheerful text alongside a gloomy image, I’ve been fascinated by how meaning shifts when words and pictures combine. That moment sparked my journey into multi-modal sentiment analysis. It’s clear that text alone can’t capture the full story. Consider a tweet saying “Loving my new phone!” next to a cracked screen photo. What sentiment would you assign? This gap between text and visual context drives the need for systems that understand both. Let me show you how to build one using Python and vision-language transformers.

Setting up our environment is straightforward. We’ll use PyTorch as our foundation:

pip install torch transformers datasets Pillow

For our core architecture, we’ll leverage CLIP’s pre-trained capabilities while adding custom fusion layers:

from transformers import CLIPModel, CLIPProcessor

class SentimentFusionModel(torch.nn.Module):
    def __init__(self, num_classes=3):
        super().__init__()
        self.clip = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
        self.fc = torch.nn.Linear(512, num_classes)  # Classification head

    def forward(self, images, texts):
        outputs = self.clip(images, texts)
        pooled_features = outputs.logits_per_image
        return self.fc(pooled_features)

Handling multi-modal data requires careful preprocessing. How do we ensure text and images align properly? Our dataset class handles this synchronization:

from torch.utils.data import Dataset
from PIL import Image

class MultiModalDataset(Dataset):
    def __init__(self, dataframe, processor):
        self.data = dataframe
        self.processor = processor
        
    def __getitem__(self, idx):
        row = self.data.iloc[idx]
        image = Image.open(row['image_path'])
        text = row['text']
        inputs = self.processor(
            text=text, 
            images=image,
            return_tensors="pt",
            padding=True
        )
        inputs = {k: v.squeeze() for k,v in inputs.items()}
        inputs['label'] = torch.tensor(row['sentiment'])
        return inputs

Training presents unique challenges. Should we freeze the visual encoder? Update the text transformer? Here’s our training loop configuration:

from transformers import AdamW

model = SentimentFusionModel().cuda()
optimizer = AdamW(model.parameters(), lr=2e-5)

for epoch in range(10):
    for batch in dataloader:
        inputs = {k: v.cuda() for k,v in batch.items() if k != 'label'}
        labels = batch['label'].cuda()
        
        outputs = model(**inputs)
        loss = torch.nn.functional.cross_entropy(outputs, labels)
        
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

For evaluation, we need metrics that account for modality interactions. Notice how we track confusion between text-dominant and image-dominant predictions:

from sklearn.metrics import classification_report

def evaluate(model, test_loader):
    all_preds, all_labels = [], []
    
    with torch.no_grad():
        for batch in test_loader:
            inputs = {k: v.cuda() for k,v in batch.items() if k != 'label'}
            labels = batch['label'].numpy()
            outputs = model(**inputs).cpu().numpy()
            
            preds = np.argmax(outputs, axis=1)
            all_preds.extend(preds)
            all_labels.extend(labels)
    
    print(classification_report(all_labels, all_preds))

Deploying our system requires efficient inference. This Gradio interface lets users test real-world examples:

import gradio as gr

def predict(image, text):
    inputs = processor(text=text, images=image, return_tensors="pt")
    with torch.no_grad():
        outputs = model(**inputs)
    probs = torch.softmax(outputs, dim=-1).numpy()
    return {labels[i]: float(probs[i]) for i in range(3)}

gr.Interface(
    fn=predict,
    inputs=[gr.Image(type="pil"), gr.Textbox()],
    outputs=gr.Label(num_top_classes=3)
).launch()

Seeing the model correctly interpret sarcastic posts where text and images contradict has been incredibly rewarding. What surprising interactions between visual and textual elements have you encountered? Share your thoughts below. If this approach to sentiment analysis resonates with you, give it a thumbs up and share with others facing similar challenges. I’d love to hear about your implementation experiences in the comments.