-
Ingest the image-text pair. Load the image and its associated text (social media caption, meme text, article headline, etc.) as separate inputs. Validate that both modalities are present; fall back to text-only analysis if no image exists.
-
Generate an objective image description. Pass the image through a Multimodal LLM (LLaVA-NeXT, GPT-4V, or Claude's own vision) with a prompt like: "Describe the contents of this image in factual, objective terms. Do not interpret emotions, intent, or sarcasm. Focus on what is visually present: objects, people, actions, setting, colors, text overlays." This caption serves as the semantic anchor.
-
Encode all three text signals. Using a text encoder (BERT, RoBERTa, or sentence-transformers), produce embeddings for: (a) the original text, (b) the generated image description, and (c) optionally, any text extracted from the image via OCR. Store these as vectors for downstream comparison.
-
Compute semantic discrepancy. Calculate the cosine distance (or other divergence metric) between the embedding of the generated image description and the original text. High distance indicates the text says something semantically different from what the image depicts. Normalize this score to [0, 1].
-
Compute sentiment discrepancy. Run sentiment analysis on both the generated description and the original text (using a sentiment classifier or a simple positive/neutral/negative scorer). Compute the absolute difference in sentiment polarity scores. A positive description paired with negative text (or vice versa) yields a high sentiment discrepancy.
-
Compute visual-textual fidelity. Measure how much the original text actually describes the image content. Use the CLIP similarity score between the image and the original text, or compute entailment probability between the generated caption and the original text using an NLI model. Low fidelity means the text ignores or contradicts what's in the image.
-
Extract raw modality features. Encode the image through a vision encoder (CLIP ViT, ResNet) and the text through the text encoder. These raw representations capture information beyond what the discrepancy channels measure.
-
Apply gated fusion. Concatenate the three discrepancy scores with the raw visual and textual feature vectors. Pass through a gating network: g = sigmoid(W * [d_sem, d_sent, d_fid, v, t] + b), then compute the fused representation as h = g * [d_sem, d_sent, d_fid, v, t]. This lets the model learn which signals matter for each input.
-
Classify. Feed the fused representation through a classification head (linear layer + softmax) to produce sarcasm probability. Apply a threshold (default 0.5) for binary classification.
-
Return structured results. Output the classification label, confidence score, and the individual discrepancy scores so the user can interpret why the system flagged something as sarcastic (e.g., "High sentiment discrepancy: image shows a traffic jam but text says 'Love my commute!'").
