MediumBuilding an Explainable Image Search Engine with CLIP and BLIP | by Manu Madhavan | Medium
Sitemap

Building an Explainable Image Search Engine with CLIP and BLIP

3 min readJun 30, 2025

--

In today’s content-driven world, image search is more important than ever. But most search engines stop at “showing” results without ever explaining why they were selected. In this post, I’ll walk you through how I built an explainable image search engine using two powerful AI models: CLIP (for semantic search) and BLIP (for image captioning).

Imagine typing “two humans sitting together” and getting back not only the right image but also a short AI explanation like:

“This image is somewhat relevant because it shows a couple sitting by a fire in the woods.”

That’s the experience I wanted to create — search that shows and tells

Press enter or click to view image in full size
Image by Author

Now look second image Explanation:” It Show two women hugging in a pool”. Its actually not relevant two our query two human sitting together

An AI agent powered by an LLM can easily discard this image just by reading the above explanation — right?

The AI Behind the Scenes

What is CLIP?

CLIP (Contrastive Language-Image Pretraining) by OpenAI maps both images and text into the same embedding space. This allows for powerful zero-shot image retrieval using plain natural language queries.

What is BLIP?

BLIP (Bootstrapped Language Image Pretraining) can generate captions for images. It’s a vision-language model that helps explain what’s happening in a given image.

Implementation Steps

1. Image Embedding Generation

The process begins with converting images into numerical representations:

These embeddings capture semantic meaning rather than just pixel values, enabling matches based on concepts like “waterfalls” rather than exact visual patterns.

Challenge: Hardcoded absolute paths in vector storage made deployment difficult.

Solution: Implemented flexible path resolution that extracts filenames and reconstructs paths based on current environment configuration.

2. Natural Language Search

When users submit text queries, transform them into the same embedding space:

The magic happens in CLIP’s shared embedding space, where “flowing water” as text and an actual waterfall image have similar vector representations.

3. Intelligent Similarity Matching

Chroma DB performs efficient cosine similarity searches:

Get Manu Madhavan’s stories in your inbox

Join Medium for free to get updates from this writer.

The system returns images ranked by semantic similarity, not just visual appearance.

The Explainability Layer

Why Explanations Matter

Raw similarity scores tell us what matches, but not why. For users to trust and effectively use the system, they need to understand the reasoning behind each result.

BLIP-Powered Explanations

Our explainability system uses BLIP to generate contextual descriptions:

Context-Aware Reasoning

The system goes beyond simple captions by incorporating query-specific context:

  • Query: “water fall in the woods”
  • Explanation: “This image is somewhat relevant to ‘water fall in the woods’ because it shows a river in the woods with yellow leaves. The aquatic features correspond to your search terms.”

Explainability Effectiveness

User studies showed that explanations significantly improved:

  • Trust in results: May increase user trust in AI recommendations
  • Search refinement: Users better understood how to adjust queries
  • Discovery: Users found relevant images they hadn’t initially considered.

Conclusion

Combining CLIP and BLIP creates a search experience that is both smart and explainable. It transforms image retrieval from a black box into a transparent and trustable process. I hope this inspires you to build your own AI-powered, explainable tools.

Entire code of Visual search tool is given in below link

https://github.com/manucet439/Image_Search/tree/main

Credit and Reference

https://openai.com/research/clip

https://medium.com/aimonks/semantic-search-hands-on-text-to-image-search-using-clip-and-faiss-c1d387f66d00

--

--

Manu Madhavan
Manu Madhavan

Written by Manu Madhavan

Professional with over 10 years of expertise | Master's in Data Science. |Skilled in Python, SQL, AI/ML, and Automation, driving impactful business growth.