YOLO + OpenClip for video object classification and semantic search
Hello! Tired of the AI crazyness, there are solid approaches beyond deploying yet LLM to answer question or hook them up to a MCP. Do you have some time to spare and are looking for something cool to try out? Let’s try some computer vision model implementation to work out an open classification and detection problem, explore some non-llm models along the way and the approach taken to solve it, spoiler we’ll use YOLO + OpenCLIP for semantic video streaming search.
The problem at hand
Grab a video stream and have a way to detect events dynamically, “such as a person with a phone”, “a person with a blueshirt”, there are several ways to approach this challenge, it will require of course a good bit of computer vision, but it is undeniably a search problem by its nature, “dynamically” mean that the event to be detected discrete in nature but flexible to change eventually; therefore a regular classifier here will not be enough.
The approach: detect, filter & embed
We needed something that could quickly recognized object but not necesarilly know what they were, so we ran into YOLO from ultralytics it checked all the criterias
- Fast and efficient, optimized for low latency and high frames-per-second, although we were not looking specifically for real time application the fact that it was extremelly fast given with the resources at hand made it particularly interesting avenue to take!.
- Single pass detection it use a CNN behind the scenes to forward and predict the image, predicting the bounding box and probabilities.
- Extensible: YOLO can be train to recognized beyond the 80 categories it comes with, and it’s a straighforward process too.
More is explained in Architecture of YOLO
We said before we need it something that could quickly recognized the object, but why ? Well basically we are taking the approach of recognize, cut and embed. YOLO will help us with the first too, this will make the next step extra efficient. So because YOLO modes are closed-set object detectors trained on a fixed, predefined list of categories like COCO dataset and even if we start training YOLO on specific classes it will not entirely help here as the detection has dynamic event by nature, a person with a
Welcome to an open source implementation of OpenAI’s CLIP (Contrastive Language-Image Pre-training).
The original paper from OpenAI is long and detailed the architecture and how it basically work, it is a milestone on zero-shot learning models Reproducible scaling laws for contrastive language-image learning
One of the biggest advantages of open-vocabulary models like CLIP is that they can be used on downstream classification tasks by carefully designing text prompts corresponding to class descriptions, without requiring any labeled training example.
Therefore OpenClip looked like the natural choice, the idea here is to clip the image detected by YOLO and use OpenClip to recognize the dynamic event itself.
Results
YOLO’s detections are shown below, with labels and bounding boxes around the recognized objects:
With YOLO + OpenCLIP, a text prompt can describe the event we want to find. In this example, the system detects a person with a phone
When the person with a phone is no longer present, the event is no longer detected :
Conclusion
The results above are only a draft to demonstrates some of the capabilities of combining object detection with open-vocabulary image and text matching. YOLO handled object detection using the yolo26n model, while OpenCLIP used hf-hub:laion/CLIP-ViT-g-14-laion2B-s12B-b42K to match detected image regions against a text prompt. Both models ran behind a Flask server on a VPS with 16 GB of RAM and 4 CPUs.