AI image search: a shopper photographs a product and your catalogue answers

What the vision model actually extracts

The photo is not thrown at a similarity index as pixels. It is read into a strict structure: likely product name, brand, product type, model number or SKU printed on the item, barcode digits if legible, colours, materials, a search phrase, a confidence score, and a flag for whether this is even a household product you would stock.

That last flag matters. If the customer photographs their dog, results are deliberately cleared rather than filled with the closest plastic object, and the interface switches to a "suggest this product" path instead of pretending to have an answer.

How the matching is ranked

The extracted search phrase is embedded and matched against your product vectors, producing a candidate pool. Those candidates are then merged with hard identifier lookups and scored in tiers: barcode or supplier barcode first, then SKU, then brand-plus-name text matches, and finally visual and semantic similarity.

Duplicates collapse to their highest score, so a product that matches on both barcode and similarity appears once, at the top. If a branch is selected, results can be filtered to what is actually in stock at that branch before the shopper commits emotionally to a product they cannot collect today.

• Barcode and supplier barcode: highest confidence tier
• SKU printed on the product: second tier
• Brand plus product name text match: third tier
• Vector similarity from the vision caption: final tier

Captioning, not a separate image index

It is worth being precise about the architecture, because plenty of vendors are not. GoGee does not maintain a second parallel image-vector index at query time. The photo is captioned into text by the vision model, and that text is embedded with the same embedding model used for typed search.

The practical consequence is good: image search inherits every improvement made to text search, including exact-identifier scoring, price filters and stock awareness. Product photos are also captioned in bulk on the catalogue side, and those captions feed each product's embedding, which is what makes the visual half of the match work.

Guardrails before the AI call

Uploads are validated by magic bytes rather than trusting a file extension, capped by size, and rate-limited per visitor and globally. If structured extraction fails, a second call runs in strict JSON mode and the response is parsed manually. Every search is logged with its input mode so you can see how many people are choosing photo over typing.