Search by Photo or Voice: How the AI Search on Crazy Plastics Actually Works

Most online stores still expect a shopper to know the right words. Type "storage tub" when the store calls it a "utility crate" and you get nothing. On the Crazy Plastics store we removed that problem in two ways: a shopper can take a photo of the product, or press a microphone button and just say it out loud. Both routes end up in the same place, a ranked list of real products that are actually in stock at the branch nearest them.

This article explains how it works under the bonnet, because the interesting part is not the buzzword, it is the plumbing.

Searching with a picture

The shopper taps the camera button and either snaps a photo or picks one from their gallery. On a phone that is often a photo of something already in their kitchen, a cracked bin, a lid that has gone missing, or a product on a shelf in another shop.

1. The photo is shrunk before it leaves the phone

South African mobile data is not free, so the image never gets uploaded at full size. In the browser the photo is redrawn onto a canvas, capped at 1600 pixels on the longest side, and saved as WebP where the phone supports it, dropping quality in steps until it fits under about 2MB. A 6MB camera photo typically lands around 200KB. That single step is the difference between a two second search and a twenty second one on a weak signal.

2. The server checks that it really is an image

Before any AI is involved, the upload is validated: size limits, an allow list of image types, and a check of the file's actual magic bytes, the first few bytes of the file that identify a genuine JPEG, PNG or WebP. A file that claims to be a photo but isn't gets rejected there. Rate limits are applied per visitor and globally, so nobody can point a script at the feature.

3. A vision model reads the picture like a person would

The image is then sent to a vision-capable Gemini model. Crucially, we do not ask it "what does this look like". We ask it to fill in a strict form, and it returns structured data: likely product name, brand, product type, any model number or stock code, visible text on the packaging, barcode digits, colours, materials, packaging type, size or quantity, and a plain-English search phrase describing the item. It also returns a confidence score and a flag for whether this is even a household product at all.

That last flag matters more than it sounds. Photograph a car or a person and a naive system will still cheerfully return "closest matching plastic bucket". Ours returns nothing, and says so.

4. Four ways of matching, merged

The extracted attributes are then matched against the live catalogue in four layers, each with its own confidence weighting:

Barcode. If the vision model could read the barcode digits off the packaging, that is an exact hit and it wins outright.
Stock code. A readable SKU or model number is the next strongest signal.
Brand and name. The detected brand is resolved against the real brand table, then combined with the product name text.
Meaning. The plain-English description the model wrote is converted into an embedding, a long list of numbers that represents meaning rather than spelling, and compared against embeddings of every product using cosine similarity inside Postgres with pgvector.

All four candidate sets are merged, de-duplicated by best score, and optionally filtered down to what is physically in stock at the shopper's branch. The strongest handful become the top matches, the rest become "related products", so a near miss still gives the shopper somewhere to go.

This is why photo search works on a scuffed, unbranded, ten-year-old container: even with no barcode and no legible text, the description of shape, colour, material and use is enough for the meaning-based layer to find the modern equivalent.

Searching with your voice

The microphone button is the same idea with a different input. Voice tends to be longer and more natural than typing: shoppers say "a big plastic box with wheels that fits under a bed", which is a far better query than the two words they would have typed.

1. Recording

The browser's own recorder captures audio, preferring Opus in a WebM container and falling back to MP4 on Apple devices. Recording is hard-capped at 15 seconds, which keeps files small and stops accidental open microphones.

2. Two transcriptions at once

At the same moment, the browser's built-in speech recognition is started in parallel, tuned to South African English. That is deliberate belt-and-braces: the browser cannot transcribe a recording after the fact, so if we want a fallback we have to capture it live.

When the shopper stops speaking, the recorded audio is validated the same way images are, including a container signature check, then transcribed server-side by a speech-to-text model. If that call fails, times out, or the shopper is on a flaky connection, the transcript quietly captured by the browser is used instead. The shopper never sees a failure, they just see their words appear.

3. Straight into the normal search brain

Once there is text, voice stops being special. The transcript is handed to exactly the same search pipeline a typed query uses, tagged as voice so we can measure it separately.

What happens to any query, typed or spoken

The text pipeline is where most of the intelligence lives:

Understanding intent. A fast Gemini model turns the sentence into structured filters: category, brand, colour, features, price range. There is a strict timeout, an automatic retry on a faster model, and a plain keyword fallback, so a slow AI provider can never leave a shopper staring at a spinner.
Not trusting the AI with the hard numbers. Price limits like "under R200" are extracted with plain code, not the language model, and always override whatever the model guessed. Money is too important to leave to a probability.
Meaning plus spelling. Vector similarity runs alongside exact name matching, stock code lookup, brand lookup and traditional Postgres full-text search. Each layer contributes candidates with a score.
Deterministic re-ranking. Colour and feature matches boost or penalise a product's score by fixed amounts, so ranking is explainable. If a shopper asked for a red one, red rises and grey falls, and we can show exactly why.
Refinement. "Now show me only the ones with lids" narrows the previous result pool rather than starting a brand new search.
An honest note. A small final check compares what was asked against what was found. If the shopper wanted "hanging" and nothing hanging came back, the page says so instead of pretending.
Off-topic questions. If the query has nothing to do with the catalogue, the system short-circuits before touching the database at all.

Learning from real shoppers

Every search, typed, spoken or photographed, is logged with its filters, its results, its scores and how long it took. Shoppers can thumb a result down. Staff review the bad ones and turn them into short written lessons, and those lessons are injected into the search prompt itself. The system genuinely gets better at this specific catalogue over time, and the improvements are human-reviewed rather than mysterious.

Why we built it this way

Three principles ran through the whole build, and they are the same ones behind every tool we make:

Never leave the shopper stuck. Every AI call has a fallback, a timeout and a cheaper alternative behind it.
Use AI for language and vision, use a database for facts. Prices, stock and branch availability come from Postgres, never from a model's imagination.
Measure everything. If you cannot see what shoppers searched for and failed to find, you are not running a search engine, you are guessing.

Picture search and voice search are part of the ecommerce websites we build, and sit alongside the rest of the GoGee.ai tool set, including competitor price tracking. If you sell physical products and your shoppers struggle to describe what they want, this is the fix.

Want the same on your store? Talk to us or build a brief and we will show you how it would work on your catalogue.