Skip to content

(05)Sep 2026

Identifies a product from a photo, and asks for a better photo when it isn't sure.

Role

Solo build

Timeline

Sep 2026

Stack

Next.js, TypeScript, Tesseract OCR, CLIP embeddings

Runs locally for now, there's no public deployment yet. Try the decision logic in the interactive demo below.

The agent's upload and camera screen
AI Product Support Agent upload screen

(01)Overview

Upload or photograph a product and the agent works out the brand, product family and exact model, but only when the evidence supports it. It checks for visible damage and tells you exactly which extra photo would help.

(02)Problem solved

Support chats waste time on 'which model do you have?'. A photo should be enough, as long as the system is honest about what it can't confirm.

(03)My contribution

  • The whole pipeline: OCR, image embeddings, matching, verification
  • A local vector store for catalogue matching
  • Camera capture and follow-up photo requests in the UI

(04)Features

  • 01Reads label text with OCR (Tesseract)
  • 02Compares the photo to the catalogue with CLIP image embeddings
  • 03Verifies a match against both kinds of evidence before answering
  • 04Checks for visible physical damage
  • 05Asks for a specific follow-up photo when confidence is low

(05)How it works

Five steps from photo to answer.

Step 01 of 05

Photo in

The customer uploads a picture or takes one with the camera, right in the browser.

image: kettle_front.jpg
size: 1280×960
source: camera

Illustrative output

Uncertainty, playable

The part that says “not sure yet.”

The decision rule, simplified and running in your browser. No model is called here.

combined confidence

71

Not sure yet. “Could you take a closer photo of the label or model sticker?”

Technology stack

Next.jsTypeScriptTesseract OCRCLIP embeddingsVector searchVision models

Development challenges

Knowing when not to answer

A wrong model number is worse than no answer. Evidence is combined into one confidence score, and below the threshold the agent asks for the photo that would help most.

Pluggable sources

The local catalogue and open-world search sit behind one provider interface, so new sources plug in without touching the orchestration.

Next project

SmartLiv & ABHEE →