Local AI for video studios: how to build your own search system

Any studio that's been running for a few years accumulates more material than anyone can remember: hours of footage, interviews, repeated takes across different projects. Finding a specific shot ends up depending on the memory of whoever recorded it — and that memory fades, changes studio, or goes on holiday. AI that runs on your own hardware, without going through any cloud service, lets you index that archive and search it by content instead of by folder name.

The basic idea is simple: separate storage from processing. A NAS server with real redundancy (RAID) stores and serves the files; a second machine with a decent GPU — a Mac Studio, a PC with a dedicated graphics card — does the heavy analysis work. That way the server never becomes a bottleneck, and you can keep editing while the system processes material in the background.

On top of that you can build a pipeline that does several things at once for every video. It starts by transcribing interviews and statements with precise timecodes (with WhisperX), storing a confidence score for each segment: below a certain threshold, the system flags it for review instead of treating it as final. Every manual correction is stored alongside the original text, without overwriting it, and that history of corrections feeds a project-specific dictionary of names and terms — boats, races, uncommon surnames — that reduces errors in later transcriptions.

The same pipeline generates visual descriptions of frames that let you search by concept — "rough sea, someone on deck" — even if no one ever tagged that shot (with CLIP). Each of these vectors is tied to the exact model version that produced it, so when a better version comes out you don't have to choose between losing the history or living with old results: both coexist, and you can migrate in batches, at your own pace. In our archive that's already tens of thousands of frames indexed as vectors, searchable in milliseconds thanks to a similarity index built into the database itself — without sending a single image to an outside service.

It also detects and outlines objects with pixel-level precision — not just "there's a boat", but its exact outline and position in every frame — using YOLO for detection and SAM for the cut-out. Every detected object stores its timecode, its confidence score and which model produced it, so you can always tell which engine generated a given label and filter by reliability. In our case that's already several hundred thousand detected objects across the archive.

And it recognises faces to group clips by person (with Immich and DeepFace): footage of the same person shot at different times ends up under one profile, even as the angle, the light or the years between recordings change.

Tagging isn't flat either. Each tag belongs to a layer — hand-curated taxonomy, an AI suggestion still awaiting confirmation, an editorial flag, or a mandatory manual review — and every client has their own tag space, so an NGO's vocabulary never mixes with a sports federation's even though both run on the same database. A per-video job log tracks which processes have run, which are pending and which have failed, so the archive never quietly ends up half-processed. All of it lives in your own database — PostgreSQL with the pgvector extension for semantic search — instead of depending on a third-party app as the single source of truth.

What matters for any small or freelance studio is that every one of these tools is open-source: WhisperX, CLIP, YOLO, SAM, DeepFace, PostgreSQL. There's no monthly subscription, no bill per API call, and no risk of a provider changing its terms or shutting the service down overnight: the only cost is the hardware, and in many cases you already own it. The system runs on electricity, not a credit card.

For work with institutional clients — NGOs, public bodies, sports federations — it's worth adding a tagging layer that tracks usage rights for each piece of material: owned, client-owned, licensed, restricted. That's the difference between reusing a shot with confidence and having to ask every time whether it's allowed.

You don't need to build all of this at once. You can start with automatic transcription of interviews alone — which already saves hours of manual searching — and add layers over time: visual search, face recognition, an agent that answers questions about the archive. What doesn't pay off is putting it off: the longer an archive grows unorganised, the more expensive it becomes to sort out later.

At Video Synthesis we apply this logic to the archives of our long-term clients, where years of footage stop being a folder sorted by date and become a resource we can reuse quickly. If you're managing an audiovisual archive that's become unmanageable, let's talk.