Vaelen: Self-Hosted AI Camera System, No Cloud, Open Source

Vaelen: A Self-Hosted AI Camera System That Actually Understands What It Sees

ambient perception

Ambient Perception — awareness that just runs, quietly, in the background, without you ever going looking for it.

That’s Vaelen: it watches your cameras and listens to your mics around the clock, and actually understands what’s happening — not just records it. No scrubbing through hours of footage. No staring at a dashboard waiting for something to happen. Something did happen? Ask Vaelen, and it just tells you.

Every other camera system on the market is selling you the same thing: a slightly smarter way to record video. Motion detection instead of nothing. Object detection instead of motion. Maybe a scene description bolted on top. But the product is still the video — you’re still the one who has to watch it, search it, and decide what mattered.

Vaelen starts from a different premise entirely: the video is raw material. What actually matters is the understanding several specialized AI models build on top of it, working together — and increasingly, what you can just ask about that understanding instead of scrubbing a timeline yourself.

Type a real question into the dashboard — “was anyone at the door yesterday afternoon who wasn’t a delivery?” — and Vaelen doesn’t hand you a list of clips to check. It searches everything it’s seen by meaning, hands the relevant recordings to a local LLM, and answers you directly, in a sentence, with the actual footage linked underneath if you want to check its work. Ask it in German, Finnish, or a dozen other languages, and it works exactly the same — more on why that wasn’t always true below.

That’s the whole philosophy in one feature. Here’s everything underneath it.

All of it runs on one machine you own, with one GPU you already have. No cloud, no subscription, no monthly fee, nothing phoning home.

Ask Vaelen: the feature that makes the point

This is the one to lead with, because it’s the clearest expression of what Vaelen is actually for. A prominent, single input box sits at the top of the dashboard — not a search bar with autocomplete, an actual question box. Ask it something in plain language, and here’s what happens under the hood: your question runs through the same semantic search index that already powers regular search, pulling the most relevant recordings by meaning, not keyword match. Those recordings — timestamps, cameras, AI-written descriptions, but also recognized people, transcribed speech, and visible text — get handed to a locally hosted LLM with one explicit instruction: answer only from what’s actually there, and say so honestly if nothing found actually answers the question.

No new model. No new infrastructure. The search index and the Ollama connection already existed for other features — this just recombines them into something you can have a conversation with. The answer that comes back has the source recordings linked directly underneath it, so it’s checkable, not a black box you have to trust blindly. Ask something that spans both recent and archived footage, and the answer says so explicitly — “found in recently saved videos” and “the archive also has recordings showing…” — instead of quietly blending the two together as if location never mattered.

Ask it something right after a fresh recording, and it waits for that recording’s analysis to actually finish first — up to ninety seconds — rather than confidently answering from a search index that doesn’t know about the thing you’re asking about yet. Past that, it tells you honestly it’s still busy and to try again, instead of quietly giving an incomplete answer dressed up as a complete one.

And when the first answer isn’t specific enough, you can push it further. Four recordings match “car” — none of the descriptions mention color, or who was driving, because the original analysis was never written to answer that specific question. Deep Research closes exactly that gap: ask a targeted follow-up on the recordings Ask Vaelen already found — “does a woman sit at the wheel in any of these?” — and instead of re-running the same search and getting the same generic descriptions back, Vaelen re-sends each recording’s actual frames through the vision model with your precise question. Each one gets its own targeted answer; those get folded into one direct, conversational response, with the individual per-recording findings still available underneath. It’s genuine extra work — one more vision-model call per recording — so it only runs when you actually ask for it, never automatically tacked onto a normal question.

You don’t have to type it, either. A microphone button next to the question box dictates your question using the browser’s own speech recognition — no server involved for that part at all. Ask by voice, and the answer gets read back to you automatically; ask by typing, and a small speaker icon on the answer lets you hear it anyway if you’d rather listen than read. And if what you actually have is a short audio clip — a voice memo, something recorded elsewhere — you can attach that instead of typing anything: it gets transcribed locally, translated for your own reading if it’s not already in your language, and folded straight into the question as context. The translation is purely a courtesy for you, by the way — the search itself works directly off the original transcript, since the multilingual embedding model doesn’t need a translation to understand it.

What Vaelen does that nothing else does

It hears what it can’t see. A dedicated audio model (CLAP) listens continuously, completely independent of the camera image. Type “glass breaking” or “dog barking” or “someone shouting” into the dashboard in plain English, and it starts triggering on that sound immediately — no training data, no retraining, no fixed list of categories to pick from. A window breaking two rooms away, in the dark, with nothing in frame — Vaelen still knows something happened, and that knowledge feeds into the same searchable, askable pool as everything else.

It reacts to what’s actually said, not just what kind of sound it is. CLAP recognizes categories — a bark, breaking glass — but has no idea what someone actually says out loud. A live transcription pass, running continuously in small chunks while a recording is already underway, closes that gap: a configured word heard in a just-transcribed chunk fires its own independent trigger, catching content no sound-category model or camera image could — a shouted word at night, with nothing visible and nothing that sounds obviously alarming. A second, separate list of words never starts anything on its own; it’s purely for flagging what got mentioned somewhere in a recording, after the fact.

It has a plain fallback for when the clever stuff doesn’t fire. Object detection needs to actually recognize something — motion doesn’t care what’s moving, just that something changed. Background subtraction (not a naive frame-by-frame difference, which fires on every passing cloud) builds a running model of what a scene normally looks like and flags real, sustained change against it — a genuinely independent second trigger, off by default, that catches whatever a configured object category or sound model happened to miss.

It notices when someone’s in trouble. A separate pose-estimation pass runs whenever a person is already detected — at zero extra GPU cost, since it reuses the same frame. From that one pass, Vaelen reads six independent signals: a fall, a raised-hands distress gesture, someone loitering in the same spot too long, a person moving unusually fast, two people in sustained close proximity, and which way someone’s facing. Every one of them is its own on/off switch with its own threshold — turn on just fall detection, or all six, or none.

Five independent ways to start a recording sounds like a lot until you actually try to pick one for a specific spot. Object detection is the obvious default — something recognizable showed up. CLAP is for what you’d notice by ear before you’d ever see it: glass breaking, a smoke alarm, a dog barking while nobody’s home — categories of sound, not specific words. Motion detection is the honest fallback for a place where “something changed at all” is the whole point, no matter what it turns out to be — a storage room, a yard at 3am, a camera angle nothing else handles well. Pose and behavior detection is safety, not security — a fall matters regardless of who caused it or why. And the spoken trigger word is for the one channel none of the others cover: content, not category, for the specific case where audio is all you’ve got — dark, out of frame, nothing that sounds obviously alarming until someone actually says the word that matters. Topics, in either their visual or spoken form, are a deliberately different tool — they never start anything, they just make what already got recorded, for whatever reason, findable afterward.

It doesn’t always wait for the tape to stop rolling. Everything above happens after a recording finishes — fine for review, too slow if something’s actually unfolding right now. Realtime Mode collects frames continuously while a camera is still recording, in batches, and checks each batch immediately with a second, deliberately small and fast vision model — Moondream or Qwen2.5-VL 3B, not the larger model doing the careful final analysis. The checkpoint prompt itself is kept to a handful of words on purpose: once an image is processed, token-generation time is what actually dominates a small model’s latency, so asking for a short list of matching keywords instead of a description keeps the whole checkpoint fast, not just the model. A match against your configured topics fires an immediate, clearly-labeled preliminary notification — and a negative topic suppresses it even if a positive one also matched, for the specific “I know about this expected thing, don’t alert me on it” case (topic: “person,” negative topic: “known family member”). The full, careful analysis still runs normally once the recording ends and always has the final word; Realtime Mode is a faster first look, never a replacement for it. Off by default, opt-in per camera, and it shares the same GPU-lock awareness as everything else — a checkpoint waits briefly for a busy GPU rather than fighting a real analysis for it, and skips gracefully rather than stalling the recording if the wait times out.

It reads anything, not just plates. OCR isn’t a license-plate feature with a narrow job — it reads any visible text on a sample of frames: house numbers, package labels, street signs, storefront names, whatever’s actually there. Every fragment it finds feeds straight into the search index, in its own field, kept separate from the AI description and the Whisper transcript so none of the three ever silently overwrite each other. License plates are a filter on top of this, not a separate scan: text whose position overlaps a detected vehicle gets checked against a plate-like pattern and, if it matches, stored and matched against named vehicles by exact text — the same “named entity survives forever” model that runs face recognition. Name a vehicle once, every future sighting auto-links to it.

It doesn’t describe four things — it describes one situation. Every signal used to live in its own silo: CLAP knew a sound happened, Whisper knew what was said, OCR knew what text was visible — but the vision model writing the scene description never heard about any of it. It just looked at frames and guessed. Now, whatever the other sensors already know about this specific recording gets folded into the same prompt the vision model uses: the sound that triggered it, a transcribed word, visible text, a detected fall. The description that comes back can be “a person arrives at the door, the microphone picks up glass breaking moments later, and a package on the step reads ‘DHL Express'” — one connected account, not three facts that happen to share a timestamp. Nothing new to install, no extra GPU cost — it’s a prompt, not a model. But it’s the difference between perceiving several things and actually understanding the situation they add up to.

It finds the recording that doesn’t fit — without you defining a single rule. The same text embeddings used for search get recycled into a per-camera Isolation Forest model. It learns what “normal” looks like for that specific camera over time and flags statistical outliers automatically. Zero extra GPU cycles, because the embedding was already being computed anyway.

It knows who’s in frame — permanently, and even when the model changes. Face recognition runs in-process (no external microservice, no network hop — the embeddings are extracted from frames already sitting in VRAM), clusters unknown faces automatically, and once you name someone, their identity is archived independently of any single recording. Delete the video that first introduced them to the system a year later, and Vaelen still recognizes them tomorrow. Switching between InsightFace’s model packs — say, from the fast default to the more accurate antelopev2 — used to quietly break clustering entirely, for a reason that took real measurement to track down (see below); it’s fixed now, and the clustering sensitivity adjusts itself to whichever pack you’re actually running.

It treats a live stream like a live stream, not a file to wait for. Point Vaelen at a watchfolder, and it doesn’t just sit there waiting for a finished video file. If what lands there is actually a growing, streamable source — a live camera export, an MPEG-TS feed, even a YouTube or Twitch URL — it connects immediately and runs the exact same detection pipeline a real camera would. Platform sources ship with their own JS-challenge-solving runtime alongside yt-dlp, since YouTube increasingly demands one to resolve some live streams correctly — and optional cookie support for the streams that go further and ask you to prove you’re not a bot.

It understands footage that’s already years old, not just what’s happening right now. Everything above works the same whether a recording just happened or has been sitting on a drive since last decade. Drop a folder of existing footage into the watchfolder and every file gets the full pipeline — description, transcription, faces, plates, visible text, search indexing — the same treatment a live camera’s recording gets, no live source required. Submit files one at a time over the REST API instead, and you get the same enriched result back on completion via webhook — the shape a media asset management system or a digital archive would actually want to script against. Files submitted this way used to disappear into their own invisible folder, reachable nowhere in the dashboard; they now land in the exact same place as anything recorded live and show up in a dedicated Remote Videos view with the same archive, export, delete, and reanalyze actions as everything else. And if a job’s video analysis happens to fail for some unrelated reason — a database hiccup, a busy GPU — the file still gets fully imported and searchable instead of the whole submission being thrown away over one failed step; the specific error comes back with the job status instead of a flat “failed.”

It captures on a schedule too, not only on detection. A permanent live stream — a public webcam, a demo feed — might genuinely never produce a real trigger of its own. Periodic Segments force a recording at whatever interval and duration you set, running through the exact same post-processing pipeline as any other recording, tagged with its own “Scheduled Segment” badge instead of a fabricated detection reason.

It can tap you on the shoulder directly. Personal push notifications via ntfy — free, no account — with the trigger screenshot attached as an actual image, not a link that assumes your dashboard is reachable from the internet (it usually isn’t, and shouldn’t be). Choose what fires it: an anomaly, a flagged fall or distress signal, a matched topic, or every recording. At most one notification per recording even if several conditions overlap at once, and behavior alerts go out the moment they’re detected, not after the full analysis pipeline finishes — a fall notification shouldn’t wait on Whisper.

An AI agent can run it for you — and increasingly, run a real conversation with it. Vaelen ships a full MCP server exposing the same tools any script can call over the REST API, so an agent like Claude can check what a camera saw, toggle cameras on or off, kick off a recording, search past events, ask a real question the same way you would, submit an existing file by local path for full processing, and read logs — system-wide or per camera — to answer “why hasn’t this triggered today” without shell access to the box. The per-recording detail tool used to only hand back the description, despite its own documentation claiming otherwise; it now genuinely returns recognized faces, visible text, and the exact trigger reason too. A layered permission system decides exactly what any of this is allowed to touch, off by default, with deletion intentionally left unreachable no matter what’s toggled on.

It tells you before your disk fills up — and heals itself when a mount briefly misbehaves. Recordings can live on an entirely different drive than the app itself, and a live Storage Status view shows free space and write access for every configured folder — a warning before it’s a crisis, not an error message after. A flaky mount at boot used to be able to send one camera process down a different storage path than its siblings, permanently, for the rest of that process’s life — recordings would quietly split across two locations with no visible sign anything was wrong. It now retries a few times against exactly that timing race, and if it still can’t get through, says so loudly in the dashboard instead of staying silent about it.

Vaelen vs. Motion vs. Frigate

Choose Motion if you want a simple daemon on minimal hardware with no GPU. Choose Frigate if you want a mature, broadly-supported NVR with wide accelerator support (Coral, Hailo, OpenVINO) and a bigger community. Choose Vaelen if the thing you actually want is an answer, not a recording.

How it’s built

Vaelen chains small, specialized models instead of asking one giant model to do everything — each one gets handed exactly the context it needs from the stage before it.

[ Triggers, in real time ]
├─ YOLO (v10 / v12 / v26)
├─ CLAP audio
└─ Pose: fall, distress, loitering, movement, proximity, gaze
        │
        ▼
[ Zero-CPU ingestion ]  RTSP / RTMP / MJPEG / USB / Watchfolder / live platform sources
        │
        ├──▶ [ Realtime Mode ]  fast model checks frame batches DURING recording, preliminary only
        │
        ▼
[ Finished recording ]
        │
        ├─ Whisper           speech → text
        ├─ EasyOCR           visible text (plates: a filter on top)
        ├─ InsightFace       people, in-process
        │
        ▼   (transcript + OCR text + CLAP label + behavior event, folded in)
[ Ollama VLM ]  one connected description + topic scoring
        │
        ▼
[ sentence-transformers, multilingual ]  semantic search index
        │
        ├──▶ [ Isolation Forest ]  anomaly detection, recycled from the search embedding
        │
        └──▶ [ Ask Vaelen ]  question in → matched recordings → LLM-synthesized answer out
                    │
                    └──▶ [ Deep Research ]  targeted follow-up → same recordings, vision model again
        │
        ▼
[ Integrations ]  MCP server · Home Assistant/MQTT · REST API + webhooks · ntfy notifications

Whisper and OCR deliberately run before the vision model now, not after — their output has to already exist for the fusion above to have anything to fold in. Getting that ordering right, without either step silently overwriting the other’s part of the same metadata file, turned out to be its own small lesson (more on that below).

Search that finally speaks your language

The sentence-embedding model doing semantic search — the thing that lets “someone at the door” match a description that never uses those exact words — was, it turned out, effectively English-only. A German question against an English-language description scored −0.04 on cosine similarity: statistically no relationship at all, despite describing the exact same event. The same English question against the same description scored 0.45. Every non-English question was quietly failing the relevance threshold, every time, regardless of how good a match it actually was — and the failure mode was indistinguishable from “nothing like that happened,” which is a much worse thing to be silently wrong about than a crash would have been. Switching to a genuinely multilingual embedding model closed the gap entirely: the same German question against the same description now scores 0.52 — matching, not just improving on, the English-to-English score. Same 384-dimensional output, so nothing about the storage format had to change, but every previously-indexed recording needed its embedding recomputed from scratch, since a vector from the old model and a vector from the new one aren’t comparable just because they’re the same shape.

Realtime Mode: a fast look while the tape is still rolling

The frame buffer this needed had to be genuinely independent of the filmstrip mechanism already running during a recording — that one exists to sample evenly across an entire event for later review, using reservoir sampling so a ten-second clip and a thirty-minute one both get sensible, spread-out coverage. Realtime Mode wants the opposite: the most recent N frames, checked the moment there are enough of them, then start over. Reusing the filmstrip’s own sampling for this would have fought its actual purpose. The two now run side by side in the same recording loop, sharing nothing but the raw frame itself. Getting genuine speed out of a small vision model turned out to be less about the model and more about the prompt: asking for a short list of matching keywords costs far less generation time than asking for a description, even once the image itself is already processed just as fast either way — a detail easy to miss if you only ever benchmark the image-encoding step.

The context window nobody had actually measured

A 27-billion-parameter model started rejecting every reanalysis request with request (44274 tokens) exceeds the available context size (32768 tokens) — a real error, with a real number in it, that had simply never been checked against what a full frame batch actually costs in tokens for that specific model. The fix that stuck wasn’t picking a bigger fixed number and hoping — it was reading the exact token count straight out of Ollama’s own error message and setting the context size to the smallest round value that actually covers it, no guessing. That same logic now runs automatically the moment you change the vision model or how many frames go into an analysis: a real test call against existing footage, a context error handled by reading the number and fixing it, and — for the rarer case of a model that simply can’t take that many images in one call at all, context size aside — a search for the largest frame count that actually works, applied automatically instead of surfacing as a mysterious failure days later.

A GPU error that used to mean “never again,” now means “try again”

Face recognition sharing a GPU with several live camera-detection processes means it sometimes loses the fight for VRAM at exactly the wrong moment — a transient condition, gone as soon as another process finishes its own work. The existing retry logic, though, had been written for a different failure entirely: a corrupted or incomplete model download, where retrying without changing anything is pointless and permanently disabling the feature is the right call. A GPU-memory error was hitting that same “give up forever” path, for a problem that had nothing to do with the downloaded files and everything to do with bad timing. It’s now told apart by the error text itself — a memory error clears its own cache and tries again with a real chance of succeeding on the very next recording, while a genuinely broken download still gets the full permanent-disable-and-redownload treatment it actually needs.

Zero-CPU ingestion, real encoding only when it’s actually needed

The single biggest cost in local video processing is decoding and re-encoding footage you didn’t need to touch. When a camera’s source codec already plays back fine in a browser (H.264, VP9, AV1), Vaelen writes the compressed stream straight to disk — packet copy, no re-encoding, roughly 1000× cheaper per frame than a real encode pass. CPU stays near idle during recording. Cameras that need it (MJPEG, most raw USB feeds) fall back to real NVENC-accelerated encoding automatically, decided per camera at connection time, not configured by hand.

The 580% that turned out to be one line

A busy machine running Vaelen alongside a dozen other services would occasionally spike hard for about a minute, then go quiet again — the kind of intermittent load that’s nearly impossible to catch with a single snapshot. A continuous profiler running through one of these spikes, though, doesn’t miss it: one function, _get_video_duration, was responsible for 580% of the cumulative CPU time sampled — more than five cores’ worth of work, for a value that never changes once a recording is finished. Every dashboard refresh was re-opening every visible recording’s video container from scratch just to read how long it was, because the function’s own comment, written when the dashboard only ever showed a handful of recordings, assumed that was “fast enough to call for every video on every refresh.” It hadn’t been, for a while. The fix isn’t clever — a duration keyed by the file’s own path, modification time, and size, computed once and never again for that exact file — but finding it required actually measuring where the time went instead of guessing, since the fallback camera connections, a completely unrelated live-preview mechanism, looked like a much more obvious suspect and weren’t the cause at all.

A sort order that actually tells you something

Recordings sorted by date get a wide divider between days instead of a thin label — not just “Tuesday,” but how many recordings that day, the actual span from the first one’s start to the last one’s end (accounting for how long each ran, not just when it started), and same-day totals for transcripts, recognized faces, and matched topics. Recordings can also be sorted by size, duration, topic-match confidence, anomaly score, or how many faces turned up — entirely client-side, re-sorting whatever’s already loaded rather than asking the server again. The day divider only makes sense chronologically, though, so any sort besides date just shows a flat list — recordings from the same day aren’t necessarily still next to each other once you’ve sorted by something else.

A growing file is a live camera, not a waiting game

Most watchfolder-style imports work the same way: drop a file in, wait for it to stop growing, then process it as a finished recording. That’s fine for a file transfer, but it means a genuinely live source — a security export streaming in real time, a platform URL — sits idle for its entire duration before Vaelen even looks at it. Now, a fast streamability probe checks whether an incoming file is already in a playable format (MPEG-TS, or an MP4 with its index up front) the moment it appears. If it is, Vaelen connects to it as a live camera source immediately, running the exact same detection pipeline a real RTSP feed would — no waiting for the recording to end first. YouTube, Twitch, and Vimeo URLs get bridged the same way through yt-dlp, auto-reconnecting if the underlying stream URL expires, decoupled entirely from platform-specific stream lifetimes.

Sensor-fused prompting, not four disconnected signals

The old version of this section was “detection-informed prompting” — ai_analyze.py telling the vision model what YOLO saw, so the description stayed anchored to the actual trigger reason instead of guessing. That’s still true, but it undersold what’s possible once the other sensors are in the room too. Whisper’s transcript, OCR’s visible text, CLAP’s trigger label, a confirmed fall — all of it gets read and folded into the same prompt as a short “here’s what the other sensors already know” paragraph, right alongside the images. Getting the order right mattered more than the prompt text itself: Whisper and OCR have to run before the vision model now, not after, or there’s nothing yet to fold in.

General text recognition, plates as a filter on top

OCR used to be scoped narrowly to vehicles: detect a car, crop it, read whatever text sits inside the box. It’s now a full-frame pass — house numbers, package labels, signage, anything legible on a sampled set of filmstrip frames — with the results feeding the search index directly, in their own field. Plates are what’s left when you filter that same text by “does this overlap a detected vehicle, and does it look plate-shaped” — no separate scan, no separate model.

Face detection that tries harder before giving up on a frame

Switching InsightFace’s model pack from the default to antelopev2 for better accuracy instead produced almost nothing — a handful of faces found where the default pack had found dozens on the same footage. Measuring the actual embedding distances between faces of the same known person told the real story: the clustering threshold had been tuned against the default pack’s typical spread, and antelopev2 simply produces a different one, wide enough that the old threshold treated even genuine matches as unrelated. Separately, and for a completely different reason, a fixed detection resolution was missing faces that were small relative to the frame — common on any wide shot of a square or a harbor rather than a doorbell’s close range. Both are fixed now: clustering sensitivity is chosen per model pack instead of one value for all of them, and a frame that comes back with zero faces at the default resolution gets retried at progressively smaller sizes before it’s written off — a documented behavior of the underlying detection library, not something specific to this project, that’s easy to never notice until you go looking at the actual numbers.

Ad-hoc recording, independent of the pipeline

Sometimes you just want thirty seconds from a camera right now, whether or not the detection pipeline happens to be running. quick_record bypasses the whole trigger/state-machine layer entirely — connects directly, records for the exact duration you asked for, and runs it through the same AI analysis afterward. Works even with the camera disabled everywhere else.

Daily and weekly summaries, for free

Descriptions already exist for every event, so turning a day’s worth of them into “two deliveries, the dog let out once, nothing unusual” is a pure text-summarization call against the same Ollama endpoint already running — no new model, no new infrastructure, just a different prompt over data that was already there.

Periodic segments, for sources that never trigger on their own

A permanent live stream doesn’t reliably produce a “something happened” moment the way a residential camera does. Periodic Segments force exactly that: an interval and a duration, both configurable independently, feeding into the same recording state machine as a real detection so the resulting clip gets the full treatment instead of being a second-class capture mode.

Nothing gets stuck once it’s in the archive

Archiving a recording used to be a one-way door — reanalyzing or re-exporting something already filed away meant deleting it and re-importing it from scratch, if you’d even kept a copy to re-import. Unarchive now does the exact reverse move, with the same care taken over its sidecar files and filmstrip folder as archiving itself: the recording, its metadata, and its thumbnails all move back together, correctly, not just the video file on its own.

One button, and the transcript speaks a different language

A transcript is only as useful as the language you actually read. Translation runs the existing transcript through the same local LLM already doing everything else, caches the result permanently the first time so it’s never redone, and gives you a toggle back to the original at any point — no separate service, no per-use cost, because the model doing it was already running for other reasons.

Bulk actions, and reanalyze that actually works

Selecting several recordings at once now extends to reanalyze, not just archive, export, and delete — each one starts with a short stagger rather than all at once, so a batch doesn’t send every recording’s postprocessing straight into the same GPU-contention problem described above. Reanalyze itself used to have a quiet dead end: a recording whose filmstrip folder had gone missing — most commonly after moving storage to a new drive without carrying the thumbnails along — simply couldn’t be reanalyzed at all, with delete as the only remaining option. It’s regenerated automatically from the source video now, the moment it’s needed, instead of being treated as a permanent loss.

Knowing exactly what the AI was actually asked

Every dynamically generated prompt sent to the local LLM — a scene description, a topic check, a daily summary, an Ask Vaelen question, a Deep Research follow-up — gets logged alongside the response it produced. None of these prompts are static text; they’re built fresh each time from whatever context is available for that specific recording, which means the exact wording genuinely changes call to call. Without this, there was no way to answer “what did we actually ask it” after the fact — only what came back.

Personal notifications, with the picture attached

A push notification is only as useful as how fast you can tell what it’s about, so the trigger screenshot goes out with it — not as a link, as the actual image bytes uploaded directly in the request. Four independent triggers — anomaly, a flagged behavior, a matched topic, any recording at all — collapse to at most one notification per event even if several conditions happen to overlap.

Real bugs, honestly documented

A recording interrupted by a restart sat forever with no description. Stopping the pipeline cleanly while a camera was actively recording closed the video file correctly — moov atom written, fully playable — but the code path that does that never called the same step that hands a finished recording off for analysis. Only the normal end-of-recording path did that. The result looked deceptively fine: a real, playable video sitting in Recent, just permanently missing a description, a filmstrip, everything past the raw file itself — easy to mistake for “still processing” when it was actually never going to finish on its own. Fixed at the same cleanup point that already closes the file, plus a startup sweep that catches anything still stuck this way from before the fix existed.

Playback sped up 3x after a network stall. PTS was being generated sequentially (pts = frame_number), so a delayed frame just got the next number in line — compressing real elapsed time into fewer frames. Fixed with wall-clock-relative timestamps (pts = elapsed_seconds * target_fps), so a stall shows up as a pause, not fast-forward.

Named people lost their identity once every video mentioning them was deleted. The per-person recognition embedding was being recomputed from whatever face rows currently existed in the database — delete all the source videos, and the embedding got wiped to null. The person stayed named, but became permanently unrecognizable. Fixed by archiving a person’s face photo to permanent storage the moment they’re identified, and by no longer deleting a named person’s embedding row when their source video goes away.

Nine files quietly wrote to the wrong database after a folder reorganization. Moving the pipeline’s modules into their own subfolder meant every file that computed its own database or config path relative to its own location started resolving that path one level too deep. Nothing crashed. Each of those nine files simply created a fresh, empty copy of its database and kept working against that, while everything from before the move sat untouched at the old location. Caught by comparing file sizes and timestamps between the two locations side by side, then merging what had accumulated in both.

A search index location, once wrong, stayed wrong forever — including across an entire storage migration. Moving a recording from active storage to archive, or moving the whole storage drive itself to a new mount, could leave the search index pointing at a location that no longer had the file — and every future question about that recording came back as “found a match, but it seems to have been deleted,” even though the footage was sitting right there, just not where the index still thought it was. The fix checks the file’s current, real locations directly by name rather than trusting the stored path at all, correcting the index quietly the moment it notices — the difference between an index that degrades permanently over time and one that heals itself as it goes.

A missing database file turned out to be a directory instead. A Docker bind-mount for a file that doesn’t exist yet on the host gets a directory created in its place instead — silently, correctly, and completely wrong for what the application actually needed there. One un-created file this way meant a single analysis step crashed with unable to open database file, which by itself would have been a two-minute fix — except that unhandled crash was taking three unrelated, otherwise-successful analysis steps down with it, because none of postprocessing’s stages had ever been isolated from each other. One broken step now fails on its own, loudly, in the logs, without preventing the rest of a recording’s analysis from completing.

Manual Record never got a thumbnail. The lightbox’s open-ended “start recording now” button used a raw ffmpeg subprocess, completely bypassing the normal detection loop — which meant it also bypassed the trigger screenshot, the filmstrip, and the AI analysis. Fixed by running the same post-hoc extraction the retroactive backfill tooling already used for older recordings, then kicking off the normal analysis pipeline afterward.

A video submitted through the external API never got a thumbnail either, for a completely different reason. Files coming in this way generated a filmstrip, since that path had been copied from the regular pipeline — but the actual card thumbnail that every other recording gets was simply never written, an omission from whenever this submission path was first built rather than a regression in anything that used to work. The symptom that surfaced it looked unrelated: a portrait-orientation video showing a solid black tile, which pointed everywhere except the real cause — no rotation-metadata bug, no aspect-ratio bug, just a missing file. A landscape recording missing the same file would have shown a perfectly reasonable placeholder icon instead of anything alarming, which is probably why it took a portrait video specifically to notice.

Moving a submitted job’s file could crash the whole job outright. A plain OS-level rename can’t move a file across a filesystem boundary — and a video uploaded through the external API physically lands in one location before moving to permanent storage, which on some setups live on genuinely different filesystems. The failure mode was a dead job with a cryptic “Invalid cross-device link” error and nothing to show for it, reported independently by an AI agent that had been given upload access and hit it on a real submission. The fix is the standard one for this exact situation — fall back to copying the file instead of renaming it when a rename isn’t possible — but it’s the kind of failure that only shows up once files start living on genuinely different disks, which a single-drive development setup will never surface.

A button that silently stopped working until the page was refreshed. The pipeline’s Start/Stop button disables itself the moment it’s clicked, and only re-enables once the server responds — except the code path handling that response had no fallback for the request simply failing outright. A brief network hiccup, or the server being too busy to answer quickly, meant the promise never resolved, the re-enabling code never ran, and the button stayed disabled forever with no error shown — indistinguishable from “did nothing” unless you already knew to suspect this. The same helper function backs three dozen other buttons across the dashboard, all with the identical gap. Fixed once, centrally: a failed request now resolves with a clear error instead of leaving the caller hanging indefinitely.

A close button that vanished against dark footage. The lightbox’s × sits directly on top of the video’s corner, styled in a muted gray with no background of its own — which works fine against bright footage and disappears almost completely against a dark scene, since there’s nothing distinguishing the icon’s color from the video behind it. Now rendered on a small dark circle regardless of what’s playing underneath, the same fix any interface overlaying arbitrary media eventually needs.

Typing a note could delete the recording you were writing about. The lightbox’s keyboard shortcuts — space to play/pause, arrow keys to navigate, Delete to remove the recording — listened globally, with no check for whether a text field currently had focus. Typing a space in the notes field toggled playback instead of typing a space; pressing Delete while editing a note would have deleted the whole recording instead of a character. Now checked once, cleanly: any of the three defers to normal text-editing behavior the moment an input or textarea has focus.

Some MJPEG cameras recorded and analyzed correctly but never showed a live preview. Recording goes through a different connection path than the dashboard’s live thumbnail does, and the thumbnail path was letting the underlying library auto-detect which backend to use for a given camera URL. That auto-detection doesn’t reliably land on the more capable option for older MJPEG-over-HTTP multipart cameras — the classic CGI-style endpoints many budget and older IP cameras still expose — even though the exact same URL worked fine through the path that doesn’t leave the choice to guesswork. Forcing the same backend explicitly, rather than letting it be inferred, was enough.

Where this is headed

Vaelen today is ambient perception — it senses and understands continuously, in the background, without asking for attention until something’s actually worth it. It doesn’t act on any of that yet; it only ever tells you or an agent you’ve authorized. Whether “understood something happened” ever grows into “did something about it” is an open question, not a roadmap item. For now, the honest scope is perception, not action.

Realtime Mode is a step toward the “act faster” side of that boundary without actually crossing it — a preliminary notice, never an automatic response. The deeper open question is the same one it’s always been: the individual signals — description, topics, transcript, faces, plates, pose, anomaly score — are still fundamentally per-recording. The obvious next step isn’t another detector; it’s making that understanding compound across recordings, cameras, and time the way a person’s memory does. A named face and a named vehicle linking into one tracked entity. Ask Vaelen reasoning across a whole day instead of a single clip’s context window. An anomaly that isn’t just “statistically unlike this camera’s history,” but “unlike what this specific person or vehicle normally does.” The pieces already exist. What’s missing is the layer that stitches them into one continuous understanding instead of several separate ones.

Frequently asked questions

What is Vaelen? Vaelen is a self-hosted, AI-powered camera system that records on detection and understands what it recorded — in plain language, searchable by meaning — instead of leaving you to scrub through raw footage yourself.

Is Vaelen free and open source? Yes. The full source is on GitHub under an open license, self-hosted on your own hardware — no subscription, no monthly fee, no account required.

Does Vaelen need the cloud or an internet connection to work? No. Detection, analysis, transcription, face recognition, and the conversational search all run locally on your own machine and GPU. Nothing about a recording or a question you ask it leaves your network, ever.

What hardware does Vaelen need? Any NVIDIA GPU from the Turing generation (RTX 20-series) through Blackwell (RTX 50-series) works unmodified — the same codebase self-tests FP16/cuDNN safety and adapts at startup. A GPU makes AI analysis fast; recording itself works even without one.

How is Vaelen different from Frigate or Motion? Frigate and Motion are recording systems with detection bolted on — you still watch and search the video yourself. Vaelen treats the video as raw material for a small team of specialized AI models (vision, speech, faces, sound, anomaly detection) working together, and lets you ask a plain-language question and get a direct, sourced answer instead of a list of clips to review by hand.

Can I ask Vaelen questions about what my cameras have seen? Yes — that’s the core of it. Ask Vaelen anything in plain language (“was anyone at the door yesterday who wasn’t a delivery?”) and it searches everything by meaning, then answers in a sentence with the source recordings linked underneath, in whichever language you ask in.

Does Vaelen support RTSP, RTMP, MJPEG, and USB cameras? Yes, along with local USB/V4L2 webcams and even live platform sources like YouTube or Twitch streams — the same detection and recording pipeline runs regardless of the source.

Get it

Bare-metal or Docker install instructions, the full settings reference, and the complete source: Vaelen on GitHub. (Private at the moment due to an internal security review)

Ambient perception was supposed to be the modest part. It’s not supposed to notice this much — it does anyway.