Vigil: The Camera System That Actually Watches Back

Most home camera systems are glorified tape recorders. Something moves, a clip gets saved, and now it’s your job to scrub through hours of footage to figure out if it mattered. Vigil doesn’t work like that.

Vigil watches your cameras, decides what’s actually worth keeping, and then tells you — in plain English — what happened. It notices when someone falls. It hears glass breaking in a room the camera can’t even see. It flags the one recording out of a thousand that doesn’t look like anything else on that camera, without you ever telling it what “weird” means. And when you’re not around to check it yourself, you can hand the keys to an AI agent and let it watch your perimeter for you.

All of it runs on one machine you own, with one GPU you already have. No cloud, no subscription, no monthly fee, nothing phoning home.

Here’s what that actually looks like.

What Vigil does that nothing else does

It hears what it can’t see. A dedicated audio model (CLAP) listens continuously, completely independent of the camera image. Type “glass breaking” or “dog barking” or “someone shouting” into the dashboard in plain English, and it starts triggering on that sound immediately — no training data, no retraining, no fixed list of categories to pick from. A window breaking two rooms away, in the dark, with nothing in frame — Vigil still knows something happened.

It notices when someone’s in trouble. A separate pose-estimation pass runs whenever a person is already detected — at zero extra GPU cost, since it reuses the same frame. From that one pass, Vigil reads six independent signals: a fall, a raised-hands distress gesture, someone loitering in the same spot too long, a person moving unusually fast, two people in sustained close proximity, and which way someone’s facing. Every one of them is its own on/off switch with its own threshold — turn on just fall detection, or all six, or none.

It tells you what happened, in words. After a recording finishes, a vision-language model looks at the footage and writes an actual description — “a delivery van pulls up and a package is left at the door” — not a bounding box, a sentence. It knows what YOLO detected and what audio triggered it, so the description is anchored to the actual reason the camera fired, not a generic guess. Ask it to watch for specific things (“break-in,” “mail carrier,” whatever you want) and every recording gets scored against them, 0 to 100.

It finds the recording that doesn’t fit — without you defining a single rule. The same text embeddings used for search get recycled into a per-camera Isolation Forest model. It learns what “normal” looks like for that specific camera over time and flags statistical outliers automatically. Zero extra GPU cycles, because the embedding was already being computed anyway.

You can search it by meaning, not keywords. “Person carrying a box” finds “individual holding a package.” It’s the same semantic search under the hood as everything else, over descriptions, transcripts, topics, and named people together.

It knows who’s in frame — permanently. Face recognition runs in-process (no external microservice, no network hop — the embeddings are extracted from frames already sitting in VRAM), clusters unknown faces automatically, and once you name someone, their identity is archived independently of any single recording. Delete the video that first introduced them to the system a year later, and Vigil still recognizes them tomorrow.

An AI agent can run it for you. This is the one nobody else has. Vigil ships a full MCP server — 21 tools, the same ones any script could call over the REST API — so an agent like Claude can check what a camera saw, toggle cameras on or off, kick off a recording, search past events, and read the system’s own health, all through a real, typed interface instead of screen-scraping a dashboard. A layered permission system decides exactly what it’s allowed to touch, off by default, with deletion intentionally left unreachable no matter what’s toggled on.

Vigil vs. Motion vs. Frigate

	Motion / MotionEye	Frigate NVR	Vigil
Trigger	Pixel difference	Object detection (YOLO)	YOLO + audio (CLAP) + pose/fall
Model interaction	None	Isolated / parallel	Detection feeds the description prompt
Face recognition	None	External / third-party	In-process, embeddings survive video deletion
Scene description	None	Optional VLM	VLM + structured topic scoring
Anomaly detection	None	None	Isolation Forest, zero extra GPU cost
Agent control	None	None	Native MCP server, 21 tools
Ad-hoc recording	—	—	Works even with the pipeline stopped
Home Assistant	Basic webhooks	Deep UI integration	Native MQTT auto-discovery
Photo manager export	Standard MP4	Internal DB/API	Immich-compatible XMP sidecars

Choose Motion if you want a simple daemon on minimal hardware with no GPU. Choose Frigate if you want a mature, broadly-supported NVR with wide accelerator support (Coral, Hailo, OpenVINO) and a bigger community. Choose Vigil if you want one strong GPU doing real reasoning about what it sees and hears, and you want the option to let an agent operate it.

---

How it’s built

Vigil chains small, specialized models instead of asking one giant model to do everything — each one gets handed exactly the context it needs from the stage before it.

[ Triggers ]
├─ YOLO (v10 / v12 / v26)
├─ CLAP audio
├─ InsightFace (in-process)
└─ Pose: fall, distress, loitering, movement, proximity, gaze
│
▼
[ Zero-CPU ingestion ] RTSP / RTMP / MJPEG / USB / Watchfolder
│
▼
[ Ollama VLM ] description + topic scoring, prompted with what triggered it
│
▼
[ sentence-transformers ] semantic search index
│
▼
[ Isolation Forest ] anomaly detection, recycled from the search embedding
│
▼
[ Integrations ] MCP (21 tools) · Home Assistant/MQTT · REST API + webhooks

Zero-CPU ingestion, real encoding only when it’s actually needed

The single biggest cost in local video processing is decoding and re-encoding footage you didn’t need to touch. When a camera’s source codec already plays back fine in a browser (H.264, VP9, AV1), Vigil writes the compressed stream straight to disk — packet copy, no re-encoding, roughly 1000× cheaper per frame than a real encode pass. CPU stays near idle during recording. Cameras that need it (MJPEG, most raw USB feeds) fall back to real NVENC-accelerated encoding automatically, decided per camera at connection time, not configured by hand.

Detection-informed prompting, not a generic “what’s in this video?”

Instead of handing the VLM a clip and hoping, ai_analyze.py builds the prompt around what actually fired: “YOLO detected a person near a vehicle. Focus on what they’re carrying, their direction of movement, and whether this matches the watch-topic ‘delivery.'” The model isn’t guessing what matters — it’s already been told. Topic scoring runs as a structured JSON call against your own categories, each one a 0–100 confidence score, not a single label.

Pose estimation, one model, six independent signals

All six behavior evaluations — fall, raised-hands, loitering, movement speed, proximity, gaze direction — come from the same pose keypoints extracted once per person already spotted by the main detector. None of them fire on a single frame: a fall needs several consecutive frames reading as “lying down” before it counts, precisely so that tying your shoes doesn’t read as a medical emergency. Loitering and movement are pure position-tracking, no pose model needed at all — they work off the same bounding boxes the main detector already produces.

Ad-hoc recording, independent of the pipeline

Sometimes you just want thirty seconds from a camera right now, whether or not the detection pipeline happens to be running. quick_record bypasses the whole trigger/state-machine layer entirely — connects directly, records for the exact duration you asked for, and runs it through the same AI analysis afterward. Works even with the camera disabled everywhere else.

Daily and weekly summaries, for free

Descriptions already exist for every event, so turning a day’s worth of them into “two deliveries, the dog let out once, nothing unusual” is a pure text-summarization call against the same Ollama endpoint already running — no new model, no new infrastructure, just a different prompt over data that was already there.

---

Real bugs, honestly documented

Playback sped up 3x after a network stall. PTS was being generated sequentially (pts = frame_number), so a delayed frame just got the next number in line — compressing real elapsed time into fewer frames. Fixed with wall-clock-relative timestamps (pts = elapsed_seconds * target_fps), so a stall shows up as a pause, not fast-forward.

Long recordings lost their last few minutes of thumbnail coverage. Reservoir sampling picks frames uniformly across a clip’s whole duration, which sounds fine until you do the math on a low sample count and a multi-hour event: the tail end can end up statistically unrepresented. A second, independent bug compounded it — a later frame-selection step for what actually gets sent to the vision model used a formula that mathematically never reached the last index in the list, silently dropping the last several frames from what the model even saw. Fixed both: a guaranteed final-frame slot in the filmstrip, and an index formula that provably includes the first and last frame.

Named people lost their identity once every video mentioning them was deleted. This one mattered the most. The per-person recognition embedding was being recomputed from whatever face rows currently existed in the database — delete all the source videos, and the embedding got wiped to null. The person stayed named, but became permanently unrecognizable. Fixed by archiving a person’s face photo to permanent storage the moment they’re identified, and by no longer deleting a named person’s embedding row when their source video goes away. Now the system stays learning, not forgetting.

---

Get it

Bare-metal or Docker install instructions, the full settings reference, and the complete source: vigil on GitHub.