Somewhere around day four of building a doorbell-camera AI for a hackathon,
I caught my own system lying to me. Not maliciously — it was just guessing,
the way all of us do when we don't want to say "I don't know" out loud. The
fix for that turned out to be more interesting than anything else in the
build, so this post is mostly about that, with some genuinely funny/painful
bug stories along the way.
The problem nobody built a camera to solve
Every doorbell camera on the market answers one question: was there
motion? That's the wrong question. The right question — the one a human
actually wants answered at 11pm when their phone buzzes — is should I
care about this? Cameras don't answer that, so people train themselves to
swipe the notification away without looking. The alert becomes noise. The
one time it isn't noise, you've already stopped paying attention.
So the project — I called it Kenbunshoku (見聞色, roughly "the art of
perceiving," a term from certain martial traditions for reading a
situation before it becomes obvious) — tries to add one missing layer:
context. Not "something moved." Something closer to "a person in a
delivery uniform is standing at your door with a package," or "someone is
lingering by the entrance at an unusual hour without a clear reason to be
there."
The architecture, in one breath: a local YOLOv8n detector watches a camera
stream and only forwards frames that actually contain a person. Those
frames go to a cloud backend that calls Qwen-VL (Alibaba Cloud's
vision-language model) to reason about what's actually happening in the
image. A lightweight memory layer checks whether this is a recurring
pattern. A push notification goes to the homeowner's phone with the
reasoning attached, in plain language. That's the whole system. No part of
it ever acts on your behalf — no locks, no alarms, no calls to anyone. It
tells you things. You decide.
That last sentence is doing more work than it looks like, and I'll come
back to it.
Here it is actually running, filmed on my phone with zero production
values whatsoever — a real detection, a real Qwen-VL classification, a
real push notification landing on a real device:
(Judge the system, not the cinematography.)
The lie I caught it telling
Early on, the classification prompt just asked Qwen-VL to look at a frame
and decide: is this person "familiar," "delivery-like," or "anomalous"? I
tested it against a deliberately ugly, blurry, low-detail image — barely a
silhouette — expecting either a refusal or an honest "unclear." Instead I
got:
"familiar" — "...suggesting they are a known individual."
Read that again. The model had no reference photo of anyone who lives at
this house. It had never seen this "known individual" before in its
existence. It could not possibly know that. And yet it produced a
confident, specific, entirely fabricated claim of recognition — because I
had asked it a question shaped like an identity question, and it answered
in kind, filling the gap with plausible-sounding language instead of
admitting the gap existed.
This is, if you squint, the exact same failure mode as a person who
doesn't want to look uninformed in a meeting. The information isn't
there, but the shape of a confident answer is easy to produce, and
confident-shaped answers get rewarded more often than "I don't know"
ones — right up until the moment they're wrong in a way that matters.
The fix wasn't a bigger model or a longer prompt lecture. It was
redefining what "familiar" was even allowed to mean:
"familiar" means the visit presents as expected and low-concern —
calm demeanor, ordinary daytime approach, no unusual items — not that
you recognize the person's identity. You have no reference photos.
And one more line that did almost all of the actual work:
If the visual evidence doesn't clearly support "familiar" or
"delivery-like," choose "anomalous" rather than guessing.
Same blurry test image, same model, new prompt:
"anomalous" — "The image is too blurry to determine any clear
behavior, carried items, or context, making it impossible to classify
as familiar or delivery-like."
That's not a worse answer. It's a categorically more honest one. The
model went from confabulating a memory it didn't have to correctly
reporting the limits of what it could see. For a home security system,
"I'm not sure, so I'm erring cautious" is strictly better than a
confident wrong guess in either direction — and it only took giving the
model permission to say so.
Refusing to build the "creepier, more expensive" version
The natural next question, if you're chasing "familiar" as identity
recognition, is: why not just do real face matching? Store embeddings,
compare against known visitors, get an actual identity-confidence score
back.
I sat with that question for a while and landed on: no, deliberately,
permanently, not out of laziness but out of principle.
Here's the reasoning. A system that stores biometric-adjacent data about
everyone who walks up to your door is a fundamentally different kind of
object than one that doesn't — different risk if it leaks, different
ethical weight, different relationship to the people it observes (some of
whom never consented to being in anyone's facial recognition database,
let alone a hackathon project's). And critically: it wasn't necessary. The
actual product need — "don't wake me up for the same delivery guy every
Tuesday" — is fully satisfied by a much cheaper, much more honest
mechanism: pattern-matching on visits, not identity.
The memory layer just asks: has this camera seen a similar classification,
on the same day of week, at a similar time, repeatedly, in the last two
weeks? If yes, that's a "recognized pattern" — still logged, but the push
notification gets quietly suppressed instead of buzzing your phone for the
fifth Tuesday in a row. If a genuinely unrecognized, anomalous visit
happens even in a pattern-matched slot, it still gets through — the system
never goes quiet on "anomalous," patterned or not.
This costs nothing extra. No embeddings, no vector database, no additional
model calls, no image storage beyond what's needed for the one live
classification. It's just a SQLite table and a modest SQL query. And it
never claims to know something it doesn't. "This time slot has a routine"
is a true, verifiable, honest claim. "I recognize this specific human
being" is not a claim this system is entitled to make, so I made sure it
never could.
Cheap and honest turned out to be the same design decision twice.
Restraint as the actual feature
The system never takes autonomous action. Not "rarely" — never, as a hard
rule I refused to relax even when it would have been a more impressive
demo. No auto-locking doors on "anomalous," no auto-calls to anyone, no
control signal of any kind, ever. Just a notification with reasoning
attached, to a human, who decides.
It would be easy to read that as a limitation — "the AI could do more but
we held it back." I'd argue the opposite: the restraint is the design.
A system that watches your front door and occasionally gets it wrong (all
of them do) is tolerable precisely because the cost of being wrong is "a
slightly annoying notification," not "the system did something to a real
door because a vision model was 73% confident." Keeping a human in the
loop isn't a feature you bolt on later. It's the thing that makes
everything upstream of it — including the confident-sounding mistakes —
survivable.
The bugs that only existed once I stopped trusting the diff
Every single one of these passed a code review. All of them looked fine on
the page. None of them were fine.
-
An em dash crashed a push notification. The alert title used a
plain em dash for style. HTTP header values have to be ASCII/latin-1.
The request didn't fail loudly in a way that pointed at the cause — it
just threw a
UnicodeEncodeErrordeep inside a networking library, the first time a real alert actually tried to fire. Nothing about the code looked wrong. -
A mobile OS silently ate all outbound network traffic. The
notification client connected to nothing, showed no error, gave no
signal that anything was wrong — because macOS's App Sandbox was
quietly blocking every request at the entitlement level, before it ever
reached a socket. Android had the mirror-image problem: no
INTERNETpermission declared, same silent nothing. Static analysis had nothing to say about either one. Only running the actual app on actual hardware surfaced them. -
The camera loop got stuck reprocessing the past. Point the camera
at your hand, then a pen, then your face — and the system kept
reporting the pen. Not broken, exactly:
cv2.VideoCapturebuffers incoming frames faster than a multi-second classification round-trip can drain them, so the loop was dutifully working through a growing backlog of old frames while the live scene moved on without it. The fix was a background thread that keeps only the newest frame and lets everything older get silently, deliberately discarded — the standard answer to this exact class of problem, but only obvious once you've watched it happen with your own hand in front of your own camera. - The system got slower the more careful it became. A single request was quietly doing two sequential network calls before responding — one to the vision model, one to the push service — with no timeout set on either. Most of the time it was fine. Occasionally, both hops had a slow moment at once, blew past the timeout budget, and the system fell back to a generic "cloud unreachable" alert instead of the real classification it had actually computed a few seconds too late.
None of these are exotic. Every one of them is a well-known category of
bug with a well-known fix. What they have in common is that reading the
code gave no signal anything was wrong. The only way to find any of them
was to run the actual thing, on actual hardware, against an actual
network, and watch it fail in front of me.
The recursive part
I built this with an AI pair programmer doing most of the typing. There's
an irony there I didn't fully appreciate until I was three days in:
I spent this entire project teaching one AI system to stop confidently
guessing and start admitting uncertainty — while collaborating, in real
time, with a different AI system whose suggestions also needed
constant grounding against reality rather than being trusted on the
strength of how plausible they sounded. Code that compiles is not code
that works. A plan that reads well is not a plan that survives contact
with a real phone, a real Alibaba Cloud console, a real curl command
against a real server.
The throughline for the whole build, human-authored parts and
AI-authored parts alike, ended up being the same single sentence: verify
against reality, not against your own confidence. That sentence is the
entire fix for the "familiar" hallucination. It's the entire fix for
every bug in the list above. It's also, probably, decent advice for
anyone shipping anything, with or without a vision model attached.
Kenbunshoku was built for the Global AI Hackathon Series with Qwen
Cloud, running on Alibaba Cloud ECS with Qwen-VL for visitor reasoning.
Camera-agnostic by design, human-in-the-loop by principle, and — as of
this week — a little more honest about what it doesn't know.
Top comments (7)
The “I don’t know” path is the product feature here. Vision systems get safer when uncertainty is explicit, especially if the UI can show what evidence was missing instead of turning low confidence into a confident wrong answer.
Yeah, exactly — "anomalous" isn't a leftover bucket, it's the uncertainty slot, and that was the actual design decision, not an afterthought. Your point about surfacing what evidence was missing in the UI rather than just the label is a good one I hadn't taken far enough — right now the reasoning text hints at it ("too blurry to determine...") but it's not structured or shown distinctly from a confident classification. That's a real gap. I appreciate you naming it plainly.
That evidence-missing UI is where uncertainty becomes useful. A label like anomalous is easy to ignore or overtrust; showing which signal was absent helps the operator decide whether to investigate, retry, or wait for more context.
The prompt fix works because you had somewhere to put the uncertainty. That is the part I would flag for anyone porting this idea to speech: a transcription decoder has no abstain token. Whisper cannot return "no speech here" the way you got Qwen-VL to return "anomalous". It emits tokens by construction, so on a clip of pure silence it produces the most probable thing a human would have said, which is why anyone who has run it on dead air has watched a stray "thank you" or a subtitle credit line appear out of nowhere. Same confabulation, same confident shape, except there is no prompt line I can write that gives it permission to say nothing.
So the equivalent of your fix in audio sits one layer earlier: you do not ask the model to abstain, you stop handing it ambiguous input. Voice activity detection, or in my case a push-to-talk key so a human decides where the clip starts and ends. Same move as your "if the visual evidence does not clearly support familiar, choose anomalous", except the gate has to live outside the model, because the model has no gate.
Your frame backlog bug has an exact audio twin, and it bites harder because you cannot see it. A stale video frame is obviously wrong the second you wave your hand at the camera. Stale audio just sounds like a plausible sentence that happens to be the previous one, and nothing about the output tells you it is late.
This is a sharper framing than what I had, honestly — I hadn't drawn the distinction between "a label set with room for an uncertainty class" and "a decoder that has no abstain token by construction." That's exactly right and I didn't think about it in those terms while writing this. The push-to-talk/VAD point makes sense as the audio-side equivalent of what I did — moving the gate upstream of the model instead of trying to get the model to self-report, since there's no token path for it to do that even if I asked. Going to go read more around this (decoder abstention / OOD behavior in ASR specifically) — thanks for laying it out this clearly, it's a better mental model than the one I had.
If you do go read around it, the terms that actually surface the useful work are worth knowing, because "abstention" is not what the speech literature calls it. Look for hallucination on non-speech segments, and for confidence estimation in end-to-end ASR. The practical proxies people use are the no-speech probability that Whisper already returns per segment, and average log probability across the segment. Both are available and mostly ignored.
Fair warning on those two: they are miscalibrated exactly where you need them. A clean hallucination on silence often comes back with a comfortable log probability, because the model is genuinely confident that "thank you" is what a human would have said there. The score measures fluency, not evidence, which is the same failure your blurry image produced before you changed the label set. It is why I lean on the upstream gate rather than on a threshold: the gate looks at the audio, and the threshold only ever looks at the model's opinion of itself.
This is really useful, thank you — "abstention" was clearly the wrong search term and I'd have wasted time on it. Noting: hallucination on non-speech segments, confidence estimation in end-to-end ASR, no-speech probability and average log-prob as the existing (if underused) proxies.
The miscalibration point is the one that actually reframes it for me though. A fluent hallucination scoring well on log-probability is the exact same failure shape as my blurry image scoring "familiar" with confidence before I changed the label set — the number and the word are both measuring "does this look like a normal thing to output," not "is there evidence for it." I hadn't connected that a confidence score can be just as ungrounded as a label if it's derived from the same fluency signal. Makes the case for the upstream gate over a threshold pretty airtight — you can't fix a model grading its own fluency by asking it to grade its own fluency more carefully. Appreciate you walking this all the way through, genuinely learned something concrete here.