Coding with AI agents carries a quiet trap: the illusion of competence.
Tasks close in record time, architecture moves forward, and everything builds cleanly. But at the end of the day, an uncomfortable question remain: Did I actually learn any of this, or was I just acting as a prompt supervisor?
In an intend to prevent AI from turning into a mindless "code vending machine," I designed an integrated learning system where the agent doesn't just write software—it documents my learning journey, maps concepts, and actively tests my comprehension.
Here is how the system works:
The Core Rule: The Vault Stores Comprehension, Not Code
I maintain an Obsidian vault (Engineering/) directly connected to my AI agent sessions via automated hooks and skills.
The foundational rule is strict: the vault never duplicates repository docs or generic boilerplate. It only captures:
- Mental models and architectural analogies.
- Real trade-offs and discarded alternatives.
- Current Architecture Decision Records (ADRs) and their change history.
- Especific markers distinguishing verified facts
[V]from assumptions[A].
System Architecture: 3 Key Layers
Architecture of the learning pipeline: from session classification to milestone exams.
1. Contextual Project Modes (learn, log, off)
Whenever an agent session starts inside any repository, it checks the configured mode:
-
learn: Full pedagogical depth. Generates architecture notes, links concepts, and triggers milestone exams. -
log: Lightweight mode. Only tracks architectural decisions (ADRs) and concept links. -
off: Zero overhead. No notes written to the vault.
2. Background Concept Distillation
Whenever I ask a conceptual question during a session—such as "What is backpressure?" or "Why prefer this concurrency pattern here?"—the agent answers in chat and quietly reports:
Saved "Backpressure" to Concepts.
Concepts start small: a 3–5 line Term (definition, quick example, and the context where it surfaced). As the same pattern appears across different projects, it matures in place into a deeper Concept. Links are embedded organically in the body text, weaving a knowledge web over time.
3. Milestone Exams (Active Recall)
Passive note-taking doesn't build long-term retention. To counter this, whenever a milestone is reached in a learn project, the agent offers an exam:
- 10 to 12 open-ended questions (no multiple choice—multiple choice tests recognition, not recall).
- Grounded strictly in the actual repository code, decisions made, and real bugs encountered during development.
- Covers real code reading, failure scenarios, and change propagation ("If we swap dependency X tomorrow, what could break and why?").
- Supports an oral mode where questions are evaluated through speech and voice transcription.
If I cannot defend and explain why an architectural path was chosen, the task is not considered truly learned.
What Changed in My Daily Routine
- True Retention: The codebase doesn't feel foreign when returning to it two weeks later.
- Cross-Project Synergy: A concept learned while fixing a bug in Rust or Go naturally connects to an architectural decision made months later in a TypeScript API.
- AI as a Socratic Partner: The agent transitions from an autocomplete utility into a technical partner that demands conceptual clarity before and after shipping code.
If you are using AI agents to write software, consider designing your workflow to amplify your own understanding, not replace it.
Do you have a system for retaining what you learn while coding with AI? Let's discuss your setup in the comments below.
Top comments (2)
I think a lot about how easy it is to build things with AI without necessarily retaining everything you should be learning along the way. I use agents all the time in my own projects, and I’ve found that the things that really stick are the things I take the time to investigate and understand, not just the code that gets generated.
I especially like your distinction between storing comprehension rather than just duplicating documentation. The milestone exams are a great idea too. Having to actually explain why something works or why a particular decision was made is a pretty good way to find out whether you’ve learned it or just watched it happen.
Really thoughtful system you’ve put together here.
Thanks for your words Jessica! What you said, "learning it vs just watching happen" nails the exact point that pushed me to build this. When you are building fast, as it is with agents, it becomes really easy to trick yourself into believing you understood an architectural decision just because you approved it.
If you are interest about how the milestone exams work in practise, I recently open-sourced the tool I use to run the oral mode: Socra on GitHub. It is a local, private voice layer (powered by Piper TTS and local Whisper STT) that lets the agent speak questions aloud and evaluate your spoken answer in real time.
The project is still early and Windows-first, so any feedback, suggestions, or contributions to shape the workflow are more than welcome :). Would love to hear your thoughts if you check it out!