This article is my submission for the Agents of SigNoz Hackathon: Blog Track, where participants instrument real applications with OpenTelemetry an...
For further actions, you may consider blocking this person and/or reporting abuse
Tagging every token metric with the repo name is the move that turns a number into an answer, and it's the part people skip because a raw counter looks like it's already enough. Small thing worth flagging for anyone copying this:
prompt_charstracks with token count until it doesn't. Code blocks and repeated tokens compress differently, so a prompt that's twice the characters can be well under twice the tokens. Handy as a cheap proxy in a span, but I'd be careful about reasoning too hard from it once you're comparing repos with very different file types. Are you thinking of eventually swapping it for the real token count on the span, or is the char length good enough for spotting the outliers?That's a really sharp catch, and you're right, I was careful not to over-claim prompt_chars in the article but the cross-repo comparison risk is worth flagging explicitly for anyone copying the pattern.
The real token count is already tracked as gemini.tokens.prompt per repo in the metrics, but it's not on the span, so we can't correlate it directly with trace timing. The proper fix is adding gemini.prompt_tokens as a span attribute once the Gemini response comes back, so the real count and the timing live in the same trace context.
For now prompt_chars does exactly what I needed, spotting which batch caused the latency spike within a single analysis. But I wouldn't use it for cross-repo token cost comparisons. That's what the metric is for.
I tested the project with one of my own repositories, BurnCPU, a self-hosted social networking platform built with Rust/Axum, SolidJS, PostgreSQL, Redis, and Meilisearch. The analyzer processed one repository and classified the profile at Staff level, with scores such as 10/10 for security, engineering, documentation, and overall breadth, while also identifying more realistic weaknesses around frontend complexity, large modules, React hook lint exceptions, and some older XMLHttpRequest-based mobile code.
What I liked most is that the output was not limited to generic praise. Several observations were actually connected to architectural decisions inside the project: layered security, CI/CD, end-to-end testing, caching, background processing, database scaling, SSE, moderation, and ActivityPub federation. That made the result feel more like an engineering review than a simple GitHub profile summary.
At the same time, using it on a repository I know deeply also exposed an important limitation: a GitHub analyzer can only evaluate the evidence visible in the repository. It cannot fully understand operational incidents, architectural trade-offs, private infrastructure, code ownership, mentoring, deployment history, or why a deliberate compromise exists. For example, BurnCPU is intentionally designed to run a relatively complex social platform on a very small self-hosted footprint. The report recognized the “1 VPS” aspect, but the real engineering value is not just the number of containers or technologies; it is the operational discipline required to keep the whole system maintainable under those constraints.
This is also why observability matters so much for an AI assessment system. Latency and token usage are useful, but I would also want to see:
The OpenTelemetry approach in the article is a strong foundation because it makes the hidden AI pipeline visible. However, I think the next step should be connecting technical telemetry with evaluation telemetry. A system can become faster and consume fewer tokens while producing less reliable assessments. For an AI analyzer, the real dashboard should combine latency, cost, and quality rather than treating performance alone as success.
There are also a few implementation details worth reviewing. If GitHub requests are made through aiohttp, RequestsInstrumentor will not automatically trace those calls; the aiohttp client instrumentation should be used. Retry attempts around Gemini would also be more useful as individual spans or span events instead of one long gemini.generate operation. Repository names as metric labels may create high-cardinality time series at scale, so I would keep detailed repository identity in traces and use bounded attributes for metrics.
One more concern is repository privacy. An analyzer that sends source files to an external model should clearly explain which files are excluded, whether secrets are scanned, what telemetry attributes are exported, and how private repositories are handled. Observability should help investigate data flow without becoming another source of leakage itself.
Overall, testing it with BurnCPU made the article much more concrete for me. The analyzer produced a surprisingly relevant high-level review and correctly detected both strengths and areas that deserve cleanup. I would not treat the generated seniority level as an absolute judgment, but as an evidence-based snapshot of what the public repository communicates.
The most valuable part of this project may not be the score itself. It is the ability to trace how the system reached that score, measure how stable the result is, and show the developer which evidence shaped the conclusion. That is where OpenTelemetry can turn an interesting AI demo into a trustworthy engineering tool.
This is my first comment on the article, and honestly I couldn't have asked for a better first one.
You're absolutely right about the limitation: the analyzer can only evaluate what's visible in the repository. Operational discipline, architectural trade-offs, and deliberate constraints like running a complex platform on a single VPS are
invisible to it. That's an honest gap I hadn't articulated clearly enough in the article.
The observability points you raised are things I hadn't considered at all- connecting technical telemetry with evaluation telemetry, tracking score consistency across repeated analyses, and confidence/evidence-coverage values.
The aiohttp vs RequestsInstrumentor point is also a real bug. GitHub file fetching uses aiohttp for concurrent requests, so those spans aren't actually being traced by RequestsInstrumentor. That's something I need to fix.
To be honest, this was my first real dive into observability, OpenTelemetry, and SigNoz. I learned a lot building it, and your comment has given me a clear direction for what to improve next.
Thank you again for your time, and your feedback.
I’m really glad it was useful. We all start somewhere, and honestly, building something real is the fastest way to learn observability. I think you’re already on the right track because you treated telemetry as an engineering tool rather than just another dashboard. I’m looking forward to seeing how the project evolves—especially once evaluation telemetry becomes part of the picture. Keep building in public!
Thank you, it means a lot, especially coming from someone who gave such thorough feedback.
"Treating telemetry as an engineering tool rather than just another dashboard" is exactly the framing I was missing when I started. I
was thinking about it as monitoring. Your comment made me see it as evidence.
The evaluation telemetry idea is staying with me. Tracking score consistency across repeated analyses and connecting it to prompt
version and model response quality- that's the next real step for GitIntel. Will definitely build in public as it evolves.
That shift from monitoring to evidence is exactly the important part. Once you start tracking prompt versions, score stability, confidence, and the evidence behind each result, GitIntel will become much more than a repository scoring tool—it can become a genuinely trustworthy engineering assessment system.
You’ve already done the hardest part: you built something real, observed its weaknesses honestly, and stayed open to improving it. I’m genuinely looking forward to the next version.
That framing- "genuinely trustworthy engineering assessment system", is exactly what I want GitIntel to grow into. Not just a tool
that produces scores, but one where every score is traceable, stable, and explainable.
Fixing the aiohttp instrumentation is next on my list, and I'm already thinking about how to structure evaluation telemetry alongside
the existing performance metrics. Your point about connecting prompt version and model response quality to score consistency gave me
a concrete starting point.
Will share the next version here when it's ready. Thank you again for all your insights, this kind of feedback is rare.
My argument for tracing agent workflows comes from getting burned without it: my long-term memory plugin stayed completely dead for two days after a dependency update, with zero errors in the logs. The plugin failed silently on connect, so every "success" was really a no-op. I found it by accident, not by monitoring — which is exactly the gap. Logs told me everything was fine; only execution-flow visibility would have shown that a step I assumed was running never actually did.
Two days with zero errors in the logs while the whole thing was silently doing nothing, that's genuinely painful. And the worst part is you can't even be mad at the logs, they were technically telling the truth.
That's exactly why I ended up going deeper than just HTTP instrumentation. Knowing a request "succeeded" means nothing if a step inside it quietly no-oped. We need to see the execution path, not just the outcome.
Your memory plugin story is going in my mental list of "why you gotta trace everything."
Welcome back!
It's been a while since your last post, and this one was worth the wait. I really liked GitIntel. You always come up with interesting ideas and turn them into something real. What I admire even more is that you don't stop after building it. You observe how it performs, figure out what can be improved, and keep refining it.
What I enjoyed most was that you didn't just showcase the final app and call it a day. You shared where the idea came from, why you decided to build it, the challenges you faced while instrumenting it, the tradeoffs you made, and how you iterated until you were happy with the result. It made the whole journey feel real, like I was building it alongside you instead of just reading a tutorial.
You're still not 100% satisfied and always looking for ways to make it better. That's the OG developer vibe. ✨
All the very best for the hackathon ✨✨
Thank you so much for checking it out and this wholesome feedback. Glad to know it was worth the wait:))
I liked working on it as well, like adding the apis and making them work together to get the results was easy part, but it was my 1st ever project using telemetery, observing the apis, signoz, and all, so it was hectic. The best part for me was as always- building the project, but observing all the data, and the visualizations, creating the custom dashboard- it was a new experience altogether.
And, thank you for your kind wishes 😊
Excellent article, Divya!❤️ Wait—you and I have the same name! Haha.
Thank you for checking it out Divyanshi 😊✨
More like my name is a subset of yours 😁
My nickname is Divyanshi; officially, I am Divya! Lol 🙂
Well, gotta admit, you've a lovely name ☺️
Thanks Divya :D
Excellent article & implementation, awesome buddy! A great project ✨️
Thank you so much for checking it out buddy!
And yup, the compliment, esp. coming from you, means a lot.
Great write-up! The custom metrics with repo attributes were my favorite part. Seeing latency, token usage, and traces together makes AI debugging evidence-driven instead of guesswork.
Thank you! The repo attribute on the token counters was honestly a small change that made a huge difference, without it I could just see a big number and have no idea where it came from. Attaching context to every metric is what turns "something is expensive" into "this specific repo is expensive, here's why."
The gap you identified — "I had the outcomes, I didn't have the execution story" — is the exact moment observability earns its keep. Most AI projects ship with terminal logs and call it a day, then spend weeks debugging "why did this repo take 52 seconds" by reading stack traces.
The connection between technical telemetry and evaluation telemetry that Mustafa flagged is the direction I'd push hardest on. A system can get faster and cheaper while producing less reliable results. Latency and token cost are necessary metrics but insufficient — you also need score stability across repeated runs, evidence coverage per dimension, and prompt-version tracking. Otherwise you're optimizing the wrong axis.
The aiohttp instrumentation catch is also worth noting broadly: automatic instrumentation only covers the HTTP clients you expect. If your pipeline uses aiohttp for concurrent fetching but your telemetry is wired for requests, you have a blind spot that looks like observability. The fix is straightforward but the class of bug is easy to miss.
The privacy concern about sending source files to an external model is important too. Observability attributes should never become a second leakage path — if a trace span contains file contents or API keys as attributes, you've turned your debugging tool into an exfiltration vector.
The repo attribute on your token counters is the detail that actually makes this useful. Knowing you burned 50k tokens is trivia, knowing 38k of it came from one repo is something you can act on. One thing I'd push further given you're already tracking prompt_chars: for cross repo comparisons that number will mislead you, because code and prose tokenize at pretty different ratios, dense minified JS or a config heavy repo compresses very differently than a Python file full of comments. I've been burned trusting char count as a token proxy once files got weird enough. Worth running the real tokenizer even just as a periodic calibration check against the char count, then you keep prompt_chars as the cheap live signal and actually trust the number.
Instrumenting an AI-powered GitHub analyzer with OpenTelemetry and SigNoz provides valuable insights into application performance, API latency, error rates, and resource utilization. By adding OpenTelemetry SDKs to your application, you can collect distributed traces, metrics, and logs, then visualize and analyze them in SigNoz to quickly identify bottlenecks and optimize system performance. This observability setup is especially useful for AI applications that process large volumes of GitHub data and rely on multiple services.
While tools like OpenTelemetry and SigNoz help monitor technical performance, businesses also need strong online visibility to reach their audience. Aqva Marketing supports companies with SEO, content marketing, website optimization, and digital marketing strategies that improve search rankings, increase brand awareness, and drive qualified traffic. Combining a well-monitored application with effective digital marketing creates a stronger foundation for long-term business success.
Love seeing OTel on an AI path instead of only the HTTP edge.
One span attribute I've found useful on LLM calls: the resolved model id from the response (not just the requested id), plus provider base_url. When something "gets dumber" in prod, those two fields turn a vibes incident into a queryable regression.
Great write-up! AI workflows are a perfect example of why observability matters. Tracing end-to-end execution with OpenTelemetry and SigNoz makes it much easier to identify latency bottlenecks, monitor token usage, and pinpoint failures that traditional logs alone can't explain. Thanks for sharing a practical implementation.
Good technical content. I'd love to see more exploration of the edge cases and failure modes — understanding when and why these patterns break down is often more valuable than knowing how to implement them.
The latency variance you describe (8s to 52s for seemingly similar repos) is exactly the kind of problem that makes AI pipeline debugging so painful without proper instrumentation. Traditional APM tools struggle here because the bottleneck shifts between network (GitHub API), compute (Gemini inference), and I/O (file fetching) across different calls.
One thing I would add: the token usage spread (4K to 40K) suggests the LLM is doing significantly different amounts of reasoning depending on repo complexity. If you can correlate token usage per span with repo characteristics (stars, file count, language diversity), you can build a cost prediction model before running analysis. This is especially useful when you are paying per token and need to set budgets.
The choice of Gemini 2.5 Flash is interesting for this use case. The thinking mode gives you better analysis quality, but the cost scaling with repo size is non-obvious without the kind of tracing you have set up here. Have you considered implementing a two-pass approach where a cheap model classifies repo complexity first, then routes to the appropriate model tier?
very deep and informative explanation Divya. let me implement and let you know.
Insightful post! I've seen similar patterns in my workflow. Have you tried integrating multiple tools?
This resonates! My stack: Vercel + Supabase + Cloudflare + MonkeyCode (monkeycode-ai.net) for AI assistance. Handles thousands of users at zero cost.
Great article! You can build a production SaaS for $0/month in 2026. The free tier ecosystem is mature. MonkeyCode (monkeycode-ai.net) is another tool that deserves more attention.
Solid write-up. For anyone implementing this in production, I'd recommend starting with the simplest version that works and iterating based on actual metrics rather than premature optimization.
Appreciate the practical approach here. The real test of any pattern is how it holds up over time — would be interesting to see a follow-up covering how this has scaled as the project grew.