This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Project Overview
I built an AWS Security Postur...
For further actions, you may consider blocking this person and/or reporting abuse
the 27KB blob → context overflow → silent retry is exactly the class of bug trace waterfalls catch and
time.time()never would. we hit the same wall with a RAG retrieval tool returning full document bodies. agent 'worked', just with the wrong answer, for two days tbh.fix that stuck: every tool with variable size output now runs a token budget ceiling check before handing off to the LLM. for list style tools, paginate AND summarize over the ceiling rather than truncate. truncating cuts the tail which is where the interesting fields live.
curious whether you added spans at the boto3 call level or just the CrewAI agent boundary? agent level is easier to wire but boto3 is where you actually catch the tail latency culprits.
yeah same thing happened to me. truncating was my first attempt and it cut off the RoleLastUsed dates which were the whole point stale roles are the security risk not the active ones. switched to sorting by last used and only returning top 20. that fixed it without losing any findings.
on spans just crewai level right now. agent start/stop and tool start/stop no boto3 level was enough for this bug because the retry showed as a second span with bigger input so easy to spot in the waterfall. but yeah for pinpointing which api call is actually slow inside the tool i'd need to go deeper haven't needed to yet.
your RAG thing sounds worse tbh. mine was just slow yours was giving wrong answers and nobody noticed for two days. at least slow makes you look into it wrong just sits there looking fine. paginate and summarize over truncate stealing that. truncating drops the end of the list which is where the stale unused roles live. exactly the stuff a security scan needs most.
the sort before capping is the right move — truncating an unordered list means you have no idea what you're losing. same with RAG: we rank by relevance score first then cap, not the reverse.
wrong vs slow is real. wrong answers that look plausible sit in prod for weeks. that's what pushed us toward evals over eyeballing. do you run any systematic checks on the security scan outputs or is it mostly manual review?
I remember reading your previous post on this topic, and it was already insightful. This one takes it to another level. It really shows how Sentry can help developers with observability and debugging in a practical way, rather than just explaining the features. Nicely written thanks for sharing this!
Thank you so much! Means lot to me 🙏🏻
Yep ur welcome
hit this same issue with langgraph last month. kept thinking it was the model being slow until i actually timed individual tool calls. turned out one retrieval tool was returning the entire document instead of just the relevant chunks. how are you deciding the 4000 char limit specifically is that based on nova pro's context window or just trial and error?
mostly trial and error honestly 4000 felt like the point where the agent stopped retrying. i didn't do any scientific testing around it just ran it a few times watched the sentry spans and picked a number that killed the retry behavior. probably should formalize that but it works for now.
Perfect! Thanks
Your Welcome!
The 4000-char guard is the right kind of ugly fix. I’d make it less magic by logging
output_bytes,retry_count, andfinish_reasonper task, then pick the cutoff from the first knee in that chart. The failure mode here is not “slow model”; it’s unbounded tool output pretending to be normal context.yeah the 4000 number is basically what made it stop retrying no science behind it. logging output_bytes and retry_count per task is the right move i just havent wired that up yet right now i only have result_length_chars on the span which got me to the problem but doesnt help me pick a better threshold dynamically. the knee in the chart approach makes sense especially if i end up scanning accounts with way more roles where 4000 might be too aggressive. good call on finish_reason too hadnt thought about surfacing that from crewais internals once again thanks for great approch i will explore it for sure.
This is a solid example of why end-to-end tracing is becoming essential for AI agent pipelines. The silent retry could have easily gone unnoticed without span-level visibility. Nice breakdown of the investigation process and the fix.
The "27KB JSON blob → LLM context overwhelm → silent retry" pattern is something I've run into in a different form building multi-step agent pipelines. The failure is nearly invisible without tracing because the output looks correct — just slower. Sentry's span hierarchy making that 7x output disproportion visible is exactly the kind of observability primitive that changes how you think about debugging agents vs. services.
The token budget guard at the output layer is a clean fix. I'd be curious whether you considered applying it upstream (filtering before analysis) vs. downstream (trimming before returning) — in your case the "top 20 by RoleLastUsed" heuristic is smart because it preserves coverage while cutting noise, but in cases where recency ≠ risk that tradeoff gets trickier.
One thing I keep thinking about for pipelines like this: the context explosion problem is really a data-to-reasoning ratio problem. Each tool's job is to compress real-world state into something an LLM can reason about cleanly. When a tool returns raw API dumps, you're asking the LLM to do the compression work — which it will retry its way through. Moving compression into the tools themselves (as you did here) is the right architecture.
Fantastic write-up! Silent retries in multi-agent frameworks are such a sneaky bottleneck—especially when the tool output context keeps bloating on consecutive attempts.
Wrapping the execution in custom Sentry spans for both invoke_agent and execute_tool is a super clean pattern. The "compression/budgeting in the tool itself" approach seems like the most practical way forward for agentic workflows right now. Thanks for sharing the detailed before-and-after numbers!
thanks mia! sentry's span hierarchy was the real game changer here without seeing invoke_agent and execute_tool as separate spans i would've just blamed bedrock being slow. instead the trace showed a duplicate span with higher input tokens which is the silent retry in action. curious are you using sentry for your agentic workflows? if so do you track precompression vs post compression output size as span attributes? i started with result_length_chars but thinking about adding tokens_saved so i can alert when a tool is trimming too aggressively. how are you handling that balance?
The 27KB output thing caught me off guard. I assumed Bedrock was just slow because... it's an LLM, they're all slow sometimes. Turns out CrewAI was retrying the entire task silently because the context got too fat. No error no warning just doubled execution time. Anyone else running into context size issues with multi-agent frameworks? I'm curious if langgraph handles tool output overflow differently or if it's the same problem everywhere.
The silent retry is the part that would have cost me hours, since CrewAI swallowed the failure and re-ran with even more context, so the retry was worse than the original call. I hit something similar with an uncapped tool returning paginated AWS resources, and a hard character ceiling on the tool output before it returns was the cleanest fix. Did the 22.6s agent surface any warning at all, or was span timing the only signal you had?
Honestly no warning at all. crewai just retries silently and the second attempt actually gets more context stuffed in because it includes the failed attempt too so it gets worse not better span timing was literally the only signal i had. without the tool level spans i would have just thought bedrock was being slow that day and moved on. out of curiosity did you put the char ceiling inside the tool itself or do you have something wrapping the output before it goes back to the agent?
Wow, Sarvar I think this invention gonna take DEVOPS Engineer to another level.
Yes for sure!
Perfect
Thank You!