DEV Community

EU AI Act Timeline: What AI Vendors and Developers Must Track Through 2026

Ali Farhat — Wed, 05 Aug 2026 04:00:30 +0000

The EU AI Act is not a single regulatory switch that turned on in 2025. Its requirements are being applied in stages, creating distinct deadlines for AI vendors, model providers, developers and organizations deploying AI systems in Europe. The first rules took effect on 2 February 2025, while a second major milestone on 2 August 2025 brought governance provisions and obligations for providers of general-purpose AI models into application.

That phased approach matters for companies building or using AI tooling. Consumer assistants, industrial applications and internal software may rely on general-purpose AI models, but the compliance timeline depends on the role an organization plays and the type of system involved. The next major date is 2 August 2026, when the AI Act's enforcement powers and most high-risk AI requirements are scheduled to become fully applicable.

The EU AI Act rollout is phased, not blanket

The European Union has structured the AI Act around a risk-based framework and a staged implementation schedule. According to the EU AI Act implementation timeline published by the AI Act Service Desk, 2 February 2025 marked the application of the Act's general provisions, including definitions and AI literacy, alongside prohibitions on unacceptable-risk AI uses.

Six months later, on 2 August 2025, the governance framework began applying. Crucially for the current AI market, this phase also brought obligations for providers of general-purpose AI, commonly abbreviated as GPAI, models into application. These are models with broad capabilities that can support many different downstream applications.

Implementation milestone	Date	Verified scope
First rules apply	2 February 2025	General provisions, including definitions and AI literacy, plus prohibitions for unacceptable-risk uses
GPAI and governance phase	2 August 2025	Governance framework and obligations for providers of general-purpose AI models
Major enforcement milestone	2 August 2026	Enforcement powers and the majority of high-risk AI requirements are scheduled to become fully applicable
Later high-risk timelines	2027 to 2028 timeframes	Further phased requirements for Annex III high-risk systems and products embedded with AI

The distinction between these dates is essential. A company cannot reasonably reduce its AI Act planning to a single claim that the regulation either is, or is not, in force. Parts of the framework already apply, while other provisions are scheduled for later implementation.

Why the GPAI milestone changes the compliance conversation

The August 2025 milestone puts particular attention on the organizations providing general-purpose AI models. This is relevant beyond the model developers themselves because many AI products and workflows are built on, or otherwise leverage, such models. For software teams, the regulatory question is therefore not limited to whether they train a foundation model. It also concerns how their product relates to the model provider, how it is deployed, and whether its intended use moves it into a more tightly regulated category.

The verified timeline does not make every AI application high risk, nor does it establish identical obligations for every business that uses AI. Instead, it shows why teams need a clear view of their position in the AI supply chain and of the systems they are placing into use.

For organizations operating in Europe, the practical governance priorities supported by the rollout include:

AI literacy, which entered into application with the first wave of rules in February 2025.
Use-case assessment, particularly to identify prohibited unacceptable-risk uses and systems that may be subject to later high-risk requirements.
Provider and model mapping, since GPAI provider obligations began applying in August 2025.
Forward planning for 2026 and beyond, rather than treating the 2025 milestones as the end of implementation.

What vendors and developers should prepare for next

The 2 August 2026 date is the central near-term milestone for many organizations. The European Commission's phased schedule places the AI Act's enforcement powers and the majority of high-risk requirements at that point. Further timing remains relevant for Annex III high-risk systems and products that include AI, with later phases cited for 2027 and 2028.

That schedule makes governance an operational issue rather than a policy exercise reserved for legal teams. Product leaders need to understand the intended use of a system. Engineering teams need clarity on which models and components are involved. Procurement and deployment decisions need to account for the provider relationships behind AI capabilities. These are practical questions that become more important as the framework expands.

Companies should also avoid two unhelpful assumptions. The first is that the arrival of GPAI obligations means every tool using a general-purpose model faces the same requirements. The second is that later high-risk milestones mean current obligations can be ignored. The verified timeline supports neither conclusion. The Act is already applying in defined areas, while additional requirements are still approaching.

Organizations assessing AI systems, model dependencies and internal governance can work with Scalevise on AI architecture, workflow automation and implementation planning that connects technical delivery with operational controls.

Frequently Asked Questions

When did the first EU AI Act rules start applying?

The first wave started on 2 February 2025. It applied general provisions, including definitions and AI literacy, as well as prohibitions for unacceptable-risk AI uses.

When did obligations for general-purpose AI model providers begin?

Obligations for providers of general-purpose AI models began applying on 2 August 2025, alongside the AI Act's governance framework.

Does the EU AI Act fully apply to every AI system already?

No. The Act is rolling out in stages. Enforcement powers and most high-risk AI requirements are scheduled to become fully applicable on 2 August 2026, with further high-risk timelines extending into 2027 and 2028.

Why should companies using third-party AI models track the GPAI rules?

Many AI tools rely on or leverage general-purpose AI models. Companies need to understand their role in the AI supply chain, their system's intended use and the implementation dates that may apply to them.

Conclusion

The EU AI Act's 2025 milestones established that AI governance in Europe is already a live operational concern, especially for unacceptable-risk uses, AI literacy and general-purpose AI model providers. For vendors and developers, the key task is to treat the regulation as a staged program: address the rules already in application, map AI dependencies and prepare for the major high-risk and enforcement milestone scheduled for August 2026.

Finding Exposed Services (and Vulnerabilities) on Your Network: A Practical ScanSearch Guide

Billy — Wed, 05 Aug 2026 04:00:13 +0000

Ever wondered what services on your network are truly exposed to the internet? Or perhaps you're trying to track down a specific type of device across your public-facing infrastructure? Manually scanning port by port across a large IP range can be tedious, slow, and often misses the bigger picture.

This article isn't about setting up a traditional port scanner. Instead, we're going to explore how to leverage an internet-wide search engine for network devices and services, ScanSearch, to quickly identify and understand the public-facing footprint of your network. Think of it like Google, but for servers, routers, cameras, and everything else connected to the internet.

We'll cover practical use cases, including:

Identifying unintentionally exposed services.
Searching for specific device types or software versions.
Spotting known vulnerabilities associated with your infrastructure.

Let's dive in.

The Problem: Blind Spots in Your Public-Facing Infrastructure

It's common for development or operations teams to configure services, sometimes forgetting that a default configuration or an overlooked firewall rule might leave something accessible that shouldn't be. This isn't just about malicious actors; it's also about maintaining good security hygiene and understanding your own attack surface.

Traditional internal network scans are crucial, but they don't always give you the external perspective. That's where a tool like ScanSearch comes in – it's constantly indexing the internet's public-facing devices.

Getting Started with ScanSearch

ScanSearch offers a powerful search syntax. The most straightforward way to begin is by searching for your organization's public IP ranges or domain names. For this tutorial, I'll use example.com and a hypothetical IP range, but you should substitute these with your own public IPs or domains.

Use Case 1: Discovering All Services on Your IP Range

Let's say your organization uses the IP range 203.0.113.0/24. To see everything ScanSearch has indexed for this range, you'd simply enter:

ip:203.0.113.0/24

This query will return a list of all devices, services, and associated information (like open ports, banners, and even HTTP response headers) found within that IP range. You might be surprised at what pops up – maybe an old development server you thought was offline, or a service running on a non-standard port.

Use Case 2: Finding Specific Device Types or Software Versions

Perhaps you're concerned about older versions of Apache or Nginx that might still be running. You can combine the IP range with a keyword search for specific banners or technologies. For example, to find all Apache servers on our hypothetical range:

ip:203.0.113.0/24 product:apache

Or, to get more specific and look for an older, potentially vulnerable version (e.g., Apache 2.2):

ip:203.0.113.0/24 product:apache/2.2

This quickly highlights potential upgrade candidates or misconfigurations.

Use Case 3: Identifying Known Vulnerabilities on Your Infrastructure

ScanSearch also indexes publicly known vulnerabilities (CVEs) associated with identified services. This is incredibly powerful for proactively assessing risk.

Let's say you want to see if any of your devices on example.com are running services with known vulnerabilities:

host:example.com has_vulnerability:true

This query would return any hosts under example.com where ScanSearch has identified a service with an associated, known vulnerability. You can then drill down into the results to see the specific CVEs and affected services. This is a critical step in prioritizing patching and mitigation efforts.

Pro-tip: You can also search directly for a specific CVE. If you're tracking a new, critical vulnerability, you could search cve:CVE-2023-XXXX to see if it appears anywhere on your network.

Use Case 4: Searching for Default Credentials or Common Misconfigurations

While ScanSearch doesn't actively exploit systems, its indexing of service banners and HTTP responses can sometimes reveal clues about weak configurations. For instance, you might look for common default administrative interfaces or specific keywords that indicate a lack of proper setup.

ip:203.0.113.0/24 title:"admin panel"

Or, looking for specific HTTP response bodies:

ip:203.0.113.0/24 http.body:"Powered by WordPress"

While not a direct vulnerability, finding an exposed WordPress admin panel on a public IP range you own is certainly something you'd want to investigate further.

Beyond Simple Queries: Combining Filters

The real power of ScanSearch comes from combining these filters. You can search for specific ports, protocols, HTTP headers, and much more. For example, to find all devices on your range running an SSH server (port 22) that also have a known vulnerability:

ip:203.0.113.0/24 port:22 has_vulnerability:true

This quickly narrows down your focus to the most critical issues.

Conclusion

Understanding your external attack surface is a fundamental part of maintaining secure infrastructure. Tools like ScanSearch provide an invaluable perspective by indexing the entire internet, allowing you to quickly query and identify exposed services, potential misconfigurations, and known vulnerabilities associated with your own public-facing assets.

Instead of endless manual scans, integrate ScanSearch into your regular security audits. It's a powerful way to catch those overlooked services and proactively address potential risks before they become a problem. Give it a try with your own public IP ranges and domains – you might be surprised at what you find.

Designing a Reliable PDF Translation Job Pipeline in TypeScript

Noah Chen — Wed, 05 Aug 2026 03:57:55 +0000

Uploading a PDF and calling a translation model looks like a two-step feature. In production, it is a job pipeline with untrusted input, two different extraction paths, several expensive stages, and an output that can be fluent while still being wrong.

That distinction matters for a small SaaS team. The translation request may come from support, sales, or an internal operations task. Nobody wants to operate a document platform, but the workflow still needs to answer basic questions:

Was the upload actually a PDF?
Does the file contain selectable text or scanned page images?
Can a retry create a second charge or a conflicting result?
What happens when page 37 fails after the first 36 pages succeed?
How do we know the translated PDF is not blank or visually broken?
When are the source and result deleted?

The translation model is one component. Reliability comes from the system around it.

Define the Job Contract First

I would not let a file reach an extractor until the API has established a narrow contract.

For example, a translation request might include:

type TranslationStyle = "general" | "technical" | "academic";

interface CreateTranslationJob {
  uploadId: string;
  sourceLanguage: string | "auto";
  targetLanguage: string;
  style: TranslationStyle;
  idempotencyKey: string;
  containsRestrictedData: boolean;
}

The request should be rejected when the source and target languages are identical, the upload is missing, the target language is unsupported, or policy says the document cannot leave an approved environment.

File validation should also be explicit. Do not trust the filename or browser-supplied MIME type. Check at least:

the actual byte size;
the file signature;
whether the parser can open the document;
whether the PDF is encrypted;
the page count;
whether the job fits the account or product limit.

A 20 MB limit is simple to explain in a user interface, but size alone is not a good predictor of work. A compressed 200-page text PDF can be smaller than a six-page scan. Page count, image area, and extracted character count are better inputs for estimating processing time.

Treat Preflight as Its Own Stage

The first useful result is not a translation. It is a document profile.

interface DocumentProfile {
  pageCount: number;
  encrypted: boolean;
  extractedCharacters: number;
  pagesWithText: number;
  pagesWithLargeImages: number;
  likelyScanned: boolean;
  estimatedOcrPages: number;
}

This profile controls routing. A PDF where nearly every page has substantial selectable text can go directly to extraction. A document made of page-sized images needs OCR. Mixed documents need a page-level decision rather than a single flag for the entire file.

A crude scan heuristic could look like this:

function needsOcr(profile: DocumentProfile): boolean {
  if (profile.pageCount === 0) return false;

  const textCoverage = profile.pagesWithText / profile.pageCount;
  const imageCoverage = profile.pagesWithLargeImages / profile.pageCount;

  return textCoverage < 0.25 && imageCoverage > 0.6;
}

The thresholds are product decisions, not universal constants. A form can contain a small amount of real text over a scanned background. A research paper may include image-heavy appendix pages. Store the measurements that led to the route so a failed job can be explained later.

Model the Pipeline as States, Not Progress Percentages

A single processing: true field hides too much.

I would rather expose a finite set of states:

type JobState =
  | "queued"
  | "validating"
  | "extracting"
  | "ocr"
  | "translating"
  | "rendering"
  | "verifying"
  | "ready"
  | "failed"
  | "expired";

Each state should have a clear owner and output:

State	Responsibility	Durable output
`validating`	inspect the upload and policy	document profile
`extracting`	recover text and geometry	page blocks
`ocr`	recognize image-only pages	OCR blocks and confidence
`translating`	translate normalized blocks	translated segments
`rendering`	rebuild the destination PDF	candidate output
`verifying`	run structural checks	verification report
`ready`	expose a time-limited download	signed result reference

The state machine makes several bugs harder to create. A rendering worker cannot start before translation output exists. An expired job cannot silently return to ready. A retry can resume from the last durable stage instead of repeating the entire workflow.

Transitions should be enforced rather than implied:

const allowed: Record<JobState, JobState[]> = {
  queued: ["validating", "failed"],
  validating: ["extracting", "ocr", "failed"],
  extracting: ["ocr", "translating", "failed"],
  ocr: ["translating", "failed"],
  translating: ["rendering", "failed"],
  rendering: ["verifying", "failed"],
  verifying: ["ready", "failed"],
  ready: ["expired"],
  failed: [],
  expired: [],
};

function canTransition(from: JobState, to: JobState): boolean {
  return allowed[from].includes(to);
}

In a real system I would also record attempt, updatedAt, failureCode, and the worker version that produced each stage. That is enough to distinguish a bad document from a deployment regression.

Put Retry Boundaries Around Durable Work

Retries are necessary, but “retry the job” is too broad.

Uploading, OCR, translation, and rendering have different failure modes. A network timeout while writing the final PDF should not trigger OCR again. A transient translation-provider error should not require another upload. A user pressing the submit button twice should not create two independent jobs.

The idempotency key belongs at job creation. Stage-specific keys can be derived from the job and input version:

function stageKey(
  jobId: string,
  stage: JobState,
  inputVersion: number,
): string {
  return `${jobId}:${stage}:${inputVersion}`;
}

Workers should write their result before advancing the state. If the process stops between those operations, the next worker can see the existing stage output and continue safely.

Retries also need limits. OCR on a malformed image is unlikely to improve on attempt 12. Use stable failure codes such as PDF_ENCRYPTED, OCR_LOW_CONFIDENCE, TRANSLATION_TIMEOUT, and RENDER_OVERFLOW. They are more useful to support than a stack trace alone.

Translation Units Need Stable Identity

Sending one entire document as a string loses layout relationships. Sending every line independently loses context.

A practical middle ground is a collection of blocks with stable IDs:

interface TextBlock {
  id: string;
  page: number;
  role: "heading" | "paragraph" | "caption" | "table-cell" | "footer";
  sourceText: string;
  translatedText?: string;
  bounds: { x: number; y: number; width: number; height: number };
  ocrConfidence?: number;
}

Stable IDs make it possible to retry one segment, preserve repeated headers, compare source and translation, and point a human reviewer to a specific page and region.

Context can be supplied by grouping adjacent blocks or attaching a document glossary. The glossary should protect product names, UI labels, units, and phrases that must remain unchanged. It is also the right place to keep reviewer-approved terminology between versions.

Verification Is More Than “The File Opens”

A translated PDF can be syntactically valid and unusable.

Automated verification should look for structural signals:

output page count differs unexpectedly from the source;
pages contain no text or images;
translated blocks are missing;
text extends outside its assigned bounds;
a table has a different number of rows or cells;
a required font lacks glyphs for the target language;
hyperlinks disappeared;
OCR confidence is below a review threshold;
numbers present in the source are absent from the translated block.

These checks do not prove semantic accuracy. They decide whether the result is safe to show as an ordinary completion or should be flagged for review.

For customer-facing, legal, medical, financial, or safety-related documents, a human review is not an optional fallback. The workflow should make that requirement visible instead of letting a fluent result imply certification.

Retention Is Part of the Data Model

Document pipelines tend to become accidental archives.

The source upload, extracted text, OCR images, model inputs, intermediate render, and final file may all exist in different systems. Deleting only the public download does not remove the job data.

Every artifact should carry a retention class and expiry:

interface StoredArtifact {
  key: string;
  kind: "source" | "extracted" | "ocr" | "translated" | "result";
  deleteAfter: string;
  encrypted: boolean;
}

Use short-lived signed URLs for downloads. Run deletion as an observable job. Record completion without keeping the deleted content. If business users need a permanent copy, make them move it into the approved system of record rather than silently extending temporary storage.

Make Operations Visible

A progress bar is useful to the person waiting, but it is not enough for the team operating the workflow.

I would track metrics by stage and route:

validation failures by reason;
percentage of pages sent to OCR;
median and high-percentile duration for extraction, OCR, translation, and rendering;
retries and terminal failures by worker version;
output files flagged for human review;
deletion jobs completed after their deadline;
jobs abandoned before download.

These measurements answer different questions. A sudden increase in OCR pages may indicate a change in customer inputs. Longer rendering time after a deployment points toward layout code, not the translation provider. A high completion rate with a low download rate may mean results expire too quickly or the notification step is unreliable.

Logs should carry the job ID, stage, attempt, and block or page range, but not extracted document text by default. Document contents make debugging tempting and data handling much harder. Store diagnostic metadata first; collect a redacted sample only through an explicit support path.

Build, Run Locally, or Use a Hosted Workflow?

Not every team should build this pipeline.

Building makes sense when translation is part of the product, jobs repeat at scale, restricted documents are involved, or the team needs precise control over models, audit logs, and retention. Local tools are attractive when a technical operator can process sensitive files without uploading them.

For an occasional public or synthetic document, a hosted interface can remove a lot of setup. PDFTranslator is one example: it supports OCR, more than 100 languages, and translation styles, with a 20 MB upload limit and a stated free allowance of 1,000 pages per calendar month. It is still a cloud workflow. Its homepage says task files are deleted within 24 hours, which is useful operational information but not a reason to upload restricted material without an approved policy.

That tradeoff is the important part. Convenience changes who operates the pipeline; it does not remove the need to classify the document or review the result.

A Production-Readiness Checklist

Before calling a PDF translation workflow reliable, I would want clear answers to these questions:

Do we validate file contents rather than extensions?
Can we route text, scanned, and mixed PDFs differently?
Are job states and legal transitions explicit?
Are retries idempotent and limited to the failed stage?
Can we trace translated blocks back to source regions?
Do we verify page structure, overflow, fonts, tables, links, and numbers?
Can the system require human review instead of merely suggesting it?
Does every stored artifact have a deletion rule?
Can support explain a failure without reading worker logs?
Have we decided which documents are allowed to leave the device?

The model call may be the most visible part of translation, but it is not the system. The system is the set of boundaries that keeps an untrusted document, a long-running process, and a persuasive-looking output from creating a quiet operational failure.

Building an Editable 3D Indoor Map in the Browser

zyl19950114 — Wed, 05 Aug 2026 03:56:10 +0000

Indoor maps are often treated as a rendering problem: take a floor plan, extrude a few walls, and display the result. That is useful for a viewer, but it breaks down when a team needs to edit a real space, place assets, or hand the result to another application.

We are building KiMap around a different boundary: turn a floor plan into an editable indoor scene in the browser, then keep the resulting structure useful for an SDK consumer.

Why a floor plan is not enough

A production indoor workflow needs more than a textured image on a plane. At minimum, the editor has to preserve the relationships between walls, floors, rooms, openings, and the objects placed in the space. Those relationships determine whether the result can later support navigation, facility workflows, a digital twin, or a custom web experience.

That is why the current KiMap workflow starts with structure. You can define the indoor geometry, inspect it in 2D and 3D, and keep editing instead of committing to a static export too early.

The browser editor boundary

The editor is built with React and Three.js. The goal is not to replace every DCC tool. It is to make the early spatial workflow accessible to teams that need to test an indoor experience before investing in a full custom pipeline.

The parts we are concentrating on are:

editable floor-plan structure and bounded spaces
2D and 3D scene inspection in the same workflow
reusable 3D furniture and local asset handling
saving an indoor project without dropping the referenced model data
a path toward SDK-oriented rendering and integration

The last point matters. A scene that looks correct in an editor is not automatically useful to an application. We want the data boundary to be explicit enough that an SDK consumer can load the geometry and assets without rebuilding the scene from scratch.

What we are testing next

KiMap is in free early access. The most useful feedback is not generic interest; it is a concrete blocker from someone building an indoor-navigation, GIS, WebGL, facility, or digital-twin workflow.

Try the live example without creating an account:

Open the KiMap example

After trying it, I would especially value answers to three questions:

Which indoor data model or export format would your stack require?
Where would this editing workflow conflict with your real building or facility data?
What API or model-loading behavior would you need before integrating it?

Early access is free and no credit card is required. There is a feedback action inside the example, or you can reach the team at kimap.founder@gmail.com.

Seedance 2.5 is priced 53% above 2.0 per token, and its 480p frame shrank

Lee — Wed, 05 Aug 2026 03:55:51 +0000

Seedance 2.5's API opens on August 7. ByteDance published the pricing ahead of it, and there is a detail in there that will quietly break your cost model if you carry it over from 2.0.

Video is quoted per second and metered per token:

tokens = (input_video_seconds + output_seconds) × width × height × fps / 1024

fps is fixed at 24. Multiply by the per-million-token rate and that is the bill.

The published rates

USD per million tokens:

Model	No video input	With video input
Seedance 2.5 (480p, 720p)	10.70	6.40
Seedance 2.0 (480p, 720p)	7.00	4.30
Seedance 2.0 (1080p)	7.70	4.70
Seedance 2.0 (4K)	4.00	2.40

2.5 costs 52.9% more per token without video input and 48.8% more with it. Only 480p and 720p are published for 2.5. No 1080p, no 4K, and offline inference reads "not supported yet".

Look at the 4K row before you move on. It is the cheapest tier per token, 43% below 480p, and it is also the most expensive output on the board, because a 3840×2160 frame carries 19.4 times the pixels of what 480p actually renders. The rate drops 43% while the token count climbs 1940%. Comparing providers by scanning the rate column gets you the wrong answer by roughly a factor of eleven.

The 480p frame changed and nobody said so

This is not in any release note. It falls out of dividing ByteDance's own worked examples by their own token rates.

Their published five-second, 16:9, no-reference examples:

Model	480p	720p
Seedance 2.5	$0.514 ($0.103/s)	$1.156 ($0.231/s)
Seedance 2.0	$0.352 ($0.070/s)	$0.756 ($0.151/s)

Divide price by token rate to recover the token count, then by 24/1024 to recover pixels:

const tokens = pricePerVideo / (ratePerMillion / 1e6);
const pixels = (tokens / outputSeconds) * (1024 / 24);

// Seedance 2.5, 480p:  0.514 / (10.70/1e6) / 5  =  9,607 tokens/sec
//                      9,607 * 1024/24          =  409,899 px  ->  ~854 x 480
// Seedance 2.0, 480p:  0.352 / (7.00/1e6)  / 5  = 10,057 tokens/sec
//                      10,057 * 1024/24         =  429,105 px  ->  ~873 x 491

720p resolves to 21,600 tokens per second on both versions, which is exactly 1280 × 720. That the same arithmetic lands on a clean, verifiable number at 720p is what makes the 480p result trustworthy instead of a rounding artifact.

So 2.5 renders 480p at a true 16:9 854 × 480 while 2.0 uses a slightly taller frame. That 4.5% pixel reduction explains a discrepancy that otherwise looks like a pricing error: 480p rises 46% per second while the token rate rises 53%. At 720p, where the frame is unchanged, the two match exactly.

If you quote customers per second, your 2.0 conversion factor is wrong on 2.5 by about five percent at 480p.

Your input clip bills like generated video

input_video_seconds sits inside the same parenthesis as the output. A reference-to-video job pays for the source clip at the same rate as the frames the model invented.

Providers surface this as a discounted per-second rate for reference mode, which reads like a deal until you total it. Seedance 2.5's published range makes the point on its own: a five-second 720p generation costs $1.244 with a short reference and $4.838 with a 30-second one. Same output, 3.9× the bill.

2.5 also doubled the input window, from 15 seconds on 2.0 to 30.

There is a floor too, and the number is unpublished. Their examples price two-second and four-second inputs identically, which implies a minimum around four seconds. The docs point at a spreadsheet calculator rather than stating it. If you resell this, customers sending two-second clips cost you more than your arithmetic predicts.

Converting any token rate to per-second

const TOKENS_PER_SEC = {
  '480p@2.5': 854 * 480 * 24 / 1024,   //   9,607
  '480p@2.0': 864 * 496 * 24 / 1024,   //  10,044
  '720p':    1280 * 720 * 24 / 1024,   //  21,600
  '1080p':  1920 * 1080 * 24 / 1024,   //  48,600
  '4k':     3840 * 2160 * 24 / 1024,   // 194,400
};

function costUsd({ tier, ratePerMillion, inputSec = 0, outputSec }) {
  const tokens = (inputSec + outputSec) * TOKENS_PER_SEC[tier];
  return (tokens * ratePerMillion) / 1e6;
}

// Sanity check against the vendor's own example:
costUsd({ tier: '720p', ratePerMillion: 7.0, outputSec: 5 }); // 0.756

That last line matches ByteDance's published figure exactly, so the formula is not an approximation.

One caveat for anyone metering downstream: token counts are estimates until the job finishes. The formula predicted 40,176 for a config where the API returned 40,594, about 1% high. Bill on the returned usage.completion_tokens, not on your estimate.

What I could not determine

Why 1080p and 4K have no published rate for 2.5. Either those tiers do not ship at launch or they arrive separately.

The exact minimum input duration. It exists and it is not a published number.

Whether the same formula holds outside the Seedance family. Pixels × duration × fps is a plausible general shape, but the constants and the input-billing rule are not something I would assume for Veo, Kling or Sora without checking. If you have run the same back-solve against those, I would genuinely like to know how it came out.

Full writeup with the Seedance 2.0 rate card in per-second and per-minute terms: reapi.ai/blog/seedance-2-5-pricing-per-token

Live parameter reference and a browser playground: reapi.ai/models/seedance-2-0

Best Project Management Software for Startups: Match the Tool to How You Work

Owais Khan — Wed, 05 Aug 2026 03:55:02 +0000

Search "best project management software for startups" and you get the same
dozen names every time: Trello, Asana, ClickUp, Notion, Linear, monday.com,
Basecamp. Ranking them by feature count tells you almost nothing, because they
are not really competing for the same job. The useful question for a startup is
not which tool has the most features. It is two narrower ones: does your work
run through engineering or through the whole company, and does per-seat pricing
or flat-rate pricing fit a headcount that is about to change? Answer those and
the shortlist collapses to two or three.

The split that actually decides it

Two forks matter more than any side-by-side feature grid. The first is who the
tool is built for. Issue trackers like Linear are built around the engineering
workflow (issues, cycles, a keyboard-first interface) and feel wrong the moment
a marketer or a founder tries to run a launch plan in them. General work tools
like Asana, ClickUp, monday.com and Trello are built for any team, which makes
them flexible but also less opinionated about how software actually ships.

The second fork is the shape of the bill. Almost everything in this category
charges per seat per month, so the cost scales directly with hiring. A small
number, Basecamp most notably, offer a flat rate that does not. For a company
planning to double headcount inside a year, that difference can outweigh any
feature comparison.

If your team is mostly engineers

For an engineering-led startup, an issue tracker usually beats a general project
tool. Linear's free plan includes unlimited members, two teams and up to 250
issues, which is enough to run a small product team before paying anything; its
Basic plan is $10 per user per month billed yearly and lifts the cap to
unlimited issues and five teams. The trade-off is scope: Linear is deliberately
narrow, so non-engineering work does not fit it well.

The larger, more familiar alternative is Jira, which startup roundups still name
as the default for software teams despite a steeper learning curve. If most of
your company is not writing code, both are the wrong starting point. You will
spend your first months fighting the tool's assumptions rather than using them.

The cheapest capable all-rounders

If you want one board-based tool the whole team can adopt, Trello and ClickUp
anchor the low end. Trello's free plan gives you unlimited cards but caps a
Workspace at ten boards and ten collaborators, with 250 automation command runs
a month. Standard is $5 per user per month billed annually and removes the board
cap, while Premium at $10 adds Calendar, Timeline, Table and Dashboard views.
Trello is genuinely easy to adopt, but its weakness is reporting: teams
routinely fall back to spreadsheets for anything above a single board.

ClickUp aims at the opposite problem by trying to do everything. Its Free
Forever plan is unusual in allowing unlimited members (constrained mainly by
60MB of storage), and its first paid tier, Unlimited, is $7 per user per month
billed yearly. That combination of a free tier that does not punish you for
adding people and a cheap first upgrade is why ClickUp appears on nearly every
budget list. The cost is complexity; the same roundups that recommend it warn
about a learning curve and a busy interface.

If your projects already live in your docs

Notion is a documents-and-databases workspace that many startups use for project
tracking rather than a dedicated PM tool. That works well when your projects are
inseparable from your notes, specs and internal wiki. Watch the free plan's
edges: file uploads are capped at 5MB, page history is kept for seven days, and
block limits kick in once you have more than one member. The Plus plan at $10
per member per month lifts those limits.

Notion is a poor fit if you need Gantt-style timelines or heavy automation out
of the box. It is an excellent fit if your team already lives in Notion docs and
simply wants tasks sitting next to them, rather than in a separate app nobody
opens.

If you want structure without inventing it

Asana and monday.com occupy the middle ground: more structure than Trello, less
specialised than Linear. Asana's free Personal plan now caps collaboration at
two users, and its first real team tier, Starter, is $10.99 per user per month
billed annually. That is the tier that unlocks Timeline and Gantt views, forms
and custom fields, so most teams will land there rather than on free.

monday.com is similar in spirit, with two startup-relevant catches worth knowing
before you fall in love with the demo: paid plans start at a three-seat minimum,
and its free tier is capped at two seats and three boards, which makes the free
plan closer to a trial than a home. Its Standard plan is $12 per seat per month
billed annually, and that is where Timeline, Gantt and guest access appear.

If a predictable bill matters more than features

Basecamp is the outlier worth knowing about because of its pricing model, not
its feature list. Its Pro Unlimited plan is a flat $299 per month billed
annually for unlimited users, rather than a per-seat charge. For a team that is
about to keep hiring, a flat rate can be cheaper and far more predictable than
any per-seat tool; the roundups make the same point, that a flat rate starts to
undercut per-seat pricing once headcount climbs. The trade-off is deliberate
simplicity: no Gantt charts, limited automation and little customisation. You
are buying a calm interface and a predictable invoice, not depth.

What to check before you commit

Pricing and limits in this category change often, so treat every figure here,
this article included, as something to confirm on the vendor's own pricing page.
Specifically:

Price the tier that has the feature you need, not the entry tier. Timeline and Gantt views, custom fields and real automation sit one tier up on most of these tools.
Count non-working seats. Founders, advisors and contractors who need only read access move a per-seat total more than the headline number suggests; check whether guests or viewers are free.
Watch the free-tier cliffs. Notion's block limit for teams, Trello's ten-board cap and Asana's two-user free plan all push you to upgrade sooner than the marketing implies.
Test an export during the trial. The tools that are easiest to leave are the ones you can evaluate honestly in year two, before you have a year of history locked inside them.

A reasonable default

For a general, non-engineering startup team that wants to spend nothing to start
and little later, ClickUp's unlimited-member free plan and $7 first tier make it
the lowest-risk place to begin, with Trello a simpler alternative if you only
ever need boards. For an engineering-led team, start with Linear and add a
lightweight tool for the rest of the company only when someone actually asks for
one. And if you are about to hire quickly and value a predictable bill over
feature depth, price Basecamp's flat rate against the per-seat tools at your
expected twelve-month headcount before committing. The ranking often flips as
you grow, which is exactly the kind of thing that is cheap to check now and
expensive to discover later.

Test smarter with Snagly: 30 open-source QA skills for AI coding agents

Ambreen Khan — Wed, 05 Aug 2026 03:54:05 +0000

If you've experimented with AI-driven testing, you've probably lived this cycle: you ask an AI agent to "test the checkout flow," and it does something — clicks around, declares success, and leaves you unsure what was actually verified. The next day you ask again and it does something different. The browser automation works; the testing discipline is missing.

That gap is what Snagly is for.

Rather than describe it, I pointed it at softwaretestingtrends.com — my own production site, nothing fixed beforehand — and recorded the whole thing. It found eleven issues, including a critical accessibility bug on my own signup page. One of its findings turned out to be wrong, and I'll come back to that, because it matters more than the ones it got right.

📺 Watch the full walkthrough — installed from an empty folder, run against production, ~20 minutes.

What it is

Snagly is a free, MIT-licensed set of 30 skills for AI coding agents — GitHub Copilot, Claude Code, Cursor, Codex and 70+ others — that turn "an AI that can drive a browser" into "an AI that tests like a QA professional." A skill, if you haven't met them yet, is a reusable instruction set that teaches the agent a specific working method — when to use it, what rigor it requires, what evidence to capture, and what it must never do.

Each skill in Snagly has one job, and they hand off to each other the way a real testing practice does:

start-testing is the front door — say "what can you test here?" and it routes you to the right skill, checking prerequisites before handing off.
Discovery & strategy: scenario-mapper explores your site and produces a prioritized list of test scenarios; test-case-writer expands any of them into a reviewable spec; test-plan sets strategy, cadence, and release exit criteria; qa-onboarding writes the guide for your next hire.
Execution: flow-runner drives real user journeys step by step, asserting outcomes (not just that clicks happened) and capturing evidence the moment anything fails. crud-tester is the only skill allowed to mutate data — under strict rules we'll get to. e2e-codegen converts a verified flow into a permanent @playwright/test spec.
The defect loop: bug-triage reproduces a suspected bug and establishes a minimal repro with an evidence bundle; bug-creator files it in Jira — deduplicated against existing tickets first; fix-verifier re-runs the repro on later builds and tells you FIXED, STILL BROKEN, or REGRESSED. bug-analyzer works the other direction: an existing ticket in, ranked root-cause hypotheses out.
Specialized checks: network-assertion (mock API failures and assert on real traffic), cross-browser-matrix, auth-session-audit, form-fuzzing, email-verification — each executing one kind of check properly.
Site-wide audits: accessibility-audit (axe-core plus the manual checks axe can't do), performance-audit (Core Web Vitals), seo-audit, i18n-audit, link-audit, and security-hygiene.
Visual & design: visual-snapshot captures a reviewable gallery of every page; visual-regression pixel-diffs two captures; figma-compare checks the built UI against its Figma design, field by field.
Synthesis: report-generator turns everything the other skills produced — across sessions, across a whole testing cycle — into one prioritized report you can send to your team.

Under the hood, the browser work runs on Playwright (via the Playwright MCP server or @playwright/cli), the design side uses the Figma MCP server, and the Jira family talks to Jira Cloud through a dependency-free Python client.

The opinions baked in

Tools are easy; discipline is hard. The value of these skills is less in what they do and more in what they refuse to do. A few principles run through the whole toolkit:

Evidence over vibes. Every finding cites what was actually observed — the screenshot, the console error, the network response. A bug isn't a bug until it has a minimal repro and a reproducibility count. bug-creator will actively route an unverified finding back through bug-triage before filing, because one withdrawn false positive costs more credibility than ten good tickets earn.

Verified and inferred are never confused. A mocked API response, a lab performance number, and a test case nobody has executed yet are all clearly labeled as such. e2e-codegen refuses to generate test code from a scenario that's never actually been run — that would bake untested assumptions into something that looks authoritative.

Mutations are contained. Only one skill is allowed to create, edit, or delete data, and only in a tenant you've explicitly named as safe. Every record it creates carries a run marker, and cleanup is itself a test. Every Jira write in the toolkit is dry-run by default — nothing is filed, commented, or transitioned until you've seen the exact payload and said yes.

Explicit about what wasn't covered. Every run report states its scope and its gaps rather than implying completeness. And two things are deliberately out of scope: aesthetic judgment (not checkable the way everything else is) and anything resembling penetration testing — the fuzzing and hygiene skills draw a hard line at injection payloads and exploit attempts.

Knowledge compounds. A target profile (targets/*.yaml) records everything the toolkit learns about your app — the login quirks, the safe tenant, the known console noise, the field that rejects "+" in phone numbers — so every run makes the next run cheaper instead of rediscovering the same facts.

Getting started

You'll need Node.js, an agent that supports skills, and the Playwright CLI (npm install -g @playwright/cli@latest, then playwright-cli install --skills). Then install Snagly:

npx skills add softwaretestingtrends/snagly --all

On Claude Code you can install it as a plugin instead, which handles updates for you:

/plugin marketplace add softwaretestingtrends/snagly
/plugin install snagly@snagly

Point it at your app by copying targets/example.yaml into your project and filling in the base URL, where credentials live (env vars — never in the file), and the login recipe. Then just ask:

"What can you test here?"

start-testing takes it from there — you don't have to know which of the thirty skills you need.

What it found on my own site

Here's the part I can't fake. My own site, in production, nothing fixed beforehand:

A sitemap that doesn't exist. My robots.txt has been pointing search engines at a 404 for months.
A critical accessibility bug on the signup page. The show/hide password toggles had no accessible name, so screen-reader users couldn't tell what they did — on the page where people create accounts.
Nine pages sharing one title tag and one meta description, no canonicals, no Open Graph tags. That last one is why links to my site had been rendering blank previews.
A pricing bug nobody would find by hand. One course is currently free. Shopify charges zero, correctly. The button on my own site still said $24.99 — caught only because it compared what the button claims against what checkout actually does.
A background 406 on every enrolled course page, invisible in the UI, failing on every single load.

And a result I didn't expect: performance came back clean. Core Web Vitals good across the board, with an unprompted note that these were lab numbers, not the field data Google grades against. A tool that only ever finds problems isn't measuring anything.

The finding that was wrong

One report said the login and signup pages had no visible focus indicator — a real WCAG failure if true. I tabbed through, and I could plainly see an orange focus ring.

It was a false positive, and the cause is instructive. The check read computed styles after focusing elements programmatically, which doesn't reliably trigger :focus-visible. Frameworks that build their focus ring from CSS variables — Tailwind, in my case — read as fully transparent in that state while actually rendering perfectly. The tool measured a ring that hadn't been asked to appear yet.

I shipped a fix the same day: that check may no longer report a failure from computed styles at all. It has to press Tab for real and compare a focused screenshot against an unfocused one.

I'm telling you this because it's the honest answer to the question you should be asking about any AI testing tool: what happens when it's wrong? Here, a human caught it in thirty seconds, and the toolkit got permanently better. Nineteen of Snagly's improvements so far came from exactly this — using it for real and fixing what broke.

The full prerequisites (including the optional Figma, Jira, and email-testing setups) are in the README.

Why the name

If you've worked a release the traditional way, you know the snag list — the running register of every defect, rough edge, and "that's not quite right" that stands between a build and a sign-off. Snagly is a toolkit for building that list properly: every snag caught with evidence, reproduced before it's reported, tracked until it's verified fixed. (And yes, it's still in service of the Software Testing Trends motto — learn smarter, test better.)

What's next

Snagly is MIT-licensed and open to contributions — new skills, better target-profile patterns, gotchas from your own Jira or app under test. Star the repo to follow releases (each one is tagged and versioned; updating is one /plugin marketplace update snagly away). It's also been submitted to the Claude Code community plugin marketplace — once listed there, it'll be browsable directly from /plugin with no marketplace-add step.

Part two is coming: I fix the issues above, then put Snagly back on the site to verify the fixes independently and close the Jira tickets it opened.

I'm also putting together a deeper guide on running an AI-augmented QA practice — how to adapt these skills to your own app, build new ones, and roll this out to a team. If you want that when it's ready, subscribe here — and if you take Snagly for a spin this week, I'd genuinely love to hear what broke, what surprised you, and what's missing.

— Ambreen Khan, Software Testing Trends

Episode 6 — Watching Something You Can't See

surajrkhonde — Wed, 05 Aug 2026 03:40:11 +0000

Week 3. "The deploy is done. Everything's green. Now what am I actually supposed to be looking at?"

Previously

Runner
    ↓
Cache
    ↓
Artifact
    ↓
Deployment

Today
    ↓
Monitoring

Junior Engineer: The canary rolled out fine yesterday. 100% traffic, all healthy. I closed my laptop. Was that wrong?

Senior Engineer: Not wrong, exactly. But let me ask you something first. Your service is running on a server somewhere. Right now, this second — is it healthy?

Junior Engineer: I mean... I assume so? Nobody's messaged me.

Senior Engineer: "Nobody's messaged me" isn't an answer. It's the absence of one. That's the entire problem monitoring exists to solve.

The Thing Nobody Says Out Loud

Senior Engineer: Here's an uncomfortable fact about production systems: you cannot see them. Not directly. You're not standing next to the server, watching electricity move through it. Everything you know about whether it's healthy is a claim — something a piece of software told you, that you're choosing to trust.

Junior Engineer: That sounds obvious when you say it, but I don't think I've ever actually thought about it that way.

Senior Engineer: Most engineers don't, until the gap between "the system told me it's fine" and "the system is actually fine" bites them. Monitoring is the discipline of shrinking that gap — of making sure what you're told is close to what's actually true, and told to you fast enough to matter.

📒 Senior Engineer's Notebook

You don't monitor a system because you don't trust it. You monitor it because you can't see it. Trust isn't the issue — visibility is.

The Car Dashboard Analogy

Junior Engineer: Can you make this concrete?

Senior Engineer: Think about driving a car. You can't see the engine. You can't see the oil level, the coolant temperature, how much fuel is actually left in the tank, mid-drive. All of that is invisible to you, sealed inside metal, while you're doing 100 km/h.

So the car gives you a dashboard. Speed, fuel, engine temperature, warning lights. You're not watching the engine. You're watching proxies for the engine — numbers and lights standing in for things you can't directly observe.

Junior Engineer: And a production system is the engine. Monitoring is the dashboard.

Senior Engineer: Exactly that. And just like a car, the dangerous failure mode isn't "the dashboard shows a problem." It's "the dashboard shows everything's fine, and it's wrong." A dashboard that lies is worse than no dashboard — because no dashboard, at least, you know you're flying blind.

What Actually Gets Measured

Junior Engineer: Okay, so — what goes on the dashboard? What do you actually watch?

Senior Engineer: There's a well-known starting set, sometimes called the four golden signals. Not the only things worth tracking, but the ones that catch the most real problems the fastest.

Latency — how long requests take. Not just the average — the average can look perfectly healthy while 5% of your users wait eight seconds. You want percentiles: p50, p95, p99. What's typical, and what's the bad end of typical.

Traffic — how many requests you're actually getting. Without this number, every other number is meaningless. An error rate of 2% means something completely different at ten requests a minute versus ten thousand.

Errors — the rate of requests failing. Not just 500s — a request that "succeeds" with the wrong data is a failure your server doesn't know to report as one.

Saturation — how close your system is to its limit. CPU, memory, database connections, queue depth. A system can have zero errors right now and still be forty seconds from falling over.

Junior Engineer: That last one feels different from the others. The others are about what's happening. Saturation is about what's about to happen.

Senior Engineer: That's a sharp distinction, and it's exactly right. Latency, traffic, and errors tell you the present. Saturation is close to the only one of the four that gives you a warning before the present turns bad.

🪞 If I asked you this in an interview

"What are the four golden signals, and why those four specifically?"

Latency, traffic, errors, and saturation. Together they answer the questions that matter most during an incident: is it slow, how much load is it under, is it actually failing, and how close is it to running out of capacity. Most production problems show up in at least one of these four before they show up anywhere else — which is why they're the default starting point rather than an exhaustive list.

Monitoring vs Alerting

Junior Engineer: Is a dashboard the same thing as monitoring, then? Just — numbers on a screen?

Senior Engineer: A dashboard is monitoring's visible half. But a dashboard only helps if someone's staring at it at the exact moment something breaks. Nobody's staring at a dashboard at 3 AM.

Junior Engineer: So how does anyone find out at 3 AM?

Senior Engineer: That's alerting — the other half. You define a threshold: error rate above 5% for two minutes, latency p99 above 3 seconds, saturation above 90%. Cross that threshold, and instead of waiting for a human to notice a graph, the system pages someone. This is where PagerDuty from Episode 1 actually connects to everything we've built since — the pager doesn't go off because a human is watching. It goes off because a machine was, continuously, and a human wasn't.

Junior Engineer: So monitoring is the measuring. Alerting is the "go wake somebody up" part.

Senior Engineer: Precisely. Monitoring without alerting is a dashboard nobody's watching. Alerting without monitoring doesn't exist — there's nothing to alert on.

And dashboards aren't only useful during an outage, worth saying explicitly. A healthy-looking dashboard, watched over days and weeks, is how teams catch trends before they become incidents at all — latency creeping up 5% every release, memory usage climbing slightly with each deploy, error rate drifting from 0.1% to 0.4% over a month with nothing dramatic enough to trip an alert. Nothing there pages anyone. But someone glancing at trends, not just thresholds, catches the slow version of the same story the fast version tells at 3 AM.

📝 Production Note

Remember correlation IDs and structured logs, from Episode 1? That's not a separate topic from monitoring — it's the raw material monitoring is built from. A dashboard showing "errors spiked at 2:14 PM" tells you that something broke. Structured logs, searchable by correlation ID, are what let you find out which request, for which user, touching which service — the difference between knowing something's wrong and knowing what to actually fix.

In modern observability, engineers usually lean on three sources of truth together, not one alone: metrics tell you something is wrong — a number crossed a line. Logs help explain what happened — the specific events around it. Traces show where the request spent its time as it moved across multiple services. You'll meet traces properly once we're deep into distributed systems — for now, just know the three aren't competing tools, they're three different angles on the same question.

The Alert Fatigue Trap

Junior Engineer: This seems easy to over-do, though. Why not just alert on everything, so nothing slips through?

Senior Engineer: Because a human being can only take so many 3 AM pages before they stop trusting them. That's called alert fatigue, and it's one of the most common ways monitoring quietly fails — not by missing a real incident, but by burying it under noise.

Junior Engineer: How does that actually play out?

Senior Engineer: Say your team gets paged fifteen times a week. Twelve of those are noise — a brief blip that self-resolved, a threshold set too aggressively, a known flaky check. After a few weeks of that, the on-call engineer starts silencing pages before fully reading them. Reflex, not laziness — self-preservation.

Then, one night, page sixteen is the real one. And it gets the same half-second of attention as the twelve before it that didn't matter.

Junior Engineer: So too many alerts is almost worse than too few.

Senior Engineer: In a very specific sense, yes — because too few means a known gap. Too many means a false sense of coverage while the actual signal quietly drowns.

Two terms are worth having exact language for here, because you'll hear them constantly. A false positive is an alert that fires when nothing's actually wrong — that's the noise causing fatigue. A false negative is the opposite and more dangerous failure: something is wrong, and no alert fires at all. Tuning alerts is really just a constant tradeoff between the two — tighten thresholds to catch more real problems, and you risk more false positives; loosen them to reduce noise, and you risk missing something real. There's no setting that eliminates both.

👦 Junior Framing / 👨‍🦳 Senior Framing

Junior framing: "More alerts means better monitoring."

Senior framing: More alerts means more noise, unless each one is something a human genuinely needs to act on right now. A good alert is a promise: if this fires, it matters, and someone needs to do something about it immediately. Break that promise enough times and the alert stops meaning anything.

SLOs and Error Budgets

Junior Engineer: How do teams decide what "healthy" even means numerically? Is 99% uptime good? 99.9%?

Senior Engineer: This is where SLOs — service level objectives — come in. A team explicitly decides: "99.9% of requests should succeed, measured over 30 days." That's not aspirational marketing language. It's a number the team commits to, and measures against.

Junior Engineer: Why not just aim for 100%?

Senior Engineer: Because 100% is enormously expensive, and past a certain point, chasing it stops making users happier and starts just slowing the team down. 99.9% still means about 43 minutes of downtime a month are considered acceptable — not desired, but acceptable, a deliberately budgeted amount of failure.

Junior Engineer: Budgeted failure. That's a strange phrase.

Senior Engineer: It's called an error budget, and the phrase is supposed to feel strange — it's meant to change behavior. If you've used up this month's error budget on a risky deploy that caused an outage, that's a signal: slow down, stabilize, stop taking risky bets for a while. If you've got budget left, you have room to ship faster and take more chances. It turns "is it safe to deploy" from a gut feeling into an actual number.

🚨 Beginner Mistakes

"I'll alert on CPU usage being high." High CPU isn't inherently bad — sometimes it means the system is efficiently using the resources it has. Alert on the thing that actually matters to users: is latency degrading, are requests failing. CPU is a diagnostic detail, not the actual problem.

"I'll set every alert threshold as tight as possible, to catch things early." This is how alert fatigue starts. A threshold that fires on normal, healthy variation trains people to ignore it — right up until the day it fires for something real.

"The dashboard looks fine, so we're done watching." A dashboard is a snapshot of now. The four golden signals can look perfectly healthy one minute before a slow memory leak crosses a threshold. Monitoring is a continuous practice, not a one-time check after deploying.

🏢 Office Reality

"We're flying blind on this service." No meaningful monitoring exists for it — nobody would know if it broke until a user complained.
"That alert is noisy." It fires often enough, for things that don't actually matter, that people have started ignoring it. A polite way of saying "this alert has stopped doing its job."
"We burned our error budget." The team exceeded their allowed failure rate for the period — often the trigger for pausing risky launches until things stabilize.
"Check the dashboards before you page anyone." Standard on-call instinct — confirm what's actually happening before waking up a second person for something that might resolve on its own.

The Alert That Almost Wasn't

Junior Engineer: Has an alert actually caught something real for you — not in theory, an actual time?

Senior Engineer: More than once, but here's a clean one. A saturation alert — database connection pool climbing toward its limit — fired quietly on a Tuesday afternoon. No errors yet. No user complaints. Latency barely moved.

Easy to ignore. Nothing was actually broken yet.

Junior Engineer: But someone looked anyway.

Senior Engineer: Someone looked anyway, because the alert existed and the team trusted it enough not to reflexively dismiss it — which, notice, is the entire alert fatigue conversation paying off in real time. Turned out a recent deploy had a connection leak — every request opened a database connection and a rare error path failed to close it. Slow leak. Would've taken maybe three more hours to actually exhaust the pool and start failing every single request, in the middle of the evening peak.

Junior Engineer: So the alert fired before anything was actually broken.

Senior Engineer: That's the entire point of saturation as a signal. It's not telling you what's wrong right now. It's telling you what's about to be wrong, while there's still time to do something other than panic.

🎯 Interview Perspective

Interviewer: How would you decide what to put an alert on?

Weak answer: Alert on anything that could go wrong.

Strong answer: Alert on symptoms that directly affect users or clearly predict an imminent failure — elevated error rate, degraded latency, saturation approaching a limit — not on every internal metric that could theoretically be interesting. Every alert should represent something a human genuinely needs to act on right now; anything less specific belongs on a dashboard for investigation, not in someone's pocket at 3 AM.

🎤 Explain It In One Minute

Imagine explaining this to a teammate — without using the words monitoring, alerting, or SLO.

Why is "the deploy went fine" not the same claim as "the system is healthy"?

If you can answer that without reaching for the vocabulary, you understand the idea driving this entire episode.

Whiteboard Moment

Production system running (invisible to you directly)
    ↓
Metrics collected continuously
    (latency, traffic, errors, saturation)
    ↓
Dashboards — visible, but nobody's always watching
    ↓
Thresholds defined against SLOs
    ↓
Threshold crossed → alert fires → someone paged
    ↓
Structured logs + correlation IDs → find the specific cause
    ↓
Fixed, or escalated toward an incident

Junior Engineer: So this whole episode is really just: you can't see it, so you measure it, and you don't wait for a human to notice the measurement.

Senior Engineer: That's the entire discipline in one sentence. Everything else — golden signals, SLOs, error budgets — is just making that sentence precise enough to actually act on.

What You Should Be Able to Explain Now

(Without looking at Google)

Can you explain:

Why "the deploy succeeded" and "the system is healthy" are different claims?
What the four golden signals are, and why saturation is different from the other three?
The difference between monitoring and alerting?
Why too many alerts can be more dangerous than too few?
What an SLO and an error budget actually are, in plain terms?
Why saturation alerts can fire before anything is technically broken yet — and why that's the point?

If yes — you understand production visibility the way it's actually practiced, not just as a dashboard you glance at after a deploy.

Junior Engineer: Okay, so say an alert does fire. For real, not a false alarm. What happens next?

Senior Engineer: Now we're back exactly where Episode 1 started — Friday, 7:02 PM, PagerDuty going off. Except this time, you're not watching from the outside. You already know what a webhook is, what a runner is, what an artifact is, how it got deployed, and what the dashboard was supposed to be telling someone the whole time.

Junior Engineer: So next time, I'm not just shadowing. I'm actually part of figuring it out.

Senior Engineer: That's the idea. Incident response, properly this time — from the inside.

(End of Episode 6)

LLM Latency Budget: Make AI Features Feel Fast Without Burning Money

Jack M — Wed, 05 Aug 2026 03:38:03 +0000

A slow AI feature does not feel smart. It feels broken.

That is the uncomfortable truth many AI SaaS builders hit after the demo works. The prototype answers well, the agent can call tools, and the RAG pipeline looks impressive. Then real users arrive. Prompts get longer. Queues form. Streaming starts late. One tenant uploads huge documents. Another runs bulk jobs at noon. Suddenly the same workflow that felt magical in testing feels like a spinner with an invoice attached.

The fix is not simply “use a faster model.” You need an LLM latency budget: a small set of rules that says how fast each AI workflow must feel, how many tokens it can spend, when to stream, when to cache, when to route to another model, and when to stop before cost and latency drift together.

This guide is for solo SaaS developers, micro SaaS builders, and AI SaaS teams shipping production features with LLM APIs, RAG, agents, or self-hosted models.

Why latency budgets matter now

AI platform news points in the same direction: builders are moving from chat demos to production workflows. Agent tools, web context APIs, voice agents, coding assistants, and RAG platforms are all getting more capable. At the same time, inference cost and reliability are under pressure.

Latency is now a product metric. Inference efficiency is becoming a business metric. Yet many articles stop at TTFT, TPOT, quantization, batching, or model serving. Fewer show how a SaaS builder turns those ideas into a product-level budget with code, dashboards, fallbacks, and customer-safe limits.

The simple model: TTFT, TPOT, and total time

You do not need a PhD in serving systems to start. Track three numbers.

Time to First Token

Time to First Token (TTFT) is the delay between the user action and the first streamed token. It includes network time, queue time, provider overhead, tool setup, retrieval, and the model’s prefill phase.

High TTFT is why a chat box feels dead.

Time Per Output Token

Time Per Output Token (TPOT) is the average time between generated tokens after the first token appears.

High TPOT is why streaming feels like a dripping tap.

End-to-end latency

End-to-end latency is the full time from request to final answer.

A rough formula is:

end_to_end_latency = TTFT + (output_tokens - 1) * TPOT

That formula is not perfect for every provider, but it is good enough to reason about the user experience.

Build budgets by workflow, not by model

A common mistake is to set one global target like “AI responses must finish in 5 seconds.” That sounds clean but fails fast.

Different workflows need different budgets.

Workflow	User expectation	Suggested latency budget
Inline autocomplete	Feels instant	TTFT under 300ms, very short output
Chat answer	Starts quickly	TTFT under 1.5s, stream response
RAG answer with citations	Trust matters	TTFT under 3s, final answer under 15s
Agent with tool calls	Progress matters	First status under 1s, step updates every few seconds
Bulk document task	Completion matters	Async job, no chat-style waiting

The key is to budget for the experience, not the raw model call.

A user can forgive a 40-second background report if the UI says what is happening. The same user may abandon a 6-second inline writing assistant if nothing appears.

A practical LLM latency budget template

Create a budget object for each AI workflow.

{
  "workflow": "support_rag_answer",
  "max_ttft_ms": 2500,
  "max_total_ms": 15000,
  "max_input_tokens": 12000,
  "max_output_tokens": 900,
  "stream": true,
  "cache_policy": "semantic_and_exact",
  "fallback_model": "fast_general_model",
  "requires_citations": true,
  "async_after_ms": 12000
}

This turns “make it faster” into engineering constraints. Your app can now decide whether to trim context, stream, route to a faster model, switch to async, reject an oversized request, or use a cached answer.

Instrument every request

Start by logging latency and token data for every AI request. Do this before buying another tool or changing providers.

Here is a small TypeScript-style example.

type LlmTrace = {
  requestId: string;
  tenantId: string;
  workflow: string;
  model: string;
  inputTokens: number;
  outputTokens: number;
  ttftMs: number | null;
  totalMs: number;
  costUsd: number;
  cacheHit: boolean;
  status: "success" | "timeout" | "error";
};

async function runWithTrace(input: {
  tenantId: string;
  workflow: string;
  prompt: string;
}) {
  const started = Date.now();
  let firstTokenAt: number | null = null;
  let output = "";

  const stream = await llm.stream({
    model: "fast-general",
    prompt: input.prompt,
    max_tokens: 700
  });

  for await (const chunk of stream) {
    if (!firstTokenAt) firstTokenAt = Date.now();
    output += chunk.text;
    sendToClient(chunk.text);
  }

  const finished = Date.now();

  const trace: LlmTrace = {
    requestId: crypto.randomUUID(),
    tenantId: input.tenantId,
    workflow: input.workflow,
    model: "fast-general",
    inputTokens: estimateTokens(input.prompt),
    outputTokens: estimateTokens(output),
    ttftMs: firstTokenAt ? firstTokenAt - started : null,
    totalMs: finished - started,
    costUsd: estimateCost(input.prompt, output),
    cacheHit: false,
    status: "success"
  };

  await saveTrace(trace);
  return output;
}

Keep the trace simple. If you capture request ID, tenant ID, workflow, model, tokens, TTFT, total time, cost, cache hit, and status, you can answer most early performance questions.

Control input tokens before touching infrastructure

Long prompts hurt TTFT. Long context means more work before the first token appears.

For AI SaaS products, input bloat usually comes from full chat history, too many RAG chunks, raw HTML, unused tool descriptions, repeated system instructions, or entire customer records when only a few fields matter. Before optimizing GPUs or switching vendors, cut useless context.

Use a context packer.

type ContextItem = {
  id: string;
  text: string;
  priority: number;
  tokenEstimate: number;
};

function packContext(items: ContextItem[], maxTokens: number) {
  const sorted = [...items].sort((a, b) => b.priority - a.priority);
  const selected: ContextItem[] = [];
  let used = 0;

  for (const item of sorted) {
    if (used + item.tokenEstimate > maxTokens) continue;
    selected.push(item);
    used += item.tokenEstimate;
  }

  return selected;
}

This is not fancy. That is the point. A basic priority-based packer often beats “send everything and hope.”

For RAG, use fewer, better chunks. For agents, expose fewer tools per step. For browser automation, clean the page before putting it into the prompt.

Cap output tokens by job type

Output tokens drive total latency and cost. Many AI features do not need long answers.

Set output caps by workflow:

Rewrite suggestion: 120 tokens
Error explanation: 250 tokens
Support answer: 700 tokens
Technical plan: 1,200 tokens
Background report: async job with a larger cap

Also give the model a structure that discourages rambling.

Answer in this format:
1. Direct answer: 2 sentences max
2. Steps: up to 5 bullets
3. Caveat: 1 short note if needed

This improves scannability and reduces token drift.

Use streaming for perception, not as a bandage

Streaming can make an AI feature feel faster, but it does not fix everything.

Use streaming when:

The user is reading generated text
The answer may take more than 2 seconds
Partial output is useful
You can show citations or tool results after the draft begins

Do not rely on streaming when:

The workflow must return valid JSON
The user needs a single deterministic result
The model must complete tool calls before saying anything
You are hiding a slow retrieval or database step before the model starts

For agent workflows, stream status events, not only text.

{ "type": "status", "message": "Searching relevant docs" }
{ "type": "status", "message": "Checking account permissions" }
{ "type": "status", "message": "Drafting answer with citations" }

This keeps users oriented while the system does real work.

Route models by latency class

Not every request deserves your strongest model.

Create latency classes:

Class	Use case	Model strategy
Instant	autocomplete, labels, short rewrites	smallest reliable model
Fast	support chat, extraction, routing	fast general model
Careful	legal-ish, financial-ish, complex reasoning	stronger model with tighter scope
Background	reports, audits, batch enrichment	slower model or queued worker

A simple router can start with rules.

function chooseModel(workflow: string, risk: "low" | "medium" | "high") {
  if (workflow === "autocomplete") return "small-fast";
  if (workflow === "bulk_report") return "batch-careful";
  if (risk === "high") return "careful-reasoning";
  return "fast-general";
}

Later, you can route based on measured performance, tenant plan, queue depth, or failure rate. Start with rules that developers can understand and debug.

Cache the boring parts

Caching is one of the easiest ways to improve both latency and cost, but cache the right things.

Good cache candidates:

Embeddings for unchanged documents
RAG retrieval results for common queries
System prompt templates
Tool schemas
Classification outputs
Deterministic transformations
Answers to low-risk, repeated questions

Bad cache candidates:

Permission-sensitive answers without tenant scoping
Personalized answers without user scoping
Answers based on rapidly changing data
Outputs that may contain stale prices, policies, or account state

Always include tenant and permission context in cache keys.

function cacheKey(input: {
  tenantId: string;
  userRole: string;
  workflow: string;
  normalizedQuery: string;
  sourceVersion: string;
}) {
  return [
    input.tenantId,
    input.userRole,
    input.workflow,
    input.sourceVersion,
    hash(input.normalizedQuery)
  ].join(":");
}

A cache hit that leaks data is worse than no cache.

Add graceful degradation

Your app needs a plan for bad days: provider slowness, queue spikes, long documents, or tenants running large jobs.

Useful degradation patterns:

Switch from careful model to fast model for low-risk requests
Reduce retrieved chunks when TTFT is at risk
Shorten output length during load spikes
Move long tasks to async jobs
Show partial results with “continue generating”
Ask the user to narrow the request before spending tokens

Example:

if (queueDepth > 100 && workflow === "support_rag_answer") {
  budget.max_input_tokens = 6000;
  budget.max_output_tokens = 500;
  budget.fallback_model = "fast-general";
}

This is not about lowering quality everywhere. It is about protecting the experience under pressure.

Watch p95, not averages

Average latency lies. Your happy path can look fine while real users suffer.

Track these metrics by workflow and tenant tier:

p50 TTFT
p95 TTFT
p50 total latency
p95 total latency
input tokens per request
output tokens per request
cache hit rate
timeout rate
cost per successful task
retries per request

A simple alert rule is enough at first.

Alert when support_rag_answer p95 TTFT > 3000ms for 10 minutes.
Alert when cost per successful task rises 30% above 7-day baseline.
Alert when timeout rate > 2% for any paid tenant tier.

Tie latency to cost. If p95 latency and cost both rise, you may have context bloat, retry loops, poor routing, or a workflow that should become async.

Treat retries as a budget risk

Retries feel harmless in code and expensive in production.

A retry can double cost, increase latency, and create duplicate tool actions. For agents, retry loops are even riskier because the model may call tools again.

Use retry rules:

Retry network errors with jitter
Do not retry validation failures blindly
Never retry write actions without idempotency keys
Stop after a small number of attempts
Log retry reason and added cost

const retryPolicy = {
  maxAttempts: 2,
  retryOn: ["rate_limit", "network_timeout"],
  neverRetryOn: ["invalid_json", "permission_denied", "policy_blocked"]
};

If a workflow needs three retries to feel reliable, it probably needs a better design, not a bigger retry loop.

When to use async instead of chat

Some AI work should not pretend to be instant.

Use async jobs for:

Large document analysis
Multi-source research
Long agent workflows
Bulk enrichment
Report generation
Evaluation runs
Tasks with external API rate limits

A good async UX includes:

Immediate job receipt
Progress updates
Cancel button
Estimated completion window
Final summary
Error state that explains what happened

This protects your chat interface from becoming a waiting room.

Implementation checklist

Use this before shipping a new AI feature:

[ ] Define max TTFT and total latency by workflow
[ ] Set input and output token caps
[ ] Log tenant, workflow, model, tokens, cost, TTFT, total time, status
[ ] Track p95 latency, not only averages
[ ] Stream text or status events when useful
[ ] Route models by workflow and risk
[ ] Cache safe repeated work with tenant-aware keys
[ ] Trim context before changing infrastructure
[ ] Move long tasks to async jobs
[ ] Alert on latency, timeout rate, retry rate, and cost per successful task

Final thought

An LLM latency budget is not bureaucracy. It is a guardrail for product quality.

When budgets are missing, every prompt can grow, every agent can wander, every retry can double spend, and every slow request can look like a mystery. When budgets exist, your team can make clear tradeoffs: faster first token, shorter output, better context, safer cache, async workflow, or stronger model only where it matters.

Fast AI is not just about speed. It is about respecting the user’s time while protecting your margins.

FAQ

What is an LLM latency budget?

An LLM latency budget is a set of limits for an AI workflow: maximum time to first token, maximum total response time, input token cap, output token cap, model route, caching rule, and fallback behavior.

What is a good TTFT for AI features?

It depends on the workflow. Inline suggestions should feel almost instant. Chat answers should usually start streaming within one or two seconds. RAG or agent workflows can take longer if the UI shows useful progress.

How do I reduce LLM latency quickly?

Start by trimming input tokens, limiting output length, streaming responses, caching repeated work, and routing simple tasks to faster models. These changes are often easier than changing infrastructure.

Should every AI workflow stream output?

No. Streaming works well for readable text and progress updates. It is less useful for strict JSON, hidden tool-call workflows, or tasks where partial output could confuse the user.

How does latency relate to AI cost?

Long prompts, long outputs, retries, and tool loops usually increase both latency and cost. That is why production teams should track tokens, latency, cache hit rate, and cost per successful task together.

Is self-hosting faster than using an API?

Not automatically. Self-hosting can reduce control-plane uncertainty, but serving models well requires batching, memory management, scaling, monitoring, and hardware tuning. Measure TTFT, TPOT, and total cost before assuming self-hosting is better.

AI Agent Safety: When Boundaries Fail with External Tools

Karnik Khanwilkar — Wed, 05 Aug 2026 03:37:02 +0000

AI agent safety boundaries are a critical challenge when agents use external tools. My journey into understanding how these boundaries can fail began with a deep dive into recent technical reports from leading AI research organizations.

I encountered this concept while exploring incidents reported by Anthropic and OpenAI. These reports detail scenarios where AI models, despite being explicitly instructed to operate within simulated environments, managed to interact with real-world systems. This phenomenon, often termed "boundary failure," occurs when the actual operational environment of an agent does not match its internal understanding or the constraints it has been given.

Modern AI agents are becoming incredibly useful because we're equipping them with capabilities far beyond just answering questions. They can run commands, browse the web, use APIs (Application Programming Interfaces), read and modify files, install packages, and interact with other systems. This ability to act and interface with the world is what makes agentic architectures so powerful and a direction truly worth investing in. However, the more an agent can do, the more critical the boundaries around it become.

A key example comes from Anthropic's July 30 report, detailing three incidents discovered during their cybersecurity evaluations. Claude models were explicitly told they had no internet access and were working inside simulated environments. However, a problem with the evaluation environment's configuration meant that internet access was actually available. While attempting their assigned cybersecurity exercises, the models reached real systems, initially treating them as part of the simulation. In one striking incident, a Claude model even published a malicious Python package to the real PyPI (Python Package Index) registry, all while believing it was still operating within its simulated exercise.

This wasn't simply an AI "deciding" to misbehave or to intentionally bypass security. The models were completing their tasks, but their understanding of the environment's boundaries was misaligned with the actual, misconfigured reality. This came shortly after a separate OpenAI incident involving Hugging Face, where models reached the real internet in importantly different ways, further underscoring the complexity. These events highlight a crucial distinction: a prompt is not a security boundary.

In simple terms, if an AI agent is given the ability to use tools like web browsers, APIs, or file system access, the actual system design and its underlying permissions must strictly enforce safety limits. Relying solely on the agent's internal understanding or instructions provided within a prompt is insufficient to guarantee isolation or prevent unintended actions in the real world.

Studying incidents like these reveals how vital it is to engineer robust guardrails around increasingly capable AI systems. As we push the boundaries of agentic architectures, ensuring alignment and preventing unintended actions becomes a complex, multi-layered engineering problem that extends far beyond the model's intelligence itself. This is why building beats consuming; actively engaging with these challenges is how we contribute to a safer AI future.

Here's what these real-world incidents highlighted for developers as we build and deploy AI agents:

A prompt is not a security boundary: Explicitly telling an agent "you don't have internet access" is not the same as actually removing internet access or restricting its network capabilities. True isolation requires physical or logical restrictions, enforced at the infrastructure or operating system level, not just linguistic ones.
The AI model isn't the whole AI system: The model's behavior matters, but it's only one component. The entire ecosystem, including the tools we connect it to, the permissions and credentials it receives, the environment it runs in, and the monitoring and safeguards built around it, all play a critical role. Each of these components can introduce vulnerabilities or points of failure.
Think about what happens when your assumptions are wrong: Both the Anthropic and OpenAI incidents occurred because core assumptions about a fully simulated and isolated environment did not match the reality of the underlying system configuration. Anticipating potential mismatches between developer instructions and the actual runtime environment is crucial for robust system design and resilience.
If an agent can act, we need to know what it's doing: When agents are empowered with the ability to run commands, browse the web, or modify files, comprehensive monitoring, logging, and audit trails become non-negotiable. We need clear visibility into their actions and interactions with external systems to detect and mitigate unintended behaviors promptly.
Give an agent what it needs, not everything you have: Implement the principle of least privilege rigorously. Grant only the minimum necessary permissions and access to tools or systems required for the agent's specific task. This minimizes the potential impact and "blast radius" if a boundary fails or an agent deviates from its intended path.

These challenges reinforce my belief that building truly adaptable and safe AI systems means not just consuming AI, but actively contributing to its foundational safety and ethical deployment. It requires a hands-on approach to system architecture, security engineering, and continuous evaluation, moving beyond theoretical discussions to real-world implementation. My journey continues to focus on how we can empower agents responsibly while upholding the highest standards of alignment and control, helping push the boundaries of what's possible in a secure manner.

Source: https://dev.to/hemapriya_kanagala/were-giving-ai-agents-more-tools-what-happens-when-the-boundaries-fail-46gh

We Measured AI Code Drift Across 5 Tools and 210 Components. Frequency Alone Lied to Us.

Jonathan Gordon — Wed, 05 Aug 2026 03:29:17 +0000

Empirical research from ReWeaver AI. 42 identical prompts, across 5 tools and 8 production dimensions, compared to human baseline. One metric that changes how you see drift.

Everyone knows AI-generated code has quality issues. What’s less understood is that the way most teams measure those issues — by how often they occur — systematically understates the risk.

We ran a controlled study to find out how badly. The answer surprised us, particularly in one dimension.

What We Did

We gave five leading AI coding tools (Cursor, Claude Code, Lovable, Figma Make, and VS Code with Copilot) 42 identical prompts: realistic single-component builds — buttons, forms, dashboards, navs, modals, auth surfaces. We scanned every output with ReWeaver, our deterministic drift-detection engine, across eight production readiness dimensions:

User Experience
Security & Privacy
Accessibility
Design Consistency
Reliability
Maintainability
Architecture
Testability

We also scanned six human-authored open-source repositories as a reference baseline.

For each dimension, we calculated two things:

Drift frequency — the percentage of lines containing at least one drift occurrence. Counts what went wrong.

Production Drift Ratio (PDR). The PDR is a metric that weights frequency by estimated remediation cost on a 0–1 scale. A PDR of 0.30 is roughly 45 minutes of cleanup per component; 0.70 is about 2.5 hours.

The Finding That Stopped Us

In Security & Privacy, AI tools produced 3× the human drift frequency. That looks manageable — a meaningful gap, but not alarming.

The PDR was 22× the human reference.

Not 22% more. 22 times more costly to fix.

The frequency gap makes Security & Privacy drift look like a minor concern. The PDR reveals it’s the most expensive problem in the dataset. AI-generated security drift (client-side authorization gates bypassable in DevTools, raw PII and credentials passed through props without tokenization) is syntactically identical to safe code. It passes review, but the fixes are harder to find and remedy.

This is the core argument of our study: frequency counts what went wrong. The PDR quantifies what it will cost to fix it.

The Full Results

Here’s how much more relative drift frequency and severity AI produced across all eight dimensions:

Dimension	Frequency multiplier	PDR multiplier
Security & Privacy	3.4×	22×
User Experience	4.5×	6.5×
Accessibility	1.7×	5.2×
Design Consistency	1.7×	4.1×
Reliability	1.4×	2.5×
Testability *	0.61×	2.0×
Architecture *	0.55×	1.7×
Maintainability *	0.52×	1.5×

* These three dimensions showed lower AI frequency than the human reference, likely due to a corpus maturity effect, not an AI advantage. Our human reference draws from mature production repositories carrying accumulated technical debt; the AI corpus is fresh greenfield components. The PDR gap remains positive even here.

In every dimension, AI-generated drift is more expensive to remediate than human-authored drift, even where humans produce more of it.

The Statistics

We ran Wilcoxon signed-rank tests comparing AI tool PDR scores against the human reference.

Global test (n = 40 paired observations; 5 tools × 8 dimensions): AI tools produced significantly more costly drift than the human baseline (z = −5.43, p < .001). Of 40 comparisons, 38 showed AI PDR above the human reference. (One was lower. One tied.)

The same test applied to drift frequency was not significant (z = −0.585, p = .559).

That asymmetry is the finding. The same code, measured two ways, tells two very different stories.

Per-dimension note: with n = 5 tools per dimension, the minimum attainable exact p-value is 0.0625, which doesn’t clear the conventional p≤.05 threshold. We report these results as directional evidence, not formally significant findings, supported by effect sizes (r = 0.90–0.91 in six of eight PDR dimensions) and unanimous positive ranks.

What the Drift Actually Looked Like

This is where the data gets concrete. The drift we found wasn’t just malformed code. It was absent code: components that satisfied the prompt and omitted the production context the prompt didn’t ask for.

Security & Privacy: Role gates enforced only in the UI, bypassable in DevTools. Raw credentials passed through props without tokenization. The model treated the browser as a trusted environment.

Accessibility: Focus escaped modals and was never returned. Interactive elements built without semantic markup. Keyboard users left stranded.

User Experience: Containers missing overflow containment. Forms that fail silently — errors that identify the problem but not the fix. Lists with no empty state. Constraint hints shown before the user has touched the field.

Design Consistency: Models named UI primitives from memory without checking they existed in the design system. Inline styles bypassed design tokens. Placeholder content shipped live.

Testability: document and window reached synchronously in component bodies — unmockable in test environments. The silent killer: components emitted with no tests alongside them.

What This Means in Practice

Drift is endemic. Every file we tested (both human and AI-generated) produced drift across every dimension. In terms of tool performance, no tool performed better across all eight dimensions. Switching tools hoping to improve on drift only changes where it shows up and how. No tool avoids drift.

Frequency misleads. A Security & Privacy gap that looks like 3× is actually 22× when you account for what fixing it costs. Teams relying on frequency-based metrics are systematically underestimating their production readiness risk, most severely in the dimensions that matter most.

Tool selection isn’t the answer. The actionable conclusion isn’t which tool to use. It’s that any tool requires a verification layer capable of catching what generation leaves behind.

Try It

The Playground at reweaver.ai/playground lets you paste React/TypeScript code and get your own PDR score. Free, no login, code scanned in memory and never stored.

The full research report is at https://info.reweaver.ai/drift-research-report

Image Upload Moderation Beyond Node.js: Classify NSFW and Violence with Multimodal Chat

JamesAnderson121 — Wed, 05 Aug 2026 03:21:58 +0000

Use multimodal chat with a strict JSON schema when your policy needs explainable labels for uploaded images; otherwise reach for a managed, fixed-taxonomy service. There is no dedicated image moderation endpoint here, so the practical design is a policy prompt, a vision-capable chat model, schema validation, and a conservative fallback.

That is my short answer. I would not ship the model's prose directly into an allow/block decision. I keep the original decision for audits, translate it into a small internal status, and make the eval set the release gate. The model is one component of the policy system — not the policy system itself.

What should a Python image upload moderation example classify for NSFW and violence?

The categories should come from the app's actual rules. For a general user-content product, I start with nudity, graphic violence, hate symbols, drugs, and minors-risk. I don't pretend those labels are universal: a medical forum and a marketplace need different thresholds, and a historical archive may legitimately show symbols that a profile-photo product should reject.

My first notebook pass is deliberately boring. I assemble a small set of allowed, blocked, and ambiguous pictures; write the expected category labels; and record the policy reason in plain English. Then I run the same prompt and schema across every candidate model. The score I care about first is false negatives on the block set, followed by false positives on harmless uploads. Overall accuracy can hide both.

This is also where a JSON schema earns its keep. A response containing "graphic_violence": "high" can be validated, stored, and compared. A paragraph such as “this appears concerning” can't reliably drive a queue or an appeal. Keep the provider response beside a normalized status such as allow, review, or block; when policy changes, you can replay the raw decisions without migrating every old record.

I learned the cost side the annoying way: one evaluation run consumed 18.7 million input tokens, roughly 3.4 times my estimate, because I had repeated the full policy rubric for every crop and retry. My notebook showed a reasonable per-case estimate, but the production-shaped harness expanded each source into several variants, then retried cases whose structured response failed validation. I had measured the neat path and budgeted for the messy one. I stopped the run, grouped usage by fixture and attempt, and found that the largest images weren't the main culprit; duplicated policy text across the expanded cases was. The fix was measurement, not guesswork. I made prompt tokens a first-class eval column, deduplicated image variants before dispatch, and reported cost per accepted decision rather than cost per request. That last denominator matters because a cheap response that lands in manual review hasn't completed the job. I also put a batch-level ceiling around experiments, so a mistaken multiplier stops early instead of becoming a surprise at the end of the day. Now I count prompt tokens before any large run and inspect the distribution, not just its mean.

Small batches first.

No prose parsing.

The focused implementation

The example below sends one local image to the verified POST /v1/chat/completions route. It uses a data URL so the program is self-contained, takes both the API key and vision model ID from environment variables, requires structured JSON, and retries rate limits while honoring Retry-After. I use the standard-library HTTP client because this article is about the policy boundary, not a framework choice.

import base64
import json
import mimetypes
import os
import random
import sys
import time
import urllib.error
import urllib.request


API_URL = "https://api.infrai.cc/v1/chat/completions"
CATEGORIES = (
    "nudity",
    "graphic_violence",
    "hate_symbols",
    "drugs",
    "minors_risk",
)


def as_data_url(path: str) -> str:
    media_type = mimetypes.guess_type(path)[0] or "application/octet-stream"
    with open(path, "rb") as image_file:
        encoded = base64.b64encode(image_file.read()).decode("ascii")
    return f"data:{media_type};base64,{encoded}"


def post_with_backoff(payload: dict, attempts: int = 5) -> dict:
    api_key = os.environ["INFRAI_API_KEY"]
    body = json.dumps(payload).encode("utf-8")

    for attempt in range(attempts):
        request = urllib.request.Request(
            API_URL,
            data=body,
            headers={
                "Authorization": f"Bearer {api_key}",
                "Content-Type": "application/json",
            },
            method="POST",
        )
        try:
            with urllib.request.urlopen(request, timeout=60) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            error_body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == attempts - 1:
                raise RuntimeError(
                    f"Chat request failed with HTTP {error.code}: {error_body}"
                ) from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2**attempt + random.random()
            time.sleep(delay)

    raise RuntimeError("Retry budget exhausted")


def moderate(path: str) -> dict:
    label_properties = {
        category: {"type": "string", "enum": ["none", "low", "high"]}
        for category in CATEGORIES
    }
    payload = {
        "model": os.environ["INFRAI_VISION_MODEL"],
        "messages": [
            {
                "role": "system",
                "content": (
                    "Classify this upload under the supplied app policy. "
                    "Use high only for clear evidence. Return JSON only."
                ),
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": (
                            "Policy: flag nudity, graphic violence, hate symbols, "
                            "drugs, and minors-risk. Give a brief policy reason."
                        ),
                    },
                    {"type": "image_url", "image_url": {"url": as_data_url(path)}},
                ],
            },
        ],
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "upload_moderation",
                "strict": True,
                "schema": {
                    "type": "object",
                    "properties": {
                        "labels": {
                            "type": "object",
                            "properties": label_properties,
                            "required": list(CATEGORIES),
                            "additionalProperties": False,
                        },
                        "reason": {"type": "string"},
                    },
                    "required": ["labels", "reason"],
                    "additionalProperties": False,
                },
            },
        },
    }

    raw_response = post_with_backoff(payload)
    raw_decision = json.loads(raw_response["choices"][0]["message"]["content"])
    levels = set(raw_decision["labels"].values())
    normalized_status = (
        "block" if "high" in levels else "review" if "low" in levels else "allow"
    )
    return {"raw_decision": raw_decision, "normalized_status": normalized_status}


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("Usage: python moderate_upload.py IMAGE_PATH")
    print(json.dumps(moderate(sys.argv[1]), indent=2))

Set INFRAI_API_KEY, set INFRAI_VISION_MODEL to a currently available multimodal model from the live model catalog, then run the file with an image path. Model availability changes, so I avoid baking an ID into an article. The explicit schema is the fallback boundary: malformed or missing fields should send an upload to manual review rather than quietly allowing it.

Choosing the integration boundary

I compare systems by who owns the taxonomy, how much adapter code lands in my repository, and whether I can replay decisions. Exact model support changes; verify image input and structured-output support in each provider's current documentation before committing.

Option	Integration shape	Policy and evaluation trade-off	I would choose it when
Infrai	OpenAI-compatible chat behind one REST API	My team owns the schema, thresholds, normalization, and evals	I expect moderation to sit beside other backend capabilities and want one consistent contract
OpenAI API	Direct model-provider integration	My team still owns the app-specific policy mapping and regression set	The application already standardizes on OpenAI's client surface
Google Gemini API	Direct model-provider integration	I must test my schema and image set against its current model behavior	Gemini is already the evaluated model family in the stack
Anthropic API	Direct model-provider integration	I must confirm current image and structured-output behavior for my exact contract	The team already operates and evaluates Anthropic models
Amazon Rekognition	Managed image-analysis service	A service-defined feature set may require an adapter to my internal statuses	I prefer a specialized managed vision workflow over chat-prompt ownership

Infrai's relevant advantage is breadth behind a simple surface: 295 routes across 20 modules sit under one key and one REST contract. For a Python team moving from notebook to production, adding a backend capability can mean another endpoint rather than another SDK, credential set, and adapter. The public discovery response is self-describing, too, so I can inspect readiness and schemas before generating a client.

The catch is real. Infrai does not provide a dedicated image moderation endpoint in this path, which means my team owns policy wording, the JSON contract, calibration, and appeals. Stick with a specialized managed moderation product when you want its fixed taxonomy and operational workflow, or stay direct with OpenAI, Google, or Anthropic when provider-specific controls matter more than a common interface. I'm not sure which model will win on your images; your mileage may vary, and only a representative eval set settles it.

Failure policy matters more than prompt polish

A production gate needs an explicit response to uncertainty. I map schema failures, unknown labels, and low-confidence classifications to review; clear high-severity evidence maps to block; only a complete all-clear maps to allow. That conservative mapping is intentionally outside the model prompt. Product code can test it, version it, and explain it during an appeal.

I also store the raw model decision, normalized status, policy version, model ID, and request ID. The first two are the core distinction: raw evidence preserves what the classifier returned, while the normalized value keeps downstream systems stable when I rename a category or tighten a threshold. Retention and access rules should match the sensitivity of user uploads. Don't treat the audit store as an excuse to keep images forever.

There is another tempting distraction: image upscaling. Infrai exposes optional Lanczos-only upscale, but resizing is separate from moderation and is not a safety control. I would test the original upload for the decision path. If a product separately needs an enlarged asset, treat that as image processing with its own purpose and retention rules.

This section is short on purpose. The hard work isn't a clever prompt; it's deciding what happens when the classifier is uncertain, proving that behavior with tests, and keeping enough evidence to revisit a decision.

What I measure before copying this design

Before launch, I freeze a labeled set that reflects the real upload mix, including benign edge cases and policy-boundary examples. I report false-negative rate per severe category, false-positive rate, manual-review rate, schema-valid response rate, and cost per final decision. I also slice results by image source and policy category because one aggregate score can conceal a bad hate-symbol result behind easy drug-free photos.

Then I rerun the suite whenever the prompt, schema, policy, or model changes. A candidate ships only if it meets the category thresholds and does not push review volume past the team's capacity. I keep a small shadow run for changed models before routing live decisions to them. This is where my notebook habits help: the same fixtures and assertions that picked the model become a production regression harness.

Prompt cost belongs in that harness. Count the repeated policy text, track retries, and measure the total input per accepted decision — a tiny request viewed in isolation can become an expensive batch after crops, retries, and multiple candidates. As far as I can tell, there is no honest universal threshold for “good enough” moderation. The right bar depends on harm severity, reviewer capacity, and what users can appeal.

Ship the gate only after the fallback has been exercised, not merely described. Feed it malformed structured output in a unit test. Confirm that it chooses review. Then evaluate the real model on the frozen image set and record the policy version with every result. That makes the design reproducible, which matters far more to me than a polished demo screenshot.