What "AI-Powered" Should Actually Mean Before You Sign an Agency Contract

"AI-powered" has become the "synergy" of web development proposals. Every agency deck has it. Few agencies can tell you what it costs to run, how they know it's working, or what happens when it confidently makes something up.

That's not cynicism — it's the pilot-to-production gap, and it's the defining failure mode of AI projects in 2026. As one recent industry guide put it: a demo that impresses in a meeting often collapsles on real traffic, messy inputs, and live permissions. The fix is to build for production from the start, not to bolt it on after the demo wins approval. (Technource, September 2026)

If you're evaluating an agency for an AI feature, here are the questions that separate the ones who've shipped from the ones who've only demoed.

1. "How will you measure whether it works?"

This is the single most revealing question you can ask. Listen for specifics: task success rate on your data, accuracy benchmarks, latency targets. Be wary of fluency as a metric — "it sounds smart" is not a measurement. One AI consultancy's list of PoC anti-patterns starts with exactly this: measuring only fluency instead of task success, and testing on synthetic toy documents that never appear in production. (Advanta Labs)

A serious answer sounds like: "We'll build an evaluation set from your real tickets/documents/queries, run every prompt change against it, and show you the pass rate." If the agency can't describe their eval process, they don't have one.

2. "What does it cost at 10x usage?"

Token spend scales with usage, and an AI feature that's cheap in a pilot can become genuinely expensive at scale. Cost controls — caching, prompt optimization, model tiering (cheap model for easy cases, strong model for hard ones) — belong in the architecture, not in a cleanup pass six months later. (Technource)

Ask the agency to model it: what does 1,000 requests a day cost? What about 100,000? If they can't sketch that math, you're signing up to discover the answer from your cloud bill.

3. "What does the software around the model look like?"

Models are not products. The software around them is. A production AI feature needs the unglamorous engineering that never appears in the demo:

• Authentication and authorization on every AI endpoint — especially anything with retrieval or tool access
• Request budgets: timeouts, concurrency limits, provider fallbacks for when the model API is down or slow
• Structured outputs wherever downstream systems need reliable JSON, not free prose
• Versioning of prompts, retrieval configs, and model IDs together, as a release unit
• Human-in-the-loop review paths for edge cases and high-stakes actions

Ask to see how they've done this before. "We call the API from the frontend" is not an architecture.

4. "What do you need from our data?"

Dirty, inconsistent, or ungoverned data produces unreliable outputs no matter how good the model is. An agency that glosses over your data is either naive or avoiding an uncomfortable conversation. Expect questions about where your content lives, who can access it, how it's structured, and how it gets updated — because retrieval over a mess returns a mess, confidently.

Two ownership questions belong in every contract, AI or not: you should own the repo, the cloud accounts, and the CI pipeline in your name — and secrets should live in environment variables, never in prompts. If an engagement ends, everything needed to run and maintain the system stays with you.

5. Red flags that predict a rebuild

Watch for these in proposals and early conversations:

• Acceptance criteria like "model deployed." Demand functional terms instead: accuracy thresholds, latency budgets, throughput. "Deployed" is a milestone; "correct 95% of the time in under 2 seconds" is a contract.
• Fixed-price bids on vague AI scope. AI projects surface complexity as they go. A fixed price against an underscoped proposal leads to change orders or quietly reduced quality.
• Production credentials "just for the pilot." Read-only integrations first; write access is earned, not assumed.
• No escalation path. Every AI feature needs a visible, measured path to a human when the model is uncertain. "Suggest, don't silently overwrite" should be the design principle for anything touching real records.

What good scoping looks like

None of this means AI projects are doomed — it means they're engineering projects, and they should be scoped like engineering projects: functional acceptance criteria, an evaluation harness built from real data, cost modeling before launch, monitoring from day one, and humans in the loop where mistakes matter.

The agencies worth hiring will be relieved you're asking these questions. It means you've been burned before, or you're smart enough to avoid it — either way, you're the kind of client good work gets done for.

That's how we scope AI work at Radsolve: eval-first, cost-modeled, and production-shaped from the first sprint. If you're planning an AI feature and want a second opinion on a proposal — ours or anyone's — get in touch.