AI · Architecture
LLM Features in Production: What Survives Contact With Real Users
An impressive LLM demo takes an afternoon. An LLM feature that behaves well for real users — with real latency, real costs, and real failure modes — is an engineering project. Here's what we've learned shipping AI features in our own products and client work.
The demo-to-production gap
Large language models are the easiest technology in history to demo and among the trickiest to productionize. The demo hides everything that matters operationally: responses that take eight seconds, outputs that don't match the format your code expects, token costs that scale linearly with success, and users who type things no prompt author anticipated.
Our own AI-powered project estimator — live on this site — taught us these lessons firsthand: parsing model output defensively, handling malformed responses, and keeping the experience acceptable when the model is slow are where most of the engineering actually went. The prompt was the easy part.
Patterns that hold up
Across our AI work — LLM-powered features, agent workflows, and integrations built on OpenAI, Anthropic, LangChain, and n8n — the features that survive production share a shape:
- —Narrow scope: one well-defined job (estimate this project, summarize this document) beats an open-ended chat box in reliability, cost, and user trust
- —Server-side calls only: API keys never live in a mobile or web client — calls go through a backend layer (Cloud Functions in Firebase-based stacks) where keys, rate limits, and logging live
- —Structured outputs: constrain the model to a schema and validate before use; never feed raw model text directly into program logic
- —Graceful degradation: the feature must have an answer for 'the model failed or timed out' that isn't a broken screen
- —Cost ceilings: per-user and per-day limits from day one, because a viral moment shouldn't be a billing incident
Where the model should not be
The most common mistake we see is using an LLM where deterministic code belongs. Price calculations, VAT rules, data validation, anything with a legally correct answer — these are not model territory, and 'the AI got it wrong' is not an acceptable incident report for an invoice total. The productive division of labor: the model handles language and judgment at the edges — understanding a messy user description, drafting a summary — while deterministic code owns everything with a right answer.
This is also the honest answer to reliability concerns: an LLM feature designed so that model errors are survivable — reviewed by a human, validated by code, or low-stakes by design — can ship confidently. One where a hallucination reaches a financial document cannot.
Start with a feature, not a strategy
Companies increasingly feel they need an 'AI strategy'. Our advice is smaller and more useful: find one real friction point in your product where language understanding would genuinely help, ship that one feature with the discipline above, and learn from it. One working AI feature in production teaches an organization more than a quarter of strategy documents — and it compounds: the plumbing built for the first feature (server-side calls, validation, cost controls) makes the second one cheap.
Takeaways
- ✓Demos hide latency, cost, and failure modes — production LLM work is mostly engineering around the model, not prompting it.
- ✓Keep API keys server-side, constrain outputs to validated schemas, and design the failure path first.
- ✓Narrow, well-scoped AI features beat open-ended chat in reliability, cost, and user trust.
- ✓Never put the model where deterministic code belongs: anything with a legally or financially correct answer stays code.
- ✓Ship one disciplined AI feature before writing an AI strategy — the plumbing compounds.
Building something where this matters?
This is how we work on every project — mobile, web, and AI. If you want a team that reasons this way about your product, let's talk.
Get a free estimate