Week one wasn't about models — it was about the ground truth. We collected 400 real user questions, wrote reference answers for all of them, and built the eval harness before touching a single API. That harness became the steering wheel for the whole project.
Retrieval quality, not model choice, was the difference. We chunked the client's 12,000-page knowledge base by semantic section, embedded with a tuned model, and hybrid-searched with keyword signals. Retrieval accuracy went from 62% to 91% before we ever switched the LLM.
Guardrails shipped as product, not as an afterthought: source citations on every answer, refusal behavior for off-topic queries, and a 'confidence below threshold' path that hands off to a human. Enterprise buyers don't buy magic; they buy accountability.
The honest numbers: 6 weeks, 3 engineers, 91% retrieval accuracy, 99.2% eval pass rate, and a feature whose usage doubled weekly for a month. The models are commodities. The system around them is the product.
Success is decided in week zero, by feature selection. Pick work that is high value, low ambiguity, and forgiving of a wrong answer. Summarising documents beats drafting contracts. Routing support tickets beats giving legal advice. The teams that fail are usually the ones that picked a feature where hallucination is expensive before they had any guardrails.
Treat prompt engineering as engineering. Version your prompts, review their diffs, and generate structured outputs against a schema instead of parsing prose. The moment a prompt is load-bearing, it deserves the same change control as the code that calls it.
Guardrails come in layers. Validate inputs before they reach the model, constrain outputs to a schema the UI can render, offer graceful fallbacks when confidence is low, and keep a human approval step for anything that spends money or sends email. None of this is optional; it is the difference between a demo and a product.
Latency and cost are product features. Stream tokens so the wait feels shorter, cache aggressively for repeated questions, and route: a small fast model for easy traffic, the big model for the hard twenty percent. Most teams halve their bill without touching quality just by refusing to send easy work to expensive models.
Then launch like it is still an experiment, because it is. A beta cohort, quality monitoring on live traffic, and a weekly review of failure modes with the prompt owner in the room. Six weeks gets you to credible; the teams that win keep iterating in production with feedback loops the demo stage never had.
Enjoyed this? Let's talk about applying it to your product.
