Retrieval-Augmented Generation (RAG) is widely used in LLM-based applications to connect models with external knowledge sources. It’s often presented as the default solution for improving factual accuracy and grounding responses.
But in practice, there are mixed opinions on how effective and scalable it really is.
Some argue RAG adds unnecessary complexity and cost, while others see it as essential for any production-grade AI system that relies on dynamic or domain-specific data.
This raises a few discussion points:
Is RAG still the best general approach for grounding LLMs?
Are simpler prompting or fine-tuning strategies sometimes more effective?
Where does RAG start to break down in real-world systems?
Is the industry over-relying on vector search as a solution?
Would be interesting to hear different perspectives from people building with LLMs.
I have been thinking about this while working with LLMs in real projects.
Two engineers can use the same model, same tools, and even similar prompts, but still end up with completely different outputs in terms of product quality, usability, and polish.
It makes me wonder where that difference is actually coming from.
Is it in how they structure context and constraints for the model, or is it more about product thinking before they even start prompting?
Some possible factors I keep noticing:
How clearly the problem is framed before using the model
How much context is provided vs left implicit
The ability to break work into smaller, model-friendly steps
Taste in what to accept, refine, or discard from outputs
If most of the raw capability is the same because the model is the same, then the real gap might not be coding skill anymore, but how well someone can shape inputs and decisions around the model.
Curious how others see this. Is it mostly context design and taste now, or are there still deeper technical differences showing up in the final product?
I’ve been building with OpenAI APIs and there’s a clear gap between “it works in a prototype” and ‘it survives production load.’
Once you move to real users, things like latency variability, rate limits, token cost spikes, and inconsistent outputs become real engineering problems, not edge cases.
Curious how others in this community are handling it in production:
How do you structure retries and avoid duplicate tool/actions?
Do you use caching layers or semantic caching for cost control?
How do you version prompts or manage changes safely across environments?
What fallback strategies do you use when responses are slow or fail?
Any patterns for observability (logging, tracing, evals) that actually work well?
Would love to learn what’s working in real systems, not demos or toy apps.
Every call to OpenAI is wrapped in a sidekiq job with standard exponential back-off.
I’ve worked on a contract recently where I implemented a prompt table with slug and version number. At runtime you retrieve the prompt with the specific slug and the highest number, and if you want to revert to last prompt, you can simply delete the newer row.
Over the past year, AI development has become dramatically faster. With modern APIs and foundation models, teams can build chatbots, copilots, recommendation engines, and automation workflows in days instead of months.
But shipping a demo is not the same as shipping a product.
Many teams discover that the real challenges begin after the first prototype:
Managing API costs as usage grows
Handling hallucinations and unreliable outputs
Maintaining context across conversations
Integrating AI into existing business systems
Monitoring performance and model quality
Ensuring security, compliance, and data privacy
The conversation around AI often focuses on models, benchmarks, and prompts. In practice, successful AI products depend just as much on architecture, observability, evaluation pipelines, and user experience.
For developers building with AI APIs today:
What has been your biggest challenge after the initial prototype stage?
Was it cost, reliability, scaling, user adoption, data quality, or something else entirely?
I’m interested in hearing real-world experiences from teams that have moved beyond demos and into production.
Thanks for sharing this. Sidekiq with exponential backoff makes sense, and I agree that idempotent, re-runnable jobs are the most important part to avoid duplicate tool/actions during retries.
I also like the prompt table approach with slug and versioning. One thing I’d probably do slightly differently is keep the newer rows and switch an active/current version for rollback, so the prompt history stays intact.
I sometimes use Ollama with Codex when I want to be more creative and do not want to worry about usage limits. For those tasks, running a Kimi and GLM model with Ollama feels really useful because I can experiment freely.