Navigating the world of large language models (LLMs) can feel overwhelming. As the tooling ecosystem expands, developers face a critical decision: how to structure an application that relies on an LLM. The fundamental challenge is that LLMs on their own are stateless text predictors. They process a prompt and return a result, but they don’t remember past interactions. To build useful applications—like document question-answering systems or multi-step AI agents—you need an orchestration layer. This layer manages memory, retrieves relevant data, and chains operations together.
When approaching this orchestration, there are three distinct paths: using broad general-purpose frameworks, opting for specialized retrieval-focused tools, or building the system entirely from scratch.
### The Broad Framework Approach
General-purpose frameworks offer an enormous surface area of features. They provide pre-built components for connecting to a wide variety of external tools, databases, and APIs. Their primary strength is breadth; they can connect your LLM to vector databases, search engines, and custom APIs with minimal effort. They also handle complex, multi-step agentic workflows well, allowing an LLM to decide between tools and loop on results without you having to build the decision logic yourself.
These frameworks are ideal for early-stage exploration. If your team is testing assumptions about what an LLM application could do, the abstraction layers accelerate development. However, breadth comes at a cost. The deep abstraction layers make debugging difficult—when something breaks, tracing the failure through multiple wrapped classes is non-trivial. Additionally, the rapid evolution of these frameworks means documentation can lag behind the current API, and upgrading versions occasionally introduces breaking changes, which is a real concern for production stability.
### The Specialized Retrieval Approach
Some tools take a narrower, more opinionated focus. They are purpose-built for retrieval-augmented generation (RAG), the process of pulling relevant documents into an LLM’s context window to answer questions. These frameworks excel at building knowledge base assistants and enterprise search applications. They provide coherent abstractions for data connectors, indexing strategies, and advanced retrieval patterns like re-ranking and hybrid search. Because they focus on a specific problem, their tooling is generally more aligned with document-heavy use cases than their broader counterparts.
The tradeoff is flexibility. If your application needs rich agent behavior, complex tool use, or workflows that don’t center on document retrieval, you may find yourself fighting the framework’s assumptions rather than working with them. The more a framework assumes about your architecture, the harder it becomes to deviate from that expected pattern.
### The Custom Scratch-Built Approach
“Building from scratch” doesn’t mean inventing a new AI model. It means calling the model provider’s base API directly, implementing your own retrieval logic, and constructing only the specific abstractions your application requires. This approach has become increasingly accessible as base APIs and vector database SDKs have matured.
The primary advantage is complete visibility and control. There are no hidden prompt injections from framework templates, no unexpected retry logic, and no opaque abstraction layers intercepting your requests. When something behaves unexpectedly, you know exactly where to look. Building from scratch also forces clarity about what your application actually needs, naturally preventing over-engineering. In production environments, this control pays dividends in debuggability, performance optimization, and adaptability as requirements evolve.
The friction here is upfront investment. You will write more code initially and make more explicit decisions. Furthermore, building a robust retrieval pipeline from scratch requires hands-on knowledge of concepts like embedding models, vector similarity, and chunking strategies—knowledge that frameworks might otherwise paper over.
### How to Choose
The decision comes down to three variables: project stage, team expertise, and application type.
* **If you are in early exploration**, use a framework. They let you test assumptions faster than building from scratch, and the abstraction overhead matters less when the goal is learning.
* **If you are building a retrieval-heavy production system**, a specialized framework is worth serious consideration. Its focused scope means the abstractions are more likely to align with your needs, and its evaluation tooling will serve you well.
* **If you have clear, specific requirements and a team with strong engineering foundations**, building from scratch is often the better long-term investment. The initial overhead pays off in flexibility and maintainability.
Remember that this is not a permanent decision. A common and effective pattern is to prototype with a general-purpose framework to validate your idea, then rebuild the core retrieval and orchestration logic from scratch for production while keeping framework integrations only for peripheral components.
***
### Frequently Asked Questions
**Q: Why are LLMs described as “stateless”?**
A: LLMs treat every prompt as a standalone interaction. They do not natively carry over information from previous conversations. To create memory or context, you must explicitly pass historical data or retrieved documents into each new prompt, which is why orchestration is required.
**Q: What is the biggest hidden cost of using a framework?**
A: Abstraction drag. While frameworks speed up initial development, they create layers of hidden logic. When a bug occurs deep within those layers, tracing the root cause requires understanding the framework’s internal mechanics, which can severely slow down debugging and customization.
**Q: Does building from scratch mean writing an LLM from scratch?**
A: No. It means bypassing intermediary orchestration frameworks and using base APIs directly. You still rely on external model providers and vector databases, but you write the code to connect and manage them yourself rather than relying on pre-built abstractions.
**Q: Is it possible to switch approaches after starting a project?**
A: Yes, though it requires refactoring. It is often more efficient to start with a framework to prove a concept, then intentionally plan to replace the framework’s core components with custom code as the application matures.
***
### Conclusion
The tools available for LLM development—whether broad frameworks, specialized platforms, or custom implementations—are not competing philosophies. They are instruments with different tradeoffs, suited to different contexts. The teams that struggle most are usually those that select a tool before they fully understand their own problem. By focusing on where your application’s complexity lives, how much control you need, and what stage of development you are in, you can make a deliberate choice that sets your project up for success. The specific tools will continue to evolve, but the core principles of clear software architecture remain constant.
Thank you for reading
