## Building Scalable Vector Search: ANN Algorithms and Cost Trade-offs
Vector search has become a critical piece of AI infrastructure, powering use cases from RAG and semantic search to agentic memory and context layers. With the rise of agentic systems, companies are trying to provide as much context to the agents as possible, which requires vector DB indexes to grow from an initial million or dozens of millions scale to the hundreds of millions or even billions. At this scale, storing indexes and associated data in RAM will cost thousands of dollars per month, and HNSW can become a scalability bottleneck.
In this article, we dive into the details of what actually makes semantic search fast and efficient: approximate nearest neighbor (ANN) algorithms, what different options exist, and their trade-offs.
—
### How Vector Databases Work
Vector databases consist of three main components:
– **Embeddings** – the numeric representation of the corpus
– **Search algorithm and index structure** – defines search quality and speed
– **Storage** – how data is stored (in memory, on disk, etc.), which determines cost and latency at scale
While embeddings are well-covered elsewhere, this article focuses on search algorithms, specifically **Approximate Nearest Neighbor (ANN)** methods.
Exact search guarantees perfect recall but does not scale well for large indexes or high-dimensional data. ANN algorithms, by contrast, trade off a small amount of accuracy for dramatically improved speed and scalability—making them the standard in production systems.
—
### Types of ANN Algorithms
ANN algorithms generally fall into two categories:
– **RAM-based** – optimized for keeping most or all of the index in memory (e.g., HNSW)
– **On-disk** – designed to minimize RAM usage and rely on disk storage (e.g., DiskANN, SPANN)
Although implementations vary, many modern ANN algorithms use graph structures to enable fast, approximate searches.
#### In-Memory ANN: HNSW
HNSW (Hierarchical Navigable Small World) is the most widely used ANN algorithm. It uses a layered graph to enable fast, in-memory searches with extremely low latency. It performs exceptionally well for small to medium indexes that fit comfortably in RAM.
However, once the index outgrows available memory, performance degrades sharply. Moving large HNSW indexes to disk introduces non-sequential I/O and high latency, making it expensive to scale.
**Example vector databases:** Qdrant, Milvus, pgvector, OpenSearch, Weaviate, Redis
#### On-Disk ANN
When index size and cost dominate, on-disk ANN algorithms offer a compelling alternative.
**SPANN** organizes vectors into clusters, keeps centroids in RAM, and stores vectors on disk. This approach reduces random I/O by grouping vectors that belong to the same centroid.
**DiskANN** uses a single-layer graph (Vamana) and stores highly quantized vectors in RAM while keeping full-precision vectors on disk. It minimizes disk seeks through careful routing, making it efficient even at billion-vector scale.
**Example vector databases:** Turbopuffer (SPANN-based), ChromaDB (cloud), Milvus (supports DiskANN), PostgreSQL (via pg_diskann)
—
### Economics: RAM vs Disk Costs
Because vector search at scale is expensive, it helps to compare storage options:
– RAM: ~$5 per GB
– Cloud disk (e.g., EBS): ~$0.08–$0.10 per GB
– Local NVMe SSD: ~$0.20–$0.25 per GB
For 100 million vectors in 1024 dimensions (float32):
– Raw storage: ~1.2 TB
– With 3x replication: ~3.6 TB effective
– After scalar quantization (roughly 4x compression): ~900 GB
Estimated monthly costs:
– Non-quantized in RAM: ~$6,000
– Scalar-quantized in RAM: ~$1,500
– Remote disk: ~$120
– Local disk: ~$300
As indexes grow to hundreds of millions or billions of vectors, these costs widen further, making on-disk approaches significantly more economical.
—
### The Trade-Off
On-disk ANN reduces cost but introduces latency, since data must be read from disk rather than RAM. In warm-cache scenarios, latency might be acceptable; in cold starts or real-time agent workflows, it can be problematic. Real-world latency varies widely based on hardware, caching, and index structure, but disk-based ANN will almost always be slower than in-memory HNSW.
—
### Choosing the Right Algorithm
Choosing between in-memory and on-disk ANN depends on your use case:
– **HNSW** is ideal when latency matters and indexes fit in RAM
– **DiskANN or SPANN** make sense for very large indexes where RAM cost is prohibitive
– **Exact search** is best for small collections or evaluation workloads
Consider vector dimensionality, query volume, budget, and latency requirements when making your decision.
—
### FAQ
**What is vector search, and why is it important for AI?**
Vector search enables AI systems to find semantically similar items by comparing numeric embeddings. It powers retrieval-augmented generation (RAG), semantic search, recommendation systems, and agent memory, making it foundational for modern AI applications.
**What is an approximate nearest neighbor (ANN) algorithm?**
An ANN algorithm finds vectors close to a query vector quickly without checking every entry. It sacrifices exactness for speed and scalability, enabling real-time search over very large vector collections.
**When should I use HNSW vs DiskANN or SPANN?**
Use HNSW when your index fits in RAM and low latency is critical. Choose DiskANN or SPANN when your index is very large, cost sensitivity is high, and slightly higher latency is acceptable.
**How much does vector search cost at scale?**
Cost depends heavily on storage medium. Keeping vectors in RAM is the most expensive, while remote or local disk storage can reduce costs by 10–50x, with trade-offs in retrieval speed.
**Can I mix exact and ANN search in production?**
Yes. Many systems use exact search for small collections or testing and switch to ANN for production-scale indexes to balance accuracy, speed, and cost.
—
### Conclusion
Vector search is a cornerstone of modern AI infrastructure, but scaling it comes with significant cost and performance considerations. By understanding the differences between in-memory and on-disk approximate nearest neighbor algorithms—and the trade-offs between latency, accuracy, and expense—you can make informed decisions that align with your application’s needs. Whether you prioritize speed, scale, or economy, choosing the right ANN strategy is essential for building efficient, production-ready vector search systems.
