Youâve decided to use vector search in your application, product, or business. Youâve researched how and why embeddings and vector search make a problem solvable or can enable new features. Youâve dipped your toes into the hot, emerging area of approximate nearest-neighbor algorithms and vector databases.
Almost immediately upon productionizing vector search applications, you will start to run into very hard and potentially unanticipated difficulties. This blog attempts to arm you with some knowledge of your future, the problems you will face, and questions you may not know yet that you need to ask.
1. Vector search â vector database
Vector search and all the associated clever algorithms are the central intelligence of any system trying to leverage vectors. However, all of the associated infrastructure to make it maximally useful and production ready is enormous and very, very easy to underestimate.
To put this as strongly as I can:Â a production-ready vector database will solve many, many more âdatabaseâ problems than âvectorâ problems. By no means is vector search, itself, an âeasyâ problem (and we will cover many of the hard sub-problems below), but the mountain of traditional database problems that a vector database needs to solve certainly remain the âhard part.â
Databases solve a host of very real and very well studied problems from atomicity and transactions, consistency, performance and query optimization, durability, backups, access control, multi-tenancy, scaling and sharding and much more. Vector databases will require answers in all of these dimensions for any product, business or enterprise.
Be very wary of home-rolled âvector-search infra.â Itâs not that hard to download a state-of-the-art vector search library and start approximate-nearest-neighboring your way towards an interesting prototype. Continuing down this path, however, is a path to accidently reinventing your own database. Thatâs probably a choice you want to make consciously.
2. Incremental indexing of vectors
Due to the nature of the most modern ANN vector search algorithms, incrementally updating a vector index is a massive challenge. This is a well known âhard problemâ. The issue here is that these indexes are carefully organized for fast lookups and any attempt to incrementally update them with new vectors will rapidly deteriorate the fast lookup properties. As such, in order to maintain fast lookups as vectors are added, these indexes need to be periodically rebuilt from scratch.
Any application hoping to stream new vectors continuously, with requirements that both the vectors show up in the index quickly and the queries remain fast, will need serious support for the âincremental indexingâ problem. This is a very crucial area for you to understand about your database and a good place to ask a number of hard questions.
There are many potential approaches that a database might take to help solve this problem for you. A proper survey of these approaches would fill many blog posts of this size. Itâs important to understand some of the technical details of your databaseâs approach because it may have unexpected tradeoffs or consequences in your application. For example, if a database chooses to do a full-reindex with some frequency, it may cause high CPU load and therefore periodically affect query latencies.
You should understand your applications need for incremental indexing, and the capabilities of the system youâre relying on to serve you.
3. Data latency for both vectors and metadata
Every application should understand its need and tolerance for data latency. Vector-based indexes have, at least by other database standards, relatively high indexing costs. There is a significant tradeoff between cost and data latency.
How long after you âcreateâ a vector do you need it to be searchable in your index? If itâs soon, vector latency is a major design point in these systems.
The same applies to the metadata of your system. As a general rule, mutating metadata is fairly common (e.g. change whether a user is online or not), and so itâs typically very important that metadata filtered queries rapidly react to updates to metadata. Taking the above example, itâs not useful if your vector search returns a query for someone who has recently gone offline!
If you need to stream vectors continuously to the system, or update the metadata of those vectors continuously, you will require a different underlying database architecture than if itâs acceptable for your use case to e.g. rebuild the full index every evening to be used the next day.
4. Metadata filtering
I will strongly state this point: I think in almost all circumstances, the product experience will be better if the underlying vector search infrastructure can be augmented by metadata filtering (or hybrid search).
Show me all the restaurants I might like (a vector search) that are located within 10 miles and are low to medium priced (metadata filter).
The second part of this query is a traditional sql-like WHERE clause intersected with, in the first part, a vector search result. Because of the nature of these large, relatively static, relatively monolithic vector indexes, itâs very difficult to do joint vector + metadata search efficiently. This is another of the well known âhard problemsâ that vector databases need to address on your behalf.
There are many technical approaches that databases might take to solve this problem for you. You can âpre-filterâ which means to apply the filter first, and then do a vector lookup. This approach suffers from not being able to effectively leverage the pre-built vector index. You can âpost-filterâ the results after youâve done a full vector search. This works great unless your filter is very selective, in which case, you spend huge amounts of time finding vectors you later toss out because they donât meet the specified criteria. Sometimes, as is the case in Rockset, you can do âsingle-stageâ filtering which is to attempt to merge the metadata filtering stage with the vector lookup stage in a way that preserves the best of both worlds.
If you believe that metadata filtering will be critical to your application (and I posit above that it will almost always be), the metadata filtering tradeoffs and functionality will become something you want to examine very carefully.
5. Metadata query language
If Iâm right, and metadata filtering is crucial to the application you are building, congratulations, you have yet another problem. You need a way to specify filters over this metadata. This is a query language.
Coming from a database angle, and as this is a Rockset blog, you can probably expect where I am going with this. SQL is the industry standard way to express these kinds of statements. âMetadata filtersâ in vector language is simply âthe WHERE clauseâ to a traditional database. It has the advantage of also being relatively easy to port between different systems.
Furthermore, these filters are queries, and queries can be optimized. The sophistication of the query optimizer can have a huge impact on the performance of your queries. For example, sophisticated optimizers will try to apply the most selective of the metadata filters first because this will minimize the work later stages of the filtering require, resulting in a large performance win.
If you plan on writing non-trivial applications using vector search and metadata filters, itâs important to understand and be comfortable with the query-language, both ergonomics and implementation, you are signing up to use, write, and maintain.
6. Vector lifecycle management
Alright, youâve made it this far. Youâve got a vector database that has all the right database fundamentals you require, has the right incremental indexing strategy for your use case, has a good story around your metadata filtering needs, and will keep its index up-to-date with latencies you can tolerate. Awesome.
Your ML team (or maybe OpenAI) comes out with a new version of their embedding model. You have a gigantic database filled with old vectors that now need to be updated. Now what? Where are you going to run this large batch-ML job? How are you going to store the intermediate results? How are you going to do the switch over to the new version? How do you plan to do this in a way that doesnât affect your production workload?
Ask the Hard Questions
Vector search is a rapidly emerging area, and weâre seeing a lot of users starting to bring applications to production. My goal for this post was to arm you with some of the crucial hard questions you might not yet know to ask. And youâll benefit greatly from having them answered sooner rather than later.
In this post what I didnât cover was how Rockset has and is working to solve all of these problems and why some of our solutions to these are ground-breaking and better than most other attempts at the state of the art. Covering that would require many blog posts of this size, which is, I think, precisely what weâll do. Stay tuned for more.