Token 1.12: What is Vector Database's Role in FMOps?
We explain what vector databases are and how they work, explore alternative solutions and provide expert insight on security
What's been hot recently? Vector databases! An essential part of the Foundation models/Large Language Models operations cycle, or FM/LLMOps.
The "Vector Database Market" research report forecasts significant growth in the sector from $1.5 billion in 2023 to $4.3 billion by 2028, a CAGR of 23.3%.
In this Token, we discuss why these databases matter for AI, how they work, and their role in handling complex data; we also explore alternative solutions and provide expert insight on security. Plus you get a curated list of open-sourced vector databases and search libraries. Let's start!
Introduction
Vector databases have their roots in information retrieval concepts and high-dimensional data indexing techniques developed in the late 20th century. The idea began in the 1960s and 1970s with the vector space model, a method for representing documents not just as plain text but as a series of points in a space with many dimensions, almost like plotting dots on a complex graph. This approach was key for understanding how similar different documents were.
Then, in the 1990s, new techniques like R-trees, KD-trees, and Locality-Sensitive Hashing (LSH) came along. These methods were better at organizing and handling intricate data, paving the way for today's vector databases.
Traditional databases, which most people are familiar with, were great for simple, structured data like numbers and text. However, as the world started dealing with more complex types of data, particularly from fields like machine learning and deep learning, a different kind of database was needed. This is where vector embeddings come into play. These are essentially lists of numbers that represent complex data patterns, like a snapshot of the information a computer learns from data. In the 2010s, vector databases were developed specifically to manage these vector embeddings. They make it easier to store, search, and analyze this advanced data, helping computers to understand and work with it more effectively.
Therefore, Vector databases became an essential part of FMs/LLMs operations or FM/LLMOps.